Technologies for E-Learning and Digital Entertainment: Second International Conference, Edutainment 2007, Hong Kong, China, June 11-13, 2007, Proceedings (Lecture Notes in Computer Science, 4469) 3540730109, 9783540730101

Citation preview

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Moshe Y. Vardi Rice University, Houston, TX, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

4469

Kin-chuen Hui Zhigeng Pan Ronald Chi-kit Chung Charlie C.L. Wang Xiaogang Jin Stefan Göbel Eric C.-L. Li (Eds.)

Technologies for E-Learning and Digital Entertainment Second International Conference, Edutainment 2007 Hong Kong, China, June 11-13, 2007 Proceedings

13

Volume Editors Kin-chuen Hui Ronald Chi-kit Chung Charlie C.L. Wang The Chinese University of Hong Kong, China E-mail: {kchui,rchung,cwang}@mae.cuhk.edu.hk Zhigeng Pan Xiaogang Jin Zhejiang University, China E-mail: {zgpan,jin}@cad.zju.edu.cn Stefan Göbel ZGDV e.V. - Computer Graphics Center Darmstadt, Germany E-mail: [email protected] Eric C.-L. Li The City University of Hong Kong, China E-mail: [email protected]

Library of Congress Control Number: 2007928440 CR Subject Classification (1998): K.3.1-2, I.2.1, H.5, H.3, I.3 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/ Web and HCI ISSN ISBN-10 ISBN-13

0302-9743 3-540-73010-9 Springer Berlin Heidelberg New York 978-3-540-73010-1 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com © Springer-Verlag Berlin Heidelberg 2007 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12075249 06/3180 543210

Preface

To learn while being entertained is always an effective means in education. With the advance in technologies, in particular graphics, multimedia, and virtual reality technologies, this has evolved into the specific area of “Edutainment”. The second international conference on edutainment, Edutainment 2007, aimed to provide a forum for practitioners and researchers in the field to share their experiences and findings in this fast growing area. Following the success of Edutainment 2006, which was held in Hangzhou, China, Edutainment 2007 was held during June 11–13, 2007 in Hong Kong. This year, we received 393 submissions from 29 different countries and areas, including China (including Taiwan), USA, UK, Germany, Italy, France, Australia, Canada, Switzerland, Korea, Japan, Singapore and Malaysia. A total of 90 papers were selected, after peer review, for this volume. Topics of these papers fall into six different areas ranging from fundamental issues in geometry and imaging to virtual reality systems and their applications in entertainment and education. These topics include Virtual Reality in Games and Education, Virtual Characters in Games and Education, E-learning Platforms and Tools, Geometry in Games and Virtual Reality, Vision, Imaging and Video Technology, and Collaborative and Distributed Environments. We are grateful to the International Program Committee and the reviewers for their effort to get all the papers reviewed in a short period of time. We would also like to thank everyone who contributed to organizing the conference.

April 2007

Kin-chuen Hui Zhigeng Pan Ronald Chi-kit Chung Charlie C.L. Wang Xiaogang Jin Stefan Goebel Eric C.-L. Li

Organization

Acknowledgements and Sponsoring Institutions The international conference series on E-Learning and Games (called Edutainment) was initiated by the Chinese University of Hong Kong. The success of Edutainment 2007 was due to the financial and practical support of various institutions.

Sponsors: •

VT Committee of the China Society of Image and Graphics

Co-sponsors: • Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong • The International Journal of Virtual Reality • Microsoft-CUHK Joint Laboratory for Human-centric Computing and Interface Technologies • KC Wong Education Foundation • Digital Entertainment Society of China Simulation Association • Eurographics We would like to thank all of them for offering the opportunity to organize Edutainment 2007 in a way that provided a diversified scientific and social program. Especially, we would like to thank all members of the International Program Committee and Organizing Committee for their great job in defining conference topics, reviewing the large number of submitted papers, and managing to put all the materials together for this great event.

Committee Listings: Honorary Chairs José L. Encarnação, Technical University of Darmstadt, Germany Yeung Yam, The Chinese University of Hong Kong, China Ruth Aylett, Heriot-Watt University, UK Conference Chairs Kin-chuen Hui, The Chinese University of Hong Kong, China Zhigeng Pan, Zhejiang University, China L. Miguel Encarnação, The IMEDIA Academy, USA

VIII

Organization

Program Chairs (E-Learning Track) Ronald Chi-kit Chung, The Chinese University of Hong Kong, China Stefan Goebel, ZGDV, Germany (Game and Animation Track) Charlie C.L. Wang, The Chinese University of Hong Kong, China Xiaogang Jin, Zhejiang University, China Publicity Co-chairs Jim X. Chen, George Mason University, USA Maiga Chang, Chung-Yuan Christian University, China Gerry J. Kim, POSTECH, Korea Abdennour El Rhalibi, Liverpool John Moores University, UK Workshop Co-chairs Holger Diener, Fraunhofer IGD, Germany Fong-Lok Lee, The Chinese University of Hong Kong, China Ryohei Nakatsu, Kwansei Gakuin University, Japan Tutorial Co-chairs Woontack Woo, GIST, Korea Lars Kjelldahl, KTH, Sweden Nuno Correia, New University of Lisbon, Portugal Publication Co-chairs Li Li, Hangzhou Dianzi University, China Eric C.-L. Li, The City University of Hong Kong, China Organizing Co-chairs K.M. Yu, The Hong Kong Polytechnic University, China Yanghseng Wang, Institute of Automation, Chinese Academy of Sciences, China Zhiyang Yao, The Chinese University of Hong Kong, China Yi Li, Nanjing Normal University, China Executive Committee Members Ruth Aylett, Heriot-Watt University, UK Jim X. Chen, George Mason University, USA Stefan Goebel, ZGDV, Germany Kin-chuen Hui, The Chinese University of Hong Kong, China Xiaogang Jin, Zhejiang University, China Zhigeng Pan, Zhejiang University, China Charlie C.L. Wang, The Chinese University of Hong Kong, China

Organization

International Program Committee Isabel Machado Alexandre, Portugal Jasir Alherbish, Saudi Arabia Elizabeth Andre, Germany Ruth Aylett, UK Mark Billinghurst, New Zealand Gloria Brown-Simmons, USA Daniel Burgos, Netherlands Yiyu Cai, Singapore Tak-Wai Chan, Taiwan Maiga Chang, Taiwan Yam San Chee, Singapore Jim X.Chen, USA Adrian David Cheok, Singapore Carola Conle, Canada Nuno Correia, Portugal Holger Diener, Germany Stéphane Donikian, France Jayfus T. Doswell, USA Miguel Encarnação, USA Bernd Fröhlich, Germany Lisa Gjedde, Denmark Gernot Goebbels, Germany Martin Goebel, Germany Stefan Goebel, Germany Lynne Hall, UK Kin-chuen Hui, Hong Kong Masahiko Inami, Japan Woochun Jun, Korea Hirokazu Kato, Japan Gerry J. Kim, Korea Lars Kjelldahl, Sweden Christian Knöpfle, Germany Jarmo Laaksolahti, Sweden Cecília Sikné Lányi, Hungary Rynson Lau, Hong Kong Fong-Lok Lee, Hong Kong Jimmy H.M. Lee, Hong Kong Jong Weon Lee, Korea Eric C.-L. Li, Hong Kong Sandy Louchart, UK Maic Masuch, Germany Wolfgang Müller-Wittig, Singapore Ryohei Nakatsu, Japan Stéphane Natkin, France Malcolm Padmore, UK

IX

X

Organization

Ana Paiva, Portgual Zhigeng Pan, China Edmond Prakash, Singapore Marc Prensky, USA Shengfeng Qin, UK Abdennour El Rhalibi, UK Paul Richard, France Albert "Skip" Rizzo, USA Judy Robertson, UK I-Fan Shen, China Ulrike Spierling, Germany Kurt Squire, USA Gerard Subsol, France Mohd Shahrizal Sunar, Malaysia Naoko Tosa, Japan Ming Hsin Tsai, Taiwan Jorge Posada Velásquez, Spain Frederic Vexo, Switzerland Yangsheng Wang, China Kevin Kok-Wai Wong, Australia Woontack Woo, Korea Zhiyang Yao, Hong Kong Kai Ming Yu, Hong Kong Gino Yu, Hong Kong Pao-Ta Yu, Taiwan Shaochun Zhong, China

Table of Contents

Keynote Speeches (Abstracts) An Interactive Folktale System and an Authoring System That Supports Its Realization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ryohei Nakatsu, Yurika Nagai, Kozi Miyazaki, Yurika Nagai, and Takenori Wama Edutainment and Serious Games – Games Move into Professional Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jos´e L. Encarna¸c˜ ao

1

2

Virtual and Augmented Reality in Game and Education An Efficient Method for Real-Time Ocean Simulation . . . . . . . . . . . . . . . . . Haogang Chen, Qicheng Li, Guoping Wang, Feng Zhou, Xiaohui Tang, and Kun Yang

3

The Dynamic Textures for Water Synthesis Based on Statistical Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Qian Zhang and TaegKeun Whangbo

12

Constructing a Convenient and Efficient System for Making Virtual Human Accentuated by Customized 3D Face and Body . . . . . . . . . . . . . . . Seongah Chin, Jongkyeong Park, and Moonwon Choo

22

The Design and Implementation of VBS . . . . . . . . . . . . . . . . . . . . . . . . . . . . Liang Wang, Yue Qi, and Xukun Shen

34

View-Dependent Hierarchical Foliage Simplification . . . . . . . . . . . . . . . . . . Qingqiong Deng, Xiaopeng Zhang, and Marc Jaeger

44

A Novel Optical See-Through Head-Mounted Display with Occlusion and Intensity Matching Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ya Zhou, Jin-Tao Ma, Qun Hao, Hong Wang, and Xian-Peng Liu

56

An Immersive Game Using a New Interface: The Well–Tep . . . . . . . . . . . . Donggang Jung, Chulsang Jang, Junhyung Oh, Junggoo Kang, Ingul Oh, and Keechul Jung Teaching Structured Development of Virtual Reality Systems Using P-VoT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jinseok Seo and Gerard J. Kim

63

69

XII

Table of Contents

VR@Home: An Immersive Contents Creation System for 3D User-Generated Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wonwoo Lee, Chinta Ram Babu, Jongweon Lee, and Woontack Woo A 3D Virtual Learning Environment to Foster Communication for Long Term Ill Children . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fabian Di Fiore, Pieter Jorissen, Gert Vansichem, and Frank Van Reeth Interactive Storytelling and Gaming Environments for Museums: The Interactive Storytelling Exhibition Project . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Danks, Marc Goodchild, Karina Rodriguez-Echavarria, David B. Arnold, and Richard Griffiths

81

92

104

Virtual Environments for Training: From Individual Learning to Collaboration with Humanoids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . St´ephanie Gerbaud, Nicolas Mollet, and Bruno Arnaldi

116

Balancing Pedagogy, Game and Reality Components Within a Unique Serious Game for Training Levee Inspection . . . . . . . . . . . . . . . . . . . . . . . . . Casper Harteveld, Rui Guimar˜ aes, Igor Mayer, and Rafael Bidarra

128

An Operational VR Platform for Building Virtual Training Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nicolas Mollet, St´ephanie Gerbaud, and Bruno Arnaldi

140

Usability Test of Immersion for Augmented Reality Based Product Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Taejin Ha, Yoonje Chang, and Woontack Woo

152

The Development of a Virtual Cycling Simulator . . . . . . . . . . . . . . . . . . . . . Yuk-Ming Tang, Mathew Ho-Cheong Tsoi, Daniel Tik-Pui Fong, Pauline Po-Yee Lui, Kin-Chuen Hui, and Kai-Ming Chan

162

Virtual Characters in Games and Education Visualization Learning for Visually Impaired People . . . . . . . . . . . . . . . . . . Patrick Salamin, Daniel Thalmann, and Fr´ed´eric Vexo

171

Interactive Composition of 3D Faces for Virtual Characters . . . . . . . . . . . Haeyoung Lee and Sungyeol Lee

182

Real-Time Face Pose Tracking and Facial Expression Synthesizing for the Animation of 3D Avatar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Junchul Chun, Ohryun Kwon, Kyongpil Min, and Peom Park

191

Continuous Motion Graph for Crowd Simulation . . . . . . . . . . . . . . . . . . . . . Mankyu Sung

202

Table of Contents

XIII

LoD-Based Locomotion Engine for Game Characters . . . . . . . . . . . . . . . . . Zuoyan Lin and Zhigeng Pan

214

Using Virtual Characters as TV Presenters . . . . . . . . . . . . . . . . . . . . . . . . . . David Oyarzun, Maider Lehr, Amalia Ortiz, Maria del Puy Carretero, Alejandro Ugarte, Karmelo Vivanco, and Alejandro Garc´ıa-Alonso

225

Grid Environment for Fabricating Custom Artificial Human Bone . . . . . . Yefei Li and XiangHong Xu

237

Method of Motion Data Processing Based on Manifold Learning . . . . . . . Fengxia Li, Tianyu Huang, and Lijie Li

248

A Hybrid of Plot-Based and Character-Based Interactive Storytelling . . . Yundong Cai, Chunyan Miao, Ah-Hwee Tan, and Zhiqi Shen

260

E-Learning Platforms and Tools A Context Model Based on Multi-agent in U-Learning Environment . . . . SooJin Jun, SeonKwan Han, SooHwan Kim, HyeonCheol Kim, and WonGyu Lee

274

Hierarchical Petri-Nets Model for the Design of E-Learning System . . . . . Feng He and Jiajin Le

283

The Design of Coopetitive Pedagogy in an Integrated Web-Based Learning Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fong-Ling Fu, Ya-Ling Wu, and Hsi-Chuan Ho

293

An Observational Study of Children Interacting with an Augmented Story Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andreas D¨ unser and Eva Hornecker

305

Adaptive QoS for Educational User Created Content(UCC) . . . . . . . . . . . Hee-Seop Han, Seon-Kwan Han, Soo-Hwan Kim, and Hyeoncheol Kim

316

ISRST: An Interest Based Storytelling Model Using Rhetorical Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arturo Nakasone and Mitsuru Ishizuka

324

Mining Learners’ Behavior in Accessing Web-Based Interface . . . . . . . . . . Man Wai Lee, Sherry Y. Chen, and Xiaohui Liu

336

Approach Comparison on Context-Aware Computing with Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zongjie Wang, Degan Zhang, Aili Li, Xiaobin Huang, and Hongtao Peng

347

XIV

Table of Contents

Implementation of a Web-Based Hybrid Educational System for Enhancing Learning Efficiency of Engineering Experiments . . . . . . . . . . . . Dongsik Kim, Kwansun Choi, Sunheum Lee, Changwan Jeon, and Jiyoon Yoo ImaginationTOOLST M : Made to Play Music . . . . . . . . . . . . . . . . . . . . . . . . Francesca Bertacchini, Eleonora Bilotta, Lorella Gabriele, Vincenzo Mazzeo, Pietro Pantano, Costantino Rizzuti, and Stefano Vena

357

369

PIGP: A Pen-Based Intelligent Dynamic Lecture System for Geometry Teaching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuanyuan Liu, Qiang Lin, and Guozhong Dai

381

A Conceptual Framework of Affective Context-Aware Interactive Learning Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Horace H. S. Ip and Belton Kwong

391

Ontology Based Situation Analysis and Encouragement in E-Learning System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yongzhao Zhan, Liting Xu, and Qirong Mao

401

A Standardized Learning Resources Retrieval System Based on Ontology Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xia Li, Jianlin Zhang, and Tao Huang

411

An Integrated Virtual Environment for Active e-Learning in Theory of Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mohamed Hamada

422

An Interactive Bomberman Game-Based Teaching/ Learning Tool for Introductory C Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wai-Tak Wong and Yu-Min Chou

433

A Study of Virtual Manipulative and Whiteboard System for Improving Multi-presentation Transformation of Geometry Problem Solving . . . . . . Wu-Yuin Hwang, Jia-Han Su, Jian-Jie Dung, and Yi-Shien Su

445

Supporting Multi-device for Ubiquitous Learning . . . . . . . . . . . . . . . . . . . . . Byoungchol Chang, Mokhwa Lim, Dall-ho Ham, Yoonsoo Lee, Sooyong Kang, and Jaehyuk Cha

457

A Medical Aided System for Evaluating Crura Rehabilitation . . . . . . . . . . Fangyu Li, Yefei Li, Shouqian Sun, Zhanxun Dong, and Xianghong Xu

468

Using Metaphors in Game-Based Education . . . . . . . . . . . . . . . . . . . . . . . . . Pedro P. G´ omez-Mart´ın, Marco A. G´ omez-Mart´ın, Pablo Palmier Campos, and Pedro A. Gonz´ alez-Calero

477

Table of Contents

XV

UML in Action: Integrating Formal Methods in Industrial Design Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jun Hu, Philip Ross, Loe Feijs, and Yuechen Qian

489

An Agent-Based Adaptable and Configurable Tutoring Module for Intelligent Virtual Environments for Training . . . . . . . . . . . . . . . . . . . . . . . . Leticia S´ anchez and Ricardo Imbert

499

A Multimodal 3D Storytelling System for Chinese Children . . . . . . . . . . . Danli Wang, Jie Zhang, Jie Li, Guozhong Dai, and Qiang Lin

511

Real World Edutainment Focusing on Human-Human Interaction . . . . . . Hiroyuki Mitsuhara, Yasumoto Hirakawa, Kazuhide Kanenishi, and Yoneo Yano

527

Research on Personalized Community E-Learning Recommendation Service System by Using Improved Adaptive Filtering Algorithm . . . . . . . Qi Luo and Zhigeng Pan

539

Efficient Embedding of Information and Knowledge into CSCL Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Santi Caball´e, Thanasis Daradoumis, and Fatos Xhafa

548

Geometry in Games and Virtual Reality Progressive Streaming of Irregular Meshes Using Geometry Images . . . . . Bin Sheng and Enhua Wu

560

A QoS Controller for Adaptive Streaming of 3D Triangular Scenes . . . . . ZhiQuan Cheng and ShiYao Jin

572

Geometry-Based Muscle Forces and Inverse Dynamics for Animation . . . Jihun Park, Sunghun Park, and Yoohun Won

584

Connectivity Compression for Stripified Triangle Meshes . . . . . . . . . . . . . . Hao Huang and Fan Zhou

596

Interpolating Solid Orientations with a C 2 -Continuous B-Spline Quaternion Curve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wenbing Ge, Zhangjin Huang, and Guoping Wang

606

Texture-Meshes in Digital Museum: Octree-Initialized Progressive Transmission with Feature Preservation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xuewei Zhao, Xukun Shen, and Yue Qi

616

A Robust Watermarking Based on Feature Points . . . . . . . . . . . . . . . . . . . . Xiaoqing Feng, Li Li, and Zhigen Pan

628

Surface Animation Based on 2D Editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jian Gao, Caiming Zhang, and Zhiquan Feng

638

XVI

Table of Contents

Kinematic Inbetweening for Motion Animation . . . . . . . . . . . . . . . . . . . . . . Dan Li, Hai-Yin Xu, Rui Huang, and Jian Wang

647

Real-Time Visualization of Tire Tracks in Dynamic Terrain with LOD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Guojun Chen, Jing Zhang, Xiaoli Xu, and Yuan Yin

655

GPU-Based Real-Time Deformation with Normal Reconstruction . . . . . . Yinghui Che, Jing Wang, and Xiaohui Liang

667

Interpolation to Data Points in Plane with Cubic Polynomial Precision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Huijian Han, Hui Liu, and Xiuhua Ji

677

ISMAS: An Interactive Sketch Modeling and Animation System . . . . . . . Xin Li, Guangzheng Fei, and Yongfeng Zhu

687

Point Sampled Surface Reconstruction Based on Local Geometry . . . . . . Lanfang Miao and Qunsheng Peng

701

An Integer Incremental AOI Algorithm for Progressive Downloading of Large Scale VRML Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jinyuan Jia, Ping Wang, Sen Wang, and Yuying Wang

711

Vision, Imaging and Video Technology Quantization of Colors Using Median of Pixels for Color Correlogram . . . Jongan Park, Youngeun An, and Pankoo Kim

723

Feature-Based Texture Design Using Deformation Techniques . . . . . . . . . . Jianbing Shen, Xiaogang Jin, and Hanli Zhao

730

Novel Method for Computer Create Van Gogh’s Oil Style Painting . . . . . Liang Xiao, Zhihui Wei, and Huizhong Wu

740

A New Approach Supporting Perception Guiding and Multi-objective Decision Making for Behavior Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xiaojuan Ban, Daoping Jiang, Shurong Ning, Dongmei Ai, and Yixin Yin

751

3D Visual Feedback in Learning of Piano Posture . . . . . . . . . . . . . . . . . . . . Javier Mora, Won-Sook Lee, and Gilles Comeau

763

An Automated Registration Method for Range Images . . . . . . . . . . . . . . . . Shen Yang, Xukun Shen, Yue Qi, and Qinping Zhao

772

Optimizing the Oriental Painting Shader for 3D Online-Game . . . . . . . . . Sung-Soo Kim, Hyuna Jang, and Won-Hyung Lee

784

Table of Contents

XVII

NBR: A Content-Based News Video Browsing and Retrieval System . . . . Huayong Liu, Tingting He, and Hui Zhang

793

An Interactive Snowboard Game in Virtual Environment . . . . . . . . . . . . . Gye-Beom Jeon, Won Cheol Kim, Amarjagal Enkhsaruul, Eun-Ju Kim, Jong-Wan Lee, Sun-Jeong Kim, and Chang Geun Song

801

Relevance Feedback and Latent Semantic Index Based Cultural Relic Image Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Na Wei, Ming-Quan Zhou, and Guo-Hua Geng

812

Collaborative and Distributed Environments Controllable Multi-party Audio/Video Collaboration Based on Multicast over CERNET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xuan Zhang, Xing Li, and Chongrong Li

819

A Synchronous Scaffolding Environment for Collaborative Technical Writing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shiou-Wen Yeh, Jia-Jiunn Lo, Jeng-Jia Huang, and Zhen-Yu Fan

829

Data Transmission for Haptic Collaboration in Virtual Environments . . . Yonghee You, Mee Young Sung, and Kyungkoo Jun

841

Message Synchronization Mechanisms Based on Waiting Period for Massively Multi-player Online Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Doowon Paik, Chung-Ha Yun, and Jooyeon Hwang

853

An Efficient Distributed MMOG Server Using 2Layer-Cell Method . . . . . Su-Min Jang and Jae-Soo Yoo

864

Filling the Gap Between Middleware and User Interactions in a Discovery Learning Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ma Antonia Mart´ınez-Carreras, Eduardo Mart´ınez-Graci´ a, and Antonio F. G´ omez-Skarmeta

876

Augmented Galaga on Mobile Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anjin Park and Keechul Jung

888

A Unified Shader Based on the OpenGL ES 2.0 for 3D Mobile Game Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tae-Young Kim, Jongho Kim, and Hyunmin Hur

898

LoI-Based Flow Control on Low-Bandwidth Federates . . . . . . . . . . . . . . . . Zhong Zhou and Qinping Zhao

904

Dynamic Game Level Generation Using On-Line Learning . . . . . . . . . . . . . Jongyeol Yang, Seungki Min, Chee-Onn Wong, Jongin Kim, and Keechul Jung

916

XVIII

Table of Contents

Development of Online Game-Based Learning for TRIZ . . . . . . . . . . . . . . . Wai Lap Leung and Kai Ming Yu A Distributed System for Augmented Reality Experiences in Science Centers and Museums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Wittk¨ amper, Anne-Kathrin Braun, Iris Herbst, and Jan Herling Federate Job Mapping Strategy in Grid-Based Virtual Wargame Collaborative Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jing Zhang and Hai Huang A Platform for Motivating Collaborative Learning Using Participatory Simulation Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gustavo Zurita, Nelson Baloian, Felipe Baytelman, and Antonio Farias Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

925

936

947

957

971

An Interactive Folktale System and an Authoring System That Supports Its Realization Ryohei Nakatsu, Yurika Nagai, Kozi Miyazaki, Yurika Nagai, and Takenori Wama Kwansei Gakuin University, School of Science and Technology 2-1 Gakuen, Sanda 669-1337, Japan

Abstract. Along with the advancement of three dimensional computer graphics technology and interaction technology, new types of education system as well as entertainment system are expected to emerge. In this paper as one example of such a system, we propose “Interactive Folktale System,” This system makes it possible for people, especially children, to enjoy folktales in various ways. They can appreciate the regeneration of original tales and its variations. Also they can interact with the story, by communicating with the characters in the story or by playing a role of one of the characters. Moreover, they can easily create a new folktale by changing the original story or by describing a script for the new story. Also the architecture of an authoring system that supports that realization of the interactive folktale system is described.

K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, p. 1, 2007. © Springer-Verlag Berlin Heidelberg 2007

Edutainment and Serious Games – Games Move into Professional Applications José L. Encarnação Chairman, INI-GraphicsNet Foundation Professor, Technische Universität Darmstadt Rundeturmstrasse 10 64283 Darmstadt [email protected] http://www.inigraphicsnet-stiftung.de/

Abstract. The talk will provide an overview of the symbiosis of Computer Graphics, Interactive Digital Storytelling and Entertainment technology as basis for Storytelling based Edutainment Applications and Serious Games for business and market oriented applications. Since human age, stories have been told and used as media to document history and transmit personal experiences. In the digital age, Computer Graphics is used to build interactive virtual worlds for training and simulation, game and storytelling technologies are used to motivate learners and provide playful, story-driven information and communication environments. This is now starting to be used more and more for professional application, that means "using games on and for the job"! Within the talk, underlying methods, concepts and technologies will be illustrated by current and recent projects of the INI-GraphicsNet in that domain: The "Virtuelles Autohaus" and "Virtual IGD Rostock" as examples for playful interactive virtual environments for training and simulation, "Virtual Human" and the "Servingo Story Generator" as examples for Storytelling based edutainment applications and "INSCAPE", "U-CREATE" addressing the authoring process for the creation of these edutainment scenarios and serious games for a given application domain.

K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, p. 2, 2007. © Springer-Verlag Berlin Heidelberg 2007

An Efficient Method for Real-Time Ocean Simulation Haogang Chen, Qicheng Li, Guoping Wang, Feng Zhou, Xiaohui Tang, and Kun Yang Dep. of Computer Science & Technology, Peking University, Beijing, China [email protected] {lqc,wgp,zf,txh,yk}@graphics.pku.edu.cn

Abstract. Ocean stimulation is important for computer animation, games, and other virtual reality applications. In this paper, a real-time method for simulating integrated ocean environment, including sky sphere, atmospheric system and ocean wave model, is proposed. By this method, the effect of cloud floating in sky sphere is obtained through texture perturbation, while the atmospheric system realizes the air scattering and absorbing effect. The ocean wave model constructs ocean surface mesh by Sine wave and realizes the bumping effect of ocean surface through normal disturbance of bump map method. For the lightening computation of ocean wave, the incident ray can be obtained by sampling the sky sphere and the reflecting light ray can then be calculated through the principle of mirror reflection. The proposed algorithm can be easily accelerated by GPU hardware. Experiments show that the method is easy to implement, and is effective to render ocean at high frame rate

1 Introduction The modeling and rendering of water has been a traditional problem in computer graphics during the last two decades. It has always been a topic of many applications, such as games and animation. In particular, ocean water is difficult to be simulated due to the shape and combination of multiple waves, in addition to the reflection of the clouds and the sun in the sky. Recently some efforts have been paid on the concept and the different levels of realism (physical realism, photo realism and functional realism) within the framework of computer graphics [1] and fluid visualization [2]. In this paper, our focus is the techniques for simulating ocean, but not those intended for fluid visualization. Therefore, a functional realism for ocean simulation, being more effective and easy than physical realism, is utilized in our work. That is to say, we are simply concerned with generating a visually convincing appearance of motion whether or not this motion actually follows the laws of physics. Of course, this computer graphics approach sometimes yielded unrealistic effects, making it unsuitable for real experiments. However, it is still quite useful in computer games and entertainment since very beautiful images and animations can be produced. As Fournier and Reeves said [3]: We do not expect any “physical” answer from the model. In fact, we do not ask any question except “does it look like the real thing”. K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 3–11, 2007. © Springer-Verlag Berlin Heidelberg 2007

4

H. Chen et al.

The paper is organized as following. In Section 2 some related work is reviewed. Then, in Section 3, we present the theory and implementation of our ocean environment model. In Section 4 and 5, experimental results are introduced, followed by conclusions.

2 Related Work Here we do not review those work dedicated to running water or to rivers such as [4, 5, 6, 7, 8], which focus on other kinds of water surfaces and are not adapted to the simulation of ocean waves. In the middle of 1980s Fournier and Reeves [3] proposed an ocean wave model based on the Gerstner–Rankine model. Roughly speaking, this model establishes that particles of water describe circular or elliptical stationary orbits. Peachey [9] proposed a similar idea, with fewer refinements (e.g., no trochoids). More recently, several models [10, 11] have proposed more precise ways to solve the propagation (wave tracing). Note that noise is generally used in all the models above in order to avoid the visual regularity due to the fact that only one or two wave trains are simulated. The spectral approaches are first introduced in CG by Mastin et al. [12]. The basic idea is to produce a height field having the same spectrum as the ocean surface. This can be done by filtering a white noise with Pierson-Moskowitz’s or Hasselmann’s filter, and then calculating its Fast Fourier Transform (FFT). The main benefit of this approach is that many different waves are simultaneously simulated, with visually pleasing results. However, animating the resulting ocean surface remains challenging. Tessendorf [13] shows that dispersive propagation can be managed in the frequency domain and that the resulting field can be modified to yield trochoid waves. A positive property of FFTs is the cyclicity: the solution can be used as a tile, which allows to enlarge the usable surface region as long as the repetitiveness is not obvious. The corresponding negative aspect of FFTs is homogeneity: no local property can exist, so no refraction can be handled. It should be noted that this model is the one implemented by Areté Image Software and used for the special effects of many movies such as Waterworld or Titanic. In recent years, with the development of hardware, today’s commodity graphics hardware begins to provide programmability both at the fragment level and the vertex level. More and more efforts turn to graphics processing unit (GPU) to solve generalpurpose problems [14]. And a lot of GPU-based ocean simulation method has been brought out. ATI Research presents a multi-band Fourier domain approach to synthesizing and rendering deep-water ocean waves entirely on the graphics processor[15]. Mark Finch [16] provides an explicit solution for simulating and rendering large bodies of water on the GPU. His method has proven suitable for real-time game scenarios, have been used extensively in Cyan Worlds’ Uru: Ages Beyond Myst. Yang adapts GPU acceleration to realize a multi-resolution mesh model of the ocean surface based on a straightforward terrain LOD scheme[17]. Based on such related work, our contributions are primarily due to: a) providing an integrated ocean environment model; b) improving the efficiency so that the ocean environment can be rendered at over 200 frame rate; c) shortening the implementation cycle because of the simplicity of our ocean model.

An Efficient Method for Real-Time Ocean Simulation

5

3 Ocean Environment Model Our ocean environment model includes three parts: sky sphere, atmospheric scattering system and ocean wave model. As shown in Figure 1, sky sphere realize a wispy clouds effect, and the atmospheric scattering model computes the incident ray from the sky sphere to the ocean wave. Then we model the ocean wave surface, and compute the reflex ray based on the law of refection.

Fig. 1. The ocean environment model

3.1 Sky Sphere The recent generation of graphics hardware has brought about a new level of per-pixel programmability for real time graphics, especially with regards to dependant texture reads. With the ability to perform per-pixel math both before and after texture fetches, a variety of new texture based effects are now possible. Texture perturbation [18] is that use the results of a texture fetch to change the texture coordinates used to fetch another texel. Here we adopt texture perturbation effect to simulate the wispy cloud. The idea is to scroll two tileable textures past each other in opposite directions. The first texture is the perturbation texture, which has u and v perturbations stored in the red and green channels of the texture. The second texture is the cloud texture, which has its texture coordinates perturbed by the perturbation map. In order to make the clouds look much more realistic however, its best to use multiple layers of clouds. The cloud layers are scrolled past each other at different speeds to give the effect of depth through parallax. 3.2 Atmospheric Scattering The atmospheric scattering effect plays a significant role at the render of outdoor scenes. It directly decides the color and lightness of sky. However, it also woks on the color of the objects in the scene, and make users feel the distance and hierarchy of scene objects, and this is the most effect of atmospheric scattering.

6

H. Chen et al.

The traditional outdoor render always uses fog model in hardware to simulate the effect. An equation can express the fog model:

L = L0 (1 − f ) + C fog f Where

(1)

L0 is the intrinsic color, C fog is the color of fog, and f is the fog coeffi-

cient. This model has three deficiencies: first, the fog coefficient f is monochrome, but actually scattering influence the rays at various wavelength differently; secondly, f, being a function of distance, can not express the physical principle of scattering, so the precision of the model can not be guaranteed; finally, the color and lightness of C fog can not change based on viewing direction. Therefore, a model provided by Hoffman which is more physically accurate [19] is adopted in this work.

Fig. 2. Atmospheric scattering model

Based on the physical model, we can get this equation as follows:

L = L0 f Ex + LIn

(2)

where f Ex is the term of attenuation， LIn is the incidence ray from sky. As shown in Figure 2, we can see that L0 which is the reflex ray with intrinsic color is absorbed by atmospheric partly and deviate from primary direction as a result of outscattering, and these two parts compose the attenuation of ray. However， the ray from sky and other directions can radiate along the direction of the reflex ray, and this is the inscattering ray. The equation of absorption and outscattering is that:

L( x) = L0 e − β Ab x e − β Sc x = L0 e − ( β Ab + β Sc ) x

(3)

where β Ab is the factor of absorption, β Sc is the factor of scattering. β Ab and β Sc are based on the wavelength of ray, and in RGB system they can express as a triple

An Efficient Method for Real-Time Ocean Simulation

7

vector. In practice, the β Ab is set to the value suggested in [20], and β Sc is the sum of Rayleigh coefficient and Mie coefficient. However there is also phenomena which can add light to a ray. One case is inscattering, where light which was originally headed in a different direction is scattered into the path of a light ray and adds to its radiance. For computing the inscattering ray, we define the scattering phase function Φ(θ) which gives the probability of scattered light going in the direction (θ), θ is the angle of incidence ray and scattering ray. Since this is a probability function, when integrated over the entire sphere of directions the result is 1. So we get this equation:

ΔL = − β Ex ⋅ L + E Sun ⋅ φ (θ ) ⋅ β Sc Δx

(4)

ESun is a scalar factor and express the energy of sun. From (4), we can get:

L ( x ) = E Sun ( L0 e − β Ex x + Since

β Sc

β Scφ (θ ) (1 − e − β x )) β Ex Ex

(5)

is the sum of Rayleigh coefficient and Mie coefficient, the function

φ (θ ) depends on the particle doing the scattering. For particles much smaller than the wavelength of light (r < 0.05 λ), φR (θ ) is based on Rayleigh coefficient. For larger particles, φHG (θ ) is based on the Henyey / Greenstein function which is an approximation to the Mie coefficient.

φ R (θ ) =

φHG (θ ) =

3

(1 + cos 2 θ )

(6)

(1 − g ) 2 4π (1 + g 2 − 2 g cos(θ ) )3 / 2

(7)

π

g is the eccentricity of Mie coefficient, we use an approximation to express it, which scales with

1 / λ2 . Hence, the equation of β Scφ (θ ) is that: β Scφ (θ ) = β R

Substitute the

3

π

β Scφ (θ )

(1 + cos 2 θ ) + β M

(1 − g ) 2 4π (1 + g − 2 g cos(θ ) ) 3 / 2

(8)

2

of expression (5) with expression (8), we can wok out the

inscattering ray. 3.3 Ocean Wave Our ocean system combines geometric undulations of a base mesh with generation of a dynamic normal map. First, we generate the combination of sine waves to form the ocean wave mesh, and produce normal disturbance on the ocean wave mesh based on

8

H. Chen et al.

bump map [22]. Then the sky sphere is sampled to get the incidence ray. Finally the reflect ray is obtained based on Fresnel reflection formula. Obviously this ocean wave model is very effective, it is easy to form a ocean wave surface, and for using normal bump maps to generate composite normals on each mesh face, the ocean grid can be very coarse, but a high sense of reality can be generated. For example, Tessendorf [13] form an ocean surface based on FFT, and need sample a 2048*2048 grid for building an ocean environment; while, the system that we realize only need sample a 50*50 grid. To obtain more reality of ocean scene, we realize the atmospheric scattering model, and calculate the scattering effect for the reflex ray of ocean wave.

+

= Fig. 3. Ocean wave model

The algorithm is implemented on GPU using vertex shader and pixel shader. The vertex shader is responsible for generating the combination of sine waves that perturb the position. First, a height map is used to allow artist control of how high the waves are in different parts of the ocean. This can be useful for shorelines where the ocean waves will be smaller than those further out to sea; Second, the height of every position is computed based on the input sine waves, and due to the SIMD nature of vertex shaders, 4 sine waves are calculated in parallel; for perturbing the bump map’s tangent space normal, vertex shader also need compute the transform matrix from the tangent space into world space. The pixel shader is responsible for producing the bump-mapped reflective ocean surface.First, it transforms the tangent space composite normal into world space and calculates a per-pixel reflection vector Second, the reflection vector is used to sample the sky sphere;At last, the reflex ray is computed based on Fresnel reflection formula.

C water = F * Creflect + (1 − F ) * Crefract F=

(8)

(g − k)2 (k ( g + k ) − 1) 2 (1 + )(k = cos α , g = n a nb + k 2 − 1) 2( g + k ) 2 (k ( g − k ) + 1) 2

Where F is the Fresnel factor, α is the angle of incidence, na and nb are the refractive index of air and water. The color of the reflex ray is not the final color, because we calculate the atmospheric scattering model, and the reflex ray is the initial ray L0 of the atmospheric scattering model, so we must compute the absorption and scattering effect for the reflex ray.

An Efficient Method for Real-Time Ocean Simulation

9

Fig. 4. The ocean wave

4 Results Since mostly computation is carried on the GPU, the performance of proposed method is relatively sensitive with the GPU power. Our experimental PC is assembled with 1.6GHz Intel P4 CPU, 512M system RAM and nVidia GeForce 6600 with 128 M of video memory and the test program window is at the resolution 640*480. With this environment the frame rate of render ocean scene is over 200. Another experimental PC is assembled with AMD Athlon(TM) XP 1800+ CPU, 512M system RAM and ATI RADEON 9550 with 128M of video memory. With this environment the frame rate of rendering ocean is 100 almost.

Fig. 5. The integrated ocean environment

10

H. Chen et al.

Fig. 6. The image of ocean war

Our method has been implemented in an ocean war simulation platform, we also use the texture perturbation effects to simulate the fire and smoke. Snapshots of ocean scene are showed in Figure6.

5 Conclusion and Future Work A novel ocean environment model, including sky sphere, atmospheric system and ocean wave model, is proposed in this paper. Texture perturbation is employed on the sky sphere mesh to realize wispy clouds. We specifically give the explanation that how atmosphere system affects light and realize the absorption and scattering effect. Also we present an ocean wave model, which combines sine waves to form the ocean wave mesh, and produce normal disturbance on the ocean wave mesh based on bump map. Our algorithm is accelerated by GPU hardware, which makes the method easy to implement and efficient in rendering with high sense of reality. Although being with several advantages, this method has some deficiencies. Since we use sine waves to simulate ocean wave, the ocean wave is seasonal. Secondly, we do not consider the wakes of objects, and the effect is important for simulate the sail of warship. These issues will be in-depth addressed in our future work.

Acknowledgement This research was supported by the National Grand Fundamental Research 973 Program of China (Grant No. 2004CB719403), and the National High Technology Development 863 Program of China (Grant No. 2004AA115120).

An Efficient Method for Real-Time Ocean Simulation

11

References 1. J.A. Ferwerda, Three varieties of realism in computer graphics, in: Cornell Workshop on Rendering, Perception and Measurement, 1999. 2. A. Iglesias Computer graphics for water modeling and rendering: a survey , Future Generation Computer Systems, 20(8), Nov. 2004. 3. A. Fournier, W.T. Reeves, A simple model of ocean waves, in: Proceedings of SIGGRAPH’86, Comput. Graph. 20 (4) (1986) 75–84. 4. Kass, M., and Miller, G. Rapid, stable fluid dynamics for computer graphics. In Computer Graphics (SIGGRAPH ’90 Proceedings), F. Baskett, Ed., vol. 24, 49–57. 5. Foster and Metaxas, D. Realistic animation of liquids. In Graphics Interface ’96, W. A. Davis and R. Bartels, Eds., 204–212. 6. Foster, N., and Fedkiw, R. Practical animation of liquids. Proceedings of SIGGRAPH 2001 (August), 23–30. 7. Neyret, F., and Praizelin, N. Phenomenological simulation of brooks. In Eurographics Workshop on Computer Animation and Simulation, Springer, Eurographics, 53–64. 2001. 8. Thon, S., and Ghazanfarpour, D. A semi-physical model of running waters. In Eurographics UK. 2001. 9. D.R. Peachey, Modeling waves and surf, in: Proceedings of SIGGRAPH’86, Comput. Graph. 20 (4) (1986) 65–74. 10. TS’O, P., and Barsky, B. Modeling and rendering waves: Wave-tracing using beta-splines and reflective and refractive texture mapping. ACM Transactions on Graphics 6, 3 (July), 191–214. 1987. 11. GONZATO, J.-C., and SAËC, B. L. On modelling and rendering ocean scenes. The Journal of Visualization and Computer Animation 11, 1, 27–37. 2000. 12. MASTIN, G. A., WATTERBERG, P. A., AND MAREDA, J. F.Fourier synthesis of ocean scenes. IEEE Computer Graphics and Applications 7, 3 (Mar.), 16–23. 1987. 13. J. Tessendorf, Simulating ocean water, in: SIGGRAPH’99 Course Notes, 1999. 14. GPGPU http://www.gpgpu.org/ 15. Jason L. Mitchell, "Real-Time Synthesis and Rendering of Ocean Water," ATI Technical Report, April 2005. 16. Finch M. "Effective Water Simulation from Physical Models," GPU Gems: Programming Techniques, Tips, and Tricks for Real-Time Graphics. Edited by Randima Fernando, Page 5-29. 2004. 17. Xudong Yang, Xuexian Pi, Liang Zeng, Sikun Li, "GPU-Based Real-time Simulation and Rendering of Unbounded Ocean Surface" Ninth International Conference on Computer Aided Design and Computer Graphics (CAD-CG'05), Hong Kong, Dec 07-10 2005. 18. John Isidoro and Guennadi Riguer, "Texture Perturbation Effects, " ShaderX, WordWare Inc., 2002,ISBN 1-55622-041-3. 19. Naty Hoffman, Nathaniel and Arcot J. Preetham, Rendering Outdoor Light Scatter in Real Time, Proceedings of Game Developer Conference 2002. 20. A.J.Preetham, P.Shirley, B.E.Smit. A Practical Analytic Model for Daylight. Computer Graphics(Proceedings of SIGGRAPH 1999):91-100, August 1999. 21. Jan Kautz, Wolfgang Heidrich, Hans-Peter seidel. Real-time bump map synthesis. Proceedings of the ACM SIGGRAPH/EUROGRAPHICS workshop on Graphics hardware Aug.2001.

The Dynamic Textures for Water Synthesis Based o n Statistical Modeling Qian Zhang and TaegKeun Whangbo Department of Computer Science, Kyungwon University, Sujung-Gu, Songnam, Kyunggi-Do, Korea [email protected] [email protected]

Abstract. The synthesis for dynamic flowing water is of relatively high practical value in design of virtual reality, computer games, digital movies and scientific computing etc. On one hand, the physical model cannot help people to produce the photorealistic and easy edited flowing scene; on the other hand, digital products can be used to show the flowing scene in the world easily. This paper presents a novel algorithm for synthesizing dynamic water scene based on a sample video. To obtain video textons, we analyze the sample video automatically using dynamic textures model. Then we utilize linear dynamic system (LDS) to represent the characteristic of each texton. By further hard constrains we synthesize a new video for dynamic water flow which is prolonged and non-fuzzy in vision. We provide test examples to demonstrate the effective and efficiency of our proposed method.

Keywords: Textons, Dynamic textures, LDS, Statistics method, Textures synthesis.

1 Introduction The synthesis for dynamic water flow is of relatively high practical value in design of virtual reality, computer games, digital movies and scientific computing etc. It is one of the research focuses in computer graphics and virtual reality all the time to utilize the flowing water result of computer simulation. The existing methods about simulation flowing water are divided into two kinds. One is based on Navier-Stokes equation in hydrodynamics, which produces the flowing water result through calculating [1].The other is a synthetic method based on video, which adopts the textures synthesis technology to produce longer flowing water video [2]. The former cannot output the photorealistic results; it is carried out by the high performance computer, and can produce the scene in which flowing water does not change much, such as calm sea level, the river, etc. A lot of flowing water scene has not reached the lifelike result on the vision, for instance the simulation of vortex. The latter analyzes the dependence between frames mainly, reconfigures frames and obtains the flowing water result. It is with high efficiency, but the method is apt to produce the textures of videos to repeat and to be fuzzy. K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 12 – 21, 2007. © Springer-Verlag Berlin Heidelberg 2007

The Dynamic Textures for Water Synthesis Based on Statistical Modeling

13

In this paper, we propose a simulation method based on dynamic textures. Our research utilizes the video sample to synthesize water scene flowing smoothly in vision at fixed time. The method is based on video sample as well, but we refer to the MRF (Markov Random Field) in 2-D texture synthesis, and use the dynamic textures to solve the fuzzy phenomenon in method [2]. The textures describe the pixels’ distribution model on gray level scale, and show the quality of the object, for instance roughness , smooth , particle degree , randomness, etc. The structural techniques (one of the three principal approaches used to describe texture) characterize texture as being composed of simple primitives called "texton" (texture element), which is regularly arranged on a surface according to some rules. Our method is regarded the video sample as textons. We calculate the distribution of video textures through analyzing automatically, and then normalize the flowing water results. The method utilizes linear system (Linear Dynamic System, LDS) to describe texton. Synthesis technique can control the results by giving restrained conditions, in order to avoid the fuzzy phenomenon appearing in the method [2]. The remainder of this paper is organized as follows. After reviewing related work in Section 2, we introduce several concepts in Section 3, and show how to analyze and synthesis the video in Section 4. Applications of dynamic texture including several kinds of flowing water synthesis are shown in Section 5. We conclude our paper in Section 6.

2

Related Works

Initially, people in terms of hydrodynamics, produce the visual effects such as sea level, wave and river, through the modeling method of physics. Limited by the CPU’s performance, it takes a lot of time to edit the flowing water. After that, researcher improved the method using parameter models. For example, in paper [3] wave function showed a series of linear association of wave type, and adopted particle system to simulate spray. In order to show folding result of surface of water, paper [4] based on Gerstner model adopted Lagrangian particle to imitate the wave parameter surface, which controls sinusoid through the depth and slope of the sea floor. But the methods above two papers, show particle and net of water are just moved near its initial position. So they are all unable to display the real flow result, and can't deal with the influence brought to surface of water at the border either. In recent years, a lot of scholars use this method to produce the cartoon [5] of the flowing water result, and get better result. Mastin et al. [6] proposed marine statistics and experience model in 1987. The method adopted the superposing of a large number of sine curves to imitate the sea, through FFT (Fast Fourier Transform) to produce the true wave high field. This method well described the horizontal smaller in amplitude. But comparing with the above models, people think that it is very difficult to control, and can't imitate some complicated results. Then, people began to try to utilize video to produce the flowing water scene in recent years, such as Bar-Joseph [7], Wei [8], Schodl [2] [9]. The method analyzes the video samples, and rearranges the frames with high efficiency. Paper [10] rearranged the frames by analyzing the relationship between frames. The method could handle the synthesis of both unstructured textures as well as the structured textures. It could cause the duplication in vision, and cannot used to get the better result in waves and

14

Q. Zhang and T. Whangbo

body motion. The main difference between [7], [8] and [2] is [8], [2] using different ways to reconstruct every frame. Methods [8] [2] improve the quality of the video result, but efficiency is very low. Stefano Soatto proposed the Dynamic Texture’s theory in paper [9]. In the paper author got a dynamic texture through dynamic model, calculated the model’s parameters, and got video prolonged. Li Yan et al. [11] in 2002, proposed a motion texture system to synthesize the body motion video, and obtained the better results. Kiran S. Bhat [12] in 2004 adopted human-computer interaction to format the dynamic scene according to the route that users mark. The method which chases frames and assigns an editor to train the sample pictures one by one, has made the good experiment result. Our research utilizes the three thoughts above to synthesize the flowing water.

3

Several Concepts

Texton[13] and dynamic texton. Textures show the luminance (gray palette) in 2-D images which have three standards: repeat constantly in a certain array area larger than this array; the array is made up at random by the basic part; every part is the same entity in general and there is roughly the same dimension in any locality of the texture area. The basic part of this array is usually called texton. The dynamic textons proposed in the paper is one part of the video, which show the vision consents. For being simple, we call dynamic texton as texton in the paper. LDS[14]. Generally, the system means any entities which accept input, and output corresponding results. In a linear system the phase space is the ν-dimensional Euclidean space, so any point in phase space can be represented by a vector with ν numbers. The analysis of linear systems is possible because they satisfy a superposition principle: if u(t) and w(t) satisfy the differential equation for the vector field (but not necessarily the initial condition), then so will u(t) + w(t). In the paper we use LDS to describe dynamic texton. Xt (at time t in frames of texton) shows the state variable which the system inputs, and Yt is the system’s output, where Yt is the frame at time t. Dynamic texture [9]. Dynamic textures are sequences of images in moving scenes that exhibit certain stationary properties in time; these include sea-waves, smoke, foliage, and whirlwind but also talking faces, traffic scenes etc. Let {I(t)}, t=1…τ. Suppose that at each instant we can measure a noisy version y(t)=I(t)+ω(t), where ω(t) is an independent and identically distributed sequence drawn from a known distribution. We say that the sequence is a (linear) dynamic texture. Dynamic textures model. Dynamic textures model is a two-level statistical model with textons[13] and their distribution, as shown in Figure 1. Each texton is represented by a LDS [14] and the initial state distribution. Texton distribution can be represented by a matrix Mij.

Each texton is represented by a LDS with the following state-space model:

⎧ X t +1 = AX t + Bvt ⎨ ⎩ Yt = Ct X t + ωt

(1)

The Dynamic Textures for Water Synthesis Based on Statistical Modeling

15

Where, Xt is the hidden state variable, Yt is the observation, and vt and ωt are independent Gaussian noises at time t. Then the parameters of an LDS can be represented by θ = {A, B, C}. Video Sample Texton

1

Y0 Y1Y2 ···Ym

Texton

Ym+1 Ym+2 ···

2

Texton

Yi

Yt-s··· Yt-2 Yt-1

n

Yt

Fig. 1. With the learnt dynamic texture, a video sequence can be divided into multiple segments, labeled as texton 1, texton 2… texton n. Each segment may have a different length. We use Y0, Y1… Yt to represent continuous frames of the vide sample.

We assume that the distribution of textons satisfies the first-order Markovian dynamics, which could be represented by a transition matrix. Mij= P (lk= j | lk-1= i)

(2)

Where two continuous textons are labeled as lk and lk-1, frame j and frame i in the two textons are labeled as j and i. P is the transition probability between lk and lk-1. The scale of transition matrix Mij is m*n, and m and n labeled as the amount of frames in textons lk and lk-1 respectively.

4

Synthesis with Dynamic Textures

The synthesis of dynamic textures is including two stages: analyzing and synthesis. The analyzing is the process in which we will divide the video sample into textons, and got the initial state distribution between textons; the aynthesis reconstructs the textons, and got the smooth, prolonged video according to the user’s demands. The algorithm is summarized as follows: Step 1: get textons and their distribution We find the textons in video sample using greedy approach, and get the distribution between two random textons, which is labeled by a matrix Mij. Step 2: estimate the LDS for every texton We describe the texton with a second-order linear dynamic system, and estimate the each LDS’s parameters. Step3: synthesize the new video We get the new order among textons using dynamic programming approach and incorporate the two textons with hard constrains.

16

Q. Zhang and T. Whangbo

4.1 Analyzing Dynamic Textures Automatically Given Y1:T = {Y1, Y2, . . . , YT }, or the observation Yt from frame 1 to frame T, our system calculates the model parameters Θ {A, B, C}. Set p (t)= xt / Xt, here p(t) is the confidence value which decided the frame’s quality at time t., xt is the state variable when we using LDS to calculate, and Xt is the variable got from Yt. First, analyze dynamic textures We take a greedy approach to incrementally initialize our model. First, we use Tmin frames to fit an LDS i, and incrementally label the subsequent frames to segment i until the fitting error is less than p(x). Then all existing LDS’ (from 1 to i) learnt from all preceding segments (possibly more than i) are tested on the remaining unlabeled Tmin frames, and the best-fit LDS is chosen. If the smallest fitting error exceeds the given threshold, i.e., none of those LDS fits the observation well, we introduce a new LDS and repeat the above process until the entire sequence is processed. Second, parametric inference of LDS In order to capture richer dynamics (velocity and acceleration), we use a second-order linear dynamic system:

＝

⎧ Yt = Ct X t + ωt ⎨ ⎩ X t +1 = A1 X t + A2 X t −1 + D + Bvt

(3)

Where vt ~N (0, 1), ωt ~N (0, R), and R is the covariance matrix. A closed form approximated estimation of the model parameters could be derived as follows [9]: 1. We calculate the SVD of the observation sequence Y1: T, [U, S, V] = SVD(Y1:T ), and set: C＝U，X= SVT

(4)

2. The maximum likelihood estimation of A1, A2, D, and B are given by:

[ A1 , A 2 ] = [ R 00 , R 01 ] ⋅ [ 1 [R τ − 2

R 11 R 12

2

∑

R 21 − 1 ] R 22

(5)

AiR i ]

(6)

1 T Ri R j τ −2

(7)

) ) ) ) ) ) ) Where, BB T = Q , and vˆ(t ) = xt +1 − A1xt − A2 xt −1 − D . Q matrix. We could reduce the rank through SVD. ΣQ = diag (σ Q (1),⋅ ⋅ ⋅, σ Q (k )) , k≤n.

is not the full rank ) Q = U Q Σ QU Q T ,

D =

0

−

i=1

τ

Ri = ∑ X t −i t =2

τ

Rij = ∑ X t −i ( X t −i )T − t =2

The Dynamic Textures for Water Synthesis Based on Statistical Modeling

17

Third, normalize distribution of textons. When we obtain Mij, the matrix M is then normalized such that ∑ M ij = 1 . i =1

In the paper, an important parameter that the user needs to determine is Tmin. Tmin must be long enough to describe the original wave. In our system, it is suitable to choose 20~30 frames as Tmin. The threshold of p(t) is determined by users, and it is related with the video sample’s quality. 4.2 Synthesis the Dynamic Textures

As analyzing the dynamic textures, new video can be synthesized. Moreover, we can edit the video interactively, both at the texton level and at the distribution level. First, rearrange the dynamic textons. We rearrange the textons according to the textons and the user’s demand. The flowing water in the nature may be influenced by the wind, artificial force, etc. As the system performed, we rearranged the textons by transition matrix. In our system, we choose the textons randomly when they have nearly the same matrix. Second, synthesize the textons. In this paper, we divide the flowing scene into two types: regular dynamic texture and random dynamic texture. When we take experiment with different sample, we utilize the different method to solve them. The regular dynamic texture means that the change of flowing water is repeated and regular, such as calm sea level, lake level and so on. The main character is that we can incorporate the two textons directly without producing the new frames at the junction. The random dynamic texture means that the change of flowing water is irregular, such as spring, brook, river, and so on. The main character is that we cannot get the smooth video in vision if we incorporate the two textons directly. So we incorporate the two textons with hard constrains. When we have the texton sequence, the second step in synthesis is conceptually straightforward. In principle, given a texton and its key frames, a video sequence can be synthesized frame by frame with the learnt LDS and sampled noise. However, the prediction power of LDS decreases after some critical length of the sequence as LDS approaches its steady state. In the system, we called the unjointed textons as synthesis texton and constrained texton, according the time. A by-product of the constrained synthesis is smooth transition between two textons because the two starting poses of the second texton are guaranteed from the synthesis processes of both the first and the second textons. Let’s Wi and W j are the two unjointed textons, and {xi1 , xi 2 } , {x j1 x j 2 } are the key frames in the two textons respectively. In order to joint the two textons smoothly in video, we set the hard constraint as follows:

⎧ x1 = xi1 , x2 = xi 2 ⎨ T T ⎩ xl −1 = Ci C j x j1 , xl = Ci C j x j 2

(8)

We should switch C n to CiT C j through the projection as the hard constraint matrix. The synthesized frames between two textons are recorded as x3 , x4 , x5 ,...xl − 2 . We calculate the transition in the linear system of the first texton based on the LDS for the two textons we have established before.

18

Q. Zhang and T. Whangbo

5

Experimental Results

We utilized sea level video and spring video to experiment separately on an Intel Pentium IV 3.0GHz computer with 1G memory. The two results are shown as follows according to the algorithm: 5.1

Analyze the Dynamic Textures

We use our algorithm to analyze the simple video, and show the results in Table 1. Table 1. Analysis results of simple video.

Video(frame) 1.Sea level

Textons (frame)

CPU time(s) Synthesis result(frame)

Texton1:1~13,22~35 46

Texton2:14～21

34.486

134

264.420

238

Texton3:36～44 Texton1:1～44

2.Spring

180

Texton2:45~156 Texton3:157~180

5.2

Synthesize the Video

Sea level

We only show the part of textons and the synthesis results in Fig.2 and Fig.3 limited by the space.

Fig. 2. Continuous frames in a texton, 128*128

Fig. 3. Synthesis textons, 128*128

The Dynamic Textures for Water Synthesis Based on Statistical Modeling

19

Spring

We show three continuous frames in original video sample in Figure 4, and we got three continuous frames using LDS we have calculate in Figure 5. We show the synthesis with two adjacent textons in Figure 6 .

Fig. 4. Three continuous frames in original video sample,320*240

Fig. 5. Continuous frames using LDS we have calculate, 320*240

Flowing water

We divided flowing water into regular and random, according flowing water’s texture character. There will produce the different result according to the different type of flowing scene. It is boundless to synthesis the video for regular dynamic texture as long as the sample is satisfied the shortest frames. But for the random dynamic texture, when we use the shortest frames, it will cause reiteration in the boundless time. So there is a longest time for synthesized video. The conclusion is shown in Table 2: Table 2. The results between different type of flowing scene

Flowing water

regular

random

Shortest frame as

Synthesis the longest in time

synthesis(frame)

with shortest frame(frame)

Lake

35

boundless

Sea level

70

boundless

Spring

65

174

River

73

248

20

Q. Zhang and T. Whangbo

A

D

G

B

E

H

C

F

I

Fig. 6. Synthesize with two adjacent textons. A is the last frame in first texton, E … H is the smooth connection between two adjacent textons, and I is the first frame in second texton.

6

Conclusions

In the paper, we proposed a method to synthesis flowing water based on dynamic textures model. The method can handle these cases mixed with synthesizing photorealistic or cartoon flowing scene. In our system, dynamic textures are represented by a set of textons and their distribution. As calculating LDS of each texton, we can synthesize and edit video easily. In the end, we got the satisfied results by several experiments. We were trying the find a reliable method to imitate flowing scene which would show the dynamic landscape in digital TV or computer games, such as city landscape show, the ditch work demonstration, etc. Our approach inspired by the proposed model for dynamic textures. We realized the model by effective analysis, optimized LDS and hard constraint. In addition, we find that different decoding methods for video will bring little influence for our results. LDS is the essential to synthesize the high quantity video. Currently we need go further research on the interaction of the system and improve the control of LDS.

The Dynamic Textures for Water Synthesis Based on Statistical Modeling

21

References 1. Ying-qing Xu, J. :The imitation for flowing water and waves based on physical model, Chinese Journal of Computer (supplement) , (1996) 153 160 2. Arno Schodl , Richard Szeliski , David H. Salesin , Irfan Essa, C.: Video Textures, In: Proceeding of SIGGRAPH’00, New Orleans, (2000) 489 498 3. Darwyn R. Peachey, J.: Modeling Waves and Surf, Computer Graphics, (1986) 20(4):65~74 4. Alain Fournier, William T. Reeves, J.: A Simple Model of Ocean Waves, Computer Graphics, 20(4), (1986) 75~84 5. Nick Foster, Ronald Fedkiw, C.: Practical Animation of Liquids, In: Proceedings of SIGGRAPH, New York: ACM Press, (2001)23-30 6. P A Watterberg, G A Mastin, J F Mareda, C.: Fournier synthesis of ocean scenes, IEE Computer Graphics and Applications, 7(3), (1987) 16-23 7. Z. Bar-Joseph. D.: Statistical Learning of Multi-dimensional Textures, Master’s thesis, The Hebrew University of Jerusalem, June (1999) 8. Li-Yi Wei, Marc Levoy, C.: Fast Texture Synthesis using Tree-structured Vector Quantization, Proceedings of SIGGRAPH’2000, August (2000) 479-488 9. Stefano Soatto, Gianfranco Doretto, Ying Nian Wu, C.: Dynamic Textures. In: Proceedings of ICCV'01, Vancouver, BC, Canada, (2001) 439 446 10. Jos Stam.C.: Stable Fluids. In: Proceedings of SIGGRAPH, New York: ACM Press, (1999)121~128 11. Yan Li, Tianshu Wang, Heung-Yeung Shum, C.: Motion Texture: A two level statistical model for character motion synthesis, In: SIGGRAPH’2002 Conference Proceedings, San Antonio, Texas, (2002) 465 472 12. Jos Stam, C.: Real-Time Fluid Dynamics for Games, In: Proceedings of the Game Developer Conference, (2003) 13. Precept J BeCK, J.: Effect pf orientation and of shape similarity on perceptual grouping,Perceptual psychophysics,(1966) 300-302 14. Theodore E Djaferie, Irvin C Schick, M.: System Theory: Modeling, Analysis and Control Springer published on 1 Jan, (1999) 15. Bui TD, Chen G, J.: Translation invariant de-noising using multiwavelets, IEEE Transactions on Signal Processing, 46(12), (1998) 3414 3420 16. Jerry Tessendorf, C.: Simulating Ocean Waves, SIGGRAPH '99 Course Notes & SIGGRAPH'2000: Course Notes 25: Simulating Nature: From Theory to Practice, (2000) 3.1-3.18 17. Brand M, C.: Hertzmann A.Style machines, Proceedings of SIGGRAPH’00, New Orleans, (2000) 183-192 18. Gianfranco Doretto, Stefano Soatto, J.: Dynamic Shape and Appearance Models, IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 28, No. 12, December (2006) 2006-2019 19. K. S. Bhat, S. M. Seitz, J. K. Hodgins, P. K. Khosla, J.: Flow-based Video Synthesis and Editing. ACM Transactions on Graphics, ACM Press, 23(3), (2004) 360-363

～ ～

～

：

～

～

Constructing a Convenient and Efficient System for Making Virtual Human Accentuated by Customized 3D Face and Body Seongah Chin1, Jongkyeong Park2, and Moonwon Choo1 1

Division of Multimedia, College of Engineering, Sungkyul University, Anyang, Korea {solideo,mchoo}@sungkyul.ac.kr 2 Department of Computer Science and Engineering, Korea University, Seoul Korea [email protected]

Abstract. In this paper, an efficient system generating a customized 3D virtual human reflecting user’s features by using the frontal and lateral pictures captured through a PC camera is suggested. To render the complete 3D virtual human figure, a customized virtual face is defined based on predefined facial feature points, followed by the generation of virtual body and accessories by mapping through Body Mass Index (BMI), which is a measure of body fat defined as the weight divided by the square of height. The basic emotion elements also can be expressed using special muscle modules. The proposed system can generate several 3D formats such as AKF, ASE, and WRL as outputs for various types of displays. The better performances in terms of easiness of usage, speed of generation, and the usability are validated by comparing with those of similarly implemented systems. The test results are shown.

1 Introduction In recent years 3D virtual human modeling is considered as promising research fields for next generation Human Computer Interaction. In the various areas such as digital human modeling, 3D avatar, game character and entertainment, the methodology of creating virtual human has been one of challenging research topics. An overall description of simulating a face is addressed by Parke and Waters [2]. In addition, P. Ekman and W. V. Friesen introduce a guide to recognizing emotions from facial clues [4]. Those preliminary theories provide us with fundamental approaches to specification of a customized face modeling. In particular, a customized virtual human defined by a personalized face model and its body is one of significant issues in this area, which facilitates creation of similar model of existing human in real as players appearing in FIFA game and definitely draws more attention from users. However, it is not easy for general users to implement a customized 3D modeling. Moreover currently existing systems such as FaceGen [9], 3DMeNow [10], Dantirum [11], Zeb [12] and Qavatar [13] are not sufficiently satisfactory for creating a customized face and body model. Some of them take longer running time or do not provide the function to customized face model. K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 22–33, 2007. © Springer-Verlag Berlin Heidelberg 2007

Constructing a Convenient and Efficient System

23

As a solution to this problem, this paper presents an efficient modeling system providing users with easy access to generate a customized 3D virtual human with basic emotional expressions. The proposed system has several advantages compared with commercially available systems which were developed for similar functions. In this system, 3D virtual face based on predefined facial feature points is quickly generated and customized using two pictures captured from frontal and lateral views. Then 3D virtual body is configured by referring to BMI (Body Mass Index) specifying 7 degree measures of body fat based on height and weight that applies to both adult men and women. In addition, the basic emotional elements implemented using muscular modules can be expressed through user’s direct manipulation. We expect this system could be positioned in commercial sectors related to HCI applications such as PDA interface, 3D authoring tools, and Web3D content, etc.. The system consists of five main modules as shown in figure 1. The function of each module is specified in the following sections. The PC version is developed on Pentium 4 (3GHz) PC equipped with LogiTec SphereMP PC Camera and PDA(iPAQ-2210, 2790) is used for mobile version.

Fig. 1. Data flow diagram of the proposed system

2 3D Virtual Face Generation Module The front-end module accepts predefined facial feature points adapted from MPEG-4 defined facial feature points created by user’s picture inputs. These feature points are transformed into a customized 3D virtual face by applying special transformation algorithm to Korean canonical facial model standing for Korean male, configured by the fuzzy c-mean clustering method [3],[7]. 2.1 Feature Points Input Module and Scaling Process User’s frontal and lateral pictures are taken by a PC Camera. The distance between user and cam can make the size of the captured object distorted. This distortion can be easily calibrated by a standard credit card whose normalized dimensions are precalculated, which guarantees the exact facial mapping without worrying about the user’s varying dimensions of faces that could be changed by ages and figural characteristics. As shown in figure 2 the credit card is pushed and located firmly just under

24

S. Chin, J. Park, and M. Choo

chin. The box shaped edge of credit card is extracted and used for calibrating the distance between the object and camera. After that, user can input the anatomical feature points, 56 frontal and 19 lateral points through rubber band method. This simple method gives the useful information for calculating the relative ratio of the size of face to the height.

Fig. 2. Input pictures and marked feature points calibrated by a credit card image

Users have different sizes of faces, determining the proportional ratio of the size of face to body. To project the size of user’s face onto the 3D virtual face, the canonical face is transformed using the coordinate of feature points extracted from the captured pictures. The input feature points determine the width, height, and the depth. These facial characteristics are compared with normalized 3D face, scaled to fit for user’s real face by the simple linear transformation [3] using equation of proportional ratio. 2.2 Generation of the Customized 3D Virtual Face The 3D Virtual Face is customized through the transformation [6] based on feature points corresponding to all vertices. This transformation operation is characterized by the feature points as control points and applied to normalized 3D face. The equations (1) to (10) are to derive new vertices after applying transformation based on the arbitrary control line defined by user input. For deriving the 3D feature based transformation, let us define variables as below:

Ai = [ Ai ( x), Ai ( y ), Ai ( z )] , Bi = [ Bi ( x), Bi ( y ), Bi ( z )] Ai ' = [ Ai ' ( x), Ai ' ( y ), Ai ' ( z )] , Bi ' = [ Bi ' ( x), Bi ' ( y ), Bi ' ( z )] P = [ P ( x), P ( y ), P ( z )] , P ' = [ P ' ( x), P ' ( y ), P ' ( z )] The following equation (1) give us to acquire ui , the ratio defined by the length of Ci over the orthogonal point projected by P onto Ci . And li is the length between P and the orthogonal point onto Ci .

Constructing a Convenient and Efficient System

ui =

25

( P( x) − Ai ( x))( Bi ( x) − Ai ( x)) + ( P ( y ) − Ai ( y ))( Bi ( y ) − Ai ( y )) + ( P ( z ) − Ai ( z ))( Bi ( z ) − Ai ( z )) Ai − Bi

2

(1) l i = Ai − P 2

2

−

( P ( x) − Ai ( x))( Bi ( x) − Ai ( x)) + ( P ( y ) − Ai ( y ))( Bi ( y ) − Ai ( y )) + ( P ( z ) − Ai ( z ))( Bi ( z ) − Ai ( z )) Ai − Bi

2

(2) How to derive a transformation in 3D space? The final form of the transformation matrix is as below:

Ψ = T ( Ai ' ) Rx (−θ x ' ) R y (−θ y ' ) R y ( −θ y ) Rx (θ x )T ( − Ai )

= T ( Ai ' ) Rx (−θ x ' ) R y (−θ y '−θ y ) Rx (θ x )T (− Ai )

(3)

where T and R are for translation and rotation matrix respectively, and δ , θ are distance factor and angle for transformation respectively. A and A' represent the vertices before and after transformation respectively. We can obtain the following equations using a unit vector v from Ai , Bi. and trigonometric functions. The final form of the transformation is the equation (4). Any vertex P can be transformed to P′ with respect to another control line Ci′.

⎡ψ 00 ⎢ψ Ψ = ⎢ 10 ⎢ψ 20 ⎢ ⎣ψ 30 Where

ψ 01 ψ 11 ψ 21 ψ 31

ψ 02 ψ 12 ψ 22 ψ 32

ψ 03 ⎤ ψ 13 ⎥⎥ ψ 23 ⎥ ⎥ ψ 33 ⎦

ψ 00 = cos(−θ y '−θ y ) ψ 01 = − sin(−θ y '−θ y ) sin(θ x ) ψ 02 = − sin(−θ y '−θ y ) cos(θ x ) ψ 03 = −(−δ x ) cos(−θ y '−θ y ) + (−δ y )sin(−θ y '−θ y )sin(θx ) + (−δ z )sin(−θ y '−θ y ) cos(θx ) + δ x '

ψ 10 = sin(θ x ' ) sin(−θ y '−θ y ) ψ 11 = cos(θ x ' ) cos(θ x ) + sin(θ x ' ) cos(−θ y '−θ y ) sin(θ x ) ψ 12 = − cos(θ x ' ) sin(θ x ) + sin(θ x ' ) cos(−θ y '−θ y ) cos(θ x ) ψ 13 = − ( − δ x ) sin( θ x ) sin( −θ y '−θ y ) − ( − δ y ) cos( θ x ' ) cos( θ x ) − ( − δ y ) sin( θ x ' ) cos( −θ y '−θ y ) sin( θ x ) + ( − δ z ) cos( θ x ' ) sin( θ x ) − ( − δ z ) sin( θ x ' ) cos( −θ y '−θ y ) cos( θ x ) + δ y '

ψ 20 = cos(θ x ) sin(−θ y '−θ y ) ψ 21 = − sin(θ x ) cos(θ x ) + cos(θ x ' ) cos(−θ y '−θ y ) sin(θ x )

(4)

26

S. Chin, J. Park, and M. Choo

ψ 22 = sin(θ x ' ) sin(θ x ) + cos(θ x ' ) cos(−θ y '−θ y ) cos(θ x )

ψ 23 = −(−δ x ) cos(θ x ) sin(−θ y '−θ y ) + (−δ y ) sin(θ x ) cos(θ x ) − (−δ y ) cos(θ x ) cos(−θ y '−θ y ) sin(θ x ) − (−δ z ) sin(θ x ' ) sin(θ x ) − (−δ z ) cos(θ x ' ) cos(−θ y '−θ y ) cos(θ x ) + δ z ' ψ 30 ψ 31 ψ 32 ψ 33

=0 =0 =0 =1

Upon completing 3D transformation of control lines, we employ feature based transformation [7] making it possible to calculate the converting position of the vertices, which are determined by combination of moving control lines. In addition, Phong Shading has been applied into the created face model, enabling us to visualize a face model as volumetric as possible by combination of 4 vectors such as view direction, light source, norm and reflection of rays. Table 1. Locations of linear and sphincter muscle location

No. 1, 2 3, 4 5, 6 7, 8 9, 10 11, 12 13, 14 15, 16 17, 18 D E F

Muscle name Zygomatic major Depressor anguli oris Frontails inner Frontails major Frontails outer Levator anguli oris Levator labii superioris Orbicularis oculi Procerus Right Orbicularis oculi Left Orbicularis oculi Orbicularis oris

3 Facial Expression Module The created customized 3D virtual face can be manipulated to express basic 6 emotions defined by Ekman [1],[4] using 18 linear and 3 sphincter muscles. 3.1 Facial Expressions Ekman and Friesen suggested facial muscle-based FACS(Facial Action Coding System), which categories six basic human emotions after analyzing the muscular movement systemically including facial muscle locations and their effective ranges of

Constructing a Convenient and Efficient System

27

subtle dislocations [1] differentiating the types of emotion. Based on this research, six emotion elements are identified by determining the anatomical positions of linear and sphincter muscles in this paper. 3.2 Linear Muscle We employ 18 linear muscles as shown in Table 1, associated with muscle names. The contraction and expansion movement of linear muscle can be described by fan shaped geometry. The beginning point is taken as a center stake of fan. The angle made by each muscle and its ending points are used to mark the 3D fan-typed area. All vertices located in this area are divided into two regions by distances. By using cosine functions, the movement of vertices is determined by the distance from beginning point and the degree of angular changes [2]. 3.3 Sphincter Muscle 3 Sphincter muscles annotated by D, E and F in Table 1, are capable of expressing eye blinking and mouth movement by using the ellipsoidal principles, which may not be possible with linear muscles. Eye blinking can be effectively expressed by allowing linear dislocations of minor axis while fixing the center of ellipsis. However, an eye is not ideally shaped as ellipsis. Accordingly, the some variation in terms of distortions of eye shape should be allowed, resulting in the equations (5) and (6).

(a)

(b)

Fig. 3. Sphincter Muscle in an anatomical picture in (a) and a parametric ellipsoid description in (b) Px 2 Py 2 + 2 =1 lx 2 ly

1 1 W (θ i ) = ( − cos 2(θ i ) + ) n 2 2

M = kW (θi )

(O − Pi ) O − Pi

(5) (6) (7)

Where k is the value for contraction of muscles and O is a center of ellipsis. The final movement vector M is determined by the equation (7).

28

S. Chin, J. Park, and M. Choo

3.4 Opening Mouth Chin joint is used to express the openness of mouth faithfully, which is not possible only with muscular movement. As shown in figure 4, the region of chin is determined by drawing lines from the lower end of ear to lower lips. The openness of mouth is calculated as the rotation angle of this line. As a result, the lower chin, lower tooth and tongue are accordingly rotated. This rotation is formulated as the following equation (8) reflecting the transformation matrices respect to all vertices located at A region shown in Figure 4 that can be warped by the rotation of chin joint.

Fig. 4. The region of chin joint and rotation

0 0 0 ⎤ ⎡1 ⎢0 cos(η ) sin(η ) (cos(η ) • S ) + (sin(η ) • S ) − S ⎥ y z y ⎥ (8) Ω=⎢ ⎢0 − sin(η ) cos(η ) (− sin(η ) • S y ) + (cos(η ) • S z ) − S z ⎥ ⎥ ⎢ 0 0 1 ⎦ ⎣0 Where S = [Sx, Sy, Sz]T and η is the rotation angle along x axis. Table 3 shows the contraction of linear (labeled by 1 to 18) and sphincter muscles (marked by D, E and F) specified in figure 4 to accurately generate six universal emotions. The G entries represent the degree of chin rotation and its unit is radian.

4 Generation Module of Customized 3D Virtual Body The complete 3D virtual human is generated by adding virtual body to the virtual face. BMI and the canonical body measure for Korean male are utilized for this process. BMI is a measure of body fat obtained by dividing weight (kg) with squared height (m) without any special devices. The correlation of BMI and body measure is computed by Spearman correlation factor, resulting in relatively high correlation coefficient 0.7 – 0.8 on the 1.0 scale [8]. Based on this research, Korean young males are classified into seven classes by the relations imbedded in BMI and body measures in this paper (see table 2). Table 2. BMI based on body classes class 1 BMI ≈18

2 18-20

3 20-22

4 22-23

5 23-25

6 25-27

7 27≈

Constructing a Convenient and Efficient System

29

Table 3. Six universal expressions created by PC version defined by contracting value of linear and sphincter muscles Happy

Angry

Surprised

Sad

Fear

Disgust

0.5 0.5 -0.56 -0.56 -0.19 -0.19 0.34 0.34 0 0 0.68 0.68 0 0 0 0 0 0 -0.1 -0.1 -0.24 -0.04

0.85 0.85 0.4 0.4 0 0 0.17 0.17 0 0 1.0 1.0 1.7 1.7 -0.5 -0.5 0.06 0.06 -0.09 -0.09 -0.06 0.02

0.5 0.5 0.1 0.1 0 0 0 0 0 0 0 0 -0.34 -0.34 1.7 1.7 0 0 0.5 0.5 0.24 -0.11

0.5 0.5 0.4 0.4 -0.35 -0.35 0 0 0 0 0.17 0.17 0 0 1 1 -0.06 -0.06 -0.06 -0.06 -0.18 -0.03

3.0 3.0 0.1 0.1 1 1 1 1 1 1 0.17 0.17 0 0 0 0 -0.06 -0.06 0.3 0.3 -0.29 0.02

-0.52 -0.69 0.06 0.3 -0.52 -0.52 -0.52 -0.52 -0.18 -0.18 1.96 0.51 0 0 1.7 1.7 -0.52 -0.52 -0.28 -0.2 -0.1 0.01

Facial expressions

Degree of contraction and rotation of chin joint annotated by A

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 D E F

η

These BMI class indices provide the criteria for designing 3D body model. Seven types of 3D body model is proposed by considering the degree of obesity with fixed height. Table 4. 3D body model based on body classes class

1

2

3

4

5

6

7

model

The body model is adjusted progressively by the data input such as age, sex, height, and, weight and composed with virtual face to make the complete virtual human. Also the accessories like glasses, earring and shoes can be added with 8 kinds of hair styles and 5 kinds of hair colors according to user’s taste. Figure 5 shows the optional hair styles to be chosen by users.

30

S. Chin, J. Park, and M. Choo

Fig. 5. Eight kinds of hair styles can be chosen

5 Exporting Module of Virtual Human and Applications The reuse and interoperability of 3D virtual human are taken into consideration by generating several file formats such as ad-hoc AKF, ASE 3D format for 3D modeling and animation, and WRL for Web3D. Figure 6 shows the whole procedures for creating 3D face in PDA devices. The ASE format can be fed into game developing stage helping a programmer to merely import ASE made by the proposed system without using 3D authoring tools as shown in Figure 7. In addition, WRL format well known as Web3D standard can be employed in order to build Web3D content just by bringing an exported WRL shown in Figure 8.

Fig. 6. PDA application

Fig. 7. Game character applied

Fig. 8. Web3D content

Constructing a Convenient and Efficient System

31

6 Performance and Conclusion Figure 9 is the part of interface of this system showing the parameter control panel for generating emotions. The proposed system also empowers users to interact for altering of emotional levels by handling control bars shown in Figure 9 (a). Figure 9 (b) displays partial experimental face models with pictures taken by a PC cam. Delphi survey is used for evaluating the results by comparing real pictures with virtual counterparts and checking the similarity. The result was approximately 8 point out of 10 validating very promising. Figure 10 shows 3D full body formulated by BMI and Figure 11 displays 6 universal expressions created by PDA devices.

(a)

(b)

Fig. 9. System interface for generating emotions with parametric controls in (a) and examples of 3D faces

Fig. 10. Examples of 3D Full body formulated by BMI

32

S. Chin, J. Park, and M. Choo Happy

Angry

Surprised

Sad

Fear

Disgusted

Fig. 11. 6 universal expressions created by PDA devices Table 5. The results of system performance

Product FaceGen 3DMe Now Dantirum Zeb Qavata Proposed system

Generation of customized model O

Body CommerUsage of Execution time Facial implemen- cial availpicture (sec) expression tation ability O 3000 X X O

Output file format 3ds, VRML

O

O

430- ∞

X

O

O

3ds

X X

X X X

X X X

O O O

X X O

O

O

480 600 600-∞ less than 1(PC) 13-20(PDA)

O

O

O

ad-hoc 3D-based UCC ad-hoc VRML AKF ASE, VRML

△

Table 5 shows the result of comparing the performance characteristics of proposed system with commercially available systems, which have similar functions. The proposed system has several advantages over other similar systems by comparing functional features. Firstly, it provides simple interface and easy way to use for beginners. Secondly, it generates customized 3D whole virtual body. Third, the relatively short execution time is another significant advantage. Fourth, it provides the emotion generation and modification modules. Finally, the usage of the output is covered wide range of applications such as game characters, Web3D contents, and 3D modeling tools.

Constructing a Convenient and Efficient System

33

References 1. P.Ekman and W.V.Fresen, "Facial Action Coding System", Consulting Psychologists Press Inc., 577 College Acenue, Palo Alto, California 94306, (1978) 2. Frederic I. Parke, Keith Waters, Computer Facial Animation, A K Peters. Ltd, (1996) 3. S. Chin and J. Park, “A PDA based Personalized 3D Facial Expression System”, LNAI Vol. 4114, (2006), 468-475 4. P. Elkman and W. V. Friesen, Unmasking The Face : A guide to recognizing emotions from facial clues, Malor Books,Cambridge, MA, (2003) 5. V. Blanz, I. Albrecht, J. Haber, and H.-P. Seidel, “Creating Face Models from Vague Mental Images”,EUROGRAPHICS 2006, Vol. 25. Number 3. (2006) 6. Beier & Neeley, “Feature- Based Image Metamorphosis”, SIGGRAPH (1992) 7. S.Chin, & S. Kim, “Deriving Facial Patterns for Specifying Korean Young Men’s 3D Virtual Face from muscle based features”, LNCS Vol. 3309, (2004) 251-257 8. Y Cheong et al, "Correlation of Somatotype Drawing and Anthropomertic Values”, Journal of Korean Academy of Family Medicine Vol. 18, Vo. 9 Sep. (1997) 918-926 9. FaceGen : http://www.facegen.com 10. 3DMENow : http://www.biovirtual.com 11. Dantirum : http://www.idanty.com 12. Zeb : http://www.zeb.co.kr 13. Qavatar : http://www.qavatar.com

The Design and Implementation of VBS Liang Wang, Yue Qi, and Xukun Shen Ministry of Education Key Laboratory of Virtual Reality Technology, Beihang University, Beijing, 100083 {wangl,qy,xkshen}@vrlab.buaa.edu.cn

Abstract. Badminton is a fast and complicated sport under diverse occasions. It requires players to be swift, which make its simulation challenging. This paper proposes a sophisticated method to implement a Virtual Badminton System (VBS) using motion capture technique. We establish a database by collecting a variety of actions after analyzing and concluding basic badminton motions. Editing and controlling methods are adopted on motion sequences to simulate more occasions. Our system combines motions together by a high-level Finite State Machine (FSM) to describe the badminton logic of the virtual athlete, and create movements like the virtual athlete reacting automatically with a series of characteristic swings when interacting with user. Physical collision model is also added to enhance the immersion of our system.

1 Introduction The shuttle’s speed in badminton can be up to 288 km per hour, which is always faster than tennis and table tennis. In play, the shuttle’s flight trajectory is affected by factors such as velocity, height, distance, etc. Players need to strive to hit the shuttle back to the other side by changing their positions and moving properly. Due to its complexity and ever-changing properties, it is difficult to simulate and develop an interactive badminton system. There is a need to design and develop a system for user to play badminton in the project of Virtual Olympic Museum for the Games of the XXIX Olympiad in 2008. Users can enjoy the game, learn badminton laws, and get to know some standard badminton postures. In this paper, we present a motion-capture based solution which combines editing and controlling methods to reconstruct badminton according to its features. It must meet two requirements: remaining close enough to the data to keep the important original characteristics, while deviating sufficiently to accomplish the given task. These important characteristics include styles such as swing, hitting and movement. And the deviations allow for moving to a particular location, reacting reasonably to user interactions. The effectiveness of our approach is demonstrated with a virtual athlete that hits and reacts under variable occasions in play. Section 2 briefly reviews motion capture related work. We present the overview of our Virtual Badminton System (VBS) in section 3. The main components and methods in VBS are introduced in details in the following sections. We conclude the paper in section 7 by discussing our implementation. K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 34 – 43, 2007. © Springer-Verlag Berlin Heidelberg 2007

The Design and Implementation of VBS

35

2 Related Work Motion capture technique is popular. It encodes a specific performer of a special action in a fixed way which makes it difficult and expensive to record new motions. Researches mostly focused on topics of editing motions to fit it onto a given scene and to improve its reusability. Editing tools, including displacement mapping[1], warping[2], spacetime[3] and timescaling[4] edited data and generated seamless motions. Productions[5][6] enabled animators to blend dissimilar motions together and build relationship among them. However, these methods lack some of the qualities: the ability to present variations such as personality and style, and the potential for physically realistic motions and interactions. More researches are focusing on combining editing methods and controlling methods as we do to explore potential advantages of these two sets, thus to generate more complicated human motions, such as examples [7][8]. And Zordan[9] combined a physical simulation and a specialized search routine to generate a physics-based response transiting between two motion capture segments. An advantage of these methods is that they introduce physics to motions, rather than capture expensive new ones. To consider the problem of creating an interactive system of a dynamic model responding to diverse occasions, our challenge locates on how to make the model move with reasonable velocity along proper path, and hit the shuttle accurately.

3 Overview The main frame of VBS has 7 main components, shown is Fig.1. To accomplish them, the first step we should do is to figure out the badminton logic and its patterns. Then we apply editing tools to enrich the database. The FSM schedules the athlete to create behaviors that continuously acts. Badminton International Laws are also adopted to regular our game. Better results can be obtained by using physical collision model.

Fig. 1 . System overview

In view of shuttle flight trajectory, velocity and relevant positions to the athlete, we divide human actions into three sets: Wait action, Serve actions and Hit actions. Serve actions have forehand serve and backhand serve. Hit actions include ten sets to cover

36

L. Wang, Y. Qi, and X . Shen

most of the court region, such as Fore Left, Fore Right, Back Right, etc, shown in Fig.2. Note that each Hit actions is composed of continuous movements: waiting, moving to the target location, swinging, hitting the shuttle, and moving back to the central spot.

Fig. 2. Classification of virtual athlete’s motions. Leaves of the tree are labeled in pray

We use motion capture device (Vicon 512 of 12 cameras) to capture all the leaf data. Each leaf data includes BVH (Biovision Hierarchical data) format human data and relevant TXT(Text) format racket data. It means that each human sequence corresponds to one racket sequence.

4 Database in VBS When capturing motions, we simplify the racket as three points to form a plane. In this way, we can capture the racket’s information at every time step, shown in Fig.3. Then, we manually cut every original data clip and select a certain length of data frames which integrates motion features. Then we save it as a new one with a suitable identifier, and store it into the database. All the identifiers in our database are unique to conveniently retrieve a special motion.

(a) motion capture model;

(b) virtual athlete

Fig. 3. The three points in (a) on the racket circled in yellow are used to record positions and directions of the racket. Picture (b) shows the virtual athlete and his racket driven by motion capture data.

Due to the limitation of motion capture, the model in left picture can not simulate badminton in a real court and can only be driven to a given location. So, it causes

The Design and Implementation of VBS

37

deviations between simulations and real actions. For instance, the velocity of virtual athlete is slower than the shuttle’s. We offer an edit tool to modify data and locate virtual athlete in his side of court. This location is 1.5 meters away from the former serve line in the center and faces the user. We can see it in Fig. 4.

Fig. 4. Initial position and direction of virtual athlete in badminton game

5 Physically Based Motions Implementing a consistent VBS of physics to be more realistic and flexible is crucial to its success. To do so, however, commercial games always follow this: games should strive to be playable above all else. This does not mean values used in our system bear no relation to real-world counterparts, but to simplify physical parameters. We apply physical force and physical collision in VBS. Object motion seems still realistic through plenty of experimentations. Assume that the shuttle is only impacted by the gravity, its motion parameters at each time step is described in Fig. 5. Once we set the shuttle’s time in the air and its dropping position, the initial parameters can be calculated, and the shuttle’s flight trajectory can be determined. Then we can use the methods in chapter 6 to control the moving speed and path of the virtual athlete.

Fig. 5. Motion Analysis of shuttle at each time step

There are 4 types of collisions in VBS: shuttle vs. ground, shuttle vs. net, shuttle vs. racket, and player vs. net. We simplify the racket, net and court as planes, and the shuttle as a point which makes the shuttle trajectory as a certain line at each time step. Then, we use line-and-plane algorithm to do collision detection, shown in Fig.6. Our physical collision model consists of detection and reaction to enhance realism a lot, similar to the one[10].

38

L. Wang, Y. Qi, and X . Shen

Fig. 6. Line-and-plane collision

6 Motion Editing and Controlling Methods 6.1 Motion Editing Methods Badminton has special actions in moving, swinging and hitting. Approaches we applied add small adjustments on key frames to make sure the virtual athlete can follow a given path and hit the shuttle accurately with reasonable velocity. Here we use time scaling algorithm to control the velocity of the virtual athlete. This method ensures the athlete can move to the destination on time where the shuttle will drop. Time Data Clip 1 Data Clip 2

Fig. 7. Time scaling algorithm

A data clip can be considered as a time sequence of poses. Time scaling method resamples the data in time by interpolations to a greater or a less number of poses. We interpolate two motions pose by pose at each point in adjusted time. Points are between original poses at different points. An interpolation coefficient between zero and one is used to generate a consistent pose at any time in Fig.7. This method can control virtual athlete’s velocity and compensate the constraints caused by motion capture. Local time scaling and global time scaling varies in the interpolated time duration. We use linear interpolation to calculate each vector position component of the representation (are vectors to be interpolated, is the result) (1  u ) u a  u u b, u  [0,1]

c

(1)

For each joint orientation vector, we use spherical linear interpolation (are vectors to be interpolated, is the result) r W

sin((1  u )W ) sin(uW )  pu , u  [0,1], sin(W ) sin(W ) arccos(q < p )

qu

(2)

The Design and Implementation of VBS

39

In general, we obtain the interpolation coefficient by the linearity of the inputs with respect to the outputs in each dimension. Given target lying between grid points (We use the same coefficient in spherical linear interpolation). u

t l hl

(3)

An example in Fig.8 demonstrates the result of right shoulder when the virtual athlete is moving to the back part of court on his right. As we can see, the motion curve after modification also keeps the main features.

Fig. 8. Chart shows the right shoulder’s motion curve. The darker one is the original curve, the lighter one is the result after time scaling.

Capture data can only drive the virtual athlete to follow a fixed way while positions of the shuttle vary all the time in play. We use our path editing algorithm to adjust trajectory to ensure the shuttle is within his hit scope.

Fig. 9. Path editing algorithm

Path editing moves and rotates the root node’s 6 DOF parameters which present the position and direction of the virtual athlete globally, as illustrated in Fig.9. We define a key frame s , and apply the direction adjustment of

to the following

40

L. Wang, Y. Qi, and X . Shen

frames after s . Then we update all the new position and direction vectors with formula 4 and 5. Note that our system uses BVH format in which the rotation sequence is ZYX. Rn

T xc

tan 1 ( Rn (1,2) Rn (2,2) ),

T yc T zc JJG pn

Rc Ry ('T ),

(4)

sin 1 ( Rn (0,2) ),

tan 1 ( Rn (0,1) Rn (0,0) ), JJG JJG JJG ps  Ry ('T ) u ( pc  ps )

(5)

where ps , pc , pn are positions individually in key frames s, c, n . The variables three Rc , Rn correspond separately to the rotation matrix in frames c , n . The parameters θ x′ ,θ y′ ,θ z′ indicate the rotation components in frame n . This scheme allows one motion to fit several paths without redundant noises like foot skate, and generates a denser motion database from 13 clips to 47 clips which enriches responses for the virtual athlete. However, Fig.10 shows shortcomings caused after editing, like the fracture in framewhich cuts a motion into two.

(a) Original motion; (b) New motion after path editing Fig. 10. (a) shows the original motion sequences from motion capture device, (b) illustrates the result after path editing. A fracture can be seen in the circle.

The fracture in Fig.10 is caused by the large deviation after path editing where the data in frame s still keeps its original value, but not the same case in frame c, n. Here,we introduce displacement mapping method [1]. It adopts signal processing theory to locally adjust the signal for desire while keeping the continuity and global shape of the signal. After the modification, we eliminate the fracture locally and make the virtual athlete move in a new path seamlessly. At any time in play, the virtual athlete is driven by a certain sequence which requires to transit between sequences. However, the motions in database are discrete gestures which have posture and velocity deviations at the end and beginning of the

The Design and Implementation of VBS

41

segments. If we transit directly from one to another, it will cause problems like shaking or sliding, etc. In this case, we use motion transition[4] to avoid the problems. Transitions interpolate the end and beginning of the adjoining segments with a simple ease in/ease out ramp at the moment when interacting with user. We demonstrate with an example in Fig.11. In our system, we consider the proper motion delay time (0.1-0.2s) which makes virtual athlete’s response to the interaction much more realistic.

Fig. 11. The dotted curve shows the deviation between two motion sequences. Solid one is the result after motion transition which is smooth from the end of one to the beginning of the other.

6.2 Motion Controlling Method We build a high-level finite state machine to describe the badminton logic, and cycle by performing actions to generate variable response like swing or movements, seen in Fig.12.

Fig. 12. The high-level FSM in badminton system. The virtual athlete stays waiting firstly, when user hit back the shuttle, he will move and hit to finish the interaction. There are 21 synthetic swings in our database.

42

L. Wang, Y. Qi, and X . Shen

In section 4, we have established a motion-capture database with unique identifiers. Therefore, we build our high-level FSM based on states in accordance with identifiers. Assume the states of the virtual athlete are changed according to the alterations of the shuttle’s states. In process of interactions, the system provides inputs to the FSM according to the current situation of the shuttle, then use outputs to drive the virtual athlete to react. In this manner, we reach our purpose of controlling.

7 Discussion While a motion-capture driven character could accomplish many realistic actions, we should also acknowledge the shortcomings, such as its lack of interaction and the finite ability to adapt to the changes. We implement a system to make it possible to let user use mouse device to play badminton with a virtual athlete driven by data in motion-capture database. Physical force and collision model are also applied to simulate more realistic object motions and interactions like in the real world. The VBS can be used as a software tool for sport education. By using editing and controlling methods, we get plenty of similar motions to enrich the database and drive the virtual athlete under more variable occasions. We adopt the current international badminton laws [11]: (a). A match shall consist of two of three games. (b). Player who first wins two games will be the winner. (c). In singles a game is won by the first side to score 21 points. (d). Both the serving side and the receiving side can add a point to its score. (e). The side winning the score or the game serves first in the next serve or the next game. The system explores the function of tracking and moving user viewpoint to enhance interaction and immersion. Whenever and wherever the flying shuttle is hit back to user, the viewpoint will move simultaneously due to the shuttle. This function eases the interaction and lets user feel like it is himself who is moving in the virtual scene. The system result is shown in Fig.13.

(a) The virtual athlete selects proper response according to the shuttle to interact with user

(b) The virtual athlete has the right of next serve when user loses point. He serves, keeps waiting, and respond when user hit back the shuttle. Fig. 13. The system result

The Design and Implementation of VBS

43

Our approach contributes to improve motion-capture data reusability, and endow the virtual athlete with reaction ability. It is an important research achievement and a technical accumulation in the project of Virtual Olympic Museum. However, what we accomplish is only a basic collection of badminton, which does not include other actions like spiking, and jumping up to hit. This could be compensated through capturing more actions to implement a more diverse, immersive, and vivid badminton. Moreover, simulations in our system are about 10 times lower than those used in current commercial games which can be solved through further process. Acknowledgments. The authors would like to thank Virtual Reality Laboratory, Institute of Computing Technology, Chinese Academy of Sciences, for the motion capture equipment used. This work is funded by NSFC No. 60533070 of the "Virtual Olympic Museum Key Technologies Research and Implementation".

References 1. Bruderlin A., and Williams L.: Motion signal processing. In: ACM SIGGRAPH 1995, Annual Conference Series, Los Angeles, California (1995) 97-104 2. Witkin A., and Popovic Z.: Motion warping. In: ACM SIGGRAPH 1995, Annual Conference Series. Los Angeles, California (1995) 105-108 3. Gleicher M.: Motion editing with spacetime constraints. In: Proceedings 1997 Symposium on Interactive 3D Graphics (1997) 139-148 4. Wiley D. J., and Hahn J. K.: Interpolation synthesis of articulated figure motion. IEEE Computer Graphics & Applications 17, 6, 39-45 5. Kovar L., and Gleicher M.: Motion Graph. In: Proceedings of ACM SIGGRAPH 2002. San Antonio (2002) 473-482 6. Rose C., Guenter B., Bodenheimer B. and Cohen M. F.: Efficient generation of motion transitions using spacetime constraints. In: Proceedings of SIGGRAPH 1996, ACM SIGGRAPH (1996) 147-154 7. Oshita M., and Makinouchi A.: A dynamic motion control technique for human-like articulated figures. Computer Graphics Forum 20 (2001) 8. Zordan V. B., and Hodgins J. K.: Motion capture-driven simulations that hit and react. In: ACM SIGGRAPH Symposium on Computer Animation (2002) 89-96 9. Zordan V. B., Majkowska A., Chiu B., and Fast M.: Dynamic Response for Motion Capture Animation. In: Proceedings of ACM SIGGRAPH 2005, (2005) 24, 3, 697-701 10. David M. Bourg. Physics for Game Developers. O’REILLY (2002) 11. LAWS OF BADMINTON. [Online]. Available: 12. http://www.internationalbadminton.org/index.asp

View-Dependent Hierarchical Foliage Simplification Qingqiong Deng1,2 , Xiaopeng Zhang1,2 , and Marc Jaeger3 1

2

Sino-French Laboratory LIAMA, CAS Institute of Automation, Beijing, China National Laboratory of Pattern Recognition, CAS Institute of Automation, Beijing, China qqdeng, [email protected] 3 INRIA-Rocquencourt,Project DigiPlante, CIRAD AMAP, Montpellier, France [email protected]

Abstract. High compression of plant geometry is an important aspect in fast realistic visualization of plants. Hierarchical structuring plant morphology is a key factor for real time plant rendering, especially when pedestrian views, including both close-ups and far views, are requested. We present here a new geometric simplification method, called Viewdependent Hierarchical Foliage Simplification (VHFS). It aims to construct efficient multi-resolution models, faithful to botanical knowledge for sparse organs of trees, such as leaves, flowers and fruits. Both preprocessing and view-dependent rendering processes are considered hereby. In the preprocessing phase, sparse organs are simplified hierarchically with respect to the topological structure of the plant, i.e. to the plant branching order hierarchy and the phyllotaxy (or anthotaxy) groups. In the rendering phase, the simplification degrees for organs in different locations in the crown are defined from the current viewpoint. The selection of the different simplification levels is based on the distance to the viewer and a visibility coefficient of the considered organs. This visibility coefficient is an approximate occlusion based on the relationship between the viewing direction and direction of each first order branch. Compared with other foliage simplification methods, the main advantages of VHFS lay in the respect of consistent botanical structure at any stage of compression, ensuring realistic foliage appearance, higher efficiency in preprocessing stage and higher data compression.

1

Introduction

Vegetation is an important element in outdoor and indoor scenes. Fast and realistic plant rendering becomes a common request for numerous applications concerning computer animation, flight simulation, urban visualization, virtual environment, and entertainment. But real-time rendering of such scenes is a challenge since brute force geometry rendering far exceeds current CPU and GPU capabilities. Software acceleration techniques of rendering have thus been widely developed aiming to decimate the geometric complexity. A powerful way is the level of detail (LOD) technique. However, plants and trees are much different K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 44–55, 2007. c Springer-Verlag Berlin Heidelberg 2007


View-Dependent Hierarchical Foliage Simplification

45

from classical continuous objects from topological structure point of view. Classical model simplification approaches fail on sparse organs due to their primitive number, sizes and distributions. Organs and more precisely leaves are of interest; foliage often determines the appearance of a tree. Various approaches have thus already been developed for foliage simplification[1,2,3,4]. They can diminish the number of leaves in a crown while maintain its appearance. But there are still many spaces for improvements. First, almost all of these methods represent leaves of one tree using a single unique choice of detail. The only exception is the approach of Remolar et al. View Dependent Multiresolution Model for the Foliage (VDF)[4], where the selection of different resolutions of foliage geometric models of a single tree follows a simple criterion, such as the distance from the viewer or importance of the object in the scene. Second, methods do usually not take benefit of the plant topological structure; leaves are considered independent from each other, leading to high processing costs when considering possible polygon fusion for instance. Finally leaf distribution and more precisely occlusion are not considered either. We aim to contribute to the definition of efficient LOD plant foliage models using the underlying branch structure in order to render high realistic 3D natural scenes. The proposed approach extends past contributions of VDF. Leaves are first clustered on a local level, then on hierarchical levels defined from branch structures. Using a simple heuristic to evaluate foliage occlusion, we allow then the use of lower LOD models for occluded or partially occluded foliage parts. The implementation of the proposed approach is then illustrated on complex foliage of stand-alone plants and on a natural scene.

2

Related Work

Plant modeling has been extensively explored. At present, there are many mature systems. L-system[5], AMAP[6] and Xfrog[7] are the typical ones. In these systems, plants are usually designed on a strong topological base, mainly defined by the branching order. However, as the result of realistic plant modeling, the number of the polygons is often too big for a normal plant. So the representation of a small scene or forest with hundreds of plants is usually very huge. Thus, real time rendering of outdoor trees with original full geometric models is impossible. Acceleration techniques of rendering have been widely researched, and people have proposed many methods. The methods can be classified into three general categories: image-based, point-based and polygon-based. Image-based rendering is widely used due to its good property that its performance is independent of the geometric complexities of objects. There are various approaches in this field, such as billboard[8], billboard clouds[9,10], multi-layer Z-buffers[11], layered depth images[12], hierarchical bi-directional textures[13] and volumetric textures[14,10]. They are efficient for and can get fantastic results when rendering plants and trees at a distance. However, the methods need

46

Q. Deng, X. Zhang, and M. Jaeger

a huge memory to store the texture images. The parallax effects are usually bad, and artifacts are obvious on close views. Point-based rendering uses a point to represent a leaf when the projected size of the leaf on image space is smaller than a pixel[17,18]. It is efficient on plants and trees that are far from viewer, but require costly sampling stages otherwise. Polygon based rendering is the traditional and predominant rendering method. It has a well support of hardware acceleration and easy texture mapping. But the data complexity through polygon model is usually too high for fast rendering. There are many geometric simplification methods[15,16] dealing with polygon models. They usually work well on continuous manifold objects. However, when applied on trees, they may produce acceptable results for trunks, but will fail on foliage. Some specific approaches have been proposed for the simplification of foliage[1,2,3,4]. The Foliage Simplification Algorithm (FSA)[1] was proposed by Remolar et al. in 2002 for quadrilateral leaves, it diminishes the number of leaves but maintain the appearance of the crown by leaves collapse; - two leaves are replaced by a new larger leaf with similar position and shape as the original ones. This method has been improved in Progressive Leaves Union (PLU)[2] using more reasonable measurements of similarity to select the best leaf pair to implement leaves decimation. In addition, the PLU is view-dependent, all simplification information is recorded on the hard disk, and appropriate approximation models will be chosen for different viewpoints automatically. Hierarchical Union of Organs (HUO)[3] was recently developed on PLU to hierarchically and respectively simplify tree organs with respect to some basic botanical structure as phyllotaxy and anthotaxy. So that the botanical structure of the plant in the HUO algorithm is better kept for visualization than that of the FSA and the PLU, but all the topological structure is not fully applied. However, the three algorithms, FSA, PLU and HUO, do not discriminate the leaves that locate in different regions of the crown, thus their simplification degrees are uniform all around the crown. Remolar et al did propose a specific algorithm VDF[4]. But it needs to maintain an active list of visible polygons obtained by using a conservative visibility algorithm, and modify the active list taking advantage of temporal coherence when the user’s viewing position shifts, so it is costly both in computation time and storage.

3

Hierarchical Foliage Simplification

The idea of View-dependent Hierarchical Foliage Simplification (VHFS) is to build LOD models for sparse organs on the base of the underlying plant topological structure. This topological structure reflects the plant design process or growing process. It covers usually two levels: a local level and a global level. The local level defines the organ topology itself and its insertion on a given point on a branch for its distribution in the crown. The global level corresponds to the topology of all branches. There are four steps in VHFS. The first step is to construct organ clusters corresponding to the phyllotaxy structure, or the anthotaxy structure, for each plant.

View-Dependent Hierarchical Foliage Simplification

47

Fig. 1. Progressive union of leaves in a leaf cluster

The second is to simplify complex mesh model of each organ in each cluster to a quadrangle by using Multiphase Surface Simplification algorithm [19]. The third step recursively implements Progressive Union of Complex Organs [3] within each leaf cluster, or phyllotaxy group, until all the quadrangles in the leaves cluster are united as a single quadrangle, which is called as the representative quadrilateral of the leaves cluster (see Fig. 1). The forth step simplifies all representative quadrangles hierarchically according to the topology information of branches and the position of each leaf on a branch. VHFS is intuitive but very effective. In the following sections, we will introduce step two and step four in detail. 3.1

Simplification of Complex Leaves

Multiphase Surface Simplification method[16] is used to simplify the complex mesh of each sparse organ to a quadrangle. A leaf mesh is often topologically equivalent to a patch, and it needs nice boundary preservation also. For each leaf of the leaf class, or phyllotaxy group, we classify the vertices of a leaf mesh into three categories: inner, boundary and corner points. The definitions of inner points and boundary points are the same as that of ordinary mesh simplification methods, while a special attention is needed to pay only on the definition of corner points, which is fixed during the simplification. To get better preserved boundary, we determine two vertices as corner points for each leaf according to its feature. One is the vertex whose distance to the vertical-axis is the longest. Since the leaves of phyllotaxy group are arranged around the vertical-axis, it is the tip of the leaf. The other is determined through the symmetry property of the leaf, and it is corresponding to the vertex that connects leafstalk although the leafstalk is usually omitted in plant models. All the simplifications process and errors due to simplifications are recorded, where the error is defined as the one from a newly generated vertex to the old mesh.

Fig. 2. Leaf Simplification

Fig. 2 illuminates the simplification of a willow leaf to a quadrilateral with VHFS, where the red vertices are corner points. VHFS simplifies complex leaves smoothly.

48

3.2

Q. Deng, X. Zhang, and M. Jaeger

Hierarchical Simplification According to Topology of Branches

As the last step of foliage simplification, all representative quadrangles are simplified hierarchically according to the topology of branches. Each branch is evaluated with a level value at first. The level value of the trunk is set as 0. The level value of each branch born on a level-k branch is k+1. We also define a subtree as the set of all successor branches and associated leaves, and a k-subtree is a subtree located on a level-k branch. After simplifying all leaf clusters, or all phyllotaxy groups, to a quadrilateral, Progressive Union of Complex Organs[3] is performed inside each different subtree. The order of this performance is the subtree levels with upper levels first. If the maximal level of branches of a tree is n, the progressive organs union is applied first inside every n-subtree until only a single quadrilateral left, and then we move down to all (n-1)-subtrees. Such process will be recursively implemented until 0-subtree is reached and the whole foliage is united as one quadrilateral. The selection of the best leaf pair is usually the most time-consuming part in pre-processing. By simplifying organs hierarchically with respect to the botanical concept of phyllotaxy and topologic properties, VHFS is much more efficient in preprocessing than other existing foliage simplification methods. The reason is that in each step of decimation, the selection of the best organ pair is performed in a very limited scope, i.e., all organs grown on a branch. Fig. 3 shows some simplification steps of a 15-year-old holly tree, whose maximal branch level is 3. Fig. 3(a) is the original model; Fig. 3(b) is the result after simplifying each 3-subtrees to a single quadrangle; and the corresponding results of 2-subtrees and 1-subtrees are showed in Fig. 3(c) and Fig. 3(d) respectively. Using VHFS, we can simplify very big trees with a great number of leaf polygons in a few second. In Fig. 7, less then 5 seconds are used to simplify of a big tree crown, with more then 412 thousand foliage polygons. This tree will make all other existing foliage simplification methods, like HUO, much exhausted. In the end of preprocessing, all the simplification data and simplification process data are saved in the hard disk as a binary tree. In each step of organ union, the new generated organ is recorded as the father of the collapsed organs.

(a) Original

(b) 3-subtree

(c) 2-subtree

(d) 1-subtree

Fig. 3. Foliage simplification according to branch topology

View-Dependent Hierarchical Foliage Simplification

49

For each node of the binary tree, the corresponding error due to simplification is recorded also.

4

View-Dependent Multi-resolution Model

While rendering, the selection of the appropriate LOD foliage model is defined from a spatial error threshold. The binary tree is traversed until the recorded error values are higher then the threshold or until lowest detail description is reached. The spatial error threshold value is computed from both the distance and a foliage occlusion factor. In our implementation, distance from crown to viewer is defined as that between the tree bounding box center and the viewer. Nevertheless, in order to increase the rendering efficiency when leaves are far away from the viewer, lower details may be selected for leaves in different areas on the crown, based on the fact that some leaves held by ”hidden branches” of the crown are often occluded by the front foliage. The idea is to mimic some how the techniques used by traditional artists to paint trees. Painters use initially large rough strokes to paint the leaves that are hidden, and then add detail using finer strokes over them to represent visible leaves. The selection of the best leaf pair is usually the most time-consuming part of pre-processing. By simplifying leaves hierarchically with respect to the botanical concept of phyllotaxy and topologic properties, VHFS is much more efficient in preprocessing than other existing foliage simplification methods. The reason is that in each step of decimation, the selection of the best leaf pair is performed in a very limited scope, i.e., all leaves grown on a specific branch. The critical question is how to efficiently determine whether a leaf is visible or not. Assuming that the visibility of leaves belonging to the same 1-subtree are uniform, we do propose here a simple non-conservative method. It provides a fast approximation visibility factor for any leaf, avoiding the recording and traversing of the active list. This assumption makes sense since the main axis (the main branches, building the skeleton of the tree) does generally keep the same direction. Leaves of a 1subtree are rendered using a same LOD. It is also practicable, since the leaves of different 1-subtrees are simplified independently before the simplification of the 0-subtree. The proposed visibility value is therefore a simple dot product between the viewing direction and the 1-subtree direction, defined by its branch direction. To summarize, the leaves are rendered with models of different LODs controlled by the following special error threshold E(B), defined with equation(1) according to its branch direction:
[(C − d)/(C − 1) + 1]ε; if d ≥ C E(B) = (1) ε; if d < C where d = b · v; b is the normalized direction of the branch B; v is the normalized viewing direction; C is the constant used in our visibility criterion to classify leaves into visible and invisible categories; and ε is the spatial error threshold

50

Q. Deng, X. Zhang, and M. Jaeger

(a)Holly original (b)Holly simplified

(c)Maple original

(d)Maple simplified

Fig. 4. Branch improper visibility

d0, i is the number of focus number which satisfies the condition ): mˆ ( Ai)=δm(Ai) ，Ai ≠Θ

mˆ (Θ)=δm(Θ)+(1-δ)

(9) (10)

whereδ ∈[ 0,1] is context reliability factor after assessment according to specified case, ∑ mˆ (Ai) ≤1, mˆ is basic probability assignment , then E is called the

ˆ is mapped evidence of E. original evidence, and , E Let the function mass m (•) is a certain evidence information (context-aware information) E at the time-point t 0, then we can define the exchange form of the function mass: mˆ ( Ai, t)=ξ(t- t 0 )m(Ai) ， Ai ≠Θ mˆ (Θ, t)=ξ(t- t 0 )m(Θ)+[1-ξ(t- t0 )]

(11)

Where ξ(t- t 0) =δ f ( t - t 0 ),δ is reliability factor, f (t - t 0) is function of timeefficiency which is supplied by the expert of the special field of the interested object and can be tuned after being assessed. The form of this function of time-efficiency is various and changeable, in different field, the description may be different, such as subsection function, trigonometric function, and so on. An example of trigonometric function is: f(t - t 0)=|Sin(t - t 0)| an example of subsection function is: ⎧(t − t0 ) /(t1 − t0 ), t0 ≤ t ≤ t1 ⎪ f (t − t0 ) = ⎨ 1 , t1 ≤ t ≤ t2 ⎪(t − t ) /(t − t ), t ≤ t ≤ t 3 2 2 3 ⎩ 3

352

Z. Wang et al.

Where t 0 ~ t 3 is time point each, and the value of them may be determined by a certain condition or restriction rule. The time-efficiency factor ξ ∈[0, 1], ∑ mˆ (Ai， t)=1, mˆ is basic probability assignment funsion, mˆ is time-efficiency belief function of m. If t= t0, when ξ=1, the evidence of focus is the whole efficiency evidence, and m= mˆ is right. When ξ =0, the evidence of focus is invalid evidence, we can get mˆ (Θ, t)=1, which means the case is unknown totally, in another word, the belief degree is uncertain absolutely. Let the energy function ψ (E) of a evidence E can be defined as: ψ (E ) =

n(E )

∑

i =1

(12)

m ( A i ) / | A i |, A i ≠ Θ

Where Ai is the set of focus elements, | Ai | is the radix of A , i n (E) is the number of elements and their set of power, m(Ai)= mˆ ( Ai, t) / ξ (t- t 0) , Ai ≠ Θ , m(Θ)=( mˆ (Θ, t) - [1-ξ(t- t0)])/ξ(t- t 0), ξ(t- t 0)≠0, mˆ ( Ai, t), mˆ (Θ, t), ξ(t- t 0) is defined above. If the function mass m1 (•), m2 (•) are basic probability function of two evidences E1,E2, Their focus element is Ai, Bj, respectively, obviously, some focus elements of E1 and E2 may be relative, and the relativity degree is decided partly by the number of focus element and its basic probability assignment. For example, E1 = {A，B，AB}, | Ai |=2, n(E1)=3. E2 = {B，C，D，BC}, | Ai |=3, n(E1)=4. E1∩E2 = B}, E1∪E2 = { A，B，C，D，AB，BC}, | Ai |=4. So we define the relative degree as follows: The coefficient of relativity μ 12 (which is from E1 to E2), μ 21 (which is from E 2 to E1) is defined, respectively:

μ12 = 1/2ϕ(E1，E2) ψ (E2)/ ψ (E1), μ21 = 1/2ϕ (E1，E2)ψ (E1)/ ψ (E2)

Where ϕ ( E 1，E 2 ) is the relativity degree of evidence E1 and E2 which can be computed as:

ϕ (E1，E2)=2 ψ (E1，E2) / (ψ (E2) + ψ (E1))

5 Computing of Aware Context Based on EDS Let the mass function m1 (•), m2 (•) are basic probability functions of two evidences E1,E2 in the space U, {Ai} and {Bj} are its set of focus elements ,then the context fusion computing method considering context relativity is as follows: ∧

m (A) =

∑

Ai ∩ B

m 1' ( A i ) m

j

= A

' 2

(B

j

),

A ≠ Φ ,Θ

(13)

∧

m (Φ ) = 0 ∧

m (Θ ) = (

∑

Ai ∩ B

m 1' ( A i ) m

j

=Θ

' 2

(B

j

)) + η

Approach Comparison on Context-Aware Computing with Uncertainty

353

Where ⎧ m 1 ( A i )( 1 − μ 12 ), A i ≠ Θ ⎪ m 1' ( A i ) = ⎨ 1 − ∑ m 1' ( A i ) , A i = Θ ⎪⎩ Ai ⊂ Θ

' 2

m

⎧ m 2 ( B j )( 1 − μ 21 ), B j ≠ Θ ⎪ ( B j ) = ⎨1 − m 2' ( B j ) , B j = Θ ∑ ⎪⎩ B j⊂ Θ

∑

η =

Ai ∩ B

m 1' ( A i ) m

j

=Φ

' 2

(B j)

In the following, we give the fusion computing method of n evidences under considering context reliability. Similarly, suppose the function mass m1 (•), m2 (•) ,…mn(•) are basic probability function of n evidences in the space U , the mapped function is mˆ 1, mˆ 2, …, mˆ n, respectively, then the context computing method mˆ is: ∧

m (Φ ) = 0 ∧

m (A) = c

−1

∑

∩ Ai = A ∧

m (Θ ) = (

∑

∩ Ai = Φ

∏

1≤ i ≤ n

∏

1≤ i≤ n

(14)

m i' ( A i ) , A ≠ Φ

m i' ( A i ) ) + η

Where

⎧ m i ( A i )(1 − μ i ( n − i ) ), A i ≠ Θ ⎪ m ( A i ) = ⎨1 − ∑ m i ( Ai ), Ai = Θ ⎪⎩ Ai ⊂ Θ η = ∑ ∏ m i' ( A i ) ' i

∩ Ai = Φ 1 ≤ i ≤ n

c =1-

∑

∩ Ai = Φ

∏ m (A )= ∩ A∑≠ Φ ∏ m (A ). ' i

1≤i≤n

i

i

' i

i

1≤i≤n

6 Tests and Evaluation As an experimental example of our active space, suppose determining a person’s identity is based on computing two kinds of context information tracked by face recognition agent and voice recognition agent, then decide the identity of the person and track his/her activity. For 200 persons, if reliability factor of the voice recognition agentδ1=0.8, reliability factor of the face recognition agentδ2 =1. According to the gathered the voice of 200 persons, the decision result of identity made by the voice recognition agent is m1({S，Z})=0.875, which means the probability p of S or Z for

354

Z. Wang et al.

the person’s identity determined by voice recognition agent is 87.5%. But according to the collected image information of 200 persons by camera, the decision result of the person’s identity by the face recognition agent is m2 ({S})=0.9, which means the probability p of S for the person’s identity is 90%. If taking the approach ((13), (14)) based on EDS mentioned above, the activitytracker agent computes the probability about the person’s identity, and the process of computing and the results are as follows: mˆ 1({S，Z})=δ1m1({S，Z})=0.8*0.875 = 0.7, mˆ 1(Θ)=1- mˆ 1({S，Z})=1- 0.7

= 0.3, mˆ 2({S})=δ2 m2({S})=1*0.9=0.9, mˆ 2(Θ)=1- mˆ 2({S}) =1 - 0.9 = 0.1.

The mass (probability) for decision of the person’s identity and the multiplication of intersection set have been given, each item is from multiplying by the intersection item. When the mass is known, we can get them together as follows:

ˆ mˆ 2({S}) = 0.63 + 0.27 = 0.9 m3({S}) = mˆ 1 ⊕ ˆ mˆ 2({S，Z}) = 0.07 m3({S，Z}) = mˆ 1 ⊕

probability of the person is S

probability of the person is S or Z

ˆ mˆ 2(Θ) = 0.03 m3(Θ) = mˆ 1 ⊕ probability of the person is uncertain Where m3 ({S}) expresses the probability of the person‘s identity is S. If taking the approach (7) based on Random Set Theory mentioned above, we are generating aggregate statistics for a group of objects in context-aware computing. Because there is addition probability in the m3({S，Z}) and m3(Θ) , which means the addition information about S or Z, both for S or Z, the addition probability is 0.1, we can determine that probability of S is 0.9, so the probability region of S is [0.9，1], that is to say, the belief of S about the person’s identity is more than 90%. Based on this computing result and a decision rule of threshold, we can decide the person’s identity is S. The computing result is consistent of our experiences and no conflict, so we can believe the efficiency of context-aware computing approach based on RST. In experimental examples, with the increase of random finite sequence, the mean error ratio will decrease, which is from 0.247% to 0.089%, our result is shown this relationship, one curve is based on RST mentioned above by us, another curve is based on Evidence Theory method, which is from 0.298% to 0.125%. Based on comparison, the advantage of RST is apparent. The reason is that Evidence Theory method is depended on energy function that is selected by the user, so it is with additional uncertainty, but Random Set method is based on statistics with principle of applied mathematicians, so additional uncertainty is avoid. From the comparisons in Fig.1, we can see that RST is the most efficient, but Fuzzy Set is worst. The reason is that Fuzzy Set Theory method is depended on membership function selected by the user, so it is also with additional uncertainty, and it is larger then EDS, but RST method is avoid. Of course, the fusion computing approach of context-aware computing for active service is different, which can lead to different efficiency. By comparisons, as we know, the more validity of new context-aware computing approach based on RST than EDS with uncertainty information has been tested successfully.

Approach Comparison on Context-Aware Computing with Uncertainty

355

0.3

Error ratio (%)

0.25 0.2 0.15 0.1 0.05 0 2

4

6

8

10

12

14

16

Number of sensors

RST

EDS

Fuzzy Set

Fig. 1. Comparison result of RST, EDS and Fuzzy Set

7 Related Works In pervasive computing paradigm, In order to realize attentive seamless service, many researchers [3-5] have recognized that context-aware process with uncertainty must be considered. The expressing and fusing method for context information is general, such as first-order probabilistic logic, Bayesian Network, D-S Evidence Theory, which can permit the aware context be understood by different process module or agent [8-12]. Mori [4] studied context-aware computing with uncertainty based on probability model with Bayesian network. Kastella [3] and Mahler [6] and studied the application of Random Set theory in information with uncertainty. Saha [5] used classic Dempster-Shafer Evidence Theory in context-aware process with multi-sensor track fusion. In D-S Evidence Theory, there is a belief degree function mass to process the combination computing, which is more freedom than traditional Probability Theory, that is m(Θ) may not be 1, if X⊆Y, m(X) may not be less than m(Y), meanwhile , m(X) and m(X′) may not have a certain amount relationship. But the sensed multi-source data as dynamic evidence context-aware information is with noise and uncertainty, the application in fact requires high reliability, we must consider context reliability factor during context-aware computing, it means if the classic D-S Evidence Theory is used as context-aware computing method and reasoning theory, we must modify it. Paul Castro [7] studied reasoning of context parameters and relative state based on Bayesian network, the shortcoming of Bayesian network is slow. Zhang [8] adopted D-S method of Evidence Theory during the study of seamless migration.

356

Z. Wang et al.

8 Conclusions In order to solve the attentive service problem of pervasive computing paradigm, we have studied context-aware computing during seamless mobility based on RST and EDS. In this paper, we focus on modeling and computing of aware context information with uncertainty for making dynamic decision. Our insight is to deal with dynamic context-aware computing with RST and EDS. We have reexamined the formalism of random set, argued the limitation of the direct numerical approaches, given new modeling mode based on RST for aware context, proposed our computing approach of modeled aware context, extended D-S Evidence Theory after considering context’s reliability, time-efficiency and relativity, enumerated experimental examples of our active space and given the evaluation. Based on the comparisons, the validity of our new context-aware computing approach for ubiquitous active service with uncertainty has been tested successfully. In fact, our approach is abstract, so it can be used in many domains.

References [1] D G Zhang, Zhao H. Random Set Theory Applied to Electric Fault Fusion Forecast in Monitoring System of Hydropower Plant [C]. The 4th information fusion international conference, Motreal, 2001, 8:ThC1-11. [2] Fisher J. Fast JPDA multi-target tracking algorithm[J]. Appl.Opt.28 (Jan.1999):371-375. [3] R. Mahler. Random Sets as a Foundation for General Data Fusion. Proc. the Sixth Joint Service Data Fusion Symposium, Laurel, 2002, pp. 357-394. [4] Reid D B. An Algorithm for Tracking Multiple Targets [J]. IEEE Transaction on Automatic Control, 1999, AC-24(6):843-854. [5] Saha F T, Chang T C. An efficient algorithm for multi-sensor track fusion[J]. IEEE Trans. [6] Aero-space Electron. Systems, 1998,34(1): 200-210. [6] S. Mori, C.-Y. Chong and R.P.Wishner, “Tracking and Classifying MultipleTargets without A Priori Identification,” IEEE Transaction on Automatic Control, Vol. AC-31, No. 5,1998, pp. 401-409. [7] Paul C, Richard M. Managing context data for smart spaces [J]. IEEE Personal Communications, 2000, 10:44-46 [8] Degan Zhang, Guangyou Xu, Yuanchun Shi. Extended Method Of Evidence Theory For Pervasive Computing [J]. Chinese Journal of Computer, July, 2004 (in Chinese). [9] Xu Ling-yu. Applicaion of Fuzzy Fusion to Accident Forecast in Hydropower station, Proceedings of The Second International Conference on Information Fusion, Vol 2,1999. [10] C.L. Morefield, “Application of 0-1 Integer Programming to Multi-target Tracking Problems,” IEEE Trans action on Automatic Control, Vol. AC-22, June, 1997, 302- 312. [11] I. R. Goodman, R. P. S. Mahler, and H. T. Nguyen, Mathematics of Data Fusion, Kluwer, 1997. [12] S. Musick, K. Kastella, and R. Mahler, “A Practical Implementation of Joint Multi-target Probabilities,” SPIE Proceedings, Vol. 3374, pp. 26-37, 2005. [13] Donald J. P, Lin L, Dieter F, Inferring High-Level Behavior from Low-Level Sensors. Ubicomp2006.

Implementation of a Web-Based Hybrid Educational System for Enhancing Learning Efficiency of Engineering Experiments Dongsik Kim1, Kwansun Choi1, Sunheum Lee1, Changwan Jeon1,and Jiyoon Yoo2 1

Department of Electrical and Communication System Engineering, College of Engineering, Soonchunhyang University, Korea 2 School of Electrical Engineering, Korea University, Korea {dongsik,cks1329,sunheum}@sch.ac.kr, [email protected]

Abstract. To enhance the excellence, effectiveness, and economical efficiency in the learning process, we implement a creative hybrid educational system for engineering experiments where web-based virtual laboratory systems and distance education systems are properly integrated. In the first stage, we designed client/server distributed environment and developed web-based virtual laboratory system for electrical/electronic circuit experiments. The proposed virtual laboratory system is composed of three important sessions and their management system: concept-learning session, virtual experiment session, assessment session. With the aid of the management system every session is organically tied up together to achieve maximum learning efficiency. In addition, every activity done during the virtual laboratory session are recorded on database and will be provided to the learners as the printout report included their own experimental information and results. With this new system structure, the learners can compare theoretical and experimental data; develop their capability in designing and analyzing the circuits; and make use of auxiliary educational tools for understanding complicated concepts. In the second stage, we have implemented efficient and cost-effective distance laboratory systems for practicing electrical/ electronic circuits, which can be used to eliminate the lack of reality occurred during virtual laboratory session. The use of simple and user-friendly design allows a large number of people to access our distance laboratory systems easily. Thus, self-guided advanced training is available even if many expensive types of equipment will not be provided in the on-campus laboratories. During distance laboratory session, the learners change the settings of the input signal and the values of circuit elements, observe the output waveform from the web camera, and obtain image information regarding their experimental results. The proposed virtual/distance laboratory systems can be used in stand-alone fashion, but to enhance learning efficiency we integrated them and developed a creative hybrid education system for engineering experiments. Our hybrid education system provides interactive learning environment and a new approach for the delivery of engineering experiments. Keywords: Web-based Virtual/Distance Laboratory System, Web-based Hardware Control Technologies, Multimedia Contents, and Java Applets. K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 357–368, 2007. © Springer-Verlag Berlin Heidelberg 2007

358

D. Kim et al.

1 Introduction In addition to enhancing traditional educational methods, information technology (IT) can also enable new ways of education delivery and innovative pedagogic strategies. Teaching is no longer confined to a time and a place. The time and physical boundaries of the traditional classroom are stretched to a learning space. A growing number of universities worldwide are now offering virtual education problems. Several companies are also providing online training for their employees. A simple search on the World Wide Web will result in hundreds of sites offering virtual courses or resources for developing and delivering such courses. Electrical/electronic experimental study is a very important component in engineering education. It not only acts as a bridge between theory and practice, but also solidifies the theoretical concepts presented in the classroom. In the classical approach, most of electrical/electronics experiments performed at real on-campus laboratories are accompanied by a complete manual, a detailed guideline for design and simulation steps, experiment procedures and a presentation of the technical report. Before the laboratory session, the learners should re-enforce basic concepts, prepare some design and simulation steps, and acquire a clear idea on what they should expect from the experimental work they will be carrying out in the laboratory. At the laboratory session, the learners are required to assemble the circuits, connect the equipment, make the measurements, compare the data to the expected behavior, and deliver a partial or complete report to the professor at the end of the session. This classical way of experimenting clearly has the following shortcomings. • The classroom lectures or the handouts are generally not sufficient for the learners to be fully prepared for a hands-on experiment or to appreciate the significance of the previously explained theory in the experiment to be performed. • When the learners are passive observers or a semi-active part of an experiment, they will understand neither the correspondence nor the difference between theory and practice. • When practicing electrical/electronic circuits on virtual world, the learners feel the lack of reality. It is necessary to build bridge between virtual world and real world so that they can understand real experiment procedure easily during virtual laboratory session. To cope with some difficulties we implement a creative hybrid educational system for engineering experiments where web-based virtual laboratory systems and distance education systems are properly integrated. We are focusing our initial efforts on electrical/electronic experiments for the reinforcement of undergraduate teaching, which is taken by most undergraduate engineering students. In the first stage, we have implemented virtual laboratory system, which provides the learners with improved experimental methods. If the learners have access to the virtual laboratory system through signing up procedure, they can acquire the fundamental concepts on the related experiment and make a virtual experiment on basic electrical/electronic circuits according to the guided experiment procedures. Equipped with theoretical knowledge acquired by executing flash animations and Java applets, the learners can easily understand the important principles and the

Implementation of a Web-Based Hybrid Educational System

359

significance in the experiment to be performed. All of these activities will be carrying out in the virtual laboratory system by clicking the menu buttons in it and filling out some text fields to change the values of experimental components. The proposed virtual laboratory system is composed of three important sessions and management system: Principle Study Sessions, Virtual Experiment Session, Assessment Session, and Management System. With the implementation of the proposed virtual laboratory system, it has become to intensify the work during the laboratory session and to provide the learners with better understanding of the significances related to the electrical/electronic experiments. In the second stage, we have implemented an efficient and cost-effective distance laboratory system for practicing electrical/electronic circuits, which can be used to compensate the absence of reality, occurred during virtual laboratory session. The use of simple and user-friendly design allows a large number of people to access our distance laboratory systems easily. The proposed virtual/distance laboratory systems can be used in stand-alone fashion, but to enhance learning efficiency we integrated them and developed a creative hybrid educational system for engineering experiments. We developed communication system, prototype of control module, and control panel under graphic user interface environment to combine the virtual laboratory system with the distance laboratory system. Our hybrid educational system provides interactive learning environment and a new approach for the delivery of engineering experiments. Our hybrid educational system is designed to support from elementary electrical experiments to advanced electronic experiments included in the curriculum of electrical engineering. Every activity done during the virtual laboratory session is recorded on database and will be provided to them as a printout report form included their experimental information and results. The educators check the printout form turned in to estimate how well they understand the experimental contents and methods during virtual laboratory session. In Section 2, our virtual laboratory system for practicing electrical/electronic circuits is described. In Section 3, our distance laboratory system and hybrid educational system are described. In Section 4, the conclusions are discussed.

2 Web-Based Virtual Laboratory System 2.1 Implementation of Virtual Experiment Equipment Since our virtual laboratory system is implemented to describe the real on-campus laboratory, widely used experimental equipments such as oscilloscopes, multimeters, function generators, and power supply etc. are implemented by Java applets. All activities done in the virtual laboratory will be carrying out by clicking or dragging the mouse and filling out some text fields to change the values of experimental components. In addition, by previewing how to use some equipment on the Web the learners can reduce their waste time and labor during real laboratory session.

360

D. Kim et al.

2.1.1 Virtual Analog Multimeter Widely used multimeters can be divided into two types: digital and analog. Though digital type multimeter is more common, we implement an analog type multimeter for engineering educational purpose. The implemented multimeter has two separate scales to avoid complicatedness (Fig. 1). Ammeters and Voltmeters are read from left to right and have a linear scale. However, Ohmmeters are read from right to left and have a nonlinear scale. As shown in Fig. 2, the range selector switch selects the function to be used by dragging the mouse to desired range and releasing.

Fig. 1. DCV, A and ACV scales

Fig. 2. Range selector switch

2.1.2 Virtual Function Generator The virtual function generator is composed of four main parts: frequency adjustments by mouse dragging, waveform mode part, frequency adjustment by slide menu or choice box, and amplitude adjustment by dragging as shown in Fig. 3. Output

(a)

(c)

(b)

(d)

Fig. 3. (a) Frequency adjustment (b) Waveform mode (c) Frequency adjustment (d) Amplitude adjustment

Implementation of a Web-Based Hybrid Educational System

361

frequency range of generated signals is 1Hz to 1MHz and amplitude range is 2V to 20 V peak-to peak. 2.1.3 Virtual Oscilloscope Oscilloscope is very important equipment in the area of engineering experiments. We implemented a virtual oscilloscope that allows signal voltages to be viewed on the Web, usually as a two-dimensional graph of one or more electrical voltages plotted as a function of time. Our virtual oscilloscope is implemented to describe typical commercial oscilloscope in view of some functions. It is not the same as commercial oscilloscope, but it is good enough to get educational effects in the area of engineering experiments. The virtual oscilloscope is composed of three main parts: oscilloscope body, control box, circuit box. Once the learners click on the button “CH1” or “CH2”, virtual marker and available marker position in the circuit box will be displayed together as shown in Fig. 4(a). If the markers are placed correctly on the circuit, its corresponding output waveforms will be displayed on the oscilloscope body. Clicking on the button “ZOOM” in the oscilloscope body, the learners can observe output waveforms in details on the separate window. Information that the markers have will be transferred when clicked on “ZOOM”. The Java source code makes the markers store frequencies, voltages, currents, phases associated with their

(a)

(b) Fig. 4. (a) Virtual oscilloscope and its structure (b) A detailed window when clicked “ZOOM”

362

D. Kim et al.

positions in the circuit. The detailed oscilloscope window is shown in Fig. 4(b), where several functions such as voltage, time adjustment, trigger control, cursor adjustment, summing. Furthermore, output waveforms can be stored, loaded, and printed out by clicking the buttons in the “STORE” box. 2.2 Virtual Laboratory System for Electrical/Electronic Circuits The web-based virtual laboratory needs, in general, various interactive multimedia components such as Java Applets, Flash animations with useful educational effects. In order to achieve this goal, we suggest that our virtual laboratory include three important sessions and management system for effective experiments on the World Wide Web. The material of our virtual laboratory system is appropriate for advanced courses on electrical/electronic circuit experiments. Each course consists of 15 chapters and each chapter comprises the Principle Study Session to explain the concepts and theories of circuit operations, the Virtual Experiment Session to provide the learners with making virtual experiments on several electrical/electronic circuits. The Management System assigns the username and password to the eligible authorized persons and provides printout service for all information about the experiment done in the Virtual Experiment Session. 2.2.1 Principle Study Session The Principal Study Session is responsible for making the learners understand the concepts and theories of the circuit operations included in each chapter. Interactive flash animations with creative and intuitive ideas for each subject lead the learners to understand them easily. For example, Fig.5 shows an interactive Java Applet for understanding the important concepts of JFET characteristic curves. The learners can easily understand the related concepts to the JFET characteristic curves by increasing/decreasing VGG by mouse clicking.

Fig. 5. Java applet for JFET characteristic curves

2.2.2 Virtual Experiment Session The Virtual Experiment Session provides virtual experimental environment to the learners. Widely used experimental equipments such as oscilloscopes, multimeters,

Implementation of a Web-Based Hybrid Educational System

363

function generators, and power supply etc. are implemented by Java Applets. During this session, the learners can build circuits for each subject, set the values for each circuit element, and measure voltages or currents etc. using several types of the virtual experimental equipment. When finishing the virtual experiment on the Web, they can print out all information related to their virtual experiments and submit it as their preliminary report to the educators in their on-campus laboratory classes. For example, Fig. 6 shows a captured image of a Java applet for virtual experiment on JFET transconductance and drain characteristics.

Fig. 6. Java applet for virtual experiment on JFET characteristics

A virtual experiment is performed according to the following procedure: (1) Assembling and connecting the circuits (2) Applying input voltages (3) Making the output measurements (4) Transmitting experimental data to the database (5) Printing out the preliminary report as shown in Fig. 7(a)-(d). The learners build a given circuit by placing proper circuit elements from “ELEMENT CHOICE” menu. With this menu, the learner can select circuit elements and change their types or values. In Fig. 7, VDD is set to have 7.6[V]. They can change the value of DC power supply by double-clicking the DC power supply symbol. In addition, they can insert a voltage and/or current markers into the circuit by using “MEASURE” menu. The learner can also measure several outputs for the various values of Vcc using the virtual oscilloscope. In this paper, the virtual experiments are designed to be performed for the fixed structure with variable circuit parameters. Therefore, with the virtual experiment equipment, the learners make virtual experiments for the fixed circuits which are shown in the experiment workbook, but they can change circuit parameters (R, L, and C), values of voltage sources, and selection of outputs. Though the virtual experiments are not arbitrary, we can get enough educational purpose for electrical/ electronic circuits. 2.2.3 Assessment Session It is very important to provide the educators with useful information on experiments done in the virtual laboratory by which the educators evaluate how well the learners are doing. Every activity done during the virtual laboratory session is recorded on the database and will be provided to them as the printout form included their

364

D. Kim et al.

(a) Assembling and connecting a circuit

(c) Making the output measurements

(b) Applying input voltages

(d) Transmitting experimental data

Fig. 7. Procedure of making virtual experiments

experimental information and results. The educators check out the submitted printout form to estimate how well the learners understand the overall experimental process. The management system supports communications between the educators and the learners in the ways mentioned above, and different setups for each learner. Good instructional development is an iterative process by which the educators and the learners perform formative assessments and summative evaluations to improve a course continually. Effective instructors use a variety of means, some formal and others informal, to determine how much and how well their students are learning. In addition, in order to show the validity of our virtual laboratory system we investigated the damage rate of real experimental equipment during class and assessed student performance on the five quizzes for one semester. The students were divided into two groups: Group 1(G1) not using the virtual laboratory system, Group 2(G2) using the virtual laboratory system. The students also were asked to evaluate the virtual laboratory environment in terms of process effectiveness, degree of interactivity, and enjoyment. More specifically, for our virtual laboratory environment the students in Group 2 had to rate on a 5-point Likert scale their level of agreement with the following statements. • The virtual laboratory system was effective in supporting my learning method. • The virtual laboratory system provided me with the appropriate level of interactivity with the real experiment. • I enjoyed using the virtual laboratory system to learn.

Implementation of a Web-Based Hybrid Educational System

365

As shown in Table 1 we have obtained several affirmative effects such as reducing the damage rate of real experimental equipment, and increasing learning efficiency. The results of our survey show strong evidence of the superiority of the virtual laboratory environment over the classical on-campus laboratory environment. In addition, we can conclude that the virtual laboratory environment enables the learners to interact not only with the learning material but also with the educators. Table 1. Between-group comparisons on the virtual laboratory system

G1 G2

Damage Rate of Real Equipment

Average Score of Five Quizzes

Process Effectiveness

Degree of Interactivity

Enjoymen t

25.4% 4.8%

64.5% 81.4%

N/A 4.31

N/A 4.01

N/A 4.13

Group1 : The students not using the virtual laboratory system Group2 : The students using the virtual laboratory system Scale: Strongly disagree 1 2 3 4 5 Strongly agree

3 Web-Based Distance Laboratory System Due to high cost of on-campus laboratories at the universities with a large number of students, much interest in the web-based virtual laboratory has been drawn. However, there are still some drawbacks: (1) the virtual laboratory environment lacks feeling of reality (2) it is very difficult for the virtual laboratory systems to describe the exact and precise operations of real experimental devices. To cope with these drawbacks, we have implemented an efficient and costeffective distance laboratory system for practicing electrical/electronic circuits, which can be used to compensate the absence of reality, occurred during virtual laboratory session. During distance laboratory session, they change the settings of the input signal and the values of circuit elements, observe the output waveform from the web camera, and obtain moving image information regarding their experimental results. Thus, self-guided advanced training is available even if many expensive types of equipment will not be provided in the on-campus laboratories. 3.1 Structure of Distance Laboratory System Our distance laboratory system is composed of Local Measurement System (LMS), Data Processing Server (DPS), and Remote Measurement Server (LMS). The operation of our distance laboratory system is largely divided into channel selection of Data Acquisition (DAQ) board and transmission of measured data. Once the learners have chosen the desired channel at the RMS, information on the selected channel will be transferred to the DPS and the LMS sequentially. The LMS with DAQ board starts measuring the experimental data and then makes the measured data transferred to the DPS and the RMS sequentially. Fig. 8 shows structure of the proposed web-based distance laboratory system and its related data flows. LMS plays

366

D. Kim et al.

important roles to measure several experimental data from the distance laboratory and then transmit the data to the DPS. The LMS is composed of data acquisition module, screen control module, socket communication module.

Fig. 8. Structure of the web-based distance laboratory system

DPS plays important roles to transmit the data obtained from the LMS to the RMS. The DSP is composed of client socket module, user access module, and data communication module. We implemented RMS to provide every user in the distance laboratory with overall experiments in progress through the web camera. The RMS is composed of text/image management module, server connection module, message transmission module, and data acquisition module. 3.2 Some Experiments for Electrical/Electronic Circuits Distance laboratory for electrical/electronic circuits is composed of three main parts: experimental circuit part, experimental equipment part, and web camera part. Through the web camera, the learners observe the circuits to be performed, The experimental circuit part is connected to the DAQ board to get experimental data and a GPIB controller card is installed to control experimental equipment properly through PCI bus. The web camera takes pictures the circuits to be performed and experimental equipment to offer the learners in real-time. During measurements of the experimental circuits, the web camera takes the pictures and transfers to the learners by using socket communication programs. The socket program in the RMS side transfers the experimental data measured during a distance laboratory session to the socket program in the client side. The transferred data will be made as moving images and be viewed to the learners. Thus, without the presences in the real on-campus laboratory, the learners can conduct their distance experiments with the moving image viewed. To make experiments in the distance laboratory for electrical/electronic circuits the learners need to sign in with their user IDs and passwords to the distance laboratory. Once signing in has been completed, they can observe the moving images of the circuits to be performed and experimental equipment. In addition, they can check out

Implementation of a Web-Based Hybrid Educational System

367

the real experimental results from the RMS and can change the channels they want to observe. Fig. 9 shows a distant experiment for the RLC series circuit. Note that a little time delay in data transmission is occurred as shown in Fig. 9(c). In general, the more the time is occurred, the more channels the learners select. This problem can be solved if we design the distance laboratory system by connecting the commercial database.

(a)

(b)

(c) Fig. 9. (a) Distance laboratory for RLC series circuit taken by the web camera (b) Output waveforms for RLC series circuit measured by an oscilloscope (c) Output Waveforms measured by the LMS(Left) and RMS(Right)

4 Conclusions An efficient virtual laboratory system with creative and interactive multimedia contents was implemented, which can be used to enhance the quality of education in

368

D. Kim et al.

the area of electrical/ electronic circuit experiments. This new and innovative structure will be used for eliminating the difficulties of classical experimental system. With this new system structure, the learners can compare theoretical and experimental data; develop their capability in designing and analyzing the electrical/electronic circuits; and make use of auxiliary educational tool for understanding complicated concepts. The results of our survey show strong evidence of the superiority of the virtual laboratory environment over the classical on-campus laboratory environment. Therefore, we can conclude that our virtual laboratory system enables the learners to interact effectively not only with the learning material but also with the educators. Furthermore, we have implemented efficient and cost-effective distance laboratory systems for practicing electrical/electronic circuits, which can be used to eliminate the lack of reality occurred during virtual laboratory session. Although main objective of our distance laboratory system is the reinforcement of experiment education on basic electrical/electronic circuits, it also can be extended to different engineering fields in similar methods. The proposed virtual/distance laboratory systems can be used in stand-alone fashion, but to enhance learning efficiency we integrated them and developed a creative hybrid educational system, which makes two types of experiments for electrical/electronic circuits. Our hybrid educational system will be used for the teaching of undergraduate and postgraduate courses. In teaching, the possibility of anytime anywhere access allows undergraduate students who are not able to finish the real experiments to continue to work on the subject after the formal laboratory session on their own time through the Internet. Acknowledgments. This research was supported from the professor support program of LG Yonam Foundation in 2004. I would like to thank LG Yonam Foundation.

References 1. Mohamed K. and Rinky L.: Web-based Learning: Effect on Learning Process and Outcome, IEEE Transactions on Education, Vol. 45, No. 4, (2002) 350-356 2. Ahmet B. K. and Okyay K: An Internet-Assisted Experimental Environment Suitable for the Reinforcement of Undergraduate Teaching of Advanced Control Techniques, IEEE Transactions on Education, Vol. 44, No. 1,(2001) 24-28 3. Dongsik K. and Kiwoon K.: A Web-based Virtual Experiment Kit for Digital Logic Circuits Using Java Applets, Journal of Engineering Education Research, Vol. 6, No. 2, (2003) 34-39 4. Denise C. and Antonio C. S.: A Modern Approach to Teaching Basic Experimental Electricity and Electronics, IEEE Transactions on Education, Vol. 44, No. 1,(2001) 5-15 5. Kazutake K.: Distance Real Laboratory using Internet at Kumamoto University, International Conference on Information Technology Based Higher Education and Training, Istanbul, Turkey (2000) 264-267 6. Sam H., Bassem A., and Mohammad I.: A Java-based Remote Laboratory for Distance Education, International Conference on Engineering Education, Taipei, Taiwan, (2000) 14-16, 7. Gillet D., Salzmann C., and Crisalle O.D.: Advances in Remote Experimentation, 19th American Control Conference, Chicago, Illinois, U.S.A. (2000) 2955-2956 8. Roehrig C., Jochheim A. :The Virtual Lab for Controlling Real Experiments via Internet, Proceedings of IEEE International Symposium on Computer-Aided Control System Design, Hawaii, USA (2003) 279-284

ImaginationTOOLST M : Made to Play Music Francesca Bertacchini1 , Eleonora Bilotta2 , Lorella Gabriele1 , Vincenzo Mazzeo1 , Pietro Pantano3, Costantino Rizzuti1 , and Stefano Vena1 1

Longo&Longo s.a.s. c.da Lecco, 87036, Rende, CS, Italy Department of Linguistics, University of Calabria, Ponte P. Bucci, Cubo 17/b, 87036 Arcavata di Rende, CS, Italy Department of Mathematics, University of Calabria, Ponte P. Bucci, Cubo 30/b, 87036 Arcavata di Rende, CS, Italy {bilotta,lgabriele,piepa,costantino.rizzuti}@unical.it, {francesca bertacchini}@yahoo.it, {stefano.vena,vmazzeo}@gmail.com http://www.imaginationtools.it

2 3

Abstract. In this paper we present ImaginationTOOLST M a new musical software useful to generate, analyze and synthesize sounds using mathematical models based on the Dynamical Systems and the Theory of the Chaos. The system allows to build and explore acoustical and musical concepts in a manipulatory way, creating network of object called TOOLS which behave as intelligent agents of different kinds (input, output, processes). By joining a specific agent with other agents, the user builds emergent acoustical/musical 3D graphs, both in synchronous and in asynchronous ways. This process of building musical compositions in a 3D environment in real time embodies the manipulation of objects and gives to the user the possibility to create, by using a process of auditory feed-back, aesthetical choices which well adapt with his/her musical ideas, so reducing the complexity of music composition. We think that the innovative characteristics of the software can really facilitate and improve the musical learning. Keywords: Music education; Collaborative learning; Modeling, Signal analysis, synthesis, and processing; Artificial, augmented, and virtual realities; Three-Dimensional Graphics and Realism.

1

Introduction

The advent of the computer and the progressive application of technologies in every field of the human life have determined great changes. In short time, the new technologies have been adopted in various fields, from educational, to musical, to medical one and so on. In the 1956 Lejaren Hiller and Leonard Isaacson completed the first musical composition generated by a computer: “Illiac Suite” for string quartet. This was the first step of a new form of artistic expression now known as computer music. K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 369–380, 2007. c Springer-Verlag Berlin Heidelberg 2007 

370

F. Bertacchini et al.

Computer music is an application of science, computer technology and electrical technology in which music is synthesized electronically, orchestrated using computer software, and played through an audio reproduction system. It allows to simulate the sounds and duplicate the orchestration of past musical compositions and performances. In addition new orchestrations are simply accomplished using computer software, and the creation of new instrumental sounds and structures becomes possible [1]. It also refers both to the study of how technologies can improve musical cognitive processes (from composing, to performing to listening), modelling the generation algorithms on the results obtained by studies on psychological performances with humans and on defining which are the design principles in building up augmented software/hardware environments that can be used by musicians in order to extend their potential. The consequence is that Computer Music is changing the process of music making. As a result of the progressive technological development and the increasing of computation capability of computers, Personal Computer are used for the sounds synthesis and elaboration sounds, first in deferred time than also in real time. There are a lot of efforts in creating new ways of composing either by using specialized score processing as in FINALE (www.codamusic.com) or through integrated score applications in Audio/MIDI software sequencer such as Cubase (www.steinberg.net) or Sonar/Cakewalk (www.cakewalk.com). These software have been developed trying to reproduce on PC the a real record studio environment (Cubase is based on the VST-Virtual Studio Technology) and adding a series of Plug-Ins (external components of the program that allow to synthesize, or to play sounds -Virtual Instrument- or to elaborate sounds - Virtual Effects). Numerous software and Plug-Ins for sound synthesis exist, such as Reason (www.propellerheads.se), B4 (www.native-instruments.com), Halion (www.steinberg.net/155_1.html). They implement different techniques of synthesis and supply a series of useful instruments for the generation and new sound experimentation or finalized to emulate the analogical instruments sound, such as Moog synthesizer or Hammond organ. But these software, even if are powerful and flexible, often present a great limit: they are closed programs that implement technical of sound synthesis and elaboration at an high level that do not allow to the user to manipulate the sound in a direct way. Powerful and flexible programs exist, generally with very complex functionality, that allow to realize systems for sound generation and elaboration guaranteeing an enormous versatility. These software packages are often based on a LabVIEW-like interface, from Max/MSP (www.cycling74.com/) to other programming environments such as the Audicle [2], interfaces for musical expressions (www.viktoria.se/fal/events/mobilemusic/2005.html), from the work of Machover [3] BrainOPERA at the MIT Media Lab, to the React-Table (mtg.upf.edu/reactable), which is a state of the art of the Tangible User Interface (TUI).

ImaginationTOOLST M : Made to Play Music

371

The computer applications for music in regular educational contests can represent a great advantage. Side by side, at the intelligence tests development were developed musical ability tests in order to assist music teachers in the selection of those pupils considered gifted musicians [4]. Recently, the concept of musical ability has been severely criticised [5], [6]. Skills can be developed through playful practice and playing in groups, not only through deliberate practice. Social factors such as parental support, teacher’s personality, and peer interactions have also been shown to be more important than amount of practice time in achieving a high level of musical performance [7]. Perhaps, is more correct to talk about music education/training, rather than only about musical ability. The contribution of technology can really support and encourage musical ability and musical passion. It comes of primary importance the kind of technologies, interfaces and representations to use. Some of the most important studies on effectiveness of music software in educational contexts reports positive implications for every experimentation which used composing software [8]. We think that users can obtain wider benefits using enjoyable, involving and learnable musical software. According to this consideration we are implementing a musical software, ImaginationTOOLS trying to create a metaphor in which the user is immersed in an emotional environment, with a rich quantity of objects, places and educational activities. This paper is organized as follows: after the introduction in section 1, we describe the software environment and the activities the user can do in this software in section 2, follow a short compare among Max/MSP and ImaginationTOOLS in section 3 and finally, the conclusions.

2

ImaginationTOOLS: The City of Imagination

ImaginationTOOLST M embodies a new philosophy of emergent musical software; it presents a 3D graphical interface. The landscape of the City of Imagination is a very nice place on the sea, with some buildings which seem to be hanging from a rock. In front of this place, a sky-line made by Chua’s attractors closes the view on the explorable world. In the software are presents several environments that offer to the final user different form of entertainment and interaction: from the basic functions of editing, manipulation and resonant reproduction, to the intermediate functions that preview the use of scripts already done and present into the libraries, to advanced functions involving the experimentation of contemporary science models through the simulation and realization of intelligent programs that allow to artist-experimenter to verify his own hypotheses and eventually to change them. We are implementing a musical software in which users can acquire specific competencies about scientific research and to test their own hypotheses, until the creations of programs and simulations that incorporate these laws. The main idea is to return at the scientist-artist experimenter who can find in a software environment the possibility of experiment new sonorities.

372

F. Bertacchini et al.

In The City of Imagination (Figure 1 shows a picture of the world) are present four typologies of environments: – – – –

The The The The

Edutainment Environment; Laboratory; Virtual Immersive Room (IVR); Computer Supported Collaborative Music (CSCM).

Fig. 1. The City of Imagination landscape

2.1

The Edutainment Environment

Actually, in the Edutainment Environment we have inserted a museum organized as a real immersive museum [9] in which it is possible to visit a collection of Chua’s attractors, to listen the music and sound they produce and to visit an artistic gallery on the same topic. The external structure of the museum is a building made by Chua’s attractor, undergone at a process of complexity reduction especially the external texture (Figure 2 shows the building). In this place, it is possible to manipulate patterns coming from Chua’s dynamical systems, discovering the fractal nature of these objects. Currently, two prototypes of the Museum exist; the first prototype is made up in Java3D (java3d.dev.java.net) and the other in C++. The first prototype of the museum is a small Applet that will be insert into the website: it was been necessary to implement a 3D engine to allow the interactive navigation from keyboard and to manage the collisions. The second prototype of the museum has been developed using the graphical open source engine Irrlicht (irrlicht.sf.net) that incorporates an advanced 3D engine and a complete system of interaction. The museum has been thought as a virtual museum

ImaginationTOOLST M : Made to Play Music

373

exhibit where users can navigate inside the environment, can interact with some pedagogical agents and with the exposed objects. Users can listen to and read the information about concepts of scientific models and music by using hands-on activities. Furthermore, an agent guide the users in the city tour.

Fig. 2. In the figure you can notice the internal of the museum

2.2

The Laboratory

The Laboratory (Figure 3 shows the Laboratory building), the heart of the software, is an interactive 3D environment. Users can produce sounds and music by manipulating some objects, called Tools, developed to carry on specifics works like sound generation, signal processing, MIDI data elaboration. The Tools are shown in the 3D software environment through small spheres of different colours. Users can arrange a series of Tools, in according to some simple rules, joining together by using cylinders to create a network of musical elements, which can be played together. Each Tool can have inputs, outputs and parameters that can be modified by users or others Tools, in order to create digital signal. Finally, users can listen to this digital signal or analyse it graphically through an oscilloscope and/or a spectroscope. The agents (Tools) linked in a network (Figure 4 shows an example) allow to create sounds and music. The Laboratory is structured on three layers: audio, graphic and interactive (Figure 5 shows the different levels). The audio engine is a particular library self-contained and self-organized that maintains trace of all Tools present in the laboratory and manages the rendering resulted from their interaction and is based on an internal Plug-Ins engine.

374

F. Bertacchini et al.

Fig. 3. In the figure is possible to admire the Laboratory environment

Fig. 4. In the figure it is possible to see an example of agents linked in a network

The graphical engine is based on Irrilicht, an open source library that allows to manage the 3D graphic; it is a wrapper for low level functions offered by OpenGL and Microsoft DirectX; it has an internal rendering engine and it is cross-platform. The graphical engine is the same as the one used for the Museum upgraded under some aspects improving the interaction of the objects from a physical point of view. Spheres (that represent Tools) are connected ones each others according to the physical particle-springs [10] model in which the spheres are particles positive charged while each cylinder contains a spring. The final result will be an

ImaginationTOOLST M : Made to Play Music

375

Fig. 5. Laboratory’s implementation layers

auto-organizing environment: the spheres will be positioned at the maximum distance possible in order to ensure a good visibility of the 3D structures. The interactive engine is an interface between the graphical and the audio engines: it allows, in fact, to insert new tools into the environment (associating them a sphere), to manipulate Tools’ parameters and to join them together. The manipulation is also allowed by a StarSlider that lets to modify the parameters (also in real-time) by standard input devices (mouse) and by advanced input devices like gloves. The linking phase is done in two steps: the first, insert a cylinder into the environment that links spheres graphically; the second let the user to route the digital signal between Tools by the audio engine. All the Tools, developed to carry on specifics works like sounds generation, signal processing and so on, are realized in the software architecture as PlugIns allowing the extention of the software capabilities in an easy manner. The Plug-Ins development process was planed realizing a progressive improvement of the program functionalities starting from the primitive functions of the sound synthesis. At the beginning, we have defined a list of about 70 Plug-Ins we want to implement using ImaginationTOOLST M ; obviously this list represents only a starting point and it will expanded both with the increasing of the software complexity and with the appearing of new requirements in the sounds and music creation process. We have already realized numerous Plug-Ins, among which twelve generators implementing both traditional sound synthesis primitive (Sinusoidal Oscillator, Squared Waveform Generator, Triangular Waveform Generator, Saw tooth Waveform Generator, White Noise Generator, Impulses Generator with Limited band, Impulses Generator with spectral contents dynamically controlled, single Pulse generator) and generator coming from the chaos theory (Logistic’s Map, Henon’s Map, Chua’s Oscillator). We have also created a Metronome Plug-In generating a time-clock signal useful for Plug-Ins synchronization. We have also realized six operators Plug-Ins realizing basic operation between signals (Sum, Signals sum, Subtraction, Multiplication, Division, Mixer) and three functions Plug-Ins (Four stage linear envelope generator: attach, decay, sustain, release; Linear Interpolator, Delay line). In order to realize signal processing functionalities we have started realizing three simple filters (One pole

376

F. Bertacchini et al.

low pass filter, Two poles filter, Biquad filter) we are also working in developing Plug-Ins implementing standard sound effects and audio editing functions (Delay, Echo, Flanger, Chorus, Dynamics Processing). The last category of Plug-Ins we are developing, called renderer, contains audio and video output and functionalities that allow debugging the Tools’ network (Audio Output, Oscilloscope, Spectroscope, Inspector). 2.3

The Virtual Immersive Room

A 3D graphical environment allows go further on the classical interaction edge moving through the Virtual Reality. The Immersive Room foresees the use of advanced technologies [11]: including computers, interactive gloves, Head Mounted Displays and stereoscopic visualizations able to simulate an immersive environment, that makes virtual objects and events perceptible as the real ones. Users can access into a room in which many psycho-acoustical experiments will be carried out, from the sensation of being immersed in a multidimensional sound environment. Users can select, from a list of sound patterns, two or three multidimensional sound organizations coming from different Chua’s attractors in order to explore the creation of complex sonorous scenes, which have never been heard. While users are listening to these sound scenes, they can access to multimedia contents related to the same Chua’s attractors. 2.4

The Computer Supported Collaborative Music (CSCM)

In the City of Imagination will be inserted a seamless environment for remote and synchronous musical collaboration [12], [13]. Participants in the performance are situated in remote locations, both in the City of Imagination or outside in other places, and the interaction occurs synchronously. The communication architecture can be multi-user. Musicians will have the opportunity of collaborate synchronously over distance, creating a collaborative composition synchronously or artificial musical universes or a jam session or record and retrieve musical pieces, in an environment in which they share compositions and make experiments on the emergence of musical structures.

3

Max/MSP Versus ImaginationTOOLST M

We implemented some simple sound synthesis technique both in Max/MSP and in ImaginationTOOLST M . In this way, we would to compare the degree of complexity of the software. The first example concerns the generation of a pure sinusoidal tone. In ImaginationTOOLST M we have create a Sin oscillator PlugIn with none input and only one output (Figure 6 shows the Sin oscillator scheme and the implementation with both software). By using two parameters, amplitude (A) and frequency (F), it’s possible controlling the generation process of the sinusoidal tone. The ImaginationTOOLST M implementation is very similar to the technical scheme because we don’t need any control structure for the sinusoidal oscillator

ImaginationTOOLST M : Made to Play Music

377

Fig. 6. The Sin oscillator’s technical scheme and its implementation with both software

and we need to use only the audio out to send the signal to the sound card. Otherwise the objects provided by Max/MSP haven’t control parameters so, it must use suitable structures to realize the needed control. In the second example, we have implemented a simple additive synthesis technique. In ImaginationTOOLST M we have create a Mixer tool keeping a certain number of signals as input and sending the signal’s sum to the output (Figure 7 shows the additive synthesis technical scheme and the implementation with both software).

Fig. 7. The figure shows the additive synthesis scheme realized with both software

The third example we have implemented concerns the Amplitude Modulation and the Tremolo (Figure 8 shows the technical scheme and the screenshots of the examples realized using both the software).

378

F. Bertacchini et al.

Fig. 8. The figure shows the technical scheme and the screenshot of the Amplitude Modulation realized with the two software

The fourth example we have implemented is the Frequency Modulation (Figure 9 shows the examples).

Fig. 9. The figure shows an example of Frequency Modulation realized with both software

In figures 6, 7, 8 and 9 it is possible to notice how adopting parameters inside the Plug-Ins simplifies in a considerably way the sound process implementation. The above mentioned figures, in fact, show that the Tools of network realized in ImaginationTOOLST M are conceptually simpler and more similar to the technical scheme than those realized with Max/MSP. This important aspect makes ImaginationTOOLST M a program easier to learn and to use for inexpert user,

ImaginationTOOLST M : Made to Play Music

379

since employing parameters, each one provided with default condition, let users create simple configurations of tools able to produce immediately sounds in a fast way.

4

Conclusions

The development team of ImaginationTOOLST M is implementing an appealing software, which embodies new science models and it offers to the user the possibility of learn difficult concepts and methods in an easy and enjoyable way. User will have the chance to interact with the software using not only the classical keyboard and mouse, but also advanced technologies such as interactive gloves, Head Mounted Display and stereoscopic visualization. So, the user will have an immersive sensation, he/she can make music using the most surprising activities and the above mentioned advanced media technologies, obtaining sound and music which satisfy his/her aesthetical ideas and creativity. To achieve this goal, the team is adopting a User-Centred Approach to design, which mixes creativity and efficient methods and above all tries to look at real user requirements [14]. The website of ImaginationTOOLST M is available at the following address http://www.imaginationtools.it

References 1. Stanionis, V. A and Berberich, H.: Computer music science and technology of a new art. Stony Brook, NY: Research Foundation of SUNY (1989) Alma Mater Studiorum University of Bologna, August 22-26 2006 2. Wang, G., Misra, A., Cook, P.: Building Collaborative Graphical interFaces in the Audicle. International Conference on New Interfaces for Musical Expression, Paris, France, June 4-8 2006, http://hct.ece.ubc.ca/nime/2005/2006.html. 3. Machover, T.: Dreaming a New Music. Chamber Music, Vol 23, N 5, (2006), 46-54 4. Hallam, S.: Conceptions of musical ability. ICMPC9 Proceedings, Alma Mater Studiorum University of Bologna, August 22-26 (2006) 5. Doxey, C., Wright, C.: An exploratory study of children’s music ability. Early Childhood Research Quarterly, 5 (1990), 425-440. 6. Tempesti, L., Calabretto, R., Canizza, S.: Technological instruments for music learning. Stony Brook, NY: Research Foundation of SUNY (1989) Alma Mater Studiorum University of Bologna, August 22-26 2006 7. Moore, D., Burland, K., & Davidson, J.: The social context of music success: A developmental account. British Journal of Psychology, 94, (2003) 529-549 8. Higgins, W.. Technology. In Colwell, R., (Ed.): Handbook of Research on Music Teaching and Learning, New York: Music Educators National Conference Schirmer, (1992) 480-497 9. Lepouras, G., Vassilakis, C.: Virtual museums for all: employing game technology for edutainment. Virtual Reality (2005) 8: 96-106 10. Cruz, I. F., Tamassia, R.: Graph Drawing Tutorial. URL: www.cs.brown.edu/ rt/papers/gd-tutorial/gd-constraints.pdf 11. Pan, Z., Cheok A. D., Yang, H., Zhu, Shi, J. J.: Virtual reality and mixed reality for virtual learning environments. Computers & Graphics 30 (2006) 2028

380

F. Bertacchini et al.

12. Bligh, J., Jennings, K., & Tangney, B.: Designing Interfaces For Collaborative Music Composition. International Conference on Multimedia, Image Processing and Computer Vision. Madrid: International Association for the Development of Advances in Technology (pages 218-222) April 2005 13. McCarthy, C., Bligh, J., Jennings K,, Tangney B.: Virtual collaborative learning environments for music: networked drumsteps. Computers & Education 44 (2005) 173195 14. Ghaoui, C.: Encyclopedia of Human Computer Interaction. Idea Group reference Publisher, (2006)

PIGP: A Pen-Based Intelligent Dynamic Lecture System for Geometry Teaching Yuanyuan Liu, Qiang Lin, and Guozhong Dai Institute of Software, The Chinese Academy of Sciences, Beijing 100080, China [email protected], [email protected], [email protected]

Abstract. Computer-based lecture presentation systems have been widely used in classroom. Yet the teachers can not edit the slides in class according to the actual classroom feedback, which goes against active learning. Although Ink annotation can to some extent make up for lack of interactions, the geometry teachers’ requirements for freely drawing graphics and showing dynamic geometric figures in class can not be met. We propose a pen-based intelligent dynamic lecture system for geometry teachers, with which teachers can not only add annotations with ink, but also demonstrate dynamic graphics with pen gestures and draw graphics with sketches. A kind of Rule-And-Context-Based mode switch method is adopted to intelligently speculate the users’ intentions to make sure that using Ink, Gesture and Sketch together do not confuse the users. An informal evaluation has been setup and the result seems quite positive.

1 Introduction Research on Pedagogy indicates that active learning could remarkably enhance students’ learning efficiency. And abundant interactions can give students participant feelings and, hence, benefit students’ active learning and promote teaching effects. Since (at least) 1970s, computers entered classrooms as teaching assistants. Computer-Aided Teaching has been of great benefits to teachers and students. The electronic lecture systems, such as Microsoft’s PowerPoint, Apple’s Keynote and so on, are used widely, which is often considered as the biggest technology revolution in classrooms [1].However, both of them can only provide static slides in the project mode, which just include naïve, uneasy ink annotation functions to edit the slides in the project mode. More and more researchers have recognized that such slides make the students keep in passive receiving, not active thinking, because the students could not get any response from their teachers and are not involved in the lectures, since teachers can not interact with their students in time. [11] Besides, the interactive techniques based on keyboard and mouse can not adapt well to the situation in classrooms. Differently, pen-based interaction has some advantages of writing easily, drawing freely, taking up just one hand, and so on. Hence, pen-based interaction is particularly suitable for lecturing in class, since there is less text input, more sketches and more operations in class, and since the teacher do not have much time to sit on a chair in class in order to attract his students. As a result, to substitute keyboard and mouse with pen in classrooms and to add pen-based ink annotation in K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 381–390, 2007. © Springer-Verlag Berlin Heidelberg 2007

382

Y. Liu, Q. Lin, and G. Dai

electronic slides is a good idea so that unchangeable slides’ disadvantages could be made up temporarily. Unfortunately, it is far away from enough, especially for geometry teachers. Above all, graphics are the main objects for them to teach in class. It is too hard for humans to draw a straight line in hand writing because, in electronic slides, teachers have no rulers to use. Hence, pens’ digital ink is not suitable for geometry teachers to use to draw graphics in class. Moreover, Geometry is always considered quite hard for middle-school students since it is difficult to understand a large number of abstract geometry transformation theorems. Yet, Dynamic Geometry can help students to enhance their comprehensions [15]. Much experience in this field has indicated that dynamic geometry graphics which can be transformed according to geometric constraints are very helpful because they can obviously help students to understand the geometric objects’ relations. Students not only can see the graphics but also can see how they get together and how they can transform. Therefore, active slides with dynamic geometry, which can be changed in project mode, are very helpful for lecturing, especially for geometry teachers. Aim at these two issues, this paper introduces a kind of electronic lecturing system, PIGP, Pen-based Intelligent Geometry Platform, which implements a kind of intelligent dynamic slides. In this system, even in the project mode, with just a pen, the teacher can write words on the slides, draw graphics, and promote gestures to operate the graphics to transform, move or delete. In the remainder part of this paper, we provide a using scenario to demonstrate the system’s functionality and characters. Next, this paper introduces the method of intelligently switching modes and recognizing gestures and sketches. After that, there is an evaluation report of our system.

2 Related Work Many researchers are working on adding pen’s digital inks into the slides in the project mode. BIRD Note-taking System [2], and Classroom Presenter [3, 4] can both be used with PowerPoint and add annotations with pens freely in the project mode. And Golub built a system named Tablet Mylar Slides [5, 6] to simulate the traditional transparency, which is totally based on pens’ digital inks. Besides, Classroom 2000 [7] and Electronic Whiteboard [8] also use ink annotation. However, all of those systems do not do any recognizing work when taking the digital inks. Digital inks are totally meaningless for them. In our system PIGP, digital ink annotation is supported. Not limited to ink annotations in project mode, we also do more work in processing and recognizing the digital inks. In addition, there are some systems used in different subjects which can recognize and process the digital inks. Jiang et al have completed a system, ChemTeach [9], which can support to input Chemical Equations in the project mode. Multimodal integration methods are used in this system to recognize the digital inks. However, in this system, digital inks can only be treated as text information inputs. In our system, digital inks can be treated not only as writing words, but also as drawing graphic items and making gestures to operate the existing items. Besides, Rbeiz et al [10] have built a Classroom Learning Partner, extended from Class Presenter through making it support

PIGP: A Pen-Based Intelligent Dynamic Lecture System for Geometry Teaching

383

ink recognition. However, it just supports to recognize a small group of characters and symbols, not including any sketches. In our system PIGP, we not only provide a group of gestures for the users to operate, but also recognize a group of sketches to draw graphs.

3 Using Scenario “A Geometry teacher had prepared a lecture for teaching the relative knowledge of Circles before class. He draws a circle and a Triangle (Fig.1), before class, because he wants to teach a theorem that an angle at the circumference keeps right as long as the angle has the diameter as base.

Fig. 1. Figures in a prepared slide

“In class, when the teacher teaches at this slide, a student questions this theorem because he could not believe that wherever the vertex of the angle is, the angle is always right. Then the teacher tries to draw an arc between two lines to mark an angle, shown in Fig2 (1). An angle is marked in the screen, shown in the Fig2 (2). Then the students see with their own eyes that the angle BAC equals 90 degrees. “Afterwards, a student brings forward another question, “all of the right angles with the diameter as base are on the circumference, aren’t they?” To give him an impressive response, the teacher points on the Point A with his pen and drags it. The Angle BAC is changing with the drag gesture. (Fig2 (3) are three snapshots of the changing process.) The students find that the vertex must on the circumference whenever the angle is right, or else the angle is not right. Through this interactive process, the students get to understand the theorem more deeply. ”

(1)

(2)

(3)

Fig. 2. Some Snapshots of Active Slides in the project mode (The black line is the trace of the pen which is recognized as a sketch)

384

Y. Liu, Q. Lin, and G. Dai

The goal of our system is (1) to help geometry teachers give their instructions in class more effectively through dynamic slides and (2) to help the students understand the complex geometry knowledge more easily. After all, Geometry is a special subject which has complex graphs and relations. It is often hard for middle school students, especially for the students who just begin to learn Geometry. For the first one, our system, PIGP, can support interactive modifications with gestures, sketches and ink annotation in the project mode. For the second one, our system provides dynamic graphs which can vividly display geometry transformation process to enhance students’ comprehensions. In the scenario above, obviously, students could get immediate feedback from teachers, and they can involve in the lectures more and think more independently than passively listening when the slides are just static. Educational research findings suggest that such active strategies can enhance the lectures’ effects remarkably [1]. At the same time, the teachers can use the dynamic geometric graphics to show abstract geometric theorems to help the students understand.

4 Gesture and Sketch Process The job of Gesture and Sketch’s process is divided into three parts (shown in Fig 3). One part, called collector, is to log the pen’s movements and to collect the context information from the UI environment (Part 1 in Fig 3). Another one, called switcher, is to intelligently switch between the gesture and sketch modes (Part 2 in Fig 3). And the last one, called recognizer, is to recognize the gestures and geometric sketches according to the context information and its rule knowledge base (Part 3 in Fig 3).

Fig. 3. Recognition Work Process

4.1 Pen-Movement Input and Information Collection The main task of the collector is to get the users’ operation information. Pen is the main input device of this system. Hence, no matter what mode the system is in, the edit mode or the project mode, the pen’ s movements and some other attributes are logged. The useful information for the following recognition task includes all the points on the

PIGP: A Pen-Based Intelligent Dynamic Lecture System for Geometry Teaching

385

pen’s trace, which are organized as a stroke, the speed, the pen’s angle of inclination and so on. Besides, in order to intelligently switch between the gesture and sketch modes, this model is also in charge of collecting the environment and context information of the strokes. The environment and context information includes the existing geometric objects in the slides which have been drawn by the users, the current state of the system and so on. Additionally, the number of the selected geometric objects at that time and their types are very important context information because the selected objects are often treated as potential operation objects. The stroke and its context information are real-timely collected by the collector and sent to the switcher in time. 4.2 Rule-and-Context-Based Mode Switch In order to provide teachers abundant interactive functions in this system, we unite Gesture, Sketch and Ink annotation together. However, it is possibly confused for the users to have too many gestures and sketches for different functions to remember. Hence, we design a switcher (Part 2 in Fig 3) to automatically complete the mode switch according to the context information and a group of pre-defined rules. Besides, the users can also manually set the system’s state by a tap gesture on a corresponding button. Since the ink annotation is usually used all through the lectures, strokes are treated as ink annotation as the default mode unless some of the rules in the knowledge base are met. The majority of the Rules are about the environment when the system gets the stroke. We define a group of conditions which can cover all the possible cases of the stroke and its context and environment, and then construct some of them to define a group of rules. Take the following conditions as examples: (X is any existed geometric item.) C1(x): the item(x) is selected. C2(x, y): the stroke starts on one item(x) and ends on one item(y). C3(x): the item starts a geometric item (x). C4(x): there is a tap gesture on the drawing button before. We can define the following two rules with the conditions above as examples: Rule 1: The stroke starts and ends on one geometric, which is expressed as:

∃x∃y ( C2 ( x, y ) ∧ C3 ( x ) ∧ C3 ( y ) ∧ ¬C1( x ) ∧ ¬C1( y ) )

If this rule is met, the stroke is judged as a gesture. Because if the user put down his pen on an unselected geometric item, such as a point, a circle and a line, we suppose the user would like to operate this geometric item. In our system, point, circle, and line are defined as basic geometric items, which can be operated separately. Rule 2: The user had done a drawing tap gesture before this stroke, and this stroke does not start or end on any geometric item, which is expressed as:

C4 ∧ ∀x ( ¬C2 ( x, y ) )

If this rule is met, the stroke is judged as a sketch. This rule is designed for drawing sketches. If the user has tapped the drawing bitmap button and there is no foreseeable operation item as well, the system gives priority to treat all the strokes as sketches.

386

Y. Liu, Q. Lin, and G. Dai

4.3 Gesture and Sketch Recognition The recognizer part (Part 3 in Fig 3) is in charge of recognizing gestures and sketches. Decided by the geometry teaching requirements, the recognizer could treat with 5 different geometric sketches, such as point, straight line, angle, circle, and ellipse. Besides, it could treat with several different types of gestures, such as tap, envelop select, drag, and delete. For different contexts and environments, those types of gestures may have several different meanings. For example, the delete gesture is to delete a page in the slides when a page is active and no geometric items are selected at that time, and it is to delete geometric items when some items are selected in the current page. For sketches, the process is not as complex as gesture recognition. Shape is the only factor with an eye to. It just recognizes the shape of the sketch [14] to find what figures the user wants to draw. And then it gets the position information from some key points in the stroke to decide where the user wants to construct the geometric item. The key points are decided by the shape information. For example, the angle-shape stroke’s key points are the first point, the last point and the point just on the corner. Afterwards, the shape and the position are sent together to the interactive module to display in the screen. Fig 4 is some examples of the sketches the system can process: (The black lines are the pen traces and the blue lines are existing geometric items.)

Point straight Line Angle

Circle

Ellipse

Angle

Vertical Angle

Fig. 4. Supported Sketch Shapes

For gestures, the procedure is more complex, since gestures’ position information as a context is very important for the gesture recognition. In some cases, the same type gesture on different positions has different meanings. To give an example, Tap gesture is treated with, besides as a select/unselect gesture, as a tap-button gesture to switch in a dynamic floating tool ribbon, which can speculate the users’ intentions according to context information to display the most useful tools. Thus the position information is taken good use of. If a tap gesture is on an unselected geometric item, it is a select gesture. If on a selected geometric item, it is an unselected gesture. And if on a button and there is some selected items at that time, it is a tap-button gesture to operate the button’s function. Besides, which geometric item should be operated is decided by the position information of the stroke sometimes. For example, Envelop gesture’s shape is not very strict while the position information is more important in this case. The only requirement for shape is to be a curve or folded line which can envelop a region. However, the position of the stroke decides which geometric items are enveloped, and it is those items that are to be selected.

PIGP: A Pen-Based Intelligent Dynamic Lecture System for Geometry Teaching

387

Fig. 5. Envelop Gesture (Red line)

5 Dynamic Geometry Computations After the recognition work, the system has known the user’s intention. The next task is to transfer the user’s operation to dynamic geometric graphics. Since it is the corresponding semantic conditions that make sure the dynamic geometric graphics can hold their geometric relations, how to re-draw the graphics after the user’s operation is a sort of Geometric Constraints Resolving Problem. The basic idea of this problem is that some geometry’s position is figured out through some other geometry’s positions which have constraint relations with this geometry. [15] There is an elementary algorithm – LIM0 (Locus Intersection Method) Algorithm for such problems based on the idea above. The LIM0 Algorithm [16, 17, and 18] is a method based on Graphic Theory to deal with geometry problems. However, this algorithm is to deal with loop-free constraint problems, the problems that in the construction process, the number of equations that need to be solved contemporarily is smaller than or equals 2 [16]. Although the majority of geometry constraint resolving problems which are probably met with in Geometry Teaching can be resolved through this algorithm, there are some exceptional cases. Hence, we extend LIM0 Algorithm’s usage scope with grouping geometries. The principle of grouping is the freedom degree of the geometry, such as fixed points, dragged points, semi-free points and so on. Through this enhanced LIM0 Algorithm, any point can be set as a dragged point so that the user could drag any point in the slides to investigate the geometry transformation. Detailed information about this algorithm is described in [15]. In our system, the Geometry Manager module (shown in Fig 6) takes the responsibility of calculation according to the enhanced LIM0 algorithm. The Compute Agent in Geometry Manager re-computes the new positions and new relations of all the graphics according to their constraints stored in the Geometry House. If needed, it also adds some new figure item whose type and position are exactly identical to the recognition result. And then it sends the results, including the items and their new positions to the Interactive Module. Take the vertical-angle sketch as an example. The Geometry Manager gets the sketch-recognition result and the two lines from the recognizer (Part 3 in Fig 3). Then it finds out all of the graphics related to the two lines and their relations in Geometry House. In this example, Line DE has a parallel relationship with Line AC so it will also be re-computed. Point A is a fixed point; Point C, D, and E are semi-free points. Hence,

388

Y. Liu, Q. Lin, and G. Dai

Fig. 6. Geometry Manage Module structure

the compute agent re-computes the new positions of the two lines since one of them (Line AC) must be moved to make sure that they are vertical now. Actually just moving Point C in Fig 7 could make the Angle BAC is right. Besides, those graphics in the House which have relationships with the two lines should be re-computed since possibly they are embroiled and changed their positions as well because of the moved figures. In this example, Line DE is related to the moved line, Line AC, so it is also moved to make the parallel relationship is not ruined.

Fig. 7. An Example of Dynamic Geometry Computation

6 Informal Evaluation We had invited a group of geometry teachers in a middle school in Beijing to use and evaluate our system. Responses from teachers are quite positive. About the teaching effects, the teachers think that the students think more actively in class after using PIGP. The students would like to think and ask questions much more than before obviously, possibly because they like to see the teachers answer their questions with dynamic graphics. It seems much easier to fix their attentions on the lecture. Besides, the teachers give us a lot of valuable feedback about the software’s functions. We made a summarization and find that the following advantages are the most impressive for them:

PIGP: A Pen-Based Intelligent Dynamic Lecture System for Geometry Teaching

389

1) It is very convenient that the teachers can draw accurate geometric graphics in class without rulers. In fact, they have to draw geometric graphics almost in class in every class. With PIGP, they have no need to bring rulers to classroom or still draw with chalks. 2) In class, the geometric graphics in the prepared slides can be transformed, which is very helpful for students, especially when teaching some complex theorems. 3) It is more convenient to draw and write with a pen in class. Because they do not have to rush back and forth between the keyboard and the blackboard or stoop in order to manipulate the mouse. 4) Pen Gestures are quite fast. Hence, they have no need to waste precious classroom time to find buttons or menus. 5) The modified slides in class can be saved after class, so that the students have no need to be wrapped up in writing down the notes and they can concentrate on listening and thinking. They also give us some suggestions for improvements. They suggest PIGP should support more geometric transformations and geometric graphics, such as point’s animation, basic conics, and so on. Besides, they put forward a problem that sometimes the system wrongly processes pen’s meaningless movements for them. We are planning to improve the Intelligent Switcher in the next phase. Students also give us some feedback. They said that the dynamic slides in PIGP are very interesting and make their questions resolved in time in class. They feel important in class. And they think it becomes easier for them to understand a geometry theorem when they can directly see its impact on dynamic geometric graphics. Since the experiment lasts not a long time, we haven’t made a formal survey for the students’ learning effect. And we have no enough statistics for a teaching effect comparison now. It is our important job in the next phase.

7 Conclusion In this paper, we propose a teaching assistant system PIGP. It provides a kind of dynamic slides for geometry class in order to promote active learning. The teachers can flexibly and freely use Ink to annotate, use Gesture to operate the geometric items, and use Sketch to draw graphics in class to adjust their lectures according to students’ responses. Besides, a Rule-and-Context-based mode switch method makes this system speculate the users’ intentions to process according to the context and environment information and intelligently distinguish Gestures and Sketches. User’s feedback indicates that the system is attractive for geometry teachers and students. In the next phase, we will improve the Intelligent Switcher and the Gesture Processor to increase the gesture recognition performance. Besides, we will continue to enhance the system’s power on geometry transformations. In addition, we plan to make a formal user evaluation to discover more issues of our system.

Acknowledgements This research is supported by the National Fundamental Research Project of China (973 Project) (2002CB312103), the National Natural Science Foundation of China under

390

Y. Liu, Q. Lin, and G. Dai

Grant No. 60503054, Grant No. 60373056, and the Key Innovation Project from Institute of Software, Chinese Academy of Sciences

References [1] Bonwell, Charles C. and James A. Eison, Active Learning: Creating Excitement in the Classroom. ASHE-ERIC Higher Education Report. Washington, D.C.: 1991. [2] BIRD Note-taking System, http://www.cs.umd.edu/~egolub/AVIAN/BIRD. [3] Classroom Presenter, http://www.cs.washington.edu/education/dl/presenter/. [4] Richard Anderson, Ruth Anderson, K. M. Davis, Natalie Linnell, Craig Prince and Valentin Razmov, Supporting Active Learning and Example Based Instruction with Classroom Technology, To Appear SIGCSE 2007 [5] The Tablet Mylar Slides Classroom Presentation System, http://www.cs.umd.edu/ ~egolub/TabletMylarSlides/. [6] Evan Golub, Handwritten slides on a tabletPC in a discrete mathematics course, Proceedings of the 35th SIGCSE technical symposium on Computer science education, March 03-07, 2004, Norfolk, Virginia, USA. [7] Gregory D. Abowd , Jason Brotherton and Janak Bhalodia, Classroom 2000: a system for capturing and accessing multimedia classroom experiences, CHI 98 conference summary on Human factors in computing systems, p.20-21, April 18-23, 1998, Los Angeles, California, United States. [8] Berque, D., Bonebright, T., and Whitesell, M. Using Pen-based Computers Across the Computer Science Curriculum, SIGCSE, 2004. [9] Jiang Yingying, Wang Xugang, Wang Hongan and Dai Guozhong, ChemTeach: A Speech and Pen-based Multimodal Presentation System, Proceeding of the 7th Asia Pacific Conference on Computer Human Interaction, Beijing, 2002. 592 601 [10] Rbeiz, M., "Semantic Representation of Digital Ink in the Classroom Learning Partner", Massachusetts Institute of Technology MEng Thesis, May 2006. [11] Bransford, John, Brown, Ann and Cocking Rodney (eds). How People Learn: Brain, Mind, Experience, and School. Committee on Developments in the Science of Learning, Commission on Behavioral and Social Sciences and Education, National Research Council, National Academy Press, Washington, 1999. [12] Wright P et al. Analysing Human-Computer Interaction as Distributed Cognition: The Resources Model. Human Computer Interaction Journal, 2000, 51(1) 1~41 [13] Wang CQ, Deng CZ, Ma CX, Hua QY, Dai GZ, An extended resources model based on distributed cognition theory. Journal of Software, 2005,16(10):1717−1725. DOI: 10.1360/jos161717 [14] Ma Cuixia, Dai Guozhong, An infrastructure approach to gesture interaction computing in conceptual design, Proceeding of the 5th Asia Pacific Conference on Computer Human Interaction, Beijing, 2002. 592 601 [15] Lin Qiang, Ren Lei, Chen Ying, Fan Fengke, Dai Guozhong, The Design of Intelligent Dynamic Geometry Software Based on the Enhance LIM0 Arithmetic. Chinese Journal of Computers(accepted, in Chinese) [16] Xiaoshan Gao, Weiqiang Yang and Lei-Dong Huang, Intelligent Dynamic Geometry Software, Proceeding of the Fifth Asian Technology Conference in Mathematics. Thailand, 2000:pp.430-439. [17] J.Owen, Algebraic Solution for Geometry from Dimensional Constraints. ACM Symp. Foundation of Solid Modeling, ACM Press, New York, 397-407,1991. [18] I.Fudos & C.M.Hoffmann. A Graph-constructive Approach to solving Systems of Geometric Constraints. ACM Transactions on Graphics, 16(2),179-216,1997.

～

：

～

A Conceptual Framework of Affective Context-Aware Interactive Learning Media Horace H. S. Ip and Belton Kwong AIMtech Centre, City University of Hong Kong, 83 Tat Chee Avenue, Kowloon, Hong Kong {cship, belton}@cityu.edu.hk

Abstract. We propose an affective context-aware interactive learning media that provides dynamic feedbacks to human motions, speech, and affects in real time. The proposed framework perceives and keeps track of emotional state of student by employing affect recognition techniques in learning to encourage active participation, facilitate expressivities and bypass tool barriers through various interactive activities and dynamically adapts the interactive contents to current progress and states of individual student. Keywords: Affective computing, interactive learning media.

1 Introduction With the advent of interactive media technology and multi-modal man-machine interface, novel interactive multimedia and virtual reality games have been developed for entertainment, education, training and psychotherapy. In particular, we have previously developed multi-modal interactive multimedia systems in association with music composition [1], creation of 3D art [2], psychotherapy of abused children [4], as well as learning games [4]. For example, Body-Brush [2-3] is a real-time body-driven immersive environment that maps body motion gesture-path-energy to the audiovisual attributes and it could stimulate students’ interest to learn basic art concepts without requiring the mastery of traditional art tools. Body-Music [1] is a novel cyber instrument that enables music laypersons to dynamically control the tonality and melody through intuitive hand motions and gestures with a pair of motion-sensing gloves. Since music theories are embedded in the design, melody flow and musical expressions can be controlled in real time, so students could learn basic music concepts with ease. Building upon Body-Brush, Smart Ambience Games (SAG) [4] is developed as a new form of digital learning for children with learning difficulties. SAG creates a motivating environment for children to overcome inhibitions related to learning, and to become more active learner. Students should find it easy to engage themselves in such immersive learning environment through various interactive activities and game scenarios. Particular subjects such as history and mathematics could be taught more interestingly and efficiently. K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 391–400, 2007. © Springer-Verlag Berlin Heidelberg 2007

392

H.H.S. Ip and B. Kwong

While such interactive systems are generally able to track, interpret and respond to human gestures and body motion, an emerging challenge is to develop machine ability to recognize and interpret human emotion in order to advance the usability of such multimedia interactive systems in education and entertainment. In the following, we will focus our discussion on the potential use of affective sensing and affective context-awareness in edutainment applications. The affective aspect of learning plays a facilitating role in enhancing students’ engagement in learning because it is believed that there is an important relationship between emotions and learning [5-6]. Affective state recognition would therefore help to create an effective learning environment. Emotional unawareness could cause digital learning systems less effective. The ability of an interactive learning system to detect affective cues like posture, gaze direction, facial expressions etc., would empower the system to respond immediately according to the emotional state of the student. For example, when the student starts to become tired, distracted, or bored, and learning becomes ineffective, the system could adjust itself to arouse the student’s learning interests by producing more interesting and eyecatching contents, decreasing the amount of factual and bookish materials, or increasing the relaxation or entertainment level of its interactive content. Furthermore, the ability to detect affective state would also help to predict level of interest and affective context of the students. It is well known that digital multimedia learning contents could stimulate students’ interest. Hence, this is what most multimedia learning systems aim to offer to its users. However, it is difficult to measure and quantify learners’ level of interest while using such learning systems and the effectiveness of various interactive learning content designs is unknown. Human behaviour in learning situations should be studied and incorporated into interactive learning environment to extract and analyze multimodal behaviours for recognizing affective states in various learning situations. To achieve this goal, we need a real-time affect recognition system to classify affective states, thus level of learning interest of children. Emotional intelligence is essential in rational decision making, perception, and learning. Affective sensing techniques could be used to recognize and express emotions, respond to human affect, and even affect or regulate the users’ emotional states. Therefore, we propose a framework of affective context-aware interactive learning media and integrate the interactive multimedia systems we built so far. Such system responds to motions, speech and affects in real time. Affect recognition techniques in learning are employed to perceive emotional states of the user, so the system could dynamically adjust the interactive feedbacks and contents according to the user's progress and emotional states. We have also designed low-cost wireless sensors and 3D projection system which are affordable to most primary schools.

2 Related Works Affective states in learning are accompanied by different patterns of motions and gestures. Although current technologies do not allow us to accurately recognize human emotions, affective cues like posture, gesture, eye gaze, and facial expression

Affective Context-Aware Interactive Learning Media

393

help us to recognize whether the learner is on the track of effective learning or not. While research in affect awareness is still in its infancy, several techniques for emotion recognition have been proposed. Leon et al. proposed a user-independent realtime emotion recognition system for software agents in domestic environments [7]. The system detects emotional states by using auto-associative neural networks (AANN) and statistical probability (SPRT) in real time by measuring heart rate, skin resistance, blood volume pressure, etc. to recognize and classify emotional changes using physiological signals. They categorize emotional states into 3 classifications: Positive, Neutral and Negative and achieved recognition level of 71.4%. This affect recognition system runs in real time, however, it is not designed particular for interactive learning media. And the sensors they used are quite bulky and could hinder freedom of movements in 3D virtual canvas. Rich et al. [10] has defined symbolic postures that convey a specific meaning about the actions of a user sitting in an office which are: interested, bored, thinking, seated, relaxed, defensive, and confident. For example, learning forward towards a computer screen, looking towards the problem, tightening and widening eyes, head nodings might be a sign of attention (on-task) while slumping on the chair or fidgeting, looking everywhere else, lowering eyebrow, shaking head in sideways suggests frustration or boredom (off-task). Recently, Kapoor et al. proposed a learning companion system [8] based on Kort’s affective model, suggesting several practical methods to track and immediately recognize the affective state of a learner throughout the learning journey by sensing gaze dynamics, facial expressions, and postural changes of the learner. Surface level behaviours that suggest a transition from an on-goal state to off-goal state or vice versa are identified. Kapoor et al. proposed a multimodal affect recognition framework [9] that achieves accuracy of over 86%. Mixture of Gaussian Processes approach is used to classify interest in a learning scenario using multiple modalities. This affect recognition framework automatically extracts, processes, and models sequences of natural occurring non-verbal behaviour for recognizing affective states in learning. Kapoor’s framework aims at recognizing in real time whether the student is off-task, and whether or not its intervention succeeds in helping the student return to being on-task. There are various affective responses associated to learning. These affective responses can range from feelings of interest and excitement to feelings of confusion and frustration. The student might quit if he is not able to recover from the feeling of getting stuck. For example, signs of frustration and boredom may mean the student wants to quit, while signs of curiosity and interest may indicate active participation, and signs of enjoyment and mastery may indicate a successful learning experience.

3 Application of Affective Models in Classroom Learning Traditional classes usually lack dynamic feedback mechanism and interactivity. It is not easy for teachers to observe and monitor the learning states such as learning attitude and efficiency of each student. However, expert and skilled teachers are able to give positive guidance to improve learning efficiency, and facilitate individual

394

H.H.S. Ip and B. Kwong

stu dents’ own efforts at learning by recognizing their students’ feedbacks, emotional state of learners based on experience and observation. We will briefly review some of the theories and prototype models of emotions that have been proposed. Plutchik [11] distinguished among eight basic emotions: fear, anger, sorrow, joy, disgust, acceptance, anticipation, and surprise. Another famous emotional model is the circumplex model of affect suggested by Russell [12]. He organized emotions in a two-dimensional space of pleasure-displeasure and degree of arousal. Many interactive and multimedia systems have adopted this emotional model. For example, eMoto [13] was designed to emotionally engage users by enhancing their SMS messages in a way that they could consciously express their emotions such as excitement, boredom, and relaxation with the interaction model based on Russell’s circumplex model of affect. Affective gestures are recognized by means of an accelerometer and a pressure sensor. The affective feedback would be represented by text message enhanced by associated styles and ornaments including various combinations of colours, shapes, and animations. Facial Action Coding System (FACS) developed by Ekman and Friesen [14] is one of the most popular standards currently used to characterize the expression of emotions. This system defines 46 facial action units for expressions and motion analysis. Other previous researches have worked on the analysis, interpretation and recognition of facial expressions based on FACS [15-17]. However, the major challenge of any approach to recognize expression of emotions is that the user may intentionally hide or suppress his/her emotions which makes the task of detecting and interpreting emotional state even more difficult for the computer. Although there is a revised version of FACS by Ekman, Friesen, and Hager in 2002, such problems still cannot be eliminated. While most existing frameworks do not concern specifically the emotions commonly observed by teachers in learning experiences, Kort et al. have developed a framework that models the complex interplay of emotions and learning [18]. Table 1 shows examples of emotional states relevant to learning [18]. Figure 1 shows the circumplex model relating phases of learning to emotions and inferring the cognitive dynamics of the learning process [18]. Table 1. Examples of emotion relevant to learning. (From Kort et al [18]) Axis

-1.0

-0.5

0

+0.5

+1.0

Anxiety-Confidence

Anxiety

Worry

Discomfort

Comfort

Hopeful

Confident

Boredom-Fascination

Ennui

Boredom

Indifference

Interest

Curiosity

Intrigue

Frustration-Euphoria

Frustration

Puzzlement

Confusion

Insight

Enlightment

Ephipany

Dispirited- Encouraged

Dispirited

Disappointed

Dissatisfied

Satisfied

Thrilled

Enthusiastic

Terror-Enchantment

Terror

Dread

Apprehension

Calm

Anticipatory

Excited

4 Recognition of Affective States in Learning Our objective is to develop a computational framework that improves affective state detection efficiency and to enable real-time feature extraction and analysis in order to

Affective Context-Aware Interactive Learning Media

395

Fig. 1. Model relating phases of learning to emotions. (From Kort et al. [18])

provide dynamic feedbacks to multiple modalities, so that the appearance or the interactivity of the multimedia contents could be dynamically adjusted and modified according to the current state of the user.

Fig. 2. Overview architecture

Figure 2 shows overview architecture of our proposed framework. There are altogether four processing modules: multimodal sensing, feature extraction, multimodal pattern analysis and affect classification. Multimodal sensing module is responsible for collecting sensed data using a range of multi-modal sensors such as sensors for motion and gesture sensing, video cameras for spatial-temporal visual

396

H.H.S. Ip and B. Kwong

cues sensing and microphones for audio sensing. These multi-modal sensed data are then filtered (for noise for example) and passed to the Feature extraction and analysis module. Feature extraction and analysis module selects the relevant sensor data and extracts the low level features from video or audio input data streams such as salient spatial temporal and cumulated volumetric features [19], facial expressions [8-9] and voice in terms of pitch and frequency [20] for affect recognition. The affect classification module typically consists of statistical classifiers such as support vectors machine (SVM) to compute the probability of features that could infers certain affective states based on the training data.For multimodal sensing, we adopt wireless motion sensors, cameras and microphones in order to preserve the degree of freedom of user’s movement. Therefore, motion, gestures and voice features are captured and analyzed. Some of the extracted features may be grouped into various channels due to their nature of similarity and supplementary. For audio and gestural features, we suggest not to attempt to do voice recognition and detailed motion capture, instead we emphasize more on the prosodic changes and gestural activity levels, which would generally imply the emotional and activity state of the student. Pentland [20] suggested a computational model of social signaling, which captures gestural activities such as unconscious face, hand, and body gestures to estimate the behaviour of the user and also measures attitude and intention from amplitude, frequency, and timing of prosodic. In this proposed framework, we employ similar methods to extract audio and gestural features. Pentland’s experiments have shown that the model could achieve an accuracy of 80% to 90%. In our proposed framework, we use parts of the mathematical measures suggested by Pentland, especially activity measure and emphasis measure. Conversational activity level is measured by the percentage of speaking against non-speaking by using a HMM, and gestural activity level is measured by the percentage of high activity time. Conversational emphasis is measured by the variation in pitch and amplitude. Conversational emphasis could be measured by the sum of mean-scaled standard deviations of velocity and acceleration, and gestural emphasis could be measured by sum of the mean-scaled standard deviations of velocity and acceleration.

5 Example: Detecting Levels of Interactivity Among Students For illustration, we describes the implementation of a generic interest-level SVM which could be used for various applications. And in this example, we apply this classifier to our interactive 3D painting activity [2]. This classifier module captures primarily motion features (visual information is essential, while voice, gestures and facial expressions, could be additional) to measure level of activity to estimate the interest level of students. We defined three classifications of interest level: high, medium and low. High interest level may mean the student is enjoying the learning process or he is being excited, which means s/he is on the right track of learning process, keeps good performance, progress, and high efficiency of learning. Medium interest level may mean the student is feeling interested or relaxed, and probably is still on task. Low interest level may mean the student is getting bored, frustrated, stuck and probably needs some help, stimulations or intervention. And what we want to do is to stimulate him/her or bring him/her back to the state of “on-task”.

Affective Context-Aware Interactive Learning Media

397

Based on the Body-Brush system [2-3] developed by the authors, and using “3D virtual painting” for learning art creation as an example, the system keeps track of the interest level of the student through his/her motion, such as the speed and level of acceleration of his/her body parts. When the interest level drops to a lower level, we could produce some kinds of stimulations or guidance, for instance, altering the view angle or display mode, generating exciting and stimulating special visual effects or features based on his/her current drawing, or displaying some interesting shapes or cartoons for the student to pick up (virtually). It is up to the student’s choice whether they follows the guidance or pick up the given materials or not, but the idea is to encourage the student to continue to work on his/her own creation and stimulate the student’s interest and motivating the student’s creativity. Before enabling the system to recognize the user’s interest-level, we need to collect training data. We invite 30 students for studying their level of interest. We record and annotate one student at a time, asking each of them to create 3 different drawings with our interactive 3D drawing tool. The requested drawings are of different complexity and the timeframe of each drawing activity is 3 minutes. At the same time, expert teachers observe and classify the level of interest of each student on each drawing based on their experience. Assessments and comments from teachers will be gathered, processed, analyzed. These data finally become useful training data for the SVM classifier [22] for relating the difference of activity level to various interest-level classifications. We also video-tape the whole process for future verification.

6 Wearable Wireless Motion Sensing Human motion and gestures provide useful cues to estimating the underlying affective state of the person. In our previous projects in a laboratory environment, we have proposed an innovative motion capture and analysis system using frontal infrared illumination. Silhouette image can be computed by techniques of background subtraction, segmentation, feature extraction, and pose estimation. The major advantage of this system is that the users are not required to wear specific coloured clothing or sensor devices. However, since the system is targeted for school students, and schools are usually have limited space and budget. It would be difficult for the schools to prepare a vast and expensive light-controlled room. Moreover, several students might be involved in the activities in order to monitor the learning progress of each student. Solely using image-based real-time motion tracking techniques may not be adequate. To meet the practical needs of the school environment, we are developing tiny wearable motion sensors to capture accurate motion data for a smart ambience learning space. Our motion sensor is watch-like wireless sensors, via radio frequency and ZigBee networking technologies, such that motion data is captured wirelessly. The motion sensor contains three accelerometers. The tilt level could be calculated by finding the values of pitch - ϕ (1) and roll - ρ (2) respectively while the sensor is at rest. (The sign of pitch is the same as of ax, and the sign of Roll is the same as ay.) To enable error checking, pitch and roll can also be calculated by using only ax and ay. Heading compensation (X’ and Y’) can be calculated from the equation (3) & (4). And the magnitude (M) of acceleration is calculated by equation (5).

398

H.H.S. Ip and B. Kwong



 ax  2 2  a y + a z  

(1)



(2)

ϕ = tan −1 

 ay  2 2   ax + az 

ρ = tan −1 

X ' = X • cos(ϕ ) + Y • cos( ρ ) • sin(ϕ ) − Z • cos( ρ ) • sin(ϕ )

(3)

Y ' = Y • cos( ρ ) − Z • sin( ρ )

(4)

M = ax 2 + a y 2 + a z 2

(5)

where φ - pitch, ρ - roll and a - acceleration

The motion vector can then be estimated from the orientation and acceleration magnitude while velocity and displacement can be approximated by integration. Figure 3 shows the wireless sensor and receiver for our smart ambience learning space. Each receiver could connect up to three sensors.

Fig. 3. Wireless sensor (left-top) and receiver (bottom) compared with a watch

7 Conclusions In this paper, we have proposed a generic conceptual framework to extract and analyze multiple modalities of behaviours for recognizing affective states for interactive media that provides dynamic feedbacks to multiple modalities and particularly affect.. Not only does such interactive system could highly motivate students’ will to learn, it

Affective Context-Aware Interactive Learning Media

399

also acts like an expert teacher who could monitor and detect level of interest, and perceive students’ emotions in order to determine dynamically what kind of motivating actions and contents to be presented to the students and to maintain a stimulating environment for the learning session. The proposed framework could potentially provide a powerful learning medium for students to learn school subjects. This is because such system encourages participation, helps build confidence, facilitates authentic expressivities, and bypasses tool barriers via various VR interactive activities. Such affect detection system should also optimize the balance between learning and relaxation phases, which would also improve the learning effectiveness and performance. We are currently working with schools to develop a low-cost “Smart Ambience Learning Space” based on this framework that would become another specialized teaching room in schools like an art room or a science laboratory.

Acknowledgements This work is jointly supported by Hong Kong Research Grants Council (RGC) Competitive Earmarked Research Grant, Hong Kong Government, CERG/1137/06.

References 1. Ip, H. S., Kwong, B., and Law, C. K.: BodyMusic: Novel Framework Design for Bodydriven Music Composition. ACM SIGCHI International Conf. on Advances in Computer Entertainment Technology (ACE 05), ACM, Valencia, Spain, June 2005. (2005) 2. Ip, H. S., Young, H., Tang, A.: Body-Brush: A Body-driven Interface for Visual Aesthetics. ACM International Conference on Multimedia (2002) 664-665 3. Ip, H. S., Young, H., Tang, A.: Body Brush: Generating Virtual 3D Painting through Body Motion. SIGGRAPH 2003: Emerging Technologies (2003) 4. Ip, H. S., and Kwong, B.: Smart Ambience Games for Children with Learning Difficulties. Edutainment 2006 - International Conference on E-learning and Games. LNCS 3942, Springer-Verlag Berlin Heidelberg 2006, Hangzhou, China, Apr. 2006, pp 484-493. (2006) 5. Kinard, E. M.:. Perceived and actual academic competence in maltreated children. Child Abuse and Neglect, Vol. 25, 1 (2001) 33-45 6. LeDoux, J.: The emotional brain: The mysterious underpinnings of emotional life. Weidenfeld & Nicholson, London (1998) 7. Leon, E., Clarke,G.S., Callaghan,V., Sepulveda,F.: A User-independent Real-time Emotion Recognition System for Software Agents in Domestic Environments, Engineering Applications of Artificial Intelligence, The International Journal of Intelligent Real Time Automation (2005) pp.- [C11] 8. Kapoor, A., Mota, S., and Picard., R. W.: Towards a Learning Companion that Recognizes Affect. Emotional and Intelligent II: The Tangled Knot of Social Cognition, AAAI Fall Symposium, Nov 2001. (2001) 9. Kapoor, A., and Picard., R. W. Multimodal Affect Recognition in Learning Environments. MM’05, Nov 6-11, 2005, Singapore. (2005)

400

H.H.S. Ip and B. Kwong

10. Rich, C., Waters, R. C., Strohecker, C., Schabes, Y., Freeman, W. T., Torrance, M. C., Golding, A., and Roth, M.: A Prototype Interactive Environment for Collaboration and Learning. Technical Report, TR-94-06. (http://www.merl.com/projects/emp/index.html) (1994) 11. Plutchik, R.: A General Psychoevolutionary Theory of Emotion. In Plutchik R. and Kellerman H., editors, Emotion Theory, Research, and Experience: vol. 1, Theories of Emotion. Academic Press. (1980) 12. Russell, J. A.: A Circumplex Model of Affect. Journal of Personality and Social Psychology: 39(6), 1161–1178. (1980) 13. Fagerberg P., Ståhl A., and Höök K.: eMoto: emotionally engaging interaction. Personal and Ubiquitous Computing, v.8 n.5, p.377-381, September 2004. (2004) 14. Ekman, P. and Friesen, W.: Facial Action Coding System. Consulting Psychologists Press, Palo Alto, CA. (1977) 15. Essa, I. A., and Pentland, A. P.: Coding, Analysis, Interpretation, and Recognition of Facial Expressions. PAMI(19), No. 7, July 1997, pp. 757-763. (1997) 16. Huang, C. L., and Huang, Y. M.: Facial Expression Recognition Using Model-Based Feature Extraction and Action Parameter(s) Classification. JVCIR(8), pp. 278-290. (1997) 17. Lien, J. J. J., Kanade, T., Cohn, J. F., and Li, C. C.: Automated Facial Expression Recognition Based on FACS Action Units. AFGR98 (390-395). (http://dx.doi.org/10.1109/AFGR.1998.670980) (1998) 18. Kort, B., Reilly, R., and Picard, R. W.: An Affective Model of Interplay Between Emotions and Learning: Reengineering Educational Pedagogy-Building a Learning Companion. Proceedings of IEEE International Conference on Advanced Learning Technologies. Madison.(2001) 19. Ke, Y, Sukthankar R, Hebert, M, “Efficient visual event detection using volumetric features:, IEEE ICCV, 2005. 20. R. W. Picard, E. Vyzas, and J. Healey. Toward machine emotional intelligence: Analysis of affective physiological state. PAMI, 2001. 21. Pentland, A.: A Computational Model of Social Signaling, 18th International Conference on Pattern Recognition (ICPR'06) pp. 1080-1083. (2006) 22. Christopher J. C. Burges. "A Tutorial on Support Vector Machines for Pattern Recognition". Data Mining and Knowledge Discovery 2:121 - 167, 1998.

Ontology Based Situation Analysis and Encouragement in E-Learning System Yongzhao Zhan1,2, Liting Xu2, and Qirong Mao2 1

State Key Lab for CAD& CG, Zhejiang University, Hangzhou, China School of Computer Science and Telecommunication engineering, Jiangsu University, Zhenjiang 212013, Jiangsu, China [email protected], [email protected], [email protected] 2

Abstract. In order to describe the learning situations in the E-Learning environment reasonably and efficiently, making good use of the Ontology’s characteristics such as clear knowledge organization and explicit definition of concept and relation, a situation ontology model is presented in this paper. By deducing the different situations in the learning process in time, the learner’s emotion could be concluded. Different encouragement and helpful entertainment for learning are given according to the different emotion. Thus the learner’s learning interest would be stimulated more effectively and the learner’s learning efficiency would be raised greatly. Keywords: E-Learning, ontology, situation analysis, emotion, encouragement.

1 Introduction In view of the fact that the ontology has characteristics such as clear knowledge organization, explicit definition of concept and relation etc, the use of ontology in the computer domain has being emerged gradually. Many scholars make the research of E-Learning using ontology: Tai-Jong Kim and his colleagues presented an E-Learning system by employing ontology [1], José M. Gascue and his colleagues presented an E-Learning system based on domain ontology [2] and we presented an E-Learning system based on awareness ontology [3]. By using ontology, such systems based on ontology have obvious advantages of knowledge organization, relationship extraction and dynamic processing. But they can not stimulate the learner’s interest effectively and raise the learning efficiency greatly without sufficient consideration of the learner’s learning situation and emotion. Some learning systems without ontology had taken the learner’s emotion into consideration: Kiyhoshi Nosu and his colleagues analyzed the learner’s emotion during the learning process by the learner’s facial expressions and biometrical signals such as pulse rate, breathing rate and finger temperature [4]. WANG Wansen and his colleagues analyzed the learner’s emotion by facial expressions and body gestures [5]. Although the learner’s emotion was analyzed in such systems, the emotion affected by the different situations during the learning process was not considered. These systems did not give corresponding encouragement and entertainment according to K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 401–410, 2007. © Springer-Verlag Berlin Heidelberg 2007

402

Y. Zhan, L. Xu, and Q. Mao

the learner’s different emotion caused by different situations. So they can not stimulate the learner’s interest effectively. By making good use of the Ontology’s characteristics such as explicit definition of concept and relation, the learning situations are described reasonably and efficiently and a situation ontology model in the E-Learning environment is presented in this paper. By analyzing and deducing the different situations and the learner’s emotion in time, the corresponding encouragement and entertainment is given according to the different emotion. Thus the learner’s learning interest would be stimulated more effectively and the learner’s learning efficiency would be raised greatly.

2 Emotion Analysis of the Learner in the E-Learning System and Encouragement Currently, the learner’s emotion caused by different situations during learning in the E-Learning system isn’t thought enough, which influences the learner’s interest and learning efficiency. In this paper, the learner’s emotion is analyzed according to different situations and the corresponding encouragement and entertainment are given according to the different emotion. 2.1 Learner’s Emotion Analysis Learners have different emotions in different learning situations. For example: the learner is happy when he gets a good mark in the test and the learner is sad when he always can’t accomplish the learning objective in time. In conclusion, there are “good” and “bad” two kinds of emotions during the learning process. Each of them can be caused by many situations. Naturally, different learners have different knowledge and learning capabilities. So the situation affected emotion should be analyzed according to everyone’s different situation and the situation factors should be modified often during the learning process. 1) Personalized learning time Generally, different learners need different time to learn the same content. Every knowledge domain [6] has its corresponding given learning time range. If the time the learner cost in learning the knowledge domain is less than the lower limit of the corresponding personalized given learning time range of the knowledge domain and the learner has a good mark in the corresponding test, it means that the learner really only need so short time. Then the learner’s personalized learning time should be decreased. And the result should be used to influence the personalized given learning time range of the next knowledge domain to be learned. If the learner has a bad mark after short time’s learning, it means the time spent is too little and the personalized learning time shouldn’t be modified. If the learner always needs more time to finish the learning objective, his personalized learning time should be increased. In this way, the personalized given learning time range of the knowledge domain can be formed according to everyone’s different situation and will be modified according to the learner’s different learning state during the learning process.

Ontology Based Situation Analysis and Encouragement in E-Learning System

403

2) Personalized average mark Different learners have different learning capabilities, so different learners have different expectation of marks. For example, the learner gets a normal mark but the mark is better than before, the learner may still be happy. These personalized situation factors should be considered in the situation analysis and should be modified according to the learner’s learning state. 2.2 Situation Ontology in the E-Learning Environment The learner’s emotion will be affected by the learning activity and the test activity greatly during the learning process. In order to describe the situations that affect the learner’s emotion reasonably and effectively, the related concepts and relations are described by ontology. In the E-Learning environment, situation ontology model includes nine concepts, some relations among these nine concepts and in the concepts’ interior. These nine concepts corresponding to the 9-tuple (P, KD, ST, PEST, PST, T, TS, TM, AG) of situation information are Person, KnowledgeDomain, StudyTime, PersonStudyTime, PStudyTime, Test, TestStandard, TestMark and AverageGrade. The framework of the model is shown in Figure1. Detailed description of these nine concepts and their relations are given latter.

Fig. 1. Situation ontology model framework

2.2.1 Learning Activity During the learning process, the learner’s emotion will be changed according to the situations such as whether the learner learns reasonably, whether the learner spends enough learning time. Here, such tow aspects are considered: 1) Concepts and relations of learning action Person expresses the learner in the E-Learning environment. It can be denoted as a quintuplet: Person (Person _ID, Name, Age, Sex, Contact), which describes main individual information. KnowledgeDomain expresses a relative independent teaching objective with a special range [6]. It can be described as a quadruplet: Knowledge-Domain (ID, Description, Document, Objective). Here, ID is the identifier of knowledge domain. Description is the description of knowledge domain. Document is the document information. And Objective is the learning object.

404

Y. Zhan, L. Xu, and Q. Mao

hassequence expresses the scheduling relations among the KnowledgeDomain individuals, which has the transitivity, hassequence={| x and y are both the KnowledgeDomain individuals x should be learned before y}. hasstudied expresses the relation that Person individual has learned KnowledgeDomain individual, hasstudied={|x is a Person individual y is a KnowledgeDomain individual x has learned y }. According to these concepts and relations, we can judge many situations. For example: when the learner wants to learn a knowledge domain document, we can judge whether all the knowledge domains that need to be learned before this knowledge domain have already been learned, whether the learner has the right to learn this knowledge domain. 2) Concepts and relations of learning time StudyTime expresses the given time range for the knowledge domain’s learning. It is decided according to the time that the normal learners cost during the knowledge domain’s learning. It can be described as a triplet: StudyTime (KnowledgeDomain_ID, Min_Time, Max_Time), which describes the ID of the knowledge domain, the shortest time and the longest time that learning the knowledge domain needed. PersonStudyTime expresses the percent of the learning time the learner need and the given learning time. It can be described as a doublet: PersonStudyTime（ Person_ID, Percent） , which describes the ID of the learner and the percent of the learning time the learner need and the given learning time. PStudyTime expresses the actual time cost in the learner’s learning the knowledge domain. It can be described as a triplet: PStudyTime（ Person_ID, Knowledge Domain_ID ,Time） , which describes the ID of the learner, the ID of the knowledge domain and the actual time cost in the learner’s learning the knowledge domain. y is a hasMoreTime: hasMoreTime={|x is a PStudyTime individual corresponding individual of StudyTime z is a corresponding individual of PersonStudyTime x.Time > y.Max_Time * z.Percent }, which describes the actual time cost in the learner’s learning is more than the upper limit of the personalized time range of the learner’s learning of this knowledge domain. hasInTime: hasInTime={| x is a PStudyTime individual y is a corresponding individual of StudyTime z is a corresponding individual of y.Min_Time * z.Percent < x.Time < y.Max_Time * z.Percent }, PersonStudyTime which describes the actual time cost in the learner’s learning is in the personalized time range of the learner’s learning of this knowledge domain. hasLessTime: hasLessTime={| x is a PStudyTime individual y is a corresponding individual of StudyTime z is an according individual of PersonStudyTime x.Time < y.Min_Time * z.Percent }, which describes the actual time cost in the learner’s learning is less than the lower limit of the personalized time range of the learner’s learning of this knowledge domain. According to these concepts and relations, we can judge many situations such as whether the learner has learned the knowledge domain in his personalized time range. And the percentage of the learner needed time and the given time will be modified according to actual situation. Then a more suitable time range for the learner can be gained in the next learning activity.

∧

∧

∧

∧

∧

∧

∧ ∧

∧

∧

∧

∧

Ontology Based Situation Analysis and Encouragement in E-Learning System

405

2.2.2 Test Activity During taking a test, the learner’s emotion will be affected by the test mark. Test expresses the test of the knowledge domain. It can be described as a triplet: Test（ Test_ID, KnowledgeDomain_ID, Content） , which describes the ID of the test, the ID of the knowledge domain and the content of the test. TestStandard expresses the mark standard of the test. It can be described as a doublet: TestStandard（ Test_ID, Standard） , which describes the ID of the test and the standard mark. TestMark expresses the learner’s mark of the test. It can be described as a triplet: TestMark（ Person_ID, Test_ID, Mark） , which describes the ID of the learner, the ID of the test and the test mark the learner gets。 AverageGrade expresses the learner’s average mark. It can be described as a triplet: AverageGrade （ Person_ID, Test_Num, Mark） , which describes the ID of the learner, the number of the tests the learner has finished and the average mark of all the tests the learner has finished。 hasGoodMark: hasGoodMark={|x is a TestMark individual y is a corresponding individual of TestStandard x.Mark> y.Mark +5}, which describes the learner’s test mark is good. hasNormalMark: hasNormalMark={| x is a TestMark individual y is a corresponding individual of TestStandard y. Mark -5 means that if the current state is a and the current input tape symbol is b, then the controller changes the current state to c and moves one step to the right (right is represented by > and left by 0011 111> 1-2< 2133-4< 414< 4-5> Fig. 4. A Turing machine example to add two positive integers

3.4 Visual Examples and Games In our integrated environment, a set of visual finite state machines examples are introduced with the aim of motivating learners in courses that include such topics. These selected examples represent useful daily life machines, games, and a puzzle. We have created six examples: an elevator, a vending machine, a man, a wolf and a goat puzzle, a tennis game, a video player, and a rice cooker. In this section, we will describe the last two due to the lack of space. 3.4.1 Video Player The operations of a video player, a common and useful machine in our daily lives, can be modeled by a finite state machine. Video player operations such as play, pause, forward, fast forward, backward, fast backward and stop can be represented by the finite automaton states. The video player simulator is designed to play a movie showing a wild cat in response to the change in the finite automaton states. The operations are represented by buttons labeled with real video player symbols. When the user pushes a button, the corresponding automaton state is highlighted and the appropriate screen shot takes place. Figure 5 shows the video player simulator interface in which the operations of the video player are simulated and the corresponding finite automaton is displayed. 3.4.2 Rice Cooker Rice cooker operations can also be modeled by a finite state machine. Examples of the operations include cooking, reheating, and keeping warm. A timer is also considered in this model. The student can operate the rice cooker simulator by pressing the operations, and then the corresponding state of the underlying automaton is highlighted. Figure 6 shows a snapshot of the rice cooker simulator interface. In the rice cooker automaton model, every state represents an operation, for example, the state labeled q0 represents the waiting (initial) state, q1 represents the keep warm (final state) operation, and q2 represents the reheating operation. The input alphabet is represented by the symbols A, B, C, and D, where: 1. 2. 3. 4.

‘A’ corresponds to heating and reheating operations, ‘B’ corresponds to the keep warms and cancels operations, ‘C’ corresponds to the timer, and ‘D’ corresponds to the finish operation.

428

M. Hamada

Fig. 5. The Video Player simulator interface within the IE

Fig. 6. The Rice cooker simulator interface within the IE

In a real rice cooker, after we set the initial conditions, it completes the task and finishes automatically after a certain amount of time. The applet simulates such behavior; first the user can set the initial conditions by pressing the buttons A, B, and

An Integrated Virtual Environment for Active e-Learning in Theory of Computation

429

C; then the finish button D will take place automatically by the automaton after a certain amount of time. Thus, we do not need to explicitly add a button ‘D’ to the simulator. 3.5 Self Assessment A set of exercises with different levels is also integrated with the environment. There are various types of quizzes: some are multiple choice, some are fill in the blanks, and some test for Turing machines, finite automata or regular expressions. Learners can perform a pre-assessment, an in-assessment, or a post-assessment. The assessment interface is shown in Figure 7. First, the learner must select an exercise (upper-left corner in the window), and then a description of the test and the evaluation method will be shown in the main window. Learners can navigate among the quizzes by using the navigation buttons at the bottom of the main window. Learners can check the score at any time by clicking on the ‘score’ button. While answering a quiz, learners can get hints or click on the introduction button on the top of the window to go to the introduction component and read more about the topics related to the quiz.

Fig. 7. The self-assessment component interface in the IE

4 Evaluation We carried out two experiments in order to evaluate the effectiveness of our integrated environment tools on the learning process of engineering students. The first experiment evaluates the improvement in the students’ motivation. The second experiment evaluates the effectiveness of using the tools on the students’ performance. The purpose of introducing the visual automata examples is to enhance the students’ motivation. To measure the effectiveness of these visual examples, we performed two experiments in the automata and formal languages course. The first

430

M. Hamada

one was for students who already completed the course; the sample population included 52 students who studied the topics in different classrooms. The following question was asked: “If the course was an elective course, would you choose to study it? And, do you recommend other students to study it?” Five options were given for responses: (a) don’t know, (b) no, (c) maybe no, (d) maybe yes, and (e) yes. The responses were as follows: 3 answered a, 3 answered b, 6 answered c, 27 answered d, and 13 answered e. Then, we demonstrated our visual examples to the students and repeated the same question again. Their responses (after seeing the examples) were: 1 for a, 3 for b, 2 for c, 29 for d and 17 for e. Comparing the results from “Before” and “After” exposure to the examples, there was a slight improvement in motivation. For choices a, b, and c, if the number of responses decreased, it indicates a positive response, which is what occurred. While for the other choices d and e, the increasing number of responses indicates positive response, which also occurred. We note that there was only a small improvement in the students’ motivation, which is natural in this case because the students had already completed the course. In the next experiment we noted a better improvement in the motivation of students who were new to the course. In the second experiment, a total of 69 students were included, and they were all new to the course. The same steps, as with the pervious experiment, were repeated with a slight modification in the question. The question was “If the course was an elective one would you chose to study it?” As before, students were allowed to choose from among the five responses: a, b, c, d, and e. Their responses (before seeing the examples) were as follows: 22 answered a, 6 answered b, 10 answered c, 23 answered d, and 8 answered e. Next, we demonstrated our visual examples to the students and presented the same question to them again. Their responses (after seeing the examples) were as follows: 9 answered a, 4 answered b, 8 answered c, 34 answered d, and 14 answered e. Comparing the results “Before” and “After” exposure to the examples, we can see a better improvement in their motivation. As with the previous experiment, for choices a, b, and c, if the number of responses decreased it meant a positive response, which is what occurred. While for the other choices d and e, an increasing number of responses meant a positive response, which also occurred. These results are summarized in Figure 13. We note that the motivation in the case of junior students (second experiment) was better than that of the senior students (first experiment). This result might be explained by the fact that the juniors had not studied the course before. A preliminary study shows that the integrated environment can improve the learning process of computer engineering students who study the theory of computation course and related courses. Last semester, the students were divided into four groups, each group containing 20 students. A set of 40 randomly selected exercises was distributed among the groups, 10 for each group. Each group members could collaborate inside their group but not with any other group members. No group could see the exercises of other group. Two groups were asked to answer their assigned exercises using the integrated environment and the other two groups without using it. An equal time period was provided to all the groups. The result showed a better performance for the two groups using the IE. Then, the experiment was repeated by redistributing the exercises among the four groups. Again, the two groups with the IE showed better performance.

An Integrated Virtual Environment for Active e-Learning in Theory of Computation

431

5 Conclusion With the vast advance in technology, the traditional lecture-driven classroom is giving way to a new and more active environment, where students have access to a variety of multimedia and interactive course materials. Such interactive course materials have already been introduced for several topics in engineering courses; see for example [5, 6, 7, 14, 15, 18]. In this paper, we followed the same path and introduced an environment which integrates a set of visual tools to support interactive learning in the theory of computation course. It can also be used in other courses such as model of computations, language processing, automata and formal languages, compiler design, discrete mathematics, and other similar courses. Through the results of our experiments, we also showed that our visual tools can enhance learners’ motivation and performance. In addition an opinion poll showed a positive feedback on the environment tools from the students. In future work, we plan to enhance our visual tools by adding more features, more visual examples and games, and by performing more performance evaluation experiments. The experiments and the opinion poll showed that our integrated environment and its components are useful tools for supporting online e-learning in a variety of courses. Our tools are web-based, easy-to-use, all-in-one, and stand-alone which make it a useful tool of e-learning.

References [1] H. Bergstrom, Applications, Minimization, and Visualization of Finite State Machines. Master Thesis. Stockholm University, 1998. Related website at: http://www.dsv.su.se/ ~henrikbe/petc/. [2] J. Bovet, Visual Automata Simulator, a tool for simulating automata and Turing machines. University of San Francisco. Available at: http://www.cs.usfca.edu/~jbovet/ vas.html , 2004. [3] N. Christin, DFApplet, a deterministic finite automata simulator. Available at: http://www.sims.berkeley.edu/~christin/dfa/. 1998. [4] R. Felder and L. Silverman, Learning and teaching styles in engineering education. Engineering Education, Vol. 78, No. 7, pp. 674-681, 1988. [5] S. Hadjerrouit, Learner-centered Web-based Instruction in Software Engineering.IEEE Transactions on Education, Vol. 48, No. 1, pp. 99-104, 2005. [6] M. Hamada, Web-based Tools for Active Learning in Information Theory. To appear in the ACM SIGCSE Vol. 38, 2007. [7] M. Hamada, Visual Tools and Examples to Support Active E-Learning and Motivation with Performance Evaluation. Lecture Notes in Computer Science, Vol. 3942, pp. 147155, 2006. [8] E. Head, ASSIST: A Simple Simulator for State Transitions. Master Thesis. State Univesity of New York at Binghamton. 1998. Related website at: http://www.cs. binghamton.edu/ ~software/. [9] N. Herrmann, The Creative Brain. Lake Lure, NC: Brain Books, 1990. [10] Java2D of Sun Microsystems www.sun.com

432

M. Hamada

[11] Java Team, Buena Vista University, http://sunsite.utk.edu/winners_circle/education/ EDUHM01H/applet.html. [12] J. Keller, Development and use of the ARCS model of motivational design. Journal of Instructional Development, Vol. 10, No. 3, pp. 2-10, 1987. [13] D. Kolb, Experiential Learning: Experience as the Source of Learning and Development. Englewood Cliffs, NJ: Prentice-Hall, 1984. [14] S. Li and R. Challoo, Restructuring an Electric Machinery course with Integrative approach and computer-assisted Teach Methodology. IEEE Transactions on Education, Vol. 49, No. 1, pp. 16-28, 2006. [15] J. Masters and T. Madhyastha, Educational Applets for Active Learning in Properties of Electronic Materials. IEEE Transactions on Education, Vol. 48, No. 1, 2005. [16] M. Mohri, F. Pereria, and M. Riley, AT&T FSM Library. Software tools. 2003. Available at: http://www.research.att.com/sw/tools/fsm/. [17] I. Myers, Gifts Differing. Palo Alto, CA: Consulting Psychologists Press, 1980. [18] R. Nelson and A. Shariful Islam, Mes- A Web-based design tool for microwave engineering. IEEE Transactions on Education, Vol. 49, No. 1, pp. 67-73, 2006. [19] S. Rodger, Visual and Interactive tools. Website of Automata Theory tools at Duke University, 2006. http://www.cs.duke.edu/~rodger/tools/. [20] P. Rosati, The learning preferences of engineering students from two perspectives. Proc. Frontiers in Education, Tempe, AZ, pp. 29-32, 1998. [21] B. Soloman and R. Felder, Index of Learning Style Questionnaire. http://www.engr.ncsu.edu/learningstyle/ilsweb.html. [22] The Apache Software Foundation, Web site: http://www.apache.org. 2006. [23] Apache Tomcat, Web site: http://tomcat.apche.org. 2006. [24] Transforming undergraduate education in science, mathematics, engineering, and technology. In “Committee on Undergraduate Science Education”, Center for Science, Mathematics, and Engineering Education. National Research Council ed. Washington, DC: National Academy Press, 1999.

An Interactive Bomberman Game-Based Teaching/ Learning Tool for Introductory C Programming Wai-Tak Wong and Yu-Min Chou Deparment of Information Management, Chung Hua University Hsinchu, Taiwan [email protected], [email protected]

Abstract. Edutainment teaching/learning tools must be very “entertaining”. Otherwise, the tools will lose its attraction soon or later. In order to become formal tools in everyday teaching/learning process, they must be something more and also “something less” than an ordinary computer game in some aspects. We report our design and development of the proposed interactive Bomberman game-based teaching/learning tool that we are authoring. It brings modern education concept in concert with the classical teaching and laboratory work. While are reading/writing C codes in the proposed game-based environment, their C codes will control the movement of the Bomberman accompany with game music. This vivid learning environment can engage students spending their precious time for extensive practice since students have highly motivation to win the game. We hope that we have shown a new path for educational practice with computer games for teaching C programming course. Keywords: Edutainment, Bomberman, Interactive game, Programming Course, C Programming language.

1 Introduction The real issue of the first computer language is not what language to teach first but how to approach the teaching of the language [1]. The syntax and language constructs are really only a very small part of programming. The programming courses aim at building an effective foundation for the development of programming skills by emphasizing problem solving, logical thinking and programming process [2]. Problem solving and logical thinking involve a series of mental operations directed toward some goal [3] while programming process involves a series of computer operations. However, many students have become so weak in the development of learning strategies since their elementary education, especially for problem solving and logical thinking. Therefore, the challenge of the instructors faced is that a significant number of their students are not highly motivated to learn the programming language. Computer games have become an integral part of the popular culture in modern societies. Moreover, “game-based programming” is the latest buzz word in the computer science educational curriculum. Research [4] shows that students today have a totally different way of learning – react more to interactive learning. If they are not entertained while they learn, the instructor has lost them. However, much of K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 433–444, 2007. © Springer-Verlag Berlin Heidelberg 2007

434

W.-T. Wong and Y.-M. Chou

content that needs to be learned by students today lacks of motivation to them. The word “boring”, “dry” and “too technical” often crosses their lips [5]. Finally, it leads to frustration. A good game helps students to enhance their learning techniques, such as learning by doing, learning from mistakes, goal-oriented learning, discovery learning, task-based learning, question-led learning, and etc [6]. Although game-based learning has been made a good progress in academic research [7], using computer games for educational purposes has been rather uncommon. Although learning by playing has been reported to education [8], nevertheless, it is still less popular in postelementary education. In this paper a new approach to the student activities that supplements the ordinary auditorium lectures is proposed. We apply a computer game, called Bomberman, to develop a C programming language course. Bomberman is a strategic maze-based computer game originally developed by Hudson Soft [9]. The original game was published in 1983 and new games in the series are still being published in this day. The current version is the fifth generation called “Bomberman Land”. In this approach, students explore the problem in the form of games. The games-based approach provides an enriched interactive learning environment by asking students to come up with appropriate solutions to solve problems. The rest of paper is organized as follows. Section 2 describes how we teach introductory c programming language course in the past. Section 3 outlines the software design of the proposed Bomberman game-based teaching/learning tools. Section 4 shows some implementations of our tool. Then, we conclude with discussion of current and future work.

2 How to Teach Introductory C Programming Language C, a high-level programming language, is powerful, fast, compact and portable. After three decades of its birth, C language is still one of the most important and popular programming language today. In practice, there are seven steps to write a C program [1, 10]: (a) Discover and understand the problem (b) Design the program (c) Write the code (d) Compile (e) Run the program (f) Test and debug the logic error (g) Verify that the problem has been solved According to these seven steps, we set up our teaching procedure. First of all we teach the skill of mastering problem solving to the student. Secondly, we emphasize the logical thinking of how to solve the problem. Then, we teach the syntax and language constructs of C programming language. Followed by that, we emphasize how to translate our logic thinking into C codes. After the introduction of compiling, linking and loading process, we teach how to use the integrated development environment such as Microsoft Visual Studio to compile and run the program. Also, we demonstrate some fundamental debugging procedure.

An Interactive Bomberman Game-Based Teaching/ Learning Tool

435

However, this traditional teaching approach cannot keep our students’ engagement, not even an hour – certainly no shouting with glee at their successes and no desire to overcome their failures [5]. It is because there is no fun in the formal teaching. Draper [11] suggests that fun is associated with playing for pleasure with their own sake through freedom of choice. Carroll [12] summaries “fun” as “Things are fun when they attract, capture, and hold our attention by provoking new or unusual emotions in contexts that typically arouse none, or arousing emotion not typically aroused in a given context.” To provide an engaging learning environment, keeping students’ attention by providing fun may be a way not to lose our students. Edutainment may be a solution for teaching introductory programming courses. Many researchers have used virtual reality technology as a modern tool for fun learning of Physics Biology and even Molecular Biology [13,14]. Currently, researchers have begun to consider the edutainment approach to teach programming course. Rajaravivarma [2] proposed a game-based approach for teaching the introductory programming course. He used two broad categories of game, word games and number games, in his course materials. The course started from some simple games that utilized basic programming skill and additional programming skills were required through the course progress. However, all proposed games are textbased, not interactive and with no game music. Goschnick and Balbo [15] proposed to motivate students by giving assignments of 2D board game such as Snakes and Ladders, Ludo and Checker for the Information Systems students. However, those students had to have first well learned the SG_Board library [16] and then they could deliver the assignment. SG_Board Library is a generic board game class developed in Java language. Their approach looks like a game programming course more than an introductory programming course. The board games supported by the library are mainly for the primary school students to play. Therefore, students who take the course may not really enjoy the game that they have developed. Nevison and Wells [17] presented a case study based on a maze as an example that provided a complex framework to teach an introduction course in Computer Science. They found out that using different structures for maze could be a rich source for illustrating design patterns. Although they claimed that one of the authors had used this concept in an introductory Java programming class with great success, what they proposed was an initial framework or a stretch of their work. In the next section, we are going to present our interactive Bomberman game-based teaching/learning tool. Our tool has the following characteristics: interactive, good graphics, fascinating music in different stage, applying constructive learning theory and especially learning with fun.

3 Software Design From the opinion of Din [6], a successful education game has the following five main factors: (a) The game itself must be immersive. (b) The playability of the game must be elevated. (c) The game must be attractive, challenging, and competitive. (d) The game should offer a goal or several goals for players to achieve. (e) The game should allow players to track and manage their progress.

436

W.-T. Wong and Y.-M. Chou

Based on the aforementioned characteristics and successful factors, we design our Bomberman teaching/learning tool. There are seven components in the proposed tool shown as follows: (a) Road Map – an overview of the teaching plan of selected chapters from the text book over the semester. (b) Presentation – provide a platform for reading the teaching materials (c) Example – explore the fundamental programming paradigms through the demonstration of the Bomberman game. (d) Exercise – structured lab works for students to practice the fundamental programming concept while playing the Bomberman game. (e) Test – evaluate the learning result of students through problems solving while they are playing the Bomberman game. (f) Discussion – Provide a communication platform for students to feed back. (g) Progress Evaluation – report the achievement of students by comparing the benchmarks of participants’ progress to point out personal learning developments. 3.1 Welcome Page In order to give our students an exciting beginning, we provide a welcome page with Musical Instrument Digital Interface (midi) game music called “battle” as shown in Fig. 1. The welcome figure we used comes from an open source project “java Bomberman” administrated by Michel [18]. There are eight main functions including the seven aforementioned components and the exit function for manipulating the proposed C programming teaching/learning tool. 3.2 Road Map In order to give our students an overview of the programming course, we provide a road map as shown in Fig. 2. Each castle contains the course materials of a chapter to learn, examples to explore, exercises to practice and tests to pass. When the cursor falls at the castle, a window will be pop up to show the brief description of that chapter. When the student click the “Start” button, the sprite “Bomberman” starts to go forward to the next castle and experience the scenario of the next level if the student has already won the game in the pervious chapter. Otherwise, it stays on the same place. If the “Bomberman” reach the exit, it means the student has gone through all chapters. 3.3 Presentation We provide a function to let the instructor present the course materials in slideshow. The slideshow program was rewritten from the open source Jxpose [19]. It is a presentation making software, a "Microsoft PowerPoint" like software written in Java. We have integrated the Jxpose software into our tool. If the student have entered the first castle in the aforementioned Road Map and then hit the left side “Presentation” button, then the slideshow of the first chapter course materials will be presented on

An Interactive Bomberman Game-Based Teaching/ Learning Tool

437

Fig. 1. The welcome page including eight buttons for manipulating the proposed C programming teaching/learning tool

the screen. It also provides a full screen show capability. Fig 3 shows the slides of the chapter 1 prepared from our textbook [10]. 3.4 Example Examples can enrich the teaching contents and engage the learners. Since learning happens when learners work with the materials, we provide a platform to let students learn through example by reading (in this section) and writing new code (in the next two sections). The code is presented in the context of the game as a small piece of code that controls the movement of the sprite, “Bomberman”. The success of the change is directly determined by the behavior of the sprite in the game world. Moreover, the examples and the exercise can ignite students in the habit of learner-tolearner interaction, collaboration and mutual assistance. In our tool, we provide a component ‘Example’ which contains two sub components: “Code Reading” and “Run”. Students can switch to another subcomponent by selecting the tab of the sub component’s frame. In the ‘Code Reading’ sub component as shown in Fig. 4 we demonstrate the description of the example and the sample function written in C language that works for that example. In the ‘Run’ sub component as shown in Fig 5 a “Bomberman” locates in the top left corner of a maze which consists of a grid of floors, blocks and trees. The walker “Bomberman” in the maze can only move into floor but not blocks and trees. The bottom left corner is designated as the goal to be reached to finish the example. Once the student click the “start” button, the bomber will start to walk according to the direction of the “moveit()” sample function. Apparently, the walker can reach the goal by continuously going down or going right if it can move forward. When the walker reaches the left corner, a popup window presents to notify the achievement. Then the walker stops at the bottom left

438

W.-T. Wong and Y.-M. Chou

location. The student can stop the movement of the walker anytime and also change the speed of movement by clicking the “Decr. Speed” and “Incr. Speed” button. When the speed is changed, the label of “SPF” button which means the millisecond per frame is also updated. Each button press causes 25 milliseconds per frame increment or decrement. Current setting is 250 milliseconds per frame.

Fig. 2. The “Road Map” component of the C programming teaching plan

Fig. 3. The “Presentation” component” provides the slideshow functionality to demonstrate the C programming teaching material

An Interactive Bomberman Game-Based Teaching/ Learning Tool

439

Fig. 4. The “Code Reading” sub-component of the component “Example”

Fig. 5. The “Run” sub-component of the component “Example”

3.5 Exercise The games-based approach enables students to tie in the design with coding. The game environment lets students practice skills in virtual scenarios. The drill-andpractice in computer games context may facilitate students to enhance learning and sustain interest for given topics over time. Running their “Bomberman” under control of their written code may enlighten their exercise desires. There are also two sub components of the “Exercise” component. They are ‘Edit/Compile’ as show in Fig. 6 and ‘Run’ as shown in Fig. 7. In the “Edit/Compile” sub component, we provide the

440

W.-T. Wong and Y.-M. Chou

same layout as the “Code Reading” sub component of the “Exercise” component with four additional buttons. They are “Clear”, “Load”, “Save” and “Compile”. “Clear” button clears the central scrollable text area that is used for coding. The “Load” button let students load any file with “.c” extension while the “Save” button let students save their written codes. The “Compile” button compiles the written codes in the central text area by Java C compiler (JavaCC) [20]. Once the student finishes the compilation, they can press the tab of “Run” sub component and give a trial. The “Run” sub component of the “Exercise” component is same as that of the “Example” component except that it has one more button to show the points that the student has gained. If we examine the two scenarios of Fig. 5 and Fig. 7, the difference is that there are some fruits located in the maze of Fig. 7. The exercise requires students to write a “moveit()” function that not only can reach the bottom right corner but also can eat the fruits as many as possible. The exercises train students with the problem solving capability to further pass the evaluation in the “Test” Component

Fig. 6. The “Edit/Compile” sub-component of the component “Exercise”

3.6 Test The “Test” component provides an excellent test bed for students. During testing, students construct their own knowledge by testing ideas and concepts based on prior knowledge and experience in the “Presentation”, “Example” and “Exercise” component. By asking them to apply to a new situation, they need to integrate the new knowledge with their pre-existing intellectual constructs to solve different problems of the topics in the chapter. However, students can test their programs simply by clicking a button. The “Test” component is the same as the “Exercise” component except that it has a timer to log the elapse time for each testing question. Certainly, the testing questions are different from the exercise. They are more difficult and more

An Interactive Bomberman Game-Based Teaching/ Learning Tool

441

exciting. Due to the restriction of paper length, we only show the “Run” sub component as shown in Fig. 8. The test requires students to write a “moveit()” function that not only can reach the bottom right corner but also can eat the fruits as many as possible. However, the “Bomberman” cannot touch any bombs. Otherwise, the bomb gets blown up with a great sound and the game is over. Students are then required to revise their written programs in the “Edit/Compile” sub component.

Fig. 7. The “Run” sub-component of the component “Exercise”

Fig. 8. The “Run” sub-component of the component “Test”

442

W.-T. Wong and Y.-M. Chou

3.7 Progress Evaluation Progress Evaluation collects information to determine what the impacts of the tests are on students at various stages of the intervention. By measuring the correctness and the time spent for each test question of each student, we can show a ranking report by using JFreeChart [21] package to encourage or to warn the student about their ranks. Students with poor ranks are required for extensive practice. A sample progress report of a student is shown in Fig. 9. The top portion mentions the overall performance. The bottom report shows the number of trials ranking of the tester. The red bar means that the tester got a failure in that problem.

Fig. 9. The ranking report generates from the component “Progress Evaluation”

4 Software Development This project is written in Java language. The graphical user interface is implemented by using Java Extension Swing package. The game music is implemented by Java Extension Sound MIDI package. To accomplish each component of this project, several open source projects are integrated to our tool: (a) JavaBomberman [18] – the “Bomberman” game environment (b) Jxpose [19] – the presentation environment

An Interactive Bomberman Game-Based Teaching/ Learning Tool

443

(c) JavaCC Eclipse Plugin [20] – C compiler (d) JFreeChart [21] – the statistical reporting (e) JDBC – Java database connectivity for accessing the student’s information in the remote database The main idea why the student’s C programs can be worked as the movement of the Bomberman is the class reloading capability in Java Virtual Machine. Basically, there are two solutions. One is to use Jar Class Loader (JCL) [22]. JCL is a simple Java API that allows developers to load classes from Java Archives (JARs). It uses a custom class loader to load classes from JARs that helps in loading multiple versions of the same class file. Another solution is to use the technique involved loading the reloadable class with a separate class loader. The solution is demonstrated in a book written by Patrick Chan [23]. Each time the class needs to be reloaded, it is loaded using a new class loader and the previous class loader (with the old class) is abandoned.

Exercise (Edit/Compile)

Write C codes moveit()

JavaCC Compiler

Insert C Codes of moveit() into Java Template Class

Java Compiler

Exercise (Run)

Start

Dynamically Class Reloading

Class file Codes Injection

Invoke moveit() method()

Fig. 10. The work flow of the Exercise/Test component to provide a platform to transit the written C codes insensitively to control the movement of the Bomberman

5 Conclusions The specific goal of this study as presented by this article is to provide an interactive game-based teaching/learning platform for teaching introductory C programming language. We have successfully embedded the learning topics in the context of Bomberman game environment in such a way that the teachers can build a meaningful game environment to specify the sequence of topics for students to learn. Furthermore the difficulty of integrating the real world learning topics into the Bomberman game has been addressed by “codes injection” and “dynamically class reloading”. We have

444

W.-T. Wong and Y.-M. Chou

shown an alternated path for educational practice with computer games for teaching C programming course.

References 1. Howell, K.: First Computer Languages. Journal of Computing Sciences in Colleges archive. Vol. 18, No. 4, (2003) 317-331 2. Rajaravivarma, R.: A Games-Based Approach for Teaching the Introductory Programming Course. ACM SIGCSE Bulletin archive. No. 4, Vol. 37, (2005) 98-102 3. Mayer, R.E.: Thinking, Problem Solving, and Cognition, Freeman Publishing, San Francisco, CA (1983) 4. Feldgen, M. Clua, O.: Games As A Motivation for Freshman to Learn Programming. 34th ASEE/IEEE Frontiers in Education Conference, Vol. 3. (2004) S1H/11- S1H/16 5. Prensky, M.: Digital Game-Based Learning. ACM Computers in Entertainment, Vol. 1, No. 1, (2003) 1-4 6. Din, H.W.-H.: Play to Learn: Exploring Online Education Games in Museums. International Conference on Computer Graphics and Interactive Techniques, No. 13 (2006) 7. Squire, K., Jenkins, H., and Hinrichs, R.: Games-to-Teach Project: Envisioning the Next Generation of Educational Games. Educational Game Conference, Edinburgh, Scotland. (2002) 8. Roussou, Maria: Learning by Doing and Learning Through Play: An Exploration of Interactivity in Virtual Environments for Children. ACM Computers in Entertainment, Vol. 2, No. 1, (2004) 9. http://en.wikipedia.org/wiki/Bomberman 10. Prata, S.: C Primer Plus. 5th edn. SAMS, Indianapolis Indiana (2004) 11. Draper, S.W.: Analysing Fun as a Candidate Software Requirement. Personal and Ubiquitous Computing, Vol. 3, No. 3, (1999) 117-122. 12. Carroll, J.M.: Beyond Fun. Interactions. Vol. 11, No. 5, (2004) 38-40 13. Amon, T.: Web Virtual Reality Edutainment in Biology and Physics. Lecture Notes in Computer Science, Vol. 3942. Springer-Verlag, Berlin Heidelberg New York (2006) 810-817 14. Lu, B.F., Zheng J.M., and Cai Y.Y.: Learning Molecular Biology By VR Playing. ACM SIGGRAPH International Conference on Virtual Reality Continuum and Its Applications in Industry. (2004) 168-172 15. Goschnick, S., Balbo, S.: Game-first Programming for Information Systems Students. Proceedings of the Second Australasian Conference on Interactive Entertainment, Sydney, Austria (2005) 71-74 16. Goschnick, S.: An Object Lesson in OOP – A General Board Game Class. Proceedings of the first Australian Software Developer Conference, Sydney, Australia, (1992) 17. Nevison, C., Wells, B.: Using a Maze Case Study to Teach Object-Oriented Programming and Design Patterns. Proceedings of the sixth conference on Australasian computing education (2004) 207-215 18. http://sourceforge.net/projects/javaBomberman 19. http://sourceforge.net/projects/jxpose 20. http://sourceforge.net/projects/eclipse-javacc 21. http://sourceforge.net/projects/jFreeCharts 22. http://sourceforge.net/projects/JCL 23. Chan, P.: The Java(TM) Developers Almanac 1.4, Volume 1: Examples and Quick Reference 4th ed. Pearson Education, Indianapolis Indiana (2002)

A Study of Virtual Manipulative and Whiteboard System for Improving Multi-presentation Transformation of Geometry Problem Solving Wu-Yuin Hwang, Jia-Han Su, Jian-Jie Dung, and Yi-Shien Su* Department of Computer Science & Informatiom Engineering, National Central University, Taiwan {wyhwang,955402001,945402028}@cc.ncu.edu.tw *Graduate Institute Educational Technology, National Chia-Yi University, Taiwan [email protected]

Abstract. In this paper, one innovative VMW (Virtual Manipulatives and Whiteboard) system is proposed. It exploits multi-presentation mode transformation to improve geometry problem posing/solving process based on the pedagogical theory, “Children should construct their geometry concept from the concrete items to abstract ideas through physical or mental manipulation“. Besides some innovation in design and implementation, the proposed system had been evaluated with one pilot study to investigate its perceived acceptance, and analyzed students’ solving strategies from their whiteboard explanation. The results show our proposed system is useful and actually has positive effects for solving geometry problem. Keywords: Geometry problem solving, transformation, Virtual Manipulatives, Web3D.

Multi-presentation

mode

1 Introduction and Literature Review Geometry is a method that people use to explain and react with physical environment; it is also a tool that people learn mathematics and science. For this reason, enhancing geometry thinking is very important and helpful for high-level mathematic thinking, and it should be developed with spatial interaction in daily life [6],[16]. However, in traditional classroom, geometry learning is usually conducted only through the description of text, 2D images and mathematical formulas on whiteboard or paper. Especially in some important topics, as object area and volume measures, such teaching method has defects [16], result to most children do not understand physical meanings of the formulas on textbook. Tan [16] suggests that geometry concepts development like area and volume measure, they should be come from the experiences of covering and stacking manipulations. Promote children understanding through manipulating spatial objects (concrete experience), brainstorming in their mind (imagery concept) and writing symbolic solutions (abstract representation) [3],[6],[2]. When mathematical formulas are K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 445–456, 2007. © Springer-Verlag Berlin Heidelberg 2007

446

W.-Y. Hwang et al.

introduced and applied after these transformations, children will more understand the formulas [16]. These three kinds of presentation transformation are shown in Fig. 1. As we know, over the last several thousand years, humankind was forced to either physically build any 3D object or to map its shape into a two-dimensional drawing [4]. People usually use 2D symbols to represent 3D real objects in everyday life. Therefore, even in the 3D world, users need 2D-sketch tool like chalkboard or whiteboard to show their ideas. Therefore, in instructional activity, teachers and students use these tools to show their ideas by writing texts and drawing graphs. Before this research, Hwang et al. [9] have developed a web-based multimedia white-board system for improving mathematical problem solving with multirepresentations. With multimedia whiteboard system, students could use different representations like voice, graphics and formulas to show what they think and how they solve mathematical problems. The results showed students were satisfied with use of the multimedia whiteboard system and math-related oral explanation is significantly related to learning performance. Moyer et al. [12] define a virtual manipulative as “an interactive, Web-based visual representation of a dynamic object that presents opportunities for constructing mathematical knowledge”. Virtual manipulative promises to have great assets for classroom use, because computer materials have portability, safety, cost-efficiency, minimization of error, amplification or reduction of temporal and spatial dimensions, and flexible, rapid, and dynamic data displays features [17]. And about the learning performance, Triona et al. [17] has shown that the performance in child concept learning (CVS task) is equally well when taught with either virtual or physical materials. Now researchers have proposed a Virtual Manipulative and Whiteboard system (VMW). Children can solve geometry problems by manipulating virtual objects in 3D space, and write math equations or text on multimedia whiteboard. VMW system provides children more interactions like coordination, structuring, comparing and observing from different viewpoints than traditional teaching materials. It exploits multi-presentation mode transformation to increase the learning performance.

Fig. 1. In this research, the Virtual Manipulatives and Multimedia whiteboard tools are integrated to connect and facilitate the concrete, imagery and abstract transformation

2 System Design and Implementation 2.1 To Exploit Multi-presentation Mode Transformation To know how VMW system supports multi-presentation mode transformation, let’s imagine a fictitious instructional space in the classroom, in which the teacher and all

A Study of Virtual Manipulative and Whiteboard System

447

students take their whiteboard and stand around a table, where several kinds of geometric objects are put for studying geometrical problems and concepts. Every student can manipulate geometric objects or observe others to manipulate geometric objects, some new ideas or concepts come up in their heads; this kind of presentation transformation is from concrete manipulation or observation to imagery concept. [5] Afterward, the students write down their solutions on the whiteboard by symbol explanations, this process is from imagery to abstract. Students can also share their ideas with each other by whiteboard contents transferring. When one student studies other’s solution in the whiteboard, he or she may try to understand others’ symbol explanation (abstract to imagery). Furthermore, the student can manipulate geometric objects to validate others’ thinking (imagery to concrete). Therefore, the process is reverse transformation (abstract > imagery > concrete) (see Fig. 2).

－

Concrete to imagery (Virtual Manipulatives)

－

Imagery to abstract (Representation transformation)

Imagery to concrete Abstract to imagery Objects

Whiteboard

(Peer communication)

Fig. 2. This figure shows the process of multi-presentation mode transformation. Student manipulates virtual objects, and the transformation is from concrete to imagery. Then student expresses his idea on the whiteboard by text or graph symbols, the imagery to abstract transformation is going. Also, the transformation is reversible.

2.2 Implementation To implement the 3D models in the VMW system, the open XML-enabled 3D standard X3D is employed, which proposed by the Web3D consortium. To help X3D application development, the Web3D Consortium also promoted the open java source project Xj3D. The open source project Xj3D provides Java API and toolkit for X3D application development and SAI (Scene Access Interface) could be used to access X3D objects with programming languages. Consequently, researchers have used X3D to describe the virtual 3D models in VMW system and employed Xj3D API to access X3D objects. In previous work, since the multimedia whiteboard was

－

448

W.-Y. Hwang et al.

implemented by Java [9], it is easy to integrate multimedia whiteboard into the VMW system. In addition, for content analysis in problem solving, we implemented manipulation-recording mechanism. Any manipulations conducted by users will be automatically collected and stored in database. For resources delivery and maintenance mechanism, the Java Network Launching Protocol (JNLP) [13] is employed. When a user login and launch application by JNLPhyperlink on the web page, the client program was downloaded, installed, updated and executed automatically, After launched, the client program is communicating with server by SOAP. To collaborate with students, instructor first initials a new geometric problem by using 3d geometries and whiteboard to build a new virtual space, and then post as a discussion topic. Each student then launches the client program by clicking icon to reply and solve the problem (the upper part of Fig 3). Students can study the problem by manipulating 3D objects, and then choose appropriate viewpoints to create their own whiteboards for writing their solutions (the lower part of Fig 3). Besides, students can also revised their solutions many times by replying their answer, or give comments to others’ solution by replying others’. This asynchronous method gives some advantages that students could help each other at any time, any place [11].

3 Research Design 3.1 Purpose To study the perceived acceptance and influences of geometry concept learning with proposed VMW system. Researchers found the elementary Math teacher and students then used VMW to support the geometric problem posing and solving activity in their related Math course. 3.2 Subjects There were 23 six-grade elementary school students, 17 males and 6 females, who enrolled in this experiment. The experimental period was one and half months. The students followed Math teacher’s instruction to solve eight geometry problems. 3.3 Activity Design Since the proposed system is web-based design, teacher and students engaged in problem posing/solving activity on web site. The activity was spitted into four phases. In the first phase, teacher stacked geometric objects to build a new space and descript problem on a multimedia whiteboard for problem posing. Next phase, students started to solve problem and expressed their solutions on whiteboard individually. When all students completed their answers, the third phase is started, all students proceeded to peers reviews and solution revision works. They reviewed others’ solutions and gave

A Study of Virtual Manipulative and Whiteboard System

449

comments on the whiteboard, and then revised his own solution according the feedback from peers’ comment. In the fourth phase, researchers evaluated students’ solutions and analyzed data that system collected. To investigate the perceived technology acceptance of our proposed system [7], [8], the students were asked to fill out satisfaction questionnaires after the activity. The questionnaire consists of total 18 items involved perceived usefulness and perceived ease of use.

Into Geometry Manipulative Space.

Upload, Save Geometry Objects Cut, copy, paste

Undo Redo

statistic Add / Select Whiteboard

Multimedia Whiteboard

Manipulative Space

Text & Drawing Tool

Undo Redo Cut, copy, paste delete

I nsert Image Voice record

Fig. 3. Students clicked the icon (surrounding by blue circle) to launch client program for manipulation. The lower part was the screenshot of the 3D space, in the center is one translucent whiteboard and the 3D space can saw through it. The upper toolbar is functions for manipulating 3D objects and the lower toolbar is functions for multimedia whiteboard.

450

W.-Y. Hwang et al.

Fig. 4. Teacher used whiteboard to describe math problems and students replied asynchronously through whiteboards. The upper part is teacher’s whiteboard that describe math problem. Then lower part is two students’ whiteboards and they discussed how to solve this problem.

4 Results and Discussion 4.1 Questionnaire The questionnaires were sent to 23 students, and then 20 valid questionnaires returned. Table 1 and 2 were the results of questionnaire. The subjects agreed the VMW system was very useful and the values are more than 4.0 averagely. As for ease of use, most subjects had positive attitude toward the VMW system. However, some perceived ease of use items revealed that subjects got some troubles with using proposed system. After interviewing, the teacher found that although the whiteboard

A Study of Virtual Manipulative and Whiteboard System

451

Table 1. Perceived usefulness NO Perceived usefulness

SD D U A

SA Ave.

1

2

3

5

1.

Teacher used multimedia whiteboard and 3D 0 geometry objects for problem posing, it is clearer than using text only.

1

4

8

7

4.05

2.

VMW has free viewpoint, it is helpful to solve 0 geometry problem for me..

0

3

6

11

4.04

3.

The multimedia whiteboard provides text inputs, 0 sketching and voice recording tools etc., it meets my needs for geometry problem solving.

4

8

8

4

3.80

4.

VMW promotes me to have wide thinking.

0

0

3

10

7

4.20

5.

I can show my solution more completely in VMW.

0

0

5

9

6

4.05

6.

Using VMW lets geometry problem solving activity 0 become more fun.

1

6

5

8

4.00

7.

I think it is useful that I can put multimedia 0 whiteboard at any position in virtual space.

0

1

12

7

4.30

8.

The VMW is useful for geometry problem solving.

0

0

5

7

8

4.15

9.

The tools that VMW provided can help me to 0 discover solution.

1

7

8

4

3.75

4

SD: Strongly disagree, D: Disagree, U:Unsure, A:Agree, SA: Strongly agree, AVE.: Average.

is integrated in virtual space, some subjects preferred using traditional tools, pen and paper in reasoning process; the whiteboards was just used for describing solutions only. 4.2 Content Analysis Math teacher posed eight geometry problems in the research. They are all area and volume reasoning problems. The eight problems are divided into two categories, irregular and regular stacked geometric shapes. In the irregular category, some basic blocks are employed to build irregular shape. Since these shapes are not regular, it is not easy and straightforward to find math rules to get answers. First students need to re-stack basic blocks into regular or near regular shape and find some efficient approach to get the answer. In regular one, blocks are stacked in specific rules that students can find the stacking rule for the solutions. In regular stacking problems, the number of blocks is huge, if students use straightforward counting method to get the volume of the shape, it is time-consuming and will have high probability to make some mistakes. Two problems, one is regular and the other is irregular, were chosen for further content analysis and called problem A and B respectively (see Fig. 5).

452

W.-Y. Hwang et al. Table 2. Perceived ease of use

NO.

Perceived ease of use

SD

D U A

SA

1

2

3

4

5

Ave

1.

The view navigation function is easy for me in VMW.

0

3

7

9

1

3.4 0

2.

It is easy to manipulate objects in VMW,

0

4

8

7

1

3.2 5

3.

I think the color and transparency state change 1 functions are easy to use.

2

9

6

2

3.3 0

4.

I learned and knew quickly for manipulating objects in 0 VMW.

1

8

9

2

3.6 0

5.

The text input function of multimedia whiteboard is 0 easy to use.

3

2

9

6

3.9 0

6.

The function of 2d geometry drawing of multimedia 0 whiteboard is easy to use.

4

7

7

2

3.3 5

7.

I learned and knew quickly to use multimedia 0 whiteboard.

1

6

10

3

3.7 5

8.

Totally, I had few trouble with using VMW system.

0

6

9

5

0

2.9 5

9.

Totally, I think the VMW is easy to use.

1

3

7

7

2

3.3 0

SD: Strongly disagree, D: Disagree, U:Unsure, A:Agree, SA: Strongly agree, AVE.: Average.

Students’ whiteboard contents were analyzed to find their solving strategies. For showing solving strategies clearly, Tables were used to describe their strategies and the distribution of correct answers and wrong answers.. Their solving strategies of problem A and B were described in details as followings. Problem A: How to calculate the volume and surface area of irregular stacking blocks. In this problem, Math teacher used 19 basic blocks (cubes) to stack a three level irregular object. Since the stacking method had no specific rule, students were expected to use some more efficient way to figure out the answers rather than just straightforward counting. In respect of volume part, there was only one student getting the wrong answer due to computation mistakes. Table 3 shows that more than half of the students used restacking method to rearrange basic blocks into regular or near regular shapes , the others used straightforward counting to get their answers. By using re-stacking method, students re-stacked them into a one-level or multi-level rectangles or cuboid. However, there may be surplus cubes left or some cubes lacked. Therefore, four students first calculated the volume of cuboid (3D), and then added remaining cubes

A Study of Virtual Manipulative and Whiteboard System

453

or subtracted lacked ones to get their answers (reference Fig. 5-A). And other three students re-stacked all cubes into many one-level rectangles with different size and added up the volume of all rectangles to get the answer. Five students decomposed the object into one single-layer rectangle and calculated the answer. Table 3. The distribution of volume solution methods in problem A, the most half students used re-stacking method to simplify the computation Method

Correct

Wrong

1. Straightforward Counting method

10

0

2. Re-stacking Method

12

1

2.1 Multilayer Stack

7

0

2.1.1. Stack into rectangles with variable sizes

3

0

2.1.2. Stack into cuboid

4

0

5

1

22

1

2.2. Single-layer Stack Total

As for surface area, it is more difficult than volume reasoning part. See Table 4, there were four students who got wrong answers because of computation mistakes or misconception. The students, who got right answers, use some more efficient way to get the correct answers by applying some rules they found. The rule is among six surfaces of irregular object; they can be divided into three dimension pairs of surfaces which the same dimension have same surface area. Therefore, the sum of three different dimension surfaces' areas was first calculated and multiplied by 2 to get the answers. However, there were still five students only using straightforward counting method to get all surface area. Table 4. The distribution of surface area solution methods in problem A, the most students find some rule to simplify the computation Method 1. Counting method 2. Calculating three faces’ blocks and multiply by 2 3. Misconception Total

Correct

Wrong

3

2

16

1

0

1

19

4

Problem B: How to find the volume and surface area of pyramid. To study how students figure out the rules from regular stacking object, Math teacher stacked a seven-level pyramid. Students were asked to find its volume and surface area.

454

W.-Y. Hwang et al.

As for volume reasoning, Table 5 shows that most students used partitioning strategy. In horizontal partition strategy, the pyramid was partitioned into seven levels of squares. Each level's volume is calculated and added up together to get the pyramid's volume. The strategy was to divide the problem into several simple subproblems and conquer them. However, one student employed another approach, vertical partition strategy, to extend the pyramid volume vertically, then he can add up all slices to get the answer (show in Fig. 5-B). The result shows the versatile approaches are found in this study and VMW system would possibly promote students’ creativity. Table 5. The solution distribution of volume in problem B, as the same before most students used partition method to simplify the computation Method

Correct

1. Give up 2. Horizontal partition 2.1. Summing all levels 2.2. Specifying extra blocks in next level 3. Vertical partition (Outward extension) Total

Wrong

0

4

15

2

12

2

3

0

1

1

16

7

In surface area part (Table 6), only half of them found the correct solution. Seven students found the rule that top and bottom surface areas are equal (bottom surface area multiplied by 2) and four faces’ surface areas are equal (one face surface area multiplied by 4). The others used the method that figured out the all 6 faces' surface area. This problem seems more difficult than the previous ones; almost half students didn’t get correct answer. Table 6. The solution distribution of surface area in problem B Method

Correct

Wrong

1. Give up

0

5

2. Misconception

0

6

11

1

3.1. Bottom blocks count multiplied by 2+ one side surface area multiplied by 4

7

1

3.2. Summation of top, left and front surface area multiplied by 2

4

0

12

11

3. Specifying each faces’ surface area

Total

A Study of Virtual Manipulative and Whiteboard System

455

Fig. 5. Samples of students’ solving contents of problem A and B. Students described solution on the whiteboard

5 Conclusions In this paper, the Virtual Manipulatives and Multimedia Whiteboard are integrated to exploit multi-presentation mode transformation for enhancing geometry problem solving process. The Researchers evaluated the perceived of acceptance with satisfaction questionnaires. The result shows elementary students agreed the system is useful and has positive attitude toward ease of use. In addition, students’ whiteboard contents were analyzed for understanding how they solved problems by using VMW. The result shows students rely on stacking and partition methods to find the solutions. However, our tool needed to be improved to meet the maturity, especially in ease of use. For future research, the researchers will involve pattern recognition technology to classify students’ manipulation and compare with their solution from whiteboard, try to extract some important features from manipulation and find correlation between reasoning method, or other issues, like gender etc. It is hard to probe human’s deep mind, but it is possible to peek it from one’s outside interaction and verbal expression [1] . Therefore, to know correlation between manipulation and solving strategy is helpful to discussing geometry concept learning in children.

References 1. Ashcraft, M. H.: Cognition. Prentice Hall, (2001) 2. Battista, M. T., & Clements, D. H.: Students'Understanding of Three-Dimensional Rectangular Arrays of Cubes. Journal for Research in Mathematics Education, 27 (1996) 258-292 3. Battista, M. T., & Clements, D. H.: Research into Practice: Using Spatial Imagery in Geometric Reasoning. Arithmetic Teacher, 39 (1991) 18-21 4. Bimber O, Encarnacao LM, Stork A: A Multi-Layered Architecture for Sketch-Based Interaction within Virtual Environments. Computers & Graphics, 24 (2000) 851-867

456

W.-Y. Hwang et al.

5. Chittaro, L., & Serra, M.: A Brief Introduction to Web3D Technologies in Education: Motivations, Issues, Opportunities. Proc. of LET-WEB3D 2004, (2004) 6. Clements, D. H., & Battista, M. T.: Geometry and Spatial Reasoning. Handbook of research on mathematics teaching and learning, (1992) 420-464 7. Davis, F. D.: A Technology Acceptance Model for Empirically Testing New End-User Information Systems: Theory and Results. (1985) 8. Davis, F. D., Bagozzi, R. P., Warshaw, P. R.: User Acceptance of Computer Technology: A Comparison of Two Theoretical Models. Management Science, 35 (1989) 982-1003 9. Hwang, W. Y., Chen, N. S., Hsu, R. L.: Development and Evaluation of Multimedia Whiteboard System for Improving Mathematical Problem Solving. Computers & Education, 46 (2006) 105-121 10. Kim, W., & Shih, T. K.: Distance Education: The Status and Challenges. Journal OF Object Technology, 2 (2003) 11. Moyer, P. S., Bolyard, J. J., Spikell, M. A.: What are Virtual Manipulatives?. Teaching Children Mathematics, 8 (2002) 372-377 12. Sun Microsystems: Java Web Start Technology. 2006 (2006/11/30) 13. Triona, L. M., & Klahr, D.: Point and Click Or Grab and Heft: Comparing the Influence of Physical and Virtual Instructional Materials on Elementary School Students' Ability to Design Experiments. Cognition and Instruction, 21 (2003) 149-173 14. Wu-Yuin Hwang, Jia-Han Su, Jian-Jie Dong: Development of Web3D Virtual Classroom System for Geometry Problem Solving. GCCCE 2006, (2006) 15. Ning-Jun Tan: Analysis of Elementary Geometry Teaching Material (In Chinese). New Elementary Mathematics Curriculum 1994, (1994) 16. Triona, L. M., & Klahr, D.: Point and Click Or Grab and Heft: Comparing the Influence of Physical and Virtual Instructional Materials on Elementary School Students' Ability to Design Experiments. Cognition and Instruction, 21 (2003) 149-173

Supporting Multi-device for Ubiquitous Learning Byoungchol Chang1, Mokhwa Lim1, Dall-ho Ham1, Yoonsoo Lee1, Sooyong Kang2, and Jaehyuk Cha1,* 1

Division of Information and Communications, Hanyang University Seoul, Korea 2 Department of Computer Science Education, Hanyang University Seoul, Korea [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]

Abstract. Rapid advancement of information and communication technologies has introduces various dimension of e-Learning environment such as ubiquitous learning, mobile learning and television learning. These technologies enabled learners to access to learning contents through variety of devices with more flexibility and consistency. In order to accomplish learning under these multiple environments, it is necessary to acquire and process the platform information that contains properties and status of the web-accessing devices. In this study, we introduce the design and implementation of a Platform Analyzer which is essential for learning systems that support multi-platform environment. We also have implemented an interactive DTV-centered multiplatform learning environment framework in combination with PC and PDA. Finally, we showed the potentiality of the multiplatform learning environment with design and adoption of learning scenario and sample contents we made in this study. Keywords: e-learning, m-learning, u-learning, digital TV, device profile.

1 Introduction It is no exaggeration to say that today’s society is a ‘Knowledge Society’ because we can access to diverse information through advanced information devices. With a variety of devices, we can access to the information through mobile phone, PDA and laptop with wireless Internet support even on the street. Moreover, the interactive digital TV nowadays provides information with mere operation of remote controller[1,2]. Many studies on e-Learning are underway to achieve such primary goal of eLearning which is to realize the learning with anything, at anytime, at anywhere by utilizing this variety of devices. The results of these studies have provided learners with various e-learning environments such as m-Learning, t-Learning and u-Learning. However, such multiple e-Learning environments have some common problems. For example, there are insufficient learning resources congruent with each device and it is difficult to reuse most of existing contents[3,4,5,6]. Consequently, many studies *

Corresponding author.

K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 457–467, 2007. © Springer-Verlag Berlin Heidelberg 2007

458

B. Chang et al.

are under way to seamlessly transform existing contents into effective and suitable form onto the multiplatform while maintaining the context and consistency of the learning content of which has been developed exclusively for a specific platform. However, many of these studies are only focused on automating the conversion of contents to be reusable on different devices without earnest consideration on the characteristics of the contents and devices. In order to achieve effective learning through multiple devices, it is necessary to provide contents that suits learners’ situation in consideration of various learning style and environment of learners[7]. Figure 1 show learning that can be delivered on a variety of information device. In order for such learning scenario to be possible, a Platform Profile processing mechanism which recognizes the learner device automatically and mines the property data is necessary. It is necessary to build sophisticated learning scenario that provides congruous contents to learning environment of learner with selecting mechanism and to set up framework to implement this learning scenario. Therefore, in this paper, we developed the Platform Profile processing mechanism which is the fundamental technique designed to support multiple learning devices. We also presented DTVcentered learning framework with learning scenario that learners can switch the platform of same content from one device to another in accordance with the learner’s need.

Fig. 1. Supporting Multi-device for u-Learning

Section 2 and 3 elaborates the details of Profile Processing and Profile Processor and is followed by section 4 which shows an experimentation using learning scenario with learning scenario and sample contents we propose in this paper. Finally in

Supporting Multi-device for Ubiquitous Learning

459

section 5, we consider the result of this study and suggest further researches in the future.

2 Processing Platform Profile In order to deliver the contents that are congruous to the device, it is necessary to acquire exact capabilities and states of user device. We define ‘Platform Profile’ as the value of capabilities and states of client devices. Platform profile processing mechanisms of previous works contains many unnecessary properties. Accordingly, in this paper we propose profiling mechanism which effectively discovers and processes the essential platform profile of user device. Figure 2 shows the profile processing procedure proposed in this study.

Fig. 2. Profile Processing Flow

2.1 Platform Profile Model The client system with its operating system and web browser has different capabilities and characteristics in accordance with the screen size, multimedia compatibility and the necessity of plug-in. Because of these differences among devices, platform profile which specifies the precise capabilities of them is necessary in order to transfer suitable contents from server to client. In the early stage of studies, most device identification mechanism only used information transmitted from HTTP Request header field in order to avoid web browser modification. Consequently, client information acquired by this mechanism is considerably limited to such as browser version or type of image. In order to solve this shortcoming, W3C DIWG (Device Independence Working Group)[8] proposed and recommended CC/PP (Composite Capabilities / preference Profile)[9]. A CC/PP profile describes device capabilities and user preferences that can be used to guide the adaptation of content.

460

B. Chang et al.

A CC/PP profile is broadly constructed as a two level hierarchy: a profile consists of a number of components and each component has a number of attributes. Most of such profiles are described with XML Serialization for RDF(Resource Descript Framework). UAProf proposed by OMA(Open Mobile Alliance) is a representative applicable specification adhering to the CC/PP standard[10]. UAProf component includes HardwarePlatform, SoftwarePlatform, NetworkCharacteristics, BrowserUA and WAP-characteristics. Each of these components comprises several attributes and each attribute includes Name, Collection Type, Attribute Type, Resolution Rule and so on as described in UAProf specification. 2.2 Platform Profile Processing DELI(A Delivery context Library for CC/PP and UAProf) is an open-source library developed at HP Labs that allows to resolve user request from CC/PP or UAProf capable devices and query the resolved profile. DELI performs profile resolution by processing Profile References and Profile Differences in accordance with the CC/PP standard. However, DELI does not automatically detect and collect capabilities of client device. Consequently, device properties modified by user cannot be instantly reflected and the profile of new devices must be registered on the server in order to make applicable. For this reason, DELI is not a suitable mechanism for a system aiming to maximize efficiency of learning by transferring optimized contents to the client device. Thus, in this paper we proposed an automated platform profile acquisition model in order to achieve efficient contents delivery and implemented platform profile processing mechanism that process the acquired profiles.

3 Proposed Profile Processing Mechanism 3.1 Overall Architecture The Platform Profile Analyzer, which we propose in this paper, is designed in compatible with previous profile processing system and we implement a system including a series of profile processing for e-Learning. 3.2 Profile Checker Profile Checker is an agent collecting platform information from user’s browser. The devices, such as desktop and PDAs, are in need of a profile checker because users can change the properties of the system easily. However, users cannot change its property of such devices like mobile phones. Thus profile checker is not necessary since the vendor provided profile is enough to identify the properties. For this reason, we only take the profile checkers of desktop and PDA into consideration. The profile checker collects some of the attributes defined in UAProf schema. In this paper, we propose two types of the profile checker: One approach, namely using JavaScript[11], can be executed on all platform regardless of the operating system, and other approach, namely using ActiveX Control[12], can be executed only in

Supporting Multi-device for Ubiquitous Learning

461

Windows operating system collecting more information than the former approach. First, profile checker using JavaScript is applicable in any system with web-browser regardless of the operating system. However, acquisition of information is limited. Information that can be collected with JavaScript on the Internet Explorer are only limited to such information as BitPerPixel, ScreenSize, BrowserName, JavaScriptEnbled, JavaScriptVersion, CPU and so on. The implementation of Profile Checker with ActiveX Control is based on the following guideline. Firstly, standard Windows API must be primarily used and secondly, search the platform’s registry directly if no suitable API are found. An ActiveX Control is an application program only working on the web browser. Because ActiveX Control provides full capabilities of general application, all necessary profile information could be searched. However, it is only limited to the Windows system and there is a difficulty in distribution. Figure 3 shows the architecture of Platform Profile Analyzer.

Fig. 3. Architecture of Platform Profile Analyzer

3.3 Profile Collector Profile Collector is a function implemented within Device Identifier and called when profile checker is requested from the user device. Profile Collector use a predefined table data structure that stores the name of the Profile Checker as a value and a string that identifies platform profile from the value of ‘user-agent’ of HTTP[13] Request header transferred by user as a key. Profile Collector first search this table and retrieve a profile checker appropriate for the client, and then transmit it. Later, when receiving the request from a profile checker, the profile collector calls a profile creator using the platform information included in the request object. Figure 4 shows the flow of data and the calling sequence between profile checker, device identifier and profile collector.

462

B. Chang et al.

Fig. 4. Profile Collecting Procedure

3.4 Profile Creator Profile Creator is consisted of the three sub-processes. One is to create profile difference from the result of Profile Checker, which is sent as a HTTP POST request. Another is to reorganize the different requests from HTTP/1.1, W-HTTP, HTTP-ex protocols and profile checker into integrated module that can be synchronously processed by CC/PP process module. The other is to create associated profile of reference profile and profile difference based on the standard CC/PP specification. Figure 5 depicts the three level procedure described above. The first procedure is to create profile difference from the data sent by profile checker. However, as described above, because the profile checker transfers the attributes defined in UAProf consisting table of the values of corresponding attributes, it is impossible to know the name of the component that contains each attributes. Moreover, a collection type information associated with each attributes is required in order to resolve bag type objects of RDF in a separate way. Therefore, the Profile Schema table is created into a structure as shown in Table 1 after reading the XML file that contains UAProf Schema that reference profile is following.

Fig. 5. Profile Creating Procedure

Supporting Multi-device for Ubiquitous Learning

463

Table 1. Profile Schema Table structure Attributes BitsPerPixel

Component HardwarePlatform

Collection Type Simple

ColorCapable

HardwarePlatform

Simple

CPU

HardwarePlatform

Simple

...

...

...

CcppAccept

BrowserUA

Bag

...

...

...

A request unification module extracts a profile references and profile diffs necessary for CC/PP process from request headers in various types of HTTP protocols such as HTTP/1.1, W-HTTP and HTTP-ex and call CC/PP process module. Finally, CCPP Process Module performs the creation of associated profile with reference profile and profile diffs. 3.5 Profile Selector In this study, perceived by the fact that attributes of user’s platform has similar characteristics within the device group, we configured a standard profile with outstanding attribute that is essential in the process of contents delivery. Profile Selector analyzes profile created by Profile Creator and selects the most congruous standard profile.

4 Adapting Learning Contents to Multi Device 4.1 Learning Scenario In pursuit of u-learning, we created learning scenario which enables learning at anytime, anywhere with a variety of information devices while maintaining the context and consistency of learning. In order to achieve this ultimate goal of elearning, the contents transmission mechanism was designed to maximize the effective use of device profiles in a way to maintain the context of learning resources. In the learning scenario, we assume that a student is learning English. Figure 1 illustrates the overall concept of learning scenario. The learning procedure in this scenario is as follows: z Situation 1: A student is learning with English contents delivered on the DTV DTV English learning content begins with login screen and the learner enters with user name and password. In the course of learning, learner can check any vocabulary or grammar for future review with remote control and checked contents are stored onto the LMS through the return channel. If there is a test during the broadcasting, the questions encoded with Xlet will appear on the screen. The learner then answer the question and it will be stored onto the LMS. Learners can go back to previous stages or repeat the question if necessary.

464

B. Chang et al.

z Situation 2: A learner has to leave home away from DTV On the way to other place, the learner continues to watch the same contents on PDA from the point where he or she stopped on the DTV. The learner also reviews the vocabularies marked on the DTV and resume the test stopped in the middle. z Situation 3: Resume learning with PC on arrival at the office The learner reviews the test result on the PC and search for more information on the Internet to replenish the learning of the day. Then he accesses to the community or sends e-mail to exchange information and have discussion with other learners achieving cooperative learning. All these activities of the learner are stored and managed within the LMS. In order to realize the scenario described above, an integrated Learning Management System that provides learning materials, keeps tracing the progress of learning and manages the learning history of a particular learner. In this study, we divided the functions elaborated in Table 2 into two groups: One is the function that can be implemented within actual learning contents and connected to the LMS, the functions elaborated are divided into two categories: On the one hand, there are functions to be implemented within actual learning contents part which will be connected to the management systems. On the other hand, there are functions to be implemented within web service part which aims at learning stage management on the LMS. Table 2. Profile Schema Table structure z Learning Courses works - Course selection and registration - Record of Progress report - Questionnaire - Cyber classroom z Tests and evaluations - Course Test - Quiz or TestPrep exam - Feedback Inquiry - Glossary section for wrong answers z Assessment evaluation - Assessment notification and submission - Feedback for assignment z Online Mentoring - Q&A board for learner-instructor interaction - Q&A feedback tracer function z Supplementary Learning Materials - Glossary, Reference or More About Functions z Community - Community Function for learner-learner interaction

Supporting Multi-device for Ubiquitous Learning

465

4.2 Implementation Learning contents delivered and executed on each device can communicate with LMS through the communication protocol of each respective devices.

Fig. 6. Delivery Structure of e-Learning contents for multi-device

Contents for PC and PDA environment were developed as HTML based contents and employed ECMAScript and API Instance to manipulate learning history. The information then is transmitted through Return Channel Server shown in Figure 6 via HTTP protocol to manage the learning history. On the other hand, Interactive TV do not support such mechanism adopted for PC and PDA to transmit the learning history because contents for today’s Interactive TV were developed with DVB/J[14] based Xlet application. We embedded a module which enables socket communication for learning contents of Interactive TV. The module performs collecting learning history and logging all transmissions functioning as ECMAScript on PC platform. In this study, we designed sample learning contents for experimentation on our framework. The main function we tried to experiment are continued learning through the different devices and tests that can also be continued on different devices. Given a learner having a test on the Interactive TV, the completed answers and elapsed time are passed onto the Return Channel Server through Xlet socket communication to be stored in the middle of the test. In our implementation, if the learner did not complete the test, the record of unsolved problems is stored and learner can continue the rest of the test with PC or PDA with continuity. Figure 7, 8 shows the test result in this study.

466

B. Chang et al.

Fig. 7. Test execution environment on the real Interactive TV and Emulator

Fig. 8. Test execution environment on PDA

5 Discussion In this study, we designed and implemented the Platform Profile Analyzer which is essential for e-Learning system that supports multi-platform environment. We also proposed a learning framework using multiple devices in accordance with a variety of learning situations through DTV (Digital TV) of which service is rapidly growing driven by advancement of interactive transmission technology. In this paper we proposed an automated platform profile acquisition model in order to achieve efficient contents adaptation and implemented platform profile processing mechanism that process the acquired profiles. We have showed that we can provide learners with flexible learning environment through the framework and experiment proposed in this study. Since it can increase the efficiency in learning for learners, it is expected to help improving the learning capabilities of learners in the future.

Supporting Multi-device for Ubiquitous Learning

467

In order to show more accurate and meaningful result on the test of the Profile Analyzer in our study, it is necessary to conduct further experimentation with abundant sample contents authored for each device. In this study, we used mechanically acquired platform information to transmit the platform congruous contents. However, in order to achieve more effective e-Learning, further research on including learners’ behavioral or emotional information into the platform profile and processing them are necessary. We also present the development methodology of learning contents using digital device congruent to the learning contents as for future work. Acknowledgments. This research was partially supported by the project(A ULearning System Based on One Source Multi Use Technology), sponsored by Seoul Metropolitan Government.

References 1. A. Dosi, B.Prario, New Frontiers of T-Learning:the Emergence of Interactive Digital Broadcasting Learning Services in Europe, ED-Media 2004. 2. Paivi Aarreniemi-Jokipelto. T-learning Model for Learning via Digital TV, Proceedings of 15th EAEEIE conference, Lappeenranta, 2005. 3. Joost Geurts. Constraints for Multimedia Presentation Generation. Master's Thesis, university of Amsterdam, 2002 4. M. Jourdan, et al. Madeus, an Authoring Environment for Interactive Multimedia Documents, ACM Multimedia'98, pp. 267-272, ACM, Bristol (UK), September 1998. 5. W.Y. Ma, I. Bedner, et al., A Framework for Adaptive Content Delivery in Heterogeneous Network Environments, Proc. MMCN2000 (SPIE Vol. 3969), 2000, San Jose, USA, pp86100 6. Kinshuk, Hong, H., Patel, A. (2002). Adaptivity through the use of mobile agents in webbased student modelling. International Journal of E-Learning, 1(3), 55-64. 7. W3C, Content Selection for Device Independence (DISelect) 1.0, http://www.w3.org/TR/ cselection/ 8. W3C DIWG, Device Independence Principles, Sep 2001, W3C Device Independence Working Group, http://www.w3.org/2001/di/. 9. F.Manola and E.Miller, Composite Capability / Preference Profiles (CC/PP) Structure and Vocabularies 1.0, http://www.w3.org/TR/2004/REC-CCPP-struct-vocab-20040115/, Jan 2004. 10. OMA, User Agent Profile, http://www.openmobile alliance.org/tech/profiles/index.html. 11. D.Flanagan, JavaScript The Definitive Guide, 1998. 12. Microsoft MSDN, Introduction to ActiveX Controls, http://msdn.microsoft.com/. 13. W3C, Hypertext Transfer Protocol 1.1, http://www.w3.org/Protocols/rfc2616/ rfc2616.html. 14. ETSI, Digital Video Broadcasting(DVB); Multimedia Home Platform (MHP) Specification 1.1.1”, ETSI, 2006

A Medical Aided System for Evaluating Crura Rehabilitation Fangyu Li1, Yefei Li2, Shouqian Sun1, Zhanxun Dong1, and Xianghong Xu2 1

College of Computer Science, Zhejiang University, Hangzhou, 310027, China {fionalfy,ssq,caid3}@zju.edu.cn 2 School of Mechanical and Electrical Engineering, China Jiliang University, Hangzhou, 310018, China [email protected], [email protected]

Abstract. This paper presents an overview of the current state of development of medical education with a special attention to the fields of rehabilitation. Changes in internet technology provide opportunities for rehabilitation. In this work, we apply medical resources and e-learning service technologies to construct a medical aided learning system. There is summarized philosophy at preparing the e-learning tool, presented achieved results and pointed out some practical hints that should be considering at development of medicine based elearning modules. Then the paper describes implement of the e-learning system for crura rehabilitation. Medical resources associated with the components of the learning system are encapsulated as reusable Learning agent. Integrated rehabilitation knowledge is used within rehabilitation of disabled people and interns. An example of evaluation of crura rehabilitation is demonstrated. Finally, the paper indicates sketched challenges and ways of future development of e-learning in the presented fields. Keywords: medical resources, cooperation environment, crura rehabilitation, elearning.

1 Introduction Physicians take the responsibility of not only therapy, but also training interns. By the end of last century medical educators began to realize that the demands from medicine changed gradually, they must continually adapt to changes in the practice of medicine. There should be more emphasis on skills and understanding rather than knowledge. Traditionally, the main objective of medical education consisted in acquiring knowledge by the students. The assessment was based on testing whether students could reproduce the acquired knowledge. Presently, the main objective of teaching is the development of student skills. This means that the medical educator is a coach in the process of the student knowledge development [1]. The medical education method transformed form teaching oriented towards learning oriented. K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 468–476, 2007. © Springer-Verlag Berlin Heidelberg 2007

A Medical Aided System for Evaluating Crura Rehabilitation

469

Another problem for medical educators was how to convey interactions. The use of animations seemed to be a solution. But from the intern’s point of view, the improvement was only partial. The positive and negative effects on teaching and learning of the different support system are summarized in [2]. The advance in the learning method does not automatically yield better understanding. New advances in the learning method present more tools in the same time period. There are significant numbers of Chinese with disabilities. Today estimates of the China general population with chronic health conditions which cause any activity limitation are 24.1 million [3]. The need for rehabilitation will continue to grow. Medical Rehabilitation seeks to maximize the functional abilities of people who have a disabling impairment due to injury, acute illness, congenital abnormality, or chronic health condition. It includes not only inpatient hospital care and outpatient clinics, but also home health services, day treatment programs for work hardening, pain management, and cognitive rehabilitation [4], [5]. New technologies should be developed to help efficient rehabilitation and evaluate it. Rehabilitation has been interdisciplinary since its beginning, with collaboration among patients, nurses, physicians, restorative therapists, rehabilitation counselors, assistive technologists [6]. The Internet technology drives a profound revolution in medicine and its spreading services determine a broad discussion about e-learning. Corresponding to this situation, the number of activities and programmes in connection with the application of new technologies in rehabilitation increases. The aim of this work is therefore to apply the learning object and e-learning technologies developed at Institute of Modern Industrial Design, Zhejiang University to address these limitations. A Medical Aided Learning System for interns is being developed to Evaluate Crura Rehabilitation. The organization of this paper is as follows. E-learning solution for crura rehabilitation is discussed in Section 2. The implement of system is introduced in Section 3. Section 4 shows an example of crural rehabilitation process and diagnosis.

2 E-Learning Solution for Rehabilitation Surface Electromyography (sEMG) is one of the most useful methods to solve the practical rehabilitation [7], [8], [9], [10], [11]. There have been efforts for many years to introduce sEMG into interns training. Fig. 1 shows a solution for rehabilitation evaluation: Information of input or output to patients was measured by training aided system, the information then feedback to rehabilitation evaluation system. The rehabilitation system modifies training system according to the results of evaluation. However, a general problem in this method is the fact that it deals with rather abstract electrical signals such as frequency, phase, swing and spectrum etc.. These medical quantities and phenomena are not observable or can only be made observable by means of measurements. On the basis of the problem analysis discussed above, it was identified that a possible solution at design of the learning aided system should meet these requirements in Table 1.

470

F. Li et al.

Fig. 1. Solution overview of rehabilitation evaluation Table 1. Requirements of e-learning for rehabilitation evaluation

1

2

3 4 5 6

7 8 9

Requirements The learning aided system should be developed in such a way that it allows interns to acquire a possibly deep insight into the complex and dynamic interactions of a number of parameters measured by instruments. The focus of learning should be on the medical phenomena themselves and not on the problems inherent in measurements. The learning aided system should be structured in such way that it would face with increased complexity. The learning aided system should give a qualitative impression of level of different quantities. an appropriate instruction in the usage of the system should be provided Interns should be motivated to study these systems in more detail, so as to become skilled in analyzing such systems themselves and giving appropriate treatment. The earning aided system should allow for self-assessment of intern learning. The earning aided system should allow for including assignments as well as individual assessments. The earning aided system should be a high degree of interactivity, e.g. if simulations are used, interns should have the possibility to freeze the step or even reverse the step so as to study the causal relation between different phenomena and states of the case under study.

A Medical Aided System for Evaluating Crura Rehabilitation

471

3 Implement of e-Learning Platform for Crura Rehabilitation For saving the cost of crura rehabilitation, the patients should have the chance to choose the rehabilitation place he could afford. Home self-rehabilitation guided by physicians or special room in hospital is the common choice. Thus the learning environment for interns is naturally distributed. 3.1 Architecture of Medical e-Learning System The most important task at beginning of composing, design and development of a good e-learning tool is its philosophy. This covers the appearance of the screens, introducing animation and simulation tools, and finally, presenting a way of explanation and description of phenomena at the screen and other ideas. The architecture of the medical e-learning platform for crura rehabilitation is shown in Figure 2.

Fig. 2. Architecture overview of the medical e-learning platform

At the client site, there are two type of channel of information acquired. Patient with crura disease can do some rehabilitation training at home everyday and transmit the data via sensors, another way he can choose is be outpatient in hospital, the rehabilitation data are collected by particular medical instrument such as Foot Scan. These data are stored in database at hospital, so the interns are available for evaluating the patient’s rehabilitation state conveniently.

472

F. Li et al.

3.2 Crura Modeling In the point of mechanics, human body can be modeled as multi rigid body [12], [13]. In Figure 3, the multi rigid body model of crura is built, the connection between bones is viewed as joints with angle parameters. Then interns can compare the output parameters, such as angle, foot stress distribution to diagnose the changes after rehabilitation training.

Fig. 3. Muti rigid body modeling of crura

With the sEMG method we mentioned above, the whole crura rehabilitation evaluation process is built. The sEMG signal is measured by instrument, the data were processed by the computer model we have built. The history data of gaits of specific patient are stored in database in comparison with the new data to evaluate the effect of crura rehabilitation. 3.3 e-Learning Agents Many current medical e-learning systems are system centric instead of interns centric. That is the system adopts the same medical materials and strategies to different

A Medical Aided System for Evaluating Crura Rehabilitation

473

interns without considering their different profiles. The system is not able to take care of the diversity of patients and interns. We propose the system based on agent technology [14], [15], [16]. The agent is designed in an object oriented manner to provide personalized e-learning service that can effectively match intern’s goals. Learning agents are incorporated into the environments in order to model the adaptive behavior of patients, interns and physicians. They adapt to specific circumstances and events during the system run. Learning agents learn from their own observations and from the experience of other agents with whom they exchange information.

4 Example During the collaboration research with hospital, we developed a computed aided system to help interns diagnosis on crura rehabilitation. A patient was injured at tarsus, in 20 days he use bicycle as rehabilitation tools guided by physicians. The crura information data were collected everyday by sensors (in Figure 4) and sent to the database in hospital. The interns there can evaluate the effect of rehabilitation from the time-history of his foot stress distribution (in Figure5).

Fig. 4. Crura of patient with sensors

In short, we accomplish these goals through combing medical instrument, multi channels biomechanical sensors, internet technology.

474

F. Li et al.

Fig. 5. Crura rehabilitation evaluation interface to interns

5 Future Challenges and Opportunities: e-Learning Based Rehabilitation E-learning has introduced a new access to medical subjects learning. By using interactive animation and simulation it enables to create interactive training environment. However the danger is that instead of deep understanding and medical background, the interns could memorize visualized results or at complicated animations without previous explanation they could feel like sitting in a classroom and passive watching the running animations. The described tools therefore cannot perform to stand alone education and must be part of complete curricula. These new challenges for e-learning in rehabilitation can offer the following solutions: (1) Most of the interactive tools are focused on “what-if” simulations based on increasing or decreasing parameters of the structure. In other words, the patient situation is performed without the use of real values from the instrument’s parameters. The sense for real values is hereby very important. (2) To introduce an active way of e-learning towards patient oriented learning. Instead of using a short problem as a tool to deliver information and knowledge, a larger scope of patient information should be used. It is well suited to the medicine disciplines and the way interns in the study and therefore, it is considered as an attractive way to approach teaching and learning.

A Medical Aided System for Evaluating Crura Rehabilitation

475

(3) Rehabilitation itself deals with practical utilization of the knowledge. Introducing pictures form practical realization and short movies (e.g. animating of the stress distribution of a foot scan result) gives evidence on application of the analyzed phenomena. This is reason why a good e-learning tool has to contain as much as possible evidences on practical realization and application of diagnosis. Some other opportunities and challenges which should be addressed are in: (1) Packaging and integration as well as medical aided learning systems would require new curriculum and e-learning approaches as the technologies mature. (2) Artificial intelligence and further integration of control, monitoring and diagnostics functions to get self-healing structure and systems (intelligent structure) would unfoundedly change the way the curriculum is organized. (3) With the evolution of a consumer-driven health care system, a bundle of services considered necessary achieves particular outcomes most efficiently. We must work now to establish clinical outcome data that will demonstrate educational contribution to efficiently rehabilitation. Medical resources could be leveraged and networks can be built to assist interns and other undergraduates in the years for keeping up with developments. As rehabilitation treatment increasingly moves into nonhospital settings. Besides of outpatient rehabilitation, we have the opportunity to formulate models of therapy service which include services to home-care patients. This may involve changes of rehabilitation physicians, as well as interns. The concept of virtual learning fits very well to the e-learning. They will probably have an important role in the future, because they integrate the technical, financial and human resources by sharing data, information, documents, multimedia means, etc., that is, the knowledge-base. It should be utilized as auxiliary parts of the medical education or, in contrary, they could constitute “virtual physicians”, where interns would be even “virtual”.

6 Conclusions The rapid development of information technologies caused development of internet based e-learning educational tools. A lot of e-learning tools have appeared in last years. Based on their philosophy and purpose of utilization they are suitable for learning, teaching or for the both purposes. This paper gives an overview of the present state of development of e-learning tools in field for crura rehabilitation and tries to show some hints to design a proper tools optimized from view of purpose of its utilization. The overview of available medical resources creates a valuable part of the paper. Based on the overview the authors draw a conclusion of the future development in the filed of e-learning for crura rehabilitation. Acknowledgments. This work has been supported by the National Science Foundation of China (Grant No. 60475025), the key project of National High-Tech. R&D Program for 973 of China (Grant No. 2002CB312106); the Ph.D. Programs Foundation of Ministry of Education of China (Grant No. 20050335096); the zhejiang

476

F. Li et al.

provincial natural science fund program (Grant No. Y104553) and open fund of key research programs of zhejiang higher education (Grant No. AMT200506-007).

References 1. Baldonado, M., Chang, C.-C.K., Gravano, L., Paepcke, A.: The Stanford Digital Library Metadata Architecture. Int. J. Digit. Libr. 1 (1997) 108–121 2. McDowell, L. Assessment and Learning: Some Insights from Current Research. In: J. Daudt and O. Rompelman (eds.): “What have they learned? Assessment of Student Learning”, SEFI Document No.23, SEFI, Brussels, 7-13, 1999 3. Francis Lau, Joanna Bates, A Review of e-Learning Practices for Undergraduate Medical Education, Journal of Medical Systems, 28(1), 71-87, 2004 http://www.ycwb.com/xkb/ 2006-12/ 02/content_1302200.htm 4. Hagglund, K., Frank, R. G., Rehabilitation psychology practice, ethics and a changing health care environment. Rehabilitation Psychology, 41(1), 19-32, 1996 5. Melvin, J., & Zollar, C. (Eds.) Access Rehabilitation: A Focus for the Health Care Reform Debate. Washington, DC: National Association of Rehabilitation Facilities, 1993 6. Gray, D. B., Disability and Rehabilitation Research: From Policy to Program, a personal perspective. American Psychologist, 45(6), 751-756, 1990 7. Jeffrey E. Bolek, Use of Multiple-Site Performance-Contingent SEMG Reward Programming in Pediatric Rehabilitation: A Retrospective Review, Applied Psychophysiology and Biofeedback, 31(3), 263-271, 2006 8. Armagan, O., Tascioglu, F., & Oner, C., Electromyographic biofeedback in the treatment of the hemiplegic hand: A placebo controlled study. American Journal of Physical Medicine & Rehabilitation, 82, 856–861. 2003 9. Basmajian, J. V., Kukulka, C. G., Narayan, M. G., & Takebe, K., Biofeedback Treatment of Foot-drop after Stroke Compared with Standard Rehabilitation Technique: Effects on voluntary control and strength. Archives of Physical Medicine and Rehabilitation, 56, 231–236, 1975 10. Bolek, J. E., Surface EMG Applications in a Pediatric Rehabilitation Setting. Biofeedback, 26, 21–23, 1998 11. Jin Dewen,Terrain Identification for Prosthetic Knees Based on Electromyographic Signal Features. Tsinghua Science and Technology,11(1),76-81,2006. 12. J.M. Buades, F.J. Perales, M. Gonzalez1, A. Aguiló, P. Martinez, Human Body Analysis with Biomechanics Criteria, AMDO2004, LNCS 3179, 239-248, 2004. 13. P. Cerveri, M. Rabuffetti, A. Pedotti, G. Ferrigno, Real-time Human Motion Estimation Using Biomechanical Models and Non-linear State-space Filters, Medical & Biological Engineering & Computing 41, 109-123, 2003 14. Wilson L.F., Burroughs D.J., Kumar A., Sucharitaves J., A framework for linking distributed simulations using software agents, Proceedings of the IEEE, 89(2), 186–200, 2001 15. Simon, R.; Woan Sun Chang; Pullen, J.M., An agent architecture for network support of distributed simulation, Distributed Simulation and Real-Time Applications, 2003. Proceedings. Seventh IEEE International Symposium on, 68-75, 2003 16. Gelenbe, E.; Seref, E.; Xu, Z.;, Simulation with Learning Agents, Proceedings of the IEEE, 89(2), 148-157, 2001

Using Metaphors in Game-Based Education Pedro P. G´ omez-Mart´ın, Marco A. G´ omez-Mart´ın, Pablo Palmier Campos, and Pedro A. Gonz´ alez-Calero Dep. Ingenier´ıa del Software e Inteligencia Artificial Universidad Complutense de Madrid, Spain {pedrop,marcoa}@fdi.ucm.es, [email protected], [email protected]

Abstract. In spite of its growing popularity, due to a huge technical evolution in the last years and to the fact that new generations are more literate in games than in books, game-based teaching has been very restrictive in its application to adult education. Most applications are merely simulations built with game technology. In order to apply game-based education in domains that can not readily be turned into simulations, new ways of game design need to be explored. In this paper we describe a new approach using metaphorical worlds, and exemplify its application to teach Computer Science concepts, proposing two possible game designs for a game-based educational system devoted to teach the workings of the Java Virtual Machine.

1

Introduction and Motivation

The use of games to teach is as old as teaching. Game in animals is connected to their intellectual capacity and is used as a mechanism for training and safe learning, and, according to Dutch historian, cultural theorist and professor Johan Huizinga, the main cultural achievements are nourished by the instinct of play of the homo ludens [5]. The use of computers games to teach can also be traced back to the first applications of computers in education [1], and a number of systems have been developed under the generic label of edutainment applications, trying to entertain and educate at the same time. Nevertheless, this long story does not count with so many successful examples, mainly simple pedagogical systems intended for kids to learn the alfabet, the colors, or simple arithmetic. The amazing growth of the videogame industry in the last few years has opened new opportunities for the use of games in education. First, the technology has evolved to a point where almost photorealistic 3D virtual worlds can be seen at home with retail equipment This way, the potential of simulations that not so many years ago was reserved to huge budget military or industry projects can now be enjoyed in the living room. And, second, what, in our opinion is more determinant, the language of the videogames has evolved to become sophisticated 

Supported by the Spanish Ministry of Science and Education (TIN2006-15202-C0303), and Direcci´ on General de Universidades e Investigaci´ on de la Consejer´ıa de Educaci´ on de la Comunidad de Madrid and Universidad Complutense de Madrid (Grupo de investigaci´ on consolidado 910494).

K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 477–488, 2007. c Springer-Verlag Berlin Heidelberg 2007 

478

P.P. G´ omez-Mart´ın et al.

enough to express mental processes, social relations and to represent rich and diverse ways of interacting with the elements of the virtual worlds, while growing the number of literate people on this new language (i.e., gamers). In spite of the growing complexity of the videogame language, most of its applications to education are merely simulations built with game technology. This approach limits the application of game-based learning to those domains where real world activities can be easily simulated. For example, to teach how to build bridges one can image a game where bridges are built1 , to teach how to drive a train, trains are driven2 . The motivation for the work presented in this paper is to pursue more creative uses of videogames in order to teach abstract domains, such as Computer Science, without imposing the constrain that a real world counterpart must exist for every game mechanic and trying to make the game experience more central to the learning process. The solution that we propose is the use of metaphors where, given the domain to be taught, a virtual world is design with elements, actions and processes that correspond, as much as possible, on a one to one basis to elements, actions and processes in the problem domain. Linguist George Lakoff and philosopher Mark Johnson [6] provide convincing evidence that metaphors may actually be people’s primary mode of mental operation. They argue that because the mind is “embodied” –that is, it experiences the world through the body in which it resides–people can’t help but conceptualize the world in terms of bodily perceptions. Our concepts of up-down, in-out, front-back, light-dark, and warm-cold are all related to orientations and perceptions acquired through our bodily senses. A second source of inspiration for the metaphorical approach to game-based education and also related to the embodiment of abstract ideas is the method of loci. A technique for remembering that was invented by Greek orators in Classical times [9], as Cicero (106 BC-43 BC) describes in his De oratore: He inferred that persons desiring to train this faculty (of memory) must select places and form mental images of the things they wish to remember and store those images in the places, so that the order of the places will preserve the order of the things, and the images of the things will denote the things themselves, and we shall employ the places and images respectively as a wax writing-tablet and the letters written on it. [Quoted in [9], p. 2]

The use of the method of loci can be traced through the Middle Ages to the present times when Philip Rosedale, Linden Lab CEO, in his keynote for the Serious Games Summit within the 2006 Game Developers Conference proposed the use of Second Life as a memory tool where people can store information in a virtual 3D environment as a Memory Palace [8]. As a proof of concept, we are working in a system to teach the inner workings of the Java Virtual Machine. The user is given a piece of Java code which has to be executed by interacting with a metaphoric representation of the Java Virtual 1 2

http://www.bridgebuilder-game.com/ http://www.microsoft.com/games/trainsimulator/

Using Metaphors in Game-Based Education

479

Machine (JVM). In this paper we present two possible metaphors, one is based on adventure-based games and the other on multiplayer first person shooters. After presenting the basic ideas of the JVM in the next section, two more sections are dedicated to describe each metaphor, and the last one to conclude the paper.

2

The Java Virtual Machine

The Java Virtual Machine (JVM) is an abstract computing machine, with an instruction set that manipulates distinct memory areas at runtime. The specification of its internals [7] force their implementors to guarantee that their implementation correctly perform the operations specified in the class files. In order to define the behavior of every JVM instruction, the specification of this abstract machine defines a set of runtime data areas. – Heap. It stores the class and array instances created by all the Java trheads. Implementors may choose at their discretion the type of storage management, but it is mandatory to have a garbage collector. – Method area. This area stores the code for methods and constructors of every class loaded in the JVM. – JVM stacks. The abstract machine has a stack per thread, that stores activation records and other status information of the thread. In JVM terminology, these records are called frames, which we will describe shortly. – Program counter. Each JVM thread has its own program counter (pc) register, that contains the address of the instruction being executed within the current method for a given activation record. – Runtime constant pool. Class files have a constant pool with names and values that are referenced from the instructions of their methods. The runtime constant pool represents this information in the running JVM. Frames are the most important elements in the method execution. A method invocation is followed by the creation of a frame in the private stack of the thread, which becomes the active or current frame of that thread. The invoked method is known as the current method while the current class refers to the class where that method is defined. Each frame has an area to store the local variables of the current method, an operand stack to store partial results of the instructions or expressions being evaluated, and a reference to the runtime constant pool of the current class. They are also a central structure to dispatch exceptions. Most of the JVM instructions use one or more internal frame structures as source or target of the data they use. They can be grouped in different sets according to their functionality: – Load and store instructions. Transfer values between the local variables and the operand stack of the current frame. They usually need information about both, source and destination, that is given by the instruction arguments or taken from the operand stack itself. – Operand stack manipulation. They perform general operations over the operand stack, as pop a value, duplicate or swap the two values on the top.

480

1

3

P.P. G´ omez-Mart´ın et al. public s t a t i c void main ( S t r i n g [ ] a r g s ) { Vector2D v1 = new Vector2D ( 0 , 0 ) ; Vector2D v2 = new Vector2D ( 3 , 5 ) ; f l o a t dotProduct = v1 . dot ( v2 ) ;

5

}

2

4

6

8

public s t a t i c void Main . main ( j a v a . l a n g . S t r i n g [ ] ) new " Vector2D " dup fconst 0 fconst 0 i n v o k e s p e c i a l " Vector2D .< init >" astore 1 // More t o f o l l o w . . . Fig. 1. Source and object code example

– Arithmetic instructions. This instructions perform arithmetic operations (such as addition, multiplication, negation, comparison, shift and bitwise operations). They pop the arguments (one or two) from the top of operand stack, and push the result back on it. – Type conversion instructions. They pop the value from the operand stack and push the equivalent value in the target type. It is worth to mention the absence of an instruction to cast a reference between different classes, because the actual value of the reference is not affected by its data type. – Object creation and manipulation. This group contains the instructions for the creation of a new class instance or array, checking the runtime type of a class instance, and operations for accessing to array components and reading or writing fields of both classes (static) and objects. – Control transfer instructions. They are conditional and unconditional branches plus two exotic instructions created for the switch construction. – Method invocation. These are also quite high-level instructions, and they reflects that the JVM was designed to execute object-oriented languages. They are responsible for searching the actual method implementation in the class hierarchy of the invoked object, in other words for the dynamic binding. There are also instructions related to exception handling and dispatching, and synchronization, but, for the sake of simplicity, they have not been included in our metaphors. Native methods are also outside of the scope of the model, so the native method stacks have not been described. Figure 1 shows a small piece of Java code and its compiled counterpart. The example assumes the existence of a classic Vector2D with the well-known dot product as a method. The main method in the Main class creates and initialises two vectors and calculates this product. Lines 2-7 create the first Vector2D by first allocating a new object in the JVM Heap (new instruction) and then

Using Metaphors in Game-Based Education

481

invoking the constructor method with the corresponding parameters. The rest of the compiled code is not shown due to space constraints.

3

Adventure Game Metaphor

As said before, every level of the game requires the student to execute a piece of Java code by interacting with a metaphoric representation of the Java Virtual Machine. In this case, it is inspired on adventure games, such as the Monkey Island saga or Grim Fandango from Lucas Arts. The student can manipulate objects and interact with characters through four basic actions: “look at”, “take”, “use” and “use with”. He also owns an inventory where collected objects are stored in order to be used later. The student receives Java source code, and she must execute it manipulating metaphorical JVM structures. So, she will be forced to compile the source code on the fly, and to understand the internals of the real JVM. Besides the inclusion of a well-known game-style interaction using the previously described actions, we decorate the gameplay with a background story. It is shown at the game start, and consists on the user avatar, who used to live in the real world, waking up with a terrible hangover in a mysterious cartoon straw rick. She discovers she has been rammed into a virtual world, known as Yogyakarta, where he wants immediately run away from. But she will be only able to leave when she understands the rules which govern that strange city. Yogyakarta turns out to be a metaphorical Java Virtual Machine, so understanding it consists on executing Java programs on top of it. The environment is populated by other characters who are in similar horrible situation. The “auto impose” role of our pupil will be to study how that world works, in order to execute more and more difficult exercises to liberate more and more countrymen of their oppressors and, eventually, to open the hidden communicating door with the free world. The user has also available the invaluable assistance of a very particular inhabitant known as Javy. He is the only one who knows the secrets of the functionality of the micro-world. Using his abilities, Javy tried to escape from Yogyakarta months ago, but something was wrong and he was discovered and sent to prison without receiving a fair trial at all. He is still in prison, and he is resolute in helping anybody who had the intention of trying to escape, giving her tips and explaining uncomprehending widgets and concepts to her. We use the resource of flashback, showing how Javy solved the problem when he was trying to get away on its own. The first user’s mission is, obviously, to look for Javy. In the remaining levels, a countryman asks the user to free him. The student will have to execute a specific program (or exercise) to do it, and the slave will, eventually, join to the revolution. The exercise is selected by a pedagogical module according to the estimated user knowledge. In case the user overcomes the level, the oppressed countryman is released, and he becomes a loyal supporter. The game (and the learning process) ends

482

P.P. G´ omez-Mart´ın et al.

when the pedagogical module decides to face the user with the last level that joins every concept the user is supposed to have learnt. Instead of releasing a new countryman, this last exercise opens the secret door that let everybody leave Yogyakarta and, finally, come back to the real world. The elements of the JVM described in Section 2 are representend in the virtual world through this metaphor: – Heap. Our city has a neighborhood where there is a building per created object. The player may enter on them to get the value of their fields. – Method area. Similar to the objects neighborhood there is a classes neighborhood where all the buildings representing loaded classes reside. When the user enters into the building of a class, he can find its method implementation. – JVM stacks. The current frame of the thread (top frame stack) is represented by other building on the city. When the user enters, he can see a metaphorical representation of the operand stack and local variables. – Program counter. The player is the one that knows which instruction is being executed. Additionally, the user may display the Java code she is executing, where the last executed instruction appears highlighted. – Runtime constant pool. The JVM uses this area mainly to avoid duplication of symbols that otherwise would exist in the instruction operands. Most of those symbols appear in the player’s inventory when they are needed to execute the next instruction. Next, the JVM instruction execution using this metaphor is sketch. – Load, store and operand stack manipulation instruactions. The student must manipulate the operand stack and local variables representation in the frame building to mimic the data movements done by a real JVM. If the instruction has operands, they are automatically provided by the system and included in the user inventory. – Arithmetic and type conversion instructions. Players have to interact with the operand stack allocated in the frame building. They have to pop the values from the stack and give them to a character that plays the role of an arithmetic logic unit (ALU), that gives the result back to the user. – Object creation and manipulation. An avatar (called Objy) in the object neigborhood is responsible to the object creation. This character is able to construct a new building representing the object, using the classname that the player must have in the inventory. Object manipulation is performed within the object building. – Control transfer instructions. As the student is, in fact, executing the Java object code, he is already playing the role of program counter. For that reason, jump instructions are not explicitly executed, but the affected code portion is repeated or avoided. – Method invocation. Once the player has found out the runtime class of the invokee object, she goes to the classes neighborhood in order to get the method reference needed to execute the code. The user has in her inventory

Using Metaphors in Game-Based Education

483

tokens representing the class and methods names. When she uses them with an existing device, it immediately returns a new token representing the code. That token must be given to the avatar (Framauro) that may create a building representing the new frame. 3.1

Game Session Excerpt

Before continue with the second metaphor, we will write a hypothetical game session, a technique that has proved very useful to analyse the consistence of game mechanis. Such texts represent the commitment of designers to a well-defined game experience, the one to engage players in. In addition to this, descriptions of game sessions can serve as references during project development and may help to validate the system. In our case, two goals are considered: to present game mechanics in terms of real actions taken by players, and to relate to them the JVM concepts previously discussed. Due to limitations on space, we will cover the execution of a small part of the Java code shown figure 1. Specifically, we will restrict ourselves with a few lines, merely the call to the first constructor. We hope this would be enough to capture the essence of the game. At this point of the game, the user has overcomed some exercises. The first one will have finished with Javy freed. The remaining exercises have let the user liberate some countrymen, who will help our pupil to make repetitive tasks that have been learnt previously and finally become boring. A new unhappy contryman ask for help to the student providing an incomprehensive piece of paper that turns out to be the exercise code. In that point, the system automatically loads all the exercise classes, that will appear in the classes neighborhood. The first instruction is also automatically decoded, so the Vector2D operand is available in the user inventory. The student confronts for the first time with the new instructions, and roams trying different things. For example, she talk to Objy, the object neighborhood guardian, but he ignores and laughs at her. The student runs for help and asks Javy, who suggest she give a class name to Objy. Then the user imitates this action, using the Vector2D element in her inventory. Fortunately, Objy is much more kind now, once he has been paid, and builds a new edifice in the neighborhood (heap). He also gives the “key” (the object reference) to the student, who goes to the frame building and push it on the operand stack. The next instruction, dup, follows. It is also a new challenge for the user, who has no idea about its purpose. Once again, she browses all around the virtual world: now the situation is even worst because she has nothing in her inventory (the dup instruction has no operands) so she cannot pay anyone to work for her. When she is asked, Javy explains the dup utility, and this help guides the user to pop the top value from the operand stack in the frame building and to negociate with Alu asking for a clone of the object. After a tense conversation, he accedes and our hero push both copies again in the operand stack. The next two instructions are fload 0, that were learnt in a previous exercise. Instead of manually execute them, the student looks for the countryman that

484

P.P. G´ omez-Mart´ın et al.

was freed when the instruction was first needed. He is delighted about being useful and immediately makes the job. The last instruction that we will described here is invokespecial. The automatic decoding insert two objects in the inventory: the class name (Vector2D) and the method name () that represents the class contructor. Once again, the student confronts to a new instruction. In this case, the meaning is clear: a new method must be invoked. But the steps are obscure, so the student move around looking for responses. Framauro informs her that he would create a new frame for a method if its code is provided. Using this clue, the student goes to the Vector2D building in the class neighborhood where, using the objects in the inventory, gets the required code. The invokespecial execution ends when the code is given to Framauro who builds a new frame for the constructor.

4

Multiplayer First Person Shooter Metaphor

Multiplayer first-person shooter games, such as Counter-Strike, are team-based, in which players join one of two teams, and combat the opposing team while fulfilling predetermined objectives, such as hostage rescue or escape. Specifically, in this new embodiment of the JVM, the action takes place within a spaceship that two competing teams are trying to destroy, leaving their enemies inside. Each team is given a bomb that, when activated, can destroy the ship. The activation of a bomb usually implies the activation of smaller parts which are also explosives by themselves and can be employed as weapons. The activation of the bombs requires to compete for resources that both teams try to get in different places of the ship. In this metaphor a bomb represents a method to execute, so that only one bomb can be in the process of being activated at a given time, and the player who carries the working bomb is the only one who can access the active execution frame. Every time a method invocation is executed, a new bomb is created that becomes the working one, so that only after completing its activation can the team resume the activation of the invoking bomb. The action is developed in two main scenarios: the code rooms and the cargo bay. The code rooms are closed areas good for ambushing and short distance fight, while the cargo bay is a big open area with irregular piles of boxes of different size that can be used for long distance fight. More in detail, the elements of the JVM are represented through this metaphor: – Heap. The cargo bay in the spaceship. Values are kept within boxes, with a different type of box for every datatype. The players may get a copy of the content of a box and put it in her inventory, or modify the contents of a box using a value from the inventory. – Method area. There is one code room per class in the problem application. Once in the room, a panel appears with a 2D view including the names of the methods that the class implements, and the Java code of every method. The rooms are organized to resemble the class hierarchy, such that the room for a class is accesible from below from its subclasses.

Using Metaphors in Game-Based Education

485

– JVM stacks. A bomb represents a frame associated to a method invocation. Only one bomb per team can be in the process of being activated, and only the soldier holding that bomb can change its state. To manipulate the bomb a 2D view is used where the user may manipulate the operand stack and local memory of the frame. – Program counter. The owner of a bomb may see in her inventory a number to indicate the instruction he is decoding. The bomb also contains a reference to the Java code of the method, and, as with any other pointer, its owner may use that reference to be teletransported to the place it is pointing to, which, in this case, is the code room of the class where that method is implemented. – Runtime constant pool. Symbols such as the names of the classes and methods and the constants that are needed to execute the JVM instructions are represented as physical objets that have to be collected through the spaceship. Values appear in boxes while class and method names are items that have to be taken from non-player characters. Now, we will describe how each set of instructions is executed. – Load and store instructions. The soldier that posses the bomb being activated has to go to the storage area and get or put a value in the right box. Instance methods include a reference to the object on which the method is being executed. That reference serves to teletransport the owner of the bomb to the storage area where the object is stored. – Arithmetic, type conversion, operand stack and control transfer instructions. Are executed by the owner of the active bomb through a 2D interface. – Object creation and manipulation. Objects are created through a 2D interface that allows to build new storage boxes, one for every object field plus one for the reference to the object class room, and place them in the cargo bay. The size of every box is proportional to the number of bytes needed by the corresponding data type and they must be kept adjacent. As the game advance, unreferenced boxes may need to be distroyed in order to get room for the new ones. To get or put a value in the field of an object, one of the members of the team has to go to the storage area and get or put the value from or in the right box. Usually is more efficient that the owner of the active bomb goes to the storage since he can be teletransported there, but other strategies are also possible. – Method invocation. To invoke a method the player needs a bomb, an item representing the name of the method and a token that contains the name of a class when the method is static or a reference to the object in other case. The first bomb is ready to be taken at the beginning of each level, representing the entry point for the application. Whenever a bomb being activated, i.e. a method being executed, includes a method invocation instruction, a new bomb, which becomes the active one, is generated. To make a bomb usable, the team needs to decode it, what means that the JVM instructions corresponding to the compilation of the Java method have to be executed. Those instructions that only change the state of the frame

486

P.P. G´ omez-Mart´ın et al.

are executed by simply typing them in the 2D bomb decoding interface, while those others that interact with the state of the heap or the method area have to be physically executed by interacting with the environment, what makes them appear automatically in the bomb decoding code. Once a team has decoded a method, they get a token that can be used to decode other bombs associated with later invocations of the same method. 4.1

Game Session Excerpt

As was done with the first metaphor, we will now present a brief extract of a possible game session of the multiplayer FPS metaphor with the same set of JVM instructions. Let’s assume that two teams, Gaea and Uranus, both composed of two players, are battling each other in the mentioned spaceship. As soon as the game starts, the system randomly selects one member of each team to be the bomb carrier. Both teams start in physically different locations and are equipped with radars that report near enemies and spaceship dwellers under their coverage. To keep things simple, we will focus on the point of view of team Gaea. Team Gaea gets first to the Main class and takes a look at the steps required to decode the main bomb (Figure 1), which represents the stack frame bound to the invocation of the main method. It takes a few moments to determine the resources they need in the short term, as an invocation of a constructor takes always two steps: first, reserve some allocation space in the JVM (using the instruction new) and second, invoke the code of the constructor (special methods named , belonging to the appropriate class). This means that team Gaea needs the name of the class and a human-readable signature of the method ("Vector2D" and "Vector2D.", respectively) to decode the first sentence of the program. Thus, the first task of the team is to obtain both resources from the NPCs (Non-Playing Characters) that inhabit the spaceship, who are not willing to share their possessions with any of the players. Note that the clash with team Uranus is immediate due to the fact that the source code related to Uranus’ and Gaea’s main bomb is the same (in this sense, each team is said to have its own thread of execution, independent of the others). It is interesting to notice the sense of urgence shared by all players, intentionally provided by the metaphor, since the team that acquires the resource in first place will have some precious advantage over their rivals. After some interaction with Uranus and NPCs, the bomb carrier receives the first resource from her teammate, goes into object creation mode (a 2D interface) to build two Float boxes and one with the reference to the class. During this moments, the bomb carrier is highly vulnerable (she cannot take any other action than manipulating the bomb), so she could use some help from her teammate; however, they agree to start searching the next resource while the bomb carrier goes on with the decoding. After having another quick look at the source code, the bomb carrier executes three more instructions (dup, fconst 0 and fconst 0 again), which push onto the operand stack the appropriate arguments of the

Using Metaphors in Game-Based Education

487

constructor (the this reference, that is, the address of the newborn but still uninitialized object, and the value 0 for each of the two float numbers required in Vector2D(), respectively). Everything is in place to invoke the constructor. As soon as her teammate delivers its signature, the bomb carrier enters again in terminal mode, thinks twice about the proper instruction to use, and resolves to execute the right one: invokespecial. As frames correspond to bombs, the team is then given a new artifact to decode, which becomes the active one and must be decoded before the main one. Therefore, this bomb decoding process directly resembles the way the JVM deals with stack frames. This time, the source code of the invoked method must be found in the Vector2D room, but team Uranus has also reached this point of bytecode execution; there is no doubt that Gaea and Uranus will be confronted again in the near future.

5

Related Work and Conclusions

Regarding the use of video games to teach Computer Science related contents we can find two main approaches: turtle-graphics-in-disguise and every-projectis-a-game. The turtle-graphics-in-disguise approach is mainly aimed at teenagers and let the user to program different types of bots in a given 3D environment (CeeBot, CoLoBoT3 , Alice [2]). There is an API to move the robot and the programmer apprentice uses a programming language to write routines that manipulate the entity in the world, which may also be inhabited by avatars controlled by the users (CoLoBot, M.U.P.P.E.T.S.4 ). Although much more sophisticated, this approach does not differ much from turtle based graphics, and therefore it can only be applied as an introduction to programming. On the other hand, the every-project-is-a-game is becoming more and more popular in Computer Science Majors; professors know that students are more motivated if the learning oriented prototype they are building for a course is a game instead of, for instance, a web system for product recommendation. Therefore, be it a course on Artificial Intelligence, Computer Networks, or Object-Oriented Programing, developing (part of) a game always guarantees student satisfaction. Usually the results are not good games but, nevertheless, this, let’s say, “application sugar” provides a surplus of student motivation really useful on these days. In this paper we have exemplified how a more ambitious approach can be taken to the application of games to teach through embodied metaphors for abstract concepts. For the problem at hand, teaching the inner workings of the JVM, the FPS metaphor presents some advantages over the adventure-based one where, although disguised within the story line, the script is basically the same from one level to another: finding tokens in the inventory, giving the tokens to the avatars that provide data or code, and executing the instructions by exchanging tokens between the stack and the avatar representing the ALU. Although good dialogues 3 4

http://www.ceebot.com http://muppets.rit.edu

488

P.P. G´ omez-Mart´ın et al.

could alleviate the problem of the boring repetition and make the game more fun, the authoring cost would be prohibitive for a educational game project. The gameplay in the FPS metaphor, on the other hand, does not depends so heavily on authoring. The main authoring issue on multiplayer first-person shooters is that of the physical organization of the level: the geometry has to be thought so that fighting in there is fun. Fortunately, this problem is not so hard as the scripting of adventure games, and the user can easily be involved in its creation as has been shown by the active mod communities for the most popular multiplayer FPS. Additionally, this metaphor promotes collaborative learning since every team must coordinate its efforts based on a common understanding of the problem being solved. The obvious pitfall of this approach is the use of a metaphor that do not support the transfer of abilities and knowledge acquired in the game to the activities in the real world. Nevertheless, the point that we try to make in this paper is that any domain can be taught metaphorically, although it may be hard to find the right metaphor. After a few years developing JV2 M5 [3,4], we have a system to teach how Java code is executed in the Java Virtual Machine and to explore posible translations from Java code to JVM bytecodes using gamebased teaching strategies. This environment forces the student to understand the underlying mechanisms of attribute access, object construction, inheritance and dynamic binding. We are now in a position of being able to face users to real exercises and to evaluate the effectiveness of this tool.

References 1. R. R. Burton and J. S. Brown. A tutoring and student modelling paradigm for gaming environments. In Proceedings of the ACM SIGCSE-SIGCUE technical symposium on Computer science and education, pages 236–246. ACM Press, 1976. 2. W. Dann, S. Cooper, and R. Pausch. Learning to Program with Alice. Prentice Hall, 2005. 3. P. P. G´ omez-Mart´ın, M. A. G´ omez-Mart´ın, and P. A. Gonz´ alez-Calero. Learningby-doing through metaphorical simulation. In 9th International Conference, Knowledge-Based Intelligent Information and Engineering Systems, KES 2005, Lecture Notes in Artificial Intelligence, pages 55–64. Springer, 2005. 4. M. A. G´ omez-Mart´ın, P. P. G´ omez-Mart´ın, and P. A. Gonz´ alez-Calero. Gamedriven intelligent tutoring systems. In Entertainment Computing - ICEC 2004, Third International Conference, Lecture Notes in Computer Science, pages 108– 113. Springer, 2004. 5. J. Huizinga. Homo Ludens. Beacon Press, 1971. 6. G. Lakoff and M. Johnson. Metaphors We Live By. University of Chicago Press, 1980. 7. T. Lindholm and F. Yellin. The Java Virtual Machine Specification. Addison-Wesley Professional, 2nd edition, 1999. 8. P. Rosedale. Serious games summit keynote: You can (not) be serious, Last viewed: December 2006. 9. F. A. Yates. Art of Memory. University of Chicago Press, 1974. 5

http://gaia.fdi.ucm.es/projects/javy/

UML in Action: Integrating Formal Methods in Industrial Design Education Jun Hu1 , Philip Ross1 , Loe Feijs1 , and Yuechen Qian2 1

Eindhoven University of Technology, 5600MB Eindhoven, NL 2 Philips Research, 5656AE Eindhoven, NL

Abstract. When designing product behavior, the designer often needs to communicate to experts in computer software and protocols. In present-day software engineering, formal specification methods such as the Universal Modeling Language have been widely accepted. Teaching design students these formal methods is non-trivial because most of design students often have difficulties in programming the behaviors of complex produces and systems. Instead of programming, this paper presents a technique, namely “acting-out”, for design students to master the formal methods. The experience shows that acting-out not only worked out very well as a teaching technique, but also showed the potential for bridging the processes of industrial design and software engineering.

1

Introduction

The industrial design of embedded systems is a contemporary challenge which deserves more attention and better support by methods and tools. For a long time the worlds of embedded systems and industrial design developed separately. The early examples of embedded systems included products such as telephone exchanges, plant control, and military applications. The early examples of industrial design included furniture, radios, cars, and household tools. In other words: the embedded systems where invisible and their users were not given much room for affective or spontaneous behavior. In contrast, the industrial designers had to give high priority to the emotional associations, the styling, the appeal, the usability and even the fun associated with the product. But now most everyday personal objects contain embedded systems and the two worlds are merging. At the methodological level however, much work regarding their integration remains to be done. The present paper is meant as a contribution to this integration. In Fig. 1 a number of methods are arranged from left to right. Formal methods include Z, Object Z, SDL, VDM, CSP, ISpec, Larch, process algebraic methods etc. They are close to mathematics and they enable very rigorous syntactic and semantic analysis. Structured methods are more flexible; they developed in a pragmatic way to answer the needs of software developers. Several useful notations are integrated in the Universal Modeling Language (UML), which emerged as a synthesis and de facto standard out of a rich variety of notations one K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 489–498, 2007. c Springer-Verlag Berlin Heidelberg 2007 

490

J. Hu et al.

or two decades ago (such as Yourdon, SSAD, Objectory, OMT etc.). In the HCI (Human Computer Interaction) and Industrial Design communities, a growing interest for the behavior aspects of products arose, not just the form-giving of screens or the physical forms and materials of products. A new generation of methods is available which we summarize as “engaging methods”. What they have in common is the idea that behaviors are not stand-alone elements that have to be described with great precision. User behaviors arise in social contexts. Moreover, by letting designers engage in the behaviors themselves, their emotional, creative and social skills are activated and thus better integrated into the result as well. We mention for example: – – – –

Personas, Focus groups, Acting out, Interaction relabelling. Computer scientists System developers HCI Researchers Industrial Designers

Formal methods

Structured methods

Engaging methods

Fig. 1. Methods ranging from mathematical and formal to social and emotional

We agree with the argumentation line of Grudin and Pruitt that these methods exploit the power of people [1]. Designers are forced to think about social, political, but also emotional aspects of design that otherwise go unexamined. The challenge to which we present a contribution in this paper is to better integrate the methods of Figure 1 horizontally. One way to do this is designing another method or language, as done in the DUTCH design method [2]. Rather than proposing a new language, here we address the question how to open up the language of the formal and structured methods to industrial designers. One could try to educate them in software engineering and UML in the traditional way; but in view of the amount of other skills that industrial designers are expected to have, this is just another load to their already overloaded education path.

UML in Action: Integrating Formal Methods in ID Education

1.1

491

UML in Industrial Design Teaching

A product with sensors, actuators and network connections can offer an interesting, useful, or playful behavior to its users and to the other products, systems and services to which it is connected. The master students from the department of industrial design (ID), Eindhoven University of Technology (TU/e) take responsibility for the creation of this behavior. In many cases. if the product isn’t stand-alone, neither is the designer. Whenever product behavior is realized through computer software and protocols, the designer often needs to communicate to experts in these matters. In software design, formal methods are often used for this purpose [3,4,5,6]. In present-day software engineering, formal specification methods such as the Universal Modeling Language (UML) [7] have been widely accepted. It contains “activity diagrams”, “use case diagrams”, “class diagrams”, “state charts” and “message sequence charts” that are useful for product designers and software engineers to exchange their ideas and concepts to reach a common understanding. A module called “Formal Software Specification in Action” is now taught at ID, TU/e to help the students understand and use such formal methods for this purpose. The knowledge and skills that students gain by participating in this module helps them to express the structure and behavior of the software components of their design in a way that is understandable to other parties. The students develop an understanding and appreciation of what it means to master complexity through formal methods. The scope is widened from small programs to complex software systems. Although developing and maintaining such systems usually involves computer scientists as well, the ID Master students are expected to be well equipped to use formal methods such as UML and thus specify system structure and desired behavior. UML as a formal specification tool is widely used in an object-oriented design process in software engineering. Such a process can follow a traditional waterfall model going through object-oriented analysis (OOA), object-oriented design (OOD) and object-oriented programming (OOP) [8], testing and delivery. It can also following a spiral process that includes fast iterations, each of which going through analysis, design, implementation and evaluation. The later is more and more used in software industrial, often being referred as rapid prototyping, or in a more recent buzz word, Agile design [9]. Either method has a phase of implementation or iterations of implementations so that the ideas and concepts under design can be visualized and tested. This often requires the designers or the engineers to have enough programming skills to implement the system (waterfall model) or to prototype the behavior (agile design). In a teaching course of such methods, students with no experience of programming would find it difficult to fully understand the entire process. This is exactly the difficulty one may expect in teaching industrial design students a formal specification language such as UML. Most of the students in an industrial design education are not trained extensively in programming tasks, neither do they need to. They are not software engineers, they are designers. But as a designer who is expected to design “intelligent products and services”,

492

J. Hu et al.

understanding of and communicating about the structure and the behavior of a complex system are certainly unavoidable. To deal with this dilemma, instead of pushing extensive programming courses to the design students, we need to find a method that can be better integrated in an industrial design (not software design) process. Hence a technique called “Acting-out” was proposed and tried out in our module for the master students at ID, TU/e. 1.2

Acting-Out as a Design Technique

In design of interactive, networked products and systems the intended interaction experience often emerges as a main design criterion. “Traditional” design techniques, like storyboards, or on screen simulations seemed unfit to deal with the multifaceted and dynamic character of interaction experience. Several design and design research communities have developed approaches to deal with this complexity in different stages of the design process, based on introducing real, bodily involvement in the design process. Different communities gave different names to these techniques, e.g. “Experience Prototyping” at IDEO [10], Informance Design [11], Designing actions before product [12], Choreography of Interaction [13], Gesturing out [14]. In this paper, we refer to this collection of techniques using “acting-out”. These approaches have in common that they allow designers to make aspects of a product or system experiential and vivid by physically acting out (elements of) interaction scenarios. Buchenau et al. [10] describe the following advantages of acting-out techniques: understanding existing interaction experiences and contexts, evaluating design ideas, exploring new design ideas, and communicating designs to an audience. These last three advantages, i.e., evaluation, exploration and communication in the design process, make acting-out especially valuable in the Industial Design UML module. UML is a highly abstract language, but it is a tool that, in Industrial Design practice, referes to real products and systems that make real life experiences happen. Our assumption in this module is that through acting-out a concept UML diagram, its structures, objects, properties, connections and restraints gain an experiential dimension in an early stage. This experiential dimension could help identify possible inconsistencies and flaws in the UML diagrams and help suggest improvements. Furthermore, it may help communication to other designers or software engineers what the problems or ideas for improvement are. In the next section, the module is presented, followed by the feedback from the students.

2

Zoo of Tamagotchi Animals

In this section we describe the trial based on the principles of Section 1.1. The trial was done as one “education module”. 2.1

Participants

The module was a one-week course for 32 master students. Students were divided into five teams (about six students each). Each team was given the same task.

UML in Action: Integrating Formal Methods in ID Education

2.2

493

Procedure

The module was a one-week (5 full days) course, including morning sessions for lectures and afternoon sessions for a project. The lectures gave introductory information about software engineering and formal methods in general, objectoriented design process, selected UML diagrams for the project and acting-out as a design approach. The students were expected to use what they learnt from the lectures in their projects, going through a single process of analysis, design and acting-out. Each team was given the same task: Designing a system called Zoo of Tamagochi Animals. The module was designed such that the students would not only learn and experience writing specifications, but also to learn and experience reading the specifications made by others. In this way both direct results (the specifications) and indirect feedback results (how the other participants interpret the direct results) could be obtained. This was done as follows: – The first half of the project was for analyzing the requirements, making and specifying the object-oriented design of the system; – In the middle of the project, the teams swapped their specifications for acting-out each other’s specifications; – The second half of the project was for acting-out the specifications received. The students were asked to “implement” the system by acting the specification out to show how the system should work according to the specification. Students could play the objects, showing their behaviors and the communication in between. Stage props could also be used to represent objects, interfaces, and events, etc. Students were asked to use imagination and creativity in acting-out, since we did not have experience in acting-out UML in a design process. Finally the students were asked to reflect on the process itself, see Section 2.6. 2.3

Materials

The students were allowed to use all the materials that are generally provided to the Master’s program, which includes one laptop per student, which includes a lap-top, flip-over’s, whiteboards, etc. and several large office-spaces and class rooms. A wide variety of materials, including modelling foam, chairs, pieces of wood were readily available as well and could be used as stage props when desired. A preliminary description of the Tamagochi Zoo requirements was handed out at the beginning of the module. Students ware asked to design the objectoriented structure of the system - Zoo of Tamagochi Animals. Each team received the same task description, entitled ”Designing a Zoo of Tamagochi Animals”, described by the following text: – All animals live in a zoo. – An animal has got a life after it is born. While the life goes on, the animal moves and sleeps (if it is still alive), eats (if it moves and finds food), grows (if it eats) until one day, it dies because of hunger, illness or age.

494

J. Hu et al.

– Every animal has got a body. Different animals look different because of their different bodies. Every animal has got two eyes and one mouth on its body. When eyes are pinched, animals scream and can be hurt. – There are male and female animals. When grownup males and females meet, they may fall into love and the love may result in baby animals. – Some animals are pets. Pets have names and they wear their name plates on their bodies. People (the users) take care of their pets and feed them with food. – People may play with their pets to keep their pets fit. – Some baby animals will be selected by people and they become pets. The rest are left free in the zoo and they have to strive for food and try to survive by themselves. – When a zoo is created, it is empty, until people get some animals from somewhere else (from shops, for example). – People may exchange their pet animals. – Dogs and cats will be our favorite animals for the time being. Dogs woof and cats meow. Cats are scared of dogs and can be bitten by dogs. When big dogs bark, cats scream and start running away. When big dogs fall into sleep, cats start getting together and partying. – The zoo is open for other animals, including unknown ones. 2.4

Direct Results

An example of the acting-out itself is shown in Fig. 2. An example of a UML state diagram is given in Fig. 3. 2.5

Feedback Results

As mentioned before, during the second half of the project the teams swapped their specifications for “acting-out”. They were not allowed to consult the teams that made the specification about the design. The teams were only allowed to read the specifications in order to understand the design. At the end of the project, each team presented (acted out) the specification together as a theater play (Fig. 2). After that students were asked to give feedback on the specifications they got from other teams, their own specifications, and their implementations (acting-out) for other teams. Here an example is included as follows: Feedback of Team 4 on Team 2’s specification ··· The state diagram (Fig. 3) has an excellent division between sub- and superstates. It can be mentioned, however, that some things could be improved. For instance, you have to be able to return to the previous state. In this diagram, you could not leave the shop without selecting a pet. Exchanging pets was not very clear too. There should have been more conditions in this diagram. We did make a misinterpretation of having a shop out of the zoo, while it was specified as being

UML in Action: Integrating Formal Methods in ID Education

(a) Implementing the Pet interface on a Dog

495

(b) Feeding a Croc only if it is a Pet

Fig. 2. Acting-out

in the zoo. As these formal methods are not yet very natural to us, it is apparently hard not to make intuitive decisions, but stick to what is there in the diagram. ··· Team 2’s reaction on Team 4’s feedback and acting-out ··· In general, the implementation as acted out was helpful, even refreshing, in showing us how our system would work. The comment was that a male as well as a female animal can give birth. This was not our intention, but we agree it was not specified clearly. We should have made an activity diagram involving two animals who are about to mate, containing the condition which animal is female. This animal would then have the method giveBirth() and the attribute isPregnant. Because a male and female animal have different behaviors, we should have also specified this in the class diagram, by introducing the abstract classes Male and Female. Also, apparently the condition for the Dance() method (an animal starts dancing if his stomach is full) was not clear to the implementation team. ··· 2.6

Reflection Results

Students evaluated the module and indicated what they felt were the advantages and disadvantages of using acting-out in a design process that incorporates structured or formal methods such as UML. We summarize a few points: Acting-out as an evaluation technique. Two teams stressed the helpfulness of acting-out in finding bugs in their system design.: ’...designers can better imagine the working[s] of the system ’being’ it.’ Acting-out as a pleasurable learning technique. Two teams remarked that this module was a pleasurable learning experience: ’And, of course, it is nice to do and greatly spices up a design process’, on of the teams writes.

Selection mode

Fig. 3. State diagram of the Zoo, designed by Team 4

Dump Pet, Pet becomes ‘normal’ animal

Play with Pet Pet gets happier

Feed Pet Pet gets happier

Pet selected

Exchange Pet; own Pet given away, new Pet received

[if own pet in sight] select Pet

Leave Zoo

Keep observing

[if animal in sight] select animal

Animal selected

Observe mode

Go shopping for a new pet

Observe the Zoo

No Selection

Give animal a name; make it a Pet

Go select a Pet in the Zoo

[if answer from Pet] Pet comes

Name selected, call Pet

[if no answer from Pet] call again

Browse list of Pets

Active mode

Leave active mode

The Zoo

Go to Zoo

Keep browsing the shop

Select an animal

Shop mode

496 J. Hu et al.

UML in Action: Integrating Formal Methods in ID Education

497

Acting-out as a exploration technique. Three of four teams remarked that acting-out also has potential as a generative technique, when it would be applied earlier in the process. Acting-out as a communication technique. One team remarked that acting-out was primarily a communication technique for them: ’We discovered most flaws by reasoning, but acting-out was a good way to communicate flaws in a diagram to the audience.’

3

Discussion and Conclusion

Using acting-out as a prototyping or verification technique, students learned to understand and apply object-oriented design principles and the formal software specification methods, up to a level sufficient for basic communication with experts, in just three days. Without involving the students in heavy programming activities, this module gives more time and space for the students to concentrate on the essential issues and disciplines. Students found learning and applying formal specification methods in this module a pleasurable learning experience. We think this pleasurability is an important aspect of our module, since it may enhance the learning experience overall. Students stressed different advantages of using acting-out in a design process incorporating a formal specification method. The evaluation and communication aspect was experienced by the students and they were optimistic about a possible use of acting-out in the exploration phase. We can not say at this point whether acting-out could actually be beneficial as a generative tool in the context of formal methods in industrial design, but it seems worthwhile to explore this aspect of acting-out. Furthermore, we are looking for methods to integrate industrial design processes with software design processes in designing “intelligent products and services”, and we speculate the acting-out design approach may provide a good bridge that helps make the transition from a general concept to an engineering level smoother. However, establishing an acting-out based method for this bridging purpose, requires more research and experiences in intelligent product and systems design practice.

References 1. Grudin, J., Pruitt, J.: Personas, participatory design and product development: An infrastructure for engagement. In: Proceedings PDC. (2002) 144–161 2. van Welie, M., van der Veer, G.: Structured methods and creativity - a happy dutch marriage. In: Co-Designing 200, Coventry, England (2000) 11–13 3. Hu, J., Feijs, L.: IPML: Structuring distributed multimedia presentations in ambient intelligent environments. International Journal of Cognitive Informatics and Natural Intelligence, Special Issue on Ambient Intelligence and Art (to appear) (2007)

498

J. Hu et al.

4. Hu, J.: Design of a Distributed Architecture for Enriching Media Experience in Home Theaters. Technische Universiteit Eindhoven (2006) ISBN:90-386-2678-8. 5. Feijs, L., Hu, J.: Component-wise mapping of media-needs to a distributed presentation environment. In: The 28th Annual International Computer Software and Applications Conference (COMPSAC 2004), Hong Kong, China, IEEE Computer Society (2004) 250–257 ISBN:0-7695-2209-2, DOI:10.1109/CMPSAC.2004.1342840. 6. Feijs, L.M.G., Qian, Y.: Component algebra. Science of Computer Programming 42(2–3) (2002) 173–228 7. Booch, G., Rumbaugh, J., Jacobson, I.: Unified Modeling Language for Object-Oriented Development (Version 0.9a Addendum). RATIONAL Software Corporation (1996) 8. Taylor, D.: Object-Oriented Technology: A Manager’s Guide. Addison Wesley (1990) 9. Martin, R.C.: Agile Software Development: Principles, Patterns, and Practices. Prentice Hall (2002) 10. Buchenau, M., Fulton Suri, J.: Experience prototyping. In: Designing interactive systems: processes, practices, methods, and techniques., New York, ACM Press (2000) 424–433 11. Burns, C., Dishman, E., W., V., Lassiter, B.: Actors, hairdos & videotapeinformance design. In: CHI, New York, ACM Press (1994) 119–120 12. Buur, J., Vedel Jensen, M., Djajadiningrat, T.: Hands-only scenarios and video action walls: novel methods for tangible user interaction design. In: DIS, New York, ACM Press (2004) 185–192 13. Klooster, S., Overbeeke, C.: Designing products as an integral part of choreography of interaction : The product’s form as an integral part of movement. In: the 1st European workshop on Design and Semantics of Form and Movement, Newcastle, UK (2005) 23–55 14. Ross, P., Keyson, D.V.: The case of sculpting atmospheres: towards design principles for expressive tangible interaction in control of ambient systems. Personal and Ubiquitous Computing 11(2) (2007) 69–79

An Agent-Based Adaptable and Configurable Tutoring Module for Intelligent Virtual Environments for Training Leticia S´ anchez and Ricardo Imbert Computer Science School, Universidad Polit´ecnica de Madrid, Campus de Montegancedo, s/n 28660 Boadilla del Monte, Madrid, Spain [email protected], [email protected]

Abstract. It is no accident that VR has become one of the most employed technologies in the training and education area. It offers many interesting features that both traditional human-based learning and classical computer-based instruction are unable to provide. However, the spectacular nature of VR has, in many occasions, eclipsed the pedagogical importance of the so called Intelligent Virtual Environments for Training, producing impressive graphical systems with very low effective tutoring capabilities. We bet on improving the quality of this kind of systems by providing them with an Intelligent Tutoring System adapted to the particularities of VR. And since developing one of these systems from scratch is a hard, complex task, we propose in this paper an adaptable and configurable —agent-based— Tutoring Module, easy to be reused in different nature and purpose applications.

1

Using Virtual Reality for Training

Virtual reality (VR) or virtual environment (VE) technology is most employed in the training and education field. In addition to the potential advantages of traditional computer assisted intelligent instruction (personalization of the contents, adaptation of the tutoring strategy to the student needs. . . ), VR offers many other interesting features. On one hand, VR represents virtual scenarios, in which trainees can gain experience and familiarization with the real working environment without assuming any risk. On the other hand, VR allows students to practise training skills, reproducing processes, activating simulations, manipulating virtual objects with behaviors similar to those expected for them in real life, and receiving instant feedback on the consequences of actions. Not only the trainee, but also the tutor may have a virtual representation (virtual mannequin or avatar) in the 3D world, which can perform demonstrations about how to carry out specific tasks. In addition, VR technology potentially increases the student’s presence, even when using non immersive interface devices. However, it is important to keep on mind that VR is just a support to education and training. Many approaches pay all their attention to the technological K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 499–510, 2007. c Springer-Verlag Berlin Heidelberg 2007 

500

L. S´ anchez and R. Imbert

aspect, forgetting the pedagogical one. The result is graphical impressive systems, with advanced interactive interfaces, but with no effective tutoring capability. We bet, as alternative, on the approach of reinforcing efficient tutoring structures, such as intelligent tutoring systems (ITSs), with VE, what has been called intelligent virtual environments for training (IVETs). This kind of systems, which adapt the classical ITSs structure to the particularities of the VEs, tends to uncouple the different knowledge to be managed during the tutoring process, depending on their nature (about the student, about the subject to be taught, about the environment. . . ). The bulk of these systems, regarding the tutoring competencies, is the Tutoring Module. Still, the pedagogical capability of current IVETs is very reduced, mostly because their tutoring modules are quite limited, both in their structure and in their content. Along this paper, we will describe the efforts devoted to model an easily adaptable and configurable efficient tutoring module as part of the MAEVIF training platform [1], and how it has been tested in a pilot system to evaluate its appropriateness.

2

Previous Work

There have been some previous interesting instructional systems based on VR/ VE technology that have opted for ITSs to guide the specific tutoring process. In some cases, IVETs have been used with pure educational purposes. Some systems as, for instance, Herman the Bug [2] or Cosmo [3] make use of a virtual tutor with a physical representation in the 3D environment. The student refers to this tutor looking for help or assistance. These systems have been used to test some instructional approaches, such as the constructivism, together with the behaviors believability of the virtual tutor and the student’s avatar. One of the main contributions of these systems has been their studies about how the persona effect (the presence of a lifelike character in an interactive learning environment can have a strong positive effect on students perception of their learning experience) is favored by the using of personified virtual tutor [4]. On the other hand, IVETs have also been used in training, both of abilities and processes. For example, Steve [5], Jacob [6] and MRE [7] —just to mention some relevant applications—, present some interesting features such as just providing help to the student, the usage of secondary characters as a support of the training process, or, in some cases, a virtual tutor with learning capabilities. In general, most of the systems referred above pay a great attention to the realism of the representation and believability of the virtual characters behavior, including the virtual tutor, as the basis that will allow the student to recognize in the future the real environment simulated. However, the tutoring infrastructure leaves very few chance to be adapted to different contexts or, what is worse, to the specific necessities of the student. The tutoring limitations of these systems are specially remarkable dealing with group training: they have not yet proved its effectiveness in work group

An Agent-Based Adaptable and Configurable Tutoring Module

501

training tutoring, there is no possibility to provide more than one student with a tutor, and, although their tutoring components are able to identify wrong actions, they have some difficulties to correct them, what makes the learning process difficult whenever the mistakes of a student have some influence in the tasks to be performed by another one. The MAEVIF model [1] offers a very convenient structure to fit in it a powerful adaptable tutoring component, prepared both for individual and group instruction, and not only the usual reactive tutor. Hence, we have substituted its basic tutoring module with the tutoring module presented in this paper. Its characteristics and details will be described in the following sections.

3

The MAEVIF Platform

MAEVIF is a Model for the Application of Intelligent Virtual Environments to Education and Training. It is the result of several experiences integrating virtual environments and intelligent tutors that served to point out the problems that commonly arise in such integrations. MAEVIF defines a generic model for intelligent learning environments, based on the use of virtual worlds. That model has been developed as an open and flexible agent-based software architecture. ITSs use the information originated during the student interaction with the learning application and the tutor to manage their behavior and to adapt it to the user specificities. However, when that interaction takes part inside a VE, the student interaction with the virtual environment is an additional valuable source of knowledge for the tutor. Thus, the classical ITS structure has to be adapted to cope with that information. With that aim, MAEVIF includes a World Module, as it is shown in Fig. 1, which controls the user interaction with the environment, together with the environment dynamics, the virtual objects behaviors and any simulation needed in the VE. The reformulation of the internal structure of ITSs has forced the redistribution of responsibilities among its components. In particular, the tutoring module must manage new kinds of knowledge and has to adapt its behavior depending on new ways of student interaction with the VE. The MAEVIF model has been developed following an agent-based approach, identifying one or more agents for every ITS module (marked with dotted-squares in Fig. 1). From those agents, we will describe those directly related to the Tutoring Agent, agent which will be deeply detailed in the rest of this paper: – Planning Agent: Builds plans for every defined activity, made up of each individual action the students have to execute to reach the activity objectives. – Tutoring Agent: Receives session requests from the students, providing them with a list of the available activities in accordance with their knowledge about the subject that is being taught. It controls the team formation in the collaborative activities, together with the assignation of the role to be carried out by every student. But the main goal of the Tutoring Agent is the student tracking, controlling every one of his actions and comparing them with the ones defined

502

L. S´ anchez and R. Imbert

Fig. 1. Adaptation of the structure of ITSs proposed in MAEVIF to deal with VEs

–

– –

– –

in the activity plan, generated by the Planning Agent. The Tutoring Agent also manages the assistance and helping system, providing the student with information and answering his questions, related both to the scenario and to the process to be followed to end successfully the activity. Path-Planning Agent: Performs a tracking of the student paths along the VE. This information is important to compare the real itinerary followed by the student and the optimum one. Expert Agent: Manages the knowledge the student has to acquire about the subject, the activity and the scenario. World Agent: Maintains and controls geometrical and semantic information about the VE objets and inhabitants (students, auxiliary characters and virtual tutors). It also gathers information about the interactions in the VE and their associated consequences, which will be used to answer some of the student questions. The world structures are represented by an ontology. Communication Agent: Is the communicative interface between the students and the VE. Every student has a Communication Agent associated. Student Agent: Records the actions executed by a specific student, the evaluation of the activities carried out, the places and objects watched, and the paths or itineraries of each movement or transport action. In addition, it provides the Tutoring Agent with information to advise the students if their itinerary do not match with the optimum path calculated by the PathPlanning Agent, or to check if the student is confused or do not know how to finish the activity. There will be as many Student Agents as student connected to the system.

Figure 2 shows the internal dependencies among these agents.

An Agent-Based Adaptable and Configurable Tutoring Module

503

Fig. 2. Dependencies and relationships among the agents identified

Despite of being well defined, the internal content of some of these agents had been poorly developed to the date. This was the case of the Tutoring Agent. In the rest of the paper it will be described how we have enhanced it, providing it with a complete tutoring strategy, designed ad hoc, to define the tutor behavior. It will also guide the student learning process, and will assist him answering his questions and giving him suitable information to face the problems that will arise during the activity execution. The intelligent tutor, according to the chosen tutoring strategy, will also make a personal tracking of every student’s learning process, adapting the strategy to the subject characteristics and the student knowledge acquisition capabilities.

4

An Easily Adaptable and Configurable Tutoring Agent

The proposed Tutoring Agent has been conceived to be generic, independent from the context of the subject to be taught or to the kind of students involved in the learning process. This makes it easy to reuse it in different applications without devoting an excessive effort to its development, and without having to reinvent the wheel every time. This Tutoring Agent is, thus, easily adaptable and configurable from the designer (both pedagogical and technical) point of view. But the Tutoring Agent itself has been provided with the ability to adapt its pedagogical behavior to the student characteristics, and to dynamically configure its learning strategy to the student skills. In the following sections, the specific tutoring structures and teaching behaviors of the Tutoring Agent are described. 4.1

Syllabus Structure

The syllabus of a specific subject is defined as the phases or stages to be passed by the student to reach a certain proficiency in that subject. The student rhythm

504

L. S´ anchez and R. Imbert

of advance through the syllabus will be determined by the specific student knowledge acquisition capabilities. Thus, the student will be able to start the activities of a phase only if he has already reached the learning goals of the previous dependent ones. The syllabus the Tutoring Agent has a very intuitive structure, alternating knowledge acquisition phases and application phases. The syllabus has been designed with a very generic structure, independent of the subject and the scenario of the training course. This has proved to be very suitable to be applied in different contexts, although it has been specially thought for environments in which the students acquire abilities and knowledge by the execution of activities. The syllabus proposed consists on the following elements: Course. A course is the most general learning stage of the syllabus. The knowledge and abilities described to be acquired once it has been overcome are very specific, and the goals to be reached are related to a global objective. Each course is divided into several phases, and it will not be passed until the student has got through all of them. Since the Tutoring Agent here described can be used in different contexts, the courses will have specific, intrinsic characteristics. In addition, students taking part in a course activities may be of different ages and have different intellectual and physical characteristics and abilities. Hence, the tutor behavior should not be the same for all the students and courses. With this aim, the tutoring strategy provides several configurable parameters, although conserving always a common structure. According to the course characteristics, a concrete tutoring strategy will be selected by the human designer for the Tutoring Agent. In the future, this selection will be automatically performed by the agent itself during its interaction with the student. The aim of the courses is to divide the learning process into independent goals. The course characteristics will help to define the tutoring strategy to be followed by the Tutoring Agent. Phase. Phases are course elements that contain the activities to be performed by the student to reach a didactic objective; in other words, a phase represents one ability or piece of knowledge the student has to acquire in the course. Phases are used to group all the course’s activities with common objectives. There have been identified four kind of phases: – General knowledge acquisition phases: Their main didactic objective is to allow the student to learn what every object in the scenario is, what they are useful for and where they are inside the VE. – General knowledge application phases: The student may start one of this phase’s activities once he has achieved all the previous didactic goals associated to it. The aim of these phases is to provide the student with knowledge about the processes to be followed. In addition, the student is indirectly prompted to prove that he still keeps the knowledge acquired in previous phases, by carrying out more intricate activities in which that previously assimilated knowledge has to be combined with new one.

An Agent-Based Adaptable and Configurable Tutoring Module

505

– Advanced knowledge acquisition phase: The student acquires more advanced knowledge and learns to make more important decisions. Thus, the emphasis of the activities performed evaluation is not exactly focused in the order of its execution, but rather in the quality of the process followed for its execution. – Advanced knowledge application phase or general evaluation phase: The student must combine both basic and advanced knowledge, proving his ability to make decisions and to correct his mistaken behaviors. Depending on the nature of the course, there could be one o more phases of each type, considering that after a knowledge acquisition phase there should always be an application one. For instance, if the subject addressed is quite difficult, there could be a general knowledge acquisition phase and its associated application one, followed by two advanced knowledge acquisition phases, with didactic objectives of different difficulty levels (level 1 and level 2), followed by their corresponding application phases. Activity. Course phases consist of basic elements called activities. Activities are used to organize phases’ didactic objectives into smaller specific ones. This allows the tracing of the student knowledge acquisition evolution. Hence, a phase will no be considered overcome until the student has successfully finished all its belonging activities. All the activities have the same attributes, being defined during the design stage by the human tutor/designer. Thus, an activity consists on: the scenario in which the activity takes place; the goal(s) to be reached (a final state); the information to be provided to the student, which will be managed by the Tutoring Agent according to the student’s skills; the initial state of the scenario objects (at the beginning of the activity); the list of didactic objectives, precondition to allow the student to perform the activity; the list of didactic objectives that will be reached once the activity has been performed; the number of students needed to begin the activity, together with its associated role inside the activity; and the time limitations —maximum and average— to carry out the activity. The key point to evaluate the performance of a student during one of these phases is the sequence of activities executed, although the Tutoring Agent allows several alternative ways to achieve a phase goal (whenever it is possible). Action. An action is the list of steps the student has to perform to overcome an activity. Each syllabus activity is defined by one or more goals to be achieved by the student. The way in which the student will reach those goals is specified by a list of actions and the order in which they have to be executed. This list is dynamically generated in execution time by the Planning Agent, on-demand of the Tutoring Agent, and may be remade always that the student has executed a non-expected action, whenever the Tutoring Agent considers it convenient according to its tutoring strategy. Every time the student wants to perform an action, the Tutoring Agent may consider it as a: – right action, if the action matches the planned one;

506

L. S´ anchez and R. Imbert

– wrong action, if the action does not match the planned one, and if it will not be possible to reach the activity goal once the action will be executed. In this case, depending on the tutoring strategy, the Tutoring Agent: will finish the activity, showing the student his fatal error; will allow the student to continue executing actions, so that the student realizes by himself his mistake (a kind of Socratic Method ); or will not allow the student the execution of the wrong action, giving him a new chance to choose a right one. – replanning action, if it does not match the originally planned action, but it is still possible to create a new plan to reach the activity goal, despite of executing the selected action. Objectives. An objective is a learning goal to be reached. It shows the knowledge or abilities to be acquired by the students, it is used to create a suitable syllabus for the student and to evaluate him. In the tutoring strategy designed, three kinds of objectives have been identified: – Didactic Objectives o Phase Objectives, general goals that must be reached in the syllabus’ phases. – Specific Objectives o Activity Objectives, reached when the student finishes correctly an activity (surpassing a certain reliability threshold, configurable in the tutoring strategy). – Action Objectives, reached when the student executes successfully an action. 4.2

Tutoring Strategy

The tutoring strategy defines the virtual tutor didactic behavior during the course. Despite the structure of this strategy is fixed, it has been conceived as a very configurable element, with several customizable parameters to adapt the tutoring strategy to every specific training course and the student characteristics. Strategy Stages. The Tutoring Strategy has been divided into three stages, that every activity follows sequentially. The first stage is the Session Arrangement, in which the students are allocated for the starting session, their knowledge information stored in previous sessions is recovered, and the tutoring strategy is revisited for every single student, in order to personalize it, offering them an activity or a list of activities to begin/continue their learning process. The second stage is the the Group Organization, in which roles are assigned to the students participating in the activity, depending on their skills. In addition, the Tutoring Agent may create autonomous participants to cover the roles not assigned to any human student. The third stage is the Activity Management. This is the hardest stage, which main goal is to make a tracking of the participants learning process. The Tutoring Agent will check whether the actions are correctly executed and in the convenient order, matching the plan generated by the Planning Agent. Then, on the one hand, the Tutoring Agent will update the student evaluation and the list of

An Agent-Based Adaptable and Configurable Tutoring Module

507

the reached objectives. And, on the other hand, if the action do not match the planned one, the Tutoring Agent will make the appropriate learning decisions, according to its tutoring strategy, asking even the Planning Agent for building a new plan. In addition, the Tutoring Agent will decide when the students need any clue and will answer their questions. Finally, the Tutoring Agent will manage the activity ending, both the successful and the unsuccessful one. Assistance System. One of the most important features needed to provide a software tutor with an intelligent behavior is the ability to offer appropriate assistance to the student, not only when he requests it, but also inferring when he needs it. The assistance system is highly interactive, since the communication must be as much real as possible. With this aim, it has been designed a complete taxonomy covering the different kinds of assistance the student may need, and establishing also several levels of help detail. This classification includes: – Descriptive information: with which the Tutoring Agent describes the courses, phases and activities and their main goals and procedures associated. If the student finds this information insufficient, he can request more detailed information, but his evaluation may then be slightly affected. – Clues: used by the Tutoring Agent to provide the student with dynamic help about the execution of a specific action. Clues will be given whenever the time elapsed between the execution of two actions is too high, and also when the student seems to be confused or lost, moving away from the place where the next action has to be executed, or looking only at objects not needed for that action. The assistance system includes a complete hierarchical clue structure for every specific action. When a high level clue has no effective result on the student, new more detailed clues are offered. The last clue may be the own virtual tutor showing the student how to perform the next action. – Questions: While clues are prompted by the Tutoring Agent, questions are initiated by the student as a demand for additional information. In contrast to the clues, now it is the student who starts the search of additional information by means of questions to the tutor. In the tutoring strategy described, there have been proposed three kinds of available questions: • General questions: questions associated to the objects in the VE, such as “What is this?” (pointing to an object in the VE). • Basic questions: questions related to the actions to be executed, e.g. “Do I have to transport this object to some other place?”. • Advanced questions: questions also related to the actions the student has to execute, but, due to its more detailed content, they may only be posed once the related basic question has already been made. The prize would be a higher penalization in the final qualification of the student for that activity. An example of this kind of questions is “What is the object that I have to transport and where do I have to move it?”.

508

L. S´ anchez and R. Imbert

– Path tracking: in the case of movement and transport actions, the PathPlanning Agent oversees if the student follows the optimal route to his destination. If the student fails in following it, the Tutoring Agent is informed and warns the student of this mistake. Evaluation Strategy. The Tutoring Agent is also the responsible for the evaluation of the knowledge acquisition of the student. It makes a personal tracking of the student learning process, valuating the time invested in the activity achievement, the number and seriousness of the mistakes made, and the amount and kind of the assistance needed. Thus, the proposed Tutoring Agent includes an evaluation strategy considering, mainly the activity duration, the failed actions and the additional assistance provided. The student evaluation is related to the specific goals reached. If the student overcomes an activity, the “reliability level” of his goals is increased, and new activities may become available for the student. In subsequent activities, the tutor may check if the student has actually reached those specific goals, increasing or decreasing its “reliability level”. Configurable Parameters. As it was mentioned above, the tutoring strategy has some configurable attributes, so the behavior of the proposed Tutoring Agent can be easily adapted to courses addressing different subjects, to several VEs and to students with specific characteristics. It will be task of the human tutor to tune the value of those parameters to produce an ad hoc Tutoring Agent. Although the number of configurable parameters may be increased to create a much more versatile and generic tutoring strategy, the Tutoring Agent is being evaluating with the following ones: – Mistakes processing: definition of how the tutor must act when the student executes a wrong action (stop the activity, allow him to realize the mistake on his own. . . ). – Objectives reliability level: number of activities in which a specific objective has to be reached, to consider it definitively achieved by the student. – Evaluation marks: how mistakes and good choices affect the student evaluation. A numeric value must be assigned to the following attributes: • Replanning Actions: every replanning action executed will affect the final evaluation negatively (except for alternative action plans). • Clues: the number and importance of the provided clues will be considered to penalize the evaluation mark. • Questions: similar to the actions penalization procedure, the questions posed by the student will be weighed depending on their seriousness to affect the final mark. • Activity duration: if the student spends too much time on performing the activity, its final evaluation will be affected negatively. However, fast students will be rewarded. – Action validation: two different strategies may be followed to present the student the actions he can execute in a given moment: to show him only those which preconditions are currently satisfied, or to show him any of the possible actions existing, exposing him to select an invalid one.

An Agent-Based Adaptable and Configurable Tutoring Module

5

509

Evaluation Prototype

The Tutoring Agent described is being evaluated through an application prototype, developed with the MAEVIF platform. This prototype represents a laundry service simulation, with which the students are trained in how to use a washing machine and how to perform a correct washing cycle, learning which is the more suitable washing program, the detergent type to use, and the water temperature to be selected, in order to wash any kind of clothes, given a number of their attributes: color, number and difficulty of stains, etc. . . This prototype has been selected because it is easy to understand, it does not need a complex VE, and the scenarios can be easily modeled. This has allowed to pay more attention to the creation of a stable and well structured ITS. In addition, all its activities can be performed in a reduced space, allowing the use of short physical range immersive VR devices. This prototype has also shown to be appropriate since the training courses activities are not too linear, allowing the student to reach the activities goals following alternative sequences of actions. The prototype is also interesting because some of the executable actions may generate simulations. E.g. if the washing cycle is not performed correctly, the clothes could shrink or lose color, or if the student puts too much detergent, lather could spill from the washing machine. A complete syllabus for this prototype has already been defined. The activities of the first phase are very simple, training the student on learning where the objects to be handle are, and how to handle them: the washing machine, the cloth basket, the washing machine controls, etc. . . The activities complexity increases in the next phases. In the second one, the student has to learn the steps to be followed to perform a washing cycle. In the third phase, the student has to acquire advanced knowledge, such as separating the clothes depending on their characteristics and to choose the suitable washing program, detergent and water temperature for each heap of clothes. In the final phase, the student has to perform complete washing activities and the tutor assistance is reduced.

6

Conclusions and Ongoing Work

Although VR technology provides ITSs with interesting potential improvements, it does not mean to move the focus of this kind of systems from the pedagogical to the technological side. Their ultimate purpose is to educate and, thus, they must present efficient tutoring capabilities. The tutoring module presented offers interesting pedagogical capabilities, together with an adaptable structure, making it easy its reusability in different contexts, independently from the subject to be taught or the student skills. This module is able to perform a personalized tracking of the learning process of every student. The Tutoring Agent that implements this module in the MAEVIF platform, controls and supervises all the actions executed by the students and their behavior in the VE, and provides them with suitable assistance. The

510

L. S´ anchez and R. Imbert

agent also evaluates the educational performance of the student, identifies the goals and pedagogical objectives reached, and updates the student model. The tutoring module described has been implemented and it is currently being evaluated in a pilot prototype. Once it has been refined with the lessons learnt during this evaluation phase, it has been planned to test the tutoring module versatility, including in its strategy different pedagogical approaches. Finally, to increase the flexibility of the tutoring module, it has been thought to allow the human tutor to define certain configurable parameters as “autoconfigurable”, so that the Tutoring Agent would be able to re-adjust those parameters dynamically during the training, gaining in adaptation of its own behavior to the student skills.

References 1. A. de Antonio, J. Ram´ırez, R. Imbert, and G. M´endez. Intelligent virtual environments for training: An agent-based approach. In Procs. of the 4th Int. Central and Eastern European Conf. on Multi-Agent Systems (CEEMAS’05), volume 3690 of LNAI, pages 82–91, Budapest, Hungary, 2005. Springer. 2. J.C. Lester, B.A. Stone, and G.D. Stelling. Lifelike pedagogical agents for mixedinitiative problem solving in constructivist learning environments. User Modeling and User-Adapted Interaction, 9(1–2):1–44, 1999. 3. J.C. Lester, J.L. Voerman, S.G. Towns, and C.B. Callaway. Cosmo: A life-like animated pedagogical agent with deictic believability. In Procs. of the IJCAI’97 Workshop on Animated Interface Agents, Nagoya, Japan, 1997. 4. J.C. Lester, S.A. Converse, S.E. Kahler, S.T. Barlow, B.A. Stone, and R.S. Bhogal. The persona effect: Affective impact of animated pedagogical agents. In Procs. of the SIGCHI Conf. on Human Factors in Computing Systems, pages 359–366, New York, USA, 1997. ACM Press. 5. J. Rickel and W.L. Johnson. Animated agents for procedural training in virtual reality: Perception, cognition and motor control. Applied Artificial Intelligence, 13:343–382, 1999. 6. M. Evers and A. Nijholt. Jacob - an animated instruction agent in virtual reality. In Procs. of the 3rd Int. Conf. on Multimodal Interaction, pages 526–533, 2000. 7. W. Swartout et al. Towards the holodeck: Integrating graphics, sound, character and story. In Procs. of the 5th Int. Conf. on Autonom. Agents, pages 409–416, 2001.

A Multimodal 3D Storytelling System for Chinese Children Danli Wang, Jie Zhang, Jie Li, Guozhong Dai, and Qiang Lin Institute of Software, The Chinese Academy of Science, Beijing 100080, China [email protected]

Abstract. storytelling by a child, as an educational activity, influences significantly the child’s linguistic ability, thought process, imagination, and creativity. There presently exist many software-based storytelling applications. However, most are deemed not suitable to Chinese children due to various cultural and language barriers. In this paper, combining the characteristics of Chinese children and using the multimodal interaction pattern based on speech and pen-gesture, we present a 3D storytelling system for children. The usability of two versions of the storytelling system is evaluated. Evaluation results suggest version 2 of the system to be of higher usability, as are comments on revising the software. Keywords: Storytelling System for Children, Multimodal Interaction, Usability Evaluation.

1 Introduction Storytelling has been going on for centuries, it is a wonderful way to educate and entertain. Stories are not direct or personal, but they convey a message that can be interpreted by other world views and perspectives [1]. During the process of learning language, storytelling is essential for children to develop their language expression, logic thinking, imagination and creativity [2]. Children learn to understand the world and culture surrounding them, and to recognize and blend themselves into the social and cultural context, by telling stories during their growth [3]. With the rapid advancement of computer technology, we can inspire children’s learning interest, enhance their initiative, and cultivate their creativity by applying advanced educational methodology and providing an effective learning environment, utilizing technology. However, due to the unique weaknesses of children because of their age, e.g. intelligence, knowledge, experience and ability, it is difficult to develop such educational software for children. Recently many researchers have developed environments, tools and software products for storytelling by children. Among them are SAGE [4], KidPad [5] StoryRoom [6], Dolltalk[7], Virtual computer peers[8,9], StoryMat[10]. Each of the systems has the techniques and uses that amuse children and give them better experiences. However these systems do not appear to be completely suitable for Chinese children in certain aspects of contents and forms. The reasons include: (a) the educational contents of the software is not suitable for Chinese culture and K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 511–526, 2007. © Springer-Verlag Berlin Heidelberg 2007

512

D. Wang et al.

environment. It is a long history of China which gives China an inimitable moral criterion and culture background. Hence, parents and teachers support the education based on the traditional culture. Children are edified by the traditional Chinese story since an early age.. As a result, it is difficult for the children to understand the foreign culture and not gain so much from the above systems; (b) the difference in the educational pattern for Chinese children results in difference in arousing creativity compared to children elsewhere in the world, according to the unique traditional culture of China. Chinese children are accustomed to be arranged by their parents and teachers, and they are accustomed to accept what their parents and teachers give. As a result, they will be more introverted and timid than children elsewhere in the world. It makes Chinese children need to be more guided and inspired; (c) the interaction of some storytelling software systems is too complex to operate and learn for the children. Therefore, we absorbed the essence from existing children software, and based on the specific needs of the Chinese children, to develop a multimodal 3D storytelling system. To design application software suitable for children, not only the contents need to be considered, but also the interaction method. Since the concept of multimodal interaction was proposed by Bolt in 1980,[11] although there have been many methods and application systems developed [12], many of them still are based on the assumption that there exists no discerned error in single-channel [13, 14]. Zhang et al. proposed a good solution to fuse multimodal information [15]. Yet the interface method in that work appears not suitable for storytelling systems. Due to limitation of literacy for pre-school and lower-grade school children, speech input is the most effective way to tell stories[16]. But the recognition rate for speech input at this level has been less than satisfactory. Therefore, the multimodal interaction pattern combining speech input and pen input is used to facilitate storytelling for children on the computer as a comparatively practical and realizable method. In designing the user interface (UI), it needs to consider the cognitive level of the child, and it is better to make the interface simple and clear, so that the child can learn the software quickly and to create his or her own stories. User survey and usability evaluation are effective ways to improve the usability of software. Usability is the effectiveness, efficiency and satisfaction with which specific users achieve specified goals in a particular environment. Many researchers began to focus on this issue from the 1980’s, and provided many approaches to improve the usability with an enormous amount of research and practices [17, 18]. It was not until recently that Chinese researchers emphasized the issue of software usability, and some research groups were formed, and these made certain progress of HumanComputer Interaction in this field [19]. In this paper, the usability evaluation solution for the storytelling system is designed using interface-scenarios [20], and an eleven-user evaluation experiment was organized. The evaluation results were analyzed. Suggestions for improving the software were also provided. The organization for the remainder of this paper is as follows: Section 2 discusses the related works. Section 3 presents the multimodal 3D storytelling system. Section 4 discusses the usability evaluation of the system. Finally the conclusion and future work are provided in Section 5.

A Multimodal 3D Storytelling System for Chinese Children

513

2 Related Works 2.1 Storytelling System for Children Language ability is very important for children. Research has pointed out that the language communication between parents and children is usually via storytelling. A child between 6 to 7 months old can understand the short story with pictures told by the parent About 10 months old or so, he can answer the questions, when the parent asks. Around one and a half years old, he can speak 1-2 words to answer the question. And about 2 years old or so, he can answer simple questions. At 3 years old or so, he can draw the conclusion for a story. At about 4 years old, he can repeat the story by himself [2]. It is a good way to develop language skills by storytelling. Doing so can cultivate the faculty of the memory, learn how to use words correctly, and can inspire the imagination of the children and train his thinking ability. It provides the opportunity to cultivate the creativity and art expression of children. It is also helpful to the child in evaluating himself and finding his inner world. With the technical advancement, many researchers began to study how to provide the environment and tools of telling stories by using the computer. SAGE was developed by the MIT media lab, and it provides a storytelling tool to let children develop the interactive stories, and make them express their feelings in the game environment. SAGE can help children develop their own story from the stories they heard before [4]. KidPad, developed by Hourcade, is a coordination creation tool designed for children. It provides the drawing tool in a large two-dimensional zoomable space [5]. Children can develop a story and display it in the space using various drawing elements. The story described by language can be hyper-linked with the objects in the space. KidPad provides advanced interactive technology under a cooperative environment. StoryRoom was developed by the HCI Lab of the University of Maryland. Its size is the same as the room, and it provides the immersed environment for telling stories [6]. Children can create and share their own stories using the storytelling tool of this system. Such a system emphasizes children as the creator of the story, and not as a participant. “Swan”, developed by the Institute of Systems Science of Chinese Academy of Science(CAS), is a software system that transforms the natural language to animated cartoon [21]. It can let the user compose a fairy tale using native Chinese language with certain constraints. In order to make the story reasonable, it utilizes lots of common knowledge. The system provides abundant animated cartoon material and speech database. At present, there are several multimedia interactive child education software systems in the domestic Chinese market, for example: WaWaYaYa, etc., already with abundant contents and imaginary scenarios. Because foreign software typically cannot adapt to Chinese children in culture, but on the other hand domestic software has the poor usability, we developed a multimodal storytelling system to provide a storytelling environment with easy learning and high usability. The aim of this system is to cultivate the imagination and creativity of children.

514

D. Wang et al.

2.2 Multimodal Interaction Multimodal user interfaces allow users to interact with computer via various different sensory channels compared to traditional interface [12]. Multimodal interaction systems permit a more natural human computer interaction, where users can apply more than one modality to express their needs. A central part of these systems is the modality fusion component that takes care of combining and integrating the monomodal interpretations of multiple modalities into a single semantic representation of the intended meaning [12]. Basically, there are two major architectural approaches to the analysis part of multimodal systems: (a) early fusion — fusing modes already at signal or recognition level and (b) late fusion — multimodal semantic processing [22]. The speech and pen-gesture multimodal system proved that multimodal systems could support mutual disambiguation of errors and lead to more robust performance [23]. In this system, in order to achieve the mutual disambiguation benefits, semantic constraints between the user’s voice commands and a set of pen gestures was imposed on what constitutes a “legal” multimodal command. Much of the current multimodal research is based on the premise that each individual mode does not have recognition errors, thus no treatment is given to address such errors [13, 14]. Zhang et al. [15] proposed a gaze and speech multimodal interface, combining the two techniques overcame the imperfections of recognition techniques, and compensated for the drawbacks of single mode. The gaze and speech interface also supports mutual correction of individual input signals, and reduced the error rate and improved interaction efficiency. However, such interface technologies are more suitable to certain types of applications. For children’s storytelling systems, speech and gestures are considered more beneficial. Therefore, our design is pen gesture- and speechbased. 2.3 Usability According to the definition of IS99241-11, usability is the effectiveness, efficiency, and satisfaction with which specified users achieve specified goals in particular environments. At present, lots of interactive systems suffer from usability problem. For example, it will lose almost 50% potential sales opportunity if a user cannot find the required information under the web environment. A poor design will waste 40% duplicated transaction costs, and lose half of the potential sale. Why does this phenomenon occur? The reason is the low-quality requirement analysis, lack of participation by the customers, and poor design of user interface. Moreover, the key problem is lack of an effective usability design and evaluation method. In the usability field, Prof. Carroll proposed a usability engineering model based on scenarios [24]. This model includes 3 phases: analysis, design and prototype/ evaluation. Gabbard et al. proposed a user-centered virtual reality evaluation method that includes 4 evaluation stages: user task analysis, expert guidelines-based evaluation, formative user-centered evaluation and summative comprehensive comparison evaluation [25]. Mayhew proposed the life cycle of usability engineering in 1999[26], and his model divides the life cycle of usability into 3 stages: requirement analysis, design/testing/development and installation. Every task consists of many sub-tasks. This approach integrates the usability tasks into conventional life

A Multimodal 3D Storytelling System for Chinese Children

515

cycles of software development, and facilities the person who lacks the usability experience to use this approach [27]. Welie et al proposed the hierarchical model of usability, and divided the usability into 4 layers from low to high: knowledge, means, usage indicator and usability. These layers are related altogether, and a lower layer supports the higher layer. The system usability can be described completely using the 4-layer model [28]. Based on a great deal of experiments, Sauro provided a onedimensional index to scale usability [29], where usability can be measured by the execution time, number of errors, completion, and satisfaction. The final usability is the sum of these 4 indexes multiplied by 0.25, and the coefficient is obtained by experiment analysis. In recent years, the usability research has exhibited a new trend – “user experience”. Jordan proposed the concept of “four pleasures”, and depicted the contents of “user experience” from a different prospective. The “four pleasures” are several different ways during the interaction between human and products and service, and include: physiological, social, psychological and conscious aspects. The product design should satisfy user experience from these aspects so as to let the user feel pleasurable [30]. In China, ZheJiang University, Dalian Maritime University, Beijing Normal University, Peking University, the Institute of Software of CAS, and the Institute of Psychology of CAS carry out the related research [19]. The European usability seminar tours, organized by the Sino-Europe usability research center, made great contribution for popularizing and spreading usability research achievements. However, in general, the usability research lacks excellent methods in China. In order to ensure the usability of the storytelling system, in this paper, the usability evaluation experiment is provided, and the two current versions of the software were analyzed and tested. The idea of user-centered design is adopted, as is iterative developing method.

3 The Multimodal 3D Children Storytelling System To enhance the imagination and creativity of Chinese children, and to develop their linguistic ability, we developed a multimodality-based 3D storytelling system for children. 3.1 Framework of the System The children storytelling system mainly includes: multimodal user interface, and multimodal fusion, story design, the generation of 3D animation and story play. The framework is shown in Figure 1. 3.2 System Functionality The main function module includes telling and listening to the story. The module for telling the story includes two functionalities: the design of story background and scenarios. At the stage of the designing story background, the user can select the natural scenes (such as trees, flowers, houses, and rivers etc.) and the ground and sky

516

D. Wang et al.

Multimodal user interface

Multimedia output

Pen and speech input

Multimodal fusion

Telling story

Story exhibition

Listening story Cartoon generation

Story selection

Scenario design

Knowledge base

Resource base

fusion rule,

Texture, ole model

background knowledge

s ecial effect t ical stor

Fig. 1. Framework of the storytelling system

texture in the 3D space by pen, and place them. He can also select 3D special effects. For example, if the story occurs in the rainy whether, he can click raining special effect to simulate the rain falling. The system will find these resources in the resource library respectively and render in real time, and user can observe them from different views. It is important to select the roles of the story in the background design. Therefore, a lot of 3D little animals are designed as the roles of the story. In the scenario design of the story, user can use speech to describe the story, and use speech or pen commands to control the action of a role, and get the correct semantics by the multimodal fusion. As a result, it can generate 3D cartoon tasks, and record the action sequence of user design, and dynamically update the state information of the existing models of the story (the model type, position and size parameters etc.). Therefore, it is convenient to generate the corresponding animate effect in the generation module of cartoon.

A Multimodal 3D Storytelling System for Chinese Children

517

The module for listening to the story includes the functionality of the story selection and play. The module of story play shows users the existing stories from a list, and the whole effect of the scenarios, cartoon, and speech, and dynamically tracks the moving role in order to enhance the dynamic display effect and appeal.

4 Key Technique Multimodal fusion Focusing on storytelling problem for children, the multimodal pattern is selected, and it includes speech input and pen pointing. The speech will describe every sentence in the story, and pen-pointing is to point the information of object, position and path in the interface. The specific process of multimodal fusion is as follows: first, the information coming from two channels will be recognized and understood, then judge whether the semantics from the two sides is coincided. If it does, it can obtain the fusion semantics directly. If it does not, the semantics will be fused according to the context and the system’s existing knowledge. After that, the output task will be planned according to the fusion result, and export to the user in the appropriate form. The user can provide the information of correctness for any fusion error afterwards[33]. Generation of cartoon The 3D cartoon is a precise and strict structure, and it will describe the concrete tasks after fusing the semantics information from two channels. It is specified as a fourelement array: 3D_Task The four-element array represents the motion role, style, and whether there is position information (for example, the action of “say” does not need the position information) and concrete position information. The generation module of cartoon will produce the real-time cartoon according to the structure information of 3D_Task, and includes the following: according to the position information of the role model, it computes the moving direction, displacement of the role, and controls the walking process, and any bumps and shelters during walking process. 4.1 Realization of the System Based on the above fusion algorithm of multimodal pattern, the storytelling system for children was developed. It was implemented on a PC computer running Microsoft Visual C++ 6.0, and supports children to interact with computer using two channels: speech and pen. Among them, the speech channel adopts the Microsoft Speech SDK (SAPI) engine, and transforms the speech data to the corresponding semantics, and the pen-channel provides users to sketch freely using a tablet. Children can create the stories by natural language or sketching, and the system uses 3D cartoon as the feedback to the scenarios described by the user. The realization of 3D rendering is

518

D. Wang et al.

Fig. 2. Interface for Version 1

Fig. 3. Interface for Version 2

done with the openGL graph library. Besides different record patterns, the two versions have slightly different interface. Figure 2 and Figure 3 is the layout for the two versions.

5 Usability Evaluation of the Storytelling System 5.1 The Evaluation Method Among all of the evaluation methods, user testing and questionnaire were selected to evaluate the storytelling system. The user first finishes a group of tasks according to the requirements, then he or she answers the questions. In order to let the user learn the storytelling software quickly, the training scheme and testing tasks are devised using interface scenarios, and this method has been proven to efficiently improve learning rate of the user and finishing rate of the tasks. 5.2 Design Evaluation Based on user research and task analysis, the design evaluation scheme for the storytelling system is implemented using interface scenarios. It mainly includes

A Multimodal 3D Storytelling System for Chinese Children

519

software specification, training document, testing tasks and questionnaires. In addition, in order to compare the layout of software interface and reasonableness of record pattern, two versions of the software were designed. Interrupted (version 1,V1) and continuous recording (version 2,V2) are used, and two slightly different button layouts are adopted. Comparison test is used, and the training document and testing tasks for the two versions are provided. 4 tasks are specified: version 1-- scenario setting; version 1-- describe the story, version 2-- scenario setting, version 2-describe the story. After testing, the users can appreciate their own stories. In order to compare the evaluation results conveniently, the level of difficulty for the testing tasks for 2 groups is almost the same, and the number of the used operations is almost the same as well. Due to space limitation of the paper, the tasks of scenario setting for version 2 are shown in Figure 4; and the testing tasks of the two versions: recording sub-task for describing the story, are shown in Figure 5. The questionnaire includes two parts: the first is the whole usability evaluation for storytelling software, and it includes 20 questions, and is used to test the evaluation of easy-learning, easy use, reliability, nature, efficiency and satisfaction respectively. The second part is the comparison of the two versions, and mainly compares the interactive nature, the final effect of the story, and interface layout and satisfaction, and it includes 8 questions.

(a) Version 1

(b) Version 2 Fig. 4. Versions 1’s and 2’s testing tasks — scenario setting

520

D. Wang et al.

(a) Version 1

(b) Version 2 Fig. 5. Versions 1’s and 2’s testing tasks — recording sub-tasks during storytelling

5.3 Evaluation Process Participants 11 pupils took part in the evaluation test, five are high school pupils and 6 are postgraduates, and all pupils have learned how to use the software. In order to compare the operation time, error, and integrity of the two versions of the storytelling software directly, the load of task design for scenario setting and telling story was the same (i.e. the difficulty, amount, and the necessary time of the operation etc.). However, a problem with this approach is that the skill of a latter version testing is higher than the one of previous one. In order to reduce this effect, the testing sequence for two

A Multimodal 3D Storytelling System for Chinese Children

521

different versions is, for example, the first person tests version 1 first, and the next person will test version 2 first, and so forth. Evaluation training The staff member first introduces the purpose and arrangement of this experiment, then explains the usage of the software, and gives a demo for these two versions. And then, the instructor introduces the testing requirement and tasks. Testing and questionnaire After a 5-minute exercise, the testers began to test according to the task requirement of two versions. The staff members record the questions during testing. Except for system problems, all other questions will not be prompted. After testing, users fill out the questionnaire. 5.4 Result Analysis Testing result z The time of finishing tasks Table 1 provides the time that every tester took to finish every task, and the data are analyzed statistically. Also the average time and deviation of finishing every task were obtained. Table 1. The time taken to finish tasks Task/finishing time V1: Setting scenario V1: Describing story V2: Setting scenario V2: Describing story

mean( s) 52.50 143.10 54.58 123.34

s.d. (s) 11.22 31.44 11.92 29.57

Table 2. Integrity of finishing tasks (the highest is 1) Tasks/ The integrity of finishing tasks V1: Setting scenario V1: Describing story V2: Setting scenario V2: Describing story

Integrity rate 1 0.96 1 1

s.d.

0 0.05 0 0

The above data show that the average time and deviation for versions 1 and 2 are almost the same, but the tasks of telling story for two versions are not the same. The average time of version 1 is more than about 20 seconds longer compared with that for version 2 (1/6 total average time of telling story of version 2). Comparing the deviation of the time, the results are almost the same. Therefore, the stability of the average time for both versions is coincident. Our test results show that when finishing similar tasks, the time of finishing tasks for version 2 occupy the dominant position.

522

D. Wang et al.

z Integrity Table 2 shows the integrity that the testers finish every task, and the data are analyzed using statistical method, and the average integrity and deviation of finishing every task were obtained. The data of final integrity is good. Except the task of telling story for version 1, the integrity that the testers finish all other tasks is 1, and it means that the finish rate is 100%, and the integrity of telling stories for version 1 is 95%, it means that only for every few testers it occurs that some sub-tasks were not finished. The data shows that the evaluator does not feel that the task is difficult to understand or to implement during finishing tasks, and the tasks and operations is within their cognitive workload and operation scope. z The error condition During the whole evaluation process, when tester operates, problems may occur. The reason may be the habit of the testers, or the operation error, or it may be that the system produced the errors. The statistical results of the errors are: during the process of finishing tasks, the number of system error is 5, and that of operating error is 10. The data is shown in the Table 3. Compare the data between versions 1 and 2, the number of operation errors differ significantly. For version 1, the number of operation error is 9, but for version 2, the number operation errors was 1. Although 9 errors may not be considered many for 11 testers, but compared between the error numbers of the two versions, the predominance of version is significant. The results of questionnaire For the design questionnaire, storytelling includes two parts: one is the overall evaluation of the storytelling software, the other is the comparison of interface for the two versions. Table 3. The error conditions for both versions 1 and 2

The statistics of the errors error number

Version 1 System Operation error error

2

9

Version 2 System Operation error error

3

1

The following analyzes the result of questionnaire from both aspects. z The overall evaluation of storytelling software The questionnaire provides the overall evaluation of our storytelling software, and the results are shown in Table 4 and Figure 6. The average score for every specification is more than 3.9, which is much larger than 3 (the middle value of a degree 5 evaluation table). Particularly, the specification of easy-learning, nature and satisfaction is about 4.2 - 4.5. It means that the only specification of the average value which is lower than 4 is the reliability. This

A Multimodal 3D Storytelling System for Chinese Children

523

Table 4. The average score and deviation of every dimension of evaluation (score range is from 1 to 5) Score

mean

s.d.

Easy use

4.05

0.70

Easy-learning

4.27

0.68

Reliability

3.91

0.72

natural

4.30

0.74

Efficiency

4.12

0.56

Satisfaction

4.45

0.57

Overall Score

4.18

0.66

The average score of dimensions 5 4

4.27

4.05

4.3 3.91

4.12

4.45

3 2 1 0

Easy use

Easy-learning

Reliability

natural

Efficiency

Satisfaction

Fig. 6. The score of every dimension of evaluation for storytelling software Table 5. The average score and the standard deviation of the two versions(the score range is from 1 to 5) Version 1

Version 2

mean

s.d.

mean

s.d.

3.00

1.10

4.00

1.10

3.73

1.01

3.64

0.81

Final effect

3.55

1.04

3.73

0.79

Satisfaction

3.64

1.03

3.73

0.90

Nature Interface layout

suggests that there are some unreliable issues in the system, and this is where the system needs to improve. In addition, users provided some valuable opinions and suggestions in this aspect.

524

D. Wang et al.

z Comparison of the two versions of the storytelling system The statistical result of the questionnaire for both versions is shown in Table 5 and Figure 7. The comparison of two versions

5 4 3

4.00

3.733.64

3.73 3.55

3.643.73

Interface layout

Final effect

Satisfaction

3.00

2 1 0 Nature

Version 1

Version 2

Fig. 7. The comparison of two versions for storytelling system

The average value of naturalness for version 1 is 3, and is lower than the one for version 2 (the score is 4). This means that the naturalness of version 2 is better for the testers. The evaluation of story effect is almost the same for both versions, but the average value of version 2 is higher, and the standard deviation is smaller. That is, the users would prefer the effect of playing the story in parallel. From the comparison data of the questionnaire, it shows that user would prefer the button layout of version 1. In fact, the difference of interface layout is small, therefore, the difference between the evaluation scores is also small. From querying the user about the feeling of the two versions, and from the test results, user appears more satisfied with version 2. Therefore, the average value of version 2 is higher, and the standard deviation is smaller. Overall evaluation The finishing time, integrity, error rate, and the analysis result of the questionnaire show that overall version 2 is better than version 1. In addition, the overall evaluation of storytelling system indicates that the system achieved nearly satisfactory results. The same conclusion can also be drawn from the observation of the evaluation experiments.

6 Conclusion and Future Work Storytelling systems provide an interactive design environment for children. By using computers to help children describe the story, and using pen and speech as multimodal input, the story which children describe can be rendered to them by way of 3D cartoons. It can inspire the interest of the children creating the story, and make it fun during the creation, and improve their language ability. In this paper, a multimodal storytelling system is devised in two versions, each having gone through

A Multimodal 3D Storytelling System for Chinese Children

525

multiple testing and iterative developments. The two versions and the overall system usability were evaluated. The results show that the ease of use and naturalness for both versions are good, and the overall evaluation results for version 2 are better than version 1. However, version 2 still has some problems, and needs to be improved further. Recently we have integrated the interface of both versions, and the demo document provided is based on the operations after modifying the interface. We have several considerations for our future work. New interaction patterns should be added in order to make children create stories more freely. Classification of users according to the knowledge level at different ages, adaptive user interface, and the corresponding story materials will be provided, as is intelligent assistant to help children solve the problem of creating stories and to provide some suggestions during the process. In addition, a large amount of story materials, including background, scenario materials, animals and actions, etc. should be added. Hence we will be able to create better and larger learning and entertainment space for the children.

Acknowledgments The authors appreciate the evaluation performed by Dehua Cui, Guangyu Wu and other participants. The research is supported by National Grand Fundamental Research 973 Program (Grant No. 2002CB312103) , the National Natural Science Foundation of China (Grant No. 60373056), Innovation Fund For Technology Based Firms(Grant No. 05C26221100020) and the CAS Pilot Project of the National Knowledge Innovation Program (KIP) "an action plan to aid the handicapped with science and technology"(Grant No. KGCX2―YW―606).

References 1. 2. 3. 4.

5.

6.

7. 8. 9.

Jones R. Story Telling. http://www.uigarden.net/english/ storytelling, 2006 Wright, A. Creating stories with children. England: Oxford University Press,1995 Qu M. Intelligent Equations for Chinese Children. China Women Publishing House, 1998 Bers M and Cassell J. Interactive Storytelling Systems for Children: Using Technology to Explore Language and Identity. Journal of Interactive Learning Research, 1999, 9 (2): 603-609, AACE Hourcade J P, Bederson B B, Druin A, Taxen G. KidPad: Collaborative Storytelling for Children. In Extended Abstracts of Human Factors in Computing Systems (CHI 2002), 2002 Alborzi H, Druin A, and et al. Designing StoryRooms: Interactie storetelling space for children. In Proceedings of Designing Interactive Systems(DIS-2000), 2000, ACM Press, 95-105 Vaucelle C, Jehan T. Dolltalk: a computational toy to enhance children's creativity. CHI 2002, Kehoe C, et al. Out of the Lab and into the World: Bringing Story Listening Systems to the Classroom. AERA, 2004 Ryokai K, Vaucelle C, Cassell J. Literacy Learning by Storytelling with a Virtual Peer. In Proceedings of Computer Support for Collaborative Learning 2002

526

D. Wang et al.

10. Cassell J, Ryokai K. Making Space for Voice: Technologies to Support Children’s Fantasy and Storytelling, Personal and ubiquitous computing, Vol. 5 PP.169~190, 2001 11. Bolt R A. “Put-that-there”: Voice and gesture at the graphics interface. ACM SIGGRAPH Computer Graphics, v.14 n.3, p.262-270, July,1980 12. Oviatt S.L. Multimodal Interfaces, Handbook of Human-Computer Interface, Ed. By J.Jacko & A.Sears, Lawrence Erlbaum: New Jersey, 2002 13. Neal J.G., Thielman C.Y., Dobes A., Haller S.M., Shapiro S.C. Natural language with integrated deictic and graphic gestures, Maybury M.T. & Wahlster W. (Ed.), Readings In Intelligent User Interfaces (pp.38-51). San Francisco: Morgan Kaufmann Publishers, 1991 14. Campana E., Baldridge J., Dowding J., Hockey B.A., Remington R.W., Stone L.S.. Using eye movements to determine referents in a spoken dialogue system. In Proc. of workshop on perceptive user interface. Orland, Florida, 2001 15. Zhang Q, Imamiya A, Go K, Mao X. A. Gaze and Speech Multimodal Interface. In Proc. of the 24th International Conference on Distributed Computing Systems Workshops (ICDCSW’04). 2004 16. Nicol, A, et al. Children are Ready for Speech Technology - but is the Technology Ready for Them?. Interaction Design and Children 2002, Eindhoven,The Netherlands. 17. Carroll J M. Scenarios and Design Cognition. In: Dai G Z(ed.), Dong S H, Chen Y D, Ren X S(sub-ed,), Proceedings of the APCHI2002 User Interaction Technology in the 21st Century, China, Beijing, Science Press, 2002, Vol.1, 23-46 18. Te’eni D, Carey J and Zhang P. Human Computer Interaction: Developing Effective Organizational Information Systems. John Wiley & Sons, Inc. USA: Hoboken， 2006 19. Dong S H. Progress and Challenge of Human-Computer Interaction. JOURNAL OF COMPUTER-AIDED DESIGN and COMPUTER GRAPHICS, 2004, 16(1):1-13(in Chinese) 20. Wang D L, and Dai G Z. Pen-Based User Interface Design Using Scenarios Tree. Journal of Computer-Aided Design & Computer Graphics, 2006, 18(8):1276-1280 21. Lu R Q. Knowledge Engineering and Knowledge Science at the Turn of the Century, Beijing: Tsinghua University press, 2001 22. Pfleger N. Context Based Multimodal Fusion, In Proceedings of ICMI’04, State College, Pennsylvania, USA, 265-272, 2004 23. Oviatt S.L., Mutual disambiguation of recognition errors in a multimodal Architecture. In Proc. CHI’99 Human Factors in Computing Systems Conf., Pittsburgh, PA, 576-583. 24. Carroll J M. HCI Models, Theories and Frameworks Toward a Multidisciplinary Science, Morgan Kaufmann, San Francisco,2003 25. Gabbard J L, Hix D, and Swan II J E. User-Centered design and Evaluation of Virtual Environments , IEEE Computer Graphics and Applications, 1999, 19(6):51-59 26. Mayhew D J. The Usability Engineering Lifecycle: A Practitioner's Handbook for User Interface Design. USA:Morgan Kaufmann, 1999. 27. Mayhew J D. The Usability Engineering Lifecycle. San Francisco: Morgan Kaufmann,1999 28. Welie M, Veer G C. Breaking down Usability, Anton Eliëns， 1999 29. Sauro J and Kindlund E. A method to standardize usability metrics into a single score. CHI 2005, 401-409 30. Zhang Y S. Respect, Understanding, Interaction -- Inspiration from the American Family, Beijing Education, 2002, 377(2):16-18 31. Wang D, Zhang J, Dai G. A Multimodal Fusion Framework for Children’s Storytelling Systems. Lecture Notes in Computer Science, Volume 3942: 585 - 588, 2006

Real World Edutainment Focusing on Human-Human Interaction Hiroyuki Mitsuhara1, Yasumoto Hirakawa2, Kazuhide Kanenishi3, and Yoneo Yano1 1

Institute of Technology and Science, The University of Tokushima, 2-1, Minami-josanjima, Tokushima, Japan 2 Graduate School of Advanced Technology and Science, The University of Tokushima, 2-1, Minami-josanjima, Tokushima, Japan 3 Center for Advanced Information Technology, The University of Tokushima 2-1, Minami-josanjima, Tokushima, Japan {mituhara,hirakawa,yano}@is.tokushima-u.ac.jp, [email protected]

Abstract. Edutainment is a promising approach to motivating students to learn, that is, making learning fun. This paper describes a real world edutainment system that makes learning fun using ubiquitous computing technology. We focus on human-human interaction in the real world as a new approach to edutainment in the next generation. Human actors can teach interactively and flexibly in various ways, by considering players' (students') characteristics and the environmental conditions. Involvement of human actors as active characters of a game story in edutainment is an advantage that conventional edutainment does not currently have. Keywords: Edutainment, Real World, Human-Human Interaction, Fun, Ubiquitous Computing Technology.

1 Introduction In the past, computer games were considered nothing more than children’s toys. In addition, as only the negative aspects of computer games were highlighted, many adults concluded that computer games were detrimental to a child's mental growth. Recently, however, the learning efficacy of computer games has attracted increasing attention, and various studies have been conducted on educational computer games [1]-[3]. A current focus is on edutainment, which can be regarded as educational software that increases learning efficacy by stimulating auditory and visual perception with multimedia technology. Edutainment is advantageous in that individuals can learn while having fun. Game consoles are presently widespread in homes, and various kinds of edutainment software are available for these consoles. Considering the recent popularization of edutainment, it is likely that a paradigm shift in computer games is currently in progress. Meanwhile, computer technology is shifting to ubiquitous computing. The concept of “ubiquitous computer technology” can be represented as follows: “Anyone can receive computer network-based services anytime anyplace.” This concept has K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 527–538, 2007. © Springer-Verlag Berlin Heidelberg 2007

528

H. Mitsuhara et al.

become a reality with the breakthrough of advanced computing technology, such as pocket computers, intelligent sensors, and wireless networks. Ubiquitous computing technology has spread quickly, and is actively being integrated into computer games [4]. For example, location-based games involve people playing games in the real world, but receiving information about their current location, for example, from an electronic device [5]-[8]. Pirates! is a ubiquitous technology-enhanced game in which players hold a wireless communication-enabled PDA (Personal Digital Assistant) and attempt to solve missions using information gathered from the PDA while walking around in the real world, which corresponds to virtual islands in the game, in search of trading goods [9]. The spread of ubiquitous computing technology is mirrored in the field of computer-based education/learning systems. TANGO, for example, is a languagelearning environment in which RFID (Radio Frequency Identification) tags are attached to real-world objects and students are tested (e.g., pronunciation quizzes) about the corresponding objects on his/her PDA [10]. Musex is an experiential learning environment that provides students with opportunities to answer questions related to museum exhibitions, again using a PDA and RFID tags [11]. MobileGame focuses on game factors in orienteering activities at universities, and provides location-based awareness information, quizzes, an instant messenger, and other functionalities on a player’s PDA [12]. The use of ubiquitous devices enables authentic learning, which involves the construction of knowledge and skills through students’ experiences in the real world [13]. Furthermore, the use of ubiquitous devices may help make learning fun, as students can learn in different settings. Using these ideas as a background, this study addresses how to make learning fun using ubiquitous computing technology; that is, how to motivate students to learn by using ubiquitous computing technology. We developed a real-world edutainment system called “Realio” that integrates edutainment and ubiquitous computing technologies [14]. In Realio, students learn through interaction with the real world; in particular, it focuses on human-human interaction. Furthermore, while conventional edutainment systems operate mainly on desktop PCs or game consoles, Realio operates on a PDA. The remainder of this paper is organized as follows. Section 2 briefly reviews what makes learning fun. Section 3 illustrates the fundamental idea, composition, and learning process of Realio. Section 4 reports the practice of Realio at an educational event. Section 5 summarizes the study.

2 What Makes Learning Fun Motivation (especially intrinsic motivation) is considered to be quite important for successful learning [15-17]. Malone and Lepper identified the following factors involved in the intrinsic motivation of learning [18, 19]. Challenge: Learning activities (or learning goals) of intermediate difficulty provide the student with a feeling of accomplishment. Curiosity: Learning activities that somewhat differ from a student's prior knowledge provide him/her with new findings.

Real World Edutainment Focusing on Human-Human Interaction

529

Control: Learning activities that allow the setting of priorities provide the student with responsibility and self-direction. Fantasy: Learning activities that do not actually occur in a student's everyday life provide him/her with meaningful imagination. Fantasy is an emotional factor and may be separate from other factors, which are influenced by knowledge and learning style. These factors are believed to make learning fun. In particular, fantasy should be emphasized in edutainment, as it is an important factor of computer games, and can be stimulated in edutainment by the inclusion of appropriate triggers, such as an empathic story and realistic graphics. Of course, the appropriate triggers have to be somewhat related to the learning goal. Denis and Jouvelot indicated that edutainment often fails to make learning fun, because the interaction it provides is limited in comparison with recent computer games [20]. Conventional edutainment limits player interaction (operation) due to educational concerns, as it has been suggested that various kinds of interaction in edutainment might encourage students to focus on the interaction itself and lose sight of the learning goal. Therefore, we think that next-generation edutainment should aim to establish a balance between fantasy and education and between interactivity and education.

3 Realio 3.1 Making Learning Fun As described above, an important issue in making learning fun is controlling the balance between fantasy and education. To do so, our idea adopts story-based learning in the real world; that is, applying a role-playing game in the real world. A game story is necessary to stimulate fantasy. Undoubtedly, nothing is better than the real world in terms of reality, and natural surroundings can increase learning efficacy through authentic learning. A further issue involves controlling the balance between interactivity and education. Regarding this issue, our idea involves human actors as active characters in the game story. Nothing is better than human actors in terms of interactivity. Human actors can provide flexible instruction in various ways, while considering students' characteristics (e.g., knowledge level, language level, interest, and preferences) and the environmental conditions (e.g., place, time, and climate). With this in mind, we developed a real-world edutainment system called “Realio,” which balances edutainment through human-human interaction in the real world. 3.2 System Composition The whole Realio system includes computer software, an edutainment scenario, players (students), human actors, and objects. Players learn by answering quizzes recorded on RFID tags attached to the objects, and by communicating face-to-face with the human actors. The system composition of Realio is illustrated in Figure 1.

530

H. Mitsuhara et al.

3.2.1 Computer Software The computer software, called the “edutainment controller”, operates on a PDA. All operations of the PDA are controlled by direct-manipulation using a stylus pen on the touch screen (e.g., answering multiple-choice quizzes). Principal tasks of the software comprise: (1) RFID tag reading, (2) quiz presentation, (3) answer judgment, and (4) hint presentation. Task 1 is processed by an RFID reader connected to the serial port. Other tasks are processed according to the edutainment scenario created by the edutainment scenario creators (e.g., teachers). This software is built on the Microsoft .NET Compact Framework. Create Edutainment Scenario Creator

Object

Customize

Edutainment Scenario Creator

RFID Tag Transfer

ID No.

Upload

Download

Interact

Quiz

PDA+RFID Reader

Interact

Server

Player Edutainment Controller Edutainment Scenario File (XML)

Real World

Actor

Edutainment Scenario File (XML)

Fig. 1. System composition of Realio

3.2.2 Edutainment Scenario An edutainment scenario creator selects a learning topic and creates an edutainment scenario, which is represented as an XML file (a partial example is shown in the Appendix). The scenario creator can create new scenarios, and also download and customize scenarios already existing on the server. The scenario storyline includes the following characteristics: a linear, goal-based story providing a narrative complement to the quizzes, and the combination of multimedia support from the PDA and the explicit involvement of human beings as supporting characters of the game story. Principal data described by the scenario include (1) game story, (2) quiz ID numbers (recorded as data on corresponding RFID tags), (3) quiz content (e.g., quiz type, questions), multiple-choice items, image data, audio data, etc.), (4) correct answers (items), (5) hints for quizzes, (6) quiz presentation order, (7) speeches and actions, (8) reference information (e.g., URLs), and (9) metadata (e.g., scenario creator, learning topic, target students, etc.). The game story helps absorb the players into the edutainment world (i.e., the fictional world existing in the real world.) 3.2.3 Players Realio can be played by either a single player or one “player group” consisting of a few students. The player group is deemed more appropriate as the group can learn through collaboration.

Real World Edutainment Focusing on Human-Human Interaction

531

3.2.4 Human Actors During the execution of Realio, human actors play supporting roles by providing hints to quizzes and leading the players toward a learning goal, according to the edutainment scenario. 3.2.5 Objects Quiz ID numbers are recorded on each RFID tag (passive type). RFID tags, attached to real-world objects, are used as triggers for quiz presentation. 3.3 Learning Process Initially, players are briefed on the game story by the human actor(s). Players are then given one PDA equipped with an RFID tag reader, and subsequently start the learning (game) process as follows: i) ii) iii) iv) v) vi)

Players search for RFID tags attached to real-world objects (within a limited area). Players find an RFID tag and read it using their PDA. A quiz corresponding to the quiz ID number on the RFID tag is then presented on the PDA. Players deduce the correct answer to the quiz. If players can not determine the correct answer, hints for the quiz are provided by the human actor(s). When the correct answer is input, information regarding the next step (e.g., the location of the next RFID tag and clues for game completion) is provided. When players correctly answer all of the prepared quizzes, the learning game is completed.

4 Practice at an Educational Event Realio was practiced at the “9th Festival of Science Experience in Tokushima (2005)”, an educational event for elementary school students. The edutainment scenario creators and human actors consisted of students engaged in research on educational technology and computer science at the University of Tokushima. 4.1 Learning Topic The scenario creators selected “binary numbers” as the learning topic. Japanese education guidelines state that binary numbers should be taught at high school, which indicates that elementary school students have probably never studied and do not understand binary numbers. The scenario creators assumed that even though the players (elementary school students) do not yet understand binary numbers, they should at least understand that binary numbers are present in today's digital society. Therefore, the learning goal was that players would have fun while learning about binary numbers during the edutainment practice, but may not fully understand binary numbers on completion.

532

H. Mitsuhara et al.

4.2 Edutainment Scenario 4.2.1 Creation Policy The scenario creators determined the following creation policies of the edutainment scenario. •

Conventional quiz-based learning may not be enough to make learning fun. Therefore, an exciting game story should be created that makes learning fun. • High priority should be placed on continuous learning during the edutainment practice. Therefore, scenario creators should provide hints when players can not determine the correct answer. • Players should participate in groups so that they can learn through collaboration. 4.2.2 Game Story Based on the creation policies outlined above, the scenario creators created the following game story, a “detective game”. A criminal has stolen university inventions that were to be exhibited at the “9th Festival of Science Experience in Tokushima”! The criminal is still in the building, pretending to be a festival visitor. He left secret codes inside the building so that his partner could find him. If you can decipher the secret codes, you will acquire the information needed to find the criminal. Find and arrest the criminal before he meets his partner and takes the inventions out of the building! In this game story, players act as detectives. One of the human actors acts as the criminal. Another actor plays a policeman who accompanies the players and provides hints. To find the criminal, the players have to find RFID tags in the building and answer quizzes (secret codes) recorded on the RFID tags. This game story aims to make learning more fun by replacing conventional learning with a game where players decipher secret codes. The length of the learning game was approximately 40 min for each player group. 4.2.3 Quizzes The scenario creators selected three quiz topics (sub-topics of binary numbers), which were set as quizzes given to the players: (1) numeration of binary numbers, (2) addition of binary numbers, and (3) application of binary numbers (imaging and facsimile transmission). Three multiple-choice quizzes were produced (Table 1), and then three RFID tags were attached to the walls inside a six-story building on the university campus. The quizzes were serialized in order of RFID tag reading, and were not presented if the players read the RFID tags in the wrong order. At first, the game taught players the fundamental concept of binary numbers, that is, the fact that only combinations of 0 and 1 can be used to represent decimal numbers. The players were then taught that the addition of binary numbers can be performed in a similar way to that of decimal numbers. Finally, the players were shown how binary numbers could be applied to daily life. The quizzes could be answered using a stylus pen, and the embedding of image and audio data in the quiz content was also allowed. The PDA user interface presented in one of these quizzes is displayed in Figure 2. Information regarding the

Real World Edutainment Focusing on Human-Human Interaction

533

Table 1. Quizzes created for the practice

Quiz Topic #1 Numeration of binary numbers

Quiz Content

Option {6, 8, 10, 12, 14, 16}

Ans.

6

Decipher this secret code. #2 Addition of binary numbers #3 Application of binary numbers

10

＋ 111

{100, 110, 111, 1001,

10111,

Decipher this secret code.

11110}

10001 10001 11111 00001 00001 Decipher this secret code.

{1, 2, 3, 4, 5, 6}

1001

4

Fig. 2. Example of the PDA user interface

location of the next RFID tag (e.g., "The next code is on the 2nd floor") and basic characteristics of the criminal (e.g., "I am wearing a red T-shirt") are presented when the players input the correct answer to each quiz. 4.2.4 Hints The human actor (policeman) accompanying the players provides hints for the quizzes. For the first hint, the human actor presents video learning material created for the scenario on a tablet PC that he/she is holding. As can be seen in Figure 3, standard e-Learning materials (video and synchronized PowerPoint slides) were used. In the video learning material, a human actor playing a cryptographic technician explains the current topic. If the first hint is not effective, the human actor (policeman) then interacts with the players to teach them further, while considering the players’ characteristics.

534

H. Mitsuhara et al.

Fig. 3. Example of video material

4.2.5 Speech and Action The speech and actions of the human actors were ad lib. If the players could not find the RFID tags, the ‘policeman’ would guide them to the correct location. The ‘criminal’ was instructed to try to escape when the ‘detectives’ and ‘policeman’ were close, but was to finally be arrested. 4.3 State of Practice Thirty-eight children participated in the edutainment practice. The age of the children ranged from 5 to 14 years old. Approximately half of the children were accompanied by a parent. Prior to participation, 33 of the children were completely unaware of binary numbers. A photograph of the practice during execution is shown in Figure 4. All of the players completed the game, as the human actors (policemen) led them to the correct answers, according to the creation policies of the scenario. All players received education via the video learning material and interactive teaching. Groups consisting of children older than 9 years old tended to determine the correct answers when the interactive teaching was provided. On the other hand, groups consisting of children younger than 8 years old tended not to determine the correct answers. In such cases, where the players were too young to understand the quizzes, the human actor provided the correct answers. 4.4 Brief Evaluation We collected questionnaire data from 38 of the players and conducted a brief evaluation of Realio. The questionnaire data is summarized in Table 2. The data of for Item 1 shows that the majority of players had fun while learning. Item 2 shows that the majority of players became aware of binary numbers. Item 3 shows that the majority of players were interested in binary numbers. These data

Real World Edutainment Focusing on Human-Human Interaction

535

demonstrate that Realio can make learning fun even if the players have no knowledge of the learning topic, such as binary numbers. Item 4 indicates that many players understood binary numbers. As we did not conduct a post-test to assess their knowledge at this time, it is too early to conclude that Realio helped the children understand binary numbers. Item 5 suggests that the children thought the video learning material was useful for learning (answering quizzes). Item 6 suggests that children thought that interactive teaching was also useful for learning. These data are to be expected, though, because most of the players had no prior knowledge of binary numbers. Interactive teaching was indicated to be very effective, which demonstrates the advantages of Realio (i.e., human-human interaction). Item 7 demonstrates that the majority of players were favorably disposed toward Realio. The above data indicate that Realio makes learning fun. During the practice, we witnessed a very interesting sight that may imply the potential of Realio. After playing Realio, a few of the children were trying to numerate binary numbers using their fingers, with some advice from their parents. This observation implies that children may maintain their motivation after participation through interaction with their parents, and consequently may further their understanding of binary numbers.

Human Actor (Policeman) Player (Detective) RFID Tag

PDA Tablet PC Fig. 4. A scene during edutainment practice

5 Conclusion This paper described a real world edutainment system that makes learning fun using ubiquitous computing technology. As an approach to making learning fun, we focused on human-human interaction in the real world. This approach aims at establishing a balance between fantasy and education and between interactivity and education. Our investigation, using human actors as active members of the game story, demonstrated

536

H. Mitsuhara et al.

that edutainment can make learning fun. This conclusion was indicated by the responses of children who used our edutainment system at an educational event. Use of the edutainment system requires a large number of human resources (edutainment scenario creators and human actors) and ubiquitous devices, which makes it difficult to utilize in regular classes at schools. However, the system is well suited to educational events, such as museum exhibitions. In future we plan to develop an authoring tool that enables scenario creators to create and check scenarios easily using a graphical user interface. We also plan to compare the learning efficacy of real-world edutainment with that of traditional edutainment. Table 2. Questionnaire data. Each value indicates the number of players who selected the item. Some answers were not submitted.

Question (1) Did you learn while having fun? (2) Did you become aware of the presence of binary numbers (BN)? (3) Were you interested in BN? (4) Did you understand BN? (4-1) Did you understand how to count BN? (4-2) Did you understand the addition of BN? (4-3) Did you understand the application of BN? (5) Do you think the video learning material was useful for learning? (6) Do you think the hints from the policeman were useful for learning? (7) Do you want to try this learning (game) again ?

Options (Degree of agreement/disagreement) 1 (No) 2 3 4 5 (Yes) 0 1 4 16 17

Mean 4.28

2

2

3

12

19

4.15

1 3

3 2

7 7

10 12

17 14

4.02 3.84

4

2

11

7

9

3.45

2

3

7

9

12

3.78

1

1

5

13

13

4.09

1

1

5

7

18

4.25

1

0

3

6

23

4.51

0

0

5

7

20

4.46

Acknowledgments. This research was supported in part by the Hayao Nakayama Foundation for Science & Technology and Culture. Without the effort and assistance of the students from Yano Laboratory, who participated as edutainment scenario creators and human actors, the edutainment practice would not have been successful. Thanks are expressed to M. Kumada, T. Okada, E. Sakamaki, T. Sasada, K. Niki, T. Hirokane, T. Mori, M. Mori, S. Inoue, Y. Nishimoto, K. Fujita, and M. Yamada.

Real World Edutainment Focusing on Human-Human Interaction

537

References 1. Eck, R.V.: Digital Game-Based LEARNING― It’s Not Just the Digital Natives Who Are Restless, EDUCAUSE review, March/April (2006) 17-30 2. Kafai, Y. B.: Playing and Making Games for Learning― Instructionist and Constructionist Perspective for Game Studies, Games and Culture, Vol.1, No.1 (2006) 36-40 3. Kirriemuir, J.: Video Gaming, Education and Digital Learning Technologies: Relevance and Opportunities, D-lib Magazine Vol.8, No.2 (2002) http://www.dlib.org/dlib/february02/ kirriemuir/02kirriemuir.html 4. Joiner, R., Nethercott, J., Hull, R., Reid, J.: Designing Educational Experiences Using Ubiquitous Technology. Computers in Human Behavior, Vol.22, No.1 (2006) 67-77 5. Lonthoff, J., Ortner, E.: Mobile Location-Based Gaming as Enabler for Location-Based Services (LBS), IADIS International Conference e-Society 2006 (2006) 485-492 6. Cheok, A. D., Goh, K. H., Liu, W., Farbiz, F., Fong, S. W., Teo, S. L., Li, Y., Yang, X.: Human Pacman: a mobile, wide-area entertainment system based on physical, social, and ubiquitous computing, Personal and Ubiquitous Computing, Vol.8, Issue 2 (2004) 71–81 7. Ritzberger, A., Drab, S. A.: TeamTags: Domination – An AGPS game for mobile phones, Proc. of 2nd International Workshop on Pervasive Gaming Application (PerGames2005) (2005) 99-103 8. Broll, G., Benford, S., Oppermann, L.: Exploiting Seams in Mobile Phone Games, Proc. of 3rd International Workshop on Pervasive Gaming Application (PerGames2006) (2006) 410 9. Björk, S., Falk, J., Hansson, R., Ljungstrand, P.: Pirates! Using the Physical World as a Game Board, Proc. of Interact 2001, IFIP TC.13 Conference on Human-Computer Interaction (2001) 10. Ogata, H., Akamatsu, R., Yano, Y.: Computer supported ubiquitous learning environment for vocabulary learning using RFID tags, Proc. of Technology Enhanced Learning 2004 (TEL2004) (2004) 11. Yatani, K., Onuma, M., Sugimoto, M., Kusunoki, F.,: Musex: A System for supporting children's collaborative learning in a museum with PDAs, Systems and Computers in Japan, Vol. 35, No. 14 (2004) 54-63 12. Schwabe, G. Göth, C.: Navigating and Interacting Indoors with Mobile Learning Game, Proc. of IEEE International Workshop on Wireless and Mobile Technologies in Education 2005 (WMTE2005) (2005) 192-199 13. Brown, J. S., Collins, A., Duguid, P.: Situated Cognition and the Culture of Learning, Educational Researcher, Jan-Feb (1989) 32-42 14. Mitsuhara, H., Ogata, H., Kanenishi, K., Yano, Y.: Real World Interaction Oriented Edutainment using Ubiquitous Devices, Proc. of The 4th International Workshop on Wireless, Mobile and Ubiquitous Technologies in Education (WMUTE 2006) (2006) 150152 15. Prensky, M.: Digital Game-Based Learning, McGraw-Hill Education (2001) 16. Wang, S.: Motivation: General overview of theories, M. Orey (Ed.), Emerging perspectives on learning, teaching, and technology (2001) http://www.coe.uga.edu/epltt/ Motivation.htm 17. Dede, C., Clarke, J., Ketelhut, D., Nelson, B., Bowman, C.: Fostering Motivation, Learning, and Transfer in Multi-User Virtual Environments, Proc. of the American Educational Research Association Conference (2005)

538

H. Mitsuhara et al.

18. Malone, T. W., Lepper, M. R.: Making Learning Fun: A Taxonomy of Intrinsic Motivations for Learning, Aptitude, learning and Instruction ed. Richard E. Snow and Marshall J. Farr, Lawrence Erlbaum, Hillsdale, NJ (1987) 19. Parker, L. E., Lepper, M. R.: Effects of fantasy contexts on children's learning and motivation: making learning more fun, Journal of Personality and Social Psychology, Vol.62, No.4 (1992) 625-633 20. Denis, G., Jouvelot, P.: Motivation-Driven Educational Game Design: Applying Best Practices to Music Education, Proc. of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology (2005) 462-465

Appendix: An Example of Edutainment Scenario

Hiroyuki Mitsuhara Detective Conan Yoshinogawa

A criminal has stolen university inventions…

Policeman He always accompanies...

...

001 What’s binary numbers? OPTION Decipher this secret code. 1.bmp 6 8 10 12 14 16 How to count binary... fb1.bmp

...

Research on Personalized Community E-Learning Recommendation Service System by Using Improved Adaptive Filtering Algorithm Qi Luo1,2 and Pan Zhigeng3 1

Information Engineering School,Wuhan University of Science and Technology and Zhongnan Branch,Wuhan 430223, China 2 School of Electrical Engineering,Wuhan Institute of technology,Wuhan 430070 China 3 State Key Lab CAD&CG, ZheJiang University, Hangzhou 310027, China [email protected]

，

Abstract. To meet the needs of education in the learning community, an improved adaptive filtering algorithm for teaching resources based on vector space model was proposed in the paper. First, feature selection and pseudo feedback were used to select the initial filtering profiles and thresholds through training algorithm. Then user feedback was utilized to modify the profiles and thresholds adaptively through filtering algorithm. The algorithm had two advantages, the first was that it could carry on self-study to improve the precision; the second was that the execution did not need massive initial texts in the process of filtering. The algorithm was also used in personalized Recommendation service system based on Community E-learning. The result manifested that the algorithm was effective.

1 Introduction Nowadays, in order to realize the education idea of life long learning, initiative learning and complete learning for community residents, domestic and aboard have established many learning communities [1]. With the increase of computers and network, people pay more and more attention in community E-learning. E-learning can break through the limit of space and time, reduce learning cost and improve learning efficiency. Therefore, many community websites based on E-learning have also been constructed; community residents can get some information or some study courses. But the application of these websites is difficult to attract community resi-dents’ initiative participation [2]. The investigation indicates that personality Recommendation service system is imperfect. The validity and accuracy of providing information are low, which makes community residents feel unconfident and suspicious in community E-learning. If the community E-learning wants to attract residents, the idea of personalized design should be needed. It means that the personalized knowledge and information service should be recommended according to residents’ needs. At present, many scholars have carried on a great deal of researches on filtering algorithms, such as traditional batch filtering algorithm [3].But those algorithms have some disadvantages, For example, it needs massive initial training texts, yet the K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 539–547, 2007. © Springer-Verlag Berlin Heidelberg 2007

540

Q. Luo and Z. Pan

precision and the recall is low [4]. According to this, an improved adaptive filtering algorithm for teaching resources based on vector space model is proposed in the paper. It has improved the traditional algorithm and introduced adaptive feedback study mechanism in intelligence control. The advantage of the algorithm is that it can carry on self-study to improve the precision, and the execution does not need massive initial texts in the process of filtering. The algorithm is also used in personalized Recommendation service system based on community E-learning. The system can support community education better.

2 Improved Adaptive Filtering Algorithm Architecture Improved Adaptive filtering algorithm based on vector space model is composed of two steps such as training phase and adaptive filtering phase. Step1: training phase. The task of training stage is to get the initial filtering profile and set the initial threshold. Step2: adaptive filtering phase. In adaptive filtering, the main task is to modify the profile and threshold adaptively. We adopt the vector space model to present topics and documents, so each topic and document is transferred into feature vector. 2.1 Training Stage Fig.1 shows the architecture of the training stage. At first, feature vectors are extracted from positive and pseudo-positive documents. The pseudo-positive documents are those that have high similarity with the topic but haven’t been labeled as positive documents in the training set. The pseudo-positive documents can be obtained by several ways. We can get pseudo-positive documents by pseudo feedback. We also can get the pseudo-positive documents by using the hierarchy of categories: a topic’s pseudo-positive documents are those that have the same high-level categories provided by the training set. To get the feature vectors, we first remove stop-words and do morphological analysis on remaining words. Then, we compute the logarithm mutual information between words and topics [5]. The formula 1 is shown as follows:

log MI ( wi .T j ) = log(

p ( wi T j ) p( wi )

)

(1)

Where wi is the i-th word and T j is the j-th topic. Higher logarithm mutual information means wi and T j are more relevant. p( wi T j ) and p( wi ) are estimated by maximal likelihood method. For each topic, we select those words with logarithm mutual information higher than 3.0 and occurring more than once in the relevant documents. Logarithm mutual Information is not only used as the selection criterion, but also as the weight of feature words. After gotten the feature vectors of positive and pseudo-positive documents, we merged them into the initial profile. The initial profile is the weighted sum of positive and pseudo-positive feature vectors. Then we should set initial threshold for each

Research on Personalized Community E-Learning Recommendation Service System

541

topic. The initial threshold is set based on the similarity of each document in the training set [6]. The similarity between a profile and training document is computed by the cosine formula 2: Sim(di , p j ) = cos θ =

∑d

ik

p jk

k

∑d ∑ p 2 ik

k

2 jk

(2)

k

Where p j is the profile vector of the j-th topic and d i is the vector representation of the i-th document. dik is the weight of the k-th word in d i . dik is computed as formula 3: dik = 1 + log(tf ik avdl / dl )

(3)

Where tf ik is the term frequency of the k-th word in the i-th document, the dl is the document length counted by tokens in the document after morphological processing and stop-words removal, the avdl is the average document length gotten from training set. According to the similarities of training documents, each initial threshold is set to get the best filtering performance.

Fig. 1. Architecture of the training stage

2.2 Adaptive Filtering Architecture

Adaptive filtering is very important stage in adaptive filtering algorithm. We get the initial profile and threshold from training stage. While filtering the input documents, the topic profile and threshold is updated adaptively by using all kinds of information such as the user’s feedback, vectors of input documents. Figure 2 shows the architecture of adaptive filtering. When a document arrives, its similarity with the topic profile is calculated. If its similarity is higher than the current

542

Q. Luo and Z. Pan

threshold, then this document is retrieved and the user’s relevance judgment is gotten. If the document is really relevant to the topic, it will be considered as positive sample, otherwise negative sample. The vectors of positive and negative samples will be used to modify the topic’s profile, which is shown in formula 4:

p 'j = p j + α p j ( pos ) + β p j (neg )

(4)

Where p 'j is the topic’s profile after modification, p j is the topic’s profile before modification, p j (cos) is vector of positive samples gotten at this updating interval while p j (neg ) is vector of negative samples; α and β are the weight of positive and negative vectors respectively.

Fig. 2. Architecture of the adaptive filtering

2.3 Threshold Optimization Algorithm

A new threshold optimization algorithm is proposed in the paper. The definition for our threshold optimization algorithm uses the following notations: t: the document’s number, since the documents are processed by temporal order, t also can be considered as time. n(t ) : The number of documents processed theretofore. nR (t ) : The relevant documents retrieved theretofore. nN (t ) : The irrelevant documents retrieved theretofore T (t ) : The threshold at time t

Research on Personalized Community E-Learning Recommendation Service System

543

S (tk , tk +1 ) : The average similarity of the documents that have been rejected in

(tk , tk +1 ) interval P(tk , tk +1 ) : The precision of filtering in (tk , tk +1 ) interval, here P(tk , tk +1 ) =

nR (tk +1 ) − nR (tk ) n(tk +1 ) − n(tk )

(5)

Intuitively, we should promote the threshold if the precision is too low and lower the threshold if few documents have been retrieved. We use the S (tk , tk +1 ) and P(tk , tk +1 ) to decide either promote or lower the threshold. The algorithm of adjusting threshold is shown below: IF p(tk , tk +1 ) ≤ EP(tk +1 ) THEN T (tk +1 ) = T (tk ) + α (tk +1 )(1 − T (tk )) ELSEIF S (tk , tk +1 ) < T (tk )r THEN T (tk +1 ) = T (tk ) D1 + S (tk , tk +1 ) D2 ELSE T (tk +1 ) = (1 − β (tk +1 ))T (tk ) Where a (tk +1 ) is the coefficient for promoting the threshold and β (tk +1 ) is the coefficient for lowering the threshold, they also can be considered as function of nR (t ) . In our experiment, we adopt linear function of nR (t ) , which are shown in formula 6 and formula 7. ⎧α (u − nR (tk )) / u α (t k ) = ⎨ 0 ⎩0

n R (t k ) ≤ u n R (t k ) ≥ u

⎧ β 0 (u − nR (tk )) / u nR (tk ) ≤ u n R (t k ) ≥ u ⎩0

β (tk ) = ⎨

(6)

(7)

Here, α 0 is the initial promoting coefficient and β 0 is the initial lowering coefficient. The parameter u indicates the maximum number of positive documents should be used to adjust the threshold and modify the profile. Here we set α 0 =0.02 and u=0.1 and =300. From the Equation, we can see that as time elapsing, nR (tk ) will grow

μ

gradually so the α (tk ) and β (tk ) will decrease gradually. This reflects the trend that the filtering will became better and better and the adjusting step will became smaller and smaller accordingly. The parameter r indicates that in the condition that S (tk +1 , tk ) is lower than T (tk +1 )i r , the threshold should be demoted with the coefficient D1 and D2. In our experiment, we set r=0.1, D1 =0.8, D2 =0.2. By EP(tk +1 ) , we mean that the precision which we hope the filtering should reach at time tk . At first, we treat this parameter as constant and try several different values

544

Q. Luo and Z. Pan

to see the performance of filtering, but the results are not satisfied. Notice that it is rigorous to hope the filtering reach the final expectant precision at beginning of filtering. We adopt a gradual-ascent function. The function is shown in formula 8. EP(tk +1 ) = P0 + ( Pfinal − P0 )nR (tk +1 ) / u

(8)

Where P0 and Pfinal is the precision we hope the filtering to reach at beginning and end of the filtering. The value nR (tk +1 ) − nR (tk ) controls the frequency that the filtering adjusts the threshold. Small value of it means that the filtering adjusts the threshold more frequently. In our adaptive filtering, we set nR (tk +1 ) − nR (tk ) at 1 which means that the filtering adjusts the threshold once it retrieves a positive sample.

3 PRSSCE Personalized recommendation service system based on Community E-learning (PRSSCE) is Fig.3.

Fig. 3. The model of personalized recommendation service system based on Community Elearning

The main function of PRSSCE is to recommend teaching resources for users quickly and accurately according to their interests.

Research on Personalized Community E-Learning Recommendation Service System

545

Feature selecting module selects users’ interested teaching resources according to browsing and the feedback of teaching resources. Then, the information is stored in user personality characteristics storehouse that is tracked and updated promptly. User personality characteristics storehouse saves user personality information such as name, age, occupation, purchasing interest, hobby and so on. It tracks users’ interests dynamically. The key words of user personality characteristics are selected and recorded; it can also provide user characteristics for constructing the module of user interest model. The module of user interest model selects key words from user personality characteristics storehouse and constructs user interest model. The filtering module of personality commodities filters teaching resources according to user model. Personality recommendation module realizes automatic recommendation and request recommendation. Request recommendation is realized by calling the management module of teaching resources and the filtering module of teaching resources. Personality searching module accepts user requests. Personality searching results are obtained through filtering module. Personality cooperation module makes user get help immediately and accurately when they have problems of the system. The work process of model as follows: firstly, user interest feature information of commodities is selected by feature selecting module. The information is saved in user personality characteristics storehouse. Personality characteristics storehouse is tracked and updated promptly. Then, user interest model is constructed by user personality characteristics information. Furthermore, teaching resources are filtered by personality filtering module according to user model. Finally, personality searching service, personality recommendation service and personality cooperation service are realized by the module of personality searching, personality recommendation and personality cooperation, which according to the filtering results.

4 Experiment and Conclusion On the foundation of research, we combine with the cooperation item of personalized knowledge service system in community .The author constructs a system website recommending personalized knowledge and courses service .In order to obtain the contrast experimental result, the traditional filtering batch algorithm and improved adaptive filtering algorithm are separately used in the module of personalized teaching re-sources. The experimental data comes from computer knowledge which is in teaching resources. All texts are the XML forms, they are divided into two aggregate; they are training examples (50639 texts) and testing examples (46900 texts). 64 different subjects are used in the testing. The result of experiment is Fig4. X axis is 64 subjects that are arranged from big to small according to precision. Y axis is the precision of traditional batch filtering and improved adaptive filtering for each subject.12 related texts are provided for each subject through batch filtering. But, 3 texts are provided through adaptive filtering. In addition, only 3 related texts for each subject are provided.

546

Q. Luo and Z. Pan

Fig. 4. The performance of batch algorithm and adaptive filtering algorithm

From Fig4, the performance of batch filtering algorithm does not drop very fast .Two curves approach extremely. In fact, two average values separately are 30.9% and 25.8%, the scope drops only 16.3%. The precision of most categories drops a lot when they does not use improved adaptive filtering algorithm. The average precision is 17.6% and drops 45.2%.This fully explains the adaptive function.

Fig. 5. Comparison with other algorithms

Research on Personalized Community E-Learning Recommendation Service System

547

We also compare adaptive filtering algorithm with other algorithms such as k-NN, Rocchio [7]. The experimental results are Fig.5. From Fig.5, adaptive filtering algorithm based on vector space model is more effective than other algorithms .because; it could carry on self-study to improve the precision. Compared to other algorithms, improved adaptive filtering algorithm has a better capability of recommendation. In summary, an improved new adaptive filtering algorithm for Community Elearning based on vector space model is proposed in the paper. The algorithm is also used in personalized recommendation service system based on community E-learning. The system can support community E-learning better. The results manifest that the algorithm is effective through testing in personalized recommendation service system based on community E-learning. I wish that this article’s work could give some references to certain people. Acknowledgments. The research work in this paper was supported by National Natural Science Foundation of China (grant no.: 60533080), this research work is also supported by project of multi-channel perception and affective computing for virtual humans (grant no.: 2005CCA04400), 973 project (grant no.: 2002CB312100) and 863 project of modeling of behavior and emotion features for virtual objects in virtual environment(grant no.: 2006AA01Z303).

References 1. Luo Qi, “Research on Application of Association Rule Mining Algorithm in Learning Community”, CAAI-11, Wuhan, 2005, pp.1458-1462. 2. Wu Yan-wen and Wu –Zhonghong, “ Knowledge Adaptive Presentation Strategy in ELearning” ,Second International Conference on Knowledge Economy and Development of Science and Technology (KEST2004), Beijing, 2004, pp.6-9. 3. Lawrence R D, Almasi G S, Kotlyar V, et al. “Personalization of Supermarket Product Reommendations” Special Issue of the International Journal Data Mining and Knowledge Discovery, No.5, 2001,pp11-32. 4. Nie Xin. “Take about the Digital Individualized Information Service of Library”, Information Science Journal, vol.23, 2005, pp 1-5 5. Robertson S, Hull DA, “The TREC-9 filtering track final report.” Proceedings of the 9th Text Retrieval Conference. Gaithersburg, 2001. pp 25-40 6. Li Dun and Cao Yuanda, “A New Weighted Text Filtering Method”, International Conference on Natural Language Processing and Knowledge Engineering, Wuhan,2005 pp695698. 7. Joachims T, “Text categorization with support vector machines”, Proceedings of the European Conference on machine learning, Springer-Verlag, 2002, pp.1234-1235.

Efficient Embedding of Information and Knowledge into CSCL Applications Santi Caballé1, Thanasis Daradoumis1, and Fatos Xhafa2 1

Open University of Catalonia, Department of Computer Science and Multimedia Rbla Poblenou, 156. 08018 Barcelona, Spain {scaballe,adaradoumis}@uoc.edu 2 Department of Languages and Informatic Systems Polytechnic University of Catalonia, Jordi Girona Salgado 1-3, 08034 Barcelona, Spain [email protected]

Abstract. This study aims to explore two crucial aspects of collaborative work and learning: the importance of enabling CSCL applications, on the one hand, to capture and structure the information generated by group activity and, on the other hand, to extract the relevant knowledge in order to provide learners and tutors with efficient awareness and support as regards collaboration. To this end, we first identify and define the main types of information generated in online group activity and then propose a process for efficiently embedding this information and the knowledge extracted from it into CSCL applications for awareness and feedback purposes. The conceptual model proposed finally gave rise to the design and implementation of a CSCL generic platform, called the Collaborative Learning Purpose Library (CLPL), which serves as a basis for the systematic development of collaborative learning applications and for providing full support to the mentioned process of knowledge management.

1 Introduction Computer-Supported Collaborative Learning (CSCL) is an emerging paradigm [1] for research in educational technology that focuses on the use of information and communications technology (ICT) as a mediation tool within collaborative methods of learning. When developing CSCL environments that support online collaborative learning, several issues must be taken into account in order to ensure full support to the online learning activity. One such key issue is interaction data analysis, a core function for the support of coaching and evaluation in CSCL environments. It relies on information captured from the actions performed by participants during the collaborative process [2]. The efficient embedding of this information and of the extracted knowledge into CSCL applications sets the basis for enhancing monitoring, awareness [3] and feedback [4] to achieve a successful learning process in collaborative environments. Therefore, the success of CSCL applications depends to a great extent on the capability of such applications to embed information and knowledge of group activity and use it to achieve a more effective group monitoring. In the literature, however, K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 548–559, 2007. © Springer-Verlag Berlin Heidelberg 2007

Efficient Embedding of Information and Knowledge into CSCL Applications

549

questions related to the information and knowledge embedding have not been sufficiently investigated. An initial approach [4] considers the use of feedback in on-line learning and its effects on group activity in general. To this end, some types of information generated by the group activity are considered as relevant knowledge to be communicated to the group members for feedback purposes but the process of how to collect the information, analyze it and extract the desired knowledge is not provided. CSCL applications are characterized by a high degree of user-user and user-system interaction and hence are generating a huge amount of event information. This information can be appropriately collected and automatically processed by computers as a data source to extract relevant knowledge of the collaboration. Note that in this context information refers to the event data generated by the learning group and knowledge refers to the result of the treatment of this information in terms of analysis techniques and interpretations that will be presented to the same group that generated it. Therefore, the event information management is the cornerstone in this context, aiming at facing three main problems: (i) how to obtain and classify the necessary information gathered from the collaborative activity into several description levels; (ii) how to efficiently process the large amount of information collected during the group activity in order to facilitate its later analysis and make the extracted knowledge available to the participants even in real time; (iii) how information should be analyzed and what kind of knowledge should be extracted to be fed back to the participants in order to provide the best possible support and monitoring of their learning and instructional processes. Finally, we need to provide an efficient and robust computational approach that enables the embedding of the collected information and the extracted knowledge into a CSCL application. In order to achieve these goals, we first propose a conceptual model for data analysis and management that identifies and classifies the many kinds of indicators (variables) that describe collaboration and learning into several high-level potential categories of effective collaboration. Then, we enter a process that, as a first step, collects and classifies the event information generated by the group activity according to these indicators. For efficiency purposes, this information may then be structured in a way that facilitates its later processing and analysis. The last stage of this process consists of interpreting the analysis outcomes and communicating the knowledge extracted to the group members for awareness and feedback purposes as well as to the tutors to effectively track the collaborative learning process. Achieving a clear and well-structured conceptual model can greatly facilitate the design of effective computational models that reflect as accurately as possible task performance, individual and group behavior, interaction dynamics, members’ relationships and group support. As a consequence, a generic platform for the systematic construction of CSCL applications is provided. This platform, called the Collaborative Learning Purpose Library (CLPL) [5], translates the conceptual model and its indicators into a computational model of collaborative learning interaction and can be used to embed information and knowledge into CSCL applications in an efficient manner. The paper is organized as follows. We first present a conceptual model that identifies and classifies the three main categories of information generated in group

550

S. Caballé, T. Daradoumis, and F. Xhafa

activity. In Sect. 3 we show how the online information collected and classified can go through a process of analysis with the aim of extracting all the necessary knowledge that is to be presented to the interested actors. This process and the categories defined in the conceptual model are translated into a computational model based on a CSCL generic platform presented in Sect. 4. We conclude in Sect. 5 with some comments and point out ongoing and further work.

2 A Conceptual Model for Managing Group Activity Information The conceptual model we propose shows how the information generated in collaborative learning activities can be captured and classified at several description levels. This fact can significantly improve the way a collaborative system used for learning and instruction can collect all the necessary information produced from the user-user and user-system interaction in an efficient manner. The aim is to model different aspects of interaction and thus at helping all the actors involved understand the outcomes of the collaborative process. To this end, we classify group activity information into three generic categories of activity: the members’ contributing behavior to the task (the outcome of collaboration), the functioning of the group (the interaction processes underlying the collaborative learning activities, such as participation behavior, role playing, etc.), and individual and group scaffolding (social support and task- or group functioning-oriented help). We summarize these issues in this section (for more details, see [6]). Collaborative Learning Outcome (or Task Performance). This category represents the mid- and low-level indicators in the form of the skills and sub-skills that should characterize the students who participate in a learning collaborative situation in order to achieve effective group and individual performance of the task and thus obtain a successful learning outcome. To measure each indicator (or skill), we associate it with the students’ actions, which represent each indicator in the best possible manner. We employ a similar terminology to the one usually used in the BSCW system [7] to refer to the actions that can be carried out in any groupware platform. Even so, they are general enough to be independent from BSCW and represent all the typical and basic actions encountered in every groupware platform. Group Functioning. This category includes the mid- and low-level indicators in the form of skills and sub-skills that students should exhibit in order to enhance participation, promote better communication and coordination, and achieve the effective interaction and functioning of the group in a learning collaborative situation. To measure each indicator, we associate it with specific student actions which best describe each skill to be accomplished. Scaffolding. This category includes the different types of social support and help services [8] that have been identified and accounted for in our model. The participants' actions aiming at getting or providing help are classified and measured according to whether they refer to the task or group functioning.

Efficient Embedding of Information and Knowledge into CSCL Applications

551

3 The Process of Embedding Information and Knowledge into a CSCL Application In this section, we present a process to provide a learning group with relevant knowledge extracted from learners’ interaction data in CSCL applications for awareness, feedback and monitoring purposes. The aim is to greatly improve the effectiveness of the learning exercise. Here, we face two difficult problems: First, the problem of how to define an efficient process of embedding information and knowledge into a computer-mediated collaboration given that several essential steps need taking into account. Second, how to give relevant and semantically grounded feedback on what is happening in a collaborative learning framework to students and teachers in order to allow them eventually to modify the on-going activity. We focus on and discuss a solution to the first problem, while providing some suggestions as how to deal with the second. The process of correctly providing information and knowledge in collaborative learning applications involves three separate, necessary steps: collection of information, analysis and presentation (see Fig. 1 and [9]). The entire process fails if any one of these steps is omitted. During the first step, a structuring and classification of the generated event information is needed. This information is then analyzed in order to extract the desired knowledge. The final step is to provide users with the essential awareness and feedback from the obtained knowledge. 3.1 The Collection and Classification of Event Information The most important issue while monitoring group activity is the collection and storage of a large amount of event information generated by the high degree of interaction among the group participants. Such a large amount of informational data may need a long time to be processed. Therefore, collaborative learning systems have to be designed in a way that classifies and pre-structures the resulting information in order, on the one hand, to correctly collect the group activity and, on the other hand, to increase the efficiency during data processing in terms of analysis techniques and interpretations. Due to its importance, the processing step is treated in great detail later. As shown in the previous section classification of the information is achieved by distinguishing several high-, mid- and low-level indicators of effective collaboration. Based on this, we further categorize and specify users’ particular actions according to the following criteria: -

Who is doing something? (i.e. the originator of the event). When did s/he do it? (i.e. timestamp). Where does s/he do it? (i.e. the location of the affected resource). What is s/he doing? (i.e. the action type and the object involved). Why did s/he do it? (i.e. student intentions and motivation which are captured by the indicators associated with each action; e.g., a user performs the action “create document” in order to generate new information in the shared workspace).

552

S. Caballé, T. Daradoumis, and F. Xhafa

Fig. 1. The process of embedding information and knowledge into CSCL applications

Our aim is to provide a guideline to correctly classify the user actions on the resources during group activity. To this end, we propose a classification process in which the event information collected from the log files is handled in sequential steps consisting of extraction, identification, coding, and categorization (see Fig. 1). In particular, we first extract the specific action performed by a user on a resource (e.g. file document, debate, etc.). Second, this action is interpreted according to the type of event that was involved in (this information should be provided implicitly by the system according to the context where the action was triggered or explicitly by the

Efficient Embedding of Information and Knowledge into CSCL Applications

553

user who triggered it). This provides the basic information that is used for the identification of the real intentions or skills shown by the user (e.g. creating a contribution during a debate can be interpreted as either revision or reinforcement of the information depending on whether the contribution was created in the context of a reply or as an observation). Subsequently, we codify the user event taking into account both the user action and the event type. Doing so, we associate a unique code to the user skill identified in the context of the action. Finally, we categorize the user event into one of the group activity indicators defined in Sect. 2 and shown in Fig. 2.

Fig. 2. A hierarchy to collect and classify all events generated during the group activity

3.1.1 Efficient Processing of the Information Due to the large amount of event information generated in CSCL applications, once this information activity has been correctly collected and classified we may come across the issue of demanding computational requirements while processing this information. In order to facilitate this step, CSCL applications may structure this information as log files in a way that takes advantage of the parallelism in order to process several files (e.g. all the groups in a classroom) at the same time and thus dramatically reduce the overall computational time to process them [10], [11]. As a result, it is possible for these applications to process a large volume of collaboration activity data and make the extracted information available even in real time. Note that this step is optional within this process of embedding information and knowledge and it is proposed for efficiency purposes only. To this end, we propose the following generic steps so as to correctly structure the event information for later processing (see Fig. 1 and [10]): by classifying the event information and turning it into persistent data, we store it in the system as structured files. These files contain all the information previously collected from the system log files in specified fields. Next, we predefine the structured files in accordance with

554

S. Caballé, T. Daradoumis, and F. Xhafa

certain criteria such as time and workspace, which characterize all group collaboration. Our goal is to achieve a high degree of granularity of log files. Thus, during later data processing, it is possible to concatenate several structured files so as to obtain the appropriate degree of granularity (e.g. all groups in a classroom for each 12 hours). This makes it possible to efficiently parallelize data processing depending on the characteristics of the computational resources. To resume, the process of embedding information and knowledge into collaborative learning applications came to the point where the information generated by group activity has been collected, classified and well-structured so that it can be easily and efficiently processed and analyzed during the second stage of this process. We now turn to present this stage whose purpose is to extract relevant knowledge to be fed back to the participants. 3.2 Data Analysis and Extraction of Knowledge The second stage of this process consists in processing all the information previously collected and classified according to the indicators mentioned before by means of analysis techniques. There is a fair deal of research focused on the analysis of online group interaction. A detailed description of an integrated approach we followed to analyze the collaborative learning interaction can be found in [11]. As a consequence of this analysis, knowledge is generated providing meta-cognition about the state and evolution of interaction, which enhances awareness about the efficiency of group activity, group behavior and the individual attitudes of its members in shared workspaces. Knowledge extraction is based on criteria related to the three socio-cognitive functions that operate simultaneously during group interaction, namely production function, group well-being and member support [12] and their associated indicators. In that sense, as regards the production function, we can extract knowledge by constantly observing the members’ activities (e.g. showing each group member's absolute and relative amount of contributions) or the status of shared resources. In addition, we can obtain knowledge that is relevant to individual and group well-being by exploring the communication and interaction flow among group members (such as members' motivational and emotional state, comparative studies of effective and ineffective groups and so on). Finally, knowledge can be acquired by ill-functioning situations, such as missing or insufficient contributions, lack of participation, etc., which can reveal the need for helping individual members by providing them specific scaffolding where and when this is necessary (i.e. member support). The definition of a variety of indicators at several levels of description allows us to determine the granularity of information to be transmitted to the interested parties. In other words, based on a model of desired interaction (establishing a comparison of the current and desired state of interaction), the analysis approach detects and highlights the indicators which were not satisfied and need to be corrected by redirecting group and individual attitudes. These indicators reveal those aspects of the collaborative learning activity (task performance, group functioning, or scaffolding) that present problems and need to be corrected adequately. Thus, they set up rules and filters in

Efficient Embedding of Information and Knowledge into CSCL Applications

555

order to extract and summarize only that information which refers to the malfunctioning aspect. The summarized information is finally transformed into useful knowledge that is communicated to and acquired by the group members who use it to improve the performance of the problematic aspect. Therefore, on the one hand, our approach enables group members to become aware of the progress of their peers in performing the learning exercise both at individual and group level, as well as of the extent to which other members are participating in the collaborative process as this influences their decision making [13]. On the other hand, our approach provides tutors with information about students' problem-solving behavior, group processing [14] and performance analysis [15] for assessment and guiding purposes [16]. This approach is presented below and constitutes the last stage of the process of embedding information and knowledge into CSCL applications. 3.3 Presentation of the Knowledge Acquired Here the problem consists in identifying the roles and needs of each learner and the tutor in every moment and being able to decide what information is required to be provided, in which granularity and how to present it. For example, the knowledge obtained from the interaction analysis should be tailored in such a way that the support provided for self-regulation or peer assessment is adapted to the role the learner plays at a particular moment. In that way, scaffolding information would be different for a learner playing a coordinator role from one that plays a software designer role. Moreover, the format used to present the information could vary from case to case. Consequently, we proceed to define three different levels that dictate how the acquired knowledge is to be presented, that is, at what format and detail level: •

•

•

Awareness level. At this level, we need to inform participants about what is going on in their shared workspace, providing information about their own actions or the actions of their peers, or presenting a view of the group interaction, behavior and performance [3]. To this end we display plane indicator values that show the state and specific aspect of the collaborative learning interaction and processes that take place. The information presented to the learner can support him/her at a meta-cognitive level. Assessment level. At this level, we need to provide data and elements to assess the collaborative activity, so the indicators used are associated with specific weights that measure the significance of each indicator in the assessment process. As in the previous case, the information provided acts at a meta-cognitive level, giving the actors the possibility to evaluate their own actions and behavior as well as the performance of their peers and the group as a whole. Scaffolding (or Guiding) level. Supporting participants during collaborative activities has become a main concern of current research [4], [17], At this level, we need to produce information aiming at guiding, orienting and supporting students in their activity. This information is determined by the unsatisfied indicators and helps students to diagnose problematic situations and self-estimate the appropriateness of their participation in a collaborative activity as well as to counsel their peers whenever insufficient collaboration is detected.

556

S. Caballé, T. Daradoumis, and F. Xhafa

In this section we have discussed the conceptual process that defines how the embedding of information and knowledge can take place in a CSCL application in an efficient manner and how important it is to consider this knowledge as a means to providing awareness, feedback as well as individual and group monitoring. We now turn to discuss the implementation of this approach by a computational model based on a generic platform that is used both for the systematical construction of CSCL applications and for embedding information and knowledge from group activity into them. This platform is presented in the next section.

4 A Computational Model to Embed Information and Knowledge We propose a generic, robust, reusable component-based Collaborative Learning Purpose Library1 (CLPL) [5], [18] based on the Generic Programming paradigm [19] so as to enable a complete and effective reutilization of its generic components for the construction of specific CSCL applications. We use this platform as a computational model especially for the implementation of the conceptual model of information management and the process of embedding this information and the knowledge extracted into CSCL applications as described in Sect. 2 and 3. The CLPL consists of generic components related to user management, administration, security, knowledge management, and functionality mapping the essential needs in which any CSCL application is involved. In this context, we are especially interested in using this platform as a computational model for data analysis and management. This is mainly performed by two components, namely CSCL Knowledge Management and CSCL Functionality components, which form the core of the CLPL in the construction of CSCL applications. Due to its importance, they are described here briefly (see [5] for details). 4.1 CSCL Knowledge Management Component In developing our CLPL we paid special attention to event analysis and management. To this end, this component is made up of two subsystems, namely CSCL Activity Management and CSCL Knowledge Processing so as to support the first two stages of the process of embedding of information and knowledge described in Sect. 3. The third stage, presentation (of user awareness and feedback), is accomplished by a different component called CSCL Functionality that we will describe in the next subsection. The ultimate objective of this component is to define a bottom-up analysis approach that analyses the user events in order to decode the specific actions of the users describing their interaction during the collaboration activities. The analysis aims at identifying those sequences of actions that can be used to determine typical patterns of interactions [20]. At this point, based on a model of desired interaction, the system allows us to compare the learners’ real interaction processes with the typical interaction patterns in order to infer whether or not the process is effective for the learner. Furthermore, the knowledge revealed by this analysis can enhance self and peer evaluation, which in 1

CLPL is found at: http://cv.uoc.edu/~scaballe/clpl/api/index.html

Efficient Embedding of Information and Knowledge into CSCL Applications

557

turn improves the efficiency of group activities, monitoring group behavior and the individual attitudes of its members in the shared workspace. Finally, this knowledge is also essential in assisting the tutor by providing the necessary means to support and assess individual and group learning outcomes. Next component provides the suitable means to present the results of this analysis so that all the actors of the collaboration can consume and benefit from the knowledge acquired. 4.2 CSCL Functionality Component This component, which has five subsystems in all, defines the three elemental parts involved in any form of cooperation, namely coordination, communication and collaboration [18]. Coordination involves the organization of groups to accomplish the important objectives of members such as workspace organization and group structure and planning. Collaboration lets group members share any kind of resources while communication represents the basis of the whole component since it enables coordination and collaboration to be achieved by providing them with low-level communication support. The final objective of this component is to provide functional support to CSCL applications in terms of group organization, resource sharing, user interaction, and so on. Moreover, this component implements the last stage of the process of embedding information and knowledge into CSCL applications by presenting the knowledge generated to users in terms of immediate awareness and constant feedback of what is going on in the system.

5 Conclusions and Future Work In this paper we have shown the importance of providing efficient knowledge and information from collaborative learning activity and how to efficiently embed them into CSCL applications as crucial aspects in the enhancement of the collaborative learning process. To this end, we have first described a conceptual model that captures and classifies three main types of information generated in the group activity by means of potential indicators of effective collaboration and secondly we presented the process of embedding this information and the knowledge extracted into CSCL applications. Then, the categories defined in the conceptual model are translated into a computational model of collaborative learning interaction by means of a generic platform called Collaborative Learning Purpose Library for the construction of collaborative learning applications which is specially used to embed information and knowledge into CSCL applications in an efficient manner. The CLPL library has been evaluated by using it as a basis to develop a Web-based structured discussion forum [5] to support on-line collaborative learning activities. We plan to use this application in our real context of the Open University of Catalonia to gain experience with it. Further work will focus on how to incorporate information retrieval and document filtering techniques into the stage of information collection to automatically extract knowledge from information with a high degree of informality (e.g. to evaluate the group’s well-being function) and thus making it possible to embed it into CSCL

558

S. Caballé, T. Daradoumis, and F. Xhafa

applications. We would also like to investigate how to integrate a portable, general and reusable CSCL ontology in our generic platform as a declarative representation of the knowledge embedded into CSCL applications with the aim to both describe how these systems are built and understand how real groups work. Acknowledgements. This work has been partially supported by the Spanish MCYT project TSI2005-08225-C07-05.

References 1. Koschmann, T: Paradigm shifts and instructional technology: An introduction. In T. Koschmann (Ed.) CSCL: Theory and practice of an emerging paradigm, 1-23. Mahwah, NJ: Lawrence Erlbaum Associates (1996) 2. Dillenbourg, P.: Introduction; What do you mean by “Collaborative Learning”? In P. Dillenbourg (Ed.), Collaborative learning. Cognitive and computational approaches (p. 1-19). Oxford: Elsevier Science (1999) 3. Gutwin, C., Stark, G. and Greenberg, S.: Support for Workspace Awareness in Educational Groupware. in Proceedings of the ACM Conference on Computer Supported Collaborative Learning, Bloomington, Indiana, USA October 17-20, 1995. 4. Zumbach, J., Hillers, A. & Reimann, P. (2003). Supporting Distributed Problem-Based Learning: The Use of Feedback in Online Learning. In T. Roberts (Ed.), Online Collaborative Learning: Theory and Practice pp. 86-103. Hershey, PA: Idea. 5. Caballé, S., Daradoumis, Th., and Xhafa, F.: A Model for the Efficient Representation and Management of Online Collaborative Learning Interactions. In P. Cunningham and M. Cunningham (Eds.), Building the Knowledge Economy: Issues, Applications and Case Studies. IOS Press, Amsterdam, Netherlands. (2006) 1485-1492 6. Daradoumis, T., Martínez, A. & Xhafa, F.: A Layered Framework for Evaluating Online Collaborative Learning Interactions. International Journal of Human-Computer Studies. Special Issue on “Theoretical and Empirical Advances in Groupware Research”, Academic Press: Elsevier Ltd (2006) 7. R. Bentley, Hosrtmann, T., Trevor, J.: The World Wide Web as enabling technology for CSCW: The case of BSCW. In: Computer-Supported Cooperative Work: Special Issue on CSCW and the Web, vol. 6, 1997, 111-134 8. Webb, N.: Testing a theoretical model of student interaction and learning in small groups. In: R. Hertz-Lazarowitz and N. Miller (Eds.), Interaction in Cooperative Groups: The Theoretical Anatomy of Group Learning. Cambridge Univ. Press, NY (1992) 102-119. 9. Caballé, S., Daradoumis, T., Paniagua, C., & Xhafa, F.: A Grid Approach to Provide Effective Awareness to On-line Collaborative Learning Teams. Proceedings of the First International Kaleidoscope Learning GRID Special Interest Group Workshop on Distributed e-Learning Environments. Naples, Italy (2005) 10. Xhafa, F., Caballé, S., Daradoumis, Th. and Zhou, N.: A Grid-Based Approach for Processing Group Activity Log Files. In: proc. of the GADA'04, Agia Napa, Cyprus. LNCS 3292 (2004) 175-186 11. Caballé, S., Paniagua, C., Xhafa, F., & Daradoumis, Th.: A Grid-aware Implementation for Providing Effective Feedback to On-line Learning Groups. In: proc. of the GADA'05, Agia Napa, Cyprus. LNCS 3762 (2005) 274-284

Efficient Embedding of Information and Knowledge into CSCL Applications

559

12. Daradoumis, T., Martínez, A. & Xhafa, F.: An integrated approach for analyzing and assessing the performance of virtual learning groups. In: Proc. of the 10th Int. Workshop on Groupware (CRIWG 2004), Berlin: Springer (2004) 13. McGrath, J.E.: Time, Interaction and Performance (TIP). A Theory of Groups. Small Group Research, 22, (1991) 147-174. 14. Dillenbourg, P. (ed.) (1999): Collaborative Learning. Cognitive and Computational Approaches. Elsevier Science Ltd. 1-19. 15. Kiesler, S. and Sproull, L.S. (Eds.): Computing and change on campus. New York: Cambridge Press (1987) 16. Daradoumis, T., Xhafa, F. and Marquès, J.M.: Exploring Interaction Behaviour and Performance of Online Collaborative Learning Teams, 9th Int. Workshop on Groupware, CRIWG'03, France. Lecture Notes in Computer Science, Vol. 2806, pp. 126-134 (2003) 17. Ellis, C., Gibbs, S., and Rein, G. Groupware: some issues and experiences. Communications of the ACM, 34,1, 9-28, 1991. 18. Lund, K. Human Support in CSCL : what, for whom, and by whom ? In J.-W. Strijbos, P. Kirscher & R. Martens (Eds.). What we know about CSCL in Higher Education. pp. 167 198. Dordrecht : Kluwer Academic Publishers, 2004. 19. Caballé S., Xhafa, F., Daradoumis, T. and Marquès, J.M. : Towards a Generic Platform for Developing CSCL Applications Using Grid Infrastructure. In: Proc. of the CLAG/CCGRID’04, Chicago, USA (2004) 20. Caballé, S. and Xhafa, F.: A Study into the Feasibility of Generic Programming for the Construction of Complex Software. 5th YRWS. In: Proc. of the GPCE/NODE'03 pp.441446, Erfurt, Germany (2003) 21. Inaba, A., Ikeda, M., & Mizoguchi, R.: What Learning Patterns are Effective for a Learner's Growth? - An ontological support for designing collaborative learning. In: Proceedings of the International Conference on Artificial Intelligence in Education (AIED2003), pp. 219-226, Sydney, Australia, July 20-24 ,2003

Progressive Streaming of Irregular Meshes Using Geometry Images Bin Sheng1 and Enhua Wu1,2 1

2

Faculty of Science and Technology, University of Macau, Macao, China [email protected] State Key Lab of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing, China [email protected], [email protected] Abstract. This paper presents a novel approach for view-dependent streaming of highly detailed and irregular meshes. In contrast to the conventional progressive streaming where the resolution of a model changes in the geometry space, our server firstly maps the 3D surfaces onto the parameter space in which the geometry images (GIM) and normal map atlas are obtained by regular re-sampling. By constructing the regular quadtree-based hierarchical representation based on GIM, we accomplish an efficient compression scheme to encode the hierarchical structure that minimize the network communication overhead to facilitate real time mesh update in the large-scale model visualization from clients. Furthermore, the encoded nodes can be transmitted in arbitrary order, so the extreme flexibility in transmission could be achieved. In particular, by using normal texture atlas, the rendering result of a partially transmitted model is improved greatly, therefore only the geometry on the silhouette to the current viewpoint requires to be refined and transmitted, which can minimize the amount of data needed to transfer each frame. Experimental results indicate that our approach is efficient enough for a broadcast scenario where one server streams the multiresolution GIM data to multiple clients with different viewpoints.

1

Introduction

As the web-services expands and the processing capabilities of computers improves, our demand is growing for the high resolution 3D models from games to commercial web sites in various distributed applications, which also poses a challenge to representing, transmitting and rendering such large-scale data sets. In the Internet environment, it is impractical in an interactive speed to download a large full model with complex details because of the limitation of the network bandwidth. Currently, the dominant strategy to deal with the problem is to simplify the model and transmit it progressively. The progressive approach allows a very coarse approximation of a model to be sent at first, followed by subsequent transmission of incremental geometry to add more and more details. K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 560–571, 2007. c Springer-Verlag Berlin Heidelberg 2007


Progressive Streaming of Irregular Meshes Using Geometry Images

561

This process allows the user to obtain an early visual perception of the geometry, and incrementally with detail up to user’s satisfaction. Streaming a triangle mesh (K, V ) must process two kinds of information: the topological connectivity K (adjacency graph of triangles) and the geometry V (positions of the vertices). To the triangle meshes with arbitrary topology, progressive meshes [1][2] are served as the framework of streaming [3][4][5]. Similar in spirit to progressive meshes, some commercial systems have been developed, such as Meta MTS products [6]. However, the complex topology is not amiable to the interactive transmission. On the contrary, the models with regular topological connectivity [7][8][9] lend themselves in an advantageous position over the arbitrary topology models in the expression and transmission [10]. The transmission of the geometry V is a heavy work, because each vertex requires a vector represented by 3 float-point data. The number of vertices can be reduced through mesh simplification, and the removed details on the surfaces can be visualized by using texture mapping, which can be accelerated by even the low-end graphics hardware. Normal mapping can add 3D shading detail to the surface, and can be accelerated by modern GPU. The details, such as color and normal vector on the surface can be stored compactly (only 1 to 3 bytes for each texel required), and can be converted into textures using surface parametrization [11][12]. In this paper, we propose an extremely flexible scheme to transmit the details and geometry progressively and interactively. We employ the surface parametrization to convert the arbitrary surfaces into geometry images (GIM)[7], and the details on the surface can be mapped onto the texture atlas. Since GIM is the completely regular representation of 3D surface in the parameter space, we construct a quadtree structure based on GIM, dubbed P-Quadtree. Since the details texture atlas is utilized to enhance realism of the partial transmitted models, the key to our approach is to refine and stream the silhouette geometry view-dependently. To transmit the P-Quadtree efficiently and flexibly, we encode and encapsulate it into a localized communication packet. The topological connectivity of the node is encoded as an integer number by its localization code in pointless quadtree structure [13], and the coordinates of each vertex can be encoded as a scalar float number by using the normal meshes [8]. Since all hierarchical and topological structure of the node is packed into a communication packet, the nodes can be transmitted in arbitrary order.

2

Overview

One of the possible applications of our view-dependent streaming framework would be a flight simulation game with multiple participants. The service company may want to attract customers by providing several new map data occasionally. However, with the typical downloading process, a customer cannot play the game until the whole terrain data have been received. In contrast , our view-dependent streaming technique allows any participant to join and play the

562

B. Sheng and E. Wu

game immediately without waiting for the download of the whole data, even when several terrain maps have been newly created. 2.1

Network Configuration

The network model of our view-dependent streaming framework consists of a server and multiple clients. The server has a database of several different meshes represented in the form of geometry images(i.e. a GIM atlas and a normal texture atlas), and we assume that the server is powerful enough to deal with requests from multiple clients. Each client downloads a mesh with different view points not only during the streaming process but also after downloading the entire mesh data. We assume that the network bandwidth of a down-link from the server to a client is much wider than that of an up-link in the reverse direction. 2.2

Our Approach

Our approach contains two stages: data preprocessing stage and data transmission stage. The transmission scheme of our approach includes two parts: data preparing on server and data receiving and 3D surface construction on the clients. On the server side, the meshes with arbitrary topological connections must be preprocessed, and the geometry and details on the surface should be converted into the regular representation in the parameter space: GIM atlas and normal texture atlas. The P-Quadtree structure is constructed based on the GIM atlas. When the transmission process begins, P-Quadtree can be streamed according to the requests with different viewpoints from the clients. The data of detailed surfaces are compressed and sent to the clients through the network. Once the data are received on the client, the nodes’ structure and vertices will be decoded from the data packages, and the structure of P-Quadtree and the surface could be reconstructed. This process is progressive and view-dependent. The normal mapping can be accelerated by modern graphics hardware, and thus the client re-quires only the refinement of the silhouette to reduce the data to be transmitted. By this approach we are able to design a server-client system for viewdependent progressive model streaming which has several important features: – Minimum Redundancy: Each node in the hierarchical P-Quadtree is sent from the server exactly once and only when the view-dependent rendering demand on the client side first requires it. On the other hand , the client sends requests to the sever only when it can not satisfy the criterion using the information that it received in the past. – Maximum Efficiency: At each moment in time the server sends exactly those GIM data that add the most to the visual quality on the client side. This reduces the bandwidth requirements by a factor of about 1.5 if the client displays the complete object from a fixed perspective. This factor decreases if the user changes the viewpoint during downloading, but it even increases if the client’s display only shows the partial image of the object.

Progressive Streaming of Irregular Meshes Using Geometry Images

563

– Minimum Server Load: The server only needs to store a comparably small amount of data for each client. Only a list of binary flags for the node status(transmitted/non-transmitted) is necessary. The server can use a dynamic structure that grows proportionally to the number of really transmitted vertices rather than requiring an amount of memory proportional to the finest level of detail. This reduction of the per-client costs on the server side makes our approach applicable to multiple clients scenarios where a server broadcast GIM data to a multitude of clients.

3

P-Quadtree and the Node Encoding

P-Quadtree is a multi-resolution representation of geometry image, whose nodes can be encoded efficiently by using the regular structure. 3.1

Geometry Images

Geometry image [7] is an ”image-like” representation of surfaces. It is constructed by converting the surfaces into a topological disk using a network of cuts and parameterizing the resulting disk onto a square domain using surface parameterizations, which is a mapping U: (x, y, z )→(u, v ) from 3D space to 2D space. It samples the surface geometry as a n×n array of (x, y, z ) values, where n is the size of the image. A pixel in the geometry image maps the 3D coordinates of a vertex on the surface in object space, and four neighbored pixels correspond to the vertices of a quadrilateral in the object space. Mapping the whole surface of the model onto a square domain will introduce high distortion, especially for the shapes with high genus or long extremities. To mitigate this, the surface can be cut into several pieces, i.e. charts, and many charts can be packed into multi-chart geometry images [14] to represent the complex model. Each chart has a natural boundary, and is parameterized onto irregular polygon. Such a parametrization atlas reduces the distortion, and distributes samples more uniformly over the surfaces and therefore better captures the surface signals. The boundary must be zippered carefully [14]. 3.2

The Structure Encoding of P-Quadtree

P-Quadtree construction. P-Quadtree is constructed based on GIM. It requires the size of square to be (2n +1)×(2n +1) to construct a quadtree, where n is the number of levels. To meet this requirement, the packed atlas is sampled by (2n +1) pixels on the longer side, and then we enlarge its shorter side to be (2n +1). The new pixels in the expanded area are assigned with null values. We construct P-Quadtree top-down. The process includes construction of quads hierarchy and computation of the quad attributes, such as node’s errors, radius of bounding sphere (Fig. 2(b)) and normal cone (Fig. 2(c)). These attributes are used in view-dependent data streaming.

564

B. Sheng and E. Wu

Node structure. A node in our algorithm is a 33 pixels block in image space, whose structure is shown in Fig. 2(a), and represents a quadrilateral patch in object space (as shown in Fig. 2(b)). A node has a center vertex and four vertices at the corners, called basic vertices (the gray dots in Fig. 2(a, b)), which are indispensable to construct the surface. When we render the surfaces using PQuadtree nodes, crack may appear on the edge whose adjacent patches have the different level. In order to prevent the crack, new vertices need to be added on the middle of these edges. We call these vertices as optional vertices (the white vertices in Fig. 1(a, b)). Only the basic vertices need to be transmitted. A

4

3

2

nc nc

5

1

α

C

0

6

7

(a) In parameter space

8

r (b) In object space

(c) The normal cone

Fig. 1. The structure of a P-Quadtree node

quadrilateral patch of the node can be subdivided recursively. From the view of topology, we could use a 2-tuple (K, V ) to represent it, where K is a simplicial complex, expressing the connectivity of vertices, edges and surfaces; V = {Vi ∈ R3 | i ≥5}, which gives the vertices of node or the quadrilateral patch. The attributes of a vertex may include the geometry coordinates (x, y, z ), the normal vector n, and the radius of the bounding sphere r. Since the shape of the chart is irregular polygon, it may contain null pixels. If all of its pixels are defined pixels, the node is called a valid node. On the contrary, if the valid pixels in a node could not map at least one valid triangle in object space, it is called invalid node. The others nodes are the gray nodes. The higher level gray nodes may be subdivided into valid nodes and invalid nodes. The position of a pixel in the image space of GIM is called the image position (u, v ) of the pixel. The image position of the pixel at left-bottom corner of a node is called image position (u0 , v0 ) of the node, and its size by pixels is called image size. We could reconstruct the topological connectivity of the patch of the node by its image position and the image size. Node encoding. In order to transmit the data of patches efficiently, we must encode the data of the node for compression. A pointless quadtree [13] is a linear array of nodes in quadtree, and the topological and hierarchical structure of a node could be determined by its localization in the array, or localization code. Therefore we could encode the structure of the node into an integer number by using the localization code of the nodes in the pointless P-Quadtree. This code

Progressive Streaming of Irregular Meshes Using Geometry Images

565

is the structure code of the node, which is noted as P. The encoding process is similar to converting a quinary code into a decimal number. Besides the topological structure, we need to transmit the geometry of the vertices on the quadrilateral patches (the nodes), which corresponds to the value of pixels in GIM. We note the pixels array as VA. Normal meshes could represent the regular mesh using one float-point data per vertex. The valid nodes on the most top of P-Quadtree would be served as the base meshes for subdivision, and thus the geometry image is transformed from ”color” image (the 3D coordinates of a vertex (x, y, z )) to a series of ”gray” images (a displacement in the normal direction that is determined by the base mesh). The 3D coordinates of a vertex can be encoded as a float-point number, called vertex code of the node. To facilitate the transmission, the structure code P and vertex codes array VA are encapsulated into a communication packet as a transmission unit of data stream. Apart from the 3D coordinates on the surface in object space, the image position of each vertex must be available when P-Quadtree and surfaces are constructed on the client. In fact, the image position a vertex in the node (u, v ) can be computed from the image position (u0 , v0 ) and the image size d of the node, which can be decoded from structure code P of the node. For example, the image coordinates of the center pixel can be computed as (u0 +d /2, v0 +d /2), and the other pixels could be computed in similar way. It seems unnecessary to transmit the image coordinates, if we transmit the vertices in the order of their relative position. However, the vertices of the node may be shared with its neighbors. To avoid transmitting the vertices redundantly, we mark the vertices transmitted, and remove them from the vertex array in the following communication packets. So the number of vertices in the vertex array is variable. To determine whether a vertex has been transmitted, we employ a control code to encode the image coordinates efficiently. Each vertex maps a bit flag in the control code. If the vertex is not transmitted, the flag is set to 0, otherwise to 1. The center vertex of the node must be transmitted, and thus only the four corners need to be controlled by a 4-bits control code. If m is the number of pixels of the GIM, and n is the number of nodes in P-Quadtree, the size of transmitting data for vertices would be no more than (32 m +(4/5 )n) < 33m.

4

Streaming and Reconstruction

After the P-Quadtree is prepared, we begin to stream the nodes according to the clients’ requests. The data stream can be transmitted through the network. The data streaming is generated on the server in real time, and the surfaces are reconstructed on the client progressively. 4.1

Data Streaming

The breadth-first tree-traversal could fit the data streaming well. Since the shape of charts could be arbitrary polygons, this may cause many gray nodes across the

566

B. Sheng and E. Wu

irregular boundary. The gray nodes must be subdivided until all the children are valid or null nodes. The valid nodes would be added to the data stream, while the null nodes are discarded. Initial nodes Streaming. To encode the 3D coordinates of vertices, the base meshes are necessary to construct the normal meshes. We firstly traverse the quadtree from the root node to get the valid nodes at the most top level.The obtained nodes are the initial nodes for transmission, which also serve as the base meshes of normal meshes. Detail texture transmitting. The normal texture atlas could be utilized to improve the visual appearance of the partial transmitted surfaces. The texture can be transmitted progressively. since the texture coordinates of the texture atlas are implicit, it is unnecessary to transmit them. The geometry data stream and normal map could be transmitted interactively in our approach. Silhouette nodes streaming. The quality of the rendering image on the silhouette is still poor after normal details texture is mapped. The nodes on the silhouette would be subdivided and the information of sub-nodes be sent to the client to enhance the silhouette of the object. To transmit the nodes near the silhouette in the highest priority, the view-dependently continuous LOD technique could be employed to refine the surfaces on the silhouette. We perform the silhouette test using the algorithms in [15]. The nodes that pass the silhouette test are called the silhouette nodes. They are subdivided and packed into the transmitting stream with the highest priority. The front-face nodes toward the viewer, which could be enhanced by normal mapping, and are assigned with the mediate priority. Back-face nodes are culled in current viewpoint. In this way, our transition scheme has a limited data for each viewpoint. However, when the bandwidth is not enough to transmit a complex silhouette details in real time, we need a query procedure ncheck() to test whether the vertex should be subdivided or not, according to the view-dependent refinement criteria. In our framework, we use the criteria proposed by Hoppe, which tests the viewing furtum, the cone of normals, and the screen-space error. In this way,we can flexibly control the subdivision level and the data size of packets to facilitate the requirement of the interactive transmission. 4.2

Memory Optimization

If we store the silhouette nodes in each session on the server side, the most significant size of the data structure on the server may come from the from the silhouette nodes. The maximum length of silhouette nodes is in the same order of magnitude as the number of nodes in the original P-quadtree. When the server has several sessions with different clients to visualize very large mesh, memory requirement for the silhouette nodes can be a serious problem on the server side. To minimize the memory overhead, we discard the the silhouette nodes data structure and just use the active tags to represent which nodes in the P-quadtree

Progressive Streaming of Irregular Meshes Using Geometry Images

567

are contained in the silhouette area. Then we can arrange the P-quadtree hierarchical structure in the partial order that the parent indices should be smaller that their children’s indices. Then, the active tags for the nodes in the P-quadtree hierarchy can be represented by a bit array. To determine the the silhouette packets to be transmitted for the server, we sequentially visit the elements in the active tag array. If the current bit is ’1’, the ncheck() procedure is evaluated to determine if the corresponding not should be subdivided into its children. Since the indices of newly active are greater than the one of the currently visited node, this sequential search would not miss any vertices that must be considered in determining the silhouette nodes.

Fig. 2. Window active flag for the P-Quadtree hierarchy

This procedure can be further accelerated with a windowing technique. the idea of our windowing technique is similar in spirit to [16] which handles very large sized data by concentrating the computation or the storage only on the currently active parts. Since the P-quadtree hierarchy has been stored in an array is a linear data structure, as shown in Fig.˜reffig:slidingwin. The left part of winlef t and the right part of winright are all zero bits , and the in-between part contains ’0’ and ’1’ bits . Note that the window only grows from the left to right direction because the partial order among the nodes in the P-quadtree is preserved in the array. While the window is moved from the left to the right by performing subdivision operations, the left and right parts of the window will be deallocated and allocated, respectively. Although there remain serval ’0’ bits within the window, we can reduce the traversing time for the silhouette nodes by confining the search range from winlef t to winright . In our experiments, since we pack the P-quadtree in a bread-first manner, the windowing technique is sufficiently fast and uses much less memory than storing a vertex front itself.

568

4.3

B. Sheng and E. Wu

Surface Reconstruction and Rendering

When a communication packet in the data stream is received on the client, the corresponding node’s structure could be reconstructed by package decoding,as shown in Fig. 3. Since the transmitted nodes are in a sequential structure, we must reconstruct their hierarchical structure firstly. By decoding the structure code, we could reconstruct the topological connectivity and hierarchical structure; the vertices or pixels of the geometry image can be obtained from decoding of the vertex codes. The algorithm to decode the structure code is similar to converting a decimal digit into a quinary code, which is the reversed encoding process mentioned in Section 3.2.

Fig. 3. Surface Reconstruction and Rendering

After the data are decoded, the quadtree on the client can be re-constructed. We firstly traverse the quadtree from the root node to locate the position of the coming node in the quadtree according to the decoded quinary digits, and then the point of this position in quadtree is assigned to this node. Once the quadtree has been reconstructed, we could select the appropriate nodes to reconstruct the surface by using the quadtree traversal. One problem during rendering is that crack may come up when two adjacent nodes have different levels. Since the vertices are transmitted progressively, we check the transmitted vertices on the edge, and then triangulate the patch to prevent from the possible crack. The geometry image can be served as flags map of the received pixels. Once a vertex code is decoded, it is assigned to the corresponding pixel. A node can be decomposed as a triangle fan. Once the rendering triangles are pre-pared, the data are sent to the rendering pipeline. The normal mapping in our approach can be implemented on GPU by using the per-pixel lighting technique.

Progressive Streaming of Irregular Meshes Using Geometry Images

5

569

Results and Discussion

We investigate the performance of the view-dependent transmission method, when the target mesh is fixed. Fig. 4 shows the rendered images of the reconstructed ’Venus’ models at various bitrates. The upper and lower images are obtained using the the traditional progressive streaming and our view-dependent transmission approach, respectively. We see that our view-dependent transmission saves more than 50% the bitrates while providing similar visual quality by allocating a main portion of the transmission bandwidth to normal map and silhouette nodes. We can further illustrate the effect of our interactive transmission in Fig. 5, where (a) is the mesh of the initial nodes, and (b) is the result of its geometry rendering. Once the normal texture atlas is applied to the initial meshes, the visual effect improved greatly(see Fig. 5(c)). However, if the viewpoint is changed, the coarse silhouette is shown (see Fig. 5(d)). When the requesting silhouette nodes are transmitted, the silhouette is refined (Fig. 5(e)).

Fig. 4. Comparison of the Reconstructed ’Venus’ models with traditional progressive streaming method and our approach

The main advantage of our transmission scheme is the interactivity and flexibility. The nodes can be transmitted in arbitrary order, because only the nodes near the silhouette need to be refined. Since the normal texture atlas can enhance the realism of rendering result, the data require to be transmitted is minimal.Table 1 shows the number of the silhouette nodes per frame. Due to the regular structure of P-Quadtree, our streaming implementation has lower run-time overhead on the server, whose performance results are shown in Table 1. All of the test cases run on Windows XP system with 2.6GHz Pentium 4 CPU, nVidia GeForce 6800 GPU and 1G RAM.

570

B. Sheng and E. Wu

(a)

(b)

(c)

(d)

(e)

Fig. 5. The interactive transmission of ”Vase”

Comparing with the streaming schemes based on progressive meshes [3][4][5], the polygons number of the initial mesh in our approach is about thousands(see Table 1). However, our approach has some advantages over them. Firstly, more features can be shown and visual effect is much more better, and the transmitting rate of modern network can be enough to download meshes with thousands triangles quickly. Secondly, although the normal map atlas may be used to enhance the realism of the rendering images in other approaches, the texture coordinates are implicit in our approach because of the ”texture-like” structure of GIM. Lastly, the node encoding in our approach is more compact and flexible due to the regular structure of P-Quadtree. Table 1. Statistics of P-Quadtree streaming Model Igea Vase Bunny # Charts 2 2 9 # Total nodes 96,586 38,869 126,684 # Initial nodes 2156 1458 5986 # Silhouette nodes 1,124 438 1,897 Streaming rate for the P-Quadtree on server(fps) 45 96 31

6

Conclusions

We have demonstrated a new interactive transmission approach to highly detailed surfaces. Our approach combines the normal mapping and view-dependent continuous LOD technique. Because only the nodes near the silhouette need to be refined progressively and interactively, only a few nodes are required to be transmitted each viewpoint. Since an efficient encoding scheme is used to compress the geometry data, the transmitting efficiency and the visual effect are improved. We believe that there is much room for improving our view-dependent streaming framework. The potential extension includes the compression of the node

Progressive Streaming of Irregular Meshes Using Geometry Images

571

encoding information and view-dependent silhouette node information. Finally, it will be piratically useful to utilize the session idle time with a smart policy for packet transmission. It would be also an important future work to measure the real loads for an experimental comparison with several view-depend streaming frameworks.

References 1. Hoppe, H.: Progressive meshes. In: Proceedings of SIGGRAPH 1996. (1996) 99– 108 2. Hoppe, H.: View-dependent refinement of progressive meshes. In: Proceedings of SIGGRAPH 1997. (1997) 189–198 3. Pajarola, R., Rossignac, J.: Compressed progressive meshes. IEEE Transactions on Visualization and Computer Graphics 6(1) (2000) 43–49 4. Prince, C.: Progressive meshes for large models of arbitrary topology. Master’s thesis, University of Washington (2000) 5. Kim, J., Lee, S., Kobbelt, L.: View-dependent streaming of progressive meshes. In: Proceedings of Shape Modeling International 2004. (2004) 209–220 6. Abadjev, V., del Rosario, M., Lebedev, A., Migdal, A., Paskhaver, V.: Metastream. In: Proceedings of VRML 1999. (1999) 7. Gu, X., Gortler, S.J., Hoppe, H.: Geometry images. In: Proceedings of ACM SIGGRAPH 2002. (2002) 355–361 8. Guskov, I., Vidimce, K., Sweldens, W., Schr¨ oder, P.: Normal meshes. In: Proceedings of ACM SIGGRAPH 2000. (2000) 95–102 9. Lee, A., Moreton, H., Hoppe, H.: Displaced subdivision surfaces. In: Proceedings of SIGGRAPH 2000. (2000) 85–94 10. Labsik, U., L.Kobbelt, Schneider, R., H.-P.Seidel: Progressive transmission of subdivision surfaces. Computational Geometry 15 (2000) 25–39 11. M. Tarini, P. Cignoni, C.R., Scopigno, R.: Real time, accurate, multi-featured rendering of bump mapped surfaces. Computer Graphics Forum 19(3) (2000) 119–130 12. Sander, P., Snyder, J., Gortler, S., Hoppe, H.: Texture mapping progressive meshes. In: Proceedings of SIGGRAPH 2001. (2001) 409–416 13. Samet, H.: The quadtree and related hierarchical data structures. ACM Computting Surveys 16(2) (2000) 187–260 14. Sander, P., Wood, Z., Gortler, S., Snyder, J., Hoppe, H.: Multi-chart geometry images. In: Proceedings of Eurographics Symposium on Geometry Processing 2003. (2003) 146–155 15. Luebke, D., Erikson, C.: View-dependent simplification of arbitrary polygonal environments. In: Proceedings of SIGGRAPH 1997. (1997) 199–208 16. Kim, J., Lee, S., Kobbelt, L.: View-dependent streaming of progressive meshes. In: SMI ’04: Proceedings of the Shape Modeling International 2004 (SMI’04), IEEE Computer Society (2004) 209–220

A QoS Controller for Adaptive Streaming of 3D Triangular Scenes ZhiQuan Cheng and ShiYao Jin PDL Lab., National University of Defense Technology Changsha City, Hunan Province, P.R. China [email protected], [email protected]

Abstract. With the development of network, it’s a common requirement to perceptually access the 3D scenes at a remote server. This demand has not been filled by software standards and their implementations, leaving a wide potential for technical innovations. In the paper, we present a QoS (Quality of Service) controller for remote rendering of 3D contents, aiming at higher real-time performance just as rendering local 3D scenes. The model of our QoS controller is defined by a high-level heuristic function, which has integrated important perceptual factors. And then, guided by the QoS controller, a hybrid transcoding algorithm is developed, which adaptively partitions each 3D model into different modality: the metadata (its compressed base mesh with texture) and a series of refiner meshes. Consequently, the 3D scenes can be adaptively and progressively transmitted to a requesting client, and be fluently displayed at end user’s window in a constant frame rate. Experimental results show the soundness of the proposed controller and algorithm guided by the 3D controller. Keywords: Remote Rendering, Streaming, 3D QoS, VRML, and Adaptation.

1 Introduction Remote visualization of large, dynamic and interactive 3D scenes has been a very important research field since the advent of digital geometry processing technologies. However， this demand has not been filled by software standards and their implementations, leaving a wide potential for technical innovations. Advances in streaming and signal processing technology are necessary to enrich the perception and performance of 3D objects in distributed applications [1,2], e.g. in collaborative environments or applications for remote rendering digital geometry museum. Especially, one important requirement of these applications is that the interactive frame rate should be high enough and should be constant in order to give the user a smooth navigation in the virtual environment. In our previous work [3], we have proposed the Adaptive Progressive Remote Rendering Architecture (APRR) framework, a service-oriented architecture, for progressive delivery and adaptive rendering of 3D contents. Based on the ideas and lessons learned during the previous development, we find that we should and must successfully take human perception factors into adaptive streaming of 3D models. In this paper, a 3D QoS controller is presented in the client/server framework, and K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 572–583, 2007. © Springer-Verlag Berlin Heidelberg 2007

A QoS Controller for Adaptive Streaming of 3D Triangular Scenes

573

integrated in both client and server side. The controller guarantees the server to adaptively transmit the best quality of multi-resolution 3D objects in such a way that a minimal overall quality degradation over the scene is obtained. While at the end-user system, the user terminal is refreshed at a constant specific frame rate, by making speed-accuracy trading-off under resource constraints, based on the QoS model. Besides these, we also propose a hybrid transcoding for adaptive transmission of a triangular mesh, which combine the metadata, its basic mesh with a mapping texture, and a series of optimal refined hierarchical levels of detail. In the following paragraphs, we will articulate the related works in section 2. By examining the graphics pipeline in remote rendering of 3D object, section 3 identifies the location of 3D QoS controller, and then describes the QoS framework including a QoS architecture, 3D QoS model and its heuristic function. Section 4 presents a hybrid coding method for adaptive streaming of 3D scenes, and explains how the QoS controller avails to the execution of the streaming algorithm. Experimental results and discussion are given in section 5. Finally, we make a conclusion in section 6.

2 Related Work Remote rendering of 3D models is a hot and important research area, and lots of researches have tackled the problem. However, implementing an adaptive system for delivering 3D models over networks is not trivial, effective researches should be investigated: New pioneering designs and solutions for coding, compression and streaming/broadcasting technologies for 3D models and multimedia data as well as technologies and applications for distribution frameworks, networking technologies for efficient and robust data communication mechanisms on current and next generation networks need to be developed. On the one hand, Hesina and Schmalstieg [4] advocated the distributed framework of remote rendering for the first time. Afterwards, many infrastructures have been development from the reasonable design and development of a logical structure [3,5,6] to the complex scene organization and management at the server [7,8,9]. Especially, the methods based on VRML/X3D has been thoroughly focused by presenting various coding techniques [10,11,12,13,14,15]. On the other hand, 3D compression [16] has been a perdurable branch of data compression, aiming at the compact representation of 3D models, and a lot of excellent progressive algorithms [17] have been proposed. Existing 3D compression algorithms use both techniques adapted from the 1D and 2D cases (like wavelets, entropy coding, and predictive coding), and completely different approaches that take advantage of the properties of 3D surfaces (like vertex degree [18], geometry image [19], and subdivision surfaces [20]). For a more complete survey of techniques in 3D mesh compression, readers are referred to recent papers [21,22]. In addition to the points already discussed, several aspects outside the graphics domain should be further researched. Firstly, at the networking level, error resilience [23,24,25] is important if lossy transmission protocols like User Datagram Protocol (UDP) are used. Secondly, security [26] is paramount to the integration of the adaptive system into the real world, as it constitutes a major concern for all networked graphics applications. Thirdly, a more elaborate pre-processing [27] scheme including

574

Z. Cheng and S. Jin

a more sophisticated perceptual model that accounts for inter component dependencies and methods for seamless integration of different representations is necessary. Finally, statistical representation [28,29] of the geometry models has also been researched for efficient transmission and rendering. In general， each component is independently considered in the previous, which may lead to combinations of different representations in a single image. However, in the paper, we attempt at integrating the perceptual adaptation and real-time preservation function into the progressive framework by a 3D QoS controller.

3 QoS Framework 3.1 Graphics in a Networked World Figure 1 shows a simple abstraction of the graphics pipeline, which includes the phases in the remote rendering. The rendering pipeline describes the way that geometric data takes from modeling to the final image. In this theoretical conception, the local data, held in the main memory of the local client site, does the actual rendering, while the full geometry database is held at the remote server site.

Fig. 1. The abstract graphics pipeline of remote rendering of 3D models

1) Geometry Modeling. This stage is independent to later actual rendering method, and can be performed off-line. Note that adaptive streaming mechanism must be pre-computed to be able to perform level of detail rendering in later stages. 2) Global Scene Traversal and Simple Chosen. The stage of the pipeline includes two jobs. One is traversal of the display scene or database in immediate or retained mode, the other is trival accept/rejection classification of a 3D model by identifying whether it is wholly inside or outside of the view volume of a client.

A QoS Controller for Adaptive Streaming of 3D Triangular Scenes

575

3) The Net. The networking stage transmits the required data to the user client site, based on the demand request for particular pieces of geometric data. 4) Local Scene. The local geometry cache holds local part/whole copy of geometric data from the global geometry database. The main challenge is to stream its content at all times appropriate to the portion of the virtual environment visible to the observer. All the techniques described in this paper aim at improving performance so that this problem can be greatly alleviated. 5) Local Rendering. The rendering process that maps a model to an image on screen is referred to traditional graphics pipeline [30]: local scene traversal, polygon processing, and pixel processing. Each primitive encountered during traversal is passed through the remainder of the local rendering pipeline. The polygon processing does sequent vertex transformations to each primitive, which include model transformation, lighting calculation, viewing transformation and clipping. And the pixel processing is also called rasterization, which is composed of three subprocesses: visible-surface determination, scan conversion and shading. To optimize the stages of the remote rendering pipeline, some knowledge discussed earlier are exploited in the paper. In the infrastructure layer, the APRR framework is made more efficient and real-time performance by adding perceptual adaptive function based on a 3D QoS controller, discussed later in the section. And in the data layer, a transcoding algorithm is proposed in section 4, to adaptively transmission of the progressive meshes. 3.2 APRR with QoS Controller APRR is a service-oriented architecture for 3D content delivery and client rendering, based on three-tier framework. And its prototype has been designed and implemented for network streaming and remote interaction of complex 3D battlefield environment. To enhance the real-time interactive function of the APRR, we introduce a new 3D Quality of Service (QoS) controller in the APRR (Fig. 2) to make speed-accuracy trading-off under resource constraints. Consequently, the display on the client terminal can be rendered in constant frame rate, guided by QoS function that is a perceptual demanddriven geometry transmission strategy. In order to provide QoS at the end user terminal, the middle service tier should negotiate a QoS contract with the network and with the user client. In the following, we would focus on the QoS controller, since the others have been clearly explained in the original APRR [3]. At the server side, the relevant model becomes an adaptive progressive model, after which passes through adaptation generation and management service modular. Similar to [6], the server QoS controller estimates its performance characteristics that are synthesized by T, Q, and I parameters required by each client, where T is the estimated time that APRR deliver model from the server to the client, Q is computed based on the later QoS model and defines the quality how closely a rendering of this representation approximates the rendering of the full-resolution data for each client, and I is the level of interaction. The adaptive management mechanism, guided by the QoS controller, selects the best continuous level available for each model, which fits into the budget, and progressively delivers them to the client in binary format.

576

Z. Cheng and S. Jin

User Interface Graphics Engine Rendering Visible Geomtry View Frustum & Occlusion Culling QoS Adaptation Controller Re-selection

Global Scene Retieval

QoS Controller

Adaptation Generation

Global 3D Scene DB

Adaptation Management

EIS Tier

Network

Data Sets

User Interface Graphics Engine Rendering Visible Geomtry & View Culling

Scene Traversal

Receiver binary

Sender Receiver

Delivery Mechanism

Middle Service Tier

Local Scene Client

Fig. 2. APRR logical architecture with QoS controller: Each color represents an independent “model-view-controller” design pattern modular. The orange points out view modular, the blue denotes on model modular, and green presents controller modular. And the new appended QoS controller and its control information are saliently highlighted in the red color, although they should be masked by the green.

At the client side, the user interface allows the user to specify a desired interactive frame rate and to interact with the 3D scene. The QoS controller makes use of the resource estimator to estimate the processing time for each object based on high-level QoS parameters of the object and platform specific parameters. And then, The function of the QoS controller is to maintain a constant specified interactive frame rate by executing the control loop of Fig. 3, i.e., selecting for each viewpoint the best quality of visible object portions, taking the constraints on the resources into account. Viewpoint Changes ?

Yes

Visibility culling

Benefit/cost update

Rendering

Quality selection

No

Fig. 3. Constant frame rate control at the client

From the before discussion, it’s obvious that the 3D QoS controllers in the client and server has the same structure and QoS model, except that they have different resource constraints. 3.3 QoS Model In order to estimate accurately the streaming level L of each object O under the given resource constraints C(u) for a client user U, it is important to have an accurate 3D QoS model at the viewpoint V. Therefore, we adapt Ngoc et. al.’s method [5] by defining the QoS model as follows: Q(O, L,V , U ) = β (O) * S (O ) *cos(α ) * PSNR(O, L,V )

A QoS Controller for Adaptive Streaming of 3D Triangular Scenes

577

Where Q(O,L,V,U) is the quality of object O, rendered at streaming level L and viewpoint V, which represents the amount of perception contribution of the object to the overall user’s scene. It depends on the size of the object S(O) (in % screen coverage), the sematic meaning β (O) of the object in the scene, the viewing angle α from the user viewpoint to the object, and the quality metric PSNR(O,L,V) approximately computed as the optimization algorithm in [5]. Note that β (O) is the only viewpoint independent parameter. The other factors have to be recalculated whenever the user view volume changes.

4 Adaptive Streaming of 3D Meshes 4.1 A Hybrid Transcoding Algorithm

Generally, the idea of adaptive progressive graphics is to include such methods into a single system that allows optimal combinations to be selected and applied, depending on the specific requirement of each client user. Once each client’s information has been obtained and quantified, the server can accommodate multiple representations of the same model and select optimal ones with respect to given limitation. And then, the challenge lies in extracting the most meaningful part of the data for the client that cannot afford to receive it all. For a 3D model, we propose a hybrid transcoding algorithm to considerably improve the quality and efficiency of its transmission in distributed graphics environments. Enlighten by the idea of texture mapping progressive meshes [19], the hybrid method partitions the compressed model into parts: a series of refiner meshes, and its metadata that is the compressed basic mesh with texture. The single texture of the metadata can be used to texture all approximation in a progressive mesh sequence. z The connectivity and geometry properties of the model are compressed by our earlier anchor-based valence-driven progressive algorithm [31], which has improved the original progressive degree-driven compression algorithm [18]. To get the metadata, we adapt our modified algorithm [31] in two ways. Firstly, to preserve the conformance with the texture-mapping, a set of anchors, locating nearest to the salient texture-mapping position, are selected from the input mesh surface vertices in a feature-sensitive fashion and preserved in the base mesh. Note that adjacent faces on the surface are generally not adjacent in the texture, which is first observed by Soucy et. al. [33]. Therefore, if we don’t use the anchors to record the main texture mapping positions, we cannot use this texture map anymore. Secondly, a multi-granularity quantization method in geometry encoding is presented in [31], which allows making better of the dynamic range (different number of bits in normal/tangential components) than with a fixed-bit representation. Consequently, we achieve the basic mesh of the model and a continuous set of approximating meshes. z The Texture coordinate mapping and value is encoded and decoded in the way, described as Isenburg and Snoeyink [32].

578

Z. Cheng and S. Jin

Now we can turn on the metadata of the model. At the server, if the Oriented Bounding Box (OBB) of the model is touched by the view volume of a client user, it would be directly accepted and downloaded in a whole. And Figure 4 has shown that it faithfully captures the detail at all levels of the streaming mesh sequence. Markv warship

M1a1 tank

58 triangles

40 triangles

35 triangles

140 triangles

100 triangles

60 triangles

354 triangles

279 triangles

156 triangles

700 triangles

550 triangles

455 triangles

1197 triangles

1969 triangles

2064 triangles

Metadata

B52 plane

Level 1 Level 2 Level 3 Level 4 Fig. 4. Instances of hybrid trancoding model with its metadata and some levels

4.2 The Adaptive Transmission Mechanism

In order to avoid data redundancy between the similar models of a 3Dscene, we treat the similar models, which are identical except with different position, orientation, scale, and material properties, as one element. In the opinion of mesh compression field, the connectivity and geometry is the main shape description, and the others are appearance attributes: texture, color, and etc.. At the server, we use hierarchical element tree to organize the scene (Fig. 5), just like the common modeling tools, i.e.,

A QoS Controller for Adaptive Streaming of 3D Triangular Scenes

579

3D Max, Open Inventor. However, we take the idea of element graph [7], the nodes of an element do not directly contain the data (except the group node), but the information is stored in several pools and the nodes are only reference the entries of the pools. The main difference, between our hierarchical element tree with the element graph [7], is that the structure and function of the pools, since it is more reasonable to organize the pools according to the standard compression classification. 3D Scenes Element E1

Element En Group Node

Group Node

Texture Coordinate Node

Vertex Array Node

Indexed Triangle Node

Geometry Pool

Topology Pool

Y

Z

Z

Level 1

X

Y

Basic Mesh

X

I0

I1

Indexed Texture-coord Node

Texture Node Texture Pool *.* images

I2

Material Node

Material Pool ambient color diffuse color

I0

I1

I2

specular color

Transform Node Transform Pool

emissive color

Position Orientation Scale

Sub-group Node

Fig. 5. An example of a hierarchical element tree, which can have sub-group node. The Vertex Array Node and Texture Coordinate Node reference the geometrical information, the Indexed Triangle Node and Indexed Texture-coord Node reference the topological field, the transform node contains a set of transformation matrixes for similar models, and the others (excluding the Sub-group Node) are appearance attributes.

By far, to each requesting client, the server can transmit the metadata and optional levels of detail. Such method has the advantage that it can provide the geometric complexity of the data being transmitted in the optimal streaming way. However, determining whether a level of a model should be delivered to the client is not a trivial task and is typically left to the server’s QoS controller. According to the 3D QoS controller, the server selects the elements, which have to be transmitted to a specific client. If an element already exists on the client, then it is ignored. Each selected element gets a priority, which is calculated based on the QoS model. As mentioned before, the pools contain the complete visual information of the 3D model. With the help of the selected elements’ “hierarchical element tree”, the server is able to identify the pool entries, which are shared by multiple client users. At first, the primary data of the elements, i.e., the ID and metadata, are transmitted to the appropriate clients. And then, the optimal progressive levels in the nodes are sent and correspondingly marked. In the meantime, each client creates a similar “hierarchical

580

Z. Cheng and S. Jin

element tree” structure, and identify the pool entries that are expected progressive data, with the help of the 3D QoS controller. Consequently, the selection and progressive transmission of the information according to user’s interests are realized in the improved APRR framework, guided by the 3D QoS Controller.

5 Results and Discussion The prototype of the 3D QoS framework has been designed and implemented in a local area network with 10Mbps bandwidth, based on the old APRR architecture[3]. The server is running on one personal computer with 3.0GHz CPU, 2GB DDR memory, and 256MB graphics adaptor, while the four clients PCs have lower hardware configuration with 2.4 GHz Intel Pentium CPU. To guarantee lossless delivery of data the packets will be delivered in the same order in which they were sent, we adopt the prototype based on TCP protocol. To evaluate our implementation and compare with the old APRR prototype, we test the rendering performance of the same synthetic battlefield environment, including a number of military weapon 3D models and a 10x10km terrain area with some building constructions. In the typical application, for the four clients, we compare the two type parameters: startup time and average bitrate in the startup process. The startup time counts the latency time that one client terminal reconstructs and renders the primary scene by the metadata and the progressive levels of the streaming models, and the average receiving bitrate is evaluated by the average transmitting bitrate in the startup process. The table 1 shows the corresponding values of the parameters, and it’s easy to observe that the startup times in this test case are reduced about 20% under the mostly similar average transmitting bitrates. Table 1. Performance comparison Original APRR Client1 Client2 Client3 Client4

startup time (sec.) average bitrate (kbps) startup time (sec.) average bitrate (kbps) startup time (sec.) average bitrate (kbps) startup time (sec.) average bitrate (kbps)

120 452 148 398 85 432 198 535

APRR with QoS 89 466 124 412 71 445 161 517

To test the effect of trading off quality for frame rate on the visual quality of the whole scene, we kept the frame rate fixed and took the snapshot of the scene at several viewpoints in the client 1 (Fig. 6.a and 6.b) and client 2 (Fig. 6.c and 6.d). We then compared the quality of the scene with and without adaptive frame rate. Figure 6 shows the snapshots of the scene at two different viewpoints when the scene is rendered at the maximum quality (Fig. 6.a and 6.c) and when it is rendered at a fixed target frame rate of 30 fps (Fig. 6.b and 6.d). In Fig. 6.b and 6.d, we can see that

A QoS Controller for Adaptive Streaming of 3D Triangular Scenes

(a) 13fps

(c) 17fps

581

(b) 30fps

(d) 30fps

Fig. 6. The quality and frame rate trade-off

degradation is easily observed at the red-ellipse marked areas. Obviously, by applying an almost un-evident quality degradation, we obtain a fixed interactive frame rate. This fact proves the appropriateness of our QoS controller.

6 Conclusion We have presented a QoS framework that enables constant frame rate in the client terminals. In this paper, we have made two main contributions. By analyzing the performance of the remote rendering pipeline for 3D models, we have proposed a QoS architecture for 3D QoS and identified clearly the QoS specification and management of the overall QoS framework. In addition, we have proposed a fast hybrid transcoding algorithm for adaptive streaming of a 3D triangular object, which progressively partitions the object into parts: a series of refiner meshes, and its metadata that is its compressed basic mesh with texture. And the algorithm has been applied in the 3D QoS framework to create the hierarchical element tree of the scene at server, guided by the 3D QoS Controller. One prototype, about the virtual military battlefield environment, has been designed and implemented in the networked PC platforms. The experimental results have proved the soundness of our framework and algorithm guided by the 3D QoS controller. Most of the ideas in this article are emerged during the improvement of the original APRR [3] system. I have presented them in this article with the goals of emphasizing the advantages of adaptive delivery of 3D models and pointing out some of the challenges involved. Although our experiments have led us to solutions to some of the problems, many issues remain open, i.e., out-of-core streaming of large-scale

582

Z. Cheng and S. Jin

environments. With increasing bandwidth, some of the issues in this article may be alleviated but not entirely solved. At any given moment, more information is generated than any computer can handle. Intelligent transcoders will be necessary to exchange it over networks, visualize it, and ultimately, make sense of it. Acknowledgments. We are grateful to Isenberg Martin for the provision of demonstration code of their related paper. This work is supported by National Nature and Science Fund of Hunan province in china (No. 05JJ30124).

References 1. C. O'Sullivan, S. Howlett, R. McDonnell, Y. Morvan, and O. Keith. Perceptually Adaptive Graphics. In Proceedings of Eurographics STAR, 2004. 2. I. B. Martin, and T. J. Watson. Adaptive Graphics. IEEE Computer Graphics and Applications, 2(1): 6-10, 2003. 3. Z.Q. Cheng, S.Y. Jin, G. Dang, T. Yang, and T. Wu. A service-oriented architecture for progressive delivery and adaptive rendering of 3D content. In Proceedings of VSMM， LNCS4270， Xi’an, China, pages 110-118, 2006. 4. G. Hesina, and D. Schmalstieg. A Network Architecture for Remote Rendering. Technical report, TR-186-2-98-02, Institute of Computer Graphics and Algorithms, Visualization and Animation Group, Vienna University of Technology, 1998. 5. N. P. Ngoc, W. van Raemdonck, G. Lafruit, G. Deconinck, and R. Lauwereins. A QoS Framework for interactive 3D Applications. In Proceedings of the 10th International Conference on Computer Graphics and Visualization, pages 317-324, 2002. 6. I. M. Martin. Hybrid Transcoding for Adaptive Transmission of 3D Content. In Proceedings of IEEE Int’l Conf. Multimedia and Expo, pages 373-376, 2002. 7. J. Sahm, Ingo Soetebier, and H. Birthelmer. Efficient Representation and Streaming of 3Dscenes. Computers & Graphics, 28(1):15-24, 2004. 8. M Guthe, and R. Klein. Streaming HLODs:an out-of-core viewer for network visualization of huge polygon models. Computers & Graphics ,28(1):43-50, 2004. 9. M. Isenburg, Y. Liu, J. Shewchuk, and Jack Snoeyink. Streaming computation of Delaunay triangulations. ACM Transactions on Graphics, 25(3):1049-1056, 2006. 10. S. Olbrich, and H. Pralle. High-Performance Online Presentation of Complex 3D Scenes. In Proceedings of the IFIP TC-6 Eighth International Conference on High Performance Networking, pages 471-484, 1998. 11. A. Guéziec, G. Taubin, B. Horn, F. Lazarus. A Framework for Streaming Geometry in VRML. IEEE Transactions on Computer Graphics and Application, 19(2):68-78, 1999. 12. B.-Y. Chen, and T. Nishita. The Development of 3D Graphics and VRML Libraries for Web3D Platform by Using Java. Systems and Computers in Japan, 34(10):47-55, 2003. 13. M. Isenburg, and J. Snoeyink. Binary Compression Rates for ASCII Formats. In Proceedings of Web3D, pages 173-178, 2003. 14. E. Fogel, D. Cohen-Or, R. Ironi, and T. Zvi. A Web Architecture for progressive delivery of 3D content. In Proceedings of Web3D, Monterey, California, USA, pages 35-41, 2004. 15. J. Jessl, M. Bertram, and Hans Hagen. Web-based progressive geometry transmission using subdivision-surface wavelets. In Proceedings of Web3D, pages 29-35, 2005. 16. M． Deering. Geometry compression. In Proceedings of SIGGRAPH, Los Angeles, CA , USA, pages 13-20, 1995.

A QoS Controller for Adaptive Streaming of 3D Triangular Scenes

583

17. H. Hoppe. Progressive Meshes. In Proceedings of SIGGRAPH, New Orleans, LA, USA, pages 99-108, 1996. 18. P. Alliez, and M. Desbrun. Progressive encoding for lossless transmission of triangle meshes. In Proceedings of SIGGRAPH, Los Angeles, CA, USA, pages 198-205, 2001. 19. P. Sander, J. Snyder, S. Gortler, and H. Hoppe. Texture Mapping Progressive Meshes. In Proceedings of SIGGRAPH, Los Angeles, CA, USA, pages 409-416, 2001. 20. I. Guskov, K. Vidimce, W. Sweldens, and P. Schroeder. Normal meshes. In Proceedings of SIGGRAPH, New Orleans, USA, pages 95-102, 2000. 21. P. Alliez, and C. Gotsman. Recent Advances in Compression of 3D Meshes. Advances in Multiresolution for Geometric Modelling, 3-26, 2005. 22. J. Peng, C.-S. Kim, and C.C.J. Kuo. Technologies for 3D Mesh Compression: A Survey. Journal of Visual Communication and Image Representation, 16(6): 688-733, 2005. 23. G. Al-Regib, and Y. Altunbasak. 3TP: 3-D Models Transport Protocol. In Proceedings of Web3D, Monterey, California, USA, pages 155-163, 2004. 24. G. Al-Regib, Y. Altunbasak, and J. Rossignac. Error-Resilient Transmission of 3D Models. ACM Transactions on Graphics, 24(2):182-208, 2005. 25. A. Lall, V. Sekar, M. Ogihara, X. U. Jun, and H. Zhang. Data streaming algorithms for estimating entropy of network traffic. In Proceedings of the joint international conference on Measurement and modeling of computer systems, pages 145-156, 2006. 26. D. Koller, M. Turitzin, M. Tarini, G. Croccia, P. Cignoni, and R. Scopigno. Protected Interactive 3D Graphics Via Remote Rendering. In Proceedings of SIGGRAPH, pages 695-703, 2004. 27. S. Gumhold. Truly selective polygonal mesh hierarchies with error control. Computer Aided and Geometry Design, 22(5):424-443, 2005. 28. S. Rusinkiewicz, and M. Levoy. Streaming QSplat: A Viewer for Networked Visualization of Large, Dense Models. In Proceedings of Symposium on Interactive 3D Graphics. Monterey, CA, USA, pages 61-68, 2003. 29. A. Kalaiah, and A. Varshney. Statistical geometry representation for efficient transmission and rendering. ACM Transactions on Graphics, 24(2):348-373, 2005. 30. J. D. Foley, A. van Dam, S. K. Feiner, and J. F. Hughes. Computer Graphics. Second Edition in C, Pearson Education, USA, 1996. 31. Z.Q. Cheng, S.Y. Jin, and H.F. Liu. Anchors-based lossless compression of progressive triangle meshes. In Proceedings of Pacific Graphics, Taibei, Taiwan, 45-50, 2006. (short paper) 32. M. Isenburg, and J. Snoeyink, Compressing texture coordinates with selective linear predictions. In Proceedings of computer graphics International, pages 126-131, 2003. 33. M. Soucy, G. Godin, and M. Rioux. A texture-mapping approach for the compression of colored 3D triangulations. Visual Computer, 12(10): 503-514, 1986.

Geometry-Based Muscle Forces and Inverse Dynamics for Animation Jihun Park1, Sunghun Park2, and Yoohun Won3 1

2

Department of Computer Engineering Hongik University, Seoul, Korea [email protected] Department of Management Information Systems Myongji University, Seoul, Korea [email protected] 3 Department of Computer Engineering Hongik University, Seoul, Korea [email protected]

Abstract. In this paper, we present an animation technique based on muscle forces, inverse dynamics and a parameter optimization. We do a crude motion planning in terms of accelerations. By integrating the accelerations given an initial configuration, we obtain all essential kinematic data. We evaluate the quality of motion planning by using various constraints and a performance index evaluated using inverse dynamics. The best motion can be chosen by using a parameter optimization method. The human motion is so complicated that it needs the motions to be coordinated nicely. The planned motion is checked using a criterion which we call the footprint function: ground reaction forces if the body is on the ground, acceleration of the body center as well as energy if it is in the air. In the motion planning for a body in the air, we reduce control variables so that we work with a smaller search space with only feasible motions. Then we include human skeletal and muscle geometry in the footprint function so that we can convert robotic rotary actuators to muscles using static optimization: given a set of joint torques, we distribute them to the eight sets of human low extremity muscles. We search most human-like animated motion from an infinite set of possible motions. We compare these with experimental data. Futhermore, the muscle geometric data obtained from our linear actuator modeling can be used in tissue animation.

1

Introduction

The problem of physically based animation is that it becomes very expensive to use forward dynamics. But inverse dynamics does not give realistic motion. In this paper, we use a parameter optimization method to do realistic motion planning with the aid of tools such as ground reaction forces, musculoskeleton geometry, etc. Motion planning is done automatically by a parameter optimizer K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 584–595, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Geometry-Based Muscle Forces and Inverse Dynamics for Animation

585

with the initial aid of an end-user. Our approach is to let the animator guess or plan a gross motion in the acceleration level. In the motion guessing, we have a number of control nodes. By interpolating the values at control nodes, we can build a continuous acceleration function. Then by integrating the function, we can get the velocities and the displacements of the body given initial kinematic data. Then we calculate torques/forces using inverse dynamics based on the kinematic data we have. We need a parameter optimization method which can find the best set of motions. Then why are the ground reaction forces or the body center accelerations important measures for the body motion ? We believe that every body would agree that the body-center-acceleration becomes negative gravity when the body is in the air because there is no support for it. We can relate the body-centeracceleration with ground reaction forces. In fact a careful examination will reveal that the ground reaction force is almost the same with the body center acceleration. The only difference is their magnitude. This is also shown in the experimental result at the end of this paper. The body center acceleration is affected by the current configuration, velocities and each of the joint accelerations. So we can also say that the ground reaction force will tell us informations about the various kinematic status at a specified time. One important fact is that the ground reaction force patterns differ depending on the motions or subjects. Also we consider body energy such that the body without any actuator should have the same amount of total energy throughout the time when the body is in the air. But muscles can generate forces which cause the velocities to change. So whenever any kind of actuators are involved, the total energy can not be constant. Up to now, only rotary actuator was used for almost all physically based articulated rigid body animations. But real human body has linear actuators(muscles) which have their limits in maximum forces they can generate. By using rotary actuators, most animator restrict the maximum torque generated by the actuator. But it is different from the real human body because the joint torque generated by the human body is also configuration dependent. According to the configuration of the body, the maximum torque which can be generated at each joint changes because there is an important factor which determines the torque of the joint. It is called the moment arm[2] which represents a perpendicular distance from the joint axis to the line where muscle is acting. By solving a static optimization problem, we figure out whether the planned motion is beyond human muscle’s force limit or not. The static optimization process can be done very quickly. If the torque amount can not be generated from the human joint, then we penalize and find the best motion using a global(dynamic) optimization. So there are two kinds(static,dynamic) of optimizations used in this paper.

2

Related Work

In this section we review the major animation techniques for articulated rigid bodies. These methods fall into two categories: those based on forward dynamics [3,4,5,6,7] and those using inverse dynamics[8,9,10,11,12]. Sometimes inverse

586

J. Park, S. Park, and Y. Won

dynamics based methods also need forward dynamics for simulation[12]. Optimal control based techniques[13,14] and the sub-optimal control technique[7] use forward dynamics. The Spacetime constraints method [10,11] and the method presented in this paper are based upon inverse dynamics. Also we can classify in terms of controllers. In fact parameter optimization tries to find a controller or a best motion trajectory for us rather than let us implement a controller for the motion. The methods which use parameter optimization technique are Spacetime constraints method[10,11], sub-optimal control technique[7], and the method which was used in this paper. Some researchers directly implemented controllers by analyzing the natural motions of animals. [4,5,15] Also there are papers which implement a motion look up table to determine the next control to be made.[16,17] Motion control is one of the most difficult problems in dynamic based animation. It is usually impossible to construct exact models, and it is difficult to measure exact forces and torques. Therefore, a great deal of attention has recently been devoted to heuristic methods for control of joints [3]. But using parameter optimization method instead of heuristic methods can save a lot of time for users.[7] Another approach is to discretize continuous time dynamic systems to create parameter optimization problems. This approach is a variant of finite difference method in solving two point boundary value problem[10,11]. Inverse dynamics methods have been used to calculate joint torques given kinematic motion planning [8,9,12]. Pure inverse dynamic methods typically lack reality because the motion has been pre-planned. Especially when the free base motion is involved, the motion planning becomes difficult. The method of Spacetime Constraints[10,11] works on a displacement and joint angle level control which makes it easy to do initial guessing for a motion planning. But because they use finite difference method, their velocity and acceleration have computational errors. They need a large number of control nodes to do better motion generation. Also they need to use many constraints in order to achieve realism. Some time constraints needed for Spacetime Constraints are not natural.

3

Our Approach

The approach is based on the concept that we guess motion first and then evaluate the quality of motion. We mean the quality of motion to be evaluated by the criteria: “ground reaction forces of our model are similar to that of human body” , “body center acceleration is negative gravity in the air”, etc. If the function is badly shaped, we can enforce it to be adjusted by including a weighted function at our performance index. Because we are guessing a motion at the kinematic level, it is usually easier to work with because we can check the resulting motion through the user interface directly. Also this method is an intermediate form of suboptimal control[7] and Spacetime Constraints.[10,11] It is close to Spacetime Constraints in the sense that

Geometry-Based Muscle Forces and Inverse Dynamics for Animation

587

it uses an inverse dynamic method. But it is close to suboptimal control[7] because it needs integration and works on continuous time domain. In the process, we can calculate ground reaction forces and accelerations of the body center. Ground reaction forces and body accelerations as well as total energy provide a good tool for deciding the quality of the motion to be evaluated. 3.1

Methodology

Let us assume that we use (n+1) control nodes, ranging from 0 to n, and consider i-th state. The differential equation(system dynamic equation) holds everywhere. Here s denotes a vector of state variables(joint displacements as well as foot tip displacements at any time instance ; same with s of previous example), u denotes ¨ Here φ represents joint angle vector of a set of vector of joint accelerations, φ. the body while single dot and double dot represent velocity and acceleration respectively. The joint torque will be distributed among a set of muscle forces. But let us ignore muscle force for a while. Note that the difference from the previous example is final time(tf ). Here final time is free. T  Initially, tf and Consider the parameter vector(z) z = tf u0 u1 . . . un ui , 0 ≤ i ≤ n, vectors are given initially guessed values, and these ui s are used to calculate u(t). Here we have n time assume   ( for  simplicity   that  intervals all control nodes are equally spaced ) 0,

tf n itf n

,

tf n

,

2tf n

(n−1)tf , tf n (j−1)t belongs to [ n f

, ...,

Here jt

ui represents the control nodes at time . If time t , nf ), 1 ≤ j ≤ n, then u(t) is calculated by a linear interpolation( for simplicity ) of uj−1 and uj . Also we convert the final time, tf , into a parameter, such that normalized time τ = ttf . In this way we fix the interval of integration 0 ≤ τ ≤ 1. From the system dynamic equation, we can do inverse dynamics to calculate ¨(t), s(t) ˙ joint torque(T ) because we know s and s(t). T

∗ ¨ ∗ ˙ ) − external / gravitational forces (1) T φ (τ ) = [Iφφ ]φ(τ ) + φ˙ (τ )[Pφφφ ]φ(τ ∗ ∗ ] is a generalized mass matrix, and [Pφφφ ] is a coriolis and centripetal where [Iφφ matrix. Usually in order to calculate performance index, we need to do more integrations. Note that we need to multiply tf whenever an integration over τ is involved. Usually constraints are assigned in terms of final configuration. At the final time it should satisfy the system equality constraints(Ψ ) Ψ (tf , s(tf )) = 0, and inequality constraints(Θ), Θ(tf , s(tf )) ≤ 0. For the case of an human body which was used for our animation. we have l fixed-base degrees of freedom(l is 4 for our case), and (n+1) control nodes(we usually used 41 control nodes so n is 40). Also let us denote horizontal and vertical foot tip displacements as x and y respecT  tively. Then the state variables become s = x y φ1 φ2 φ3 φ4 and the control T T   variables become z = tf u0 u1 . . . un where ui = φ¨i1 · · · φ¨i4 , 0 ≤ i ≤ n, φ¨ij is j-th rotary joint of human body at i-th control node.

588

J. Park, S. Park, and Y. Won Y

m 4 ,I 4 RF

GMAX

4 VAS HAMS

m ,I 3

3

3

GAS

m ,I 2

2 OPF

2 SOL

m ,I 1

1

1

TA

(x,y)

0

X

Segment mi (Kg) lc (m) i Foot 2.2 0.14275 Shank 7.5 0.274 Thigh 15.15 0.251 HAT 51.22 0.343

li (m) Ii (Kgm2 ) 0.25 0.0155 0.435 0.065 0.400 0.126 0.343 6.814

(a)

Muscle Max Force(N) Min Force(N) Muscle Max Force(N) Min Force(N)

SOL 4500.1 1247.0 VAS 6854.2 676.1

OPF 4468.5 199.5 RF 972.4 11.84

TA 127.5 0.06 HAMS 2633.0 28.69

GAS 2714.6 32.21 GMAX 3438.4 25.33

(b)

Fig. 1. (a) Schematic representation of the four-segment model of a jumping human. m1 , m2 , m3 , m4 are the lumped masses of the foot, shank, thigh, and HAT(head, arms, and truck) respectively; I1 , I2 , I3 , I4 are the moments of inertia of the foot, shank, thigh and HAT respectively; l1 , l2 , l3 , l4 are the length of the foot, shank, thigh and HAT respectively; lc1 , lc2 , lc3 , lc4 are the length of from the local base to the center of the foot, shank, thigh and HAT respectively. (b) Schematic representation of musculoskeletal model for the human jumping. Symbols appearing in the diagram are : soleus(SOL), gastrocnemius(GAS), other plantarflexors(OPF), tibialis anterior(TA), vasti(VAS), rectus femoris(RF), hamstrings(HAMS), and gluteus maximus(GMAX).

4

Basic Dynamic Equations

We have implemented a symbolic recursive algorithm of Generalized Principle of D’ Alembert’s, which works on M athematicaT M . The equations are derived for the planar case, and we obtain additional equations from the ground reaction force relations. Collection of similar terms and simplification leads to Eq.(1). Figure 1(a) shows a schematic representation of the four segment model of a jumping human. Note that the generalized mass matrix is positive definite and symmetric so that we can easily recover the form of Eq.(1) with some manipulations. Also in terms of skeletal dynamics, there are dynamic discontinuities. For human jumping, the degrees of freedom(DOF) changes from three to four and to six. When the heel is on the ground it has three DOF. If the heel is off the ground while the toe is on the ground it has four DOF. When the body is in the air, DOF changes to six. We change DOF based on the ground reaction forces. If it becomes negative, we change DOF from 4 to 6.

Geometry-Based Muscle Forces and Inverse Dynamics for Animation

589

In order to follow dynamic discontinuity, we used both 4 DOF and 6 DOF equations controlled by finite state machines. The ground reaction force for heel is modeled by having a torsional exponential springs and dampers at toe, using a soft constraint for 3 DOF model.

5

Working with Skeletal/Muscle Geometry

¨(τ ), s(τ ˙ ), s(τ ), and joint torque Up to now, we explained how to calculate s which is calculated from inverse dynamic equation Eq.( 1). But we need to work on muscle force rather than joint torque using the moment arm[2] which will be explained later. As was pointed out at previous sections, human joint is actuated by linear actuator rather than by robotic rotary actuator. In terms of linear actuator, moment arm plays a very important role as was explained at the introduction. The musculotendon and its activation dynamics[7] was not used here. Now we want to convert rotary joint actuators to linear muscle actuators. The next computational segment is distributing a set of joint torques to a larger number of muscles. First we need to know how to calculate joint torque caused by each muscle. Then we use the static optimization to distribute the joint torque we get using inverse dynamics to a set of the muscles. Note that we used a rotary actuator for the foot tip because of the complexity of muscles spanning around the foot. The musculoskeletal geometric data come from Pandy’s paper[2]. 5.1

Calculating Moment Arm

In order to calculate the joint torque actuated by muscle actuators, we need to calculate the perpendicular distance from the center of a joint to a line where muscle force is acting. Unfortunately some muscles span to more than one joint. In terms of planar case we can easily apply the trigonometric rule. But the three dimensional case, it becomes too complicated to depend on the trigonometric rules.(But in this paper we are just working on planar model.) Moment arm of a muscle, r, can be defined as a time derivative of musculotendon length, lMT , with respect to a joint angle, θ. r = −d|lMT |/dθ. For the soleus, moment arm is computed(refer Figure 2) to be (l1 − r1 )r2 sinθ lMT  MT where l = (l1 − r1 + r2 cosθ)2 + (r2 sinθ)2 M Tsoleus = rsoleus Fsoleus rsoleus =

M Here Tsoleus is a ankle joint torque caused by the soleus muscle force Fsoleus . If we have m set of muscles and l rotary joints at the body, then the moment arm matrix, M, (for planar only) becomes as:

590

J. Park, S. Park, and Y. Won

MUSCLE ORIGIN

r2

SOLEUS MUSCLE INSERTION

X Y ANKLE COORDINATE

r1 l1

r

Fig. 2. An example of calculating moment arm : for soleus

⎛

T ∂|lM | 1 ∂θ 1 ⎜ ∂|lM T | ⎜− 1 ⎜ ∂θ2

−

T ∂|lM | 1 ∂θl

−

−

M =⎜ ⎜ ⎝

.. .

−

T ∂|lM | 2 ∂θ1 MT ∂|l | − ∂θ2 2

··· −

T ∂|lM | 2 ∂θl

··· −

.. .

··· .. .

T ∂|lM m | ∂θ1 ∂|lM T | − ∂θm2

.. .

T ∂|lM m | ∂θl

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

(2)

Figure 1(b) shows a lower extremity of human body with eight muscles used for our animation. Also table shows maximum/minimum muscle forces in Newton(N ). The muscle geometric data obtained from our linear actuator modeling provides us muscle length. The muscle length data can be used in tissue animation. 5.2

Solving Static Optimization Problem

Let us assume that the human body has l rotary joints(l is 4 for our case) and m muscles.(m is 8 for our case) Usually l < m. We need to distribute l joint torques, which were calculated from the Eq.( 1), among m muscles. Because the human body is an actuator redundant system, and because big muscles are recruited first, we need to distribute them appropriately. This problem is called the static optimization. At any time instance, we may be able to solve the static optimization problem. If it can not satisfy all the constraints, then we can include the error term at the performance index of Eq.( 3). We can convert the

Geometry-Based Muscle Forces and Inverse Dynamics for Animation

591

above static optimization problem into a normalized version to make the static optimization easier. In fact the static optimization is fairly easy. Then the final performance index can include muscle forces as variables. 5.3

Performance Index for Dynamic Optimization

We have a following performance index for a dynamic(global) optimization. m  tf 

 J= 0



+

i=1 tf 

FiM M FiMax

2

MFM − T

 dt + 0

m tf  i=1



F˙iM M FiMax

2 dt

T   M F M − T dt + functions of body energy (3)

0

The first term of performance index includes comportability during the entire simulation. Human body feels in terms of muscles not joint torques. When they feel pains, it is in terms of the muscle forces. The second term tries to enforce a smooth muscle force curve. Because the real muscle force is a first order differential equation of a smooth curve, we want to penalize the sharp slopes. The third term is an error term due to the static optimization. If a joint torque is too big to distribute all the muscles involved, then we penalize it at this term. Also we have other term involved with the total body energy which will be explained later.

6

Results

Figure 3 shows Animation of backward flip jump and forward flip jump. For the performance index of dynamic optimization, we also included body energy constraints. Our total body energy consists of kinetic as well as the potential energy excluding the muscle energy. In our case, the potential energy becomes zero when the body lies on the ground. The total body energy changes when the body is in the air because muscles generate forces. By using our method (as

Fig. 3. Animation of forward flip jump (a-d) and backward flip jump (e-h)

592

J. Park, S. Park, and Y. Won

well as any other inverse dynamic based method), we can not calculate an exact muscle energy stored during the jump-up phase. For the jump up phase, the performance index tries to maximize the total body energy. In terms of flying in the air and after hitting the ground, our performance index tries to minimize the changes of a total body energy as small as possible. In terms of constraints, we implemented joint limits such that whenever the limits are violated we penalized them. Joint angles consist of ankle, knee and hip. The body leaves the ground only if the vertical ground reaction force becomes negative. The jump up phase has some special constraints such that we enforce the maximum and minimum of the ground reaction forces so that the ground reaction function will look like that of human body. Also we need to give a constraint of the ground for a human jumping. But in terms of a diving, it was removed. We have constraints of final configuration of the body. In terms of the body center acceleration, we do not need to take care of it because our motion planning automatically get the exact acceleration of the body if the body is in the air. As you can find from our simulation results, the y directional(vertical) body acceleration is always negative gravity, while x directional(horizontal) body acceleration is zero when the body is in the air.(Refer zoomed-in graph for better view.(figure 4(a))) The acceleration of a body on the ground changes based on the motion it performs. When the body hits the ground, there are violent changes of motions. Also it is possible that the ground reaction force becomes negative which means that the body bounces back. The dynamic optimizer will penalize(try to reduce the amount) if a ground reaction force is far from that of stand still body. We made several animation of a human jumping with flipping( both forward and backward), a human platform diving with flipping. Because the constraints used are almost the same except the ground removal for diving, we do not find any need to explain separately. Also we included real human jumping data. (figure 4(b)) As was pointed out at the beginning of the paper, it is very difficult to measure human motion in the air. So the experimental data provided is only a jump-up and a jump-down phases missing a flying phase. Also the real data only includes normal(not violent) jumping. Note that the performer is not a athletic person and the motion is normal jumping while our model simulation is based on violent motions. Also the ground reaction force varies so much from a person to a person, but we can get a sense that the ground reaction force function of both real human and model is close although the magnitude changes. (We believe that it is due to the violent motion we animated.) The real human ground reaction forces were scaled to match with that of our model. The measurement was done using a force plate. Also note that experimental data is in a real time while model simulation is in a normalized time. Usually we played with 21 or 41 control nodes which is pretty a small number compared with that of Spacetime Constraints[10,11]. The evaluation of performance index and constraints function is done fairly quickly. Total computational time depends on the number of the control nodes as well as integration time interval. Because we work in a normalized time, the integration interval used was

Geometry-Based Muscle Forces and Inverse Dynamics for Animation

593

Backward Flip Jumping Ground Reaction Force

Ground Reaction Force

vertical

horizontal

Body Energy 2500

8000

5000 2000

5000 2000

-1000

-1000

-4000 0.0

-4000 0.0

0.5

1.0

2000

Energy ( J )

Force ( N )

Force ( N )

8000

1000 500

0.5

0 0.0

1.0

time ( normalized )

time ( normalized )

Acceleration

Acceleration

y direction(vertical)

1500

ZOOM IN

x direction(horizontal)

150

0.5

1.0

time ( normalized ) force acc (x) acc (y)

vertical force/acceleration

150

m/sec^2

100

50 0 -50

0

0.5

-100 0.0

1.0

time ( normalized )

0.5

10

0

-10

-20 0.0

1.0

time ( normalized )

0.5

time ( normalized )

Human Experimental Data Ground Reaction Force

Ground Reaction Force

vertical / jump up

Vertical /jump down

4000

4000

3000

3000

Force ( N )

Force ( N )

2000 1000 0

2000 1000 0

-1000 0.0

2.0

4.0

-1000 -2.0

6.0

0.0

2.0

4.0

time (sec)

time (sec)

Ground Reaction Force

Ground Reaction Force

Horizontal / jump up

Horizontal/jump down

4000

4000

3000

3000

Force ( N )

-100 0.0

50

-50

Force ( N )

m/sec^2

100

Force/acceleration

20

2000 1000 0 -1000 0.0

2000 1000 0

2.0

4.0

time (sec)

6.0

-1000 -2.0

0.0

2.0

4.0

time (sec)

Fig. 4. Comparison of simulation results and real data

1.0

594

J. Park, S. Park, and Y. Won

0.025 for 41 control nodes. To get a motion for a movie, we varied the time interval appropriately. The movie was made only once due to equipment availability. We used GRG2 for dynamic(global) optimization, while VF02AD for static optimization. We believe that our method is superior to Spacetime Constraints in the sense that we get a smooth(differentiable) velocity and displacement. Our method can include general performance index which includes integration. Also the final time is a free variable so that we do not restrict our model in terms of time.(for example a jump up time, a touch down time etc.) This free final time can be used for the interaction of motions among several animated figures, which is our next research topic. Our simulation is guaranteed to satisfy the physical law. In this paper, we provided a good criterion for checking the quality of body motion. That is the ground reaction forces if a body is on the ground, and the acceleration of a body center if it is in the air. Generally the problem we solve has so many trigonometric terms involved in dynamic equations that the optimization problem we solve is highly non-linear due to its nature. Basically we aid the optimizer by giving a gross motion(initializing control variables). Also we believe that Spacetime constraints may also find it difficult to find a good motion trajectory without any initial motion guessing due to complicated dynamic equations. Futhermore, the muscle geometric data obtained from our linear actuator modeling can be used in tissue animation.

Acknowledgement This work was supported by 2006 Hongik University Research Fund. Additional expenses of this research were supported by Seoul Research and Business Development Program (10555).

References 1. Dobsik, M., Frydrych, M.: Biomechanically Based Muscle Model for Dynamic Computer Animation. In: ICCVG 2004. (2004) 2. Pandy, M., Anderson, F., Hull, D.: A parameter optimization approach for the optimal control of large-scale musculoskeletal systems. Submitted to the Journal of Biomechanical Engineering (1991) 3. Wilhelms, J.: Using dynamic analysis for realistic animation of articulated bodies. IEEE Computer Graphics and Applications 7 (1987) 12–27 4. McKenna, M., Zeltzer, D.: Dynamic simulation of autonomous legged locomotion. Computer Graphics 24 (1990) 29–38 5. Raibert, M., Hodgins, J.: Animation of dynamic legged locomotion. Computer Graphics 25 (1991) 349–358 6. Hahn, J.K.: Realistic animation of rigid bodies. Computer Graphics (SIGGRAPH ’88 Proceedings) 22 (1988) 299–308 7. Park, J., Fussell, D., Pandy, M., Browne, J.C.: Realistic Animation Using Musculotendon Skeletal Dynamics and Suboptimal Control. TR 92-26, Computer Science Department, University of Texas at Austin , (also at Third Eurographic Workshop on Animation and Simulation, Sept. 1992, Cambridge, U.K.) (1992)

Geometry-Based Muscle Forces and Inverse Dynamics for Animation

595

8. Isaacs, P., Cohen, M.: Controlling dynamic simulation with kinematic constraints, behavior functions and inverse dynamics. Computer Graphics 21 (1987) 215–224 9. Isaacs, P., Cohen, M.: Mixed methods for complex kinematic constraints in dynamic figure animation. Visual Computer (1988) 296–305 10. Witkin, A., Kass, M.: Spacetime constraints. Computer Graphics 22 (1988) 159– 168 11. Cohen, M.: Interactive spacetime control for animation. Computer Graphics 26 (1992) 293–302 12. Stewart, J., Cremer, J.: Animation of Human Locomotion Climbing Stairs and Descending Stairs. In: Third Eurographic Workshop on Animation and Simulation. (1992) 13. Brotman, L., Netravali, A.: Motion interpolation by optimal control. Computer Graphics 22 (1988) 309–315 14. Pandy, M., Zajac, F., Sim, E., Levine, W.: An optimal control model for maximumheight human jumping. Journal of Biomechanics 23 (1988) 1185–1198 15. Miller, G.S.: The motion dynamics of snakes and worms. Computer Graphics 22 (1988) 169–178 16. van de Panne, M., Fiume, E., Vranesic, Z.: Reusable motion synthesis using statespace controllers. Computer Graphics 24 (1990) 225–234 17. van de Panne, M., Fiume, E., Vranesic, Z.: Control Techniques for Physically-Based Animation. In: Third Eurographic Workshop on Animation and Simulation. (1992)

Connectivity Compression for Stripified Triangle Meshes Hao Huang and Fan Zhou Computer Application Institute, Sun Yat-sen University, Guangzhou, China [email protected]

Abstract. Modern graphics application systems have to render many 3D triangle mesh models, thereby facing problems of memory and bandwidth. A general significant solution is to compress the static 3D triangle mesh model at the preprocessing phase, especially the topological information. This paper presents a new triangle meshes compression algorithm, which encodes connectivity information between vertex chains in the meshes that are already stripified. The proposed algorithm provides a new representation that is guaranteed to encode any stripified meshes in less than 1.67 bits/T, and 1.05 bits/T on average. Furthermore, the proposed algorithm employs a short compression/decompression time. Hence it will satisfy real-time rendering constraints.

1 Introduction Graphical display of a 3D surface is the most effective mean for communication. In a 3D scene, object surfaces are tessellated as meshes, consisting of polygonal patches, the simplest of which is triangle. Triangle meshes are widely used due to graphics hardware, which often optimizes its performance in rendering triangles. As the volume of triangle mesh models is increasing, many geometric datasets require a large amount of space. A common technique to reduce this data is to send long runs of adjacent triangles. Such technology which called triangle stripification [1] [2] is widely supported by today’s graphics software and hardware. The problem of constructing good triangle strips has been considered in several papers [3] [4]. The objective is to minimize the number of swaps and restarts, thereby minimizing the total number of required vertices. Although the use of indexed triangle strips reduced the amount of data needed to represent the mesh connectivity by a factor between two and three compared to indexed triangle sets, it still needs at least 2lognV bits. For storage and transmission purposes it is often necessary to have a more compact representation of a mesh. However, currently there are only a few available mesh compression techniques are designed to encode stripified triangle meshes. Different connectivity compression methods for triangle meshes had been proposed in many literatures. “Generalized Triangle Mesh (GTM)”was first introduced by Deering [5]. This technique needs (1/8[log 2 (V )] + 8)V bits to represent the connectivity information for a given triangle mesh where V denotes the number of vertices. Chow [6] improved the performance of GTM using some heuristics. Taubin and Rossignac’s method [7] had applied topological surgery for producing a simply connected polygon, whose compression result was about 4.4V bits. But this method consumes K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 596–605, 2007. © Springer-Verlag Berlin Heidelberg 2007

Connectivity Compression for Stripified Triangle Meshes

597

large amount of internal memory and takes a longer time for compression and decompression processes. Gumhold and Strasser [8] introduced a compression algorithm that uses seven building operations to assign the traversal order for the mesh. Using Huffman encoding, the compression was between 3.4V and 4.3V bits. Touma and Gotsman [9] proposed a new triangle mesh compression method created by exploiting triangle marching structure. But it focused on regular meshes mainly. In his literature [10], Rossignac presented an algorithm that encodes one triangle at a time using five operation codes. King and Rossignac [11] claimed a guaranteed 3.67V bits for compressing the connectivity of planar triangle graph using an improved Edgebreaker’s encoding. But Edgebreaker was either multiple pass traversals or operated in reverse order. Multiple pass traversals had taken a long time to be executed. Comparatively the valence-driven method [12] was a vertex-based method but it had required an extra pass to calculate the valence of vertices when the triangle mesh does not have vertex degree information. Kubola [13] described a new algorithm with nearly 1 bits/triangle and a fast handling speed. However it missed the problem of the connectivity among vertices in the same row which was referred and solved in literatures [14] and [15]. However, most of available connectivity compression techniques do not support the encoding stripified meshes. Using strips and vertices’ valence, B’oo [14] and Park [16] suffered from high compression rate. Isenburg introduced Triangle Fixer [17], which was a simple and efficient scheme for encoding the triangle strips connectivity. Enhancing algorithm has been proposed in literature [18]. But the algorithms suffer restriction that operation in reverse order should work only off line since their decompression order followed the reverse order of the compression. This paper proposes a new method for coding the connectivity information of stripified meshes with trade-offs between compression rate and compression/decompression time. In this research report, the strips’ connectivity information are encoded between vertex chains. The proposed algorithm encodes and decodes 3D models strips with single traversal for real-time rendering. The proposed algorithm provides a new representation that is guaranteed to encode any stripified meshes in less than 1.67 bits/T, and 1.05 bits/T on average. Furthermore, the lossless compression algorithm employs a short compression/decompression time, which will satisfy real-time rendering constraints. This paper does not address the compression of vertex locations. For this, any predictive [7] [9] or spectral [19] coder may be used in combination with our method. The new connectivity compression procedure is described in the next section. Decompression procedure is proposed in section 3. Section 4 gives special cases and the corresponding solution. Compression result analyses and experimental results are given in section 5. Finally, conclusions are drawed in section 6.

2 Compression Procedure 2.1 Triangle Strips Decomposition Although strips generated by all the available stripification algorithms can be encoded by this algorithm, the algorithm presented by Xiang et al. [4] is strongly recommended.

598

H. Huang and F. Zhou

Fig. 1. Triangle strips and stripified triangle mesh model

Not only because it can generate less and longer strips, but also because this algorithm is very fast and it does not need too much memory [20]. In figure 1, the triangle mesh model is decomposed as a set of triangle strips, which is denoted as {T0, T1, … , Tn}. Triangle strip Ti is defined as a consecutive sequence of triangles with two vertex chains. In triangle strip Ti, the left vertex chain of the stripification direction is denoted as C1 and the right vertex chain is denoted as C2. Denote the ith vertex of C1 as V1i and the ith vertex of C2 as V2i. 2.2 Basic Cases and Rules in Triangle Strip Encoding Here define an edge between vertex A and B as A→B. Although the edges are undirected, A is defined as outpoint and B as inpoint. The explicit encoded edges are defined as directed-edges and those encoded implicitly as indirected-edges.

Fig. 2. Illustrations of basic cases (solid edges are directed-edges and dashed are indirectededges)

Case A In triangle strips, a quadrangle with two vertices in chain C1 and other two vertices in chain C2 is the most common case (nearly 65%). In figure 2a, “1→3; 2→4” are encoded. Three adjacent vertices represent a triangle, such as ∆（ 1, 3, 2） and ∆（ 3, 2, 4） . Here 1→3 and 2→4 are directed-edges, while 1→2, 2→3 and 3→4 are indirected-edges. It is evident that both directed-edges have the same “direction”, which means their inpoints belong to the same vertex chain and so their outpoints do. Case B When one vertex in C1 or C2 has connectivity relationships with three or more vertices in the other vertex chain in a strip, it was defined as a reused-vertex as vertex 1 in figure 2b. Here “1→2, 3, 4, 5” are encoded. Outpoint 1 is a reused-vertex, and inpoints 2, 3, 4, 5 have connectivity relationships with vertex 1. That is, 1→2, 1→3,

Connectivity Compression for Stripified Triangle Meshes

599

1→4 and 1→5 are directed-edges, while 2→3, 3→4, and 4→5 are indirected-edges. It is evident that there are no connectivity relationships between unadjacent inpoints. For example, there is no “2→4” in figure 2b. Case C (operator RE) Case A requires both directed-edges with the same direction, but there may be different direction between two adjacent directed-edges in special condition as in figure 2c. As in figure 2c, the left quadrangle should be encoded as “1→4; 2→5” and the right quadrangle as “5→2; 6→3”. These two parts code cannot be connected directly since different direction occurs. Operator RE (reverse edge) is proposed to solve this problem. RE indicates the direction of directed-edge on its left should be reversed when decoded with directed-edge on its right. Hence figure 2c should be encoded as “1→4; 2→5; RE 6→3”. Case D (operator RV) Sometimes inpoint of one directed-edge is also outpoint of the next directed-edge as in figure 2d. In figure 2d, the left quadrangle should be encoded as “1→6; 2→7” and the rest part as “7→2, 3, 4, 5”. These two parts code cannot be connected directly due to different allocations of vertices 7 in two parts. Hence operator RV (reverse vertex) is proposed to solve this problem. That is, inpoint of the directed-edge on its left should be viewed as outpoint of the vertices on its right. Hence figure 2d should be encoded as “1→6; 2→7; RV 3, 4, 5”. Apparently, vertex 7 is a reused-vertex of vertex 3, 4, 5 on RV’s right. Detailed use of RV is described in Rule 3.

Fig. 3. Illustrations of encoding rules

Encoding rules are set as follows: 1) Directed-edges only exist between C1 and C2. Namely directed-edges with two vertices on the same vertex chain are never allowed. 2) Case B must be followed an operator RE as in figure 3a, except Case B occurs at the last part of a triangle strip. 3) RV just appears in two conditions: ¾ Vertex before RV is a reused-vertex, as in figure 2d. ¾ Vertex before RV is the last vertex of its vertex chain, as in figure 3b. Ta in figure 4 are encoded as the following tokens: 7→1 8→2 3 RE 4→9 5→10 11 12 RE 13→6 Tb in figure 4 are encoded as the following tokens: 1→7 2→8 RV 3 RE 4→9 10 RE 11→5 12→6 RV 13

600

H. Huang and F. Zhou

Fig. 4. Example of triangle mesh encoding

2.3 Substitution with R and O and Operators Elimination For more compact representation, R and O are introduced in this section to substitute the original vertex token. Vertices in C1 are substituted for O and vertices in C2 are substituted for R. Furthermore, the following rules can be framed based on section 2.2: 1) Operator RE only appears between two directed-edges with different direction. 2) Operator RV just appears after a reused-vertex or before the last vertex of chain. The difference between RE and RV is that the vertex before RE has the same kind (either both R or both O) with the vertex next to RE, and the vertex before RV has a different kind (R or O) with the vertex next to RV. Based on these rules, operator RE and RV can be eliminated in the final code, and can be resumed during decoding progress (check more details in section 3). Consider figure 4 for example, the vertex tokens can be transformed into sequences with R and O as follows (“→” tokens are eliminated): The final codes of Ta in figure 4 are as follows: ROROOORORRRRO The final codes of Tb in figure 4 are as follows: OROROORRROROR

3 Decompression Procedure Parts that have not yet been decoded are defined as dangling-parts. Dangling-parts will be decoded sequentially. Initially decoding rules are set as follows: 1) If the dangling-parts alternate between single R and single O, the anterior code is decoded as outpoint of a directed-edge, and the next one as inpoint. For example, “R O R O R O …” is decoded as “R→O; R→O; R→O; … ”. 2) If R is followed by three or more O in the beginning of dangling-parts, R is a reused-vertex to these O. 3) This rule is about the resumption of RE: ¾ When there is “R O O R” or “O R R O” in the beginning of dangling-parts, operator RE should be inserted between the adjacent tokens with the same kind (both R or both O), such as “R→O; RE O→R” or “O→R; RE R→O”.

Connectivity Compression for Stripified Triangle Meshes

601

¾

If R is decoded as a reused-vertex based on decoding rule 2 or decoding rule 4.1, and the following O sequence is followed by R, a directed-edge should be made with the last O in O sequence and the later R. Before this directed-edge an operator RE should be inserted. For example, “R O O O O O R” should be decoded as “R→O, O, O, O; RE O→R”. 4) This rule is about the resumption of RV: ¾ When R is decoded as inpoint of a directed-edge, and the subsequent dangling-parts have two or more O, an operator RV should be inserted after this R, which means R is a reused-vertex to these O. ¾ When R is decoded as inpoint of a directed-edge, and the subsequent O is the last vertex of its vertex chain, an operator RV should be inserted after this R. 5) Rule 1, 2, 3, and 4 cannot be used across strips. The above rules also work when R and O are replaced by each other. Consider the final codes of Ta and Tb in section 2.3, they can be decoded as follows: ¾ First resume the operator RE, RV, and →: R→O R→O O RE O→R O→R R R RE R→O O→R O→R RV O RE O→R R RE R→O R→O RV R ¾ Then transform O into original vertices: R→1 R→2 3 RE 4→R 5→R R R RE R→6 1→R 2→R RV 3 RE 4→R R RE R→5 R→6 RV R ¾ Finally transform R into original vertices: 7→1 8→2 3 RE 4→9 5→10 11 12 RE 13→6 1→7 2→8 RV 3 RE 4→9 10 RE 11→5 12→6 RV 13

4 Special Cases: Intra-bifurcate Although most strips generated by the available algorithms are regular, but some degenerated cases need to handle since computing an optimal set of regular triangle strips is NP-complete. To solve this problem and obtain higher compression rates, the restriction of stripification result is loosened here. This section presents intrabifurcate, one kind of degenerated case in triangle strips, and proposes heuristics to improve the compression performance. If one triangle strip has some parts of vertex chain that are not adjacent to the next vertex chain, intra-bifurcate may exist in this triangle strip. Definitions as follows: ¾

Intra-bifurcate - a set of triangles whose vertices all belong to the same vertex chain as grey triangles shown in figure 5.

The basic assumption of the proposed method is that all triangles in a strip should be related to both C1 and C2. But triangles in intra-bifurcates are not related to the other vertex chain, which must be processed distinguishingly from the normal triangles. 4.1 Encoding of Intra-bifurcate Figure 5 illustrates how intra-bifurcate is encoded from the original vertex chain. Intra-bifurcates can exist in any triangle strip. Using a newfangled bracketing algorithm, the intra-bifurcate in triangle strip Ti-2 is encoded as follows:

602

H. Huang and F. Zhou

1) All vertices in intra-bifurcate are arranged orderly in a queue. 2) Each triangle in the intra-bifurcate can be represented as a pair of brackets. The bracket on the left, which means the first vertex of this triangle, is denoted as 0 and the bracket on the right, which means the third vertex of this triangle, is denoted as 1. If a vertex is shared by n intra-bifurcate triangles as their first/third vertices, there will be n left/right brackets that correspond to this vertex. The results of intra-bifurcate encoding are a set of brackets (bit stream of 0 and 1). Therefore, the connectivity of intra-bifurcate triangle can be encoded with 2 bits per triangle.

Fig. 5. Examples of intra-bifurcates encoding

4.2 Decoding of Intra-bifurcate The triangle strip Ti-2 with intra-bifurcate is decoded as follows: 1) Travel the vertices in the intra-bifurcate queue orderly until the first right bracket is found. 2) Denote the vertex corresponding to the right bracket as A. 3) Denote the vertex without any bracket on A’s left in the queue as B. 4) Denote the nearest vertex with left bracket on B’s left as C. 5) Denote three vertices of one intra-bifurcate triangle as A, B and C, and then eliminate one right bracket from A and one left bracket from C. 6) Obviously, each vertex could be a second vertex (namely B in step 3) of one intra-bifurcate triangle at most. Keep on decode the intra-bifurcate vertex queue until all brackets have been eliminated from every vertices.

5 Compression Rate Analyses and Experimental Results Apparently, in a strip without “intra-bifurcate” only R and O corresponding to its vertices should be encoded. Hence the number of codes is: R + O =V '

In a strip with V’ vertices, the number of triangle is:

1)

Connectivity Compression for Stripified Triangle Meshes T = V '−2

603

2)

Since the code just comprises R and O. Using Huffman method, the coding rate is 1 bit per R and 1 bit per O. Combined with formula 1) and formula 2), the relationship between T and coding volume is as follows: R *1 + O *1 = T + 2

3)

So the compression rate of a triangle strip with T triangles is: BPT = (T + 2) / T

4)

which means the compression rate of a triangle strip is nearly 1 bit/T when T is large enough. However, if we allow both sequential and fan strips, we will need at most ⎡T / 4⎤ strips [4], which means the shortest strips has 3 triangles, hence the worst compression rate is 1.67 bits/T. When it comes to triangle strips with intra-bifurcates, one intra-bifurcate triangle must be encoded with 2 bits, hence the compression rate will rise. In the worst case, a strip S with V’ vertices has the largest number of intra-bifurcate triangles when vertices in C1 or C2 all related to intra-bifurcate. Suppose the number of vertices in C1 is V1 and the number of vertices in C2 is V2 (both V1 and V2 are larger than 2). Since each vertex could be a second vertex of one intra-bifurcate triangle at most except for the first and the last vertices, the number of intra-bifurcate triangles in C1 is V1-2 and the number of intra-bifurcate triangles in C2 is V2-2. Hence coding volume of intrabifurcate is 2 * (V1 − 2 + V2 − 2) = 2V '−8

5)

Combined with formula 3), the compression rate of a triangle strip with T triangles is: V '+2V '−8 3V '−8 1 = = 1.5 + V '−2 + V '−4 2V '−6 2V '−6

6)

Suppose both V1 and V2 are 3, namely V’ is 6, the compression rate of a triangle strip is also less than 1.67 bits/T. In experimental results (measured on a Genuine Intel(R) CPU T2400 at 1.83GHz.), compression rate and compression/decompression time will be shown to indicate the performance. According to all these tests in [20], the algorithm presented by Xiang et al. Table 1. Compression rate Model

Vertex

Triangle

Strips

Total bits

Bits/T

teapot

1,071

1,992

8

2,008

1.01

lake

4,225

8,192

64

8,320

1.02

Mexico

14,150

28,104

95

28,294

1.01

skull

20,002

40,000

2,617

45,234

1.13

Bunny

35,947

69,451

2,508

74,467

1.07

Scanned Buddha

186,127

378,726

25,504

429,734

1.13

LandScape

211,524

419,870

123

420116

1.00

604

H. Huang and F. Zhou Table 2. Compression/decompression time Model

Encode time (s)

Readcode time (s)

Decode time (s)

teapot

0.003256

0.029615

0.000653

Lake

0.011760

0.096236

0.002007

Mexico

0.047818

0.387826

0.009647

skull

0.121468

1.016242

0.022921

Bunny

0.147128

1.267552

0.026172

Scanned Buddha

0.786141

5.681413

0.113161

Land Scape

0.871843

5.195572

0.129984

(FTSG [4]), provides a stripification with less and longer strips that is most suitable for the rendering. Hence FTSG is selected as the stripification method used before the compression algorithm. First the compression rate is analyzed. As in table 1, the average compression rate is 1.05 bits/triangle; as in the worst case it is less than 1.67 bits/triangle. From table 2, the proposed algorithm employs a short compression/decompression time, and the time applied in decoding is 2% of that applied in reading code. Furthermore, the proposed algorithm only needs single pass traversal hence gives better performance compared with Edgebreaker [10] and Kubola’s algorithm [13].

6 Conclusion This paper presents a new triangle meshes compression algorithm, which provides a new representation that is guaranteed to encode any meshes in less than 1.67 bits/T, and 1.05 bits/T on average. Furthermore, the proposed algorithm employs a short compression/decompression time, will satisfy real-time rendering constraints, and has a good representation.

References 1. F.Evans, S. S. Skiena, and A. Varshney. Optimizing triangle strips for fast rendering. In Visualization’96 Conference Proceedings, pp. 319-326, 1996 2. M.Woo, J. Neider, and T.Davis. Open GL Programming Guide. Addison-Welsley, Reading, MA 1996 3. B.Speckmann and J.Snoeyink. Easy triangle strips for TIN terrain models. In Proceedings of 9th Canadian Conference on Computational Geometry, pp. 239-244, 1997 4. X.Xiang, M.Held,and J.Mitchell. Fast and efficient stripification of polygonal surface models. In Proceedings of Interactive 3D Graphics, pp. 71-78, 1999. 5. M. Deering. Geometry compression. In SIGGRAPH ’95 Proc., pages 13–20, Aug. 1995. 6. M. Chow. Optimized geometry compression for realtime rendering. In IEEE Visualization ’97 Proc., pages 347–354,Oct. 1997. 7. G. Taubin and J. Rossignac. Geometric compression through topological surgery. Technical report, Research Report RC-20340, IBM Research Division, 1998.

Connectivity Compression for Stripified Triangle Meshes

605

8. S. Gumhold and W. Strasser. Real time compression of triangle mesh connectivity. In SIGGRAPH ’98 Proc., pages 133–140, July 1998. 9. C. Touma and C. Gotsman. Triangle mesh compression. In Proc. of Graphics Interface ’98, pages 26–34, 1998. 10. J.Rossignac. EdgeBreaker: Connectivity compression for triangle meshes. IEEE Transactions on Visualization and Computer Graphics,pp.47-61,1999 11. D King, J Rossignac. Guaranteed 3.67 v bit encoding of planar triangle graphs. 11th Canadian Conference on Computational Geometry, 1999 12. P.Alliez and Desbrun. Valence-driven connectivity encoding for 3D meshes. Eurographics’01, pp.480-489, 2001 13. K. Kubola and Henry Chu. Connectivity Compression for Three-dimensional Planar Triangle Meshes.Proceeding of the Ninth International Conference on Information Visualization.2005 14. M. B´oo, M.Amor, M.Doggett, J.Hirche, and W.Strasser. Hardware Support for Adaptive Subdivision Surface Rendering. Proc.of ACM Siggraph/Eurographics Workshop on Graphics Hardware,pages 33-40,2001 15. P.N.Mallon,M. B´oo,M.Amor and J.D.Brugiera.Concentric Strips:Algorithms and Architecture for the Compression/Decompression of Triangle Meshes.Proceedings of the First International Symposium on 3D Data Processing Visualization and Transmissin,2002 16. D.G.Park, Y.S.Kim, H.G.Cho, Triangle Mesh Compression for Fast Rendering. IEEE Proceedings of the 1999 International Conference on Information Visualization,1999 17. M Isenburg. Triangle fixer: edge-based connectivity compression. In Proceedings of 16th European Workshop on Computational Geometry, pp. 18-23, 2000. 18. Isenburg M. Triangle strip compression. In Proceedings of Graphics Interface, 197-204. 2001. 19. Z. Karni and C. Gotsman. spectral compression of mesh geometry. In SIGGRAPH’00 Conference Proceedings, pp. 279-286, 2000 20. P. Vanecek. Triangle Strips For Fast Rendering.Technical Report No.DCSE/TR-2004-05. Apirl 2004.

Interpolating Solid Orientations with a C 2 -Continuous B-Spline Quaternion Curve Wenbing Ge*, Zhangjin Huang, and Guoping Wang Dept. of Computer Science and Technology, Peking University, Beijing, China Phn:. +86-010-62765819-816 [email protected] {dgewb,hzj,wgp}@graphics.pku.edu.cn Abstract. An algorithm is presented to construct a C2-continuous B-spline quaternion curve which interpolates a given sequence of unit quaternions on the rotation group SO(3). We present a method to extend a B-spline interpolation curve to SO(3). The problem is essentially to find the quaternion control points of the quaternion B-spline interpolation curve. Although the associated constraint equation is non-linear, we can get the accurate quaternion control points according to two additional rules for quaternion computations in S3. In addition, we provide a point insertion method to construct interpolation curves that have local modification property. The effectiveness of the algorithm is verified by applying it to some examples. Keywords: C2-continuous, interpolation, quaternion, B-spline curve, computer animation.

1 Introduction Quaternions have been introduced as a useful representation for interpolating 3D orientation in computer animation; especially the unit quaternions are most suitable for controlling the rotation for both object models and virtual cameras [6]. The unit quaternions have many advantages such as they are free from singularities; and they are computationally more efficient than the 3×3 matrix representation of a 3D rotation [8]. Thus, the question of how to construct unit quaternion curves plays an important role in computer animation [3-6, 8-13]. In computer animation, it is a fundamental problem to generate a smooth motion for a rigid body so that the generated motion interpolates a given sequence of keyframe positions and orientations [8]. The rigid motion can be represented as continuous translational and rotational motions. The rotation group SO(3) is obtained as a projective space of the unit quaternion space S3. For the interpolation of key frame positions in R3, there are many well-known techniques available such as Bspline and Bezier. However, it is rather difficult to extend them to S3. Shoemake [6] provided a scheme for deriving Bezier control points from a sequence of quaternions. This provides an interpolating spline for quaternions, but the construction is somewhat more complicated. Nielson [3] presents a new method to construct a *

Corresponding author.

K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 606–615, 2007. © Springer-Verlag Berlin Heidelberg 2007

Interpolating Solid Orientations with a

C 2 -Continuous B-spline Quaternion Curve

607

quaternion w-spline curve. Lee and Shin [4] present a new scheme that applies a filter mask (or a convolution filter) to orientation data. But many of common properties in R3 in geometric modeling break down even in the simple non-Euclidean space S3, for example, the de Casteljau type construction of cubic B-spline quaternion curves does not preserve C2-continuity [8]. There are also many other methods to construct curves in S3, but most of the curves are C1 continuous quaternion curves [6,10,13]. Because of the discontinuity in second derivatives, there may occur large angular accelerations at the curve joints, which have undesirable effects on generating naturally-looking rotations, thus, high degree continuity is also an important factor for the quaternion curves in computer animation [8]. In this paper, we present one new approach to construct the C2 continuous B-spline quaternion curve by using of the additional quaternion operators in S3. Kim etc. [9] presented a general curve construction scheme that extends a spline curve in R3 (represented by a basis form) into a similar one in S3. But when constructing an interpolation quaternion curve [8], they don’t provide a method to compute the accurate control points because of the non-linearity of the equation. Instead they approximate the control points. With our method, we can get the accurate control points. In addition, we provide a point insertion method to construct interpolation curves that have local modification property. The rest of this paper is organized as follows. In section 2, we review the previous construction method of B-spline quaternion curves. Section 3 describes our algorithm for compute the control points of unit quaternion curves using proposed rules. Section 4 demonstrated some experimental results. Finally, we conclude in section 5.

2 Preliminaries Given n+1 control points {pi}(i=0, …, n), the B-spline curve P(t) of order k is defined by: n

P(t ) = ∑ pi Bik (t )

(1)

i =0

where Bik(t) is the basis function of B-spline of order k, and the corresponding knot vector is T={t0, …, tn+k-1}. The B-spline curve can be reformulated as follows: n ~ ~ P(t ) = p0 B0k (t ) + ∑ ( pi − pi −1 ) Bik (t )

(2)

i =1

where ⎧ i +k k ⎪∑ Bi (t ) if (t i < t < t i +k −1 ) ⎪ j =i n ~k ⎪ k Bi (t ) = ∑ B j (t ) = ⎨1 if (t ≥ t i +k −1 ) j =i ⎪ ⎪ if (t ≤ t i ) ⎪0 ⎩

(3)

Kim etc. [9] presented a general curve construction scheme that extends a spline curve in R3 (represented by a basis form) into a similar one in S3. By replacing the

608

W. Ge, Z. Huang, and G. Wang

curve P(t) with the quaternion curve Q(t), and the control point pi with the quaternion control point qi, and the summations with the quaternion multiplications, the corresponding B-spline quaternion curve with a cumulative basis form is formulated as follows:

Q(t ) = q0

~ B0k ( t )

n

∏ (q i =1

~ Bik ( t ) −1 i −1 i

q)

(4)

The B-spline quaternion curve is Ck-2 continuous and locally controllable by moving the control points {qi} [9]. Based on the above equation, Kim etc. [8] presented a method that can extend a Bspline interpolation curve to a similar one in S3. Given a sequence of data points Pi (i=0, …, n), the point interpolation can be done by constructing a uniform cubic Bspline curve P(t) which interpolates each point Pi at t=i. The cubic B-spline P(t) with n+3 control points pi (i=-1, 0, …, n, n+1) is defined by: n +1

P (t ) = ∑ pi Bi4 (t )

(5)

i = −1

The relationship P(i)=Pi forms a system of linear equations:

1 4 1 pi −1 + pi + pi +1 = Pi , for i=0, …, n. 6 6 6

(6)

With proper boundary conditions for p-1 and pn+1, there are n+1 equations for n+1 unknowns. It’s easy to compute the exact solution in R3. But when extended to Bspline quaternion curve, the system of equations is non-linear. Kim [8] doesn’t offer a method to compute the exact solution. Instead they use an iterative method to approximate the solution. In the next section, we introduce a new method to solve these equations and get the exact solution.

3 Quaternion Spline Interpolation 3.1 Local B-Spline Interpolation As mentioned equations (6) in section 2, there are n+1 equations for n+1 unknowns xi with proper boundary conditions for x’-1 and x’n+1: ⎛2 1 ⎞ 0 0 " 0⎟ ⎜ 3 6 ⎟ 1 ⎛ ⎞ ⎜ ⎜ x0 − x−′ 1 ⎟ ⎜ 1 2 1 0 " 0 ⎟⎛ x0′ ⎞ 6 ⎟ ⎟⎜ ⎜ ⎟ ⎜6 3 6 x1 ⎜ ⎟ ⎜ 0 % % % " 0 ⎟⎜ x1′ ⎟ ⎟⎜ # ⎟ ⎜ ⎟=⎜ 1 2 1 # 0 ⎟⎜ ⎟ ⎜ ⎟ ⎜0 0 6 3 6 ⎟⎜ xn′ −1 ⎟ ⎜ xn −1 ⎟ ⎜ ⎜ ⎟ ⎜ # # 0 1 2 1 ⎟⎜ x′ ⎟ 1 ⎜ xn − xn′ +1 ⎟ ⎜ 6 3 6 ⎟⎝ n ⎠ 6 ⎝ ⎠ ⎜ 1 2⎟ ⎜0 0 0 0 ⎟ 6 3⎠ ⎝

(7)

Interpolating Solid Orientations with a

C 2 -Continuous B-spline Quaternion Curve

609

Where xi (i=0,1,…,n) are the given points to be interpolated and x’i (i=0,1,…,n) are the control points we want to compute. Since the system is strictly diagonally dominant, there exists a unique solution. But we can see that the interpolation Bspline curve doesn’t have the local modification property: if we change any given point, all the control points will be changed. So we suggest that additional control points should be interpolated at every three or four points. These additional control points will give the quaternion interpolation curve local modification property. Also it will simplify the computation of the above equations. We introduce one method to produce such additional points. We use 4-point method [2] which is considered as a classic method in the curve generation. The 4point scheme works by repeated refinement of S={P0i , i Z}, according to the rule

∈

j +1 2i

P

P2 ji ++11 =

= Pi ,

(8)

j

1 9 ( Pi j + Pi +j1 ) − ( Pi −j1 + Pi +j 2 ) 16 16

(9)

We just select the second rule to get additional points, That is, additional control points ak at every three control points is defined by: ak =

9 1 ( x3k + 2 + x3k +3 ) − ( x3k +1 + x3 k + 4 ) 16 16

(10)

Now, the computation of each control point just related to the three given points and two additional points: ⎛2 1 ⎛ ⎞ ⎜ ⎜ x3 k − ak ⎟ ⎜ 3 6 ⎜ ⎟ ⎜1 x3k +1 ⎜ ⎟=⎜6 1 ⎜ ⎟ ⎜ x3k + 2 − ak +1 ⎟ ⎜ 0 6 ⎝ ⎠ ⎜ ⎝

1 6 2 3 1 6

⎞ 0⎟ ⎟⎛ x3′ k ⎞ ⎟ 1 ⎟⎜ ⎜ x3′ k +1 ⎟ ⎟ 6 ⎜ ⎟ 2 ⎟⎝ x3′ k + 2 ⎠ ⎟ 3⎠

(11)

We can get the solution: x3′ k =

3 15ak + ak +1 , (15 x3 k + x3 k + 2 − 4 x3k +1 ) − 28 56

x3′ k +1 = x3′ k + 2 =

(12)

3 a + ak +1 , (4 x3 k +1 − x3 k − x3k + 3 + k ) 7 6

(13)

3 a + 15ak +1 ( x3 k + 15 x3k + 2 − 4 x3 k +1 ) − k 28 56

(14)

From the above solution, we can see that if we change x3k+i (i=0,1,2), it will affect the control point x’3k+i (i=0,1,2) and ak (or ak-1). The ak (or ak-1) will affect the control points x’3(k+1)+i (or x’3(k-1)+i) (i=0,1,2). So one given point will totally affect 6 points round it. Next, we will apply our above method to construct the B-spline quaternion interpolation curves. We introduce two additional computation rules in S3 according to R3.

610

W. Ge, Z. Huang, and G. Wang

3.2 The Rules in S3 According to R3 (1)

(2)

The control polygon rule: the edge of control polygon in S3is the great inferior circle arc between them. That arc is the curve where the sphere intersects a plane through the two points and the origin. The addition rule: sin uθ sin(1 − u )θ , q1 + q2 sin θ sin θ

uq1 ⊕ (1 − u )q2 =

where cosθ = q1 ⋅ q 2 (3)

(1 + u )q1Θuq2 =

(4)

(15)

The subtraction rule: sin uθ ( q1 cosθ − q2 ) + q1 cos uθ sin θ

The equation’s solution rule: the solution to the equation q1=(1-u)x u x = q1 cos θ− 1− u

u θ 1 − u ( q − q cos θ ) 2 1 sin θ

sin

(16)

⊕u q

2

is: (17)

The rule (1) and rule(2) have been described by Shoemake[6]. The addition rule is just a formula for spherical linear interpolation from q1 to q2. The spherical linear u q2 interpolation is the angle interpolation between q1 and q2: uq1

⊕(1- )

q3 q2 q1

(1-u)a

ua

⊕

Fig. 1. The computation of uq1 (1-u) q2

You can get such equation: q2 − q3 cos uα sin uα = q3 cos(1 − u )α − q1 sin(1 − u )α

(18)

It can work out q3 as follows: q3 =

sin uα sin(1 − u )α q1 + q2 sin α sin α

(19)

where cosa= q1·q2. x’3k+i Also we can describe the subtraction rule and get the equation’s solution in the same way as follows

Interpolating Solid Orientations with a

Θu q

Subtraction rule: (1+u) q1

C 2 -Continuous B-spline Quaternion Curve

611

2

q1 q3 q2

a

ua

Θu q

Fig. 2. The computation of (1+u) q1

2

q3 − q1 cos uα sin uα = sin α q1 cos α − q2 It can work out q3 as follows:

q3 =

(20)

sin uα (q1 cos α − q2 ) + q1 cos uα sin α

where cosa= q1·q2. Now we define the equation’s solution rule: q1=(1-u)x

⊕u q

(21)

2

q1 q2 x

ua/(1-u)

ua

⊕

Fig. 3. The solution of equation: q1=(1-u)x u q2

q2 − q1 cos α sin α = u u q1 cos α − x sin α 1− u 1− u

(22)

It can work out x as follows: u sin α u (23) x = q1 cos α − 1 − u (q2 − q1 cos α ) 1− u sin α where cosa= q1·q2. By the above four rules, we can compute the solution of the following equation, though it is non-linear:

612

W. Ge, Z. Huang, and G. Wang

1 2 1 ⎧ ′ = ⊕ ⊕ x a x x3′ k +1 3 k k 3 k ⎪ 6 3 6 ⎪⎪ 1 2 1 ⎨ x3k +1 = x3′ k ⊕ x3′ k +1 ⊕ x3′ k + 2 6 3 6 ⎪ 1 2 1 ⎪x = x′ ⊕ x′ ⊕ a ⎪⎩ 3k + 2 6 3k +1 3 3k + 2 6 k +1

(24)

(25)

(26)

ak is computed using 4-point scheme: 18 1 1 2 1 1 (27) ak = ( x3k + 2 ⊕ x3k +3 )Θ ( x3k +1 ⊕ x3k + 4 ) . 16 2 2 16 2 2 operation has Commutative law and Associative law: For (24)+(26): 1 x3k ⊕ 1 x3k +2 = 2 ( 1 x3′k ⊕ 1 x3′k +2 ) ⊕ 1 x3′k +1 ⊕ 1 ( 1 ak ⊕ 1 ak +1) 2 2 3 2 2 6 6 2 2 1 1 1 2 (25): x3k +1 = ( x3′ k ⊕ x3′ k + 2 ) ⊕ x3′ k +1 . 3 2 2 3 1 1 We can treat x3′ k ⊕ x3′ k + 2 as one quaternion, and work out x3′ k +1 : 2 2 12 5 6 1 1 1 1 1 (28) x3′k +1 = x3k+1Θ ( ( x3k ⊕ x3k+2 )Θ ( ak ⊕ ak +1)) 7 7 5 2 2 5 2 2 Then, we can compute x’3k from (24) and x’3k+2 from (26). From the control points {x’k}and {ai}, we can get the B-spline quaternion interpolation curve use formula (4). The curve is C2 continuous as shown in Figure 4, which is proved by Kim etc. [9].

⊕

Fig. 4. Quaternion interpolation: the yellow curve is a polyline in S3, while the red curve is a Bspline quaternion interpolation curve

4 Experimental Results From the above results, we can apply the B-spline quaternion interpolation curve to produce the sweep motion and sweep solid. By adapting the construction method

Interpolating Solid Orientations with a

C 2 -Continuous B-spline Quaternion Curve

613

designed by [9] for B-spline quaternion curves, the C2-continuity is guaranteed which is useful in the construction of the sweep surface/solid. Example 1: given a sequence of object orientations shown as the Figure 4(a). We get the sweep solid as shown in Figure 4(b): (sweeping with cubic B-spline quaternion curve):

(a) the orientation of objects

(b) the sweep solid by the sweep motion Fig. 5. Example 1

Example 2: given a sequence of object orientations as Figure5 (a), we get the result shown as (b): (sweeping with cubic B-spline quaternion curve):

(a) the orientation of objects Fig. 6. Example 2

614

W. Ge, Z. Huang, and G. Wang

(b) the sweep solid by the sweep motion Fig. 6. (Continued)

5 Conclusion In this paper, we focus on the B-spline interpolation problem: how to find the Bspline control points so that the B-spline quaternion curve interpolates a given sequence of unit quaternions. We introduce two additional rules for the computation of quaternion in S3: the subtraction rule and the equation’s solution rule. By the four rules, we can compute the control points accurately in S3. Furthermore, we present a method to produce the local modification property for the B-spline quaternion interpolation curve. Here, there are some issues we should consider in the future, we had used the 4point scheme to produce new control point, but we can also use the 6-point scheme or other method to produce the new control point. In general, these interpolated quaternion B-spline curves are different by using of different scheme. Therefore, we will continue to research their relationship and some conditions.

Acknowledgements This research was supported by Grant No 2004CB719403 from The National Basic Research Program of China(973 Program), Grant No. 60473100 and Grant No. 60573151 from National Natural Science Foundation of China, Grant No. 908-03-0110 from Ocean 908 Special Program. Grant 20060390359 from China Postdoctoral Science Foundation, Also was supported by Grant No. 2006AA01Z334 from The National High Technology Research and Development Program of China(863 Program)

References [1] A.Barr, B.Currin, S.Gabril, and J.Hughes. Smooth interpolation of orientations with angular velocity constraints using quaternions. In Computer Graphics (Proc. of SIGGRAPH'92), pages313-320, 1992.

Interpolating Solid Orientations with a

C 2 -Continuous B-spline Quaternion Curve

615

[2] Dyn,N., Levin,D., Gregory,J., 4-point interpolatory subdivision scheme for curve design. Computer Aided Geometric Design 4, pages: 257-268, 1988 [3] Gregory M. Nielson. v-Quaternion Splines for the Smooth Interpolation of Orientations. IEEE Transactions on Visualization and Computer Graphics (Volume 10, Issue 2), 2004. [4] J.Lee and S.Y.Shin. General Construction of Time-Domain Filters for Orientation Data. IEEE Transactions on Visualization and Computer Graphics (Volume 8, Issue 2), 2002. [5] J.Schlag. Using geometric constructions to interpolate orientation with quaternions. Graphics GEMS II, pages 377-380, Academic Press, 1992. [6] K.Shoemake. Animating rotation with quaternion curves. Computer Graphics (Proc. of SIGGRAPH'85), pages 245- 254,1985. [7] Kenjiro T.Miura. Unit quaternion integral curve: A new type of fair free-form curves. Compueter AIded Geometric Design 17, pages:39-58, 2000. [8] M.J.Kim and M.S.Kim. a C2-continous B-spline Quaternion Curve Interpolating a Given Sequence of Solid Orientations. Computer Animation '95., Proceedings, 1995. [9] M.J.Kim, M.S.Kim, and S.Y.Shin. A General Construction Scheme for Unit Quaternion Curves with Simple High Order Derivatives. Computer Graphics (Proceedings of SIGGRAPH 95 ),29: 369-376, August 1995. [10] M.S.Kim and K.W.Nam. Interpolation solid orientations with circular blending quaternion curves. Computer-Aided Design,27(5):385-398,1995. [11] Ravi Ramamoorthi and Alan H. Barr. Fast construction of accurate quaternion splines. Proceedings of the 24th annual conference on Computer graphics and interactive techniques, Pages: 287 – 292, Year of Publication: 1997. [12] Samuel R. Buss and Jay P. Fillmore. Spherical averages and applications to spherical splines and interpolation. ACM Transactions on Graphics (Volume 20, Issue 2), 2001 [13] W.Wang and B.Joe. Orientation interpolation in quaternion space using spherical biarcs. In Proc. Of Graphics Interface’93, pages 23-32, 1993.

Texture-Meshes in Digital Museum: Octree-Initialized Progressive Transmission with Feature Preservation Xuewei Zhao, Xukun Shen, and Yue Qi Ministry of Education Key Laboratory of Virtual Reality Technology, Beihang University, Beijing, 100083, China {zhaoxw,xkshen,qy}@vrlab.buaa.edu.cn

Abstract. In order to decrease the data volume of 3D textured models in digital museum with significant features preservation, a new simplification method for texture-mesh based on triangle collapse is presented. By introducing the concepts of geometry and texture boundary, the quadric error matrices are improved, and constraint strategies are adopted to retain the feature of geometry boundaries and texture attributes during the simplification process. In addition, a method combining octree-based compression with progressive mesh is given to provide a continuous multi-resolution representation of 3D models. The proposed methods have been successfully applied to progressive transmission and representation system of 3D models in digital museums.

1 Introduction Digital Museum is an information system serving over the Internet. With modern information technology, digital museums are able to collect and manage all aspects of cultural heritage, and provide a variety of services including exhibition, education and research. 3D model is considered as the fourth form of digital media[1] apart form audio, video and image. Therefore, in addition to traditional multimedia form, using interactive 3D technology to present precious relics or historical scenes is one of the most popular and competing technologies used in digital museums in recent years. However, the data volume of 3D models is usually very large, which leads to difficulties in storage, transmission and rendering. Therefore, freeing the restraint of the network transmission is an urgent demand for the 3D display in digital museum. The traditional transmission mode of single-resolution models is “Download-andPlay”. All relevant data of 3D models will be downloaded to the client before rendering, so the disadvantage is that the response time is too long. Hoppe developed an idea of progressive mesh[2]. The method uses multi-resolution models, that is, to transmit a rough outline of the original model first, and then a gradually refined data to improve the details of model. In this way, it is possible to respond to users within a very short time. This can be classified and termed as a progressive transmission method with the mode of “Download-while-Play”. To divide multi-resolution models, simplification is required. Schroeder[3] used a vertex decimation method to decrease of triangle meshes; Hoppe[2][4] presented K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 616–627, 2007. © Springer-Verlag Berlin Heidelberg 2007

Texture-Meshes in Digital Museum

617

progressive meshes with edge collapse method based on optimization algorithm; Isler[5] used edge collapse and triangle collapse to produce real-time LOD(Level-ofDetail) models; Garland[6] used quadrics to control simplification error; Zhou[7] combined Isler and Garland’s method to provide a new triangle collapse algorithm; Fahn[8] made a reservation of boundary edge and color using quadric error; Yang[9] improved progressive transmission through an optimum geometry/texture multiplexing method. The idea related to simplification is compression, which means using fewer bits to express 3D models with mesh topology retained in order to lower the transmission time. Some researchers combined progressive and compression encoding to present a concept of progressive compression meshes. However, most methods mentioned above mainly handle regular models without attributes. In practice, 3D models comprise geometry, topology as well as color, texture and other attributes, and there may also be boundaries and holes. All the feature information is important for maintaining the appearance and reality of 3D models. Hence, they must be retained during processing. To solve the above problem, this paper presents a simplification algorithm based on triangle collapse to retain the geometry and texture features. Besides effective control of the geometric error between coarse mesh and original mesh, the information of boundaries and texture attributes of the model is also preserved better. In addition, with the principle of progressive mesh and geometry compression, an octree-initialized progressive texture-mesh is constructed, which supports smooth transition of various LOD models. The method can be applied to the digital museum with progressive transmission solutions of 3D textured models. Section 2 illustrates the simplification algorithm. The construction and transmission of octree-initialized progressive texture-mesh are described in Section3. The last 2 sections conclude the paper with experimental results and list the tasks remaining to complete the project in the near future.

2 Simplification with Boundary Preservation For the sake of simplicity, we introduce some basic definitions first. 2.1 Definitions We define a group of triangles along the same public edge and the adjacent vertex as triangle meshes TM , thus TM can be expressed by duple (V , T ) composed of a vertices set V = (v0 , v1 , " , vn ) and a triangles set T = (T0 , T1 ," , Tm ) . Definition 1. For each vertex

vi in TM , the collection of all triangles Tik that uses

vi as a vertex, is called a triangles set Pi that is correlative to vi . Definition 2. For each triangle Ti in TM , the union of all triangles set correlative to

3 vertices of Ti , is called a triangles set Ci that is correlative to Ti .

618

X. Zhao, X. Shen, and Y. Qi

Definition 3. For each edge in TM , if the edge is owned by one triangle only, it is a boundary edge, and the owning triangle is called a boundary triangle. Definition 4. For each non boundary triangle Ti in TM , if there is at least one vertex on the boundary edge, it is called a corner triangle, and the vertex is called a corner vertex. Definition 5. For each triangle Ti in TM , if none of its vertices lies on the boundary edge, it is called an internal triangle. Definition 6. For each triangle Ti in TM , if the color difference between Ti and its correlative triangles exceeds a pre-specified threshold τ , it is claimed as a colory triangle. 2.2 Triangle Collapse

This paper exploits a mesh simplification algorithm based on triangle collapse. The implementation of triangle collapse operation merges 3 vertices into 1 vertex, and thus simplifies the original mesh, as shown in Fig. 1. v6

v 7

v6

3

4

v5

3

4

tcol

2

5 Ti

6

v7

v5

v0

1

6

1

vi

v2 v8

v1 7

8 9

v4

vsplit

7

v8

9

v4

v9

v9

Fig. 1. Triangle collapse. 3 vertices v0 , v1 , v2 of triangle Ti merge to 1 vertex vi . The reversal operation is vertex split.

Our algorithm equips each Ti in TM with a quadric matrix Qi . If Ti collapses to one vertex vi = [ xi yi zi 1]T , the quadric error will be calculated and then a Ti with the smallest error is chosen for the current collapse operation. Here we define the quadric error ε (Ti ) as the square sum of distance between vertex vi and each triangle plane in Ci :

ε (Ti ) = ε ([ xi yi zi 1]T ) =

∑ (p

p∈Ci

T

vi )

2

(1)

In (1), p = [ a b c d ]T stands for the plane equation ax + by + cz + d = 0 ( a 2 + b 2 + c 2 = 1 ) of each triangle in Ci correlative to Ti . So (1) can be converted into standard quadric form as below:

Texture-Meshes in Digital Museum

ε (Ti ) =

∑ (v

p∈Ci

i

T

p)( pT vi ) =

∑v

i

p∈Ci

T

( ppT )vi = vi T ( ∑ M p )vi p∈Ci

619

(2)

In (2), M p = ppT is a symmetric 4×4 matrix denoting the distance between an arbitrary vertex in TM and plane p :

⎡ a 2 ab ac ad ⎤ ⎢ ⎥ ab b 2 bc bd ⎥ M p = ppT = ⎢ ⎢ ac bc c 2 cd ⎥ ⎢ ⎥ 2 ⎣⎢ ad bd cd d ⎦⎥

(3)

The quadric matrix Qi is the sum of M p form all triangles correlative to Ti ,

∑M

p∈Ci

p

; and vi is selected to make ε (Ti ) minimum.

2.3 Geometry Boundary Preservation

Some models contain feature information such as boundaries and other special holes, e.g., eye socket of skull, and these geometry hole boundaries have special meanings that need to be retained in the simplification process. For this reason, we mark the original triangles by 3 types according to different geometry feature by Definition 3, 4 and 5, as shown in Fig. 2.

Fig. 2. Mark all the original triangles into 3 types: internal, corner and boundary

In order to retain the boundary edges, and thus control the boundary triangles not to be folded, we adjust the weight of quadric of each triangle according to the different markings. The quadric error of boundary triangles should be increased so that boundary triangles have the biggest errors while internal triangles achieve the smallest ones:

ε (Tboundary ) = Max(ε (Ti )) i

p∈Ci

(4)

However, the disadvantage of this method is that the collapse of corner triangles will lead to the emergence of the gaps. Such problems can be resolved by restricting

620

X. Zhao, X. Shen, and Y. Qi

the new vertex to be a corner vertex of the triangle. In this way, corner triangles will be contracted to points on the boundary, thus lead to no collapse of the border. 2.4 Texture Boundary Preservation

For each triangle in the original mesh, the greater color difference between it and its neighborhood triangles, the more it contributes to the overall attribute. So it could be counted as a color boundary. To distinguish this feature form geometry boundary triangle, here we define it as a colory triangle. Usually, for textured models, texture mapping is mapping a texture image to the surface of the geometry mesh. There are 2 types of mapping modes between geometry mesh and texture: vertex correspondence and triangle correspondence. As shown in Fig. 3.

B

A

a

d C

b c D

(b) Triangle Correspondence

(a) Vertex Correspondence

Fig. 3. 2 types of mapping between geometry/texture: (a) texture coordinate corresponds to geometry vertex; (b) texture triangle corresponds to geometry triangle

For the former type, we use texture coordinate to find the color vector [ri gi bi ]T ( 0 ≤ ri , gi , bi ≤ 1 ) corresponding to the vertex in the texture image, thus to obtain a color value for each triangle. Here we use the average pixel value of three vertices for the texture color value of the triangle: 2

[r g b]T = (∑ [ri gi bi ]T ) / 3

(5)

i =0

And then we mark all the triangles according to the criteria of Definition 6. The threshold τ can be selected by different requirements of preservation priority on geometry/attribute. For Ti and a correlative triangle Tp , with corresponding color vectors [ ri gi bi ]T , [rp g p bp ]T , if

∑ (| r − r

p∈Ci

i

p

| + | gi − g p | + | bi − bp |) > τ

(6)

We can label Ti as a colory triangle, and use the method similar to that of dealing with boundary triangles. Thus its collapse error is increased to preserve important attributes. For the latter type, centers of texture triangles that correspond to geometry triangles are calculated first through texture coordinates. And then texture colors of geometry triangles are substituted by pixel values of centers:

Texture-Meshes in Digital Museum

[u v]T = ([u0 v0 ]T + [u1 v1 ]T + [u2 v2 ]T ) / 3

621

(7)

Here [ui vi ]T (i = 0,1, 2) stands for the texture coordinate of each vertex, while [u v ]T for the center coordinate of texture triangle. After this, we use the same method to preserve colory triangles. 2.5 Algorithm Outline

1) Pretreatment on the original mesh: establishing vertex list, triangle list and correlativity; all triangles will be divided into boundary, corner or internal triangles; 2) For each Ti in TM , calculate quadric matrix Qi and color value C (Ti ) ; 3) Calculate color differences and mark all colory triangles; 4) For each Ti in TM , calculate the new vertex position vi through its quadric Qi , and calculate collapse error by vi T Qi vi ; increase errors of boundary triangles and colory triangles; 5) Arrange triangles by collapse errors; 6) Select the triangle with the smallest error in sequences and implement collapse operation; update all relevant information; 7) Repeat Step 6), until the triangle series are empty or the approximation tolerance is reached.

3 Octree-Initialized Progressive Texture-Mesh 3.1 Progressive Mesh (PM)

In this paper, we use triangle collapse to simplify mesh models. Similar to Hoppe’s approach, we implement a triangle collapse operation on M to gradually reduce the resolution, and finally to the base mesh M 0 . We record the vertex coordinates of disappeared triangles and the sequence number of correlative triangles as the detailed refinement information. With the reversal of this sequence, we implement the contrary operation of collapse, vertex split to re-insert the vertices and triangles to the base mesh, thus to reconstruct M from M 0 . The process can be described as follows: Simplification: tcoln−1 tcol0 (M = Mn ) ⎯⎯⎯ →Mn−1 " ⎯tcol ⎯1⎯ →M1 ⎯⎯ ⎯ →M0

(8)

0 1 n−1 M0 ⎯vsplit ⎯⎯ →M1 ⎯vsplit ⎯⎯ → "Mn−1 ⎯vsplit ⎯⎯ →(Mn = M)

(9)

Refinement:

Thus, the expression {M 0 ,{vsplit0 ," , vsplitn −1}} composes a representation of PM.

622

X. Zhao, X. Shen, and Y. Qi

For the textured models, the expression of vertex coordinate can be extended to vi ( xi , yi , zi , ui , vi ) , with xi , yi , zi for the vertex coordinate and ui , vi for the texture coordinate. With the new expression, attribute information can be recorded in PM file, thus to provide a recovery at the same time. 3.2 Octree-Based Comprssion (OT)

Octree-based encoding[10][11][12] is an efficient geometry compression scheme. The keystone is to quantify the geometry coordinates of the vertices by bits, including the following 3 steps: 1) Reorganize all vertices of the model by octree division, until the finest tree nodes reach the quantified accuracy. First, carry the bounding box through recursive subdivision, and generate an octree implying the spatial location of vertices. Each node records the number of vertices in the sub-trees. Nodes that do not contain vertices will not be subdivided any more, while others containing vertices will be subdivided until the node is empty or it contains only one vertex which reaches the peak level. Therefore, the accuracy of bounding box is determined by the accuracy of quantification and the coordinates can be restored by the most sophisticated level of tree nodes. A 2D example is described in Fig. 4. 8

2

0

1

0

0

1

1

0

0

0

0

0

0

1

5

0

1

1

1

2

3

0

0

0

0

0

1

1

1

0

1

0

0

0

0

1

Fig. 4. A 2D example of octree-based division. Vertices in the left are divided and organized to the tree right.

2) Traversal the octree by layers in a coarse-to-fine way, and produce 3 different types of output data flows to describe the vertex location. According to the numbers of vertices in a tree node, for each empty node, use 1bit to mark whether it is a single node; for a single node, use 3 bits to mark which sub-node contains only one vertex; for each non-empty and non-single node, use 8 bits to mark whether its sub-node is empty; Loop until reaching the most accurate level of nodes; 3) Use arithmetic encoding to encode these 3 data flows. 3.3 Progressive Transmission with PM and OT

For the models with high accuracy, the size of base mesh is still too large after simplification. What’s more, for the texture-mesh models, with restrictions imposed by the boundary-preserved method mentioned above, its base mesh will remain at a certain level of detail that can no longer be simplified. This will increase the response time and is bad for real-time browsing. In solution, we first analyze the characteristics of 3 transmission methods.

Texture-Meshes in Digital Museum

623

approximation Mn

OT

r tte be

PM

M0

a

b

c

d

time (bits)

Fig. 5. Approximation/time diagrams of 3 types of transmission methods. Vertical line for NonP, curve for PM, and ladder for OT.

Fig. 5 describes the relationship between changes in the model approximation over time. The horizontal axis shows the transmission time by gradually transmitted bits, while the vertical axis represents the similarity of the model. The higher the resolution is, the greater the similarity is. The traditional non-progressive method is shown with black line in the figure, which indicates that users need to wait a very long time (b bits) until all model’s decoding. The shortcoming is that the response time is too long. Progressive mesh is a fine-grained multi-resolution transmission program, as the black curve shown in Fig. 5. Assuming it consumes a bits to transmit the initial mesh M 0 , after the base mesh is decoded, we can begin to render the crude model and refine it. Besides, PM provides higher model accuracy than other methods, but it is slower in total transmission time (needs d bits) compared to non-progressive approach. Octree-based encoding is a coarse-grained multi-resolution transmission strategy. As shown in the gray ladder curve in Fig. 6. With smaller data volume after compression, it has a very short response time; and with the transmission and decoding of refinement information, the geometry data update in batches and the resolution improves layer by layer. The total time it consumes (c bits) is slightly longer than non-progressive approach but shorter than PM. The disadvantage is that the resolution upgrade is not continuous and the accuracy is lower than PM in the transmission process. Progressive mesh provides an effective incremental transmission method for 3D models, which construct a series of multi-resolution models with high similarity; while the compression coding greatly reduces the total transmission time better. Thus, it is very reasonable to combine the two approaches, and carry octree-based compression on the base mesh, as shown in Fig. 6. During the initial stage of transmission which demands a short response time and correspondingly a low model accuracy, we use octree decoding, until the completion of base mesh transmission; after that, we use PM transmission to maintain a relatively high and continuous upgrade in model resolution, whenever to be paused.

624

X. Zhao, X. Shen, and Y. Qi y

5

6

7

4

x

+ z

2

1

3

LOD

0

Octree

Fig. 6. Encoding with PM and OT. Octree division on the base mesh.

4 Experimental Results With the algorithm above, we have simplified and tested several groups of mesh models with different features. 4.1 Boundary-Preserved Simplification for Bunny

Fig. 7 shows a bunny with a number of holes and cracks at the bottom, it can be seen that with the boundary preservation, the geometry feature can be maintained even when simplified by 99%.

Fig. 7. Boundary preservation of bunny

4.2 Texture-Preserved Simplification for Capsule

Fig. 8 shows the simplification result of a capsule with/without texture boundary preservation, we can see that with the method, the division of the texture and color has been nicely maintained.

Fig. 8. Texture preservation of capsule

Texture-Meshes in Digital Museum

625

4.3 Octree-Initialized Progressive Transmission

Integrating our methods, we have tested several models with octree-initialized progressive transmission. Fig. 9 shows the process. After the initialization of octree, the base mesh is retrieved, which maintain not only the high geometry similarity, but also the features of boundary, contour and texture.

Fig. 9. Progressive transmission of P-47 and Kwan-yin

4.4 Results

The method has been implemented in Java and Java3D on PC with P4 3.0GHz CPU and 512MB memory. Table 1 gives the experimental results, and all models contain feature information. Table 1. Experiment data, including the triangle numbers of original mesh/base mesh, file size of source/base mesh(before compression)/base mesh(after compression)/progressive mesh, response time and total transmission time of non-progressive mode/progressive mesh mode/octree-initialized progressive mode.

Models Capsule

Triangle Numbers Org.

BM

12460

830

Transmission Time/s BM BM NonOT NonOT Src. PM PM PM (before) (after) P P +PM +PM 0.6 0.05 0.02 0.3 2.0 1.1 0.2 2.0 3.0 2.5 File Size/MB

Kwan-yin 40384 3158 2.5 Bunny P-47

69451

693

0.50

0.25

Response Time/s

1.5

8.5

2.9

0.8

8.5

12.0 11.0 15.0 13.0

2.5

0.59

0.30

1.7

8.0

2.1

0.2

8.0

96720 8428 7.8

1.10

0.47

2.3

14.0

3.0

0.5

14.0 30.0 28.0

626

X. Zhao, X. Shen, and Y. Qi

From the results we can see that with octree encoding, base mesh generated by simplification is smaller in data volume, which greatly reduces the network transmission delay and shortens the response time.

5 Conclusion and Future Work Based on the triangle collapse algorithm, we present a simplification method for texture-mesh models with geometry and texture feature preservation. Through a series of collapse operations and final compression of base mesh, a multi-resolution mesh is generated, and an octree-initialized progressive transmission used in digital museum for 3D exhibition is also accomplished. This method can not only maintain the surface quality of the geometry similarity, but also preserve the boundary edges as well as color attributes to ensure the reality of 3D models. In addition, the method can also easily be extended to the preservation of a variety of scalar attributes, such as normal, surface material, transparency. However, with the increasing volume of information, there will be bounds in terms of increasing the amount of computation and memory consumption. So how to design a more effective, rapid and efficient data structure and algorithm is a key element in the future; besides, for greater precision textured models, we need to take better strategy of texture mapping and transmission in order to display them synchronously with the geometry progressive mesh.

References 1. Schroeder P., Sweldens W. Digital Geometry Processing. Course Notes for ACM Siggraph. Los Angeles, California, 2001 2. Hoppe H. Progressive Meshes. ACM Computer Graphics (SIGGRAPH’96), 1996, 30(1): 99-108 3. William J., Jonathan A. and William E. Decimation of Triangle Meshes. Computer Graphics (SIGGRAPH’ 92), 1992, 26(2): 65-70 4. Hoppe H., DeRose T. and Duchamp T. Mesh Optimization. ACM Computer Graphics (SIGGRAPH’ 93), 1993, 27(1): 19-26 5. Isler V., Lau R., Green M. Real-time Multiresolution Modeling for Complex Virtual Environments. Proceedings of ACM Symposium on Virtual Reality Software and Technology. HongKong: ACM Press, 1996: 11-19 6. Garland M., Heckbert P. Surface Simplification using Quadric Error Metrics. Computer Graphics (SIGRAPH’97), 1997, 31(3): 209-216 7. Zhou K., Pan Z., Shi J. Mesh Simplification Algorithm based on Triangle Collapse. Chinese Journal of Computers, 1998, 21(6): 506-513 8. Fahn C. Polygonal Mesh Simplification with Face Color and Boundary Edge Preservation using Quadric Error Metric. Proceedings of IEEE 4th International Symposium on Multimedia Software Engineering. California: IEEE, 2002: 174-181 9. Yang S., Shen M., Kuo C. Progressive Coding of 3D Textured Graphic Model via joint Mesh-texture Optimization. ICASSP, 2004

Texture-Meshes in Digital Museum

627

10. Peng J., Kuo C. Geometry-guided Progressive Lossless 3D Mesh Coding with Octree (OT) Decomposition. ACM Transactions on Graphics, 2005, 24(3), 609-616 11. Schnabel R., Klein R. Octree-based Point-cloud Compression. Eurographics Symposium on Point-Based Graphics, 2006 12. Huang Y., Peng J., Kuo C. Octree-based Progressive Geometry Coding of Point Clouds. Eurographics Symposium on Point-Based Graphics, 2006

A Robust Watermarking Based on Feature Points Xiaoqing Feng1,2, Li Li3, and Zhigen Pan1 1

State Key Lab of CAD&CG, Zhejiang University, Hangzhou 310027, China 2 Institue of Computer Application Techniques,College of Information, Zhejiang University of Finance&Economics, Hangzhou, 310018, China 3 Hangzhou Institute of Electronic Engineering, Hangzhou, 310018, China {fengxiaoqing,lili,zgpan}@cad.zju.edu.cn

Abstract. The paper present a novel watermarking algorithm, which is obtained based on a watershed segmentation, one image watermarking algorithm and the addition property of Fourier transform. Firstly, some feature points of 3D meshes are extracted by using an effective watershed segmentation algorithm. Furthermore, in order to keep the mesh transparency, watermark information is preprocessed by using an image watermarking algorithm. Finally, digital watermarking is embedded into those feature points. During extraction, the target model is registered by principal component analysis. We have applied our method to a number of 3D mesh models of different geometric and topological complexity. Those experimental results prove that the algorithm is robust against a variety of attacks, such as rotation, translation, scaling, combinations of the above and re-sort. Keywords: Watershed segmentation, Feature points, Image watermarking, Fast Fourier transform.

1 Introduction With the development of internet, some information, such as text, image, sound, video and 3D model, is exchanged and transferred conveniently on internet. In this environment, an important issue is the copyright protection of these digital data and prevent unauthorized duplication of them. Digital watermarking technique is one of effective methods of protection copyright and information security. However, in comparison to audio, video, and still-image data, some challenges exist for watermarking 3D mesh models [1,2]. For example, arbitrary meshes lack a natural parameterization for frequency-based signal processing, and thus they cannot adopt the corresponding image watermarking algorithms that are based on the frequency domain, such as the wavelet transforms and the Fourier transforms which most image watermarking algorithms are based on. In 1997, Ohbuchi[3] et al. first introduces the issue of 3D watermarking, Although this algorithm is not robust against some attacks, such as mesh simplification and remeshing , it gives some important conceptions such as unobtrusive, robustness, space efficiency and embedding primitives. Beneden et al. [4] also proposed an algorithm that adds a watermark by modifying the normal distribution of the model K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 628–637, 2007. © Springer-Verlag Berlin Heidelberg 2007

A Robust Watermarking Based on Feature Points

629

geometry. Kanai et al. [5] presented a watermarking scheme for 3D polygons using the wavelet transform. Their paper is the first one that applied a transformed domain watermarking approach to 3-D meshes. Yet, this application is restricted to 4-to-1 topological class of mesh. However, to most of watermarking algorithms[6-11], watermarking is embedded into some regions that are un-special meaning according to human perception. In this paper, we propose a novel watermarking algorithm that embeds data into shapes of 3D polygonal meshes. In this algorithm , watermarking is not embedded into the whole meshes but into some feature points. Here, feature points are boundaries of some useful segments by using an effective watershed segmentation algorithm. Furthermore, in order to keep the mesh transparency, we do not directly embed watermarking into vertex coordinate of mesh but into a gray image. According to the addition property of Fourier transform, a change of spatial domain will cause a change in the frequency domain, which will allow watermarks to be extracted from the watermark information. Finally, the watermark information is scaled down and embedded in those feature points. During extraction, the target model is registered by principal component analysis. We have applied our method to a number of 3D mesh models of different geometric and topological complexity. Those experimental results have proved that the algorithm is robust against a variety of attacks, such as rotation, translation, scaling , combinations of the above and re-sort. The rest of this paper is structured as follows. In Section 2, we will present embedding pipeline and extracting one of our new mesh-watermarking algorithm based on feature points. A watermarking embedding scheme is explained in detail in Section 3, and a watermarking extracting method is discussed in Section 4 . In section 5, we display our experiments results, and we will conclude this paper with summary and conclusion in Section 6.

Fig. 1. Embedding pipeline

630

X. Feng, L. Li, and Z. Pan

2 The Watermarking Algorithm To most watermarking schemes, watermarking information is not embedded into feature points ,but into some non-essential vertexes or areas , which are according to human perception. However, these feature points are especial important and sensitive to model’s changes, even the tiniest change. So, we consider feature points’ characteristic and develop a new digital watermarking algorithm based on feature point. Firstly, some surface patches are generated by using a effective watershed algorithm. Then , we obtain some feature points , namely, the boundaries points of surface patches. Secondly, watermark information is embedded into these feature points. Finally, A mesh model embedded watermark is got. The watermarking pipeline consists of two separate pipelines: an embedding pipeline (Fig. 1) and an extraction pipeline (Fig. 2).

Fig. 2. Extracting pipeline

3 Watermarking Embedding In this section, we describe the novel framework of watermarking embedding. First we present a watershed segmentation of 3D meshes which is to detect feature points and later, in successive sections, we describe each step of watermarking embedding in details. 3.1 Watershed Segmentation of 3-Dimensional Meshes 3D mesh segmentation has had numerous applications in the past. Moreover, several existing applications benefit from segmentation. Good examples are feature extraction, decimation, adaptive subdivision[1], and surface fitting for reverse engineering. In this section, we use the feature extraction function of mesh segmentation. Of course, there are some methods of mesh segmentation, such as watershed, clustering, voronoi diagram and so on. In our watermarking system, we

A Robust Watermarking Based on Feature Points

631

adopt a watershed segmentation algorithm to partition 3D surface into useful segments ,and detect feature points, which are boundaries of useful parches on 3D meshes and used for embedding watermarking. The watershed algorithm that we shall present is based on Mangan’s method, but there are some different in detail. We use [12] to estimate the principal curvatures, k and k , and the associated principal directions T and T for each vertex Vof M. max

min

1

2

Furthermore, during the mesh segmentation , we adopt the mean curvature ( kmax  kmin ) H 2 as the height function f. Surface patches are2 obtained by H= using the watershed algorithm, and these patches have characteristic:

）

；

1 submesh is diffeomorphism to disk 2) submesh is flat enough; 3) Minimum convex closure of submesh must be approximate nicely to submesh. In real word, a mesh model is usually expressed by a pair of linear table. M={V,F},V={v :1 ≤ i≤ n } is a table of vertex, F={ f :1 ≤ k≤ n } is a table of triangle. i

i

k

F

One ring of vertex v is defined as Fig. 3 in this paper[13]. i

v j

Fig . 3. One ring of vertex vi

The watershed algorithm proposed in this paper is designed referenced to one in [14]: 1. 2. 3. 4. 5. 6.

compute normal vector ,principal curvature and the mean curvature at each vertex. attach an unique label to each local minima. classify each flat area as a minimum or a plateau. allow each plateau to descend until a labeled region is encountered. descend other remaining unlabeled vertices and join them to labeled regions. merge regions whose watershed depth is below a threshold.

632

X. Feng, L. Li, and Z. Pan

Fig. 4. Patches and feature points

The result of segmentation to cat model (4539 vertexes, 8923 triangles) is showed as Fig. 4. 3.2 Watermark Embedding The watermark embedding algorithm illustrated in Fig. 1. In our group, Li has proposed a valid image watermarking algorithms, which was based on fast Fourier transform[15].And it can be described in detail as follows: Step 1: In order to embed some meaningful watermarking information , we should encode watermarks into the binary sequence W, W = {wi (i) (0,1)},i = 1......24. The rule of encoding is that watermarks are meaningful characters and every character is encoded into a 6 bit sequence in which the total number of digital "0"is 5 and the total number of "1" is 1. For example, character "a"of watermarks is encoded as [1 0 1 0 1 0] character "b" [0 0 0 0 1 0] ; and character "" as[0 0 0 1 0 0]

∈

Step 2: In order to improve the robust of watermarking, we embed watermarking into a gray image A, which size is defined 32×32. To obtain FFT spectrum matrix B(x,y) and phase matrix (x,y), x={1, 2.....32}, y={1, 2...32} . A is made Fourier transformation. Step 3: Embed w into B(x,y) to get B*(x,y) according to formula (1) i

B*((i+16),(12 +j)) =B((i+16),(12 +j)) +d

(1)

here i =2,5, −5,−2; j =1,2...6; Step 4: To obtain watermarked image I * , we transform inversely to B *(x, y) and C(x,y)

A Robust Watermarking Based on Feature Points

633

Step 5: We get watermarking information W * according to formula (2). Here, we should make watermarking information W* enough small to maintain transparence of the mesh model. Thus, τis the weight, according to our experiments, transparence of mesh will be kept nicely when τ is 0.05.

W *(i, j) =(I *(i, j) −I (i, j)) ×τ;i =1,2...32; j =1,2...32

(2)

Step6: Sort these feature points according to x -coordinates increased by degrees, and embed watermark W * into these feature points.

4 Watermark Extracting This watermarking algorithm is a non-private one, it need original model during extraction processing. Step1: In this step, we firstly process a tested model using the principal component analysis (PCA) to register[16]. Then, comparing tested model with original one M , magnified mesh M ' can be obtained according to formula (3). g(x)x is the xcoordinate of each vertex on M , x is the x-coordinate of each vertex on M'f () . f '( )is magnified x-coordinate.

f '(x) =g(x) +( f (x) −g(x)) ×106

(3)

Step2: The watershed algorithm is carried out to mesh M ' , and some feature points are extracted. We sort these feature points according to x -coordinate increase by degrees. Step 3: A 32×32 matrix A is made up of parts of feature points. And we carry out Fourier transformation to matrix A to obtain FFT spectrum matrix B (x,y) Step 4: seletct 24 value of matrix B(x,y), and edited them into a sequence W.Then, W is divided averagely into 4 parts. Finally , we can get watermark according to the rule of watermarking encoding.

5 Results In this section, we demonstrate that our watermarking scheme is effective in the presence of various real-world attacks. We implement this watermarking algorithm by VC++6.0 and MATLAB 6.0. Watermarking information is 4 numbers, such as”1234”. Test models are cow model 4539 vertexes, 8923 triangles , cat model 4539 vertexes, 8923 triangles , horse model 5002 vertexes, 9721triangles , venus model 2793 vertex, 5552 triangles .

（

）

（

（ ）

）

（

）

634

X. Feng, L. Li, and Z. Pan

In order to resist attack of re-sort, we embed watermark into x-coordinates of feature points increased by degrees. Fig. 5 and Fig. 6 show different results of initial and re-sort x -coordinates. Our experiments show that watermarking can be effectively extracted after suffering attacks (Fig. 7). Table.1 shows results of attacks to some 3D models. These attacks include noise, rotation, translation and scaling. Furthermore, to some combined attacks of translation and rotation, translation and scaling , the watermarks are still extracted accurately. The symbol of “

√” means that watermark will be

extracted completely.

Fig. 5. Initial x-coordinates of feature points

Fig. 6. Re-sort x-coordinates of feature points

Experiments indicated that this watermarking algorithm can endure strongly attacks of scaling and translation. According to the addition property of Fourier transform, such as formula (4) and (5), a change of spatial domain will cause a change

A Robust Watermarking Based on Feature Points

rotation ( x:0q, y:45q, z:45q)

scaling ( x:0.6, y:0.6, z:0.6)

635

translation ( x:0, y:10, z:8)

Fig. 7. Experiments results of cat model (4539 vertexes, 8923 triangles) Table 1. Results of attacks to some 3D models

attack methods no attack rotation

cat

horse

venus

Ĝ

Ĝ

Ĝ

Ĝ

Ĝ

Ĝ

Ĝ

Ĝ

Ĝ

( x : 0, y : 10, z : 8)

Ĝ

Ĝ

Ĝ

combined attacks: rotation ( x : 0q, y : 45q, z : 45q) and

Ĝ

Ĝ

Ĝ

scaling ( x : 0.6, y : 0.6, z : 0.6) combined attacks: rotation ( x : 0q, y : 45q, z : 45q) and

Ĝ

Ĝ

Ĝ

translation ( x : 0, y : 10, z : 8) combined attacks: translation ( x : 0, y : 10, z : 8) and

Ĝ

Ĝ

Ĝ

scaling ( x : 0.6, y : 0.6, z : 0.6) re-sort

Ĝ

Ĝ

Ĝ

( x : 0q, y : 45q, z : 45q) scaling

( x : 0.6, y : 0.6, z : 0.6) translation

in the frequency domain[17], which will allow watermarks to be still extracted after mesh models experience some attacks of scaled and translation.

F [β×f (x1, x2)] = β×F [ f (x1, x2)] F [β+f (x1, x2)] = F [β] + F [ f (x1, x2)]

(4) (5)

636

X. Feng, L. Li, and Z. Pan

6 Conclusion In this paper, we proposed and evaluated a watermarking algorithm for a 3D model. A watershed algorithm is effectively applied to select feature points , which are robust to stand attacks. Experiments showed that the watermark produced by using our proposed method is resistant, to a certain extent, against similarity transformation, such as rotation, scaling, translation, re-sort and combined attacks. In the future, there are still many spaces to improve the watermarking algorithm, such as making the algorithm blind and reducing its computation time. Acknowledgments. This work is supported by the National Natural Science Foundation of China (No. 60473111). Thanks to the discussion with Ph.D Shusen Sun. The image watermarking algorithm is provided by Ph.D LiLi.

References 1. BenedensO. Geometry-based watermarking of 3D models.IEEE Computers and Applications 1999;19(1):46–55. 2 Praun E, Hoppe H, Finkelstein A. Robust mesh watermarking.In: SIGGRAPH Proceedings, 1999. p. 69–76. 2. Ohbuchi R., Masuda H., Aono M.: Watermarking Three-Dimensional Polygonal Models. 3. In:Proceedings of the ACM International Conference on Multimedia ’97. Seattle, USA, 4. November 10-13, (1997) 261–272 5. O.Benedens, “Watermarking of 3d-polygon-based models with robustness against mesh simplification,” in Proceedings of SPIE, vol. 3657, no. 3657, CA, 1999, pp. 329–340. 6. S. Kanai, H. Date, and T. Kishinami, “Digital watermarking for 3d polygons using multireso-lution wavelet decomposition,” in Proc. Of International Workshop on Geometric Modeling: Fundamentals and Applications, Tokyo, Japan, 1998, pp. 296–307. 7. Koller D., Turitzin M., Levoy M., etc.: Protected Interactive 3D Graphics Via Remote Rendering.In Proceedings of the 2004 SIGGRAPH Conference,23 (3),(2004): 695-703 8. Praun E., Hoppe H., Finkelstein A.: Robust mesh watermarking. in Proc. of ACM SIGGRAPH 99, New York, (1999):49–56 9. Yin K. K., Pan Z.G., Shi J. Y.: Texture watermarking in VRML scenes. Journal of Engineering Graphics, (3), 2003:126–132(in Chinese). 10. Ohbuchi R., Mukaiyama A., Takahashi S.: A Frequency-Domain Approach to Watermarking 3D Shapes. In Proc. EUROGROGRAPHICS 2002, Saarbrucken, Germany, Sept. 2-6, (2002) 11. Sun S. S., Pan Z. G., LI L., Shi J. Y.: Robust 3D model watermarking against geometric transformation. CAD/CG'2003, Oct.29-31, Macao, (2003):87–92 12. Li L., Zhang D., Pan Z. G., Shi J. Y., etc: Watermarking 3D mesh by spherical parameterization. Computers & Graphics 28(6), (2004): 981–989 13. Taubin G, etc. Estimating the tensor of curvature of a surface from a polyhedral approximation [A]1 In : Proceedings of the 5th Inter2 national Conference on Computer Vision, Washington , DC: IEEE Computer Society , 1995. 52 : 902 907 14. ShenHui cun ,Li jianhua,Zhou laishui. Estimation of triangular mesh vertex normal vector. Computer Engineering and Applications. 2005 .Vol.41. No.26. p12-15 15. Mangan A P, Whitaker R T. Partitioning 3D surface meshes using watershed segmentation [J]. IEEE Trans Visualization and Computer Graphics, 1999, 5(4): 308-321

．

～

A Robust Watermarking Based on Feature Points

637

16. Li Li, Research on Watermarking of Digital Image and 3D Models, PhD Thesis, Zhejiang University,2004. Shlens. A Tutorial on Principal Component Analysis. 17. Jonathon http://www.cs.cmu.edu/~elaw/papers/pca.pdf 18. WangyuJian, LI Li, Pan Zhi-geng. An Effective Non-blind Watermarking Subdivision Surface Algorithm. Application research of computer. 2006.Vol.23. No.4. p123-125. (in chinese)

Surface Animation Based on 2D Editing Jian Gao, Caiming Zhang, and Zhiquan Feng Department of Computer Science and Technology, Shandong University, 250061 Jinan, China [email protected]

Abstract. To perform 3D animation with carving on surfaces of 3D models is a new branch of 3D modeling. A set of new methods are carried out in this paper. One based on recursion can solve the image transformation problem with curve constrain. The edge corroding method can gain a relatively external depth image from a 2D image. Also another based on partial resampling is used to paste a 2.5D model to any other 3D model, and has made some good results. The work of this paper has greatly shortened the path from 2D editing to 3D surface animation. Keywords: 3D carving, transformation with curve constrain, edge corroding, surface overlaying, partial resampling.

1 Introduction 3D carving is a new research area in 3D modeling. According to different geometry Meta, Li [1] classified the 3D carving methods into four types. They are methods based on point constraint, curve constraint, sphere constraint and hexahedron constraint. The methods based on point constraint is the simplest one, it allows users directly dragging a mesh point to perform transformations. The methods based on curve constraint allow users to draw a curve on the surface of a 3D object, and this curve can be used as a reference to lay out concave and protruding. The sphere constraint based ones can simulate transformations containing blurred edges and corners. At the same time, the hexahedron constraint based ones can make the effects of clear edges and corners. These methods can fulfill 3D carving tasks, but they can only take the direct input of users, which is a big limitation for use. What users can use are just the fixed stroke styles. This makes it difficult to control the details. Here, we leave the rendering work aside, because there are many software for this job, for example, Adobe Illustrator, 3DS MAX, Maya, etc. Recently, 3D amalgamation techniques [2] come out. It means the cut and paste between 3D models. Arbitrary part of a 3D model can be cut out and pasted onto the surface of another one. Then a new model is created and the combination must be perfect enough to cheat human eyes. This work brings great benefits for 3D modeling. However, there is a precondition that models must be created before amalgamation. To do this, users must spend a lot of time. In our view the definition of 3D carving can be broader. It should contain every method that can perform 3D carving effects. 3D carving can be created based on 2D K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 638–646, 2007. © Springer-Verlag Berlin Heidelberg 2007

Surface Animation Based on 2D Editing

639

pictures, which is a new concept that is provided in this paper. Algorithms for 2D modeling are introduced in Chapter 2. There is a new method for image transformation with curve constraint. Chapter 3 and 4 describe the course from pictures to 3D shapes. During that, the depth image is created with our edge corroding algorithm and 3D amalgamation is fulfilled with our algorithm base on partial resampling.

2 2D Editing Now great amount of image editing software has come out to help people managing pictures. However, for modeling, one prefers to pay much more attention to the 2D transformation techniques. For instance, when we operate the strokes in 2D NPR [3, 4], most of us would like to control the shape of strokes by editing a curve. Yang [5] uses straight lines to approximate the curve, and then make transformation alone them. But there is a shortcoming in it. Good results can only be obtained when the amount of lines is great enough. It brings great deal of waste calculations and can not guarantee time cost. Here, we use our recursive algorithm and cubic Bézier curve to cover the curve constrained stripe transformation problem.

Fig. 1. 2D Image Strip Transformation Controlled by Bézier Curve

The content of Figure 1 is (from left to right): original image domain, transformation curve and target domain, original stroke and transformation result. Here are the steps. Determine the original domain and target domain. According to the definition of transformation curve, we can calculate the number of the steering points in X direction, which mean the points where the increasing relationship between t and X(t) turns different. The steering points number M can be 0, 1or 2. Assuming the control points are Pi (i= 0,1,2,3). Then M can be calculated as follows. 3

P (t ) = ∑ PB i i ,3 (t ), 0 ≤ t ≤ 1

(1)

Bi ,3 (t ) = C3i t i (1 − t )3−i

(2)

i =0

Bézier curve: Bernstein base function:

The Y derivative would be: ∂2P = 3(( − P0y + 3P1 y − 3P2y + P3y )t 2 + (2 P0y − 4 P1 y + 2 P2y )t + ( − P0y + P1 y )) ∂y∂t

(3)

640

J. Gao, C. Zhang, and Z. Feng

Setting

∂2 P =0 ∂y∂t

(4)

We can get a square equation, and then M and the parameter coordinates will be obtained easily. According to M and the parameter coordinates, the Bézier curve can be divided into M+1 parts, and then our recursive algorithm will be applied within each part. The algorithm works as follows. program GetColumn (t1,t2) var Middle : Double; begin // Middle is the average value of t1 and t2 Middle = (GetXCoordinate(t1)+GetXCoordinate(t2))/2.0; // Return, if the Middle column has been obtained if (tag[ Middle ] = true) begin return; end // calculate a column of pixel with interpolation CalculateColumn(Middle); // create the former part of the image recursively GetColumn(t1,Middle); // create the rear part of the image recursively GetColumn(Middle,t2); end This algorithm has a time cost of O(n) and n is the column number of target image. It is easy and robust enough for using. More inspiring, it can manage more than 2 corners fairly well by just connecting more than 2 Bézier curves end by end. This also avoids the management on high rank curves.

3 Depth Calculation Because there is no definite depth information in the ordinary pictures, accurate depth images can not be obtained, and we can only get some relatively external depth images. 2 algorithms are given here to show the calculating process. 3.1 Edge Corroding Method Given a piece of handwriting of a brush pen, the first thing one can think of is the writing strength during each stroke. In another word, one wants to know where the writer uses more strength and where the writer uses less strength. Of course, there are

Surface Animation Based on 2D Editing

641

many aspects which can affect the result of handwriting; however, strength is the major one. A stroke will be thick if the writer uses more strength, otherwise thin. According to this cause, we use our edge corroding method to gain a depth image in the following steps. 1) Turn the original image into black and white, thus the outline can be selected later. 2) Mark up the outline and tag the pixels on the outline. 3) If no pixel is tagged in step 2), the algorithm can end up. Otherwise, evaluate each pixel a depth value, and the value is due to the executing times of step 2). Delete the evaluated pixel and return to step 2). In Figure 2, (A) is the original image, (B) is the black and write result of (A) and (C) is the depth image generated by our algorithm.

(A)

(B)

(C)

Fig. 2. Calculating depth image with edge corroding method

3.2 Pixel Color Based Method This kind of methods has a long history, and has become many new methods. But their essence is the same. They usually seek each pixel in one picture, and find out the range within which color changes. Divide the pixels into different parts and then map them onto different positions in the depth domain. Thus a piece of depth image is obtained.

4 3D Model Generation and Applications 4.1 2.5D Model Generation Using the depth information generated in chapter 3 as the Z data in 3D space, we have got the 3D point set. The next step is to connect the points and make a mesh. Thinking of the points in the picture, we know they are adjacent to each other. So the question becomes a new one, i.e. just determine the connection within each single cell. There are two ways to do this shown in left of Figure 3. A comfortable way to determine is to calculate the difference between the two end points of the two diagonal. Then we choose the diagonal with less difference as the result. The right part of Figure 3 is the final 2.5D model generated by this theory.

642

J. Gao, C. Zhang, and Z. Feng

Fig. 3. Linking within a cell and the 2.5D model

4.2 3D Pasting The 2.5D model generated in 4.1 express the information contained in the 2D picture vividly. However, if we want to make it more valuable, more effort must be done. Naturally, it will be more useful if it can be pasted easily onto the surface of another 3D model. Usually models exist in two forms. One is mesh and the other is parameter surface. There is great difference when managing the two cases. 4.2.1 Surface Overlaying Method If the 3D model is in the form of parameter surface B(u, v), it must be turned discrete first. Because a parameter surface is continuous and the 2.5D model is discrete, we must give them a unified form. Thinking of the displaying aspect, it is better to make the parameter surface discrete. Then the 2.5D model can also be treated as a surface N(u, v). The final surface will be F(u, v) = B(u, v) + N(u, v) after a surface overlaying. 3 Steps are as follows.

⊿

1) Divide the parameter surface into tiny triangle set{ i}. 2) Use a parameterization to map the points of the 2.5D model into the parameter domain. Here we apply the famous Harmonic [6, 7] algorithm. 3) We call the area which the 2.5D model covers in the parameter domain an influenced area. Some points generated by the parameter surface will get positions within the influenced area. Calculations will just be done to these points. Given one point, an interpolation operation can get its barycentric coordinate in the related triangle i of the 2.5D model. Then another interpolation operation will calculate its depth information. Finally, its 3D coordinate is calculated according to the normal at this point. Figure 4 shows the surface overlaying result. The depth information of the 2.5D model has been fairly added to a bent Bézier surface.

⊿

4.2.2 Partial Resampling Method Now most of the 3D models are in the form of mesh. It is extremely important to make our work covering these models. Otherwise, our work will be almost useless in the actual applications. 1)

Mapping both 3D model and 2.5D model to 2D domain to simplify the cutting operation. Harmonic algorithm is applied again here.

Surface Animation Based on 2D Editing

643

Fig. 4. The Result of overlaying two surfaces and its detail

2) Resampling points from the 3D model according to the density of the points on the 2.5D model. The 3D position and normal for each point will be calculated with an interpolation operation by barycentric coordinate. 3) Determine the cutting area. Delete every triangle that has a part in the influenced area. Mark up the influenced points that are not deleted. We call these points outer ring. 4) Find out the edge points of the 2.5D mode and name them as inner ring. A triangulation between the outer ring and the inner ring will sew the two part of the model. 5) Paste the 2.5D model to the resampling area. With depth di and normal Ni, the 3D coordinate of each resampled point can be calculated. We add an offset di along the direction in Ni to the original 3D coordinate of the resampled point to get its new 3D coordinate. Jones [8] paid much attention to the point normal calculation during smoothing a mesh model. It is very important to get a set of good point normals before browsing a model. Something must be pointed out. The point normal of the final model can not be calculated directly according to the triangle adjacency information. The cause is that some triangles in the original model have much bigger size than normal size. So the resample points within these triangles will be in the same plane. This will lead the wrong normal generations. In fact, we seldom want the bigger triangles look planar. So the point normal at the resampled points in these triangles must be obtained with interpolations according to normals of the bigger triangles vertices. Otherwise, a lot of point normals will be parallel, and this will make some area look planar, not smooth. So we must obtain point normals during step 2). The points not influenced in the original model will keep the original point normal. The points bumping up in the 2.5D model can obtain their own point normals with interpolations according to the adjacency information. Thus every point in the final model can get a good point normal. Figure 5 shows a magnified detail of the final model. Area A is the resampled area, area B is the triangles between the outer ring and the inner ring and area C is the part that is not influenced in the original model. Figure 3 also shows that except the part

644

J. Gao, C. Zhang, and Z. Feng

A

B

C

Fig. 5. Detail of the final model and the whole final model

bumping up in the 2.5D model, the other areas look extremely like the original model, which can prove the validity of our algorithm.

5 Conclusions and Future Work Applying image management, depth calculations, 3D pasting, we realize the mapping from 2D editing to 3D modeling. The transformation path in image editing is expressed in the form of parameter curve. Curve can be controlled easily by use input. We can also add some control in the process of depth calculation to make many lovely effects (different concave and protruding styles). At the same time, we can operate as we like to control the shape of the parameter surface. All these things can make our algorithm much more convenient and operable, which we have expected. The work of this paper is a great promotion on 3D modeling. It provides more modeling methods. We can control the details with 2D editing. And we can get more if we apply this work also with present 3D carving techniques. The mapping from 2D to 3D with our algorithm has a linear time cost at each calculation step, so real-time 2D editing can generate real-time 3D animation, as shown in Figure 8. Figure 6 shows the flowing of fluid on the surface of a jar. Figure 7 shows that 2D editing, for

Fig. 6. Animation of fluid

Surface Animation Based on 2D Editing

Fig. 7. Controlling 3D animation with 2D editing (transformation and rotation, etc.) (S) 0.080

Time

0.0785

0.075 0.060 0.045

0.0393

0.030 0.015 0

0.0196

Resampe Resolution

0.0049

128u 128

256 u 256

512u 512

1024u1024

Fig. 8. Animation Time Cost with Different Resample Resolution

645

646

J. Gao, C. Zhang, and Z. Feng

example transformations and rotations, can directly control the shape of 3D model. We are expecting the future development of our work. There are still many shortcomings in our work. The most distinct one is the original image problem. Now we can manage images of pixels but can not do it with vector graphics. The bigger future application is direct 3D writing. The target is to allow people writing and carving easily on the surface of a certain 3D model. That will be a big vision as we guess.

References 1. Li, Leilei., Zhang, Sanyuan., Zhang, Xinyu., Ye. Xiuzi. Geometric primitives based deformation techniques for arbitrary meshes, ACM SIGGRAPH International Conference on VRCAI, Singapore, 2004. 2. Hongbo Fu, Chiew-Lan Tai, and Hongxin Zhang, Topology-free cut-and-paste editing over meshes, Proc. 3rd Int. Conf. Geometric Modeling and Processing (GMP 2004), IEEE, Apr 2004 3. M. Kaufmann, S. Francisco, T. Strothotte and S. Schlechtweg, Non-Photorealistic Computer Graphics: Modeling, Rendering and Animation, CA, 2002, 470. 4. H. Hoppe, Progressive meshes, SIGGRAPH 1996, 99-108 5. Yang Gang, Meng Xiangxu, Tu Changhe, Stroke Generation with Curve constrained image transformation, CAD and Graphics of China, Vol.15, No.1, 2003, pp65-70 6. P. Duren, Harmonic Mappings in the Plane, Cambridge Univ. Press, 2004, pp.1-7, 18-23, 66-72, 182-186; MR2048384 (2005d: 31001). 7. E. Praun, H. Hoppe, Spherical Parametrization and Remeshing, ACM Transactions on Graphics (TOG) (July 2003), Volume: 22, Issue: 3. 8. T. R. Jones, F. Durand, M. Desbrun, Non-Iterative, Feature-Preserving Mesh Smoothing, SIGGRAPH 2003, pp. 943-949

Kinematic Inbetweening for Motion Animation Dan Li1, Hai-Yin Xu2, Rui Huang1, and Jian Wang1 1

College of Computer Science & Technology, Huazhong University of Science & Technology, Wuhan, China 2 National Centre for Computer Animation, Media School, Bournemouth University, UK [email protected], [email protected]

Abstract. A novel algorithm and approach to kinematic inbetweening for motion animation is proposed. To represent motion of a character in computer animation, a parametric curve is used to specify motion path of character while a velocity profile curve describing the varied speed of character along the path. Given the motion path and speed, kinematic inbetweening technique is executed to generate motion inbetweens from which skeleton sequences for character animation are obtained by inverse kinematical technique. In this paper, the principle and a novel algorithm for kinematic inbetweening is proposed and several demonstrations provided. Keywords: motion, kinematic inbetweening, parametric curve, animation.

1 Introduction While the main research themes in computer animation includes modeling, motion animation and rendering, motion animation especially for character animation has recently attracted more and more research attentions in animation community[1]. While computer animation consists of updating a scene over time, motion animation aims to automatically generate realistic intermediate frames, i.e. inbetweens over time. While keyframing technique and motion capturing technique are still powerful in motion animation, there are quite a lot of emerging approaches to motion animation, such as motion warping, motion signal processing, motion editing with spacetime constraints, blending, motion retargeting and motion mapping[2-5]. These motion animations usually depend on keyframing and motion capturing and begin with some input frames provided by them rather than from zero. Parametric space and event space are the main two spaces in which the motion frame sequences are produced in. So-called motion animation in parametric space explicitly manipulates each motion parameters such as degree of freedom joints of the character to generate frame sequences. In this case, the character motion is parameterized into a parametric space and then manipulation can be directly undertaken in this space to obtain motion sequences[6-9]. In keyframing, each degree of freedom functions of the character are independently interpolated and used to create inbetween motion frames. The simplest form of interpolation, linear interpolation, often results in motion that appears jerky because the velocities of the K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 647–654, 2007. © Springer-Verlag Berlin Heidelberg 2007

648

D. Li et al.

moving objects are discontinuous. To correct this problem, better interpolation techniques, such as splines and radial basis function, are used to produce smoothly interpolated curves. Another important approach, motion signal processing applies elementary techniques from signal processing to handle one dimensional motion parameters to produce animation sequences. Lower level manipulation tends to be tedious and sometimes violate the constraints. Motion animation in event space implicitly specifies the motion task or goal for characters, generates the motion inbetweens in the Cartesian coordinate system, and further incorporate inverse kinematics to generate the motion sequences[10-12]. Wiley and Hahn[10] proposed interpolation synthesis for articulated figure motion based on motion capturing. The very popular method for motion animation is to set the path by a parametric curve in Cartesian space for the character to follow such as in Maya. In Maya, there are two types of approaches to motion inbetweening. The first is parametric speed method. A simple way to implement a path motion is to compute the points along the parametrically represented path with equally spaced parametric value. However, the motion of the character along the path would be unpredictable and unrealistic as the relationship between the curve parameter and the arc length of curve is usually nonlinear. The second approach used in Maya is arc length parameterization method. But the arc length parameterization is difficulty and sometimes impossible for most curves frequently used in computer animation[13-14]. In this paper we investigate kinematic motion inbetweening along parametric curves in computer animation. The motion path for character animation is specified by a parametric curve. The principle of kinematic inbetweening is put forward and outlined at first and inbetweening algorithm is then developed in section 2, and followed by some demonstrations and their analyses in section 3. At last, conclusion is given in section 4.

2 Inbetweening The motions of characters in animations such as human figures are usually represented by a set of position or orientation unknowns. The motion unknowns may be defined in three dimensional Cartesian coordinates or joint angles. Keyframing requires the animator to specify key positions for the characters being animated. The keyframing animation consists of the automatic generation of intermediate frames, called inbetweens, obtained by interpolating a set of keyframes supplied by the animators. The most popular method is to set the path by a parametric curve for the character to follow such as in Maya. In this paper, parametric curves for task specification in motion animation are used. A parametric curve is used to specify the motion path which comes from either traditional interpolation or motion capture, or high-level motion planning, while a velocity profile curve is adopted to describe the motion timing. The motion inbetweening aims to obtain a series of inbetweening positions along the parametric curve. The geometric property for the parametric curve is analyzed. Based on the relationship between the geometric and motion properties, an approach and algorithm to motion inbetweening along parametric curves is proposed. Given the motion path

Kinematic Inbetweening for Motion Animation

649

Motion path Character design

Inbetweening

IK

Skeleton sequence

Velocity profile

Fig. 1. Inbetweening principle

and velocity profile, inbetweening technique is utilized to generate inbetween motion sequences from which skeleton sequences are obtained by inverse kinematical technique. Fig 1 shows the kinematic inbetweening workflow. 2.1 Motion Path Parametric curves have succeeded in application for modeling, deformation and rendering within computer graphics. We consider application of parametric curves in motion animation. The motions of a character in animation are usually represented by a set of positional unknowns. Assume any position of the end effecter of an animated character is described as a point g ( x, y , z ) . The motion path for the character is specified by a three-dimension parametric curve,

g (u ) = [x(u )

y (u ) z (u )]

(1)

where 0 ≤ u ≤ 1 is one of position parameters. 2.2 Velocity Profile By traditional motion animation technique, the inbetweening position unknowns are evenly distributed along the motion path without including any information as to the timing of the motion. In our research, a velocity profile curve is adopted to describe the varied speed of character along the motion path. Let the speed of the motion be v(t ) . The velocity profile which is a spline curve can be obtained by interpolation and blending. 2.3 Inbetweening The motion inbetweening aims to obtain a series of inbetweening positions along the parametric curve. Assume g n represents the position inbetween sequences. As aforementioned, equation (1) represents the motion path and the user specifies the velocity profile. Assume s is the arc-length of the curve. According to the definition of the arc-length, the derivative of arc-length with respect to the position parameter can be represented as:

{ }

650

D. Li et al.

ds dx dy dz = ( )2 + ( )2 + ( )2 = gu du du du du

(2)

du be the derivative of the position parameter with respect to the arc-length, ds 1 . Let d 2u equation (2) gives h = q = 2 be the second-order derivative of the ds gu Let h =

position parameter. Differentiating equation (2) with respect to arc-length results in:

q=

g du g d 1 = − uu3 ( ) = − uu2 du g u gu ds gu

(3)

Let the first and second order derivatives of the position parameter with respect to time be denoted as:

u =

du d 2u , u = dt dt 2

(4)

The velocity and acceleration of point g along the curve (1) can be represented as

v=

ds d 2s , v = dt dt 2

(5)

The relationship between time derivative and arc-length derivative of the position parameter can be represented by:

u =

du du ds = = hv dt ds dt

(6)

u = qv 2 + hv

(7)

g n ( xn , y n , z n ) ， the and g n +1 ( xn +1 , yn +1 , zn +1 ) . The task of

Suppose the current position parameter and position is u n and parameter and position of the next is u n +1

inbetweening is to calculate the parameter and position of the next , i.e. u n +1 and

gn +1 ( xn +1 , yn +1 , zn +1 ) according to the current position parameter and position. The parameter u n +1 can be expressed as a polynomial of time using Taylor’s series expansion with terms up to the second order,that is:

1 un +1 = un + uT + uT 2 2

(8)

Where T is the frame cycle. By taking equations (6) and (7) into account, equation (8) can be rewritten as:

Kinematic Inbetweening for Motion Animation

1 1 un +1 = un + hvT + hvT 2 + qv 2T 2 2 2

(9)

Then according to the position curve (1), the next frame position obtained.

g n +1 (u n+1 ) = [x(u n+1 )

651

y (u n+1 ) z (u n+1 )]

g n +1 can be (10)

2.4 Inverse Kinematics While the position for end effecter of the character is obtained, the inverse kinematics and some other style constraints are utilized to generate all degree of freedom skeleton joint parameters. Inbetween sequences of joints parameters are solved by existing inverse kinematics algorithm in this paper. It may be noticed that realistic motions can be obtained by incorporating some stylization control over the character into standard inverse kinematics techniques. 2.5 Motion Sequences Generation By above inbetweening and inverse kinematics, a set of position inbetweens {g n } are obtained from which the skeleton sequences are generated. The frame for the animation usually comprises all degree of freedom joint parameters, that is F = g 1 (u1 ) g 2 (u 2 ) g i (u i ) ... g m (u m ) . Then the k-th frame can be

[

[

]

represented by F k = g (u k ) 1

1

]

i m g (u k ) g (u k ) ... g m (u k ) . 2

2

i

3 Illustrations We use two simple illustration examples to demonstrate this technique. Given the motion path as a parametric curve, by setting different velocity profile, different inbetween point sequences on the curve are obtained. Constant speed and variable speed are respectively used to generate inbetweening as shown in Fig 2 and Fig 3. The correlative velocity profile curves are given on the Fig 2(a) and Fig 3(a). According to the Fig 2(b), the distance between adjoining points is constant while in Fig 3(b) the spacing varies, which is dense at the starting and ending segment and sparse in the middle. For the same motion path, by considering the motion speed of the animated character, different point sequences along the curve are resulted, from which diverse frame sequences may be generated. But in traditional keyframing, once motion curve is determined, the frame sequence is fixed. It is also expected that more realistic motion animation can be generated due to the inclusion of kinematic information of the animated character. In our illustration, one of a human character hands is required to follow a parametric curve while the character stands still. The limb skeleton consists of hand, forearm and arm. By using inverse kinematics of human limb, the joint variables for

652

D. Li et al.

shoulder, elbow and wrist are obtained. Fig 2(c) and Fig 3(c) show the resulting skeleton sequences, respectively. For the same motion curve, by considering the motion speed of the animated character, different point sequences along the curve are resulted from which diverse skeleton sequences may be generated.

V(t)

(a)

(b)

(c)

Fig. 2. Inbetweening result with constant speed (a)velocity profile (b)curve inbetweening (c) skeleton sequences

V(t)

(a)

(b)

(c)

Fig. 3. Inbetweening result with variable speed (a) velocity profile(b)curve inbetweening (c) skeleton sequences

4 Conclusion Traditional keyframing animation is mainly based on geometrical interpolation. The kinematic inbetweening aims to incorporate some kinematics elements into the

Kinematic Inbetweening for Motion Animation

653

traditional motion animation technique. In this approach, a parametric curve is used to specify motion path of a character with a velocity profile curve describing the varied speed of the character along the path. Based on the motion path and velocity profile, inbetweening technique is utilized to generate inbetween motion sequences from which skeleton sequences for character animation are obtained by inverse kinematical technique. By the kinematics oriented inbetweening, with different speed profiles, diverse motion sequences can be generated for the same position curve. Thus more realistic motion animation can be obtained by considering more complex kinematic information of the animated character. In the future, we will develop more versatile inbetweening techniques by incorporating dynamics into the kinematics oriented inbetweening.

Acknowledgments The authors would like to thank financial supports from National Natural Science Foundation of China (Program No.50305007) and Hubei Key Technologies R&D Programme(Program No. 06AA101C01).

References 1. Magnenat-Thalmann N., Thalmann D. Virtual humans: thirty years of research, what next? Visual Computer 21(4)(2005)997-1015. 2. Pullen K., Bregler C., Motion capture assisted animation: Texturing and synthesis. ACM Transactions on Graphics. San Antonio, Texas (2002) 501–508. 3. Gleicher M.: Retargeting motion to new characters. ACM SIGGRAPH. Orlando, Florida (1998)33-42. 4. Popovic Z, Witkin A. Physically based motion transformation. Proceedings of ACM SIGGRAPH. LosAngeles (1999) 11-20. 5. Kovar L., Schreiner J., and Gleicher M. Footskate cleanup for motion capture editing. ACM SIGGRAPH Symposium on Computer Animation. San Antonio, Texas (2002) 97– 104. 6. Kochanek D. H. U., Bartels R. Interpolating splines with local tension, continuity and bias control. Computer Graphics 18 (3) (1984) 245 254. 7. Rose C.F., Bodenheimer B. and Cohen M.F., Verbs and adverbs: multidimensional motion interpolation. IEEE Computer Graphics and Application 18(5) (1998)32-40. 8. Bruderlin A., William L.: Motion signal processing. Computer Graphics 29(4) (1995)97104. 9. Witkin A., Popovic Z. Motion warping. Proceeding of ACM SIGGRAPH. LosAngeles, California (1995)105-108. 10. Wiley D.J., Hahn J.K. Interpolation synthesis of articulated figure motion. IEEE Computer Graphics and Applications 17(6) (1997)39-45. 11. Terra S. C. L., Metoyer R.A. Performance timing for keyframe animation. Eurographic/ACM Symposium on Computer Animation. Grenoble, France (2004) 253258.

～

654

D. Li et al.

12. Tolani D., Goswami,A., and Badler N.I. Real time inverse kinematics techniques for anthropoporphic limbs. Graphical Models 62 (5)(2000)353-388. 13. Jieqing Feng, Qunsheng Peng. Speed Control in Path Motion. Proceedings of CAD & Graphics. Wuhan， China (1995)440-445. 14. Hai-Yin Xu, Dan Li. Implicit curve oriented inbetweening for motion animation. Proceedings of ACM SIGGRAPH GRAPHITE. Kuala Lumpur, Malaysia (2006)30-35.

Real-Time Visualization of Tire Tracks in Dynamic Terrain with LOD Guojun Chen1,2, Jing Zhang3, Xiaoli Xu1, and Yuan Yin2 1 College of Computer and Communication Engineering, China University of Petroleum, Shandong Dongying, 257061, P.R. China [email protected] 2 School of Computer Science and Engineering, Beihang University, Beijing 100083, P.R. China {cgj,yinyuan}@vrlab.buaa.edu.cn 3 College of Resources Environment and Tourism, The Capital Normal University, Beijing, 100037, P.R. China [email protected]

Abstract. In off-road simulation, the terrain is being modified as a result of its interaction with the vehicles. Previous methods dealing with tire tracks are to animate tire tracks and have no interactivity. In this paper, we describe a method to render tire tracks in dynamic terrain. In the base of hierarchical structure for terrain LOD, hierarchical structure starts at a resolution sufficient to represent the initial terrain, and then increases the resolution wherever necessary at the modified terrain regions. To avoid the cracks at the boundary between terrain cells, which are modified and increased the resolution, an algorithm based on window-scanning avoids cracks between neighboring cells in the 2×2 window is given. This algorithm avoids the transfer of cracks effectively through guaranteeing the results after window scanning will not impact the edges processed previously. The results of experiments show that not only this algorithm avoids crack effectively, but also the cost of computing is very low. Keywords: dynamic terrain; tire simulation; multi-resolution; level of details (LOD); terrain visualization.

1 Introduction The real-time visualization of the terrain plays an important role in computer graphics, three-dimensional geographic information system, virtual reality and simulation. Ground vehicle simulation belongs to the type of applications that reproduce real world human experience, and it is very important that the system supports user-simulator interaction. In conventional driving simulation systems, effects as tracks did not alter the surface topology of the terrain that the tires interact with. Most noticeably, presence of vehicles affects the soft terrain and tire tracks will appear. We refer to this type of terrain surface whose geometric and other properties may change during the process of the application as dynamic terrain [1]. Certainly, K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 655–666, 2007. © Springer-Verlag Berlin Heidelberg 2007

656

G. Chen et al.

dynamic terrain is not limited to the effects of vehicles. In military, for example, explosions and the footprints of soldiers also deform the terrain. The geometry, color, and texture of the terrain surface may all be altered as a result of the entity-terrain interaction, and this brings two new complexities to simulation. The first lies in the vehicle and terrain dynamics, and the second is the visualization of the dynamic terrain. A good deal of work over the last several decades has been done toward physically based models of deforming terrain surfaces. Li and Moshell developed a visually dynamic model of soil slippage and soil manipulations [2]. In their work, a bulldozer blade serves as a local force function used to change the heights of the terrain. Sumner et al. [3] described a model of ground surfaces that could be deformed by the impact of a rigid body model. They showed footprints made by a runner in sand, mud and snow as well as bicycle tire tracks, a bicycle crash, and a falling runner. Many techniques have been developed for representation and efficient visualization of terrains and other surfaces. In particular, the recent development of view-dependent multiresolution methods have provided a strong advance over distance-based discrete level-of-detail and other simple methods aimed at minimizing rendered polygons [47]. But these methods focus either on static terrain or on time-varying geometry where all changes are known prior to any rendering. In order to create an efficient algorithm for the real-time visualization of dynamic terrain surfaces, we need to study one method to render dynamic terrain, especially tire tracks, in the car game and in the ground vehicle simulation. In this paper, our research work is focused on the visualization of terrain changes determined through real-time tire-soil simulation or other processes. The remainder of this paper is organized as follows: In Section 2 we review some related work by previous researchers. Then, in Section 3 and 4, we introduce our rendering technique and give an implementation of it. At the end of this paper, we give the results.

2 Related Work Most existing terrain surface representation methods are intended for static terrain surface, but a few methods for dynamic terrain surface, such as crater in battlefield, have been presented. Summer at al.[3] described a simulation model of ground surface that could be deformed by the impact of rigid body was not for real-time use. In 2000, Shamir et al. [8] presented a method for multiresolution dynamic surface visualization, which divided surface representation into triangulated irregular networks(TINs). The model used a DAG (Directed Acyclic Graph) as hierarchical structure and the DAG incrementally was modified as the surface was deformed. Whereas TINs have the potential to represent a surfaces with fewer triangles for a given error tolerance, the simplicity of regular subdivision hierarchies make them more appropriate for realtime rendering terrain[1,4-9]. Rami [9] proposed a physically-based approach to model the explosion event and the process of terrain fragmentation resulting in the formation of craters. The methodology is based on voxel modeling and introduces a new method that uses

Real-Time Visualization of Tire Tracks in Dynamic Terrain with LOD

657

connected-fragments to model terrain with the geology. Memory usage could be a limitation to the model, since it increases with increasing level of resolution of terrain surface. He [1] provided multiresolution view-dependent representation and display of dynamic terrain by extending ROAM (Real-Time Optimally Adapting Mesh) with efficient hierarchy updates as terrain deforms, and used DEXTER (Dynamic EXTEnsion of Resolution) to provide only-where-needed memory efficient resolution extension. This approach was suited for real-time off-road driving simulation. He just dealt with relatively small scale terrain inputs. When the dynamic terrain is divided into regions, the surface continuity between neighboring approximation mesh, as well as the continuity between them and the neighboring static objects has not been studied. In 2001, Lindstrom et al. [10] presented an elegant and simple method to implement framework for performing out-of-core visualization and view-dependent refinement of large terrain surface. The nested error metric, which can eliminate the crack in the meshes of terrain surface automatically and implicitly, was presented. Although they proposed a simple to compute index that yields substantial improvements in locality used data layouts, the framework is not suitable for rendering dynamic terrain. Wang et al. [11] presented a complete solution to visualize dynamic terrain in real time based on DEXTER [1] and the nested error metric[10]. But the modified regions on the terrain surface have not been given a high priority to be preserved. Chen et al. [12] presented an algorithm for dynamic terrain LOD (DT LOD) with regions preservation. The algorithm set boundary vertices of modified regions active, and use DAG of vertices which used to represent the relationship between vertices in the refinement. Zhang et al. [13] presented an algorithm that reuses per-vertex visibility computation from previously displayed frames efficiently during continuous view-dependent LOD visualizations. This algorithm decreases to a certain extent computing the distance from the viewpoint to vertex at each frame. Cai et al. [14] presented a hybrid multiresolution algorithm for crater based on ROAM. They selected half-ellipsoid model in the explosion theory as the physical based model of crater, and provided a procedural texture generation method for crater. They just dealt with relatively small scale terrain inputs.

3 DT LOD with DAG The mesh produced by edge bisection can be represented as DAG of its vertices. It can be seen that triangle bin-tree and DAG can implement refinement and simplification of triangle mesh. Whether vertex i is introduced into triangle mesh will determine whether terrain is to be refined or be simplified locally. 3.1 DAG For a given refinement M of a mesh, we say that a vertex i is active if it appear in M. It can be concluded that if a vertex i is active, its parents (and by induction all of its ancestors) must be active. This conclusion is the necessary condition on which the mesh forms a continuous surface without cracks. The dependency relationship of

658

G. Chen et al.

vertices is included in the refinement criterion implicitly [10, 11]. A directed edge from i to one of its children j in the DAG corresponds to a split operation, in which j is inserted on the hypotenuse and connected to the apex vertex i (Fig.1). Thus, all non-leaf vertices not on the boundary of the mesh are connected to four children in the DAG, and have two parent vertices. Boundary vertices have two children and one parent.

Fig. 1. The bin-tree of the triangles and the DAG of its vertices

The splitting and merging is related to whether the vertex i is active or not, while the error of the vertex can decide whether the vertex is active or not. The error is classified into object space error and screen space error. The introduction of vertex i, which splits a triangle into two finer triangles, brings a world space error, calculated by the vertical distance (z axis): δi |Zi-(Zk+Zl)/2|, Here Zk, Zl is the altitude of the end points of the hypotenuse. We adopt formula (1) to compute the screen space error. In order to imply the relationship among the vertexes with error, we adopt the nested errors sphere [7, 8], shown in formula (2). Where ri is the screen space error of the vertex i respectively; pi, e are the position of the vertex i and viewpoint in space respectively. d is the distance from the viewpoint to the projection plane, is the numbers of the pixel on the screen projected by the unit-length in space. Ci in the formula (2) is the set of children of vertex i in the DAG. We can conclude the result that the viewpoint move in the error sphere of vertex i, then the viewpoint must be in the error sphere of i’s father. Note that Equation 3 can be used where the viewpoint is contained in i’s bounding sphere or not.

＝

λ

pi − e =

δi × d × λ = ri . τ

(1)

⎧ ri , if i is a leaf node ⎪ . Ri = ⎨ max ⎧⎨ri , max{ p i − p j + R j }⎫⎬ others ⎪⎩ ⎩ j∈Ci ⎭

(2)

Active(i ) ⇔ pi − e < Ri .

(3)

Real-Time Visualization of Tire Tracks in Dynamic Terrain with LOD

659

Although the nested error sphere maintains explicit dependencies between each child and its parents, we, too, use a DAG of sphere for some purpose. We need to fast force some vertices, whose error don’t satisfy Equation 3 in some cases, to be activated with their descendants using DAG. 3.2 Boundary Vertices and Continuity Between Neighboring Blocks

Conventional algorithms for large-scale terrain visualization are based one data tiling. When partition large terrain into blocks and rendering them with LOD respectively, there would appear the surface discontinuity between neighboring blocks, namely neighboring blocks share the same boundary and vertices on the boundary may have different status, which results in cracks between neighboring blocks. The method in our algorithm to avoid cracks is to set the vertices on shared boundary with the uniform active status forcibly. If a vertex’s active status changed from inactive to active its ancestors must be set active, consequently cracks can be avoided. Theorem 1: Suppose a terrain is (2n+1) × (2n+1). When a vertex p(0, y) (1 0)

Fig. 6. Shading map and Mesh generate by 0-1 function

3.2 Establishing the Mesh By sampling the height field, we obtain the 3D vertices, which are then connected to build the triangle mesh. For each point in the interior region of the height map, two symmetric vertices on both sides of the view plane are created to form a closed mesh. The topology of the interior vertices can be easily built in the way like normal height field. In order to build a well-formed mesh we re-sample the original contour as shown in Fig 7. First, the circled points adjacent to the contour are collected. Then, the intersections of the contour and the paths these points moving to their interior neighbors are considered as the new vertices along the contour, such as the green points. The topology of the green points follows that of the circled points. The mesh resolution can be specified by defining the sampling density. Fig 8 shows an example of the created meshes at different resolution. The contour is resampled to keep consistency accordingly.

692

X. Li, G. Fei, and Y. Zhu

Fig. 7. Mesh creation by height field sampling

Fig. 8. Mesh created at different resolution

3.3 Adding Details on the Mesh Surface The generated mesh can be modified by editing the 2D height field. This is an advantage of our modeling method. With this method, we realize the functions of modifying/increasing details on the mesh surface also by 2D sketching. After analyzing the sketching habit of most cartoonists, we design the functions for adding/modifying details by directly sketching a closed region on the mesh surface. It is done in the following steps: (a) Sketch on model surface to define an interested region, as shown in the colored region in Fig 9(a); (b) Extend-out/cave-in this region with the height field as shown in Fig 9(b); (c) Specify the extend-out/cave-in shape of the region as shown in Fig 9(c). This function implements the following steps: (1) When constructing models in section 3.2, we reserve the height field image and corresponding relationship between the mesh vertices and pixels in height field. When users draw a closed curve on view plane, the curve is mapped on the model surface first, and then according to the relationship between mesh and height field image, the closed curve can be mapped onto the height field image. (2) By filling the pixels in the closed curve region on height field image, all the vertices affected by this closed region are collected according to the relationship between the mesh vertices and 2D pixels. The renewed height field of the filled region can be established using the methods in 3.1 and 3.2. (3) Modify the vertices in the closed region and shift the corresponding vertices along the direction of normal vector. If the region needs to be extended-out, a positive distance will be shifted along the normal direction; vice versa, shifting along the backward normal direction. By the approach, users can modify the detail shape on model surface interactively.

ISMAS: An Interactive Sketch Modeling and Animation System

693

Fig. 9. The process of describing a surface detail

4 Skeleton/Animation In the skeleton/animation system, users directly specify the character skeleton by sketching curves on the view plane. Users can draw and edit the desired skeleton curve of each key frame. The system will complete the whole animation by deforming the mesh according to the skeleton curves which are interpolated between the key frames. Fig 10 shows the deformation driven by the curve skeletons. System generates an animation by taking the following steps: (a) Users draw a line on the surface of model to define a curve skeleton directly, system initialize the sketch to an original curve skeleton. (b) Users draw a new line to define a gesture of model in a key frame, system deform the mesh as response. (c) System interpolates the skeletons for in-between frames to complete the whole animation.

Fig. 10. Deformation driven by soft skeleton curve

4.1 Skeleton Registration In our system, the skeleton is not the traditional rigid skeleton. When users draw a curve on the view plane, we project it onto the center of mesh as a curve skeleton. Here, the editing unit is the curve skeleton, not the node as in the traditional rigid skeleton system, and users could deform the model by redrawing a new curve, rather than by adjusting the angles of node.

694

X. Li, G. Fei, and Y. Zhu

4.1.1 Dividing Skeleton Region The two planes that perpendicular to the skeleton curve at the start point and end point partition the whole mesh into three parts: (a) static region – the region out of start cutting plane. The region will not be influenced by the skeleton curve; (b) deformable region – the region between two cutting planes. The region deforms with the shape of skeleton curve; (c) rigid region – region out of end cutting plane. The region performs the rigid deformation and keeps the continuation with deformable region. Fig 11 shows how to partition regions of the mesh.

Fig. 11. Partition mesh into three regions: static region (left), deformable region (middle), rigid region (right)

4.1.2 Parameterization of the Mesh Vertices To deform the deformable region, we need to set up the correspondence between the vertices in deformable region and the skeleton curve. We assign a value S to each vertex, which is a normalized arc length value from the start point to the end point, which ranges between 0 and 1, where S = 0 represents the start points and and S = 1 represents the end points. We assign S=0 to all the vertices in the static region, and assign S=1 to all the vertices in the rigid region. 4.2 Model Deformation After the definition of character skeleton, users can redraw the skeleton curve to define the deformation of model or the definition of the key frames. System will compute the current model shape according to the modification to the skeleton curve. 4.2.1 Deformation Algorithm Now we get the vertex positions in the original model, original skeleton curve, target skeleton curve and the S value of all the vertices in the original model relative to the original skeleton, based on which system can compute the coordinate of model vertices after deformation. To compute the position of each vertex, we extract the following information first: coordinate position So of the original skeleton curve at position S, coordinate Sr of the target skeleton curve, normalized tangent vector uo at

ISMAS: An Interactive Sketch Modeling and Animation System

695

position S on the original skeleton curve, and normalized tangent vector ur at position S on the target skeleton curve. Then we do following conversions:

Fig. 12. Deformation computation of a vertex

(a) Computing the vector V1 between So and vertex Vo;

V

1

= V

o

− S

(6)

o

(b) Computing rotation matrix To that rotate uo to z axis; then getting V2 by rotating vector V1 with this matrix;

T0 = Roy Rox

(7)

V2 = ToV1

(8)

(c) Computing rotation matrix Tr that rotate ur to z axis; then computing V3 by rotation vector V2 with the inverse matrix of Tr.

V3 = Tr−1V2 = Rrx−1 Rry−1V2 = RrxT RryT V2

(9)

(d) Finally, obtaining the desired vertex position by adding V3 to the position Sr.

Vr = S r + V3 Vr = S r + RrxT RryT Roy Rox (Vo − S o )

(10) (11)

4.2.2 Twisting In our system, mesh could be twisted based on the axis of skeleton. Users can define the distortion angles at the end of skeleton curve(with mouse wheel), and achieve the effect of distortion of structure along skeleton. In above deformation process, we have rotated vector u0 to z axis in the step 2; here we insert a rotation matrix Rz rotating along z axis, and get the twisting transformation as:

Vr = S r + RrxT RryT R z Roy Rox (Vo − S o )

(12)

696

X. Li, G. Fei, and Y. Zhu

Fig. 13. Twist deformation of mesh

4.2.3 Scaling Scaling allows animation effect more diversified. Users can freely define the scaleable proportion at arbitrary positions of skeleton. The scaleable center of scaling is also the skeleton curve. Scaling transformation could be added as following formulation:

Vr = S r + Ts RrxT RryT Roy Rox (Vo − S o )

(13)

4.3 Multi-skeleton Deformation Formulation 13 gives the distortion approach of vertices on single skeleton curve. Based on hierarchy of traditional rigid skeleton, we design a similar hierarchy for our curve skeleton. At the beginning of processing a multi-skeleton deformation, system will deform the mesh according to root-skeleton first, then deform according to its child-skeletons until the last leaf-skeleton. 4.3.1 Initializing Multi-skeleton In our system, the first skeleton defined by user is the root-skeleton, and the after skeletons are its child-skeleton or grandchild-skeleton. When a new skeleton is defined, system will identify relationship between the new skeleton and the old skeletons according to start points of the new skeleton: (a) Search for the leafskeletons in the old skeleton topology-tree. (b) Calculate the S value of the start point of the new skeleton relative to leaf-skeleton. (c) If S>0 new skeleton will be a childskeleton of the searched leaf-skeleton. If S=0, search for parent-skeletons of leafskeletons and recalculate the S relative to searched skeletons. (d) Search for each level of tree from leaf and calculate S value repeatedly to find parent-skeleton of the new skeleton. If no one skeleton is found to be its parent-skeleton, then the new skeleton will be assigned to be a child-skeleton of root.

Fig. 14. Topology-tree of multi-skeletons

ISMAS: An Interactive Sketch Modeling and Animation System

697

When initializing multi-skeleton, system needs to save all S values which represents the relationship between the vertices and the skeletons. Furthermore, for every skeleton, we need to save an S’ value relative to its parent-skeleton, where S’ indicates how its parent-skeletons affect it. The value S’ can be computed by the average value of vertices close to a start cutting plane. The start cutting plane is perpendicular to the skeleton curve and passes by the start point of skeleton. 4.3.2 Multi-skeleton Deformation In the beginning of multi-skeleton deformation, we deform the mesh by the rootskeleton, and then rigidly transform all its child-skeletons according to their value S’ (grandchild-skeleton deformation accord to the same S’ value of child-skeleton). The distorted skeleton should replace the original skeleton for the following deformation computing. Based on the deformed skeleton, we continue to deal with the left skeleton’s deformation by the sequence of topology-tree and completing the deformation of the whole multi-skeleton.

Fig. 15. Process of multi-skeleton deformation

5 Results Our system can create model in real-time on standard PC and user can finish his modeling work in a minute without any parameter (using mouse wheel to adjust the thickness or extend-out/cave-in). Fig 16 shows a modeling example with few strokes: Firstly, sketch contours of object, system will generate a rough model (a). Sketch on

Fig. 16. Process of creating a fish model

698

X. Li, G. Fei, and Y. Zhu

rough model to select an interested region for edit, adjust the thickness of the region (b). Sketch on the model again to select some regions, adjust these regions for adding surface-details(c). Fig 17 shows the process of creating an umbrella model which is hard to be generated by other sketch modeling tools. First (a), user draw a hexagon sketching

Fig. 17. Process of creating an umbrella model

Fig. 18. Models created by our system

Fig. 19. Animation of letters’ deformation. The red letters are key frames.

ISMAS: An Interactive Sketch Modeling and Animation System

699

and adjust the thickness to thin. Use closed surface detail tool draw another hexagon on top of mesh as (b), then choose sin extend function to create arc of the umbrella (c). Turn to bottom view, and draw a new hexagon with closed surface detail tool (d)and cave in the bottom of the umbrella (e). Use the 0-1 function to create top (f) and stand (g) of the umbrella. Fig 18 exhibits some models created by non-expert modeling users through our system. Fig 19 exhibits a letter animation created by our system.

6 Conclusion and Future Work We presented a method for modeling and animation characters based on sketching. In our system, users are able to create mesh and establish animation by just sketching curves, which makes the creation of 3D structure just like sketching on paper very naturally and easily. We also provided a method for adding mesh details on surface by sketching. The editing results can be presented to users in real-time on a standard PC. Our animation system is more adapt to non-expert users. Users establish a skeleton by sketching on mesh surface and adjusting key frame by another sketch; system interpolates in-between frames according to the skeletons of key frames, and generates an animation containing deformation, scaling and twisting. The techniques of our animation system can also be used in other interactive animation systems. The proposed method promises a lot of future improvements. (a) Meshes with structures such as pipe and cup can hardly be built here. In the further study, we want to develop a system that allows these topology structures. (b) The quality of the generated mesh is quite resolution dependent. We are looking for suitable simplification methods to solve this problem. (c) The BOOL operation of mesh by sketch and smooth of the combination need further study. (d) Users create and modify the curve skeleton need a little experience, it need time to lean how to create a proper skeleton. We will simplify the operations of the animation system.

References [1] Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka. Teddy: A sketching interface for 3D freeform design. SIGGRAPH 99 Conference Proceedings, pages 409–416, August 1999. [2] Youngihn Kho, Michael Garland: Sketching mesh deformations. Proceedings of ACM SIGGRAPH 2005. [3] Borgefors G. Distance transformations in arbitrary dimensions. Computer Vision, Graphics, and Image Processing, 1984, 27 (3) : 321–345 [4] PS Tsai and M. Shah. Shape from Shading Using Linear Approximation. Image and Vision Computing J., vol. 12,no. 8, pp. 487–498, 1994. [5] David Bourguignon, Raphaëlle Chaine, Marie-Paule Cani, George Drettakis: Relief: A Modeling by Drawing Tool [A]. Eurographics Workshop on Sketch-Based Interfaces, August 2004. [6] Karpenko O., Hughes J. F., Raskar R.: Freeform sketching with variational implicit surfaces. Computer Graphics Forum 21, 3 (2002), 585.594

700

X. Li, G. Fei, and Y. Zhu

[7] Owads S., Nielsen F., Nakazawa K., Igarashi T.: A sketching interface for modeling the internal structures of 3D shapes. In Proceedings of Third International Symposium on Smart Graphics (2003), Springer-Verlag, Berlin, pp. 49.57. [8] Igarashi Takeo, Hughes John F. Smooth meshes for sketch-based freeform modeling[C]. Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPH, San Diego,2003:139-142 [9] Nealen A,Sorkine O,Alexa M,et al. A sketch-based interface for detail-preserving mesh editing[J]. ACM Transactions on Graphics,2005,24(3):1142-1147

Point Sampled Surface Reconstruction Based on Local Geometry Lanfang Miao1,2 and Qunsheng Peng2 1

Department of Computer Science, Zhejiang Normal University, Jinhua 321004, China 2 State Key Lab. of CAD&CG, Zhejiang Univerisity, Hangzhou 310027, China {lfmiao,peng}@cad.zju.edu.cn

Abstract. In this paper, we presented a point sampled surface reconstruction method based on local geometry. First, an adaptive Binary Space Partition (aBSP) tree was built based on the local shape complexity which was judged by three factors: local main curvature, the normal cone and the number of points. Different local complexities produce different aBSP-trees. Then, each leaf node in the aBSP-tree was approximated by a weighted quadric function and the center points set of aBSP tree’s leaf nodes could be treated as one approximation to the original point set at a given complexity of the local shape. Finally, a ray tracer based on point was used to render these aBSP-trees, which avoids triangulating of implicit surfaces before they are rendered. The experimental results show that the method can reconstruct sharp features very conveniently and effectively. Keywords: surface reconstruction; point sampling; implicit surface; local geometry.

1 Introduction There are many applications that rely on the reconstruction of accurate models of objects such as sculptures, damaged machine parts, and archaeological artifacts. In 3D digitizing technologies such as laser range finding, mechanical touch probes, and computer vision techniques, some of these techniques can yield millions of 3D point locations on the object that is being digitized. Once these points have been collected, it is a non-trivial task to build a surface representation that is faithful to the collected data. Some of the desirable properties of a surface reconstruction method are very important, which include efficiency, low memory overhead, the creation of surfaces that approximate rather than interpolate the data (when noise is present) and faithful reproduction of sharp features. In this paper, we present an implicit surface representation that allows us to reconstruct surface models from a point set. The major features of our reconstruction method are that: 1) an adaptive Binary Space Partition(aBSP) tree was built based on the local shape complexity; 2) the local shape complexity was judged by three factors: local main curvature, the normal cone and the number of points; 3) each leaf node of K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 701–710, 2007. © Springer-Verlag Berlin Heidelberg 2007

702

L. Miao and Q. Peng

the BSP-tree is approximated by a weighted quadrics function of its neighboring points and all center points set of aBSP tree’s leaf nodes could be treated as one approximation to the original point set at a given complexity of the local shape; 4) the reconstructed surface was directly rendered by ray tracing point sampled surface.

2 Related Work From view of bounds of reconstruction, there are two surface reconstruction methods of point sets. They are based on local geometry and global geometry. In global method, the result of the reconstruction surface is related to every point of the whole model, i.e. adjusting the position one point will affect the whole surface representation. While the surface in local method is only related to the points in its local area, i.e. the change of one point will only affect its neighborhood area. Savchenko et al. [1], Carr et al. [2], and Turk and O’Brien[3] use globally supported Radial Basis Functions (RBFs) while Morse et al. [4] and Ohtake et al. [5] employ compactly supported RBFs to reconstruct smooth surfaces from point cloud data. While RBF-based methods are especially useful for the repair of incomplete data, they face serious difficulties in accurate reconstruction of sharp features. In addition, since RBF solutions are global in nature, processing one million points seems to be difficult. Blinn [6] presents an implicit shape reconstructions method from point sets by blending local implicit primitives. This is a local surface reconstruction method and most implicit shape reconstructions are based on this idea. Muraki [7] uses a linear combination of Gaussian blobs to fit an implicit surface to a point set. Hoppe et al. [8] locally estimates the signed distance function as the distance to the tangent plane of the closest point. A volumetric approach of Curless et al. [9] introduced for shape reconstruction from range scans is based on estimating the distance function from a reconstructed model. Projection-based approaches to shape approximation [10] have the advantage that they are local and directly yield points on the surface. However, the projection step requires the solution of a non-linear moving least squares problem, which makes most practical shape operations expensive. Ohtake et al. [11] present a MPU shape representation and can accurately represent sharp features by selecting appropriate shape functions. But the octree subdivision is made only according to the point’s density and normal error, which may make the different parts of the point surface be put into the same cell of octree and thus increase the difficulty of processing. Furthermore, when a reconstructed surface is rendered, it must first be triangulated and then be rendered using some rendering method such as ray-tracing, which also increases the processing time. The reconstruction method in this paper is a binary subdivision method that adapts to the complexity of local shape, and it completely takes the distribution of points into account and also has less node than octree subdivision [12]. Besides, we first approximated each leaf node in the aBSP-tree with a weighted quadric function and then adopted a ray-tracing reconstructed point sampled surface, which will reduce the time from surface reconstruction to surface display.

Point Sampled Surface Reconstruction Based on Local Geometry

703

3 Reconstruction Method 3.1 Surface Definition Let first define the point sampled surface. Given a set of points xi ∈ R 3 , i ∈ {1,2, , n} and corresponding normal ni ∈ R 3 , i ∈ {1,2, , n} on a point sampled surface, we can find the neighboring points p j ( j = 1,2,3, m) for each point xi . In global coordinate system, the surface near each point xi can be approximated by a quadric function, which is defined by the following formula: f ( x, y , z ) = a0 + a1 x + a2 y + a3 z + a4 xy + a5 xz + a6 yz + a7 x 2 + a8 y 2 + a9 z 2 ,

(1)

where the unknown coefficients are determined by minimizing: m

∑ w ( x, y , z ) | f ( x, y , z ) |

2

j =1

j

j

,

m

∑ w ( x, y , z ) j =1

j

where w j ( x, y, z ) is a weighted function. When computing the coefficients in formula 1, we must refer to some non-zero-value points, e.g. the points that have a surface distance ± dist _ s in the normal direction, because the coefficients can not be computed only by zero-value points on the surface. In local coordinate, we can use a bivariate quadratic polynomial to fit local shape. Let us define local coordinates (u,v,w) with the origin of coordinates at （ xi + dist _ s ) such that the plane (u,v) is orthogonal to ni and the positive direction of w coincides with the direction of ni . A quadratic shape function at xi is given by following formula: f (u , v, w) = w − a0 − a1u − a2v − a3uv − a4u 2 − a5v 2

where (u,v,w) are the coordinates of each neighboring point in the new coordinate system. The unknown coefficients are determined by minimizing m

∑ w (u, v, w) | f (u, v, w) |

2

j =1

j

j

m

∑ w j (u, v, w)

.

j =1

3.2 Normal and Curvature Estimations If the normal at every point in point model is not given, it can be estimated by following methods. One method is based on principal component analysis (PCA) to neighboring points of the point [8,13]. The eigenvector corresponding to the smallest eigen value is regarded as the normal of this point. Another is based on the least squares method of the neighboring points. It computes the approximated plane, and the normal of this plane is regarded as the normal of the point. In this paper, we compute the point normal by PCA. It is described as following:

704

L. Miao and Q. Peng

Organize the input point set as kd-tree, finding the k-nearest points of each point, estimate the eigen value λ1λ2λ3 by eigen analysis of the covariance matrix of the local neighborhood and then regard the eigenvector corresponding smallest eigen value as the normal of this point. But above eigenvector can not tell whether the vector direction points in or out of the point set surface. So, we must make the normal consistent to all the points, which can be seen in Hoppe et al. [8]. Suppose three eigen values λ1λ2λ3 satisfy the relation: λ1 < λ2 < λ3 . The main curvature can be defined as: σ = λ1 /(λ1 + λ2 + λ3 ) , where the largest is 1/3 and the smallest is zero. The smaller the curvature is, the flatter the distribution of points is. When the curvature is zero, the distribution looks like ellipse; when the curvature equals to 1/3, the distribution of point looks like sphere. 3.3 Weighted Function The weighted function is used to determine the coefficients of quadrics function. In fact, it is a contribution factor of each neighboring point of xi for reconstructing the surface near point xi . The closer the neighboring point is from xi , the bigger the weighted factor is, and the greater the effect on surface reconstruction is. We adopted the bilateral filter as the weighted function because this filter can be used to de-noise in feature preserving surface: w j ( xi ) = a j f (| p j − xi |) g (| ( p j − xi ) ⋅ n p |) , j

⎛

| r |2 ⎞ is used for 2 ⎟ ⎝ 2 ⋅σ ⎠

where a j is the disk area of the point p j , Gaussians gauss(r ) = exp ⎜ −

both the spatial weight f and the influence weight g in this paper and σ = Rmax /(2 2 ~ 3 2) , Rmax = max(| p j − xi |, j ∈ (1, 2,3, m)) . Other robust influence weights could also be used such as the following single value function: 2

⎡ ( R − | xi − ray ( xi ) |) + ⎤ s j ( xi ) = ⎢ i ⎥ , Ri ⎣ ⎦ where | xi − ray ( xi ) | is the vertical distance from xi to ray and Ri is bound sphere

radius of the point xi . ( Ri − | xi − ray ( xi ) |)+ is zero if its value is less than zero, and it is real value if its value is greater than zero. 3.4 Building Adaptive BSP-Tree View the whole point set as the root node of BSP-tree, and compute the complexity of the local shape that is regarded as the subdividing condition. If the condition is satisfied, divide the node into two nodes (left and right) and recursively subdivide these nodes based on the complexity of local shape, until the complexity of local shape does not satisfy the subdividing condition. We call this BSP-tree as an adaptive BSP-tree based on a given local shape complexity, simply as aBSP-tree. The complexity of local shape depends on three factors: curvature, normal cone and the number of points in aBSP-tree node. Based on the complexity of the local shape

Point Sampled Surface Reconstruction Based on Local Geometry

705

of a point model, an adaptive Binary Space Partition (aBSP) tree was built. Each leaf node in the aBSP-tree was approximated by a weighted quadric function or bivariate quadratic polynomial as described in section 3.1. The center points set of all leaf nodes of the aBSP tree could be treated as one approximation to the original point set at a given complexity of the local shape. When there is no complexity limit in the process of building aBSP-tee, binary space partition will continue until there is only one point in its node. In this case the center points set of all leaf nodes in the aBSP tree is just the original points set, and the reconstructed surface in all its leaf nodes can be looked as the original point sampled surface. When different complexity of local shape is added in aBSP building process, the number of center points of leaf nodes in the aBSP-trees must be less than the number of original points set. Different complexity of local shape could generate different aBSP-tree. In this way, a multi-resolution representation for point sampled surface could be generated. 3.5 Sharp Features Reconstruction The coefficients of the quadrics function at each point are estimated by its neighboring points. Suppose xi be the center of one leaf node in an aBSP-tree and its neighboring point be x j { j = 0,1, 2,...m} . When xi is in flat area, all its neighboring points are included to calculate the coefficients of the quadrics function. But when xi is near sharp features, some of its neighboring points may be not included to calculate the coefficients of the quadrics function. Because near sharp feature there must exist some points that the angle between their normal n j and normal ni is greater than a given normal cone and normal cone is one of local shape complexity factors. Therefore according to the dot product of the normals n j ini , we can determine which points are not included in estimating the coefficients of quadrics function. This method is very simple and practical to reconstruct sharp features such as edges and corners. It is just by one method that any kind of sharp feature can be reconstructed. However, the method used in [11] is difficult because it is hard to know in advance that the sharp features is an edge or a corner, and therefore it can not tell how many parts in dividing its neighboring points. In our experiment, the dot product n j ini is in the range between 0.78 and 0.90. When the dot product threshold is set to be small, only highly sharp features can be reconstructed, and when the threshold becomes greater, more finer sharp features can be reconstructed. Therefore, dot product threshold can control the sharp feature reconstruction in different degrees. 3.6 Error Estimation Pauly et al. designed a method for estimating numerical and visual error between a simplified point set and its original point set by resampling points on a MLS local surface [13]. In this paper, we design an error estimating method by ray casting the reconstructing surface respectively from original point set and from aBSP-tree leaf nodes set with given complexity of local shape. The method is described as following:

706

L. Miao and Q. Peng

For each ray ri that starts from the point xi on original point sampled surface with its direction being the normal at xi , compute the intersecting points qi , qi ' with the reconstructed implicit surfaces respectively from original model S and the approximating model S ' ; and then calculate the distance d (qi , qi ') between these two intersecting points; define this distance as the approximating model’s error at point xi , shown in fig. 1; compute all such error and then calculate average error and max error which are defined as an average error Δ avg ( S , S ' ) and a max error Δ max ( S , S ' ) between the two models: Δ avg ( S , S ' ) =

1 ∑ d (qi , qi ') n i

Δ max ( S , S ' ) = max ri {d (qi , qi ')} .

qi’

ni

ni

qi’

qi S’

d(qi,qi’)

xi

qi xi

S

Fig. 1. Error estimation between two implicit surface

3.7 Rendering Point Sampled Surface When the model surface is being reconstructed in a aBSP-tree, we directly use a ray tracing method to render the reconstructed point sampled surface [14], which will save time from surface reconstruction to its display, because it avoids the process where the reconstructed implicit surface is triangulated first and then is rendered by some rendering method such as ray tracing.

4 Experimental Result and Analysis We implemented the above method on a commodity PC with P4 2.5 GHz CPU and 256M memory. Fig. 2(a) is the distribution of center points of all leaf nodes in the aBSP-tree of point model Icosahedron. From this figure, we can see that there are more points in edges and corners than in the flat areas. Fig. 2(b) is the result of ray casting reconstructed surface from fig. 2(a), while fig. 2(c) is the result of ray casting reconstructed surface from the original point set. Although the points in fig. 2(b) are just only one tenth of fig. 2(c), the rendering effects in the two cases are almost same even if there are some edges and corners.

Point Sampled Surface Reconstruction Based on Local Geometry

707

Fig. 2. (a) distribution of points in a aBSP (points:17972) (b) ray casting rendering result to reconstructed surface from (a) (c) ray casting rendering result to reconstructed surface from original point set (163842)

Fig. 3 is the results of ray casting the reconstructed surface from some original point set models. Fig. 4 and 5 are respectively the results of ray casting the reconstructed surface for point models vase and horse at different complexity of local shape. In these figures, from left to right, with complexity of local shape becoming simpler, the finer features gradually disappear and the average errors displayed in the caption below the figure are becoming bigger(where the average errors are relative to the diagonal length of envelope box of model and that is 3.464).

Fig. 3. Some original models rendered by ray tracing point sampled surface

Fig. 6 is zoom in results of ray casting reconstructing surface from the original model igea (left:50k) and one aBSP model igea (right:35k) with a given complexity of local shape. Although the number of points is decreased from 50k to 35k, the effect of rendering is almost the same and the average error is just 2.4030*(10-4)/3.464. So, this surface reconstruction method is very effective. Table 1 displays the time of reconstructing surface and average error of different models between their original point model and the simplified point model at different complexity of local shape. In this table, the local shape complexity of aBSP leaf node satisfies： main curvature 0, then its corresponding grid still belongs to AOI after moving. In this case, if the checkpoint (xp , yp) is beyond the scope of two AOI circles, then continue to check its next adjacent grid (i-1, j), otherwise, check the grid (i+1, j-1) in its lower row.

716

J. Jia et al.

In above, we have presented the major idea for a dynamic incremental AOI algorithm already. It is based the fact that we have known all the grids overlapped by AOI circle initially. Therefore, a static culling algorithm of grids overlapped by AOI should be designed in advance. Assuming that the checkpoint of current grid is P1 (x, y), the checkpoint of its adjacent grid is P2 (x, y+d), by introducing them into the discriminent D1 respectively, we get the following incremental form, which is fundamental for culling algorithm of grid overlapped by static AOI:

－

D1 (p2) = D1(p1) + 2d(y y0) + d2.

(4)

So, static incremental discriminant (4) is a function of width of grid d, and dynamic incremental discriminant (3) is a function of the step size of an avatar c. 3.3 Integer Incremental AOI Algorithm Both two incremental discriminants (3) and (4) above involve multiplications. It is necessary to optimize them by eliminating multiplications e.g. 2dy, 2dy0, d2, 2cy, 2cy0, c2 on c and d. we set c =1, 2, 4, …, and d = 2m (m is an natural number). Thus, not only multiplications can be avoided, but also our algorithms can be implemented in hardware. 3.4 Incremental AOI Algorithm Based on above ideas and formulation presented, we illustrated the detailed algorithm only in the case of first quadrant The C code of viewpoint moving upward in the first quadrant:

int d,r,c,D1,D2 int x0,y0,x,y,i,j; D1=(x-x0)^2+(y-y0)^2-r^2; D2=D1+2*c*(y-y0)+c^2; while(x>=x0) { if (D1>=0 && D2>=0) { D1 = D1+2d2-2*dx+2*dy-2*d*(y0-x0); D2 = D1+c2+2*c*y+2*d*c-2*c*y0; i++; j--; } else { if (D1>=0 && D2

sl( i ,o ) for

remote object instance

△t(l))

accept the update;//for remote object instance. //or “send out the update” for local object instance ELSE drop the update; ENDIF In this way, the bandwidth usage can be cut down and update fidelity of every LoIs is guaranteed. It should be noted that attributes of L0 is subscribed by all subscribers. When an attribute value update brings value of L0 attributes, the bit of L0 is certain to be 1 in the AND operation. So the critical attribute value updates won’t be dropped. It will be send out by the sending federate or be accepted by the receiving federate.

5 Experiment Evaluation In this section, simulation experiments are conducted to evaluate the flow control on low-bandwidth federates. We use four computers to conduct the experiment. The computer setup is shown in Table 4. The sender computers, S1 and S2, are responsible for a BH RTI containing 500 object instances individually. Each receiver computer, R1 or R2, starts a BH RTI receiving all the update messages of total 1000 object instances. To investigate the flow control independently in the receiving computers, flow control is disabled in the sender computers. In this way, we can see what ability the receivers own to treat the traffic. We assume the maximum bandwidth of R1 is 30 Mbps and that of R2 is 10 Mbps. All the incoming packets are looked on as “original data”, and packets after flow control are looked on as “data after flow control”. Then we can see the flow control efficiency in host R1 and R2. Table 4. Computer Setup for Experiment

Host Id

CPU

RAM

OS

S1

P4 3.2G P4 3.2G P4 3.0G P4 3.0G

1G 1G 512M 512M

winXP winXP winXP winXP

S2 R1 R2

LoI-Based Flow Control on Low-Bandwidth Federates

913

The experiments are carried out in 100M Ethernet network using Huawei Switch Quidway S3050. According to the bandwidth restriction of R1 and R2, f(l) is set for them. Here are the settings. Packet size: 220 bytes, including 10 attribute values, 8 bytes for each. S1, S2: no flow control. 500 objects in L2 R1: 400 objects in L2 and 600 objects in L1; f(L2) = 20, f(L1) = 5. R2: 400 objects in L2 and 600 objects in L1; f(L2) = 10, f(L1) = 2. The bandwidth usage results are shown in Figure 7. From the figure we can see that R1 and R2 have endured the total bandwidth more than 70 Mbps. But after flow control, the bandwidth is cut down to about 20 and 10 Mbps respectively. Then we select Bandwidth Usage 80

70 R1 original data R1 data after flow control R2 original data R2 data after flow control S1 data S2 data

bandwidth usage (M bit/s)

60

50

40

30

20

10

0 0

50

100

150

200

250

300 time(s)

Fig. 7. Bandwidth Usage Host R2's Object In LAYER__ABOUT 40

35

35

30

25

20

15

10

incoming data original data data of L1 after flow control

5

0 0

total updates per second (packet size: 220 bytes)

total updates per second (packet size: 220 bytes)

Host R1's Object In LAYER__VISION 40

30

25

20

15 original data data of L2 after flow control data of L1 after flow control

10

5

50

100

150

200

250

0 0

300

50

100

150

(a) R1’s Object in L1

200

250

300 time(s)

time(s)

(b) R2’s Object in L2

Fig. 8. Object Update Results

914

Z. Zhou and Q. Zhao

two object instances from R1 and R2 to check the update fidelity. From Figure 8 we can see that each object instance maintains a successive update frequency. The total update frequency of object instance in L2 is frequency of L1 plus that of L2 plus that of L0. The update frequencies of object instances of L2 are high enough in each LoI, L1 or L2, for the application to treat with. The update frequencies of object instances of L1 are also successive. The successive update in each LoI guarantees real-time update for each attribute with the help of LoI. We have implemented the LoI-based flow control in BH_RTI 2.2. The screenshot of an application is shown in Figure 9. Three computers are used to simulate 16 tank entities, including one wireless laptop. A PDA is used for the observer federate, which is based on a wireless connection of the bandwidth about 20 Mbps.

(a) Simulation Scenario

(b) Federate on mobile PDA

Fig. 9. Screenshot of the Interaction between wired and Mobile Federates

6 Conclusion This paper suggests to reduce the object updating frequency and to restrict the update interval based on LoI according to the federate bandwidth. A flow control model on low-bandwidth federates was built on the basis of LoI. This model depicts the way to conduct flow control with fidelity guarantee. Flow control algorithms are proposed to drop update packets according to the object LoI and packet LoI. Experiments illustrated that the flow control model can control the bandwidth of flow on federate bandwidth requirements.

Acknowledgement This work is supported by the National 863 High-tech program of China under grant number 2006AA01Z331 and National Natural Science Foundation of China under Grant No. 60603084.

LoI-Based Flow Control on Low-Bandwidth Federates

915

References 1. Defense Modeling and Simulation Office, High Level Architecture interface specification version 1.3, April 1998. 2. IEEE standard for modeling and simulation (M&S) High Level Architecture (HLA) – Framework and rules. (IEEE Std 1516-2000). New York:The Institute of Electrical and Electronics Engineers Inc., 2000. 3. IEEE standard for modeling and simulation (M&S) High Level Architecture (HLA) – Federate interface specification (IEEE Std 1516.1-2000). New York:The Institute of Electrical and Electronics Engineers Inc., 2000. 4. IEEE standard for modeling and simulation (M&S) High Level Architecture (HLA) – Object Model Template (OMT) Specification (IEEE Std 1516.2-2000). New York:The Institute of Electrical and Electronics Engineers Inc., 2000. 5. Pullen M, Myjak M, Bouwens C. Limitations of Internet protocol suite for distributed simulation in the large multicast environment. RFC 2502, 1999. 6. S.Bachinsky, J.Noseworthy, F.Hodum, Implementation of the Next Generation RTI, Spring Simulation Interoperability Workshop, Orlando, FL., USA, 1999. 7. Defense Modeling and Simulation Office, Department of Defense. 2002. RTI 1.3 – Next generation programmer’s guide version 6. http://www.dmso.mil/. 8. Mikael Karlsson, Lennart Olsson, pRTI 1516- Rationale and Design. Fall Simulation Interoperability Workshop, Orlando, FL., USA, 2001. 9. Douglas D Wood, Len Granowetter. Rationale and Design of the Mak Real-Time RTI. Spring Simulation Interoperability Workshop, Orlando, FL., USA, 2001. 10. J.O. Calvin, C.J. Chiang, S.M. McGarry, S.J. Rak, D.J. Van Hook, M.R. Salisbury, Design, implementation, and performance of the STOW RTI prototype (RTI-s), Proceeding of Spring Simulation Interoperation Workshop, 1997, Paper 97S-SIW-019. 11. Suiping Zhou, Stephen John Turner, Wentong Cai, Hanfeng Zhao, Xiaolin Pang. A Utility Model for Timely State Update in Distributed Wargame Simulations. Proceedings of the 18th workshop on Parallel and Distributed Simulation. pp. 105-111. 2004. 12. BH RTI 2.3 User Guide. http://www.hlarti.com/ 13. Zhou Z, Zhao QP. Reducing time cost of distributed run-time infrastructure. the 16th International Conference on Artificial Reality and Telexistence, Hang Zhou, Nov. 2006 14. Zhou Z, Zhao QP. Research on RTI congestion control based on the layer of interest. Journal of Software, 2004,15(1):120~130. 15. Zhou Z, Zhao QP. Extend HLA with layered priority. Proceedings of the Spring Simulation Interoperability Workshop. Orlando FL, 2003. Paper 03S-SIW-012.

Dynamic Game Level Generation Using On-Line Learning Jongyeol Yang, Seungki Min, Chee-Onn Wong, Jongin Kim, and Keechul Jung HCI Lab., School of Media, College of Information Technology, Soongsil University, 156-743, Seoul, S.Korea {yjyhorse,dfmin,cheeonn,eyezzang01,kcjung}@ssu.ac.kr

Abstract. In recent years, many researchers are attracted to computer games research. Capable gamers can easily get bored, while beginners tend to give up after trying several times because the game does not correspond to their level of interest. Therefore, this paper proposes that the user's play pattern to be modeled on the basis of probability and level designer will dynamically generates the gaming level accordingly. We analyze user's play pattern and design pattern based on GMM (probability model) and dynamically generate the level with online learning technique adapting the reinforcement technique. The play pattern is modeled using GMM and in order to create game level dynamically, the method of updating the weight of enemy creation using online script is proposed. Finally, we apply our proposed method to a 2D shooting game and introduce user’s play pattern leading to design pattern in the game.

1 Introduction For a long time, computer games have developed hand-in-hand with the development of hardware and computer graphics. In contrast our interest in computer games has emerged as a new area of research in recent years [1]. How interesting the game is to the user usually depends on the player’s gaming level. Capable gamers can easily get bored, while beginners tend to give up after trying several times because thegameis too complex for them to enjoy [2]. The games level of most computer games is controlled by level designers. Level designer composes a game based on the user’s skill. Naturally, to make up for the shortcomings resulting from this, some other games using system environment settings are also available so that the usercan choose in accordance to their own choice [3]. However, even these are not good enough as they only change the parameters. As a consequence, many researches are currently being conducted to dynamically alter the game level. In game, research using Artificial Intelligence (AI) technology has been done to showNon-Player Character (NPC) action realistically and to implement competitive strategies and tactics. Techniques such as finitestate machine, intention decision tree, and agents are the methods in which, the stateor circumstances of the game is modeled, and, on the basis of this, the next state of NPC is decided. On thewhole, these methods are easy to handle and expandable. However, since most variables are decided in advance before the beginning of the game, they often fail to provide the gamer with the diverse and complex circumstances K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 916 – 924, 2007. © Springer-Verlag Berlin Heidelberg 2007

Dynamic Game Level Generation Using On-Line Learning

917

in the game. Another disadvantage is that skillful users can easily understand the pattern of NPC’s next action and they can predict the game and easily loose interest in the game. There are some other methods to solve this problem. One of it is AI adaptation using machine learning [4]. This method was first used mainly for board games or card games, but recently, as hardware technology being constantly improving, it has been applied gradually to commercial games. In this method, Neural Network or Genetic Algorithm has been generally used and the level of the game is decided based on the user’s skills [5]. However, game level designers experience a hard time controlling the game with this method because the result varies according to the data and duration of learning, which is the natural characteristic of the algorithm. The previously mentioned methods are the ones implementing the realistic action of a NPC. They are different from the ones which offer a dynamic level of difficulty to users. Currently, researches are being carried out in order to provide a continuation of interest to the user. One of the examples is adaptive game in which the level and the circumstances of the game are based on the user’s current skill. Player modeling technique based on data mining classify the type of user by analyzing and observing the player’s play style during the game [6]. Based on this, the user is given a specific level of game that the player can handle. Also, there is a research on the method in which the circumstances or resources of the game are controlled according to the user’s skill level by analyzing the player’s behavioral data (play data) systematically and thus predicting the state of the player that will be most probable to show in the future. This paper proposes that the user’s play pattern to be modeled on the basis of probability and level designer dynamically generates the level of the game as per suitability. The play pattern is modeled using GMM and, in order to generate level dynamically, the method of renewing weight of enemy creation with an online scripting is proposed. The remainder of the paper is as follows. In section 2, we describe play pattern modeling. Section 3 describes dynamic level creation followed by section 4 describes implementation steps in general and evaluates the availability of the proposed method with the test included. Finally, we draw our conclusion and future work in section 5.

2 The Player Modeling We need to analyze player’s play pattern in order to adapt the game closely towards the level that the designer has in plan. Player parameters for both x and y position together with the sequence of the movement are the basis for the model. Using the analyzed pattern, we want to dynamically generate composition of level so that it achieves the design pattern as closely as possible. By comparing the similarities between the two models we have made, we generate the enemy pattern which will lead to closer similarities. Online learning adapting reinforcement learning with comparison of enemy generation scripting is used here as the design pattern of the game engine. We assume the correspondence of user, which was mentioned previously, is the probability in the feature space that is to be analyzed.

918

J. Yang et al.

This paper uses GMM (Gaussian Mixture Model) to model user’s play pattern. GMM can model a large number of distributions on data space, so it can analyze data more accurately [7]. Density function consists of the number of k. This shows in equation (1). k

G(X Θ) =

i =1 k

p ( i ) = 1,

p ( i ) g ( X θ i ) where,

(1)

p (i ) ≥ 0

i- 1

k In this equation, x is feature vector, Θ is { p (i ),θ i }i =1 and θ i is average, covariance of each component as coefficient vector. p (i ) indicate a prior probability that may happen in ith component. g is multivariate Gaussian distribution function. Also, this paper proposes that we use EM (Expectation-Maximization) algorithm to estimate Gaussian parameters that each component is the greatest log-likelihood of data [8]. To measure approximation of distance apart modeled GMM, previous research [reference] uses technique based on Euclidian distance based on average point of GMM distribution. However this technique doesn’t influence directly of each GMM distribution [9]. This paper also introduces metric such as KL distance that normalize the distance apart two probability models. KL distance is generally expressed like equation(2) and can get more exact value because it considers not only the average of each component, apart from Euclidian distance, but also distribution configuration(that is distribution.)

KL( f || g ) =

f log

f g

(2)

In the equation, both f and g represent GMM, respectively. However equation(2) is not closed-form, so we propose to use estimate value of KL distance on data space to implement KL distance by introducing Monte Carlo simulation technique. Equation(3) is applied Monte Carlo simulation by equation(2).

KL( f || g ) =

In equation(3),

x1 ,..., x n

f log

f 1 ≈ g n

n t =1

is sampling value from

f ( xt ) log

f ( xt ) g ( xt )

(3)

f (x) .

3 Dynamic Level Generation In game, the composition of each level is composed of a set of scripts to create enemy pattern. Generally, enemy creation scripts have been defined already by level designer before game is executed.

Dynamic Game Level Generation Using On-Line Learning

919

This paper proposes the design pattern for the game engine to adapt to player’s model. The design pattern of dynamic level generation uses a comparison of scripts that creates dynamic level to analyze user’s play pattern when game is played. Each script has the weight to create a script dynamically. Then we select a script using its weight value in probability [10]. Therefore we introduce on-line learning technique adapting the reinforcement learning technique to update the weight while the game is being played. As the game is played, proper consideration and discipline value are reflected to the weight based on the distance of design pattern after analyzing the user’s play pattern. The script which closely generate design pattern has a high weight and residual script has a low weight. Weight renewal function to update weight of script equation(4) can be written as:

∆W =

b−F b F −b 1− b

− Pmax

{F < b}

Rmax

{F ≥ b}

(4)

Updated weight is W+_W and W is current weight. In this equation, max P is maximum penalty and max R is maximum consideration. The value b is break-even value and b is a role not to change when F is to b. F is fitness function and it plays an important role on weight update function. F is material efficiency and can be written as equation(5):

1 F=

2 ( b −1)( K − K ' / 2 ) K

{K < K ' / 2}

+1 {K ' / 2 ≤ K ≤ K ' } − 2b( K − K ' ) + b {K ' < K ≤ 1.5 K '} 0 {1.5 K ' < K }

(5)

In the equation, K is the KL distance between user’s pattern of current script and design pattern. K’ is the KL distance between user’s pattern of previous script and design pattern. When K and K’ are equal, F has b value and _W becomes 0 such as equation(4). The graph of the _W and F is shown in Fig. 1. We define the function F as boundary of k so that it F exist between 0 and 1 shown in Fig. 1.

920

J. Yang et al.

Fig. 1. Weight update function and fitness function

4 Implementation This paper proposes that we model user’s play pattern applying 2D shooting game and dynamically control enemy’s pattern to create an interest or level of designer’s wish as mentioned previously [11]. We will analyze the user’s movement pattern to model based on GMM. In Shooting game, the user’s movement pattern is an importantdata which reflects variable pattern of the enemy and level. The user’s movement pattern is a set of (x,y) coordinates which users’ positions are extracted by regular cycle during the game play (Fig. 2). In Fig. 2, ellipses are example of modeling data based on GMM.

Y

X

Fig. 2. User’s position sample and GMM

Dynamic Game Level Generation Using On-Line Learning

921

As per described in the earlier sections, player pattern modeling is based on GMM during a game play. At the same time, the game design pattern generates the gaming script dynamically to alter the level of the game and use reinforcement learning technique to adapt based on weight adjustment so that the script suits the player’s model. The detailed process for dynamic script creation is as follow. 1. Set each weight equally as 1/N to the number of N script. 2. Progress the game as generating the number of N script once. We calculate KL distance between design pattern and modeled user’s play pattern of each script based on GMM. Then we update the weight using this value. Unlike equation( 4), we define maximum of weight updating and distribute weight updating value based on KL distance of each script. 3. When new script is generated, execute the process as follows. 3-1. Select a script as probability based on the weight of each script. 3-2. Model user’s play pattern which is accumulated on the selected script based on GMM. 3-3. Update the weight of selected script as mentioned in equation(4) above. 3-4. Update the weight of non-selected scripts to set the sum of all scripts. This paper uses Final Mission 1 (FM1) as the experimental scrolled based shooting game. This paper defines 7 enemy pattern creation scripts for testing and each script is composed by enemy plane, start position and attack type. The script is continued executing for a minute and new script is generated if time passes or enemy plane doesn’t exist on the screen. User’s play pattern is sampled 6 times per second and the sampled pattern is modeled GMM by EM algorithm. Fig. 3 shows the interface of the game.

Fig. 3. Interface of FM1

Fig. 4 defines design pattern and model user’s play pattern of enemy creation script based on GMM in step 2 during the step of dynamic script creation. In this way, the game is dynamically generates the suitable level based on the player’s model. Our motivation for this is to generate a higher level of interest and playability to the player so that the game is not perceived as a dull and boring game. The nature of the game is a simple shooting game, thus maintaining an interest in the game is an essential challenge for this game genre. In Fig. 5 we show the weight change based on script generation.

922

J. Yang et al.

Fig. 4. User’s play pattern of script

Fig. 5. Script weight change

Fig. 6 (b) depicts user’s play pattern of script s1 in step t and the weight increasesmore than step t-1 because b gets near the design pattern of designer’s intention as shown in Fig. 6.

Dynamic Game Level Generation Using On-Line Learning

(a)

923

(b)

Fig. 6. (a) step-by-step change of script weight (b) user’s play pattern S1 in step t.

Fig. 7 is KL distance between user’s play pattern and design pattern while creation of dynamic script is executed. The KL distance value decreases while processing the previous step as shown in Fig. 7 and user’s play patterns are gradually conversed to the design pattern.














































    

       

Fig. 7. Change of the KL distance between user’s pattern and design pattern.

5 Conclusion Among the existing games, most level designs are passively defined by level designers. Therefore the game can be boring because the user can adapt the playing pattern to suitable to themselves easily. This paper proposes that we generate levels dynamically while the game is being played by the user during game modeling. It is modeled using user’s play pattern based on GMM in order to process as the level designer intended the game to be executed. The degree of similarities between the

924

J. Yang et al.

user’s play pattern and the design pattern can be more accurately obtained by using KL distance. This paper also proposes that we continuously update the weight of enemy creation script through an introduction of on-line learning method. As indicated by tests, the user could progress in the game by adapting themselves to the various users’ play pattern. So far we have considered only one play pattern domain which is the user’s position. Future works include applying more pattern domains simultaneously. The limitation in this research is time factor being left out in the user’s play pattern. This can be very important in some games. By diversifying user’s play pattern domain, we can apply it to other type of games such as sports game or First Player Shooter (FPS) game. It can be extended to on-line game which needs updating continuous whereby analyzing of user’s play pattern can be an important data to upgrade the game’s playability and continuous interest in the game. Acknowledgments. This work was supported by the ‘Seoul R&BD Program (10581 cooperate Org 93112).

References 1. Noah Falstein, "Game Developer Magazine," The Flow Channel, 2004. 2. Raph Koster, "Theory of Fun for Game Design,"Paraglyph Press, Phoenix, 2004. 3. Daniel Johnson, Janet Wiles, "Effective Affective User Interface Design in Games," International Conference on Affective Human Factors Design, Singapore, 2003. 4. John E. Laired, "Using a Computer Game to Develop Advanced AI," IEEE Computer Society Press, pp.70-75, July, 2001. 5. Bernd Freisleben, "A Neural Network that Learns to Play Five-in-a-Row," International Conference on Artificial Neural Networks and Expert Systems, pp.87-90, 1995. 6. I.Faybish, "Applying the Genetic Algorithm to the Game of Othello," Master's thesis, Vrije Universiteit Brussel, Computer Science Department, Brussels, Belgium, 1999. 7. Todd K. Moon, "The Expectation-Maximization Algorithm," IEEE Signal Processing, Vol. 13, pp.47-60, 1996. 8. Chad Carson, Serge Belongie, Hayit Greenspan, Jitendra Malik, "Blobword: Image Segmentaion Using Expectation-Maximization and Its Application to Image Querying," IEEE Trans. On Pattern Recognition and Machine Analysis, Vol. 24, No. 8, pp. 10261038, Aug. 2002. 9. Richard O. Duda, Peter E. Hart, David G. Stork, "Pattern Classification," 2nd Edition, pp. 55, John Wiley & Sons Inc, 2001. 10. Imran Ghory, "Reinforcement learning in board games," Technical Report CSTR-04-004, Department of Computer Science, University of Bristol, May 2004. 11. Pieter Spronck, Ida Sprinkhuizen-Kuyper and Eric Postma, "Difficulty Scaling of Game AI,"International Conference on Intelligent Games and Simulation, Belgium, pp. 33-37, 2004.

Development of Online Game-Based Learning for TRIZ Wai Lap Leung and Kai Ming Yu Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University [email protected]

Abstract. The paper reports how to assist university learners to study TRIZ, which is a series of useful tools, methods and strategies for solving inventive problems systematically, by means of computer games. The aim of this project is to help players to practice and understand their basic TRIZ knowledge and technique effectively and efficiently by playing online mini games. The paper goes through several major tools of TRIZ, Inventive Principles, Trends of Evolution, Contradiction and Engineering Parameters, and finally evaluates their educational and entertainment values to the player in terms of level design, feedback and interaction. Keywords: TRIZ, online game, edutainment.

1 Introduction In order to maintain their competitiveness, most companies face a challenge that they have to develop new products and processes with limited resources, lowest cost, higher quality and shorter design time. Not only they have to solve problems during development, but they also need to have breakthroughs. TRIZ (Teoriya Resheniya Izobreatatelskikh Zadatch), which is the Soviet initiated Theory of Inventive Problem Solving, is introduced to define and then to solve any inventive problem or opportunity situation systematically. TRIZ is a philosophy, a method and a series of tools (Figure 1). To access the good solutions, problem definers and problem solvers use the large majority of the TRIZ tools in order to map their specific problems and solutions to and from a generic problem-solving framework (Figure 2). It is, however, time-consuming to learn and gather a working knowledge of the whole structure [1]. There are a few examples of studying TRIZ by games. For instance, the ideal final result concept of TRIZ can be illustrated via the tangled string game [2]. The classic floating crap game is quoted as an example for TRIZ Principle 28: Substitution for mechanical means – (c) change from static to movable fields, from unstructured fields to those having structure [3]. Creax (www.creax.com) and Hape Etzold (www.etzold.biz) [4] have both developed packs of card game on TRIZ inventive principles. However for digital game-based education of TRIZ, most of them are for children. The player may have some understanding after playing the game, but there is little text description and full of childish cartoon which it is not suitable in tertiary learning. For example, “Vanki-Vstanki Studio” (Figure 3) is a game for learning K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 925–935, 2007. © Springer-Verlag Berlin Heidelberg 2007

926

W.L. Leung and K. M. Yu

Inventive Principles of TRIZ, where the game environment is a virtual school-studio [5]. The player can learn those principles by constructing self-righting dolls with different components and seeing the principle of operation inside. It is effective enough, but not efficient in university learning.

Fig. 1. Hierarchical View of TRIZ

Fig. 2. General TRIZ Process

There is another famous PC game, Invention Highway (Figure 4), developed by Roni Horowitz (http://www.start2think.com) and marketed via Compedia (www.compedia-usa.com). This game actually introduces an Advanced Systematic Inventive Thinking (ASIT) method (may consider as a dialect of TRIZ). ASIT is also designed for teaching people to think more creatively and solve problems more

Development of Online Game-Based Learning for TRIZ

927

effectively. Invention Highway possesses appropriate explanation and many interesting simulation games that may suit tertiary learning. However, its file size is large (172MB for demo) and can only be accessed on the PC that has it installed. Another ASIT game, Puzzling Time Quest, is solely targeted for kids of ages 7 to 12 [6].

Fig. 3. Vanki-Vstanki Studio

The aim of this project is, thus, to develop a series of online mini games on TRIZ, both concept and application, for university students. Via the games, players (a) can practice the knowledge and technique of TRIZ they learnt during lessons, (b) can familiarize TRIZ concepts and their usages within a shorter period, and (c) can review on their own pace by selecting appropriate game of varied difficulty level.

2 Game-Based Learning With game-based learning, knowledge gained can be directly transferable and applicable to the real world and to unfamiliar scenarios, whereas the normal classes which have many of the limitations cannot achieve [7]. Also, game-based learning can support the development of (a) strategic thinking, (b) the use of logic, (c) memory, (d) problem solving and (e) critical thinking skills. As TRIZ is utilized in innovation of real life problems, game-based learning matches perfectly in studying TRIZ. Considering most of the students can access the games anytime and anyplace, as well as can be assessed instantly by the supervisor, all games are made online. In order to embed the games with interaction and to play the games in the browser more conveniently, Adobe Flash (http://www.adobe.com/products/flash/flashpro/) is used for the game implementation in this project.

928

W.L. Leung and K. M. Yu

Fig. 4. One of game in Invention Highway

3 Mini Games TRIZ is a huge topic composed of many concepts, such as the 9-Windows, 40 Inventive Principles, 31 Trends of Evolution, 39 Engineering Parameters, 76 Standard Solutions, Contradiction Matrix, etc. At the introductory level, the first mini game, TRIZ Search provides an overview of TRIZ terms. Then, to help the students to focus on each concept one by one, from easy to hard, games: (a) 40 Principles, (b) Trends of Evolution, (c) 9-Windows, (d) Contradiction, (e) Engineering Parameters, and so on in the future, are created for each specific concept. 3.1 TRIZ Search TRIZ Search is developed to introduce the tons of terms to the players categorically. It is a crossword puzzle game (Figure 5), where TRIZ terms are allocated to different groups according to different concepts or theories (Figure 6). Choosing one of the categories, a reasonable number of the corresponding terms will be used for generating a new puzzle, the players can then memorize those vocabularies bit by bit easily. Also, with the help of the categorization, players can become more aware of the relationship between a specific category and a specific term.

Development of Online Game-Based Learning for TRIZ

Fig. 5. Crossword puzzle game

Fig. 6. Different categories of TRIZ

929

930

W.L. Leung and K. M. Yu

3.2 40 Principles 40 Principles is developed to grab the players’ attention on one important TRIZ tool, the 40 Inventive Principles. The game runs in the way that different Principle objects are falling down and the avatar below is trying to collect them (Figure 7). However, there exist rules to determine whether the Principle object can be collected or not (Figure 8). Therefore, the players are always required to distinguish a specific Principle object from the others quickly. Also, when getting to a more difficult level, less information about the specific Principle object will be given (Figure 9). So what the players rely on will be shifted from (1) principle number to (2) principle graphic to (3) principle name to (4) principle description. This design let the players recognize the Principles as well as their meanings behind. 3.3 Trends of Evolution This game is developed to let the players be familiar with different evolution trends. There are 31 Trends of Evolution, where each trend belongs to one (some trends belong to two) of perspectives: (a) Space, (b) Time, and (c) Interface. The scene is simply composed of several bubbles and a car (Figure 10). The players control the car and then different events will be triggered depending on which kind of bubble is passed. After choosing one of the perspectives, bubbles of some daily examples of one trend as well as the explanation of the trend (Figure 11) will appear. The players can complete the trend by simply passing the examples one by one to connect them in ascending order (Figure 12). Solving a certain number of trends successfully will result in an evolution of the scene and the car in that perspective (Figure 13). Otherwise, degeneration is resulted instead. It is interesting that the players who play the game about evolution are themselves evolving too. In the process of evolution/degeneration, the players can make their decision for which perspectives come first and this may change the difficulty. 3.4 9-Windows This game is developed for players to get use to the usage of System Operator, another name of 9-Windows. In the game, those 9-Windows become nine tic-tac-toe squares, i.e. a tic-tac-toe game (Figure 14). A little bit difference from the normal tic-tac-toe is that the square can be marked with the players’ representative sign (X/O), only after the player can select and place the correct block in that square. If a wrong block is placed, a blank square, if any, will be assigned randomly and this turn is certainly missed. Those blocks are either text or pictures and they should match the corresponding system and time perspectives. Three levels of difficulty are implemented, where a minimax strategy is used in the highest level to enhance the fun.

Development of Online Game-Based Learning for TRIZ

931

Fig. 7. 40 Principles game play

Fig. 8. Rule in the game

Fig. 9. Less information given in more difficult level

932

W.L. Leung and K. M. Yu

Fig. 10. Scene of ‘Trends of Evolution’

Fig. 12. Connect in ascending order

Fig. 11. Explanation of the trend

Fig. 13. Evolution in perspective ‘TIME’

3.5 Contradiction This game is developed to let the players who have established the basic knowledge of Technical Contradictions to practice. Contradictions always take place in the process of invention, and have to be identified and then tried to eliminate. The players are asked to select a pair of improving and worsening factors which come out a contradiction (Figure 15). 3.6 Engineering Parameters This game is developed to enable the players to connect the real life conflicts to the engineering parameters. There is a large disk with 39 engineering parameter names filled in different sectors (Figure 16). According to the conflict given in the right top corner, the players are required to match it to the corresponding parameter by wheeling the disk.

Development of Online Game-Based Learning for TRIZ

933

Fig. 14. Text in the block must match the system (sub-system) and time (present) perspectives

Fig. 15. A pair of improving parameter (Power of engine) and worsening parameter (Cost) are selected

934

W.L. Leung and K. M. Yu

Fig. 16. A wheeling disk of 39 Engineering Parameters allocated in different sectors

4 Conclusions 4.1 Educational Value The TRIZ mini games provide a chance for students to review what they have learnt in lesson. The main feature is that those games of the introductory level give information bit by bit, while those games of the higher level give less information but emphasize the accuracy of content. And instant feedbacks are provided to let students to recognize their learning progress. As the second phase of development, games with strategic thinking or decision making for intermediate students, will be added in the future. 4.2 Entertainment Value Each TRIZ mini game is designed according to the theme or the usage of the particular TRIZ tool. Players have a large degree of freedom to pick whatever they prefer to play. They can also experience different scenarios and approach different objectives in each play. Different challenges are always ready for the players to overcome every time they

Development of Online Game-Based Learning for TRIZ

935

get the higher level. As the second phase of development, games with larger scale will be made in the future. Acknowledgements. This research was funded by grant no. 997F from the research fund of The Hong Kong Polytechnic University.

References 1. Mann, Darrell. Hands-on systematic innovation. Creax (2002) 2. Mann, Darrell, “Case Studies In TRIZ: Anti Red-Eye Flash Photography”, TRIZ Journal (July 2001) http://www.triz-journal.com/archives/2001/07/f/index.htm 3. Terninko, John, “40 Inventive Principles with Social Examples”, TRIZ Journal (July 2001) http://www.triz-journal.com/archives/2001/06/a/index.htm 4. Lamia, Walter Lamia, “Review of “Solution – Innovation Card Game” TRIZ Journal (December 2003) http://www.triz-journal.com/archives/2003/12/j/10.pdf 5. Kozyreva, Nelly and Novitskaya, Elena, “Methods of Teaching TRIZ Principles Through an Example”, TRIZ Journal (August 2002) http://www.triz-journal.com/archives/2002/ 08/a/index.htm 6. Micaiah C. Slocum (11 years old) and Hosannah M. Slocum (9 years old), “Two New Games That Teach Children and Their Parents ASIT (From a Kid’s Perspective)”, TRIZ Journal (January 2004) http://www.triz-journal.com/archives/2004/01/2004-07.pdf 7. Moretti, Michela and Dondi, Claudio. Guide to quality criteria of learning games. SIG-GLUE (2006) http://www.sig-glue.net/documents/SIG-GLUE_Guide.pdf.

A Distributed System for Augmented Reality Experiences in Science Centers and Museums Michael Wittkämper, Anne-Kathrin Braun, Iris Herbst, and Jan Herling Collaborative Virtual and Augmented Environments Department Fraunhofer Institute for Applied Information Technology FIT Schloss Birlinghoven, 53754 Sankt Augustin, Germany {michael.wittkaemper, anne-kathrin.braun, iris.herbst, jan.herling}@fit.fraunhofer.de

Abstract. This paper describes an Augmented Reality based approach for use in museums and science centers for explanation, which is designed to improve visitors understanding. The distributed Augmented Reality system uses hybrid tracking technologies to detect the position and orientation of the museum visitor and measures a specific exhibit state with data acquisition sensors. Furthermore, we enhanced our system with an Augmented video streaming technology for remote participation and we developed the Mixed Reality Interface Markup Language (MRIML) to support high-level, platform independent scenario specification. Keywords: Augmented Reality, AR Streaming, AR/MR User Interface.

1 Introduction Making the invisible visible is a desirable ability for many situations. In the context of a learning environment it would be useful to support the comprehension of complex scientific relations and the understanding of difficult technical processes. Augmented Reality (AR) systems, as defined by Azuma [1], are a suitable technique to make that possible. With AR, virtual objects can be superimposed in real-time into the real world. A major benefit of AR systems (as compared to Virtual Reality systems) is that they support presenting information in the spatial context of a physical environment. Users are allowed to perceive complex phenomena and underlying physical models in situ such as science centers and museums that gather and concentrate exhibits for playful learning experiences. While interacting with an exhibit, the visitor can gain additional visual and acoustic information, such as interpretations of acting forces. In that way, Bernoulli’s principle of lift could be explored by turning an airfoil model, where the force vectors and the airflow are displayed with respect to the current angle (see Figure 1). With a biological exhibit, aspects of the photosynthesis could be explored by changing the environmental conditions such as the lighting conditions. The visitor could then observe the changing amount of CO2 and O2. In this paper we present an AR system, which realizes the previously outlined scenarios. More generally, it enables visitors to explore exhibits in science centers and museums or other exhibitions, it supports exhibit designers to specify the virtual content and its behavior, and it allows external visitors to participate from remote. K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 936 – 946, 2007. © Springer-Verlag Berlin Heidelberg 2007

A Distributed System for Augmented Reality Experiences

937

Fig. 1. Augmented airfoil exhibit

The AR experience is provided by three technologies. They address the on-site experience, the creation of content as well as remote participation as follows. The distributed AR System (Section 3) includes mobile user and stationary exhibit AR units (or systems). The user AR system comprises a wearable computer mounted on a back-or belt-pack, a wireless mouse as well as a head mounted device including a seethrough display and head tracking devices. The user system detects the visitor’s position and viewing direction by using hybrid tracking combining computer vision and inertial tracking. It presents various types of media content, such as images, sound, videos and more sophisticated 3D objects and animations On entering the exhibition it is handed over to the visitor. After putting it on, the user can freely move within the exhibition hall and explore all the exhibits, either by real or virtual interaction techniques. (In the special case that the user only moves within a very limited area, the wearable computer can be stationary and the user just puts on the glasses). Exhibit specific devices such as sensors, actors and object tracking systems are shared amongst users and are connected locally to one of the stationary AR systems. All AR systems within one museum participate in one distributed AR application. As within the real exhibits in a museum the virtual components have to be created by exhibit designers. Different scenarios for each exhibit can be created, comprising representations of the real exhibit, the additional virtual objects as well as their behavior. The Mixed Reality Interface Markup Language (MRIML, Section 4) was developed to allow for a high-level definition of such scenarios. To allow external visitors to participate from a remote location, we extended our system by using an Augmented streaming technology (Section 5).

938

M. Wittkämper et al.

Section 6 concludes with a brief discussion and gives a short view to our future work. But first we provide a short overview of AR technology used in exhibitions in the following Section 2.

2 Related Work There are already various kinds of AR supported exhibitions each using a range of different sensing technologies and display types. One frequently used form of sensing technology is the marker-based tracking, which is used within an HMD-based system for geographical education ([14], [15]) or for mathematics and geometry education ([5]). In [5] a handheld tracked panel is used to interact with the scene. In [14] a component-based system for handheld applications can be found. The authors present an educational game called Mr. Virtuoso, where the players have to pick up virtual artwork, by clicking on it in the PDA display. The handheld device shows a video-see through AR scene. Several PDAs, which are connected via WiFi, can be used. The system uses OpenGL ES, ARToolKitPlus for tracking and a networking middleware. Another output-device is presented in [20]. Here the authors use a Handheld AR visor, which consists of a display screen and a small camera. For orientation and position tracking ARToolKit is used. To enable collaboration, two handheld visors can be connected to a single computer. Furthermore two large LCD panels show what is being seen in each handheld visor. Interaction with the AR exhibit is performed by using a slider. A lot of work has also been carried out with non-marker-based approaches. A mobile AR system with a head-worn display like in [17] is used in [14], where the position of the visitor is tracked with an infrared location-identification sensor which is situated inside the light fixture at the ceiling. The AR system plays audio-visual information about the exhibit, which is detected by the visitors’ position. In [16] RFID-technology is used. Each visitor has a wirelessly connected handheld device containing an RFID tag. The exhibits are equipped with RFID readers. Additionally, conventional Internet technologies are used to augment the real world. Using the sensors, the visitors can pick the suitable webaddress of each exhibit, then the information can be shown in a web browsers on the handheld device.

3 Distributed Augmented Reality System For the AR system three basic requirements can be identified leading to the following design decisions. • Advanced augmentation of exhibits by virtual objects: Simple 3D objects, complex 3D phenomena representations, images, audio, video, and text are to be displayed. This requires a strong processor with hardware graphics acceleration. • Mobility: The visitor should be able to freely move inside the museum. Wired connections to fixed points in the museum are not possible. • Scalability: Many visitors should be able to use the distributed AR system independent from each other. As the bandwidth for wireless communication is limited inside a museum, the usage of mobile thin clients connected to stationary high-end servers through wireless connections is not possible. As a consequence, the AR system consists of independent self-contained mobile systems on which all visitor specific operations are performed. They are completed by stationary AR systems, also called exhibit AR systems that connect the exhibit specific devices. All systems are connected through a 54Mbps high-speed wireless network.

A Distributed System for Augmented Reality Experiences

939

To deliver the Augmented Reality experience to the visitor, the following functionalities have to be performed by the mobile AR system: Determining the visitor’s head position and orientation (head tracking) Presenting the real view captured by a camera in a video see through display (video augmentation) Presenting virtual objects in the display that augment visualization of exhibits Supporting interaction with the scenarios through virtual head-up menu and exhibit parts Recording the visitor’s view for use after visit (image and video capturing) Recording the experiment variables for use after visit (data capturing) Streaming the visitor’s view to a server that relays the video stream to remote locations Operating a bidirectional audio communication line between visitor and remote locations In addition, the augmented exhibits offer specific functionalities, such as Determining the position and/or orientation of exhibit’s parts (object tracking) Acquiring data measured by sensors (data acquisition) Control specific variables of the environment The functionalities of the mobile AR system are provided by the hard- and software components running on a mobile system. The mobile AR system consists of the 3 hardware components: a wearable computer on back- or belt-pack, a headset with display and tracking hardware, and a wireless mouse.

3.1 Wearable Computer The wearable computer either comes as back- or belt-pack. Usually the backpack unit consists of a stripped baby carrier mounted by a high-end laptop with stereo graphics capabilities (see Figure 2). All software components are run on this system and all other user specific hardware components are connected to it. It is the heart of the system and is based on an Intel Pentium-M processor running at 2GHz with 1GB of main system memory. The graphics board is an NVIDIA Quadro FX Go 1000 with OpenGL quadbuffered stereo support. A small bag fixed at the carrier is used to hold all the wires and an additional battery pack to supply the display with power. The backpack can be adjusted according to the back dimensions of the visitor. By fastening the waist belt all the weight (approx. 5 kilograms) is resting on the hip instead of on the shoulders. Under some circumstance a belt-pack solution including a wearable worn on a belt can be used as alternative: In the case of monocular displays and rather simple 3D content onboard graphics adapters found in wearable computers are sufficient. In this respect the custom designed x4 Wearable developed by the University of Birmingham was successfully used.

3.2 Headset The headset comprises of either a binocular video see-through stereo display (iglasses SVGA Pro 3D) or a monocular optical see-through display (Shimadzu Data Glass 2/A, Liteye LE 500 or LE 750, see Figure 2). In the first case the real world is captured by a webcam attached to the front and displayed within a non see-through display. In the latter

940

M. Wittkämper et al.

Fig. 2. Mobile AR system with wearable computer on baby carrier and monocular headset

case the real world is perceived directly. In both cases the webcam is required for capturing the real world view which is sent to the remote participants. For both applications it is crucial to adjust the one captured real world image stream to the two eye positions and to the field of view of the display. This is accomplished by projecting the image onto the inner surface of a sphere located in the background of the virtual scene. The position of the sphere differs for the left and right eye and can be adjusted so that a stereoscopic experience is created and the video augmentation matches with the peripheral real view outside the display. This is important as the field of view of the displays is rather limited (20° - 25 ° horizontally) and a wider field of view, even without virtual objects, is very helpful for walking around the science center without colliding with other visitors or exhibits. The video augmented image consisting of the video background and the virtual objects can be streamed to a client using the RTSP protocol. This streaming functionality was added to the 3D viewer by the authors using DirectShow and the VideoLAN streaming library. In addition to streaming the same view can also be recorded locally as video or single images. For the registration of real and virtual objects, a precise tracking of the visitor is essential. Facing the large space in that the visitor will walk around as well as the number of exhibits at the science centres, a marker-based optical tracking like ARToolKit [4] or ARTag [3] seems to be a cost-effective solution. However, these toolkits were not designed to run in an inside-out configuration, where the markers are fixed at well-know locations and the camera is mounted on the HMD. As a consequence the precision of the tracked rotation component is not sufficient. Instead a hybrid tracking system was integrated into the AR/VR Morgan[7] framework , where the rotation is replaced by a more precise inertial tracking device, the InterSense InertiaCube3. In this hybrid approach the two systems are combined as follows.

A Distributed System for Augmented Reality Experiences

941

The user position TUser and orientation RUser are RUser = RInertia

(1) -1

TUser = TFixedMarker * RInertia * TTrackedMarker

(2)

where RInertia is the tracking result of the inertial tracking, TFixedMarker is the known 3D position of the detected marker, and TTrackedMarker is the tracking result of the optical tracking. The headset has an inertial tracking sensor for tracking head rotation and a second webcam for tracking head position. This second webcam is used to detect black and white patterns (fiducial markers) printed on the paper and scattered around the exhibits. As the pattern positions are known to the system, it can determine the user’s head position. As a consequence the visitor is restricted to an area around or in front of the exhibit. As long as the tracking camera finds patterns, the position is tracked. Opposed to that, the orientation is always tracked. Therefore, provided the visitor remains at the same position, he can look in many directions even when the camera detects no markers. However, if he moves without seeing the marker he has to bring a marker into sight of the tracking camera to update the head position.

3.3 Interaction There are two ways to interact with the exhibit. Either real parts of the exhibits are used as Tangible Interfaces or a virtual menu is used to access a number of operations. To allow for interaction with parts of the exhibit the same tracking technology is used. In most cases either the position or the orientation are required. In addition to this, physical experiments conducted at the science centers often include data such as temperature, CO2 density or lighting condition, measured by a variety of sensors. To enable the visitor to control and monitor this data, a data acquisition system (e.g. Intelligent Instrumentation EDAS CE) using the industrial OPC Data Access standard . [8] was integrated into the AR system. This standard provides a specification for communicating real-time data from data acquisition devices hosting an OPC server to any OPC client. The OPC client developed within the AR/VR Morgan framework reads from and writes to the data channels. The head-up menu is fixed to left side of the view, following the visitor wherever he looks. It is a hierarchical menu consisting of two levels. The main menu contains up to 7 submenus or entries; each submenu contains up to 7 entries, each related to a specific action. The menu structure is common to all exhibits and its content adjusts to the current exhibit. Functionalities accessible through the menu include capturing data and video images, controlling the real world and switching on or off virtual content. The head-up menu is operated using the wheel and buttons of a wireless mouse. It has to be held so that the wheel and buttons can be operated with the thumb. (A cursor controlled by the mouse is not required.) The menu is activated, i.e. made visible by pressing the wheel button. By turning the wheel, the visitor can navigate through the list; the current entry is highlighted by a white frame. An entry is selected by pressing the left button. When selecting a submenu it replaces the main menu. After selecting a (sub-) menu entry its highlighting is reinforced as long as the button is pressed. After selecting an entry that is not a sub-menu, the menu is deactivated. When activated again the same (sub-) menu becomes visible. To navigate from a submenu into the main menu the right button has to be pressed. In the main menu this deactivates the menu. Pressing the wheel button deactivates the menu at the current level.

942

M. Wittkämper et al.

To conclude, the left button is used to descend into the second level of the menu hierarchy. The right button is used to move up again to the main menu. And the wheel button is used to stay at the current level. The fact that activating the menu leads to the same (sub-) menu that was active at last, allows fast access to the same group of functions. Capturing a series of screenshots is possible without having to navigate through the main menu into the specific capture menu again and again.

4 Scenario Specification with MRIML To describe and specify the content and interaction for the augmented part of an exhibit a user interface description language was developed. Instead of creating an application specific interface for the use of AR system in education, the aim was to specify a general platform independent language for both Augmented/Mixed Reality user interfaces as well as WIMP based user interfaces. With the help of MRIML (Mixed Reality Interface Markup Language) the learning scenario can be specified outside the system in which it will be used (rendered) [3]. Opposed to the usual WIMP based user interface description languages MRIML fully supports a third dimension. UI elements are arranged in 3D space and can be of 2D or 3D nature. Also input and output elements are device independent. For example, a label can either be shown or read to the user depending on whether a display or earphones are available as output device. As all other UI languages MRIML consists of container elements (e.g. Frame, Menu) and control elements (e.g. Label, Artifact, Button). At first MRIML was specified as a vocabulary to be used in conjunction with the user interface meta language UIML (User Interface Markup Language) [18]. This meta language provides a rich infrastructure common to all user interfaces such as the behavior of UI parts including connections to the front- and backend. As a drawback of UIML we identified that although UIML is an XML language, the vocabulary does not benefit from it. For example validation is only performed on the UIML structure, as the vocabulary is found in values only and has no own XML tags. Thus, to fully profit from the many features of XML a separate XML schema was defined for MRIML and the original MRIML vocabulary.

0.1 1.0 -1.5 1.2 0.9 0.02 blue

flyingPlane.avi

Airflow.wrl

0.0 0.2 -0.5 Lift yellow

...

Fig. 3. MRIML fragment of Airfoil elements shown in Figure 1

A Distributed System for Augmented Reality Experiences

943

Figure 3 shows an MRIML fragment representing the virtual elements of the Airfoil scenario presented in Figure 1. An XSLT/XPath-based renderer was developed to translate the MRIML user interface elements into VRML’97 3D objects and Behaviors [3]. While the text labels directly correspond to MRIML elements, the airflow animation and the background video use artifact elements. The geometry of 3D objects is typically described in VRML’97, the low-level 3D description language.

5 Augmented Video Streaming Two types of augmented streaming scenarios are supported. In the first scenario, a local projection screen or display is near to the exhibit. It enables other visitors than the one wearing the mobile AR system to share his augmented view. Being at the same location and able to directly interact with him, the degree of participation is very high. With only the local area network involved the available bandwidth is comparably high and the video quality is primarily dependent upon the selected video camera. No relays are required. Delays below about one second proved to be acceptable to follow her activities.

Fig. 4. Augmented video streaming components for stream creation and distribution including data and user control flow

In the second augmented streaming scenario, observers can follow the visitor’s or a museum staff member’s activities through a live video stream (see Figure 4). They can also communicate with him through bidirectional audio connections. Here the degree of participation is reduced; remote users can either instruct the AR user to simply navigate to a specific location and look at a certain item, or suggest how to perform an experiment. Nevertheless, being able to participate was well received by users. In our implementation in the AR/VR Morgan [7] framework, augmented live video streams are created in the AR system’s visualization component and distributed to various clients through an IP network (see Figure 5).

944

M. Wittkämper et al.

The AR scene, optionally controlled by a head tracking system and an external application, is rendered and combined with a video background. To eliminate camera distortion and to adjust the camera’s field of view and pose to that of an AR display, this background is created by mapping the camera image to the inner surface of a sphere. Care has to be taken so that the camera’s field of view is both large enough to cover the whole background and small enough so that the remaining pixel resolution on the background is sufficient. At present any off-the-shelf webcam does not exceed an effective pixel resolution comparable with QVGA; this results in a rather poor quality for the real image portion. In case the camera is part of a video see-through (VST) display (e.g. a VSTHMD) the frame can be shown in the display. For optical see-through (OST) displays a second render pass without the video background is required. The combined frame is then compressed and transmitted over the network.

Fig. 5. Augmented video streaming components for stream creation and distribution including data and user control flow

RTSP/RTP (real-time streaming protocol) [11] was chosen as stream transport protocol since it is sufficiently supported by a wide range of media players and streaming servers. The challenge in augmented live video streaming is to encode and transmit each frame without limiting the real-time behavior of the system. Since the bottleneck is the frame compression the choice of the right encoder is of vital importance. We prefer MPEG4 encoders, which ensure an acceptable CPU usage, frame rate and quality as well as data size, but other encoders are supported as well. By realizing the streaming server as DirectShow™ filter we gained a high flexibility. The complete process of video streaming can be realized in a DirectShow™ graph, making it easy to test and exchange different MPEG4 encoders, e.g. 3vix and DivX. Additionally it allows forwarding the encoded frames to other filters, e.g. for recording the stream to a file. In case several clients are connected simultaneously or they are located in different networks with differing bandwidths (e.g. intra-and internet) streaming relays are employed to serve the individual clients, reduce size and frame rate, as well as support different protocols such as HTTP. Application specific augmented live video streaming user interfaces are implemented outside the framework, which offer e.g. pan/tilt/zoom camera control or choice between several video streams.

A Distributed System for Augmented Reality Experiences

945

6 Discussion and Future Work In this paper we discussed the implementation of an AR based museum experience. The system has been installed at four science centers and at numerous other exhibitions or events. It was used with six different exhibits by about 500 hundred visitors in various exhibitions over the last three years. Technical evaluation results about the wearability of mobile AR systems have been published in [6]. More technical and the full pedagogical evaluation of 16 test runs (4 runs at 4 sites) is currently undertaken by the consortium of the CONNECT project (see Acknowledgements) and due to be published in the near future. Our experiences as developers of the system can be summarized as follows. In general, the mobile AR systems performed well and provided the required degree of mobility. The main disadvantage was that at some areas of the exhibits the inertial tracking system was influenced by distortions of the magnetic fields caused by metal parts of surrounding exhibits or the building. Another disadvantage was that it required the installation of optical markers on each exhibit. It is also worth noting that the backpack did not disturb the users. In contrast the different headsets were not so comfortable to wear due to their weight or the fact that the weight was not well balanced (i-glasses display). The advent of powerful lightweight ultra mobile PC’s (UMPC) marks a change from the backpacked system and enhances wearability. The fact, that in a monocular display the virtual augmentation is only visible for one eye, made the system annoying and difficult to use for unexperienced users. Therefore neutral density filters were applied to the noncovered eye. For the future we plan to combine two monocular optical see-through displays to a stereoscopic display. Concerning the Augmented streams we observed that the quality of the video (including the physical environment and the virtual objects) plays an important role for the user acceptance. If either the physical environment or the overlaid virtual objects are shown in low-resolution or with a small update rate, users will not be willing to watch the augmented video stream for a long time. While the overall resolution and frame rate were ok, the low effective resolution of the webcams resulted in grainy images of the real exhibit parts. Replacing the webcams by board cameras with configurable lenses will solve this problem. As an alternative to streaming a user’s view, views from a fixed position can be used instead to avoid shaky pictures. MRIML and VRML’97 performed well as specification languages. In order to realize the arrows, the MRIML specification had to be extended by parameterizable visual labels. This provided the starting point for an ongoing discussion on a more general support of complex 3D user interface elements, requiring language elements beyond artifacts. MRIML was also used as interchange format between the AR system and a scenario configuration tool developed by the CONNECT project partner Intrasoft. To allow even scenario designers with non-technical skills to create exhibits, a suitable authoring interface is required.

Acknowledgements Parts of the work described in this paper were performed within the project CONNECT, partially funded by the European Commission (FP6-2002-IST-1507844). We thank the project partners for their invaluable feedback.

946

M. Wittkämper et al.

References 1. Azuma: A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments 6, 4 (1997), 355-385 2. Broll, Lindt, Ohlenburg, Linder: A Framework for Realizing Multi-Modal VR and AR User Interfaces, UAHCI 2005 3. Fiala; ARTag, a fiducial marker system using digital techniques, CVPR 2004 4. Kato, Billinghurst, Asano Tachibana: An Augmented Reality System and its Calibration based on Marker Tracking, Transactions of the Virtual Reality Society of Japan, 1999 5. Kaufmann, Schmalstieg: Mathematics and Geometry Education with collaborative Augmented Reality, Computers & Graphics, 27(3), 2003 6. Knight, Williams, Arvanitis, Baber, Wichmann, Wittkaemper, Herbst, Sotiriou: Wearability Assessment of a Mobile Augmented Reality System, VSMM 2005 7. Ohlenburg, Herbst, Lindt, Fröhlich, Broll: The MORGAN Framework: Enabling Dynamic Multi–User AR and VR Projects, VRST2004 8. OPC foundation: http://www.opcfoundation.org/ 9. Oppermann, Specht: A Nomadic Information System for Adaptive Exhibition Guidance, ICHIM, 1999 10. Perkins, C. RTP Audio and Video for the Internet. Sams, 2002 11. Real Time Streaming Protocol: http://www.rtsp.org/ 12. Shelton: Augmented Reality and Education, Current Projects and the Potential for Classroom Learning, New Horizons for Learning, 9(1), 2002 13. Shelton, Hedley: Using Augmented Reality for teaching the Earth-Sun realationship to undergraduate geography students, The first IEEE Interantional Augmented Reality Toolkit Workshop, 2002 14. Sparacino: The Museum Wearable: real-time sensor-driven understanding of visitors’ interests for personalized visually-augmented museum experiences, Museums and the Web 2002 15. Sparacino, Larson, MacNeil, Davenport, Pentland: Technologies and methods for interactive axhibit design: from wireless object and body tracking to wearable computers, ICHIM, 1999 16. Spasojevic, Kindberg: A Study of an Augmented Museum Experience, Hewlett Packard internal technical report, 2001 17. Starner, Mann, Rhodes, Levine: Augmented Reality through Wearable Computing, Special Issues on AR, 1997 18. UIML (User Interface Modelling Language): http://www.uiml.org/ 19. Wagner, Schmalstieg, Billinghurst: Handheld AR for Collaborative Edutainment, ICAT 2006 20. Woods, Billinghurst, Aldridge, Garrie: Augmenting the Science Centre and Museum Experience, SESSION: Virtual and augmented reality, 2004

Federate Job Mapping Strategy in Grid-Based Virtual Wargame Collaborative Environment Jing Zhang1 and Hai Huang2 1

College of Resources Environment and Tourism The Capital Normal University, Beijing 100037, P.R. China [email protected] 2 School of Computer Science and Engineering Beihang University, Beijing 100083, P.R. China [email protected]

Abstract. A crucial issue for grid-based virtual wargame collaborative environment is that of mapping federate jobs to grid nodes efficiently. Because there exits large amount of interactions among federates, it is a challenging work to design an efficient federate mapping strategy. In this paper, we established an objective function for federate job mapping which takes into consideration not only the load of grid nodes, but also network utilization. Experimental results show that the proposed federate job mapping strategy can contribute to load balancing of the virtual wargame collaborative environment.

1

Introduction

Virtual Wargame Collaborative Environment is able to support providing vivid virtual battlefield environment for soldiers by establishing vivid 3D terrains, landforms, surface culture, natural weather conditions, and various special effects through information access and graphic processing technology; is able to support behavior modeling by intelligent agents to simulate the tactical combats between the two sides under a variety of circumstances, evolve the everchanging battlefield situation, and meet the purpose of training the battling command abilities of tactical commanders and fighting skills of combatants. Various complex natural environments and tactical backgrounds can be set up in the virtual environment so as to reduce training costs substantially and guarantee day-and-night all-weather training. After going through three stages of SIMulator NETwork (SIMNET), DIS Protocol, and Aggregate Level Simulation Protocol (ALSP Protocol), it has evolved into High Level Architecture (HLA). HLA has been followed universally as a de-facto standard, and was adopted as the IEEE standard 1516 in September 2000. HLA defines a general technological framework, aiming to facilitate reusability of simulation components and interoperability among simulation applications. In HLA, the simulation applications and underlying supporting software organized to reach specific simulation goal are called federation. Simulation applications in a federation are called federates and they communicate with each other via Run Time Infrastructure (RTI), as K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 947–956, 2007. c Springer-Verlag Berlin Heidelberg 2007 

948

J. Zhang and H. Huang

Federate

Federate

...

Federate

Federate

RTI

Fig. 1. HLA Federation

illustrated as Fig. 1. Based on HLA, large-scale virtual wargame collaborative environment can be developed over the Internet. With the wide adoption of HLA standard, some limitations emerge. For example, there exist problems of different degree in such aspects as scalability, convenience, robustness, and component reusability. Over the past several years, the concept of Grid, proposed by Ian Foster as secure and coordinate resource sharing and problem solving in dynamic, multi-institutional virtual organizations, has become a hot topic. Some research institutions attempt to adopt grid technology in HLA simulation to solve some problems of current HLA simulation [1], [2], [3], [4], [5]. Their studies have focused on various aspects of simulation grid, such as simulation grid architecture and overall technology, dynamic comprehensive management technology of simulation grid service resources, gridenabled (service-enabled) technology of simulation resources, simulation grid collaborative modeling/simulation interoperability, simulation grid QoS and realtime implementation technology, simulation grid operation monitoring and optimal scheduling technology, simulation grid application development and implementation technology, simulation grid visualization service implementation technology, and simulation grid security support technology. In previous work, we have established the Aegis prototype [6], which validated the beneficial impact that grid brought to HLA simulation. The ultimate goal of Aegis is to establish a Simulation On Demand (SOD) environment, allowing users to customize or set up a federation to carry on collaborative simulation through a browser/client at any time and place (as Fig. 2 shows). The simulation developers upload simulation resources such as terrain, models, texture and federates onto some storage nodes in SOD, and then expose them as grid services. Users can access the grid information service and obtain a list of available simulation nodes and resources in SOD solely through a browser/client, and allocate federates to appropriate nodes to set up a simulation federation according to loads of nodes, in which federates communicate via RTI. In above process, the simulation resources reside on some nodes will be transferred to other nodes through GridFTP service if necessary. A lot of computing and data resources may be involved in large-scale gridbased collaborative simulation in SOD environment and a simulation run may last a long time. The load of nodes may vary significantly due to the uncertainty and unpredictability of federates at different nodes, and at the same time the availability of node resources is not guaranteed because of human factor or malfunction. Therefore, it is necessary to balance the load among distributed nodes so as to improve utilization of resources and make simulation advance as normal when a certain node is overloaded or unavailable.

Federate Job Mapping Strategy

949

Fig. 2. SOD Allowing Simulation Anytime and Anywhere

The most widely used methods of balancing load is selecting an appropriate destination node for a newly-joined federate to run on, which is also called job mapping, or migrating a federate at overloaded node to a lightly-loaded node. We have implemented federate migration in grid-based virtual wargame collaborative environment in our previous work [7]. In this paper, we aim to developing federate job mapping strategy for grid-based virtual wargame collaborative environment. This paper is organized as follows. In Sect. 2, related work is reviewed. In Sect.3, we introduce relations between federates and resources in HLA federation and grid-based federation. In Sect.4, we describe the federate mapping strategy in grid environment. The experimental results are presented in Sect.5. Finally, we conclude the paper in Sect.6.

2

Related Work

Many traditional distributed algorithms and Grid scheduling algorithm have some features in common, that are performed in multiple steps to solve the problem of matching application needs with resource availability while providing quality of service [8]. Solving the matching problem to find the choice of the best pairs of jobs and resources is an NP-complete problem. Many heuristics have been proposed to obtain the optimal match. In [10], a dynamic matching and scheduling scheme based on a distributed policy for mapping tasks onto HC systems is provided. A task can have several subtasks, and the subtasks can have data dependencies among them. In the scheme presented in [9], the subtasks in an application receive information about the subtasks in other applications only in terms of load estimates on the machines. Each application uses an algorithm that uses a weighting factor to determine the mapping for the subtasks. The weighting factor for a subtask is derived by considering the length of the critical path from the subtask to the end of the directed acyclic graph (DAG) that represents the application. Various resource management approaches for distributed simulation have been investigated in the research community. In [10], the resource sharing decision is

950

J. Zhang and H. Huang

made by the end user of the computer. If the user has work for the computer, she/he may opt not to participate in the resource sharing system. Otherwise, the user indicates the willingness to share the resource to a manager. The manager will ”tell” a communication federate that new resource is available. Based on the outcome of the load balancing algorithm, one or more federates are selected by the manager and migrated to the appropriate destination host, and the simulation execution proceeds after migration succeeded. When the user decides that the resource should be no longer available to the simulation, the manager performs the load balancing algorithm again to move the running federate(s) to other nodes. All the resource availability information is managed by the manager and relayed to the communication federate. All communication between the communication federate and the simulation federates is done using the RTI interactions, and the communication federate can be viewed as part of the HLA federation. An alternative approach was proposed in [11], where a federate is encapsulated in a job object. Each job, implemented with multi-threading architecture to increase concurrency, consists of two interfaces, one interface to the RTI and the other to the Load Management System (LMS). The LMS incorporates two subsystems: job management subsystem and resource management subsystem. The job management subsystem monitors the execution of federates and performs load balancing activities if necessary. The resource management subsystem, with help of the services provided by the Globus Toolkit, performs resource discovery and monitoring in the computing Grid. The major modules of Globus that are used in LMS include the Grid Security Infrastructure (GSI), the Grid Resource Allocation Manager (GRAM), and Grid Information Service (GIS). Czaj kowski et al. described the GIS service for distributed resource sharing in [12].

3

Relations Between Federates and Resources in HLA Federation and Grid-Based Federation

In the traditional HLA Federation, the federate is tightly coupled with the computing resources (see Fig. 3). When a simulation application is operated, which machine to run the task is randomly selected by the simulation administrator, or allocated to the specific machine according to the experience or knowledge of the administrator. This approach of matching between simulation tasks and computing resources is simple and easy with obvious drawbacks. Since it is not able to optimize the matching according to the characteristics of simulation tasks and the load conditions of computing resources, the load of computing resources is easily overbalanced, and the simulation advance as normal may even be affected in serious cases. In particular, a federation may involve multiple network management domains in WAN-based simulation application where it is a major QoS indicator to reduce the communication delay between federates. Therefore, the relationship between publishing/subscribing federates should be focused on in the matching of simulation tasks and computing resources.

Federate Job Mapping Strategy

951

Fig. 3. Relations Between Federates and Resources in HLA Federation

Fig. 4. Relations Between Federates and Resources in Grid-Based Federation

In grid-based federation, federate job mapping strategy is implemented as part of the grid resource management (see Fig. 4) system. Grid resource management provides a global view of grid resources through Index Service, while the job mapping module uses Index Service to map to map federate jobs to the appropriate resources so as to realize optimization of resource utilization.

4

Federate Job Mapping Strategy

In nature, job mapping need to turn the matching between tasks and resources into the objective function; and the mapping process to maximize the objective function. Since the nodes are geographically distributed in wide area network in SOD, and the communication delay among each other bears great impact on the duration of a federation run; on the other hand, network bandwidth is an important resource in wide area network, therefore we should minimize network resource cost when performing a federation run. Consequently, the CPU load, memory storage, and communications among federates will be considered in this paper when establishing the federate job mapping objective function. The node load can be computed by measuring the CPU load, memory usage, and hardware system information about the nodes, like clock frequency, processor type, amount of memory etc. The network load can be described with the occupancy of network

952

J. Zhang and H. Huang

resources, that is, be determined by both the communication traffic and router hops. Suppose there are m nodes in SOD environment (see Fig. 5), the CPU speed and memory capacity of node Ni are Ci and Mi respectively, and the corresponding CPU and memory usage during simulation execution are ci and mi respectively, where i ∈ N , 1 ≤ i ≤ m. Also, suppose the federation comprises of n federates, and let sq,p be the number of object attributes of federate fq that federate fp subscribes, hp,q be the number of hops of the shortest path routing between nodes on which federate fp and federate fq run respectively, where p, q ∈ N , 1 ≤ p, q ≤ n. Group F is defined to be composed of all the n federates, while Group G is composed of all the m nodes.

LAN LAN

Node LAN

Router

Fig. 5. Network Topology of a Grid-Based Federation

The federate job mapping strategy can be described as follows: Step 1: Select any element f ∈ F from group F, and select an element g from group G so that f is able to run on g while not overload g. If g exists, then f is distributed to run on g, and group F is updated to F-f at the same time. Step 2: This step is an iterative process, as we have proposed in [7]. First, select any element from group F again. Without loss of generality, suppose the newly selected element is fu , where u is an integer from 1 to n. The running node Nj , where j is an integer from 1 to m, is then searched for through the following approach: ¯ j are the thresholds of CPU and memory of Nj . Then, the Let C¯j and M maximum available CPU and memory capacity on Nj can be expressed as (C¯j − ¯ j − mj /Mj )Mj , respectively. cj /Cj )Cj and (M The network can be approximately comn resource cost when n fu runs on Nj  n puted by v=1,u=v hv,u sv,u + v=1,v=u hu,v su,v = v=1,u=v hu,v (sv,u + su,v ). The preference value for fu selecting Nj as the node to run on is defined as: ¯ j − mj /Mj )Mj P (fu , Nj ) = w1 (C¯j − cj /Cj )Cj + w2 (M n  +w3 /( hu,v (sv,u + su,v ) + ε) , v=1,u=v

Federate Job Mapping Strategy

953

where wk (k = 1, 2, 3) is the weight that CPU, memory and communication bears in the preference value, and ε is an infinitesimal positive number which is introduced to avoid dividing by zero. Now, the grid node for fu to run on should be Nk that can maxmize P (fu , Nk )}, where 1 ≤ k ≤ n, k = i. Step 3: If Nj is found for fu in step 2, and fu will not overload while running on Nj , then fu is distributed to Nj , and group F is updated to F−fu at the same time. Step 4: Implement step 2 and step 3 repeated to until group F become an empty group. And the whole federate job mapping process is completed. As can be seen from step 2, the grid node for fu to run on should be the one that bears the maximal available CPU and memory capacity and at the same time the network resources with the minimal costs. In other words, the implementation of step 1 to step 4 are favorable for realizing load balance of computing nodes in SOD system and for reduction of overall occupancy of network resources.

5

Experimental Results

The prototype system of a SOD simulation grid environment is established on basis of WSRF.NET and BH RTI 2.1 which is a distributed interactive simulation running platform developed by the Virtual Reality Laboratory of Beihang University. The simulation creation and task allocation processes are tested. And the testing results are compared with the random task allocation algorithm to verify the effectiveness of SOD system in load balancing. 5.1

Experimental Environment

In the experiment, four hosts are used to build a SOD simulation grid operating environment. Index Service is run on Node 1. Meanwhile, a monitoring program is run on every node to obtain the current status of the node and reflect it to Index Service. Before the test, the load assessment program should be run first under the condition that every node is light-loaded to obtain a benchmark time value as the reference for load assessment. The benchmark time value should also be written in the configuration file for access of the monitoring program run on the node. The programs of federates select helloWorld in union. This helloWorld program modifies DMSO helloWorld correspondingly to enable it to record data generated in the simulation process for collection by SOD system. 5.2

Experimental Results and Analysis

SOD Portal shows the information of various resources in SOD environment. First, the simulation resources required are selected on the basis of the information listed to form a federation. After the selection of RTI and federate programs,

954

J. Zhang and H. Huang

Table 1. Loads and variances before and after using random task scheduling algorithm

Load (%)

Status Load before task allocation (%) Load after task allocation (%)

Node 1 29 31

Node 2 9 20

35 30 25 20 15 10 5 0

Node 3 11 13

Node 4 18 30

Variance 8.12 7.44

Load before task allocation Load after task allocation

1

2

3

4

Node Fig. 6. Load changes with random task scheduling algorithm Table 2. Loads and variances before and after using federate job mapping strategy Status Load before task allocation (%) Load after task allocation (%)

Node 1 30 27

Node 2 12 21

Node 3 9 25

Node 4 15 16

Variance 8.08 4.23

SOD will enter the task allocation stage of simulation creation. The major consideration of this stage is job mapping strategy. Table 1, Table 2, Fig. 6 and Fig. 7 show different results generated by adopting different task allocation algorithms. Table 1 lists the experimental data obtained by adopting random task allocation algorithm. Table 2 lists the experimental data obtained by adopting federate job mapping strategy of SOD system. In this paper, the changes in variance of node load after task allocation are used to measure load balancing effects. Comparing Table 1 and Table 2, it can be seen that the differences between node loads are reduced significantly after the federate job mapping strategy is adopted for task allocation. Therefore, SOD system is able to balance loads in allocating simulation tasks.

6

Conclusions

As in other domains, the goal of mapping in simulation grid is usually optimizing some objective function. However, because the different characters between simulation grid and other grid-based applications, their objective functions are

Load (%)

Federate Job Mapping Strategy

35 30 25 20 15 10 5 0

955

Load before task allocation Load after task allocation

1

2

3

4

Node Fig. 7. Load changes with federate job mapping strategy

also different. Usually, the objective function of most grid-based applications are to minimize the overall completion time of the tasks or a more complex function of multiple requirements(such as Min-Min). In simulation grid, the interaction between simulation applications is the key character. How to design the mapping strategy that considers the interaction among applications in simulation grid is important for simulation grid to achieve success. In this paper, we proposed a federate job mapping strategy which can contribute to load balancing of the virtual wargame collaborative environment.

Acknowledgements This paper is supported by the National Natural Science Foundation of China under the grant No. 40571126.

References 1. K. Zajac, A. Tirado-Ramos, Z. Zhao, P. Sloot and M. Bubak: Grid Services for HLA-based Distributed Simulation Frameworks. Grid Computing: First European Across Grids Conference, LNCS Vol. 2970, pp. 147-154, 2004. 2. K. Zajac, M. Bubak, M. Malawski, and P. Sloot: Support for Effective and Fault Tolerant Execution of HLA-Based Applications in the OGSA Framework. Computational Science - ICCS 2004, LNCS Vol. 3038, pp. 848-855, 2004. 3. Y. Xie, Y.M. Teo, W. Cai, and S.J. Turner: Extending HLA’s Interoperability and Reusability to the Grid. 19th ACM/IEEE/SCS Workshop on Principles of Advanced and Distributed Simulation (PADS 2005) Monterey, CA, USA, June 2005. 4. J.M. Pullen, R. Brunton, D. Drake, M. Hieb, K.L. Morse, A. Tolk, and D. Brutzman: Using Web Services to Integrate Heterogeneous Simulations in a Grid Environment. 2004 ICCS Workshop on HLA-based Distributed Simulation on the Grid, LNCS Vol. 3038, pp. 835-847, 2004.

956

J. Zhang and H. Huang

5. R.P.Z. Brunton, K.L. Morse, D.L. Drake, B. Moller and M. Karlsson: Design Principles for a Grid-Based HLA Federation. Proceedings of the IEEE 2004 European Simulation Interoperability Workshop, Paper 04E-SIW-056, 2004. 6. Wu W, Zhou Z, Wang SF, and Zhao QP: Aegis: a Simulation Grid Oriented to Large-scale Distributed Simulation. The Third International Conference on Grid and Cooperative Computing (GCC 2004), LNCS Vol. 3251, pp. 413-422, 2004. 7. Huang H, Wu W, Tang X, Zhou Z: Federate Migration in Grid-Based Virtual Wargame Collaborative Environment. International Conference on E-learning and Games (Edutainment 2006), Springer LNCS vol. 3942, pp. 606-615, 2006. 8. HE XiaoShan, SUN XianHe, Gregor von Laszewski: QoS Guided Min-Min Heuristic for Grid Task Scheduling. Journal of Computer Science and Technology, Vol.18, No.4, pp.442-451, 2003 9. M. A. Iverson and F. Ozguner. Dynamic, competitive scheduling of multiple DAGs in a distributed heterogeneous environment. 7th IEEE Heterogeneous Computing Workshop (HCW’ 98), Mar. 1998, pp. 70-78. 10. Johannes Luuthi and Stenen Groumann: The Resource Sharing System: Dynamic Federate Mapping for HLA-based Distributed Simulation. In Proceedings of Parallel and Distributed Simulation, pages 91-98. IEEE, 2001. 11. Wentong Cai, Stephen J. Turner, and Hanfeng Zhao: A Load Management System for Running HLA-based Distributed Simulations over the Grid. In Proceedings of the IEEE International Symposium on Distributed Simulation and Real Time Applications, 2002. 12. K. Czajkowski, S. Fitzgerald, I. Foster, and C. Kesselman: Grid Information Services for Distributed Resource Sharing, 2001.

A Platform for Motivating Collaborative Learning Using Participatory Simulation Applications 1

2

1

Gustavo Zurita , Nelson Baloian , Felipe Baytelman , and Antonio Farias

1

1

Information Systems Department - Business School, Universidad de Chile 2

Computer Science Departments – Engineering School, Universidad de Chile [email protected],[email protected], [email protected], [email protected]

Abstract. Several research efforts suggest that collaborative participatory simulations improve teaching and learning, increasing motivation inside the classroom. Currently, it has been mainly applied with students of primary and secondary educational levels, leaving higher level students aside. This paper presents a platform for implementing participatory simulations, where social interactions and motivational effects are the main facilitators. An instance of this platform was implemented for medicine school undergraduate students. Its implementation is simple, lightweight, fully based on pen-based interaction, and designed to work with handhelds and tablet-PC over an ad-hoc wireless network. The platform is able to support any kind of application implementing participatory simulation based on the exchange of artifacts among the learners.

Keywords: Techniques, methods and tools for CSCW in design. Other Keywords: Handhelds. Collaborative Learning. Participatory Simulation. Learning Motivation. Gestures. Sketches. Freehand-input based.

1 Introduction Some research groups have implemented collaborative learning participatory simulations with handhelds, tablet-PCs and infrared beaming [15], and it has been found that this kind of activities provide various advantages for teaching and learning: a) they introduce an effective instructional tool and have the potential to impact student learning positively across different curricular topics and instructional activities [18], (b) they increase motivation [8], [2], (c) they improve engagement, self-directed learning and problem-solving [8], and (d) they may stem from students having “rich conceptual resources for reasoning abut and thoughtfully acting in playful and motivational spaces, and thus can more easily become highly engaged in the subject matter” [7]. Collaborative learning trough participatory simulations use the availability of mobile computing devices having the capability of simple data exchanges among neighboring devices [19], [5]. They enable students to act as agents in simulations in which overall patterns emerge from local decisions and information exchanges. Such K.-c. Hui et al. (Eds.): Edutainment 2007, LNCS 4469, pp. 957–970, 2007. © Springer-Verlag Berlin Heidelberg 2007

958

G. Zurita et al.

simulations allow students to model and learn about several types of phenomena [3], in order to improve their knowledge about human behaviors, to help them in solving conflicts, to shape interaction protocols between humans, and to learn some aspects of collective management, games and experimental economics situations. Due to the nature of participatory simulations, it can be used to support the learning and teaching of any kind of systems for which its general behavior emerges from the interaction of its agents. As stated in [13], new pedagogical practices for the whole classroom are required in order to take the most advantage of new Technologies: it not enough to provide each classroom with handhelds or tablet-PCs but the way how technology meets education in a synergic way should be reinvented [14]. Computers have been already used since long time to support learning, but traditionally, most efforts have been directed to support individual learning [1]. This approach, tough very effective for some scenarios, fails to take advantage of what a collaborative learning can contribute to the whole learning process. On the other hand, the growing acceptance of handhelds and tablet-PCs enable users to take advantage of numerous advantages in scenarios that desktop computing cannot provide [8]. Handheld’s and tablet-PC most natural data-entry mode is the stylus (pen-based or freehand-input-based system), which imitates the mental model of using pen and paper, thereby enabling users to easily rough out their ideas and/or activating different functionalities like copy, move, delete, etc.[10]. However, most currently available handheld applications adopt the PC application approach that uses widgets instead of freehand-input-based paradigms (via touch screens) and/or sketching, [4]. This paper, presents a collaborative learning experience based on participatory simulations, having two general research goals: (a) to propose a conceptual platform for specifying participatory simulations using handhelds and tablet-PCs, (b) implementation of this platform to be managed by the teacher for developing applications to the whole classroom of participative simulations supporting motivating learning objectives, and (c) to determine the feasibility of using this in undergraduate curricular contexts of the simulation activities both in terms of intended and actualized learning outcomes; particularly in the medicine area. An instance of the platform is described. Its implementation is simple, lightweight and fully based on wirelessly interconnected handhelds with an ad-hoc network.

2 Handheld and Tablet-PC in Whole-Classroom Participatory Simulations Handheld and tablet-PC computers provide advantages, which make this an especially attractive platform for developing participatory simulations for a whole class activity [12]. In order to fully take advantage of this hardware, we should harness common features of handhelds and tablet-PCS including: (a) portability – students can move around within a classroom with their computing devices, (b) social interactivity – they can exchange data and collaborate with other people face-to-face, (c) context sensitivity– mobile devices can gather data unique to the current location, environment, and time, including both real and simulated data, (d) connectivity – it is possible to connect handhelds and tablet-PCs to data collection devices, other

A Platform for Motivating Collaborative Learning

959

handhelds and tablet-PCs (ad-hoc network) that creates a true shared environment, (e) individuality – mobile devices can provide unique scaffolding that is customized to the individual’s path of investigation. Our research project is to develop and examine a new participatory simulation platform that is designed from the ground up for handheld computers (used by students) and tablet-PC (used by the teacher) and draws on the unique affordances of these mobile technologies. Implicit to our research is the belief that a powerful handheld learning environment might capitalize on the portability, social interactivity, context sensitivity, connectivity, and individuality of ubiquitous devices to bridge real and virtual worlds. The handhelds and tablet-PC then provides a window into the virtual context that is sensitive to information being supplied to it by the real world. A learning participatory simulation is a role-playing activity that helps to explain the coherence of complex and dynamic systems. The system maps a problem of the real world to a model with a fixed number of roles and rules. Global knowledge and patterns emerge in participatory simulations from local interactions among users and making decisions to understand the impact by an analysis and observation while doing and/or at the end of the activity. Foundational concepts underpinning the design and use of collaborative learning participatory simulations include (a) immersion of students in simulations of complex phenomena; (b) development of inquiry and research design skills by the reflections and analysis of the simulation; (c) negotiation, coordination and alignment of individual local behavior in order to foster group-level systems understanding. An advantage of learning participatory simulations is the fact that the activity is highly effective and motivating even in large groups (a whole class). A teacher could starts or not with an introduction where he or she explains the relevant theoretical parts of the complex problem. At least the roles and rules need to be clearly specified. The students should understand the possible activities and consequences, which are available in the simulation at the end. The exchange of experiences and a discussion in small groups within or after the simulation help to increase the understanding of the simulated reality. A whole class activity of participatory simulations can be integrated in the lecture [12], where a major idea is the concept of learning through doing [9]. Students participate in an active and motivating way, analyze information, interchange information among them, make decisions and see the outcome of their actions. 2.1 Principles of the Handheld and Tablet-PC Interface According to [10], [4] a handheld (and tablet-PC) application interface must imitate the pen-and-paper metaphor so users can interact naturally with the computer in varied situations, especially when they are or need to be in movement, thus freeing them to concentrate on the tasks at hand instead of worrying about the interface (interaction with keys, menus, widgets, etc.). A pen-based system offers a more natural and intuitive interface enabling the sharing and exchange of design information so as to improve efficiency. Essential to the functioning of pen-based user interface is the use of gestures [11]. In [5], a survey intended to illuminate the problems and benefits users experience with gestures, it was found that the most frequent actions were deleting, selecting and

960

G. Zurita et al.

moving, and that users consider these actions to be efficient as a form of interaction, as well as convenient, easy to learn, utilize and remember, and potentially an added advantage for the interface. Finally, handhelds are an appropriate technology for providing high mobility and portability in physical spaces like a classroom, and for creating ad-hoc networks through peer-to-peer connections between already incorporated WiFi components (Dell Axim X50) jointly with a tablet-PC. Such network allows deliberate information exchange between users, as well as to automatically interaction between devices (sees section 5). Proximity detection is done with infrared sensors (IrDA) combined with WiFi.

3 Developing a Platform Collaborative learning trough participatory simulations with handhelds offer an additional perspective providing off-screen, first person experience and insight into the dynamics and emergent behavior, as students become agents in a complex system. Information and conceptual knowledge circulates through peer-to-peer interaction by the interchange and negotiations of objects, which takes different forms in each simulation. We propose a platform for the specification, design and creation of mobile learning participatory simulations based on handhelds wirelessly interconnected. In order to generate, specify and implement applications of collaborative learning participatory simulation, the teacher defines on the tablet-PC): (a) learning goals of the reality simulated, (b) artifacts to be interchanged, b) behavior variables and parameters, and (c) rules and roles for playing the simulation (see section 4). Goals need to be identified in order to let students successfully learn/construct certain knowledge. These goals may include meta-objectives and they are only a mean to achieve an overall learning objective. Roles and rules are spread among students when the activity begins, but the teacher can also interact too. In order to configure the system for a collaborative learning participatory simulation, the Teacher may setup transferable objects (artifacts), their behavior parameters, rules and participant roles using his/her tablet-PC. Then, to begin the activity, the professor explains the goal of the activity to the students, also describing objects, rules and roles, and how these concepts are represented in their handhelds. Rules, roles and goals should be designed to achieve (a) a high social interaction between students, (b) negotiation instances, and (c) competition to encourage an active and motivated stance as if students where playing a game [9]. If students require assistance, our platform allows the teacher to wirelessly give them feedback and assessment. The teacher can (a) observe the simulation state of each participant device and (b) modify such state in order to solve the student inquiry. Start-up setup must ensure students play always an active and dynamic role through time. This should be based on trading between students (Negotiation component of Students, and Interchange Objects), or automatically among handhelds (Exchange Objects). These conditions depend on each learning participatory simulation application build, and may involve the following aspects: (a) type of exchange objects, (b) exchange amounts, (c) trade conditions, (d) parameters before and after the exchange, and (e) exchange objects.

A Platform for Motivating Collaborative Learning

961

If students require assistance, the platform allows the teacher to wirelessly give them feedback and assessment. The teachers though the tablet-PC can (a) observe the simulation state of each participant handheld device and (b) modify such state in order to solve the student inquiry. Start-up setup must ensure students play always an active and dynamic role through time. This should be based on trading between students (Negotiation component of Students, and Interchange Objects), or automatically among handhelds (Exchange Objects). These conditions depend on each learning participatory simulation application build, and may involve the following aspects: (a) type of exchange objects, (b) exchange amounts, (c) exchange conditions, and (d) parameters before and after the exchange. Once the simulation is done, the teacher must guide students’ understanding about the activity. In this way, the students will construct the learning objective together, analyzing different stages of the activity.

4 A Scenario for Collaborative Learning Participatory Simulation In this section we describe an instance of the platform proposed in section 3. It is oriented to support undergraduate students of medicine schools to learn aspects related to the identification of illnesses trough their symptoms and the corresponding medication. Students normally reinforce their knowledge about these aspects with a lot of real practice accompanying physicians. They have to develop skills on associating a certain set symptoms to the correct disease, since one symptom may be present in many diseases. Also medication can be different even for the same disease according to the intensity of the symptoms. We propose a scenario where a whole medicine course simulates this process in order to determine the illness and the corresponding medication based on a set of symptoms distributed to the handhelds of the students. Initially, the teacher configures a set of diseases, symptoms and prescriptions delimiting the tolerance range and possibilities they can present. This information is distributed among the 20-30 medicine students of the whole class, so each student has at least one disease and various medical prescriptions and symptoms which may not all correspond to the illness. The students have to establish face-toface encounters in order to exchange symptoms and medical prescriptions they may have in case they want to do so. This exchange of information and the discussion between the students involved should reinforce or refute their knowledge about the corresponding disease in a highly participating and motivating atmosphere. Handhelds in their turn act proactively informing students who put their devices close together if they have symptoms or prescriptions which they can exchange (according to the configuration established by the teacher at the beginning). Handhelds will only inform of this opportunity. Students should identify and decide if they exchange these elements. The teacher can monitor the process of each student using the tablet-pc. The teacher can also change the association parameters between symptoms, diseases and medical prescription during the experience. A correct solution by a student establishing the right association between a disease, it symptoms and the right medication can be automatically determined comparing the student’s solution with the configuration given by the teacher. Small deviations are allowed. It is also possible

962

G. Zurita et al.

that cases unexpected by the teacher may arise during the experience. These can be assessed in real time by the teacher giving the corresponding feedback to the students.

5 An Application Using the Platform We have implemented a lightweight application for the creation of participatory simulation applications based on the platform proposed in section 3. Using this platform we have successfully implemented an application for the scenario proposed on previous section. This application allows the teacher to organize and administrate the simulation using a Tablet. The teacher can create "Medical Conditions", “Treatments” and “Symptoms”, and let students organize and link them finding out the optimum configuration. This will encourage different students to interact with each other in order to trade their items (treatments and symptoms) in order to describe in the best way his or her assigned medical conditions. 5.1 Simulation Management Items design. The teacher can define different items, which may represent medical conditions, treatments and symptoms. In order to do this, he or she has to work under the “Define activity" mode. Here, the teacher can draw a sketch, type or hand-write the name of certain item and surround it within a rectangular shape. This will produce an "item icon", displaying an awareness of successful creation and displaying a reduced version of the original scratch. Then, additional "goods icons" may be created, as seen in Figure 1.a. Selecting a created item will display contextual options for choosing item type: "Medical Conditions", “Treatments” and “Symptoms” (Figure 1.b). Clicking such options will change the item’s type. Items of each type are displayed with different colors. Items linking. Once items of different types have been created, the teacher can relate “symptoms” and “treatments” to certain “medical conditions”. This is done drawing a line that starts from one item and ends in another. When the stylus is released, an arrow symbolizing a link between both items will replace such sketched line. Internally, the system will store the relation between these two items. When students reproduce this link they will get appropriate bonus. Activity status and real time management. Anytime before or during the simulation, the teacher can enter the “Define activity” mode and create new items, or alter their previous relationship. For instance, he or she could introduce a new treatment for an old medical condition, encouraging the students to keep searching for alternatives to their current individual configuration in their handhelds. Another mode is the “Activity overview”, which presents metrics about the activity status, overall students’ assessment, how many students are performing correctly, how many need help, and how much are students interacting with each other. Detailed watching and teacher-student interaction. Entering the “Participants” mode will display all students’ current screens in a matrix, allowing the teacher to watch exactly what students are doing, which score they have, and who are interacting – and what are they exchanging. Figure 2 shows an example participant's screens summary. All students’ handhelds screens are reproduced scaled in order to fit them

A Platform for Motivating Collaborative Learning

a

963

b

Fig. 1. a) Teacher can create items by handwriting their names on the tablet-PC. Surrounding some sketches with a rectangle shape will define a new item. b) Selecting an item will show contextual options, which allow the teacher to choose selected item’s type.

in the teacher screen. Students' current action is also being displayed: in the image, first three students are linking their concepts, while Felipe and Pedro are exchanging items. The teacher can double click a student's screen in order to access a full scale mode where he or she can interact with the participant's configuration, both altering and assessing the student's actions.

Fig. 2. On “Participants” the teacher has a quick summary of every student's screen. In this example, some students are drawing links while others (Felipe and Pedro) are exchanging items in order to achieve better individual configurations. Exchanged items appear bigger, highlighting them.

964

G. Zurita et al.

5.2 Simulation Description When the activity teacher has setup the simulation, each student has randomly assigned “medical conditions”, “treatments” and “symptoms”. Originally assigned items will not correspond to each other, so students will have to start seeking for adequate “treatments” and “symptoms” for their assigned “condition”. Once they exchange their items for the corresponding ones, they'll have to establish links between them.

Fig. 3. Students screen displays items like the teacher's screen. Drawing a link between two items will generate a link. Repeating such action will remove a previously generated link.

Sudents' screen and items linking. Assigned and exchanged items appear in the student's handheld screen, just as they would look in the teacher's tablet-PC, showing customs colors depending on the item's type. Students may establish different linking configurations trying to find the optimal relationship. In this process, each student may define and remove links between concepts, until the system highlights link as “optimal”. Student may define links in the same way the teacher does so, by drawing a line from one item to the other (Figure 3). Item exchange between students. When two students engage in a face-to-face encounter, automatically their handhelds display the exchange mode. In this screen, the lower area shows items currently owned by the student, while the upper area of the screen there are two negotiation spots: one where offered items may be dragged, and other where engaged student's offered items are shown. Dragging an item from the owned area into the offered area (Figure 4 ) will include such item into the exchange. On the other hand, when the trade is done and both students agree, dragging any of the items offered by the other switches current student's trade state to “accept”. When both accept, the exchange is done, transferring items between both students.

A Platform for Motivating Collaborative Learning

965

Fig. 4. Two students exchange items through a face-to-face encounter using their handhelds. Dragging an item from the owned area (“My items”) into the offer zone will put such item on trade.

6 Implementation The System was implemented over a platform we developed previously which supports the development of distributed applications for mobile devices over different platforms: Java and .NET. The middleware we developed consists of a set of classes implementing an API the programmer can use in order to write distributed applications easily. These classes are available in Java and C# and implement the necessary mechanisms for converting data objects from their internal representations into an XML representation, transmit them across platforms and convert the XML representation into the corresponding internal one. They also provide and start the necessary services for discovering partners present in the ad-hoc network and establish the necessary connections among the different applications in order to synchronize shared data. 6.1 Discovering Partners and Establishing the Connections Every application programmed wit our platform starts a service which will send multicast messages at regular intervals to the group in order to inform other participants of the presence of a new one. It will also start consuming multicast messages from other partners present in the ad-hoc network. This allows the application to maintain a list of active participants updated. Every time a multicast message of a new participant is received, its ID and IP number are stored in the list and a TCP/ IP connection is established with that application through which data will be shared. If a certain amount of time has passed without receiving a multicast message from a member of the list of active participants, its entry is deleted and the connection to that application closed. The Figure 5 shows the structure of the

966

G. Zurita et al.

communication node present in all applications developed with the platform. It has a module which manages threads for sending and receiving multicast packages used to maintain an active partners list. This list is used by another module which is responsible for creating TCP/IP connections with the active partners and destroying them for those partners which left the group and transmit synchronization data. 6.2 Sharing Objects The data sharing mechanism is based on a “shared objects” principle. A shared object is an abstract class which should be extended in order to create an object class whose state will be transmitted to all active participants when the object changes its state, this is when one or more variables change their value. The programmer implements a shared object by extending the SharedObject abstract class. The synchronization of the shared objects is achieved by transmitting a copy of it to all partners every time their state is changed. For this, methods for sending and receiving objects were designed and implemented. At the beginning these methods were private to the middleware, but very soon we discovered that many small yet powerful applications could be implemented very easily based on those methods without having to use the SharedObject class. Therefore we made these methods public to the API. In order to transmit an object across platforms we need a way to represent objects common to both platforms: this is XML. When the application must update the state of an object it is passed to a serializer which produces an XML representation. This description is then sent to the communication node of the other application. The receiving node uses its own de-serializer for transforming the XML representation in the corresponding internal one. TCP/IP connections to other applications for transmitting/receiving objects

TCP/IP connections manager

Communication Node

Multicast UDP traffic for discovering and discarding partners

Multicast discovery manager

Active partners list

Fig. 5. The structure of the communication node

6.3 Group Management The face-to-face edutainment scenario was the first situation that motivated us to for developing this middleware because of the need to have applications implemented and running in different platforms to share data. In this scenario, we also recognized the need to have the possibility of defining subgroups of partners inside the whole group of active participants. For example, the teacher may want to propose a task which should be accomplished by small groups which do interact among them, but she wants to keep the possibility of looking what the different groups are doing. For this we developed the necessary mechanisms inside the middleware in order to have

A Platform for Motivating Collaborative Learning

967

applications join and leave certain groups. This information is stored in the communication node and is used when the copy of an updated object has to be distributed among participants. Accordingly, we developed the methods which will send objects only to applications belonging to a certain group. An application can join more than one group, so it can receive updates coming from different groups. We also implemented methods for remotely force an application to join and/or leave a group. This was necessary because in many cases, the teacher or the moderator of a group was supposed to form smaller working groups. The teacher or moderator can then join the various in order to “see” what was happening on all of them.

7 Related Work and Discussion Throughout much of the fifty-year history of participatory simulations, computational technologies have played a central role. Nowadays, the ‘participatory’ aspect of these simulations can be enhanced by available technology, which gives support for the roles and rules to be distributed among the students. Researchers are highly interested in collaborative learning participatory simulations due to these simulations appear to make very difficult ideas around ‘distributed systems’ and ‘emergent behavior’ more accessible to students [19]. Furthermore the participatory simulation embeds student engagement and motivation in a playful social space [2] where they have rich conceptual resources for reasoning about and thoughtfully acting in playful spaces, and thus can more easily become highly engaged in the subject matter. Different hardware devices were used to implement participatory simulations. Colella et al. [3] developed wearable computational badges (or "thinking tags") that allow users to move freely while communicating information between badges. Colella, researched about student learning with one of the first instances of a participatory simulation supported by a thin layer of computing technology. Disease propagation models are natural candidates for this kind of participatory simulation and have been implemented by a number of researchers and curriculum developers [16]. Klopfer et al. [6] describes a prototype about “Environmental Detectives,” which is a participatory simulation experience where groups of students participate in a realtime simulation based on a local watershed. The real-world watershed includes streams, trees, and other natural elements. This real-world situation is then augmented by a simulation of an environmental disaster; in this case the river being polluted. Handheld computers implement a simulation where students can take air and water readings, interview people and get geographical information. Tomlinson [17] presented a novel interaction paradigm for multi-user participatory simulations which involves desktop PC screens that serve as virtual islands populated by autonomous 3D animated creatures, and mobile devices, such as tablet PCs, that serve as virtual rafts that participants may use to carry creatures between the islands. This paper described how this “Island/Raft” paradigm may serve as an interactive platform for education in several different content domains, and presented a functional implementation of the paradigm. The system is able to present content in ways that are of interest to learners and growing effectiveness as educational tools.

968

G. Zurita et al.

HubNet [19] is an architecture designed to give students the experience of participating as elements in a simulation of a complex dynamic system. HubNet is an open client-server architecture, which enables many users at the "Nodes" to control the behavior of individual objects or agents and to view the aggregated results on a central computer known as the "Hub". This network of nodes is integrated with a powerful suite of modeling, analysis and display tools that, together, give the capacity to "fly" the system in intuitive mode, to reflect on the emergent result of the simulation and to encode student strategies as rules which the system can then run independently. The network layer implements flexible communication protocols that include the ability to upload and download data sets, upload and download program, monitor key-presses at the handheld computer level, support real-time interaction as in network computer games, and form collaborative groups of various sizes: peer to peer, small groups, and whole class modes. Klopfer et al. [12] showed that the newer and more easily distributable version of Participatory Simulations on handhelds was equally as capable as the original Tagbased simulations in engaging students collaboratively in a complex problem-solving task. They feel that handhelds technology holds great promise for promoting collaborative learning as teachers struggle to find authentic ways to integrate technology into the classroom in addition to engaging and motivating students to learn science. The fact that handhelds can display results in a more attractive way suggests that they might be a more motivating platform for implementing participatory simulations. A collaborative learning participatory simulation implementing a stock exchange scenario was designed for master’s students in financial theory, using architectures based on a server and clients running on desktop PCs or laptops as well as on handhelds, [13]. Based on the literature above mentioned, we have identified that no system has yet been proposed or implemented for handhelds in a wireless ad-hoc network using a pen-based interface as main metaphor for user interaction. We believe that the most significant contribution of the work reported here is to provide a conceptual platform for applications of collaborative learning participatory simulations, which is easy to adapt to many subject-matter content knowledge and undergraduate curricular integration and encouraging the adoption of learner-centered strategies. The teachers, who pre-evaluate the application, suggest that the same technologies and ideas could be used across many subject matter areas. The design of effective learning environments of our conceptual platform have included (a) learner centered environment (learners construct their own meanings), (b) knowledge-centered environment (learners connect information into coherent wholes and embedding information in a context), (c) assessment-centered environment (learner use formative and summative assessment strategies and feedback), and (d) community-centered environments (learner work in collaborative learning norms). The next phase of our investigations will develop and explore more subject-specific applications and learning and motivational measures at the student level. In fact, another application was already developed with the goal of learning the building of confidence between buyers and sellers [20]. The first results of the testing of our prototypes indicate that the developed platform is the right approach to support the implementation of participatory simulations and that they can play a positive role supporting the learning of various systems in a motivating way.

A Platform for Motivating Collaborative Learning

969

Acknowledgments. This paper was funded by Fondecyt 1050601.

References 1. Campbell, A., Pargas, R.: Laptops in the classroom, Proceedings of the 34th SIGCSE technical symposium on Computer science education, Reno, Navada, USA, (2003), 98102. 2. Colella, V.: Participatory simulations: Building collaborative understanding through immersive dynamic modeling”. The Journal of the Learning Sciences 9, 2000, pp. 471– 500. 3. Colella, V., Borovoy, R., Resnick, M.: Participatory simulations: Using computational objects to learn about dynamic Systems, Conference on Human Factors in Computing Systems, (1998), 9 – 10. 4. Dai, G., Wang, H.: Physical Object Icons Buttons Gesture (PIBG): A new Interaction Paradigm with Pen, Proceedings of CSCWD 2004, LNCS 3168, (2005), 11-20. 5. Hinckley, K., Baudisch, P., Ramos, G., Guimbretiere, F.: Design and Analysis of Delimiters for Selection-Action Pen Gesture Phrases in Scriboli, Proceeding of CHI 2005, ACM, (2005), 451-460. 6. Klopfer, E., Perry, J., Squire, K., Jan, M.: Collaborative learning through augmented reality role playing, Proceeding of Computer support for collaborative learning, Taipei, Taiwan, ACM, (2005), 311-315. 7. Klopfer, E., Yoon, S., Perry, J: Using Palm Technology in Participatory Simulations of Complex Systems: A New Take on Ubiquitous and Accessible Mobile Computing, Journal of Science Education and Technology, 14(3), (2005), 285-297. 8. Klopfer, E., Yoon, S., Rivas, L.: Comparative analysis of Palm and wearable computers for Participatory Simulations, Journal of Computer Assisted Learning, 20, (2004), 347– 359. 9. Kopf, S., Scheele, N. Winschel, L., Effelsberg, W.: Improving Activity and Motivation of Students with Innovative Teaching and Learning Technologies, International Conference on Methods and Technologies for Learning (ICMTL), WIT press, (2005), 551 – 556. 10. Landay, J., Myers, B.: Sketching interfaces: Toward more human interface design, IEEE Computer, 34(3), (2001), 56-64 11. Long, A., Landay, J., Rowe, L.: PDA and gesture Use in Practice: Insights for Designers of Pen-based User Interfaces, Retrieved on 2006, December, from http://bmrc.berkeley.edu/research/publications/1997/142/clong.html 12. Moher, T., Wiley, J.: Technology Support for Learning Scientific Control as a Whole Class. Paper presented at the Annual Conference of the American Education Research Association, San Diego, CA., (2004). 13. Swan, K., Cook, D., Kratcoski, A., Lin, Y., Schenker, J., van ‘t Hooft, M.: Ubiquitous computing: rethinking teaching, learning and technology integration, Education and technology: Issues and applications, policy, and administration, In S. Tettegah & R. Hunter (Eds.) , New York: Elsevier, 2006, 231-252. 14. Roschelle, J.: Keynote paper: Unlocking the learning value of wireless mobile devices, Journal of Computer Assisted Learning 19 (3), (2003), 260–272. 15. Soloway, E., Norris, C., Blumenfeld, P., Fishman, R., Marx, R: Devices are Ready-atHand, Communications of the ACM, 44(6), (2001), 15–20. 16. Stor, M. & Briggs, W.: Dice and disease in the classroom. The Mathematics Teacher, 91(6). (1998), 464-468. 17. Tomlinson, B.: A heterogeneous animated platform for educational participatory simulations, Proceedings of Computer support for collaborative learning. Taipei, Taiwan, (2005), 677 – 681.

970

G. Zurita et al.

18. Vahey, P., Crawford, V.: Palm Education Pioneers Program: Final Evaluation Report, SRI International, Menlo Park, CA, (2002). 19. Wilensky, U., Stroup, W.: Learning through participatory simulations: Network-based design for systems learning in Classrooms, Proceedings of the Computer Supported Collaborative Learning Conference (CSCL’99) (eds. C. Hoadley & J. Roschelle), Erlbaum Associates, Mahwah, NJ, (1999), 667-676. 20. Zurita, G., Baloian, N., Baytelman, F., Antunes, P.: A Framework like a tool for specify motivating Collaborative Learning Participatory Simulations, Proceedings of the CSCWD Conference, Melbourne, Australia, May 2007.

Author Index

Ai, Dongmei 751 An, Youngeun 723 Arnaldi, Bruno 116, 140 Arnold, David B. 104 Baloian, Nelson 957 Ban, Xiaojuan 751 Baytelman, Felipe 957 Bertacchini, Francesca 369 Bidarra, Rafael 128 Bilotta, Eleonora 369 Braun, Anne-Kathrin 936 Caball´e, Santi 548 Cai, Yundong 260 Carretero, Maria del Puy Cha, Jaehyuk 457 Chan, Kai-Ming 162 Chang, Byoungchol 457 Chang, Yoonje 152 Che, Yinghui 667 Chen, Guojun 655 Chen, Haogang 3 Chen, Sherry Y. 336 Cheng, ZhiQuan 572 Chin, Seongah 22 Choi, Kwansun 357 Choo, Moonwon 22 Chou, Yu-Min 433 Chun, Junchul 191 Comeau, Gilles 763

225

Dai, Guozhong 381, 511 Danks, Michael 104 Daradoumis, Thanasis 548 Di Fiore, Fabian 92 Deng, Qingqiong 44 Dong, Zhanxun 468 Dung, Jian-Jie 445 D¨ unser, Andreas 305 Encarna¸ca ˜o, Jos´e L. 2 Enkhsaruul, Amarjagal 801

Fan, Zhen-Yu 829 Farias, Antonio 957 Fei, Guangzheng 687 Feijs, Loe 489 Feng, Xiaoqing 628 Feng, Zhiquan 638 Fong, Daniel Tik-Pui 162 Fu, Fong-Ling 293 Gabriele, Lorella 369 Gao, Jian 638 Garc´ıa-Alonso, Alejandro 225 Ge, Wenbing 606 Geng, Guo-Hua 812 Gerbaud, St´ephanie 116, 140 G´ omez-Mart´ın, Marco A. 477 G´ omez-Mart´ın, Pedro P. 477 G´ omez-Skarmeta, Antonio F. 876 Gonz´ alez-Calero, Pedro A. 477 Goodchild, Marc 104 Griffiths, Richard 104 Guimar˜ aes, Rui 128 Ha, Taejin 152 Ham, Dall-ho 457 Hamada, Mohamed 422 Han, Hee-Seop 316 Han, Huijian 677 Han, Seon-Kwan 274, 316 Hao, Qun 56 Harteveld, Casper 128 He, Feng 283 He, Tingting 793 Herbst, Iris 936 Herling, Jan 936 Hirakawa, Yasumoto 527 Ho, Hsi-Chuan 293 Hornecker, Eva 305 Hu, Jun 489 Huang, Hai 947 Huang, Hao 596 Huang, Jeng-Jia 829 Huang, Rui 647 Huang, Tao 411

972

Author Index

Huang, Tianyu 248 Huang, Xiaobin 347 Huang, Zhangjin 606 Hui, Kin-Chuen 162 Hur, Hyunmin 898 Hwang, Jooyeon 853 Hwang, Wu-Yuin 445 Imbert, Ricardo 499 Ip, Horace H. S. 391 Ishizuka, Mitsuru 324 Jaeger, Marc 44 Jang, Chulsang 63 Jang, Hyuna 784 Jang, Su-Min 864 Jeon, Changwan 357 Jeon, Gye-Beom 801 Ji, Xiuhua 677 Jia, Jinyuan 711 Jiang, Daoping 751 Jin, ShiYao 572 Jin, Xiaogang 730 Jorissen, Pieter 92 Jun, Kyungkoo 841 Jun, SooJin 274 Jung, Donggang 63 Jung, Keechul 63, 888, 916 Kanenishi, Kazuhide 527 Kang, Junggoo 63 Kang, Sooyong 457 Kim, Dongsik 357 Kim, Eun-Ju 801 Kim, Gerard J. 69 Kim, HyeonCheol 274, 316 Kim, Jongho 898 Kim, Jongin 916 Kim, Pankoo 723 Kim, Soo-Hwan 316 Kim, SooHwan 274 Kim, Sun-Jeong 801 Kim, Sung-Soo 784 Kim, Tae-Young 898 Kim, Won Cheol 801 Kwon, Ohryun 191 Kwong, Belton 391 Le, Jiajin 283 Lee, Haeyoung 182 Lee, Jong-Wan 801

Lee, Jongweon 81 Lee, Man Wai 336 Lee, Sungyeol 182 Lee, Sunheum 357 Lee, Won-Hyung 784 Lee, Won-Sook 763 Lee, WonGyu 274 Lee, Wonwoo 81 Lee, Yoonsoo 457 Lehr, Maider 225 Leung, Wai Lap 925 Li, Aili 347 Li, Chongrong 819 Li, Dan 647 Li, Fangyu 468 Li, Fengxia 248 Li, Jie 511 Li, Li 628 Li, Lijie 248 Li, Qicheng 3 Li, Xia 411 Li, Xin 687 Li, Xing 819 Li, Yefei 237, 468 Liang, Xiaohui 667 Lim, Mokhwa 457 Lin, Qiang 381, 511 Lin, Zuoyan 214 Liu, Huayong 793 Liu, Hui 677 Liu, Xian-Peng 56 Liu, Xiaohui 336 Liu, Yuanyuan 381 Lo, Jia-Jiunn 829 Lui, Pauline Po-Yee 162 Luo, Qi 539 Ma, Jin-Tao 56 Mao, Qirong 401 Mart´ınez-Carreras, Ma Antonia 876 Mart´ınez-Graci´ a, Eduardo 876 Mayer, Igor 128 Mazzeo, Vincenzo 369 Miao, Chunyan 260 Miao, Lanfang 701 Min, Kyongpil 191 Min, Seungki 916 Mitsuhara, Hiroyuki 527 Miyazaki, Kozi 1

Author Index Mollet, Nicolas 116, 140 Mora, Javier 763

Tan, Ah-Hwee 260 Tang, Xiaohui 3 Tang, Yuk-Ming 162 Thalmann, Daniel 171 Tsoi, Mathew Ho-Cheong

Nakasone, Arturo 324 Nakatsu, Ryohei 1 Nagai, Yurika 1 Ning, Shurong 751

Ugarte, Alejandro

Van Reeth, Frank 92 Vansichem, Gert 92 Vena, Stefano 369 Vexo, Fr´ed´eric 171 Vivanco, Karmelo 225

Oh, Ingul 63 Oh, Junhyung 63 Ortiz, Amalia 225 Oyarzun, David 225 Paik, Doowon 853 Palmier Campos, Pablo Pan, Zhigen 628 Pan, Zhigeng 214, 539 Pantano, Pietro 369 Park, Anjin 888 Park, Jihun 584 Park, Jongan 723 Park, Jongkyeong 22 Park, Peom 191 Park, Sunghun 584 Peng, Hongtao 347 Peng, Qunsheng 701

225

477

Qi, Yue 34, 616, 772 Qian, Yuechen 489 Ram Babu, Chinta 81 Rizzuti, Costantino 369 Rodriguez-Echavarria, Karina Ross, Philip 489 Salamin, Patrick 171 S´ anchez, Leticia 499 Seo, Jinseok 69 Shen, Jianbing 730 Shen, Xukun 34, 616, 772 Shen, Zhiqi 260 Sheng, Bin 560 Song, Chang Geun 801 Su, Jia-Han 445 Su, Yi-Shien 445 Sun, Shouqian 468 Sung, Mankyu 202 Sung, Mee Young 841

104

Wama, Takenori 1 Wang, Danli 511 Wang, Guoping 3, 606 Wang, Hong 56 Wang, Jian 647 Wang, Jing 667 Wang, Liang 34 Wang, Ping 711 Wang, Sen 711 Wang, Yuying 711 Wang, Zongjie 347 Wei, Na 812 Wei, Zhihui 740 Whangbo, TaegKeun 12 Wittk¨ amper, Michael 936 Won, Yoohun 584 Wong, Chee-Onn 916 Wong, Wai-Tak 433 Woo, Woontack 81, 152 Wu, Enhua 560 Wu, Huizhong 740 Wu, Ya-Ling 293 Xhafa, Fatos 548 Xiao, Liang 740 Xu, Hai-Yin 647 Xu, Liting 401 Xu, XiangHong 237, 468 Xu, Xiaoli 655 Yang, Jongyeol 916 Yang, Kun 3 Yang, Shen 772 Yano, Yoneo 527 Yeh, Shiou-Wen 829 Yin, Yixin 751

162

973

974

Author Index

Yin, Yuan 655 Yoo, Jae-Soo 864 Yoo, Jiyoon 357 You, Yonghee 841 Yu, Kai Ming 925 Yun, Chung-Ha 853 Zhan, Yongzhao 401 Zhang, Caiming 638 Zhang, Degan 347 Zhang, Hui 793 Zhang, Jianlin 411 Zhang, Jie 511 Zhang, Jing 655, 947

Zhang, Qian 12 Zhang, Xiaopeng 44 Zhang, Xuan 819 Zhao, Hanli 730 Zhao, Qinping 772, 904 Zhao, Xuewei 616 Zhou, Fan 596 Zhou, Feng 3 Zhou, Ming-Quan 812 Zhou, Ya 56 Zhou, Zhong 904 Zhu, Yongfeng 687 Zurita, Gustavo 957