Human-Computer Interaction. Design and User Experience Case Studies (Lecture Notes in Computer Science) 3030784673, 9783030784676

Citation preview

LNCS 12764

Masaaki Kurosu (Ed.)

Human-Computer Interaction Design and User Experience Case Studies Thematic Area, HCI 2021 Held as Part of the 23rd HCI International Conference, HCII 2021 Virtual Event, July 24–29, 2021, Proceedings, Part III

Lecture Notes in Computer Science Founding Editors Gerhard Goos Karlsruhe Institute of Technology, Karlsruhe, Germany Juris Hartmanis Cornell University, Ithaca, NY, USA

Editorial Board Members Elisa Bertino Purdue University, West Lafayette, IN, USA Wen Gao Peking University, Beijing, China Bernhard Steffen TU Dortmund University, Dortmund, Germany Gerhard Woeginger RWTH Aachen, Aachen, Germany Moti Yung Columbia University, New York, NY, USA

12764

More information about this subseries at http://www.springer.com/series/7409

Masaaki Kurosu (Ed.)

Human-Computer Interaction Design and User Experience Case Studies Thematic Area, HCI 2021 Held as Part of the 23rd HCI International Conference, HCII 2021 Virtual Event, July 24–29, 2021 Proceedings, Part III

123

Editor Masaaki Kurosu The Open University of Japan Chiba, Japan

ISSN 0302-9743 ISSN 1611-3349 (electronic) Lecture Notes in Computer Science ISBN 978-3-030-78467-6 ISBN 978-3-030-78468-3 (eBook) https://doi.org/10.1007/978-3-030-78468-3 LNCS Sublibrary: SL3 – Information Systems and Applications, incl. Internet/Web, and HCI © Springer Nature Switzerland AG 2021 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Foreword

Human-Computer Interaction (HCI) is acquiring an ever-increasing scientific and industrial importance, and having more impact on people’s everyday life, as an ever-growing number of human activities are progressively moving from the physical to the digital world. This process, which has been ongoing for some time now, has been dramatically accelerated by the COVID-19 pandemic. The HCI International (HCII) conference series, held yearly, aims to respond to the compelling need to advance the exchange of knowledge and research and development efforts on the human aspects of design and use of computing systems. The 23rd International Conference on Human-Computer Interaction, HCI International 2021 (HCII 2021), was planned to be held at the Washington Hilton Hotel, Washington DC, USA, during July 24–29, 2021. Due to the COVID-19 pandemic and with everyone’s health and safety in mind, HCII 2021 was organized and run as a virtual conference. It incorporated the 21 thematic areas and affiliated conferences listed on the following page. A total of 5222 individuals from academia, research institutes, industry, and governmental agencies from 81 countries submitted contributions, and 1276 papers and 241 posters were included in the proceedings to appear just before the start of the conference. The contributions thoroughly cover the entire field of HCI, addressing major advances in knowledge and effective use of computers in a variety of application areas. These papers provide academics, researchers, engineers, scientists, practitioners, and students with state-of-the-art information on the most recent advances in HCI. The volumes constituting the set of proceedings to appear before the start of the conference are listed in the following pages. The HCI International (HCII) conference also offers the option of ‘Late Breaking Work’ which applies both for papers and posters, and the corresponding volume(s) of the proceedings will appear after the conference. Full papers will be included in the ‘HCII 2021 - Late Breaking Papers’ volumes of the proceedings to be published in the Springer LNCS series, while ‘Poster Extended Abstracts’ will be included as short research papers in the ‘HCII 2021 - Late Breaking Posters’ volumes to be published in the Springer CCIS series. The present volume contains papers submitted and presented in the context of the Human-Computer Interaction (HCI 2021) thematic area of HCII 2021. I would like to thank the Chair, Masaaki Kurosu, for his invaluable contribution to its organization and the preparation of the proceedings, as well as the members of the Program Board for their contributions and support. This year, the HCI thematic area has focused on topics related to theoretical and methodological approaches to HCI, UX evaluation methods and techniques, emotional and persuasive design, psychological and cognitive aspects of interaction, novel interaction techniques, human-robot interaction, UX and technology acceptance studies, and digital wellbeing, as well as the impact of the COVID-19 pandemic and social distancing on interaction, communication, and work.

vi

Foreword

I would also like to thank the Program Board Chairs and the members of the Program Boards of all thematic areas and affiliated conferences for their contribution towards the highest scientific quality and overall success of the HCI International 2021 conference. This conference would not have been possible without the continuous and unwavering support and advice of Gavriel Salvendy, founder, General Chair Emeritus, and Scientific Advisor. For his outstanding efforts, I would like to express my appreciation to Abbas Moallem, Communications Chair and Editor of HCI International News. July 2021

Constantine Stephanidis

HCI International 2021 Thematic Areas and Affiliated Conferences

Thematic Areas • HCI: Human-Computer Interaction • HIMI: Human Interface and the Management of Information Affiliated Conferences • EPCE: 18th International Conference on Engineering Psychology and Cognitive Ergonomics • UAHCI: 15th International Conference on Universal Access in Human-Computer Interaction • VAMR: 13th International Conference on Virtual, Augmented and Mixed Reality • CCD: 13th International Conference on Cross-Cultural Design • SCSM: 13th International Conference on Social Computing and Social Media • AC: 15th International Conference on Augmented Cognition • DHM: 12th International Conference on Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management • DUXU: 10th International Conference on Design, User Experience, and Usability • DAPI: 9th International Conference on Distributed, Ambient and Pervasive Interactions • HCIBGO: 8th International Conference on HCI in Business, Government and Organizations • LCT: 8th International Conference on Learning and Collaboration Technologies • ITAP: 7th International Conference on Human Aspects of IT for the Aged Population • HCI-CPT: 3rd International Conference on HCI for Cybersecurity, Privacy and Trust • HCI-Games: 3rd International Conference on HCI in Games • MobiTAS: 3rd International Conference on HCI in Mobility, Transport and Automotive Systems • AIS: 3rd International Conference on Adaptive Instructional Systems • C&C: 9th International Conference on Culture and Computing • MOBILE: 2nd International Conference on Design, Operation and Evaluation of Mobile Communications • AI-HCI: 2nd International Conference on Artificial Intelligence in HCI

List of Conference Proceedings Volumes Appearing Before the Conference 1. LNCS 12762, Human-Computer Interaction: Theory, Methods and Tools (Part I), edited by Masaaki Kurosu 2. LNCS 12763, Human-Computer Interaction: Interaction Techniques and Novel Applications (Part II), edited by Masaaki Kurosu 3. LNCS 12764, Human-Computer Interaction: Design and User Experience Case Studies (Part III), edited by Masaaki Kurosu 4. LNCS 12765, Human Interface and the Management of Information: Information Presentation and Visualization (Part I), edited by Sakae Yamamoto and Hirohiko Mori 5. LNCS 12766, Human Interface and the Management of Information: Information-rich and Intelligent Environments (Part II), edited by Sakae Yamamoto and Hirohiko Mori 6. LNAI 12767, Engineering Psychology and Cognitive Ergonomics, edited by Don Harris and Wen-Chin Li 7. LNCS 12768, Universal Access in Human-Computer Interaction: Design Methods and User Experience (Part I), edited by Margherita Antona and Constantine Stephanidis 8. LNCS 12769, Universal Access in Human-Computer Interaction: Access to Media, Learning and Assistive Environments (Part II), edited by Margherita Antona and Constantine Stephanidis 9. LNCS 12770, Virtual, Augmented and Mixed Reality, edited by Jessie Y. C. Chen and Gino Fragomeni 10. LNCS 12771, Cross-Cultural Design: Experience and Product Design Across Cultures (Part I), edited by P. L. Patrick Rau 11. LNCS 12772, Cross-Cultural Design: Applications in Arts, Learning, Well-being, and Social Development (Part II), edited by P. L. Patrick Rau 12. LNCS 12773, Cross-Cultural Design: Applications in Cultural Heritage, Tourism, Autonomous Vehicles, and Intelligent Agents (Part III), edited by P. L. Patrick Rau 13. LNCS 12774, Social Computing and Social Media: Experience Design and Social Network Analysis (Part I), edited by Gabriele Meiselwitz 14. LNCS 12775, Social Computing and Social Media: Applications in Marketing, Learning, and Health (Part II), edited by Gabriele Meiselwitz 15. LNAI 12776, Augmented Cognition, edited by Dylan D. Schmorrow and Cali M. Fidopiastis 16. LNCS 12777, Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management: Human Body, Motion and Behavior (Part I), edited by Vincent G. Duffy 17. LNCS 12778, Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management: AI, Product and Service (Part II), edited by Vincent G. Duffy

x

List of Conference Proceedings Volumes Appearing Before the Conference

18. LNCS 12779, Design, User Experience, and Usability: UX Research and Design (Part I), edited by Marcelo Soares, Elizabeth Rosenzweig, and Aaron Marcus 19. LNCS 12780, Design, User Experience, and Usability: Design for Diversity, Well-being, and Social Development (Part II), edited by Marcelo M. Soares, Elizabeth Rosenzweig, and Aaron Marcus 20. LNCS 12781, Design, User Experience, and Usability: Design for Contemporary Technological Environments (Part III), edited by Marcelo M. Soares, Elizabeth Rosenzweig, and Aaron Marcus 21. LNCS 12782, Distributed, Ambient and Pervasive Interactions, edited by Norbert Streitz and Shin’ichi Konomi 22. LNCS 12783, HCI in Business, Government and Organizations, edited by Fiona Fui-Hoon Nah and Keng Siau 23. LNCS 12784, Learning and Collaboration Technologies: New Challenges and Learning Experiences (Part I), edited by Panayiotis Zaphiris and Andri Ioannou 24. LNCS 12785, Learning and Collaboration Technologies: Games and Virtual Environments for Learning (Part II), edited by Panayiotis Zaphiris and Andri Ioannou 25. LNCS 12786, Human Aspects of IT for the Aged Population: Technology Design and Acceptance (Part I), edited by Qin Gao and Jia Zhou 26. LNCS 12787, Human Aspects of IT for the Aged Population: Supporting Everyday Life Activities (Part II), edited by Qin Gao and Jia Zhou 27. LNCS 12788, HCI for Cybersecurity, Privacy and Trust, edited by Abbas Moallem 28. LNCS 12789, HCI in Games: Experience Design and Game Mechanics (Part I), edited by Xiaowen Fang 29. LNCS 12790, HCI in Games: Serious and Immersive Games (Part II), edited by Xiaowen Fang 30. LNCS 12791, HCI in Mobility, Transport and Automotive Systems, edited by Heidi Krömker 31. LNCS 12792, Adaptive Instructional Systems: Design and Evaluation (Part I), edited by Robert A. Sottilare and Jessica Schwarz 32. LNCS 12793, Adaptive Instructional Systems: Adaptation Strategies and Methods (Part II), edited by Robert A. Sottilare and Jessica Schwarz 33. LNCS 12794, Culture and Computing: Interactive Cultural Heritage and Arts (Part I), edited by Matthias Rauterberg 34. LNCS 12795, Culture and Computing: Design Thinking and Cultural Computing (Part II), edited by Matthias Rauterberg 35. LNCS 12796, Design, Operation and Evaluation of Mobile Communications, edited by Gavriel Salvendy and June Wei 36. LNAI 12797, Artificial Intelligence in HCI, edited by Helmut Degen and Stavroula Ntoa 37. CCIS 1419, HCI International 2021 Posters - Part I, edited by Constantine Stephanidis, Margherita Antona, and Stavroula Ntoa

List of Conference Proceedings Volumes Appearing Before the Conference

xi

38. CCIS 1420, HCI International 2021 Posters - Part II, edited by Constantine Stephanidis, Margherita Antona, and Stavroula Ntoa 39. CCIS 1421, HCI International 2021 Posters - Part III, edited by Constantine Stephanidis, Margherita Antona, and Stavroula Ntoa

http://2021.hci.international/proceedings

Human-Computer Interaction Thematic Area (HCI 2021) Program Board Chair: Masaaki Kurosu, The Open University of Japan, Japan • • • • •

Salah Ahmed, Norway Valdecir Becker, Brazil Nimish Biloria, Australia Maurizio Caon, Switzerland Zhigang Chen, China

• • • • •

Yu-Hsiu Hung, Taiwan Yi Ji, China Alexandros Liapis, Greece Hiroshi Noborio, Japan Vinícius Segura, Brazil

The full list with the Program Board Chairs and the members of the Program Boards of all thematic areas and affiliated conferences is available online at:

http://www.hci.international/board-members-2021.php

HCI International 2022 The 24th International Conference on Human-Computer Interaction, HCI International 2022, will be held jointly with the affiliated conferences at the Gothia Towers Hotel and Swedish Exhibition & Congress Centre, Gothenburg, Sweden, June 26 – July 1, 2022. It will cover a broad spectrum of themes related to Human-Computer Interaction, including theoretical issues, methods, tools, processes, and case studies in HCI design, as well as novel interaction techniques, interfaces, and applications. The proceedings will be published by Springer. More information will be available on the conference website: http://2022.hci.international/: General Chair Prof. Constantine Stephanidis University of Crete and ICS-FORTH Heraklion, Crete, Greece Email: [email protected]

http://2022.hci.international/

Contents – Part III

Design Case Studies Graphic Representations of Spoken Interactions from Journalistic Data: Persuasion and Negotiations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christina Alexandris, Vasilios Floros, and Dimitrios Mourouzidis A Study on Universal Design of Musical Performance System . . . . . . . . . . . Sachiko Deguchi Developing a Knowledge-Based System for Lean Communications Between Designers and Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yu-Hsiu Hung and Jia-Bao Liang Learn and Share to Control Your Household Pests: Designing a Communication Based App to Bridge the Gap Between Local Guides and the New Users Looking for a Reliable and Affordable Pest Control Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shima Jahani, Raman Ghafari Harivand, and Jung Joo Sohn

3 18

34

49

Developing User Interface Design Strategy to Improve Media Credibility of Mobile Portal News. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Min-Jeong Kim

67

Elderly-Centered Design: A New Numeric Typeface for Increased Legibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yu-Ren Lai and Hsi-Jen Chen

85

Research on Interactive Experience Design of Peripheral Visual Interface of Autonomous Vehicle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zehua Li, Xiang Li, JiHong Zhang, Zhixin Wu, and Qianwen Chen

97

Human-Centered Design Reflections on Providing Feedback to Primary Care Physicians . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ashley Loomis and Enid Montague

108

Interaction with Objects and Humans Based on Visualized Flow Using a Background-Oriented Schlieren Method . . . . . . . . . . . . . . . . . . . . . . . . . Shieru Suzuki, Shun Sasaguri, and Yoichi Ochiai

119

Research on Aging Design of News APP Interface Layout Based on Perceptual Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zhixin Wu, Zehua Li, Xiang Li, and Hongqian Li

138

xviii

Contents – Part III

Research on Modular Design of Children’s Furniture Based on Scene Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Junnan Ye, Wenhao Li, and Chaoxiang Yang A Design Method of Children Playground Based on Bionic Algorithm . . . . . Fei Yue, Wenda Tian, and Mohammad Shidujaman

153 173

Bias in, Bias Out – the Similarity-Attraction Effect Between Chatbot Designers and Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sarah Zabel and Siegmar Otto

184

Research on Immersive Virtual Reality Display Design Mode of Cantonese Porcelain Based on Embodied Interaction. . . . . . . . . . . . . . . . . . . . . . . . . . Shengyang Zhong, Yi Ji, Xingyang Dai, and Sean Clark

198

Design and Research of Children’s Robot Based on Kansei Engineering . . . . Siyao Zhu, Junnan Ye, Menglan Wang, Jingyang Wang, and Xu Liu

214

User Experience and Technology Acceptance Studies Exploring Citizens’ Attitudes Towards Voice-Based Government Services in Switzerland. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matthias Baldauf, Hans-Dieter Zimmermann, and Claudia Pedron Too Hot to Enter: Investigating Users’ Attitudes Toward Thermoscanners in COVID Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alice Bettelli, Valeria Orso, Gabriella Francesca Amalia Pernice, Federico Corradini, Luca Fabbri, and Luciano Gamberini Teens’ Conceptual Understanding of Web Search Engines: The Case of Google Search Engine Result Pages (SERPs) . . . . . . . . . . . . . . Dania Bilal and Yan Zhang What Futuristic Technology Means for First Responders: Voices from the Field. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shaneé Dawkins, Kerrianne Morrison, Yee-Yin Choong, and Kristen Greene

229

239

253

271

Blinking LEDs: Usability and User Experience of Domestic Modem Routers Indicator Lights. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Massimiliano Dibitonto

292

The Smaller the Better? A Study on Acceptance of 3D Display of Exhibits of Museum’s Mobile Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xinhao Guo, Jingjing Qiao, Ran Yan, Ziyun Wang, and Junjie Chu

303

Contents – Part III

Research on Information Visualization Design for Public Health Security Emergencies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wenkui Jin, Xurong Shan, and Ke Ma Comparative Study of the Interaction of Digital Natives with Mainstream Web Mapping Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marinos Kavouras, Margarita Kokla, Fotis Liarokapis, Katerina Pastra, and Eleni Tomai Success is not Final; Failure is not Fatal – Task Success and User Experience in Interactions with Alexa, Google Assistant and Siri . . . . . . . . . Miriam Kurz, Birgit Brüggemeier, and Michael Breiter Research on the Usability Design of HUD Interactive Interface. . . . . . . . . . . Xiang Li, Bin Jiang, Zehua Li, and Zhixin Wu Current Problems, Future Needs: Voices of First Responders About Communication Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kerrianne Morrison, Shanee Dawkins, Yee-Yin Choong, Mary F. Theofanos, Kristen Greene, and Susanne Furman Exploring the Antecedents of Verificator Adoption . . . . . . . . . . . . . . . . . . . Tihomir Orehovački and Danijel Radošević Are Professional Kitchens Ready for Dummies? A Comparative Usability Evaluation Between Expert and Non-expert Users . . . . . . . . . . . . . . . . . . . . Valeria Orso, Daniele Verì, Riccardo Minato, Alessandro Sperduti, and Luciano Gamberini Verification of the Appropriate Number of Communications Between Drivers of Bicycles and Vehicles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuki Oshiro, Takayoshi Kitamura, and Tomoko Izumi User Assessment of Webpage Usefulness . . . . . . . . . . . . . . . . . . . . . . . . . . Ning Sa and Xiaojun Yuan

xix

325

337

351 370

381

400

418

429 442

How Workarounds Occur in Relation to Automatic Speech Recognition at Danish Hospitals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Silja Vase

458

Secondary Task Behavioral Analysis Based on Depth Image During Driving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hao Wen, Zhen Wang, and Shan Fu

473

xx

Contents – Part III

Research on the Relationship Between the Partition Position of the Central Control Display Interface and the Interaction Efficiency . . . . . . . . . . . . . . . . JiHong Zhang, Haowei Wang, and Zehua Li

486

HCI, Social Distancing, Information, Communication and Work Attention-Based Design and Selective Exposure Amid COVID-19 Misinformation Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zaid Amin, Nazlena Mohamad Ali, and Alan F. Smeaton

501

Digital Communication to Compensate for Social Distancing: Results of a Survey on the Local Communication App DorfFunk . . . . . . . . . Matthias Berg, Anne Hess, and Matthias Koch

511

An Evaluation of Remote Workers’ Preferences for the Design of a Mobile App on Workspace Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cátia Carvalho, Edirlei Soares de Lima, and Hande Ayanoğlu

527

Feasibility of Estimating Concentration Level for not Disturbing Remote Office Workers Based on Kana-Kanji Conversion Confirmation Time . . . . . . Kinya Fujita and Tomoyuki Suzuki

542

A Smart City Stakeholder Online Meeting Interface . . . . . . . . . . . . . . . . . . Julia C. Lee and Lawrence J. Henschen Fostering Empathy and Privacy: The Effect of Using Expressive Avatars for Remote Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jieun Lee, Jeongyun Heo, Hayeong Kim, and Sanghoon Jeong PerformEyebrow: Design and Implementation of an Artificial Eyebrow Device Enabling Augmented Facial Expression. . . . . . . . . . . . . . . . . . . . . . Motoyasu Masui, Yoshinari Takegawa, Nonoka Nitta, Yutaka Tokuda, Yuta Sugiura, Katsutoshi Masai, and Keiji Hirata Improving Satisfaction in Group Dialogue: A Comparative Study of Face-to-Face and Online Meetings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Momoko Nakatani, Yoko Ishii, Ai Nakane, Chihiro Takayama, and Fumiya Akasaka EmojiCam: Emoji-Assisted Video Communication System Leveraging Facial Expressions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kosaku Namikawa, Ippei Suzuki, Ryo Iijima, Sayan Sarcar, and Yoichi Ochiai Pokerepo Join: Construction of a Virtual Companion Experience System . . . . Minami Nishimura, Yoshinari Takegawa, Kohei Matsumura, and Keiji Hirata

554

566

584

598

611

626

Contents – Part III

Visual Information in Computer-Mediated Interaction Matters: Investigating the Association Between the Availability of Gesture and Turn Transition Timing in Conversation . . . . . . . . . . . . . . . . . . . . . . . James P. Trujillo, Stephen C. Levinson, and Judith Holler Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xxi

643

659

Design Case Studies

Graphic Representations of Spoken Interactions from Journalistic Data: Persuasion and Negotiations Christina Alexandris1,2(B) , Vasilios Floros1,2 , and Dimitrios Mourouzidis1,2 1 National and Kapodistrian University of Athens, Athens, Greece

[email protected] 2 European Communication Institute (ECI), Danube University Krems and National Technical

University of Athens, Athens, Greece

Abstract. Generated graphic representations for interactions involving persuasion and negotiations are intended to assist evaluation, training and decisionmaking processes and for the construction of respective models. As described in previous research, discourse and dialog structure are evaluated by the y level value around which the graphic representation is developed. Special emphasis is placed on emotion used as a tool for persuasion with the respective expressions, pragmatic elements and the depiction of information not uttered and their subsequent use in the collection of empirical and statistical data. Keywords: Spoken journalistic texts · Spoken interaction · Persuasion · Negotiations · Graphic representations · Cognitive bias

1 Registration of Spoken Interaction: Previous Research With the increase in the variety and complexity of spoken Human Computer Interaction (HCI) (and Human Robot Interaction - HRI) applications, the correct perception and evaluation of information not uttered is an essential requirement in systems with emotion recognition, virtual negotiation, psychological support or decision-making. Pragmatic features in spoken interaction and information conveyed but not uttered by Speakers can pose challenges to applications processing spoken texts that are not domain-specific, as in the case of spoken political and journalistic texts, including cases where the elements of persuasion and negotiations are involved. Although usually underrepresented both in linguistic data for translational and analysis purposes and in Natural Language Processing (NLP) applications, spoken political and journalistic texts may be considered to be a remarkable source of empirical data both for human behavior and for linguistic phenomena, especially for spoken language. However, these text types are often linked to challenges for their evaluation, processing and translation, not only due to their characteristic richness in socio-linguistic and socio-cultural elements and to discussions and interactions beyond a defined agenda, but also in regard to the possibility of different types of targeted audiences - including non-native speakers and the international community [1]. Additionally, in spoken © Springer Nature Switzerland AG 2021 M. Kurosu (Ed.): HCII 2021, LNCS 12764, pp. 3–17, 2021. https://doi.org/10.1007/978-3-030-78468-3_1

4

C. Alexandris et al.

political and journalistic texts there is also the possibility of essential information, presented either in a subtle form or in an indirect way, being often undetected, especially by the international public. In this case, spoken political and journalistic texts also contain information that is not uttered but can be derived from the overall behavior of speakers and participants in a discussion or interview. These characteristics, including the feature of spontaneous turn-taking [31, 39] in many spoken political and journalistic texts, are linked to the implementation of strategies concerning the analysis and processing of discourse structure and rhetorical relations (in addition to previous research) [10, 22, 35, 41]. In our previous research [2, 6, 23], a processing and evaluation framework was proposed for the generation of graphic representations and tags corresponding to values and benchmarks depicting the degree of information not uttered and non-neutral elements in Speaker behavior in spoken text segments. The implemented processing and evaluation framework allows the graphic representation to be presented in conjunction with the parallel depiction of speech signals and transcribed texts. Specifically, the alignment of the generated graphic representation with the respective segments of the spoken text enables a possible integration in existing transcription tools. In particular, strategies typically employed in the construction of most Spoken Dialog Systems, such as keyword processing in the form of topic detection [13, 19, 24, 25] (from which approaches involving neural networks are developed [38]), were adapted in the functions of the designed and constructed interactive annotation tool [2, 6, 23], designed to operate with most commercial transcription tools. The output provides the User-Journalist with (a) the tracked indications of the topics handled in the interview or discussion and (b) the graphic pattern of the discourse structure of the interview or discussion. The output (a) and (b) also included functions and respective values reflecting the degree in which the speakers-participants address or avoid the topics in the dialog structure (“RELEVANCE” Module) as well as the degree of tension in their interaction (“TENSION” Module). The implemented “RELEVANCE” Module [23], intended for the evaluation of short speech segments, generates a visual representation from the user’s interaction, tracking the corresponding sequence of topics (topic-keywords) chosen by the user and the perceived relations between them in the dialog flow. The generated visual representations depict topics avoided, introduced or repeatedly referred to by each Speaker-Participant, and in specific types of cases may indicate the existence of additional, “hidden”[23] Illocutionary Acts [9, 14, 15, 32] other than “Obtaining Information Asked” or “Providing Information Asked” in a discussion or interview. Thus, the evaluation of Speaker-Participant behavior targets to by-pass Cognitive Bias, specifically, Confidence Bias [18] of the user-evaluator, especially if multiple users-evaluators may produce different forms of generated visual representations for the same conversation and interaction. The generated visual representations for the same conversation and interaction may be compared to each other and be integrated in a database currently under development. In this case, chosen relations between topics may describe Lexical Bias [36] and may differ according to political, socio-cultural and linguistic characteristics of the user-evaluator, especially if international users are concerned [21, 26, 27, 40] due to lack of world knowledge of the language community

Graphic Representations of Spoken Interactions from Journalistic Data

5

involved [7, 16, 37]. In the “RELEVANCE” Module [23], a high frequency of Repetitions (value 1), Generalizations (value 3) and Topic Switches (value -1) in comparison to the duration of the spoken interaction is connected to the “(Topic) Relevance” benchmarks with a value of “Relevance (X)” [3, 5] (Fig. 1). 2.5 2 1.5 1 0.5 0 -0.5 0

2

4

6

8

10

12

14

-1 -1.5 Fig. 1. Generated graphic representation with multiple “Topic Switch” relations (Mourouzidis et al., 2019).

The development of the interactive, user-friendly annotation tool is based on data and observations provided by professional journalists (European Communication Institute (ECI), Program M.A in Quality Journalism and Digital Technologies, Danube University at Krems, Austria, the Athena- Research and Innovation Center in Information, Communication and Knowledge Technologies, Athens, the Institution of Promotion of Journalism Ath.Vas. Botsi, Athens and the National and Technical University of Athens, Greece).

2 Association Relations and (Training) Data for Negotiation Models However, in the above-presented previous research, the “Association” relation is not included in the evaluations concerned. Furthermore, the “Association” relation is of crucial importance in dialogues constituting persuasion and types of negotiation based on persuasion [34], especially if emotion is used as a tool for persuasion [30], establishing a link between persuasion, emotion and language [30]. Emotion as a tool for persuasion may be used in diverse types of negotiation skills, apart from persuasion tactics [12, 30, 34], including “value creating”/ “value claiming” tactics and “defensive” tactics [34]. “Association” relations between words and their related topics are often used to direct the Speaker into addressing the topic of interest and/or to produce the desired answers. In some cases, the “Generalization” may also be used for the same purpose, as a means of introducing a (not directly related) topic of interest via “Generalization”.

6

C. Alexandris et al.

For negotiation applications, the identification of words and their related topics contributes to strategies targeting to directing the Speaker-Participant to the desired goal and the avoidance of unwanted “Association” types as well as unwanted other types of relations -“Repetitions”, “Topic Switch” and “Generalizations” (Fig. 2).

4 3 2 1 0 -1

0

2

4

6

8

10

12

14

-2 Fig. 2. Generated graphic representation with multiple “Association” relations. (Mourouzidis et al., 2019).

The “Association” relations between words and their related topics contribute to the analysis and development of negotiation procedures. In this case, Cognitive Bias and socio-cultural factors play a crucial role in regard to the perception of the perceived relations-distances between word-topics. For example, the word-topics “Country X” (name withheld) –“defense spending” or “military confrontation” – “chemical weapons” may generate an “Association” (ASOC) or “Topic Switch” (SWITCH) reactions and choices from users, depending on whether they are perceived as related or different topics in the spoken interaction. Diverse reactions may also apply in the case of the “Association” and “Generalization” relations, where “treaties” and “international commitment” may generate “Association” (ASOC) or “Generalization” (GEN) reactions and choices from users: “treaties” is associated with “international commitment” or “treaties” are linked to “international commitment” with a “Generalization” relation. Differences concerning the perception of the “Association” (ASOC) relations between word-topics are measured in the form of triple tuples as perceived relationsdistances between word-topics [3], related to Lexical Bias (Cognitive Bias) concerning semantic perception [36]. Examples of segments in (interactively) generated patterns from user-specific choices between topics are the following, where the distances between topics in the generated patterns are registered as triple tuples (triplets): (military confrontation, chemical weapons, 2) (“Association”), (treaties, international commitment, 3) (“Generalization”). These triplets and the sequences they form may be converted into vectors (or other forms and models), used as training data for creating negotiation models and their variations.

Graphic Representations of Spoken Interactions from Journalistic Data

7

Fig. 3. Interface for generating graphic representation with multiple “Association” relations.

Possible differences in the perceived relations with the Lexical Bias concerned may play an essential role both in the employment of negotiation tactics (based on crosscultural analysis) and in training applications. The number of registered “Association” relations in the processed wav.file or video file may be used to evaluate persuasion tactics employed in spoken interaction involving negotiations (a) and their possible employment in the construction of training data and negotiation models (b). Since the generated graphic representations are based on perceived relations, they may also be used for evaluating trainees performance (c). We note that, independently from interactive and user-specific choices, topics may be also pre-defined and/or automatically detected with word relations based on existing (ontological and semantic) databases. However, this commonly used strategy and practice is proposed to be employed in cases where persuasion and negotiation tactics are monitored and checked against a pre-defined model, either as a form to control spoken interaction or as means to evaluate the pre-defined model. The following examples in Figs. 3 and 4 depict the user interface and the generated graphic representations containing multiple “Association” relations: Chosen wordtopics and their relations in dialog segment with two speakers-participants (resulting to a “No” answer): “military confrontation”, “reckless behavior”, “strikes”, “danger”, “crisis”, “crisis”, “consequences”, “aggression”, “consequences”, “trust”. (choices may

8

C. Alexandris et al.

vary among users, especially in the international public), Data from an actual interview on a world news channel (BBC HardTalk 720- 16–04-2018).

Fig. 4. Generated graphic representation with multiple “Association” relations and respective values (including one “No” Answer (−2) – presented in Sect. 3).

3 Affirmative and Negative Answers in Negotiations In spoken interaction concerning persuasion and types of negotiation based on persuasion [12, 30, 34], perceived affirmative (“Yes”) and negative (“No”) answers are integrated in the present framework with the respective “0” (zero) and “−2” values. Specifically, an affirmative answer is assigned a “0” value, similar to the initial “0” (zero) value starting the entire interactive processing of the wav.file. An example of a generated graphic representation with multiple “Yes” answers is depicted in Fig. 5. In this case, the spoken interaction (concerning persuasion or negotiation based on persuasion) contains multiple positive answers and the respective multiple “0” (zero) values (Fig. 5). A negative answer is assigned a “−2” value, lower than the “−1” Topic Switch value (Fig. 6). Thus, a negotiation with a sequence of negative answers and several attempts to change a topic or to approach a (seemly) different topic will generate a graphic representation below the “0” (zero) value.

Graphic Representations of Spoken Interactions from Journalistic Data

9

An example of generated graphic representations below the “0” (zero) value depicting spoken interactions (persuasion –negotiations) is shown in Fig. 6. In this case, the spoken interaction contains multiple negative answers and/or multiple attempts to switch to a different topic (Fig. 6).

4 3 2 1 0 0

2

4

6

8

10

12

14

-1 Fig. 5. Generated graphic representation with multiple “Yes” answers.

4 3 2 1 0 -1

0

2

4

6

8

10

12

14

-2 -3 Fig. 6. Generated graphic representation with multiple “No” answers (and topic switches).

As in the above-described case of “Association” and “Generalization” relations, for affirmative and negative answers, the distances between topics in the generated patterns are registered and may be be used as training data for creating negotiation models and their variations. However, in the case of affirmative and negative answers, the topic and the respective answer is not registered as a triplet but is registered as a tuple: (stability, 0) (“Affirmative Answer”), (sanctions, −2) (“Negative Answer”). Similarly to the registered “Association” relations, the number of perceived affirmative (“Yes”) and negative (“No”) answers in the processed wav.file or video file may be

10

C. Alexandris et al.

used to evaluate persuasion tactics employed in spoken interaction involving negotiations (a), for the construction of training data and negotiation models (b) or for evaluating a trainees performance (c).

4 Registering Word-Topics and Their Impact in Persuasion and Negotiations 4.1 Word-Topics and Persuasion Tactics The type of word-topics concerned in the registered “Association” relations and the “Yes” or “No” answers in the processed wav.file or video file may also be used to evaluate persuasion tactics employed in spoken interaction involving negotiations. Word-topics and the registered relations and answers may be linked to positive responses and/or collaborative speaker behavior or negative responses, tension and conflict. Detecting and registering points of tension or other types of behavior and their impact in the dialogue structure facilitates the evaluation of persuasion tactics and types of negotiation based on persuasion [30, 34], especially “value creating”/ “value claiming” tactics and “defensive” tactics [30, 34] and in other cases where a link between persuasion, emotion and language is used [12, 30]. 4.2 Word-Topics and Word-Types as Reaction Triggers For negotiation applications, words and their related topics can be identified as triggers for different types of reactions (positive, collaborative behavior or tension). The words and their related topics may concern the following two types of information: (1) “Association” (or other) relations that are context-specific, connected to current events and state-of-affairs, (2) “Association” (or other) relations that concern words with inherent socio-culturally determined linguistic features and are usually independent from current events and state-of-affairs. In the second case (2) it is often observed that the semantic equivalent of the same word on one language sometimes may appear more formal or with more “gravity” than in another language, either emphasizing the role of the word in an utterance or being related to word play and subtle suggested information. The presence of such “gravity words” [1, 4] may contribute to the degree of formality or intensity of conveyed information in a spoken utterance. It is observed that these differences between languages in regard to the “gravity” of words are often related to polysemy, where the possible meanings and uses of a word seem to “cast a shadow” over its most commonly used meaning. Similarly to the above-described category, words with an “evocative” element concern their “deeper” meanings related to their use in tradition, in music and in literature and may sometimes be related to emotional impact in discussions and speeches. In contrast to “gravity” words, “evocative” words usually contribute to a descriptive or emotional tone in an utterance [1, 4]. Here, it is noted that, according to Rockledge et al., 2018, “the more extremely positive the word, the greater the probability individuals were to associate that word with persuasion” [30]. In the generated graphic representations, perceived “Gravity” and “Evocative” words are signalized (for example, as “W”) in the curve connecting the word-topics. This

Graphic Representations of Spoken Interactions from Journalistic Data

11

signalization indicates the points of “Gravity” and “Evocative” words as “Word-Topic” triggers in respect to the areas of perceived tension or other types of reactions in the processed dialog segment with two (or more) speakers-participants. In Figs. 7 and 8 the perceived “Gravity” and “Evocative” words also constitute word-topics (Figs. 7 and 8). 3 2

W

W

W

1 0 -1

0

2

4

6

8

-2

10

12

14

-2

-3 Fig. 7. Generated graphic representation with multiple “Association” relations and Word-Topic triggers (“W”).

4 3 2

W

1

W

0 -1

0

2

4

6

8

W

10

12

14

-2 -3 Fig. 8. Generated graphic representation with multiple “No” answers and Word-Topic triggers (“W”).

The detected word types may be used as training data for creating negotiation models and their variations, as in the above-described cases. The signalized Word-Topic triggers may be appended as marked values (for example, with “&”) in the respective tuples or triple tuples, depending on the context in which they occur: (sanctions, −2, &dignity) (“Negative Answer”), (military confrontation, chemical weapons, 2, &justice) (“Association”). If the Word-Topic triggers constitute topics, they are repeated in the tuple or triple tuple, where they receive the respective mark: (country, people, 2, &people) (“Association”).

12

C. Alexandris et al.

Signalized “Gravity” and “Evocative” words can be identified either from databases constructed from collected empirical data or from existing resources such as Wordnets. In spoken utterances “Gravity” words and especially “Evocative” words are observed to often have their prosodic and even their phonetic-phonological features intensified [1, 4]. The commonly occurring observed connection to intensified prosodic phoneticphonological features constitutes an additional pointer to detecting and signalizing “Gravity” and “Evocative” words [1, 4]. 4.3 Word-Topics as Tension Triggers Previous research depicted points of tension in two-party discussions and interviews containing longer speech segments. These points are detected and signalized by the implemented “TENSION” Module in the form of graphic representations [2], enabling the evaluation of the behavior of speakers-participants. In spoken interaction concerning persuasion and types of negotiation based on persuasion, detected points of tension in the generated graphic representations enable the registration of word-topics and sequences of word-topics preceding tension and the registration of word-topics and sequences of word-topics following tension. The evaluation of such data contributes both to the construction and training of models for the avoidance of tension (i) and for the purposeful creation of tension (ii). Multiple points of tension (referred to as “hot spots”) [2] indicate a more argumentative than a collaborative interaction, even if speakers-participants display a calm and composed behavior. Points of possible tension and/or conflict between speakers-participants (“hot-spots”) are signalized in generated graphic representations of registered negotiations (or other type of spoken interaction concerning persuasion), with special emphasis on words and topics triggering tension and non-collaborative speaker-participant behavior. As presented in previous research [2], a point of tension or “hot spot” consists of the pair of utterances of both speakers, namely a question-answer pair or a statementresponse pair or any other type of relation between speaker turns. In longer utterances, a defined word count and/or sentence length from the first words/segment of the second speaker’s (Speaker 2) and from the words/segment of the first speaker’s (Speaker 1) the utterance are processed [2, 11]. The automatically signalized “hot spots” (and the complete utterances consisting of both speaker turns) are extracted to a separate template for further processing. For a segment of speaker turns to be automatically identified as a “hot spot”, a set of (at least two of the proposed three (3) conditions must apply [2] to one or to both of the speaker’s utterances. The three (3) conditions are directly or indirectly related to flouting of Maxims of the Gricean Cooperative Principle [14, 15] (additional, modifying features (1), reference to the interaction itself and to its participants with negation (2) and (3) prosodic emphasis and/or exclamations). With the exception of prosodic emphasis, these conditions concern features detectable with a POS Tagger (for example, the Stanford POS Tagger, http://nlp.stanford.edu/software/tagger.shtml) or they may constitute a small set of entries in a specially created lexicon or may be retrieved from existing databases or Wordnets. The “hot spots” are connected to the “Tension” benchmark with a value of “Y” or “Tension (Y)” [2] and the “Collaboration”

Graphic Representations of Spoken Interactions from Journalistic Data

13

benchmark with a value of “Z” or “Collaboration (Z)”, described in previous research [2, 3]. In the generated graphic representations, word-topics as tension triggers are signalized (for example, as “W”) in the curve connecting the word-topics (Fig. 9). This signalization indicates the points of word-topics as tension triggers in respect to the areas of perceived tension in the processed dialog segment with two (or more) speakers-participants. The detected word types may be used as training data for creating negotiation models and their variations, as in the above-described cases. 4 3 2

W

1

W

0 -1

0

2

4

6

8

W

10

12

14

-2 -3 Fig. 9. Generated graphic representation with multiple “No” answers and Word-Topic triggers (“W”) and Tension (shaded area between topics) in generated graphic representation and “tension trigger” (“W”).

4.4 Tension Triggers and Paralinguistic Information Furthermore, in previous research [2] “hot spots” signalizing tension may include an interactive annotation of paralinguistic features with the corresponding tags. Words classified as “tension triggers” may, in some cases, be easily detected with the aid of registered and annotated paralinguistic features, where the paralinguistic element may complement or intensify the information content of the word related to perceived tension in the spoken interaction. In some instances, the paralinguistic element may contradict the information content of the “tension trigger”, for example, a smile when a word of negative content is uttered. In this case, the speaker’s behavior may be related to irony or a less intense negative emotion such as annoyance or contempt. With paralinguistic features concerning information that is not uttered, the Gricean Cooperative Principle is violated if the information conveyed is perceived as not complete (Violation of Quantity or Manner) or even contradicted by paralinguistic features (Violation of Quality). Depending on the type of specifications used, for paralinguistic features depicting contradictory information to the information content of the spoken utterance, the additional signalization of “!” is proposed, for example, “[! facial-expr: eye-roll]” and “[! gesture: clenched-fist]”.

14

C. Alexandris et al.

According to the type of linguistic and paralinguistic features signalized, features of more subtle emotions can be detected. Less intense emotions are classified in the middle and outer zones of the Plutchik Wheel of Emotions [28] and are usually too subtle to be easily extracted by sensor and/or speech signal data. In this case, linguistic information with or without a link to paralinguistic features demonstrates a more reliable source of a speaker’s attitude, behavior and intentions, especially for subtle negative reactions in the Plutchik Wheel of Emotions, namely “Apprehension”, “Annoyance”, “Disapproval”, “Contempt”, “Aggressiveness” [28]. These subtle emotions are of importance in spoken interactions involving persuasion and negotiations. Data from the interactive annotation of paralinguistic features may also be integrated into models and training data, however, further research is necessary for the respective approaches and strategies.

5 Conclusions and Further Research: Insights for Sentiment Analysis Applications The presented generated graphic representations for interactions involving persuasion and negotiations are intended to assist evaluation, training and decision-making processes and for the construction of respective models. In particular, the graphic representations generated from the processed wav.file or video files may be used to evaluate persuasion tactics employed in spoken interaction involving negotiations (a), their possible employment in the construction of training data and negotiation models (b) and for evaluating a trainee’s performance (c). New insights are expected to be obtained by the further analysis and research in the persuasion-negotiation data processed. Further research is also expected to contribute to the overall improvement of the graphical user interface (GUI), as one of the basic envisioned upgrades of the application. The presented generated graphic representations enable the visibility of information not uttered, in particular, tension and the overall behavior of speakers-participants. The visibility of all information content, including information not uttered, contributes to the collection and compilation of empirical and statistical data for research and/or for the development of HCI- HRI Sentiment Analysis and Opinion Mining applications, as (initial) training and test sets or for Speaker (User) behavior and expectations. This is of particular interest in cases where an international public is concerned and where a variety of linguistic and socio-cultural factors is included. Information that is not uttered is problematic in Data Mining and Sentiment AnalysisOpinion Mining applications, since they mostly rely on word groups, word sequences and/or sentiment lexica [20], including recent approaches with the use of neural networks [8, 17, 33], especially if Sentiment Analysis from videos (text, audio and video) is concerned. In this case, even if context dependent multimodal utterance features are extracted, as proposed in recent research [29], the semantic content of a spoken utterance may be either complemented or contradicted by a gesture, facial expression or movement. The words and word-topics triggering non-collaborative behavior and tension (“hot spots”) and the content of the extracted segments where tension is detected provide

Graphic Representations of Spoken Interactions from Journalistic Data

15

insights for word types and the reaction of speakers, as well as insights of Opinion Mining and Sentiment Analysis. The above-observed additional dimensions of words in spoken interaction, especially in political and journalistic texts, may also contribute to the enrichment of “Bag-ofWords” approaches in Sentiment Analysis and their subsequent integration in training data for statistical models and neural networks.

References 1. Alexandris, C.: Issues in Multilingual Information Processing of Spoken Political and Journalistic Texts in the Media and Broadcast News, Cambridge Scholars, Newcastle upon Tyne, UK (2020) 2. Alexandris, C., Mourouzidis, D., Floros, V.: Generating graphic representations of spoken interactions revisited: the tension factor and information not uttered in journalistic data. In: Kurosu, M. (ed.) HCII 2020. LNCS, vol. 12181, pp. 523–537. Springer, Cham (2020). https:// doi.org/10.1007/978-3-030-49059-1_39 3. Alexandris, C.: Evaluating cognitive bias in two-party and multi-party spoken interactions. In: Proceedings from the AAAI Spring Symposium, Stanford University (2019) 4. Alexandris, C.: Visualizing Pragmatic Features in Spoken Interaction: Intentions, Behavior and Evaluation. In: Proceedings of the 1st International Conference on Linguistics Research on the Era of Artificial Intelligence – LREAI, Dalian, October 25–27, 2019, Dalian Maritime University (2019) 5. Alexandris, C.: Measuring cognitive bias in spoken interaction and conversation: generating visual representations. In: Beyond Machine Intelligence: Understanding Cognitive Bias and Humanity for Well-Being AI Papers from the AAAI Spring Symposium Stanford University, Technical Report SS-18-03, pp. 204-206 AAAI Press Palo Alto, CA (2018) 6. Alexandris, C., Nottas, M., Cambourakis, G.: Interactive evaluation of pragmatic features in spoken journalistic texts. In: Kurosu, M. (ed.) HCI 2015. LNCS, vol. 9171, pp. 259–268. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21006-3_26 7. Alexandris, C.: English, german and the international “semi-professional” translator: a morphological approach to implied connotative features. J. Lang. Transl. Sejong Univ. Korea 11(2), 7–46 (2010) 8. Arockiaraj, C.M.: Applications of neural networks in data mining. Int. J. Eng. Sci. 3(1), 8–11 (2013) 9. Austin, J.L.: How to Do Things with Words, 2nd edn. University Press, Oxford Paperbacks, Oxford (1962).(Urmson, J.O., Sbisà, M. (eds.) 1976) 10. Carlson, L., Marcu, D., Okurowski, M. E.: Building a discourse-tagged corpus in the framework of rhetorical structure theory. In: Proceedings of the 2nd SIGDIAL Workshop on Discourse and Dialogue, Eurospeech 2001, Denmark, September 2001 (2001) 11. Cutts, M.: Oxford Guide to Plain English, 4th edn. Oxford University Press, Oxford, UK (2013) 12. Evans, N.J., Park, D.: Rethinking the persuasion knowledge model: schematic antecedents and associative outcomes of persuasion knowledge activation for covert advertising. J. Curr. Issues Res. Advert. 36(2), 157–176 (2015). https://doi.org/10.1080/10641734.2015.1023873 13. Floros, V., Mourouzidis, D.: Multiple Task Management in a Dialog System for Call Centers. Master’s thesis, Department of Informatics and Telecommunications, National University of Athens, Greece (2016) 14. Grice, H.P.: Studies in the Way of Words. Harvard University Press, Cambridge, MA (1989)

16

C. Alexandris et al.

15. Grice, H.P.: Logic and conversation. In: Cole, P., Morgan, J. (eds.) Syntax and Semantics, vol. 3, pp. 41–58. Academic Press, New York (1975) 16. Hatim, B.: Communication Across Cultures: Translation Theory and Contrastive Text Linguistics. University of Exeter Press, Exeter, UK (1997) 17. Hedderich, M.A., Klakow, D.: Training a neural network in a low-resource setting on automatically annotated noisy data. In: Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP, Melbourne, Australia, pp. 12–18. Association for Computational Linguistics-ACL (2018) 18. Hilbert, M.: Toward a synthesis of cognitive biases: how noisy information processing can bias human decision making. Psychol. Bull. 138(2), 211–237 (2012) 19. Lewis, J.R.: Introduction to Practical Speech User Interface Design for Interactive Voice Response Applications, IBM Software Group, USA, Tutorial T09 presented at HCI 2009 San Diego. CA, USA (2009) 20. Liu, B.: Sentiment Analysis and Opinion Mining. Morgan & Claypool, San Rafael, CA (2012) 21. Ma, J.: A comparative analysis of the ambiguity resolution of two English-Chinese MT approaches: RBMT and SMT. Dalian Univ. Technol. J. 31(3), 114–119 (2010) 22. Marcu, D.: Discourse trees are good indicators of importance in text. In: Mani, I., Maybury, M. (eds.) Advances in Automatic Text Summarization, pp. 123–136. The MIT Press, Cambridge, MA (1999) 23. Mourouzidis, D., Floros, V., Alexandris, C.: Generating graphic representations of spoken interactions from journalistic data. In: Kurosu, M. (ed.) HCII 2019. LNCS, vol. 11566, pp. 559–570. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22646-6_42 24. Nass, C., Brave, S.: Wired for Speech: How Voice Activates and Advances the HumanComputer Relationship. The MIT Press, Cambridge, MA (2005) 25. Nottas, M., Alexandris, C., Tsopanoglou, A., Bakamidis, S.: A hybrid approach to dialog input in the citzenshield dialog system for consumer complaints. In: Proceedings of HCI 2007, Beijing, People’s Republic of China (2007) 26. Paltridge, B.: Discourse Analysis: An Introduction. Bloomsbury Publishing, London (2012) 27. Pan, Y.: Politeness in Chinese Face-to-Face Interaction. Advances in Discourse Processes series, vol. 67. Ablex Publishing Corporation, Stamford, CT, USA (2000) 28. Plutchik, R.: A psychoevolutionary theory of emotions. Soc. Sci. Inf. 21, 529–553 (1982). https://doi.org/10.1177/053901882021004003 29. Poria, S., Cambria, E., Hazarika, D., Mazumder, N., Zadeh, A., Morency, L-P.: Contextdependent sentiment analysis in user-generated videos. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, July 30August 4 2017, pp. 873–88. Association for Computational Linguistics - ACL (2017). https:// doi.org/10.18653/v1/P17-1081 30. Rocklage, M., Rucker, D., Nordgren, L.: Persuasion, emotion, and language: the intent to persuade transforms language via emotionality. Psychol. Sci. 29(5), 749–760 (2018). https:// doi.org/10.1177/0956797617744797 31. Sacks, H., Schegloff, E.A., Jefferson, G.: A simplest systematics for the organization of turn-taking for conversation. Language 50, 696–735 (1974) 32. Searle, J.R.: Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press, Cambridge, MA (1969) 33. Shah, K., Kopru, S., Ruvini, J-D.: Neural network based extreme classification and similarity models for product matching. In: Proceedings of NAACL-HLT 2018, New Orleans, Louisiana, June 1-6, 2018, pp. 8–15. Association for Computational Linguistics-ACL (2018) 34. Skonk, K.: 5 Types of Negotiation Skills, Program on Negotiation Daily Blog, Harvard Law School, 14 May 2020. https://www.pon.harvard.edu/daily/negotiation-skills-daily/types-ofnegotiation-skills/. Accessed 11 Nov 2020

Graphic Representations of Spoken Interactions from Journalistic Data

17

35. Stede, M., Taboada, M., Das, D.: Annotation Guidelines for Rhetorical Structure Manuscript. University of Potsdam and Simon Fraser University, Potsdam (2017) 36. Trofimova, I.: Observer bias: an interaction of temperament traits with biases in the semantic perception of lexical material. PLoS ONE 9(1), e85677 (2014). https://doi.org/10.1371/jou rnal.pone.0085677 37. Wardhaugh, R.: An Introduction to Sociolinguistics, 2nd edn. Blackwell, Oxford, UK (1992) 38. Williams, J.D., Asadi, K., Zweig, G.: Hybrid code networks: practical and efficient end-toend dialog control with supervised and reinforcement learning. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, July 30-August 4 2017, pp. 665–677. Association for Computational Linguistics (ACL) (2017) 39. Wilson, M., Wilson, T.P.: An oscillator model of the timing of turn taking. Psychon. Bull. Rev. 12(6), 957–968 (2005) 40. Yu, Z., Yu, Z., Aoyama, H., Ozeki, M., Nakamura, Y.: Capture, recognition, and visualization of human semantic interactions in meetings. In: Proceedings of PerCom, Mannheim, Germany, 2010 (2010) 41. Zeldes, A.: rstWeb - a browser-based annotation interface for rhetorical structure theory and discourse relations. In: Proceedings of NAACL-HLT 2016 System Demonstrations. San Diego, CA, pp. 1–5 (2016). http://aclweb.org/anthology/N/N16/N16-3001.pdf

A Study on Universal Design of Musical Performance System Sachiko Deguchi(B) Kindai University, Higashi-Hiroshima Hiroshima 739-2116, Japan [email protected]

Abstract. This paper describes the development of a new UI of musical performance system based on the results of workshops in a care home, and describes the extension of the UI for non-Western music. The new UI has only 8 strings which are numbered or colored, and they can be tuned for most major or minor scales so that users do not need to use sharp/flat. A new function was added to the score display system to transpose keys in order to generate the scores which can be used for the new UI. The new UI was evaluated by an experiment, and the result indicates that the new UI is easier than keyboard for the people who have little musical experience. Also, the two notations of duration of scores were evaluated by an experiment to know which is easy for those people. The score DB is enhanced based on the result of workshops in a care home. The new systems (string UI of musical performance system and score display system) were developed for elderly people and people with little musical experience. Then, the string UI was extended so that users can choose a number of strings and each string can be any pitch. By using this function, some UIs of non-Western musical instrument can be implemented. Keywords: Numbered notation · Colored notation · UI for elderly people · UI of non-western music · Musical scale · Tuning of strings

1 Introduction Music therapy is commonly used to improve the quality of life of elderly people [1, 2]. It is difficult to use musical instruments for elderly people who have little experience of musical performance. Some instruments have been proposed and used for elderly people along with singing [3], however, people could play chords but they do not play melody on the instruments. Our aim is to provide a system on which people can play melody. Our previous research provided the musical performance system and the scores for the people who were not familiar with staff notation scores [4, 5]. Many musical performance systems have been proposed [6–9], however, the notation of scores have not been discussed enough. While, numbered notation scores are sometimes used for elderly people, children and beginners, and colored notation scores are exceptionally used for children, however, scientific and technological discussions are not enough. The aim of this research is to improve user interfaces of our musical performance system and © Springer Nature Switzerland AG 2021 M. Kurosu (Ed.): HCII 2021, LNCS 12764, pp. 18–33, 2021. https://doi.org/10.1007/978-3-030-78468-3_2

A Study on Universal Design of Musical Performance System

19

to enhance score display system. Also, the aim of this research is to use the performance system and score display system in some genres of non-Western music. The following results were presented in HCII2019 [4]. (1) We developed a musical performance system on tablet PC. The system had several UIs (with note names, numbers, colors or shapes on keyboards, or without symbols on keyboards). (2) We proposed several musical notations and developed a score database. (3) We evaluated the UIs and scores and found that the numbered notation was most useful for the people who were not familiar with staff notation scores. Also, we found that colored notation would be useful for some people.

2 Utilization of UI and Scores of Numbered Notation 2.1 Utilization in 2018 Method. We had an extension course about musical performance at our university in Dec., 2018. We used commercial electric pianos (32 keys: 19 white keys and 13 black keys) and put the labels of numbers on the keys, and we used numbered notation scores. 28 people over 50 years old attended the course (Fifties: 1, Sixties: 17, Seventies: 9, Eighties: 1). Numbered notation scores of 13 Japanese children’s songs, 9 English children’s songs and 3 pieces of classical music were used in the course. An instructor explained the scores and participants practiced the keyboard using the scores. The participants also sang the songs. Total time of explanation was around 30 min, and total time of practice was around 70 min. At the end of the course, the participants answered the questions by rating 4, 3, 2 or 1 (4:good, 3:a little good, 2:a little bad, 1:bad). Questions are as follows. Q1: Are the scores easy to understand? Q2: Is the keyboard easy to play? Q3: Is it easy to play using the scores? Q4: Is it easy to play and sing at the same time?

Table 1. Mean values of evaluation by participants of extension course. Q1 Sixties

Q2

Q3

Q4

3.94 3.71 3.71 3.53

Seventies 4.00 3.67 3.78 3.44 Eighties

3

3

1

2

Result and Discussion. The mean values of questions answered by people over 60 years old are shown in Table 1. A person in his eighties wrote in the questionnaire that he had

20

S. Deguchi

difficulties in using the keyboard, while people in their seventies played well and did not wrote about the difficulties. Therefore, we understood that we should study the usage for aged people.

2.2 Utilization in 2019 Method. We used the same electric pianos used in the extension course in 2018 at three workshops in a care home in Oct. and Nov., 2019. The number of participants and the number of people who agreed to answer the questions at each workshop are as follows. 7 people attended all three workshops and answered the questions and 3 people attended two workshops and answered the questions. – First workshop: 13 (attended), 10 (answered the questions) – Second workshop: 12 (attended), 11 (answered the questions) – Third workshop: 10 (attended), 9 (answered the questions) The ages and the numbers of the participants who answered the questions are as follows. First workshop: 75–79: 1, 80–84: 2, 85–89: 6, 90–94: 1 Second workshop: 75–79: 1, 80–84: 1, 85–89: 5, 90–94: 2, 95–99: 2 Third workshop: 75–79: 1, 80–84: 1, 85–89: 4, 90–94: 2, 95–99: 1 Numbered notation scores of 12 Japanese children’s songs and 1 English children’s song were used in the workshops. An instructor explained the scores and participants practiced the keyboard by using the scores. The participants practiced at their own pace. The participants also sang the songs. Total time of explanation was around 15 min, and total time of practice was around 30 min in each workshop. In the first workshop, participants practiced basic songs individually. In the second workshop, participants practiced basic songs again and other songs. Also, they practiced a song together. In the third workshop, participants learned several functions of electric keyboard and practiced some songs. At the end of each workshop, the participants answered the questions. Questions are as follows in each workshop. First workshop Q1: Are the scores easy to understand? Choose 4, 3, 2 or 1 (4:good, 3:a little good, 2:a little bad, 1: bad). Q2: Is the keyboard easy to play? Choose 4, 3, 2 or 1. Q3: Is it easy to play using the scores? Choose 4, 3, 2 or 1.

A Study on Universal Design of Musical Performance System

21

Q4: Is it easy to play and sing at the same time? Choose 4, 3, 2 or 1. Q5: Did you enjoy this workshop? Choose 7, 6, 5, 4, 3, 2 or 1 (7:very good, 6:good, 5:a little good, 4: neither good nor bad, 3:a little bad, 2:bad, 1:very bad). Second workshop Q1-Q5 are the same as Q1–Q5 of first workshop. Q6: Is it good for you to play with other people? Third workshop Q3–Q5 are the same as Q3–Q5 of second workshop. Q7: Is it good to change tone colors? Choose 4, 3, 2 or 1. Q8: Is it good to use the function of accompaniment? Choose 4, 3, 2 or 1. Q9: Is it good to listen to the songs stored in the keyboard? Choose 4, 3, 2 or 1. Q10: Is it good to use percussion button? Choose 4, 3, 2 or 1. Result and Discussion. The mean values of questions Q1–Q6 are shown in Table 2. In the third workshop, some people could not answer all questions because they could not try all functions of keyboard or could not practice well because of the time limit. The mean values of Q3 in the 1st and 2nd workshops are 3.60 and 3.55 (max is 4), therefore, the participants could play the keyboard using numbered scores if they practice at their own pace. While, the mean values of Q4 in the 1st and 2nd workshops are relatively low. It would be difficult for aged people to play and sing at the same time. The mean value of Q5 is around 6 (max is 7) in each workshop, therefore, we think the participants almost enjoyed the workshops. The mean values of questions Q7–Q10 are as follows. Q7: 3.63, Q8: 3.50, Q9: 3.50, Q10: 3.29 The result indicates that participants were interested in changing tone colors, playing with accompaniment and listening to the music. Therefore, we should implement these functions in our system. In these workshops, we also found the followings: (a) People took time to play notes beyond one octave. (b) People took time to play sharp or flat notes. (c) People remembered children’s songs well. We decided to develop a new system based on these findings.

3 New User Interface 3.1 Development of Basic System In our previous research, we developed several UIs on tablet PC [4]. We used HTML, CSS and JavaScript for implementing UIs. In this research we developed a simple UI

22

S. Deguchi Table 2. Mean values of evaluation by participants of workshops at a care home.

1st WS 2nd WS 3rd WS

Q1 3.90 3.82

Q2 3.80 3.64

Q3 3.60 3.55 3.25

Q4 3.30 3.09 3.50

Q5 5.90 5.91 6.00

Q6 3.55

based on the results of workshops at a care home in 2019. We found that “People took time to play notes beyond one octave”, therefore, we provide a UI of 8 notes, because, many simple songs can be played within 8 notes (one octave and 1 note). This UI has eight strings which correspond 7 notes in one octave and 1 note next to the octave. e.g., {C4, D4, E4, F4, G4, A4, B4, C5} in C major. The numbers (1, 2,… 7) are written at the strings. This UI was designed referring to a shape of Lyre of ancient Greek music [10]. A user can choose the UI of colored strings. Figure 1 shows examples. The UI with colored strings without numbers are also provided. We also found that “People took time to play sharp notes or flat notes” at the workshops, therefore, we provide a function to change keys. Most simple songs use notes on the scale, e.g., in G major, {G, A, B, C, D, E, F#} are used. If we play a song in G major on the keyboard, we have to use black key next to F for F#. While, if we play a song in G major on the string instrument which is tuned for G major, we don’t have to use any black key. In our system, the pitches of strings can be transposed to most major keys and minor keys. Also, we provide two ways to transpose keys: (1) The first note is a keynote, or (2) The first note is always C4. E.g., in G major, a user can choose (1) The strings are tuned for {G4, A4, B4, C5, D5, E5, F#5, G5}, or (2) The strings are tuned for {C4, D4, E4, F#4, G4, A4, B4, C5}. In both cases, the sequence of string numbers is always {1, 2, 3, 4, 5, 6, 7, 1’}, or the sequence of string colors is always {red, orange, yellow, green, light-blue, blue, purple}. I.e., 1/red means the first string, 2/orange means the second string, and so on. A number/color does not mean pitch, but it means string number/color. Since we use numbers or colors on strings, any pitches can be assigned to strings and we do not have to use sharp/flat to play the UI. To avoid using sharp/flat, we could transpose an original key to C major or A minor, however, the range of pitches is changed and it would be inconvenient to sing the song. It is important that we can use any key and that we can play without sharp/flat. 3.2 Outline and Method of Evaluation Experiment An experiment to evaluate the new UI was carried out in 2020. Examinees were students because we could not make an experiment in a care home in 2020. This section describes the outline and method of experiment. The aim of this experiment is to compare two UIs: UI of keyboard (using black keys) and UI of strings (tuned for a scale). We call the former UI-1, and the latter UI-2. UI-1 is shown in Fig. 2, and UI-2 used in pre-experiment is shown in Fig. 1 (left). UI-2 was modified after pre-experiment, which is described in Sect. 3.3.

A Study on Universal Design of Musical Performance System

23

Fig. 1. A simple UI of strings with numbers (left, UI-2 in the pre-experiment) and a UI with numbers and colors (right).

Parts of scores used in the experiment are shown in Fig. 3. The pitches are notated as numbers (1–7) and +/- symbols are used for sharp/flat. The duration is notated as the length of space in these scores. The melodies used in the experiment are generated as follows. – First, the intervals of each notes on the scale are determined, e.g., (1 1 −1 2 1 −2 1…) in Score-1 (used for UI-1) and (2 −1 1 1 −2 1 2…) in Score-2 (used for UI-2). Both melodies are similar. – Next, pitches of notes are determined based on the intervals, e.g. (2 3 4+ 3 5+ 6 4+ 5+…) in Score-1 and (1 3 2 3 4 2 3 5…) in Score-2. – Quarter notes and eighth notes are used in these scores. – The melody length is 8 bars in each score. The procedure of the experiment is as follows. Experiment 1: – Examinees play UI-1 twice using Score-1 (including 3 sharps, A major). – Examinees answer the questions. Experiment 2: – Examinees play UI-2 twice using Score-2 (UI is tuned for A major) – Examinees answer the questions.

24

S. Deguchi

Fig. 2. A simple UI of keyboard with numbers (UI-1 in the experiment).

The questions are as follows. Q1: Is the UI easy to understand? Choose 5, 4, 3, 2, 1, 0 (5:very good, 4:good, 3:a little good, 2:a little bad, 1:bad, 0:very bad). Q2: Is the Score easy to understand? Choose 5, 4, 3, 2, 1, 0. Q3: Is it easy to play using the scores? Choose 5, 4, 3, 2, 1, 0.

Fig. 3. A part of numbered notation score in A major used for UI of keyboard (above, Score-1 in the experiment) and for UI of strings (below, Score-2 in the experiment).

3.3 Pre-experiment and Modification of UI Method. We made a preliminary experiment. Examinees were five lab students (male, age: 22–24). Experiment 1 and Experiment 2 were carried out as described in Sect. 3.2. Result and Discussion. After the experiment, examinees said as follows. – UI-2 (Fig. 1 left) was dark and it was difficult to read the numbers. – The strings of UI-2 were thin, therefore, they felt uneasy when they played the system. – They recognized the merit of UI-2 because they did not have to think about sharp (black key), however, the design of UI-1 (keyboard) was better. Modification of UI. We decided to modify the design of UI-2 based on the discussion of pre-experiment. Figure 4 shows the new design of UI-2.

A Study on Universal Design of Musical Performance System

25

Fig. 4. A simple UI of strings with numbers, which was designed based on the result of preexperiment (UI-2 in the experiment).

3.4 Evaluation Experiment Method. The evaluation experiments were carried out several times in Dec. 2020. Total number of examinees were 33 (Male Students, Age 19–25). 31 Examinees had no experience with keyboard instrument. Experiment 1 and Experiment 2 were carried out as described in Sect. 3.2. UI-1 (Fig. 2) and the new design of UI-2 (Fig. 4) were used. score-1 (Fig. 3 above) and Score-2 (Fig. 3 below) Were used for UI-1 and UI-2. Result and Discussion. The mean values of questions are shown in Table 3. The mean value of each question for UI-2 is higher than that for UI-1. Table 3. Mean values of evaluation of UI-1 and UI-2. UI-1 UI-2 Q1 3.94 4.42 Q2 3.39 4.18 Q3 3.18 4.15

Paired sample t-test is used for the comparison of the mean values of each question for two UIs. The degrees of freedom is 32, and the critical value for significance level of 0.05 (two-tailed test) is 2.0369 and that of 0.01 is 2.7385. T-ratios of the comparisons are as follows. Q1: -4.50 Q2: -7.54 Q3: -6.07 Therefore, there is a significant difference between the mean values of each question for UI-1 and UI-2. This result indicate that UI-2 would be easier than UI-1 to play the system. Because UI-2 can be tuned for any keys, we do not see any sharp/flat in scores. While, when we use UI-1, we have to use black keys corresponding to sharp/flat in scores. We also asked the examinees: Which do you think is easier to play, UI-1 or UI-2? The numbers of examinees who answered UI-1, UI-2 or “Almost the Same” are as follows. UI-1: 1 UI-2: 25 Almost the Same: 7 This result also shows that UI-2 would be easier to play for the people who have little experience with keyboard instrument.

26

S. Deguchi

The black keys on the keyboard are asymmetric, and this layout is helpful for a user to recognize the location where she/he is playing. One examinee pointed out that there should be some mark or color on the string UI so that he could recognize the location. 3.5 Development of Application System Three Octave Version. We have extended the UI of strings (Fig. 1) for normal use. The system has 22 strings (for three octaves) and has scroll function. Figure 5 shows an example. The strings can be tuned for most major scales and minor scales. Users can choose normal size or wide size of strings. This UI also has some functions: recording user’s performance, playing the recorded performance, and playing music using scores in score database. These functions could support the practices. Since we made sounds of 5 octaves, the UI of 5 octave version can be implemented if needed.

Fig. 5. A 3-octave UI of strings.

Any String Number and Any Pitches. For non-Western music, we are now developing a system in which users can choose any number (9–21) of strings and any pitches. E.g., UI of koto (Japanese harp) can be designed by choosing 13 strings and assigning the following pitches {D4, G3, A3, A#3, D4, D#4, G4, A4, A#4, D5, D#5, G5, A5} to the strings. People can use this function to customize the UI to their musical instrument and can play the UI using original scores. Also, composers can use this function to compose music in some musical genre, even if they are not familiar with the genre, e.g., a composer usually working on popular music could compose a piece of koto music using the UI of koto. The numbered or colored UI is a universal design. The merits of using numbered or colored UI and scores instead of keyboard and staff notation scores are as follows. – In previous research, we showed that it was easy to play numbered or colored UI with numbered or colored scores for the people who had little musical experience. The notation using note names would be also easy to play, however, note names would not be useful when people play and sing at the same time. – In this research, we showed that numbered or colored UI enabled us to play musical performance system without treating sharp/flat by tuning strings and using corresponding scores.

A Study on Universal Design of Musical Performance System

27

This UI is also a cross-cultural design. Major or minor scales are commonly used in Western music and popular music in many countries today. However, 7 scales are theoretically possible and it is said that 6 scales were used in medieval music. Also, several genres of traditional music in the world use the scales other than major or minor scale. E.g., koto music uses different scale [11]. The sequences of intervals of major, minor, and koto scales are as follows, where, “w” means whole tone and “s” means semitone. Major scale: w w s w w w s Minor scale: w s w w s w w Koto scale: s w w w s w w Therefore, it is important to provide the functions to implement scales other than major or minor scales. Also, tuning is usually complicated in traditional music. E.g., in koto music, there are several tunings, and 5 notes in the scale are assigned to strings and other 2 notes in the scale are played by pushing the strings (to increase the pitches). Therefore, the function to assign a pitch to each string is necessary along with some functions for playing methods. In previous research, we provided musical notations for people with little musical experience based on koto scores, and used those notations for keyboard UI (with numbers, colors, or note names). In this research, we first developed a new UI for elderly people and people with little musical experience based on the design of ancient Greek Lyre. Then we noticed that this UI can be extended to non-Western music. It is interesting that a universal design would be a cross-cultural design and vice versa. 3.6 Sounds of Musical Performance System As described in Sect. 2.2, people are interested in changing tone colors. This musical performance system provides the sounds (the pitches of 5 octaves) of piano and two electric pianos. We also made other sounds, which should be improved before being used in the system. Usually DTM systems use MIDI sounds, but it is slow to use MIDI sounds on tablet PC. Therefore, we developed a program using C language to generate WAVE data based on the additive synthesis. In previous system, we generated data of 44.1 kHz sampling and 16-bit length. Since the data size was big, we evaluated several sampling rates and data lengths. Then, we decided to use 10 kHz sampling and 16-bit length, therefore, the data size became 1/4. Because of human auditory property, most speakers enhance the power of low frequency sounds. However, speakers of tablet PC which we are using do not implement this function. We implemented the function to enhance low frequency sounds by ourselves, and users can choose enhanced sounds or original sounds.

4 Score Display System 4.1 Improvement of the System and Evaluation of Two Notations of Duration Staff notation is widely used in Western music today. It is the standard to notate music, and it is especially important for complex music. Also, it could be used as a support tool

28

S. Deguchi

for performers. When performers play musical instrument, the pitches and the location of fingers are important for performers and staff notation is convenient because we can memorize and recognize the melody by glancing at notes on five lines. However, in some musical genres, some types of tablature are used. E.g., guitar music uses tablature scores. Koto music uses numbered notation scores and each number corresponds to each string of koto instrument. We developed score display system in 2018 [4] and improved the system in 2019 and 2020. This system can display scores of 2/4, 3/4 and 4/4 time signatures. This score display system can generate four kinds of notations about pitches (numbers, note names, note names in Japanese, and colors), and can generate two types of notations about duration (space length and symbol). Figure 6 shows example scores: two notations of pitches and one notation of duration. We already compared the notations about pitches (numbers, note names, colors, and staff notation) in 2018. In 2018, the score display system was under development, therefore, we used the scores which were made manually by using Excel. While, in 2020, we compared the notations about duration using the scores generated by our system.

Fig. 6. A score of numbered notation (above) and a score of colored notation (below).

Method. The evaluation experiments of scores were carried out several times after the experiments of UIs. Total number of examinees were 33 (Male Students, Age 19--25). The aim of the experiment is to compare two scores: Type-1 score represents the duration using space length, while, Type-2 score represents the duration using symbol. Type-1 score was designed based on the design of Ikuta-school koto score [12], while, Type-2 score was designed based on the design of Yamada-school koto score [13]. Figure 7 shows an example of each score. In Type-1 score of Fig. 7 (above), the first note is quarter note and the second note is 8th note. A user can know the duration of each note by the length of space where the note is placed. While, in Type-2 score of Fig. 7 (below), the first note is quarter note and the second note is 16th note. A quarter note has no symbol, an 8th note has an underline, and a 16th note has a double underline. We call the Type-1 score used in the experiment Score-3, and call the Type-2 score used in the experiment Score-4. The melodies used in the experiment are generated as described in the experiment of UIs. In the scores of this experiment, 16th notes are used along with quarter notes and 8th notes.

A Study on Universal Design of Musical Performance System

29

In this experiment, UI-2 (Fig. 4) was used. The procedure of the experiment is as follows. – After Experiment 1 and Experiment 2 to evaluate two UIs, examinees practiced Type-2 score twice (the same melody of Type-1 score used in Experiment 2). – After the practices, Experiment 3 and Experiment 4 were carried out as follows. Experiment 3–1: – Examinees played UI-2 once using Score-3 (Type-1). Experiment 4–1: – Examinees played UI-2 once using Score-4 (Type-2). Experiment 3–2: – Examinees played UI-2 once again using Score-3 (Type-1). – Examinees answered the questions. Experiment 4–2: – Examinees played UI-2 once again using Score-4 (Type-2). – Examinees answered the questions. Examinees used Score-3 and Score-4 alternately because they had to get used to 16th notes before they evaluated the scores.

Fig. 7. A part of numbered notation score of Type-1 (above, Score-3 in the experiment) and Type-2 (below, Score-4 in the experiment).

The questions are as follows. Q1: Is the Score easy to understand? Choose 5, 4, 3, 2, 1, 0 (5:very good, 4:good, 3:a little good, 2:a little bad, 1:bad, 0:very bad). Q2: Is it easy to play using the scores? Choose 5, 4, 3, 2, 1, 0.

30

S. Deguchi

Result and Discussion. The mean values of questions are shown in Table 4. The mean value of each question for score-3 (Type-1) is slightly higher than that of score-4 (type-2). Paired sample t-test is used for the comparison of the mean values of each question for two scores. The degrees of freedom is 32, and the critical value for significance level of 0.05 (Two-Tailed Test) is 2.0369 and that of 0.01 is 2.7385. T-ratios of the comparisons are as follows. Q1: 3.32 Q2: 2.20 Therefore, there is a significant difference (level of 0.05) between the mean values of each question for Type-1 score and Type-2 score. This result indicate that Type-1 score would be easier than Type-2 score for the people who have little experience of musical performance. Table 4. Mean values of evaluation of Type-1 score and Type-2 score. Score-3 (Type-1) Score-4 (Type-2) Q1

3.73

3.00

Q2

3.39

3.00

We also asked the examinees: Which do you think is easier to play, Score-3 (Type-1, using space length) or Score-4 (Type-2, using symbol)? The numbers of examinees who answered Type-1, Type-2 or “Almost the Same” are as follows. Type-1: 21 Type-2: 8 Almost the Same: 4 This result also shows that Type-1 score would be easier to play. However, 8 people preferred Type-2 score. If people practice Type-2 scores well, they would understand the symbols of duration well and it could be easier to use Type-2 scores. Also, the scores similar to Type-2 scores are used in Taisho-goto [14] in Japan. Therefore, we think we should provide both Type-1 and Type-2 scores. 4.2 Extension of the System for Transposition We have added a new function to the score display system. This function transposes original key to any key specified by a user. This function is useful when a user wants to change the key to sing a song. A user can choose a number from -12 to 12 in the menu on the score display system, then the key is transposed. In this function, 1 means semitone (half step) and 2 means whole tone (whole step). E.g., if a user chooses 7, C major is transposed to G major, i.e., the note numbers {1, 2, 3, 4, 5, 6, 7} on the score are changed to {5, 6, 7, 1’, 2’, 3’, 4+’}. The key of a song is not always specified in the score data, and the system could not determine the key, therefore, this system changes note numbers and put + symbol for sharp automatically. A user should choose + (sharp) or - (flat) by her/himself. E.g., if a user choose 5 in the transposition menu (from C major to F major), the note numbers {1, 2, 3, 4, 5, 6, 7} are changed to {4, 5, 6, 6+, 1’, 2’, 3’} by the system, then a user can choose flat in the menu, and the numbers are changed to {4, 5, 6, 7-, 1’, 2’, 3’}.

A Study on Universal Design of Musical Performance System

31

As described in Sect. 3.1, the strings of new UI are tuned for the scale specified by a user. Therefore, users do not need to know the pitches of the strings. E.g., in G major, the strings are tuned for (1) {G4, A4, B4, C5, D5, E5, F#5, G5} or (2) {C4, D4, E4, F#4, G4, A4, B4, C5}, but users only see the string numbers {1, 2, 3, 4, 5, 6, 7}. Therefore, the corresponding scores should be provided. In the former case (1), users need scores which are transposed from G to C, because in G major, the numbers {1, 2, 3, 4+, 5, 6, 7} are used in the scores of keyboard UI and {5, 6, 7, 1’, 2’, 3’, 4+’} should be changed to {1, 2, 3, 4, 5, 6, 7} for the string UI. For the latter case (2), another new function has been added. In G major, the numbers {1, 2, 3, 4+, 5, 6, 7} are used in the scores of keyboard UI, while, the numbers {1, 2, 3, 4, 5, 6, 7} should be used in the scores of the string UI whose pitches are {C4, D4, E4, F#4, G4, A4, B4, C5}. Therefore, the score display system provides the function to delete sharp/flat from the original scores. 4.3 Score DB The results of workshops in a care home showed that elderly people remembered children’s songs well. Therefore, we decided to enhance score DB. We made score DB of 17 Japanese children’s songs and 11 English children’s songs in Humdrum format before. In this research, we encoded these songs in our original form as described below. We are adding some songs whose copyrights were expired. Especially, we should add scores within 8 notes. Also, we will add some children’s songs in the world. MusicXML [15] is commonly used for data exchange today. E.g., MuseScore (free software to generate staff notation scores) [16] uses original data format, but it can export data in MusicXML format. Several music encoding methods have been proposed and used. MEI [17] is used by researchers of Western music today. Humdrum [18] has been used for a long time and it is a simple encoding method. We had been using Humdrum format, however, we decided to use our original format in this research. Since we need to edit text files, we use a simple and readable format as Humdrum. The following lines show examples of our format and Humdrum (**kern, **text). Our format:A4 0 8 litHumdrum (**kern, **text):8a litIn Humdrum (**kern), each note is encoded, where a number represents duration and an alphabet represents pitch. E.g., 8a means 8th note of pitch A4, 4cc means quarter note of pitch C5, 16f# means 16th note of pitch F#4, and so on. While, in each line of our format, the first element means pitch, the second element means sharp/flat (1:sharp, −1:flat, 0:without sharp/flat), the third element means duration, and the fourth element means syllable of lyric. We have developed a program by using python, which can convert MusicXML format (which was generated by MuseScore) to our data format. We are adding scores to DB using MuseScore and our converting program. This program cannot convert all MusicXML data which is available on the internet. We have also developed a program which can convert Humdrum (**kern) format to our data format. Therefore, we could convert huge resources of copyright free music (mainly classical music) [19] to our format.

32

S. Deguchi

5 Conclusions and Future Work We developed a new UI of musical performance system based on the results of the workshops in a care home. The new UI has only 8 strings and the strings can be tuned for most major or minor scales so that users do not need to use sharp/flat. The strings are numbered or colored. The evaluation experiments showed that the string UI without sharp/flat keys was easier than the normal keyboard with sharp/flat keys for the people with little musical experience. We also developed 3-octave version of string UI. We extended the system in which a user can choose string number and pitches for the strings, therefore, a user can use this system for non-Western music. We also compared the notations of duration: using space lengths, or using symbols. The result showed that both notations are usable. We added a new function to the score display system to transpose keys. This function provides scores for the string UI. We are enhancing score DB based on the result of the workshops in a care home. In this research, we showed that numbered or colored UI enabled us to play musical performance system without treating sharp/flat by tuning strings and using corresponding scores. We developed a new UI for elderly people and people with little musical experience based on the design of ancient Greek Lyre. Then we noticed that this UI can be extended to non-Western music. A universal design would be a cross-cultural design and vice versa. Future work includes the followings. As for the UI of musical performance system, we will develop vertical layout of UI, sharp function for koto UI, some functions to represent playing methods for the string UI, and a function to play with accompaniment. We would add harp sounds and some sounds of playing methods such as vibrato. The template of 6/8 time signature will be added to the score display system. Children’s songs in the world should be added to the score DB. Also, the new simple UI should be evaluated by elderly people. The extended version of UI for non-Western music should be evaluated by the people who are familiar with some genre of non-Western music and by the composers who are not familiar with the genre. Acknowledgments. The author would like to thank the manager of care home, the care staff and the coordinator of Kindai University for supporting the workshops of musical performance. The author would also like to thank N. Kimura, Y. Sugimoto, K. Sasaki, R. Takeuchi and S. Kaneda for their contribution to the development of the musical performance system.

References 1. Ridder, H.M., Wheeler, B.L.: Music Therapy for Older Adults: Music Therapy Handbook. The Guilford Press, New York (2016) 2. Davis, W.B.: Music Therapy and Elderly Populations: An Introduction to Music Therapy, 3rd edn. The American Music Therapy Association, Maryland (2008) 3. Götell, E., Brown, S., Ekman, S.L.: Caregiver-assisted music events in psychogeriatric care. J. Psychiatr. Ment. Health Nurs. 7, 119–125 (2000) 4. Deguchi, S.: Multiple representations of the UI, score and scale for musical performance system and score DB. In: Kurosu, M. (ed.) HCII 2019. LNCS, vol. 11568, pp. 177–191. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22636-7_12

A Study on Universal Design of Musical Performance System

33

5. Deguchi, S.: A Study on the UI of musical performance system and score representation. In: AAAI 2018 Spring Symposium Series Technical Report, pp. 207–211 (2018) 6. Zbyszynski, M. et al.: Ten years of tablet musical inter-faces at CNMAT. In: Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 100–105. NIME (2007) 7. Hochenbaum, J., et al.: Designing expressive musical interfaces for tabletop surfaces. In: Proceedings of the International Conference on NIME, pp. 315–318. NIME (2010) 8. Oh, J., et al.: Evolving the mobile phone orchestra. In: Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 82–87. NIME (2010) 9. Brown, D., Nash, C., Mitchell, T.: A User experience review of music interaction evaluations. In: Proceedings of the International Conference on NIME, pp. 370–375. NIME (2017) 10. Mathiesen, T.J.: Apollo’s Lyre: Greek Music and Music Theory in Antiquity and the Middle Ages, ACLS Humanities E-Book (2010) 11. Deguchi, S., Selfridge-Field, E., Shirai, K.: The temperament, scale and mode of Koto music, In: Proceedings of International Congress of Musicological Society of Japan 2002, pp. 434– 438 (2002) 12. Miyagi, M.: Rokudan no Shirabe Koto Score of Ikuta School. Hogakusha, Tokyo (2005) 13. Nakanoshima, K.: Rokudan no Shirabe. Koto Score of Yamada School, Hogakusha, Tokyo (2008) 14. Hirano, K., et al.: Nihon Ongaku Daijiten (In Japanese). Heibonsha, Tokyo (1989) 15. MusicXML: https://www.musicxml.com/. Accessed 23 Jan 2021 16. MuseScore: https://musescore.com/. Accessed 23 Jan 2021 17. MEI: https://music-encoding.org/. Accessed 23 Jan 2021 18. Huron, D.: Humdrum Toolkit. https://www.humdrum.org/. Accessed 23 Jan 2021 19. CCARH: Kern Scores. http://kern.ccarh.org/. Accessed 23 Jan 2021

Developing a Knowledge-Based System for Lean Communications Between Designers and Clients Yu-Hsiu Hung and Jia-Bao Liang(B) Department of Industrial Design, National Cheng Kung University, Tainan, Taiwan [email protected]

Abstract. Design is all about communication. Clients often have a difficult time expressing their needs to designers through the proper channels, leaving many designers struggling to meet the demands placed on them. The objective of this study was to improve designer-client communication by looking at the communication process with Lean Thinking, finding waste and eliminating it, and creating a knowledge base containing comprehensive product attributes that can be targeted to help designers and clients simplify the communication. An experiment was conducted to develop the children’s electric toothbrush knowledge base with four main stages. The first stage was analyzing the attributes of toothbrush from several aspects includes function, behavior and structure. The second stage was surveying 10 designers by questionnaire and the results were analyzed qualitatively and quantitatively. The third stage was to import individual cases and filter attribute elements to establish mapping relationships. The fourth stage was the use of Django project structure and presentation of knowledge base. Based on the findings of this study, an augmented children’s electric toothbrush knowledge base using the Kansei retrieval method was developed, the knowledge base contains comprehensive product attribute items and cases, which can be used by both clients and designers to improve the designer-client communication. Keywords: Designer-client communication · Product knowledge base · Lean communication

1 Introduction Design and communication are closely related. Maier, Eckert [1] have reviewed research on models of design communication and communication problems and divided them into mechanical and systematic arguments: the former is based on the five elements of communication proposed by Shannon and Weaver [2], pointing out that the problems in communication are mainly noise in the communication channel and incomplete communication information, etc. The latter uses Luhmann [3] conceptual model of the three elements of communication (information, expression and understanding) as an example, pointing out that the problems in communication are mainly due to the different interpretations of information by the communication subjects, and that the key to the communication process is the recipient of the communication and the process of understanding the information. © Springer Nature Switzerland AG 2021 M. Kurosu (Ed.): HCII 2021, LNCS 12764, pp. 34–48, 2021. https://doi.org/10.1007/978-3-030-78468-3_3

Developing a Knowledge-Based System for Lean Communications

35

Designers always try to extract information from their communication with clients and turn it into a design language, however, there is a huge gap between designers and clients [4]. To solve the problem of information transfer during communication, Shen, Zhang [5] reviewed various models of communication between designers and clients, summarising the basic structure of these models and the eight key issues they cover: (i) Context and characteristics; (ii) Reflective representation; (iii) Interactive interpretation; (iv) Artefactual variation; (v) Mutual awareness; (vi) Consumer engagement; (vii) Collaborative production; (viii) Collective consumption. Therefore, the biggest problem in communication between designers and clients is that it is often difficult for clients to find appropriate channels to effectively communicate their needs to designers [6], thus making it difficult for many designers to meet their clients’ requirements. There has been much research into effective design communication, including effective communication between designers and designers, effective communication between designers and engineers, and effective communication between designers and customers. Graham, Wildes [7] suggest that communication with customers in design is an embodiment of user-centredness, related to the field of human-computer interaction (HCI), through such interaction, customers and designers can become more effective in communication (especially at the technical level). As customers and designers from different backgrounds need to communicate with each other for a common set of goals, they may have different perceptions of representations and convey different information, so effective communication plays a crucial role in the design process. Many researchers have used mathematical theories and models to analyse uncertain, incomplete or redundant user requirements in order to obtain the relationship between customer requirement sequences (e.g. fuzzy mathematics [8], QFD [9], statistical methods [10], and kano models [11], where the extraction or ranking of importance of communication information is considered, but there is no mention of how these rankings or extraction affects communication and how it affects design. In particular, with the exponential increase in the amount of information available to consumers today, it is worth investigating how to improve the efficiency of the organisation of recording and managing information in the design communication process to make two-way communication between designers and customers more efficient [12]. In the above-mentioned research on design communication, only one side (client side or designer side) of the information presentation problem is addressed, but not the twoway communication between designers and clients, nor does it examine the true cause of the problem in the process of finding communication and eliminating waste in the communication process to achieve efficiency. Lean thinking, as a method of improvement that has received more attention in recent years, has been shown to work well in many process improvements [13]. The documentation and management of information in processes is also an important part of lean thinking, as is the process of communication, which requires researchers to extract and manage effective information in the process of communication, so many studies have been conducted around lean communication, for example, Colazo [14] demonstrated that lean thinking can effectively change the effectiveness of communication in teams, even when faced with a lot of information and complex relationships. Most of these studies have looked at models and methods of lean communication in production organisations, and only a few have looked at the

36

Y.-H. Hung and J.-B. Liang

value of using lean communication in the product development process. For example, Ferreira and Barbosa [15] proposed Lean Communication-centred design and point out that, compared to traditional communication approaches, in each iteration of the solution’s details, integrating pieces of the communication model created at different stages of the process, and the Lean A3 record captures these different stages, so that the design of a lean communication is easier for designers to review and reflect on. However, with the advent of artificial intelligence and the era of big data, consumers’ consumption concepts and pursuits are constantly changing, so how can Lean Communication respond to this background, solve the stagnation and waste in the communication between the client and the designer, combine with a knowledge base to effectively manage the exponentially increasing complex design knowledge, methods and cases used in the product design process, so that designers and clients can communicate more effectively? It is worthwhile to further study and think about this in order to reduce the waiting time of customers in the design development process. The aim of this paper is to (i) examine the two-way communication process between designers and clients based on lean communication, analyse the waste in designer-client communication and identify opportunities for improvement (ii) propose a lean-based knowledge base that can simplify the communication between designers and clients to improve design efficiency in the context of the information age. This study has been designed to support the communication between designers and clients by building a framework model that combines lean communication and design communication in the information age, and by identifying and eliminating wastage in the analysis and identification of user requirements and communication in the design development process.

2 Literature Review 2.1 Problems in Design Communication Design communication is the process by which information is passed and agreed between the clients and the designers, either individually or as a group, in order to achieve design objectives. In the process of design communication, there are many obstacles that affect the effectiveness of communication. Maier collates research on design communication and divides the problems in design communication into two categories: (i) the mechanisms of communication [16], including the noise of the communication channel and the incompleteness of the communication information; and (ii) the systemic nature of communication [17], i.e. the different understandings of the information by the communication subjects and the key to the communication process is the receiver of the communication and the understanding of the information. Based on the above two types of problems, three aspects of design communication can be summarised that may affect the efficiency of design communication: (i) the communication medium - the effectiveness and accuracy of information transfer and reception; Raaphorst, Duchhart [18] suggest that by using visual methodology, visual representation is the main medium of communication between stakeholders in the design process.

Developing a Knowledge-Based System for Lean Communications

37

The authors propose the use of visual methodologies such as visual discourse analysis, iconographic content analysis and social symbolism analysis to investigate these representations at different stages of meaning formation and suggest that these research methods have the potential to address issues such as miscommunication between participants. (ii) the language of communication - the distortion of information; Judge, Randall [19] reviewed the literature to identify graphic symbols, tools, language or communication attributes that are currently available to support children’s communication and inform decision making for those working with children. and (iii) the organisation of communication - the problem of transferring information at various levels of the organisation. Pedó, Brandalise [20] point out that because of the large number of internal and external stakeholders involved in the management design, a complex organisational structure is needed, i.e. an information system to support collaboration and coordination. The authors therefore examine the use of VM tools in design management and present an ongoing visual management tool developed in collaboration with a UK infrastructure design company. 8 Most of the solutions or frameworks mentioned in the above study are either specialist in nature or require prior knowledge or learning to reach consensus on certain representations before they can be used, and to some extent lack usability, ease of use and timeliness. Alternatively, there are a number of complementary approaches that attempt to improve the efficiency of design communication, but the research is often conducted with the aim of improving efficiency, without examining the issue of waste in design communication and improving its efficiency by eliminating it. 2.2 Lean Communication Industry 4.0 has brought about a wave of digitisation, and if we do not eliminate waste but simply introduce technology, then it means autonomising waste as well, so we need clear thought to guide us to eliminate waste in communication. Lean thinking, as an improvement approach that has received more attention in recent years, has shown to work well in many process improvements (Soares & Teixeira, 2014), for example, Iuga (2017) studied the role and importance of communication and types of communication for organisations and their employees, and what has remained consistent is the purpose of communication: communication for all sizes, types and cultures of organisations are critical to effectiveness and success. The authors validate the effectiveness of lean and visual communication by establishing visual communication methods on the internal shop floor of an automobile and subsequently investigating the effectiveness of the implementation of these methods. Mostly models and methods of lean communication in production organisations have been explored, with only a small number of studies focusing on the value of applying lean communication in the product development process, e.g. Chang, Lee, and Huang (2018) found that newly formed interdisciplinary teams were slow and severely behind schedule when working on a new project with poor team communication and coordination. Ferreira and Barbosa (2016) propose a lean communication-centred design and show that, compared to traditional communication methods, the design process is more effective when In each iteration of a solution, the designer redefines the definition of the previous stage, integrating the pieces of the communication model created at different

38

Y.-H. Hung and J.-B. Liang

stages of the process, and the Lean A3 record captures the important details of these different stages, making the design under Lean Communication easier for designers to review and reflect on. However, with the changing needs of customers, design and even production work is becoming less and less repetitive, if the lead time for product development and design is reduced, thereby reducing the lead time for production and increasing customer value is a question worth considering. The effective management of the exponentially increasing amount of complex design knowledge, methodologies and case studies used in the product design process allows designers and customers to communicate more effectively, thereby reducing the waiting time for customers in the design development process, and is worthy of further study and consideration. 2.3 Communication Tools for the Designer-Client Although there are various types of product design processes, such as ‘KJ method’ [21], ‘brainstorming’ [22], ‘Design ethnography’ [23] or ‘concept mapping’ [24], they all treat the design process as usually starting from a briefing stage and ending with design sketches. Archer defines the design process as involving four interlinked phases: (1) problem analysis (2) solution synthesis (3) evaluation and (4) communication. The communication between different participants usually goes through the whole design process. Participants in a typical product design process usually include the designers, clients, and consultants from many other disciplines. During the briefing and design solution development process, the designer–client communication is intensive and significant. (Wu et al. 2013). With the development of technology, computer-based tools facilitating designer– client communication gradually appear. Knowledge base (KB) also comes into being. The emergence of the knowledge base has benefited from the development of Web technologies, with the purpose of optimizing the results of search engines and improving the quality and experience of clients’ searching. Since [22] Tim Bernes-Lee [25] invented the World Wide Web (Linked Information System) in 1989, the system has undergone a transition from hypertext links to semantic links. Later, the concept of the Semantic Web was introduced, emphasizing the use of semantic relationships to describe resources and data in the World Wide Web. Semantics refers to the rich meaning of the data itself. The Web links these data to form huge network information. In this development process, artificial intelligence scholars introduced “ontology” to characterize knowledge, defined Linked Data, and proposed a large number of Knowledge Representation (KR) methods. These unified forms of knowledge constitute a knowledge base (KB). When building a knowledge base, it actually build a few basic components, including extracting concepts, instances, attributes, and relationships. From the way of construction, the construction of the knowledge base can be divided into manual construction and automatic construction [26]. Buchanan and Shortliffe [27] use expert knowledge to define computer programs that are able to solve complex problems. There are many ways to build a knowledge base by manual work. For example, by interviewing and transcript analysing to obtain case(Hart 1986); Manual construction relies on expert knowledge to write certain rules, collect relevant knowledge information from different sources, and build knowledge architecture [28]; by designing common fuzzy input space according

Developing a Knowledge-Based System for Lean Communications

39

to the expert knowledge [29]; by reading documents and visiting in the field to obtain cases [30]. Automated build is based on knowledge engineering, machine learning, artificial intelligence and other theories automatically collect and extract concepts, instances, attributes and relationships from the Internet [31, 32].Over the last decade, the construction of the knowledge base by automated work moving from pattern matching [33] and rule-based systems [34] to systems that use machine learning [35–37], DeepDive [38, 39], statistical inference [40] and so on [41, 42]. With the advent of the era of big data, more and more algorithms and data mining research are being carried out in the field of knowledge base, but often such projects are time consuming and costly, and require a long-term preparation process and postmaintenance. While manual construction can reduce the pre-learning time of knowledge base construction, it requires the expert knowledge used by the knowledge base to accurately describe the attributes and relationships of objects. The aforementioned empirical KB research show the potential of KB in supporting product design. However, there are research gaps remaining. First, the object-oriented knowledge base was mostly unidirectional, but in the development process of product design, the communication between designers and clients are bidirectional. Second, most of the research focused on the theoretical and academic level but lacked the empirical research. These two gaps suggest that (i) The knowledge base can also be bidirectional. Taking the product knowledge base as an example, it can be used for both designers and clients. (ii) It is necessary to choose the right product and attribute organization strategy to verify that the product knowledge base construction method is feasible. The research questions are as follows: • Can KB be bidirectional for both the designer and the consumer during the construction, thereby eliminating the current waste in communication and effectively simplifying the communication between designer and client? • Is it possible to select a certain type of product and use the proposed construction method to construct the knowledge base, thereby confirming the mapping relationship of the internal attributes of knowledge?

3 Method of Product Knowledge Base Based on Expert Knowledge 3.1 Objective In order to examine the whole process of communication, reduce the waste between designers and clients’ communication, so as to improve the efficiency of design and client communication, and to solve the current situation of unidirectional construction of knowledge base and few empirical cases, this paper eliminates the wastage in the communication process before improving the communication efficiency, and avoids amplifying the waste when digitising the design communication knowledge base, and then adopts an expert knowledge and case reasoning The next step is to build a product knowledge base using a combined attribute organisation strategy. Helping customers to extract effective information accurately and efficiently from the vast amount of data,

40

Y.-H. Hung and J.-B. Liang

building a bridge between effective information, facilitating designers to accurately capture client needs, shortening the development and design cycle, and enabling consumers to search the knowledge base to find products that meet their needs. 3.2 Overview of the Method of Product Knowledge Base As shown in Fig. 1, it is a model flow chart of a product knowledge base construction design method. Firstly, look at the whole process of communication and use the principle of lean thinking as a guideline to find and eliminate waste and to find reasonable solutions to improve the efficiency of communication. Secondly, the client demands need to be cognized, acquired and expressed, obtain product background knowledge and basic conditions, extract the Kansei words related to the product, filter the Kansei words through questionnaires, and obtain the final Kansei words related to the product after statistics; Thirdly, the client demands are transformed, the expert knowledge is used to analyze the product to obtain the internal properties. Questionnaires and statistical analysis may be performed at this stage. Finally, the clinet demands are presented. Through the mapping relationship established by the expert knowledge of the previous part, the product knowledge base is built by means of programming language. The construction of the knowledge base follows the expert knowledge rules and is combined with case analysis to quickly build knowledge for designers and consumers in a short period of time.

requirements transformation

requirements presentation

Fig. 1. A brief model flow chart of design method for product knowledge base

3.3 The Specific Technical Route of This Construction Method As shown in Table 1, it is a detailed flow chart of a method for establishing a product knowledge base based on expert knowledge, and the specific steps are as follows: • Improve the efficiency of the process with Lean Thinking

Developing a Knowledge-Based System for Lean Communications

41

• By examining the designer-client communication process with Lean Thinking, we can see that there are many wastes, as summarised in the table below

Table 1. waste in communication processes and corresponding improvement measures. Waste in the communication process

Improvements

Rework due to misunderstanding of requirement information

Establishing a mapping between user needs and design requirements

Rework due to a gap in expertise between client, designer and technician

Establishing a mapping between user language, design language and technical language

Rework due to discrepancies between the designer’s presentation and practice

Presentations are made using the current product library and the basic solution is confirmed and then improved

• Cognition, acquisition and expression of clients demands. Through the literature reading, market research, interviews, observation and other methods to summarize the background knowledge and basic conditions of the product, the Kansei words are extracted for the first time. Design a questionnaire, select product clients or designers related to the product to conduct a survey, and screen out a number of (fixed number) of emotional vocabulary based on the results. • Transform user demands. Select appropriate expert knowledge as the basis for rule construction. These rules can accurately analyze the attribute knowledge of the information inside the knowledge base and establish an effective relationship between these attributes. Next, a questionnaire needs to be designed, respondents have to score the matching degree of a certain attribute and a certain Kansei words by using the 5-point Likert scale from1 = Strongly Disagree to 5 = Strongly Agree. Through the statistical results, the mapping relationship between Kansei words and attribute, and the mapping between attributes are established. In this way, the attribute relationship of the internal objects of the knowledge base can be constructed through expert knowledge. • Presenting client demands. Analyze the product case that needs to be stored in the knowledge base, and determine whether the product has a certain attribute. If the product case includes a certain attribute, a mapping relationship can be established between the Kansei words matching the attribute and the product case. Use the programming language to build the logical

42

Y.-H. Hung and J.-B. Liang

framework of the product knowledge base based on the analysis of the previous steps. Design and optimize the pages of the knowledge base so that clients can get specific products by searching for vague and Kansei words. This knowledge base can be aimed at both clients and designers. Clients can search for products that best meet their needs.

4 Mapping Construction of Children’s Electric Toothbrush Knowledge Base This article chooses FBS as the expert knowledge base and children’s electric toothbrushes as the case for knowledge base construction. In 1990, Gero first proposed the Function-Behavior-Structure (FBS) [43] for the product design process. This model can accurately describe the attributes and relationships of objects by establishing models and mappings of products. As an emerging fast-moving consumer goods, demands for children’s electric toothbrush are fuzzy and changes frequently, so it has certain representativeness in product design. Therefore, this paper takes children’s electric toothbrush as an example to explain the design method of a sensory engineering product knowledge base based on FBS model. 4.1 Cognition, Acquisition and Expression of Client Needs of Children’s Electric Toothbrushes Through the literature reading, market research, interviews, observations and other methods, 20 Kansei words related to children’s electric toothbrushes were summarized. 26 people were surveyed by questionnaires on industrial design-related professional backgrounds. Each person selected 10 representative descriptions. The emotional vocabulary of children’s electric toothbrushes. The results are shown in Fig. 2. The results of the survey are analyzed and sorted. The adjectives of the high votes are selected and the results are checked and corrected. Finally, the emotional vocabulary of 10 children’s electric toothbrushes is: intelligent, safe, lively, cute, light, reliable, client-friendly, concise, qualitative, novel, convenient and professional. 4.2 Transforming Client Demands for Children’s Electric Toothbrush The function of the children’s electric toothbrush is expressed by the method of “verb+noun”. The function, behavior and structure of children’s electric toothbrush are subdivided from the functional expression of children’s electric toothbrush, and they are modeled separately from function, behavior and structure. Finally, the functional unit base, behavior unit base and structural unit base model of the product are obtained. As shown in Fig. 3, an example of the basic function modeling of a child’s electric toothbrush is shown. The questionnaire was designed to allow the respondents to score the matching degree of the children’s electric toothbrush functional base in Fig. 4 and the 12 Kansei words obtained in Sect. 4.1. Take the “warn oral problems” in the basic functional unit of children’s electric toothbrushes as an example.

Developing a Knowledge-Based System for Lean Communications

43

Fig. 2. Children’s electric toothbrush questionnaire Kansei words intention bar chart

Clear plaque Clean teeth

Remove oral food residue Promote oral blood circulation

Basic Function

Massage gums

Relieve gum discomfort Reduce the chance of illness

Prevent of oral diseases

Warn oral problems Monitor oral condition

Conform to children s operational cognition

Fig. 3. Examples of basic functional modeling for children’s electric toothbrush

The statistics of the recovered scores are shown in Table 2, Sensitive vocabulary with a mean value greater than 4.0 is: smart, safe, reliable, humanized, convenient, and professional. The smallest standard deviation is convenient, so the mapping between “convenient” and “warning oral problems” is established. relationship. Establish a mapping relationship between children’s electric toothbrush functionbehavior-structure, match the behavioral base with the functional base, and then match the structural base with the behavioral base to finally complete the function, behavior and structure mapping. When the mapping relationship is established, Fig. 4 is a diagram showing the additional behavior-additional structure mapping relationship of the children’s electric toothbrush.

44

Y.-H. Hung and J.-B. Liang Table 2. Numerical analysis of warning oral problems Mean

Standard deviation

A confidence interval estimate for 90% of the mean Lower limits

Upper limits

intelligent

4.4

0.699

3.995

4.805

safe

4.4

0.843

3.911

4.889

lively

2.8

1.032

2.201

3.399

cute

3.0

1.247

2.277

3.723

light

3.2

1.398

2.389

4.011

reliable

4.1

0.876

3.592

4.608

user-friendly

4.0

0.817

3.527

4.473

concise

3.6

1.265

2.867

4.333

qualitative

3.9

1.197

3.206

4.594

novel

3.4

1.075

2.777

4.023

convenient

4.2

0.422

3.956

4.444

professional

4.2

0.789

3.743

4.657

Attached behavior

Play music, children's songs, etc.

①

Small noise, high frequency vibration

②

Foldable for easy carrying

③

Memory chip, memorable mode

④

Pressure sensing

⑤

Brush head replacement reminder

⑥

Correct brushing posture

⑦

Intelligent timing

⑧ ①

Sound system ② Damping system Attached structure

③ Storage box structure ⑧

⑦ ⑥

⑤ ④

①

PCBA\timer

Fig. 4. Mapping of additional behavior and structure of children’s electric toothbrush

Developing a Knowledge-Based System for Lean Communications

45

5 Mapping Construction of Children’s Electric Toothbrush Knowledge Base After using the expert knowledge to construct the logical relationship of the internal objects of the knowledge base, it is necessary to construct the operational logic of the overall knowledge base. It is also the presentation of the clients needs of children’s electric toothbrushes mentioned in Sect. 3.2. The operation process of the knowledge base is shown in Fig. 5.

Individual product

Judge based on expert knowledge Knowlegde Base

User

Search

User

Use expert knowledgeFBS to build internal attributes

Fig. 5. Operation flow chart of product knowledge base based on FBS model

Analyze a case of a child’s electric toothbrush product that needs to be stored in the knowledge base, and determine whether the product has a unit base. If the product case includes a unit base, a mapping can be established between the Kansei word that matches the unit base and the product case. Use the programming language to build the logical framework of the product knowledge base based on the analysis of the previous steps. The Django project is a Python custom framework used by the knowledge base site presented in this study. The logic code for implementing the knowledge base. The admin under /tooth/app is the code for the background administrator registration management page, the models are the code for defining the model (design database), the views are the code for implementing the view logic, the urls are the code for the distribution address, and the settings for /tooth/tooth are The configuration code for the knowledge base. Design and optimize the pages of the knowledge base so that clients can get specific products by searching for vague and sensible words.

6 Conclusion A product knowledge base establishment method based on expert knowledge has the following contributions compared with the existing research: • Using Lean Thinking to guide designer-client communication to improve processes. Develop a product knowledge base based on expert knowledge to simplify designerclient communications. • By combining expert knowledge with practical cases, the method covers the product attributes comprehensively and quickly establishes relationships between the various relationships, in order to map the relationship between clients, products and designers;

46

Y.-H. Hung and J.-B. Liang

• Build a product knowledge base based on FBS model, which is fast and at the same time oriented to clients and designers. Clients can search for products that best meet their needs. Designers can design innovative products based on knowledge base products to greatly improve design efficiency. Communication is the heart and soul of design and the total object of design is to communicate in some form. The study provided reference for using Lean Thinking to improve product design communication processes and develop product knowledge base. What’s more, the development of the knowledge base transformed the fuzzy client demands to the clear design requirements and made the communication between designers and clients effectively.

References 1. Maier, A.M., Eckert, C.M., Clarkson, P.J.: A meta-model for communication in engineering design. CoDesign 1(4), 243–254 (2005) 2. Shannon, C.E.: A mathematical theory of communication. ACM SIGMOBILE Mob. Comput. Commun. Rev. 5(1), 3–55 (2001) 3. Luhmann, N.: What is communication? Commun. Theor. 2(3), 251–259 (1992) 4. Crilly, N., Maier, A., Clarkson, P.J.: Representing artefacts as media: modelling the relationship between designer intent and consumer experience. Int. J. Des. 2(3), 15–27 (2008) 5. Shen, W., et al.: The user pre-occupancy evaluation method in designer–client communication in early design stage: a case study. Autom. Constr. 32, 112–124 (2013) 6. Lertlakkhanakul, J., Choi, J.W., Kim, M.Y.: Building data model and simulation platform for spatial interaction management in smart home. Autom. Constr. 17(8), 948–957 (2008) 7. Graham, A.K., et al.: User-centered design for technology-enabled services for eating disorders. Int. J. Eat. Disord. 52(10), 1095–1107 (2019) 8. Xue, L., et al.: An approach of the product form design based on gra-fuzzy logic model: a case study of train seats. Int. J. Innovative Comput. Inf. Control 15(1), 261–274 (2019) 9. Büyüközkan, G., Güler, M., Mukul, E.: An integrated fuzzy QFD methodology for customer oriented multifunctional power bank design. In: Kahraman, C., Cebi, S. (eds.) Customer Oriented Product Design. SSDC, vol. 279, pp. 73–91. Springer, Cham (2020). https://doi.org/ 10.1007/978-3-030-42188-5_5 10. Oliveira, F., et al.: Identifying user profiles from statistical grouping methods. J. Inf. Syst. Eng. Manage. 3(1), 06 (2018) 11. Ozalp, M., et al.: Integration of quality function deployment with IVIF-AHP and Kano model for customer oriented product. Customer Oriented Prod. Des. Intell. Fuzzy Tech. 279, 93 (2020) 12. Redeker, G.A., Kessler, G.Z., Kipper, L.M.: Lean information for lean communication: analysis of concepts, tools, references, and terms. Int. J. Inf. Manage. 47, 31–43 (2019) 13. Soares, S., Teixeira, L.: Lean information management in industrial context: an experience based on a practical case. Int. J. Ind. Eng. Manage. 5, 107–114 (2014) 14. Colazo, J.: Changes in communication patterns when implementing lean. Int. J. Qual. Reliab. Manage. (2020) 15. Ferreira, D.V.C., Barbosa, S.D.J.: Lean communication-centered design: a lightweight design process. In: Kurosu, M. (ed.) HCI 2016. LNCS, vol. 9731, pp. 553–564. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39510-4_51 16. Cruz-Lozano, R., et al.: Determining probability of importance of features in a sketch. ASCEASME J. Risk Uncert. Eng. Sys. Part B Mech. Eng. 3(4), 041003 (2017)

Developing a Knowledge-Based System for Lean Communications

47

17. Zhang, Z., et al.: A design communication framework based on structured knowledge representation. IEEE Trans. Eng. Manage. (2020) 18. Raaphorst, K., et al.: The semiotics of landscape design communication: towards a critical visual research approach in landscape architecture. Landsc. Res. 42(1), 120–133 (2017) 19. Judge, S., et al.: The language and communication attributes of graphic symbol communication aids–a systematic review and narrative synthesis. Disabil. Rehabil. Assist. Technol. 15(6), 652–662 (2020) 20. Pedó, B., et al.: Digital visual management tools in design management (2020) 21. Scupin, R.: The KJ method: a technique for analyzing data derived from Japanese ethnology. Hum. Organ. 56, 233–237 (1997) 22. Sutton, R.I., Hargadon, A.: Brainstorming groups in context: effectiveness in a product design firm. Adm. Sci. Quart. 41, 685–718 (1996) 23. Salvador, T., Bell, G., Anderson, K.: Design ethnography. Des. Manage. J. (Former Series) 10(4), 35–41 (1999) 24. Novak, J.D.: Concept mapping: a useful tool for science education. J. Res. Sci. Teach. 27(10), 937–949 (1990) 25. Berners-Lee, T., Hendler, J., Lassila, O.: The semantic web. Sci. Am. 284(5), 34–43 (2001) 26. Shi, Z.Z.: Knowledge Discovery. Tsinghua University Press, Beijing (2011) 27. Buchanan, B.G., Shortliffe, E.: The MYCIN Experiments of the Stanford Heuristic Programming Project. Addison-Wasley, Reading (1984) 28. Boose, J.H.: A knowledge acquisition program for expert systems based on personal construct psychology. Int. J. Man Mach. Stud. 23(5), 495–525 (1985) 29. Guillaume, S., Magdalena, L.: Expert guided integration of induced knowledge into a fuzzy knowledge base. Soft. Comput. 10(9), 773–784 (2006) 30. H.Q, W.F.Z.: Statistical analysis and coping strategies of cases of ethnic factors in China: based on the knowledge base of chinese national emergencies in 1980–2015. J. Intell. 35, 122–128 (2016) 31. Zhong, X.-Q., Liu, Z., Ding, P.-P.: Construction of knowledge base on hybrid reasoning and its application. Jisuanji Xuebao (Chinese J. Comput.) 35(4), 761–766 (2012) 32. Niu, F., et al.: Elementary: large-scale knowledge-base construction via machine learning and statistical inference. Int. J. Semant. Web Inf. Syst. (IJSWIS) 8(3), 42–73 (2012) 33. Hearst, M.A.: Automatic acquisition of hyponyms from large text corpora. In: Coling 1992 Volume 2: The 15th International Conference on Computational Linguistics (1992) 34. Li, Y., Reiss, F., Chiticariu, L.: SystemT: a declarative information extraction system. In: Proceedings of the ACL-HLT 2011 System Demonstrations (2011) 35. Betteridge, J., et al.: Toward never ending language learning. In: AAAI Spring Symposium: Learning by Reading and Learning to Read (2009) 36. Carlson, A., et al.: Toward an architecture for never-ending language learning. In: Proceedings of the AAAI Conference on Artificial Intelligence (2010) 37. Nakashole, N., Theobald, M., Weikum, G.: Scalable knowledge harvesting with high precision and high recall. In: Proceedings of the Fourth ACM International Conference on Web Search and Data Mining (2011) 38. Chen, Y., Wang, D.Z.: Knowledge expansion over probabilistic knowledge bases. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data (2014) 39. Shin, J., et al.: Incremental knowledge base construction using deepdive. In: Proceedings of the VLDB Endowment International Conference on Very Large Data Bases. NIH Public Access (2015) 40. Niu, F., et al.: DeepDive: web-scale knowledge-base construction using statistical learning and inference. VLDS 12, 25–28 (2012)

48

Y.-H. Hung and J.-B. Liang

41. De Ville, B.: Applying statistical knowledge to database analysis and knowledge base construction. In: Sixth Conference on Artificial Intelligence for Applications. IEEE Computer Society (1990) 42. McCoy, A.B., et al.: Development and evaluation of a crowdsourcing methodology for knowledge base construction: identifying relationships between clinical problems and medications. J. Am. Med. Inform. Assoc. 19(5), 713–718 (2012) 43. Gero, J.S.: Design prototypes: a knowledge representation schema for design. AI Mag. 11(4), 26 (1990)

Learn and Share to Control Your Household Pests: Designing a Communication Based App to Bridge the Gap Between Local Guides and the New Users Looking for a Reliable and Affordable Pest Control Solutions Shima Jahani(B) , Raman Ghafari Harivand, and Jung Joo Sohn Purdue University, West Lafayette, IN, USA {jahani,rghafari,jjsohn}@purdue.edu

Abstract. One of the most crucial steps to becoming independent for young people is leaving their parents’ homes to study or get a decent job. Based on the current population reports, 4,208,601 young people from 20–24 leave their parents’ house to live independently. When Young adults start their higher education or employment by living independently, they face many challenges with housekeeping specialty pest control. In this study, researchers tried to help young adults who move to a new place to start their higher education or employment to share their pest control experiences, access updated local information, and communicate better by designing a communication-based platform application. Community-based applications consist of one or many forums in which each member could be involved in the community. Keywords: Pest control · Community-based strategy · Household pesticide · Pest management strategy · Communicative forum

1 Introduction A nationwide survey by Home Team Pest Defense, the third-largest US residential pest control company, found that in the past 12 months, 84% of American homeowners have encountered a pest problem [1]. The health and well-being of house occupants can be hurt by the presence of pests [2], especially for young adults who start their higher education or employment by living independently, they face many challenges with housekeeping chores to keep it clean and hygienic [3]. Pest is any living creature or organism which could affect human wellbeing or economy negatively [4]. According to a consumer affairs report, most of the renters and owners in the US were nervous about ants, spiders, roaches, and bedbugs [5]. There are numerous integrated Pest Management (IPM) techniques for controlling pests; [2] Since pest control management strategies rely on a broad knowledge of pest’s biology and ecology, it is very challenging to apply effective and efficient methods [6]. One of the well-known methods is using pesticides [7]; although most of them are harmful to home occupants and nature, 80–90% of households use © Springer Nature Switzerland AG 2021 M. Kurosu (Ed.): HCII 2021, LNCS 12764, pp. 49–66, 2021. https://doi.org/10.1007/978-3-030-78468-3_4

50

S. Jahani et al.

them [8–10]. This research aims to provide a communicative atmosphere for our target users who are college or university students with no pest control experience. Lack of knowledge and experience makes them unable to cope efficiently with pest issues. Having communication with the expert people in the pest management field could help users with no pest control background find efficient, effective, and affordable solutions.

2 Related Works 2.1 Importance of Knowing Pest Control Solutions During screening the literature, the researchers decided to focus on the significant problems with pest control. Customers apply the solutions to exterminate the pests, but sometimes they are not aware of applying a given solution. Lack of knowledge, time, and budget leads the target users, young adults who move to a new place to start their higher education or employment and apply appropriate pest management solutions. In the following paragraphs, researchers provide evidence to show why the preexisting solutions could not always be useful. World Health Organization (WHO) data (2010) showed that one household pesticide contains two active ingredients for distinct insecticide clusters, carbamates and pyrethroids [11]. These ingredients impact human well-being and pollute rivers and groundwater, which cause poisoning and numerous diseases [12]; the use of pesticides in the houses is dangerous for residents and can threaten public health [13]. It causes several health issues such as cancer, asthma, allergies (sensitive to chemicals), fetal disability, acceleration of bone calcification, hypertension, reproductive disorders, carcinogenesis [14] and Parkinson’s [15]. People are exposed to pesticides when used in and around the home [16], and remaining pesticides are carried into the home on the residents’ shoes or clothing from the outside or workplace [17]. Children are more subject to pesticides because they are at home most of the time. Also, they show given behaviors such as hand to mouth, playing on the floor. Moreover, the children’s metabolism is immature, and their smaller size leads to greater consumption of pesticides from foods [18]. Pests cause diseases and physical damages and damnify human beings in terms of time and money. According to the United States Department of Agriculture (USDA), US “residents spend at least $1 billion on Formosan termite control and repairs each year. Some experts estimate the number is closer to $2 billion” [19]. 2.2 Lack of Budget Effects Pest Management Strategies Since our target users are young adults (students) with can earn 13,880 over the year, which means about 1,156 per month [20]. Spending money on pest management is not reasonable under this circumstance. This financial loss is not only associated with termites but also the other pests that can cause financial damages. In many cases, people need to call pest control agencies to solve their issues, which is very costly and timeconsuming. Based on the HomeGuide website, the average cost for pest control is about 250$, and if the customers need to hire someone to exterminate the pest, they will charge

Learn and Share to Control Your Household Pests: Designing a Communication

51

between 250$ to 400$, which is more than one-fourth of the monthly income of our target users. It is worth mentioning that their price is different by region and even by zip code. Table 1 shows the average cost for pest control by agencies in Indiana. The prices are varied based on the type of pest and areas [21]. Table 1. Average cost of pest control in Indiana state [21].

2.3 Role of Time in Pest Control Time plays an important role in the pest control process. Customers need to dedicate time to assess the situation, such as where the pest is living, how they are entering the home and finding the best strategy to exterminate them. US Bureau of Labor Statistics shows that college and undergraduate students have less than 4 h of free time [22]. It means spending time on pest control, which is a time-consuming process, is not rational for our target users. The process of pest management typically costs $300–$550 per visit. The process of pest management typically costs $300–$550 per visit. Therefore, if the customer faces a seasonal pest, their house needs a periodic visit that is more expensive. A periodic visit can happen every month, every two months, or every three months, and lasts more than 2 h per visit. Here are the average cost breakdowns for each kind of periodic visit: Every month: $40 to $45. Every two months (semi-monthly): $50 to $60. Every three months (quarterly): $100 to $300 [21].

52

S. Jahani et al.

Since the existence of pests in houses can cause various physical and biological problems, and lack of knowledge about the solutions like using chemical pesticides could be ended up in serious problems, the goal of this research is to provide a context that customers could diagnose their pest troubles efficiently and make the best solutions for choosing the solutions. A communicative platform can lead people, whit pest issues to share their problems with the people who have appropriate pest control experiences. 2.4 Community-Based Strategy A community-based approach is an appropriate tool for this study. Delgado, in Social work practice in nontraditional urban settings, explains that “Community-based program design is a social method for designing programs that enable social service providers, organizers, designers and evaluators to serve specific communities in their environment. This program design method depends on the participatory approach of community development, often associated with community-based social work, and is often employed by community organizations [23].” Therefore, this research aims to help young adults who move to a new place to start their higher education or employment to share their pest control experiences, access updated local information, and communicate better by designing a communication-based platform application. Community-based applications consist of one or many forums in which each member could be involved in the community. In this research, we decided to adopt a communication-based design into the app design and engage the users to share their pest control experiences with the new users looking for a reliable and affordable solution [24].

3 User Research and Findings 3.1 User Study (Questionnaire Survey) In order to collect first-hand data from users regarding pest control, the researchers have conducted a questionnaire survey. Through this survey, researchers tried to figure out the main problems the users face regarding pest control and the severity of the issue. It was also essential to understand how users cope with pest issues and their preferences when encountering pest issues. Understanding the type of the users’ houses, pest problems, their possible solutions, and their experiences regarding pest controlling was the aim of this survey. The questionnaire can be divided into two significant sections. The first section focuses on demographic questions and the second section is about the users’ experiences regarding pest control issues. To collect the data that will lead to developing the app, researchers have provided different types of responses such as 10 points Likert, Multiple choice, and Open-ended questions. To collect the data that will lead to developing the app, researchers have provided different types of responses such as 10 points Likert, Multiple choice, and Open-ended questions. This would lead to more accurate results as it provides the responders a better degree of freedom in answering the questions.

Learn and Share to Control Your Household Pests: Designing a Communication

53

Researchers considered the target users as students or employees between 18 and 32 years old, who have started to live independently for the first time. Out of all 32 individuals who fill out the form, 34.4% were students, and 65.6% were employees. Chart 1 shows the duration that the participants have started living independently.

Chart 1: Living duration-independently

The survey shows that the majority - 62.5% - of the target users live in apartments (suite of rooms forming one residence, typically in a building containing a number of these.) Chart 2 depicts the distribution for the type of dwelling, based on the users’ response.

Chart 2: Users house types

Based on our survey, 71.9% of users have been infected with at least one kind of pests. Also, it states that 76.9% of users have faced pest infection between 0 to 5 times per season.

54

S. Jahani et al.

Users reported that the kitchen is the first place where they found most of the pests there, and their bedrooms were the second place which receives pests’ damages mostly. Chart 3 is revealing that pests damaged which parts of their houses. Based on the survey responses, the kitchen is considered the hardest place for exterminating the pest. (Chart 4)

Chart 3: Affected places by pests

Chart 4: A most challenging place for exterminating the pest percentage

There are numerous signs and symptoms which the users figure out of the pest infestation in the house. The way users understand that pests have infected their houses is shown in the chart below (Chart 5). The responses show that 21 of the participants discovered the infestation by seeing the building’s pests themselves.

Learn and Share to Control Your Household Pests: Designing a Communication

55

When those were moving around, and nine responders found body parts in their houses. In this question, participants could select multiple responses.

Chart 5: The ways that users found the pests.

The first reaction of users encounter with pests is the essential part of our survey. The majority of responses (15 responses) show that their first reaction when facing a pest issue is to seek advice from their family or their friends on the issue by sharing the problem. Chart 6 states that the second reaction uses commercial pesticides by the house owner to exterminate the pest directly.

Chart 6: Distribution of responses for encountering pest issues

Researchers asked users about their preference regarding getting help from local advisors or pest control agencies. Based on the responses, 88.9% of users trust local

56

S. Jahani et al.

advisors and 11.1% trust the pest control agencies. In addition, Chart 7 demonstrates their reasons of trust as well as the frequency of each reason.

Chart 7: Why people can trust or cannot trust local advisors

Participants provided some solutions regarding pest controlling through open-ended questions. The variation of the solutions was limited. Therefore, researchers summarized them in Table 2. Among all the responses, using pesticides was the most popular solution among the participants, which means users prefer to do the pest controlling processes by themselves. Table 2. solutions for pest control. Ranking Solutions 1

Using pesticide

2

Using DIY and non-chemical solutions

2

Setting mice trap

3

Waiting for season changing

3

Asking someone for help

4

Calling agencies

4

Searching on Google

5

Raiding

More than 53% of participants mentioned that their solutions did not work, and the pests appeared after a while. They provided why their strategies regarding pest control failed or passed (Table 3). Throughout the survey participants were asked about the platforms (applications or websites) to control the pests. More than 89% of them have not any experience with pest

Learn and Share to Control Your Household Pests: Designing a Communication

57

Table 3. Reasons of successful of unsuccessful solutions Ranking Solutions

Outcomes

1

Old house with many unsealed holes

Unsuccessful

2

New pests come back every season

Unsuccessful

2

Effective chemical pesticides

Successful

3

DIY (do it yourself) solutions worked Successful

4

Hard to find their nests

Unsuccessful

control applications or websites. Reddit, Instagram, Bobvila.com, and Facebook were the platforms that less than 11% of participants have used for pest issues. According to the responses, more than 83% of users think that the above-mentioned platforms were not helpful because their information was very general, or they did not have a chance to use them. Besides, 16% of users thought that the platforms were helpful because it was free and saved their time. Also, they believed that using the platforms was very easy and quick which made them needless to search on Google. 3.2 Conclusion of the Survey Throughout this survey, the target users were asked about the type of house they live in and the pest issues they usually face. Researchers found that approximately 62% of target users are living in apartments for more than three years. Users stated that the kitchen and bedrooms are the two most infested areas of the home. Also, they mentioned that exterminating pests from these two places are considered harder than the other areas of the houses. Besides, users mostly notify about pest issues in their home when the pest moves around, making them easy to discover. The results state that half of the individuals shared their problems with their families or friends and used pesticides as the first reaction to pests. When target users solve the problems, they prefer to apply the local advisor’s solution instead of asking pest control agencies to help. The possible reason for the users tending to apply the local advisors’ suggestion vs. using pest control agencies was that local advisors ‘services are free of charge and trustable. Although more than 11% of users prefer to get a service from a pest control agency, more than 88% think that local advisors have the same experiences with efficient solutions that could help pest control agencies. According to the survey, using chemicals and “Do It Yourself” pesticides are the practical solutions to cope with pest problems. However, these strategies are not always effective as the pests come back as the season changes or the users’ houses are old enough to have many unsealed holes.

58

S. Jahani et al.

3.3 User Interview Then after the questionnaire survey, 6 participants were recruited from the 32. participants for the two rounds of interviews are for knowing the challenges users face during pest invasions and methods they use for pest control. The interview questions are mixed and listed below: • How would you rate the severity of the pest control situation at your place? • Based on the questionnaire you submitted, you said you had a pest infection. Can you please tell me more about your past experiences with pests at your new location? • Did you plan to prevent future incidents (pest invasions)? How? • Did you try to identify the type of pest? How? • Have you ever tried the pest controlling methods yourself? How? • Have you ever asked for advice from your friend or family friend regarding the pest situation? • Have you tried to ask your questions from local people and learn from their pest controlling experience? Would you trust their advice? • Have you ever contacted a pest control agency or a professional for pest control? Why? What was the problem? • Have you ever checked a website for pest control? Do you remember the situation? Did you find the information useful? If not, why? • Do you use the general pesticide? • Why did you start to cook? For what kind of reason? • Do you inspect the house for the pest situation frequently? After the interviews, we analyzed the interview scripts listed the findings, and selected the key findings. Here are our key findings from the interviews: • In general, pest control situations are either simple or complex. There are distinct motivations for whom target users turn toward first based on the severity of their situation. • DIY Solutions are proactive and not just reactive solutions. • Climate and weather outside impact the pest control situation inside the home • Users will use social media (Instagram story question/response feature, Facebook suggestions post) to assess who has been in a similar situation and what their solutions worked, and which ones did not work. Then they will contact those people directly for more information if necessary. • Problems with finding information from search engines: – – – – –

Information on the internet is scattered (too many sources) Varying opinions Difficult to refine your search to your specific situation Which sources are trustworthy Hard to identify the specific pest with personal identification through a search engine

• Solutions should be personal and customizable to fit the customer’s needs.

Learn and Share to Control Your Household Pests: Designing a Communication

59

• The cleanliness of one’s home has a direct correlation with pest control. • Cost is a major influence on what solution someone chooses to engage with. • People trust their landlords because they are easily accessible, typically knowledgeable, and include this as part of the lease agreement (affordable) • Local professionals/people with experience are useful. • Tenants/homeowners may need emotional support from someone they trust that can soothe them if their pest control situation produces fear/anxiety and give them practical solutions that they know will work. • Users ignore pest control situations and do not regularly inspect the house until it becomes a significant issue. • Many users face short-term pest invasions, and they need efficient, cheap, and effective solutions. After analyzing the findings, we identified the emerging themes. Based on the emerging themes, we created the ten insights statements. 3.4 Insights Gained from User Research • Sharing pest control experience with the local community is effective, efficient, and affordable • Users require quick, accurate responses in emergencies • Trust is essential when giving recommendations for pest situations • Complicated pest infections require professional expertise • Solutions should be customizable to fit the customer’s needs • Comprehensive and valid pest information and methodologies provide convenience • Future pest incidents can be prevented by appropriate cleaning and homecare • Many pest infections are invisible, and users become aware when the situation is out of control • Templates and communication guides make it easier to share the information After listing general insights, we prioritized our insights based on our original Design Challenge and project goals, and we discarded any that do not directly relate to our challenge. We then refined the key insights and prioritized them based on the following criteria Well-informed research data point, memorable content, and actionable results. Key Insights • Sharing pest control experience with the local community is effective, efficient, and affordable. • Trust is essential when giving recommendations for pest situations. • Emergency pest situations require quick and accurate responses. • Complicated/dangerous pest infections require professional expertise. • The pest information on the internet is too general and does not help with identifying the pest and finding appropriate solutions.

60

S. Jahani et al.

4 Design Process After the user research, we created product attributes for our application design. Creating a community-based platform application was our primary goal. Therefore, we created a social forum to let the people share their experiences and communicate. We also created ways for users to reach out to the professionals in a fast and affordable way. As for finding accurate and credible information, we also tried to add functions to our application. We brainstormed many ideas and went through several rounds of brainstorming and ideation. For each round, we created low fidelity prototypes and refined the ideas several times. After the final product attributes were decided, we confirmed the user flow structure of our app. Afterward, we designed wireframes and the user interface and created a high-fidelity version of our application with interactive pages (Fig 1).

Fig. 1. User flow of Pesto

Learn and Share to Control Your Household Pests: Designing a Communication

61

4.1 Main Features There are five main features to help the user with the pest control process. 1. 2. 3. 4. 5. 1.

Creating a report Search for finding the pest and the solutions Posting on a Forum Adding a link on News User Profile Creating a report: By this feature, users can create a post, link, or report. The new posts go to a given forum, the new link will be sent to the News page, and the report will be sent to the selected people, other media, or even a forum by users’ selection (Fig. 2).

Fig. 2. Creating a report: The user can report any pest problems on forums, contacts, social media, or seek a quick answer from the local professionals.

2. Search for finding the pest and the solutions: The users can quickly identify the pest type by taking a picture of it or put some keywords, and the application will bring up the matched results based on the input of the user and location. The user can find specific details of the pest and share them with others. The search feature also provides relevant information on the posts shared by other users (Fig. 3). Posting on a Forum: The users can post their questions or any external links in the forum. What they see in the forum are the general hot posts that local people have posted, and it is totally location-based but the Home page includes worldwide posts. They have organized based on users’ favorite topics (Fig. 4).

62

S. Jahani et al.

Fig. 3. Search for finding pest and situations: Users can type the name of the pest in the search bar or upload a picture of it, if it is available, in order to identify the pest information

Fig. 4. Posting on a forum. The users have access to the forums to share their experience and learn from local professionals or ask a new question.

Adding a link on News: Users can add and share an external link from the other News agencies and read the outbreak news regarding pest issues (Fig. 5). 3. User Profile. The users receive a score from the application based on their activity on the forum pages and their rating from the other users. After passing a certain level, the user’s membership will transform to “professional Member,” and the user can receive Boost requests from the other users to answer them and get paid (Fig. 6).

Learn and Share to Control Your Household Pests: Designing a Communication

63

Fig. 5. Adding a link on News.

Fig. 6. User Profile. The User has access to the archive of his/her pest controlling activities through the application.

5 User Testing to Evaluate the Prototype In order to evaluate our application, we performed quantitative and qualitative user testing and used the “Think Aloud” approach. We hired 4 participants who were previously participated in the early stages of the design to test our prototype to find answers to the following questions: Effectiveness: Is it possible for the user to navigate and use the tools in the application’s interface free of errors? Efficiency: What is the average time it takes for a user to complete each task? How long does it take to correct the errors? Satisfaction: Are users subjectively satisfied when using the application? Is the application pleasing to use? Is the user interface easy to use and learn?

64

S. Jahani et al.

Since the prototype was not fully functional, we filled the contents for some steps, and the user just needed to click on the page to process the application. For example, the user could not share something on social media, but when clicking on the share button, a screenshot of a social media post would be shown to the user to get his/her reaction. We asked users to accomplish the tasks below in order to evaluate the application: • Search for finding the information about a predefined pest by uploading a preexisting photo in the gallery and inserting given information. • Sharing pest information on social media • Report/Share a pest problem • Creating a forum • Adding a link on News The feedback we received from the participants were generally positive. The final results were showing that the application is easy to understand and efficient to accomplish the tasks. The users finish most of the tasks with no problem and found the application effective as well. They found the “Report” feature very useful in emergencies. They also liked the forum page and how it’s integrated into the search results. They also found it very useful to be able to find more information about the pest by taking pictures of it when it’s hard to identify it and learn from other people’s experiences in the forum pages. There was some valuable negative feedback as well. Some users had a hard time understanding the scoring system and how they become a local professional and earning money. Some users also were not sure if networking and having connections on a pest control application is necessary and did not find it quite useful.

6 Discussion and Conclusion This paper reported findings from 32 participants through surveys and six individual interviews for pest control. Researchers created questionnaires to understand various users’ situations for each housing type during the primary research phase. We found great potential in sharing information among local community members and the importance of finding a problem early. Through the user research, we figured out five key insights and developed five main features. We designed an application based on an information-sharing platform, built a community to connect with other users, and quickly reported to local professionals. By encouraging users to share information and solutions, we promoted the interaction among the users and their engagement in the community. To get a quick solution for an urgent case, we added the paid service ‘Report’ that can get quick advice from other users and quotes from local professionals. We believe that the report service is a win-win strategy for both community users and professionals. Other features of the app, such as posting on a forum and searching a database, also help users find a control pest solution and encourage sharing information with other community members. In this study, most of the main features have no major problem, as the evaluation turned out. However, there are some details and the business model that could be polished.

Learn and Share to Control Your Household Pests: Designing a Communication

65

Also, pest problems do not happen every day. This will affect the activity of the forum and the frequency of use. Our future work may focus on making a more detailed working prototype and solution for a more active community in the app.

References 1. Home Team Pest Deafense “Survey Results on Pests”: https://pestdefense.com/hometeamreleases-survey-results-on-pests/. Accessed 24 Apr 2012 2. Centers for Disease Control and Prevention and U.S. Department of Housing and Urban Development: Healthy housing reference manual, Chapter 4. US Department of Health and Human Services, Atlanta (2006) 3. Murphy, R.G., Todd, S.: Minimizing pest risk in dwellings. Struct. Surv. 14(1), 9–13 (1996). https://doi.org/10.1108/02630809610116197 4. Bateman, P.L.G.: Household pests. Struct. Surv. 2(2), 115–123 (1984). https://doi.org/10. 1108/eb006181 5. Parkman, K.: Pest control statistics and trends. https://www.consumeraffairs.com/homeow ners/pest-control-statistics.html. Accessed 14 May 2020 6. Pélissié, B., Crossley, M.S., Cohen, Z.P., Schoville, S.D.: Rapid evolution in insect pests: the importance of space and time in population genomics studies. Curr. Opin. Insect Sci. 26, 8–16 (2018) 7. Triwidodo, H., Mudikdjo, K., Panjaitan, N.K., Manuwoto, S., Yuliani, T.S.: Studies on the behavior housewives in home pesticide usage in Special Capital Region Jakarta. IPB (Bogor Agricultural University (2012). https://repository.ipb.ac.id/jspui/handle/123456789/54270 8. Armes, M.N., et al.: Residential pesticide usage in older adults residing in Central California. Int. J. Environ. Res. Public Health 8, 3114–3133 (2011). https://doi.org/10.3390/ijerph808 3114 9. Savage, E.P., et al.: Household pesticide usage in the United States. Arch. Environ. Health 36, 304–309 (1981). https://doi.org/10.1080/00039896.1981.10667642 10. Stout II, D.M., et al.: American Healthy Homes Survey: a national study of residential pesticides measured from floor wipes. Environ. Sci. Technol. 43, 4294–4300 (2009). https://pubs. acs.org/doi/full/10.1021/es8030243 11. World Health Organization: The WHO Recommended Classification of Pesticides by Hazard and Guidelines to Classification 2009, p. 8. World Health Organization (2010). https://apps. who.int/iris/bitstream/handle/10665/44271/9789241547963_eng.pdf?sequence=1 12. Den Hond, F., Groenewegen, P., van Straalen, N.: Pesticides Problems, Improvements, Alternative, pp. 1–17. Blackwell Sciences Ltd., Blackwell Publishing company (2003) 13. Raini, M., Isnawati, A., Herman, M.J.: Paparan propoxur pada anggota rumah tangga yang menggunakan anti serangga semprot di Jakarta, Tangerang, Bekasi dan Depok. Indonesian Bull. Health Res. 37, 43–54 (2009). https://doi.org/10.22435/bpk.v37i1 14. Atkinson, H.C., Begg, E.J., Darlow, B.A.: Drugs in human milk. Clinical pharmacokinetic considerations. Clin. Pharmacokinet. 14, 217–240 (1988). https://doi.org/10.2165/00003088198814040-00003 15. Hileman, B.: The environment and Parkinson’s: If exposure to chemicals causes this dread disease, regulators may have to alter approaches to neurotoxicity testing and risk assessment. Chem. Eng. News. 79, 35–37 (2001). https://doi.org/10.1021/cen-v079n038.p035 16. Bradman, A., Whyatt, R.M.: Characterizing exposures to nonpersistent pesticides during pregnancy and early childhood in the National Children’s Study: a review of monitoring and measurement methodologies. Environ. Health Prospect 113, 1092–1099 (2005). https://doi. org/10.1289/ehp.7769

66

S. Jahani et al.

17. Coronado, G.D., Vigoren, E.M., Thompson, B., Griffith, W.C., Faustman, E.M.: Organophosphate pesticide exposure and work in pome fruit: evidence for the take-home pesticide pathway. Environ. Health Prospect 114, 999–1006 (2003). https://doi.org/10.1289/ehp.8620 18. Steer, C.D., Grey, C.N.B., Alspac Study Team: Socio-demographic characteristics of UK families using pesticides and weed-killers. J. Expo. Sci. Environ. Epidemiol. 16, 251–263 (2006). https://doi.org/10.1038/sj.jea.7500455 19. Kaplan, K.: Learning What “Wood” a Termite Prefers. https://www.ars.usda.gov/news-eve nts/news/research-news/2015/learning-what-wood-a-termite-prefers/. Accessed 12 Nov 2015 20. Collage affordability: Working at the Minimum Wage. http://collegeaffordability.urban.org/ covering-expenses/working-during-college/#/. Accessed 10 May 2016 21. Daniel, W.: How much does pest control service cost? https://homeguide.com/costs/pest-con trol-prices. Accessed 18 Mar 2020 22. U.S Bureau of Labor Statistics: Time use on average weekly for full time university and college students. https://www.bls.gov/tus/charts/students.htm. Accessed 20 Dec 2016 23. Delgado, M.: Social Work Practice in Nontraditional Urban Settings. Oxford University Press (1999) 24. Kazmi, S.S.: Create Community Based Mobile Apps. https://appsgeyser.com/blog/createcommunity-based-mobile-apps/. Accessed 8 Jul 2020

Developing User Interface Design Strategy to Improve Media Credibility of Mobile Portal News Min-Jeong Kim(B) Sookmyung Women’s University, Cheongpa-ro 47-gil 100, Yongsan-gu, Seoul 04310, Korea [email protected]

Abstract. According to various media phenomena appearing in Internet environment, portal news is becoming one of the most influential media in Korea. However, the media credibility of portal news is not very high compared to that of broadcast media or newspapers. Therefore, the focus of this study is to elicit insights for establishing strategies for improving the media credibility of portal news by relating the credibility of various news media and features of user groups. First, media credibility is identified through a factor analysis and user segments are obtained by a cluster analysis. Then, user interface design strategies of mobile portal news are provided according to characteristics of each group. The findings can contribute to classifying user groups by factor-specific media credibility and to suggesting design strategies for mobile portal news, depending on the features of user groups for improving media credibility. Keywords: Mobile portal news · Media credibility · Factor analysis · Cluster analysis

1 Introduction Journalism is undergoing a fundamental transformation, perhaps the most fundamental since the rise of the penny press of the mid-nineteenth century. In the twilight of the twentieth century and the dawn of the twenty-first, there is emerging a new form of journalism whose distinguishing qualities include ubiquitous news, global information access, instantaneous reporting, interactivity, multimedia content, and extreme content customization. At the core of transformation is the Internet. Digital technologies such as the Internet bring about not only the concept of news and media, but also structural changes such as supply and consumption of news [1]. Among the various press-related phenomena that emerged in the age of the Internet, portal news has established itself as one of the most influential journalism in Korea [2]. Portal news refers to the provision of news services through portal sites, which act as a gateway to Internet users. It differs slightly from news portals, which are the internet-enabled version of traditional news media. In portal news, the portal sites rely on other news media and news agencies for most of their news, and they retransmit the news articles of news media and news agencies. In other words, portal sites have © Springer Nature Switzerland AG 2021 M. Kurosu (Ed.): HCII 2021, LNCS 12764, pp. 67–84, 2021. https://doi.org/10.1007/978-3-030-78468-3_5

68

M.-J. Kim

sufficient influence over consumers of news in the editing process, although the sites do not produce news of their own [3]. Whether portal news can be considered the news media has been a topic of debate since the beginning of news services by portal sites in Korea in the 2000s. The argument against considering portal sites as a type of news media is that they do not carry out the actual functions of the news media but instead mediate news commercially with profit-centric intentions. Furthermore, the media-related characteristics of portal sites reveal low entry barriers and communications media that involve communication and information exchange on a broad level immediately, making them fundamentally different from print media and broadcast news [4, 5]. On the contrary, the arguments put forth in favor of viewing portal sites as news media are that portal sites (a) engage in editing, one of the major functions of journalism; (b) carry out gatekeeping and agenda-setting functions while filtering and selecting news articles; and (c) influence the formation of public opinion. Hence, they can be regarded as performing the functions of journalism in Korean society [6–8]. While conflicting viewpoints exist regarding the status of portal sites as a form of news media, portal news has already become one of the influential journalism by carrying out journalistic functions through agenda setting and formation of public opinion in the process of social communication. An examination of the news utilization rates by medium revealed that the utilization rate of news through television reduced to 85.4% in 2018 from 95.2% in 2011; printed newspaper use fell by more than half, from 44.6% in 2011 to 17.7% in 2018. However, the population using the Internet via mobile devices and PCs to access news increased from 57.0% in 2011 to 82.3% in 2018; and among this population, mobile-enabled news access increased from 19.5% in 2011 to 80.8% in 2018 while PC-enabled news access dropped from 51.5% in 2011 to 31.7% in 2018 [9]. As such, there has been a decrease in the proportion of accessing news via TV, printed newspapers, and radio over the last 7 years, while the news-mediating functions of the Internet have expanded continuously, most notably, mobile news access. Furthermore, considering the high proportion of portal sites among mobile-enabled news access [10], the role of portal news has become more significant. While an increasing number of people are using their mobile devices to access portal news in recent times, analysis has revealed that the media credibility of portal news is not very high compared to that of broadcast media or newspapers [9]. In other words, portal news users consume portal news even though its credibility is not as high as that of broadcast media or newspapers. Most of the domestic studies on portal news have focused on its legal regulation given the change in the importance of portal news. With the rising importance of portal news, there have been many studies on media credibility, similarities and differences in the credibility of various forms of media, including portal news. In addition, research on user interface design of portal news was conducted for both PC and mobile devices. However, research on user interface design in portal news has generally focused on advancing usability, and there have been hardly any studies on user interface design for improving media credibility. Therefore, this study aims to design a mobile user interface for improving the media credibility of portal news. The objective of this study is to identify user groups based on the measurements of media credibility for various news mediums. It also aims to design the user interface

Developing User Interface Design Strategy to Improve Media Credibility

69

of mobile portal news based on the characteristics of the identified groups. In the first phase of this study, factor analysis is used to identify common characteristics in the credibility of various news media, and the results are summarized into factor-specific media credibility. This is followed by cluster analysis, which is used to classify user groups by factor-specific media credibility. The next phase involves analyzing the demographic and social characteristics, frequency and method of portal news usage and providing user interface design strategies for each group based on the findings. Through the proposed user interface design strategies for mobile portal news, this study aims to improve the media credibility of portal news and contribute to creating a preferred user experience.

2 Background 2.1 Transformation in the Portal News Medium With the spread of the Internet, Korean news media underwent major changes. The environment of news distribution through newspapers and broadcasting changed to digitalcentric news distribution, and the existing news media companies strived to change by creating subsidiaries and engaging in online news distribution. In the 2000s, the existing news media companies began to supply news to portal sites along with their own online news distribution and thereafter failed to establish their own distribution methods, handing over the reins of news distribution to portal sites. This then led to dramatic changes in the importance of portal sites in Korea; news service users using portal sites in 2003 increased from 6.97 million in January 2003 to 10.97 million in November 2003 in the case of Naver (Korea’s No.1 portal), and Daum (Korea’s 2nd portal) saw an increase to 17.58 million users from 6.89 million over the same time period. Furthermore, 15 news media companies supplied news to portal sites at the beginning of the 2000s; this number then surpassed 100 in 2006, and in 2015 there were more than 200 news media distributing news through portal sites [11]. This signifies that portal sites, as platform operators, lead the online news distribution in Korea and have unparalleled influence on news media companies. Furthermore, a survey on 1,000 smartphone users aged between 20 and 60 in March 2018 to explore news utilization through portal sites confirmed that portal sites (93.3%) were the major source for news consumption. While news consumers who accessed news through terrestrial TV broadcasting and general programming channels were still significant at 81.8% and 61.6%, respectively, people accessing news through radio and print newspaper were 1.7% and 2.5%, respectively [10]. With the increasing importance of portal sites as news mediators, the Roh Moohyun Administration (2003–2008) began to take interest in the journalistic functions and public opinion formation functions based on portal sites. The government implicitly acknowledged that the Internet, including portal sites, functioned as the journalism in Korean society, by including the Internet in the laws and regulations governing the news media [12]. Furthermore, along with the institutionalization of regulation of the Internet through amendments in laws, the Ministry of Culture, Sports and Tourism established and announced the “Guidelines for News Use Agreement between news media and portals” to induce autonomous regulation of portal sites.

70

M.-J. Kim

With the rising demand for the social responsibility of portal sites in Korea, the Korean Internet Self-governance Organization (KISO) has been launched as a tool for self-regulation in 2009. In 2012, three years after its launch, KISO enacted the “Autonomous regulation on the arrangement of articles of Internet news service providers” in partnership with the Korea Internet Corporations Association, to lay the foundation for improving the fairness and credibility of socially controversial Internet news [2]. 2.2 Media Credibility Studies on media credibility have been conducted in the United States since the early 1950s. The term “credibility” has been studied as two concepts: audience credibility and source credibility. Hovland and Weiss explained source credibility as a factor for enhancing communication effectiveness [13]. Gaziano and McGrath, who studied measurement the concept of credibility, showed that 12 items grouped together in a credibility factor and created newspaper and television credibility scores based on scores on the 12 items, which later formed the basic framework of research into credibility of newspapers and television news [14]. Meyer divided the 12 items of Gaziano-McGrath into 2 dimensions, which were news believability and community affiliation [15]. Among 2 dimensions, Meyer developed 5 believability indexes (fairness, bias, completeness, accuracy, and trustworthiness) to define and measure the credibility of newspapers [15], and studies that followed have since measured credibility based on Meyer’s research. Starting in the late 1990s, research comparing the credibility of traditional media (newspapers, TV) and online media (Internet) was conducted; Johnson and Kaye compared Internet and traditional source on media credibility measure, and concluded that online media tended to be judged more credible than their traditional versions [16]. Media credibility research in Korea was spearheaded by the Korea Press Foundation with user surveys taken every two years since 1984 to assess overall media credibility; surveys after 2000 have since compared the credibility of newspapers, TV, and the Internet. According to the Korea Press Foundation, in 2004, credibility ranking was in the order of TV, radio, newspaper, Internet, and cable TV [17]. Recent results also indicate that the credibility of portal news is not higher than that of broadcast media and newspapers [9]. The reason for the poor credibility of portal news is because their coverage of news on politics and economics has reduced, which are traditionally considered important, to secure higher traffic through the distribution of sensational soft news on topics such as sports and entertainment [18]. Studies that measure media credibility have begun to include speed as an additional measure of credibility, along with other major items measuring media credibility with the development of new media [19, 20]; this is because, given the characteristics of online media, speed of information delivery is an important trust factor from a user’s perspective. 2.3 User Interface Design of Online News Studies on the user interface of online news services have been conducted since the late 1990s [21–24], and along with this, research on the usability of online news websites was also carried out. Generally, the standards for usability of online news websites are

Developing User Interface Design Strategy to Improve Media Credibility

71

based on usability and web contents as defined in ISO documents such as ISO 9241Part 11 [25]. As the use of news through smartphones and tablets is increasing, research on recommendation of news content according to personalization of interface and method of providing user interface personalization has been conducted [26]. As portal news became the dominantly used media in Korea, studies on the user interface of portal news became common: Park et al. used Focus Group Interview (FGI) in their study on the user interface of Korea’s top portal news [27]; and Lee et al. comparatively analyzed the news portals of existing news media and portal news, suggesting a layout that could improve the accessibility and functionality of portal news [28]. Research on the service design of mobile portal news was conducted by evaluating news services of the two leading providers of portal news in Korea and presenting associated recommendations [29].

3 Research Method 3.1 Respondents The data are drawn from the 2017 Survey on the Opinion of the Media Audience by Korea Press Foundation [30]. The Korea Press Foundation conducted a computer-aided personal interview (CAPI) survey using tablet PC of 5,010 citizens aged over 19 nationwide in order to establish data for media-related research. Usage behaviors and media credibility of news media based on the changes in the media environment are being investigated bi-annually since 1984 and are being surveyed every year since 2010. The 2017 Survey on the Opinion of the Media Audience covers 15 news media (regional daily newspapers, regional weekly newspapers, national newspapers, economic newspapers, news agencies, news magazines, news channels, general programming channels, terrestrial TV broadcasts, radio broadcasts, SNS, messaging services, portal news, Internet news, and websites of news media) and contains data on usage behaviors and media credibility of 15 news media and usage behaviors of portal news. In this study, 2,576 respondents who responded to the portal as a news media were first selected out of the total 5,010 respondents, and 2,084 among the 2,576 respondents were finally analyzed except those who responded ‘I don’t know’ to a question on credibility of news media. 3.2 Analysis Method We subdivide the finally selected 2,084 respondents by using the media credibility of 15 news media as variables. However, using 15 variables to derive cluster characteristics can lead to distribution of cluster characteristics, thus making it difficult to identify clusterspecific differentiators. Therefore, a factor analysis was conducted to reduce the number of variables through the elimination of redundancy and a cluster analysis was employed to classify the respondents into groups for the factor scores which were obtained from the factor analysis. In this way, there are studies in which factor analysis and cluster analysis are performed together for segmentation [31, 32]. These analyses in this paper were performed by SPSS. Additionally, the respondent distributions in segments (which were obtained from the two analyses) were examined in terms of socio-demographic

72

M.-J. Kim

characteristics and usage behaviors of portal news. In this section, the factor analysis and the cluster analysis for the data are described in detail. Factor Analysis As mentioned above, factor analysis is a method for investigating whether a number of variables of interest are linearly related to a smaller number of unobservable factors [33]. It constitutes a data analytics method that is most faithful to the principle of parsimony [34]. In this paper, factor analysis was employed for the purpose of reducing the number of variables (15 type of news media) to fewer significantly meaningful factors groups. Factor analysis was performed by evaluation of principal components and computing eigenvectors. Only the eigenvalues higher than 1 (Kaiser Criterion) [35], giving a cumulative percentage of the variance above 60%, were retained. Afterwards, the rotation of the principal components was carried out by a varimax rotation method. The results, presented as factor loadings of the rotated method enables to summarize variables into factors and to detect structures in the relationships between variables. In addition, in order to be used in the cluster analysis in the next stage, the factor scores for the respondents was calculated using the factor score coefficient obtained from the extracted factors. Cluster Analysis At this stage, cluster analysis was performed to classify respondent groups by factorspecific media credibility, which is summarized by multiplying media credibility of the respondents by factor score coefficient. Punj and Stewart examined several clustering methods used for marketing practices [36]. They concluded that iterative partitioning methods tended to outperform hierarchical methods, on the basis of an extensive review of numerous empirical studies. In particular, the iterative partitioning methods were widely used for a large number of data for clusters since their computational processes were faster than those of the hierarchical methods. There are a number of techniques in iterative clustering methods, but one of the most prominent is a K-means technique, which was employed as it is appropriate for our large-size data. Through this K-means technique, user segments based on the factor-specific media credibility were formed. After the segmentation, the information on influential variables such as socio-demographic factors and usage behaviors of portal news for each segment was analyzed to better characterize the segments. The results can be used in developing design strategies for mobile portal news, which could be customized for each user group.

4 Research Results In this section, the results of applying the analytics methodology based on the respondents’ survey data are summarized. The chapter begins by explaining the analysis of credibility of 15 news media from the respondents. Then, the credibility of the 15 news media was categorized into three factors through a factor analysis. Next, three segments for factor-specific media credibility was obtained through a cluster analysis with factor scores of the respondents’ survey, and the user profiles and their characteristics in each segment were analyzed.

Developing User Interface Design Strategy to Improve Media Credibility

73

Table 1. Descriptive statistics of the raw data - mean (standard deviation). News media

Mean (StDev)

News media

Mean (StDev)

Regional daily newspapers

3.44(0.804)

Terrestrial TV broadcasts

4.08(0.832)

Regional weekly newspapers

3.49(0.807)

Radio broadcasts

3.63(0.789)

National newspapers

3.58(0.812)

SNS

3.02(0.827)

Economic newspapers

3.64(0.841)

Messaging services

3.11(0.817)

News agencies

3.71(0.858)

Portal news

3.74(0.758)

News magazines

3.35(0.823)

Internet news

3.31(0.820)

News channels

3.99(0.791)

Websites of news media

3.38(0.838)

General programming channels

4.10(0.815)

The basic statistics of the credibility of 15 news media are presented in Table 1. Media credibility was surveyed on a 5-point scale (1 point for Not credible at all – 5 points for Very credible), and general programming channels had the highest credibility at 4.10 points. This was followed by 4.08 points for terrestrial TV broadcasts and 3.99 for news channels, indicating high credibility for broadcasting media. In particular the credibility of terrestrial TV broadcasts is lower than that of general programming channels because the image of unfair and biased reporting has made a negative impression on viewers. Fourth on the list was portal news, with 3.74 points. This is because the survey respondents were selected only the respondents who answered the portal as a news media, and their media credibility of portal were relatively high. On the other hand, when we look at the news media with low scores, credibility of news from SNS had the lowest credibility with 3.02 points, followed by messaging services at 3.11 points, Internet news at 3.31 points, news magazine at 3.35 points, and websites of news media at 3.38 points. Among these classifications, websites of news media refer to media through which Korean news media distribute news online, and Internet newspapers refer to Internet-based online news companies that grew because of higher Internet penetration and installation of high-speed Internet [11] and both can be considered similar to news portals. The credibility of portal news is higher than that of other forms of online media such as websites of news media and Internet news because both websites of news media and Internet news supply portals with news, and thus portal news had higher credibility in terms of unbiasedness and objectivity compared to individual online news media, and it had the highest score for speed [19]. 4.1 Factor Analysis of Media Credibility As mentioned, the factor analysis aimed at categorizing the media credibility of 15 news media into fewer factors (factor-specific media credibility) based on responses for the degrees of media credibility. In this study, a principal component analysis was applied to group the 15 variables under consideration, and three major factors with eigenvalues of 1 or more were identified. The three factors accounted for 62.7% of the cumulative percentage of the variance. In general, the factors accounting for 60–70% or more of

74

M.-J. Kim

the variance can be considered to be a good fit to the data. A varimax rotation was then applied to the three factors, with 5 iterations. This procedure rotated the set of individual scores within the space defined by the principal component axes, thereby creating a new set of factor loadings for the factors that have already been found [33]. The rotated factor matrix is presented in Table 2 – in terms of factor loadings, eigenvalues, percent of variance (cumulative percent), and communality. Table 2. Main factors underlying credibility of news media. News media

Print media

Broadcast news

Internet

Communality

Regional daily newspapers

0.818

0.181

0.708

Regional weekly newspapers

0.802

0.197

0.153

0.705

National newspapers

0.764

0.192

0.157

0.646

Economic newspapers

0.664

0.316

0.255

0.605

News agencies

0.510

0.474

0.222

0.534

News magazines

0.448

0.420

0.358

0.505

News channels

0.187

0.817

General programming channels

0.139

0.789

0.169

0.671

Terrestrial TV broadcasts

0.274

0.707

0.155

0.599

Radio broadcasts

0.348

0.587

0.163

0.482

SNS

0.105

0.875

0.779

Messaging services

0.142

0.862

0.770

0.711

Portal news

0.183

0.359

0.615

0.540

Internet news

0.280

0.451

0.540

0.574

Websites of news media

0.419

0.407

0.482

0.573

Eigen value

3.367

3.252

2.793

% of variance

22.446

21.680

18.623

Cumulative %

22.446

44.126

62.749

The nature of each factor was determined by the characteristics of the variables. The high loadings on the factors appear in bold font in Table 2. The first factor (Component 1) had heavy loadings for six variables (i.e., regional daily newspapers, regional weekly newspapers, nationwide newspapers, economic newspapers, news agencies, and news magazines), which were mainly related to print media. This factor was called “print media.” The second factor (Component 2) was characterized by four variables (i.e., news channels, general programming channels, terrestrial TV broadcasts, and radio broadcasts) relating to broadcast news. This factor was called “broadcast news.” The third factor (Component 3) centered on five variables (i.e., SNS, messaging services, portal news, Internet news, and websites of news media) for Internet. This factor was

Developing User Interface Design Strategy to Improve Media Credibility

75

termed “Internet.” As a result, we reduced media credibility of 15 news media to three factor-specific media credibility. Table 3 shows the factor score coefficient matrix. This matrix was used to calculate the factor scores of all of the responses. As described above, the three main factors were extracted to ensure the quality of clustering instead of directly using the 15 variables. Using this matrix, the factor scores for the 2,084 respondents could be obtained and then used in forming segments. Table 3. Factors score coefficient matrix. News media

Print media

Broadcast news

Internet

Regional daily newspapers

0.383

−0.143

−0.107

Regional weekly newspapers

0.358

−0.143

−0.064

National newspapers

0.339

−0.136

−0.055

Economic newspapers

0.235

−0.052

−0.011

News agencies

0.110

0.091

−0.034

News magazines

0.073

0.054

0.057

News channels

−0.147

0.405

−0.117

General programming channels

−0.176

0.390

−0.064

Terrestrial TV broadcasts

−0.076

0.305

−0.076

Radio broadcasts

−0.003

0.213

−0.062

SNS

−0.080

−0.151

0.444

Messaging services

−0.067

−0.140

0.426

Portal news

−0.090

0.050

0.241

Internet news

−0.053

0.089

0.172

0.044

0.033

0.129

Websites of news media

4.2 User Segments Based on Factor-Specific Media Credibility To obtain the factor scores, users’ responses for the media credibility of 15 news media were multiplied by the factor score coefficients (see Table 3). The media credibility of three factors (i.e., print media, broadcast news, and Internet) obtained through the factor analysis were used as the input variables in a cluster analysis. A K-mean technique was employed for this analysis. The cluster analysis with the 2084 × 3 matrix was performed for the values of the drawn factors. The analysis results for the factor scores of the respondents for media credibility are presented in Table 4. The number of clusters was set to three through a process of trial and error, taking into consideration the significance levels of the factors. The cluster centers mean are the center values of each cluster, that is, the set of the average factor scores. F tests should be used only for descriptive purposes because the

76

M.-J. Kim

clusters have been selected to maximize the differences among the clusters, and thus the significance levels cannot be used to test the hypothesis that the cluster means are different. Rather, if the observed significance level of a factor is high, it can be relatively assured that the factor does not contribute much to the separation of the clusters [37]. Because all the significance levels for the factors in Table 4 were less than .001 in case of the three clusters, the factors were meaningful in differentiating the clusters. Table 4. Cluster analysis results. Final Cluster Centers 1

2

3

d.f

F

Sig

Print media −.37764

−.49767

.81813

2

582.355