Emotional Engineering, Vol. 8: Emotion in the Emerging World 3030383598, 9783030383596

Citation preview

Shuichi Fukuda   Editor

Emotional Engineering, Vol. 8 Emotion in the Emerging World

Emotional Engineering, Vol. 8

Shuichi Fukuda Editor

Emotional Engineering, Vol. 8 Emotion in the Emerging World

123

Editor Shuichi Fukuda Keio University Tokyo, Japan

ISBN 978-3-030-38359-6 ISBN 978-3-030-38360-2 https://doi.org/10.1007/978-3-030-38360-2

(eBook)

© Springer Nature Switzerland AG 2020 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

Most people regard emotion in a reactive way. They think their response to the outside world stimuli is emotion. But if we remember the Latin etymology of emotion is “e” = ex-out and “movere” = move, it literally means how we move out. And it is interesting to know that the word “motivation” comes from the same Latin “movere”. Thus, emotion is very much active. As our world expands rapidly and shifted to a wide, open world without boundaries, we cannot apply rational approaches in a straightforward manner as we did yesterday when our world is small, closed and with boundaries. We responded to such a change with the idea of “System Identification”. The basic idea of “System Identification” can be described in the following way. It is the same idea as we identify the name of a river. If we look at the flow, we cannot identify its name. Water is changing every minute. And we don’t have appropriate means to describe its change rationally, although computational fluid dynamic theory has progressed so much. So, instead we look around and find mountains or forests that do not move at all. Then, through these feature points, we identify its name. How we identify the feature points in the changing environments and situations is “System Identification”. We identify the system and then we will know what parameters should be considered for controlling the system. The most important point we need to realize is that rational approaches are primarily focus on “control”. The idea of control is how we predict the changes and keep our systems operating in a robust condition. If we use the recent buzzword, this is “Adaptability”. Such adaptability may be described as “staying afloat” in the river. In other words, it is “Reactive”, i.e., “Reactive Emotion”. But the changes themselves are changing rapidly. There were changes yesterday. But they changed smoothly so that we could differentiate them mathematically. Therefore, we could predict the future. But today, changes occur very frequently and extensively. And the greatest change of changes is they come to change sharply. Therefore, we cannot differentiate them anymore, i.e., we cannot predict the future. We need to “adapt” to these unpredictable changes to survive. Thus, “Adaptability” becomes more important than products and their functions. To put it another way, engineering or society itself was product-centric yesterday. But today, v

vi

Preface

processes become more important than products. Adaptability is the problem of how we process. Abraham Maslow, American psychologist, proposed “the Hierarchy of Human Needs”. He pointed out that at the lower level of the hierarchy, humans want physical or material satisfaction. But as they go up, their needs change from material to mental and at the highest level, they look for “Self-Actualization”. Emotion, if we come back to its Latin origin, means nothing other than “Self-Actualization”. It also interests us to know that Edward Deci and Robert Ryan, both American psychologists, proposed “Self-Determination Theory” and they pointed out that we feel happiest and get the maximum satisfaction when we do the job which is internally motivated. No matter how much reward is given from outside, it cannot provide such happiness or satisfaction, although the job is the same. Thus, emotion is how we would like to establish ourselves in frequently and extensively and unpredictably changing environments and situations. It is not a problem of just adaptation. No. what matters most is how we can swim against the flow to get to our destination. It is not just staying afloat. Emotion is not reactive. It is very much proactive. If we are just active. we just stay afloat. To swim ahead, we need to be proactive. Emotion is proactive. Thus, decision making becomes most important in the next generation industry as economists point out. And motivation and emotion play an important role in decision making. They constitute a loop. Emotion plays an important role what action we should take to make another step or another swim forward. In this volume, many different issues relating to this problem of today are taken up from a wide variety of perspectives. I hope the reader will understand what attempts are being made outside of their own fields and they can team up to win such a never- experienced game and bring us the brightest future. If this book serves as a trigger, that is great. Last, but most important, I would like to thank from the bottom of his heart all chapter authors who squeezed their busy time to contribute to this book. And I would like to thank Mr. Anthony Doyle, Mr. Rajan Muthu and Ms. Ritu Chandwani, Springer for making this publication possible. Tokyo, Japan November 2019

Shuichi Fukuda

Contents

1

Instinct Engineering: What We Learn from Invertebrate . . . . . . . . Shuichi Fukuda

1

2

From Work for Others to Work for Yourself . . . . . . . . . . . . . . . . . Shuichi Fukuda

13

3

The Relationship Between Attributes of Objects and Phonemes of Naming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuri Hamada and Hiroko Shoji

4

Computational Handicraft . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuki Igarashi

5

Sustainable Behaviour: A Framework for the Design of Products for Behaviour Change . . . . . . . . . . . . . . . . . . . . . . . . . Giulia Wally Scurati, Marina Carulli, Francesco Ferrise and Monica Bordegoni

6

Quantifying Trust Perception to Enable Design for Connectivity in Cyber-Physical-Social Systems . . . . . . . . . . . . . Yan Wang

27 43

65

85

7

A Study of “Waku-Waku” at Work . . . . . . . . . . . . . . . . . . . . . . . . 115 Ryotaro Inoue and Takashi Maeno

8

The Romantic Brain: Secure Attachment Activates the Brainstem Centers of Well-Being . . . . . . . . . . . . . . . . . . . . . . . 135 Yoshiaki Kikuchi and Madoka Noriuchi

9

A Neural Model of Aesthetic Preference for Product . . . . . . . . . . . 149 Kazutaka Ueda

vii

viii

Contents

10 Emotional Design: Discovering Emotions Across Cars’ Morphologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 Egon Ostrosi, Jean-Bernard Bluntzer, Zaifang Zhang, Josip Stjepandić, Bernard Mignot and Hugues Baume 11 Modeling of Aesthetic Curves and Surfaces for Industrial Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 Ichiroh Kanaya and Keiko Yamamoto 12 A Mathematical Model of Emotions for Novelty . . . . . . . . . . . . . . . 191 Hideyoshi Yanagisawa 13 The Relation Between Characteristics of Forest Sounds and Psychological Impression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 Terutaka Yana and Takashi Maeno 14 Artificial Intelligence and Virtual Reality-Based Kansei, Emotional, and Cognitive Science and Engineering . . . . . . . . . . . . 215 Keiichi Watanuki 15 Development of an LED Lighting System Through Evaluation of Legibility and Visual Fatigue . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 Keiichi Muramatsu, Keiichi Watanuki, Naoya Mashiko, Yoichiro Watanabe and Masutsugu Tasaki

Chapter 1

Instinct Engineering: What We Learn from Invertebrate Shuichi Fukuda

Abstract Our world is changing from the closed world with boundaries to open world and changes occur frequently, extensively, and unpredictably. Thus, our world is rapidly shifting from explicit and verbal to tacit and nonverbal. We have been focusing our attention on products and their functions. But today we need to adapt to such changes. Thus, adaptability becomes most important. Until now, knowledge has been considered important. But knowledge is concept, and concept varies from person to person. Take music for example. It is represented by a musical score, but music is played differently from player to player. The music itself is analog. The real-world is analog. We need more direct interaction with the analog real-world to make adequate decisions. Invertebrate survive on instinct alone. But we forgot to utilize instinct. To interact with the real-world directly and to adapt more adequately, we must make efforts to make the most of our instincts. Then, we can open the door to the world of Wisdom Engineering.

1.1 How Engineering Has Developed Figure 1.1 shows how engineering has developed. Yesterday, our world was closed with boundaries. Therefore, we could apply explicit and verbal rational approach in a straightforward manner. But the world expanded rapidly and the boundaries disappeared. Thus, our world became an open world without boundaries (Fig. 1.2). Engineers were forced to think of a new way to apply a rational approach. They came to realize how they identify the name of a river. If we look at the water, it is changing every minute, so we cannot identify its name. But if we look around, there are mountains and forests, so we can identify its name (Fig. 1.3). Engineers take notice of this idea and they developed System Identification. They realized if they can identify feature points or feature parts, then they can identify the system. Thus, they succeeded in establishing Controllable World (Fig. 1.4). S. Fukuda (B) Keio University, System Design and Management Research Institute, Tokyo, Japan e-mail: [email protected] © Springer Nature Switzerland AG 2020 S. Fukuda (ed.), Emotional Engineering, Vol. 8, https://doi.org/10.1007/978-3-030-38360-2_1

1

2

Fig. 1.1 Engineering yesterday, today, and tomorrow Fig. 1.2 Closed world to open world

Fig. 1.3 How do we identify the name of a river?

S. Fukuda

1 Instinct Engineering: What We Learn from Invertebrate

3

Fig. 1.4 Rational world and controllable world

But rapid expansion of our world brought diversification and personalization accompanied with it. Diversification progressed rapidly and our world soon became nonverbal and even System Identification approach cannot be applied anymore. In short, we are now swimming in the river and we need to adapt to the changing environments and situations. In other words, our world became not only open without boundaries, but it changes very frequently and extensively. And to make the problem difficult, changes themselves changed. Yesterday there were changes, but these changes were smooth so that we can differentiate them. Thus, we can predict the future. Engineers could foresee the operating or use conditions of their products. Thus, final products and their functions were most important. Yesterday, the product value was most important. But today changes change sharply. So, we cannot differentiate them, and we cannot predict the future. What becomes important today is how flexibly we can adapt to these unpredictable changes. To describe it another way, Process becomes important. Now, Processes become more important than Products. Therefore, Adaptability becomes a key word now (Fig. 1.5).

1.2 Challenges of Engineering Today Up to now, the idea of System Identification has been effective. Our approach has been model-based. But today, environments and situations change frequently, extensively, and unpredictably. Therefore, we are now in the flow. We need to swim against the water by trial and error to reach our goal. Swimming is one of human movements. But, let us consider human movements on the ground first for simplicity. Nikolai Bernstein is very famous for human motion control. Figure 1.6 shows his cyclogram of hammering. He is very famous for human motion control. What troubled him was human motion trajectories vary widely. Thus, the number of degrees of freedom is tremendously large. But as his cyclogram shows, our motion trajectory is fixed when we

4

S. Fukuda

Fig. 1.5 Changes yesterday and today

Fig. 1.6 Cyclogram of hammering

approach the target. Once decision is made, then our trajectory is fixed, and the number of degrees of freedom is reduced to the minimum. Why our trajectories vary widely at the first portion of our motion is because we coordinate and balance our body parts to adapt to the environment and situation [1, 2]. What makes the problem further difficult is motion trajectories vary not only regarding one human but from human to human, because our body builds vary from person to person. And strangely enough, our movements are called by different names, depending on whether we observed it from outside or from inside. When we observed our movement from outside, we call it Motion Control. And from inside, we call it

1 Instinct Engineering: What We Learn from Invertebrate

5

Motor Control. What Bernstein did was the study of motion control. As to motor control, there are many researches on individual topics, but how we control our body movement from inside as a whole is not clear at all. Anyway, we need to integrate motion control and motor control to make it clear how we coordinate and balance our body parts to adapt to the environment and situation. We, however, have no other choice than to do it by trial and error. There is no model.

1.3 Engineering Tomorrow Tomorrow, our world will change drastically. We have been distinguishing clearly living things from nonliving things. But Kevin Ashton proposed the idea of IoT (Internet of Things), where there is no distinction between living things and nonliving things. Humans and machines work together on the same team. However, most discussions about IoT are going on from the standpoint of supply chain in the current industrial framework. They focus on sensing technologies or Internet technologies. Very few discussions are made about teamworking where living things and nonliving things work together on the same team. We must remember what makes IoT revolutionary is that it removes the wall between living things and nonliving things. Then, what is Life? We think we can distinguish between living things and nonliving things easily. Is it true we can? Consider crystals; they grow. Consider materials (machines, etc.); they deteriorate; they are aging. And we say “break-in” or “grow on” when materials come to fit us. Asics, shoes maker in Japan noted that shoes deform as shown in the middle in Fig. 1.7 after we wear them for a long time. Usually, these worn out shoes are just thrown away and shoes makers keep on producing the shoes in the same, traditional way. But Asics noted that if the shoes deform this way, and if they can make the middle part more flexible and adaptable, then people can walk more comfortably. Yes, indeed. The shoes Asics developed this way as shown at left were welcomed

Fig. 1.7 Asics shoes

6

S. Fukuda

so much by people. They say “I don’t feel like walking with the shoes on. I just feel like walking on my own feet. Now, I can enjoy walking a long distance.” Asics observed the same deformation at the start of a race (as shown at right) in Fig. 1.7. So, they produced sports shoes for athletes, and they were also welcomed and appreciated very much. This is achieved through Communication between human and material, i.e., Things communicate and work together as a team. This is not through Internet, but if we consider Internet is a means of communication, IoT can be described as “Things communicate and work together on the same team to adapt to the changing environments and situations.” So, this is nothing other than IoT.

1.4 From Control to Coordination: Increasing Importance of Decision-Making The idea of control has contributed so much to the progress of engineering. But we must note that Control is focused on How. Yesterday, environments and situations did not change appreciably, and changes were smooth, so that we could differentiate them and predict the operating conditions of machines. Therefore, we could focus on products and their functions. How they can perform their functions was most important. Although our world became open without boundaries, we could still apply rational approaches by introducing System Identification approach. Once we identified the system, we could control the parameters to achieve our goal. In other words, the goal is there, and we develop the better How. But today, our world expanded tremendously, and environments and situations change so much. Therefore, we need to swim in the flow of changes. Thus, Adaptability becomes the key word. We need to Perceive what environments we are in and make Decisions on What actions or movements we should take to adapt to the changing environments and situation, which change sharply so that we cannot predict the future anymore. We must make ourselves clear What goal we are going to achieve. So, today, What becomes crucially important. Thus, as economists points out Decision-Making will be the next sector of industry. They call this sector Quinary Sector as shown in Table 1.1. Table 1.1 Five sectors of industry

Sector

Activities

5 Quinary sector

Decision-making

4 Quinary sector

Knowledge and ICT industry

3 Quinary sector

Service industry

2 Quinary sector

Transforms raw materials into products—manufacturing, etc.

1 Quinary sector

Extracts raw materials from nature—agriculture, fishing, etc.

1 Instinct Engineering: What We Learn from Invertebrate

7

It should be emphasized that Decision-Making needs Situational and Environmental Awareness. And instead of Control, we need Coordination. This can be easily understood if we look at Bernstein’s cyclogram (Fig. 1.6). In the first portion, we perceive the environment and the situation and coordinate our body parts by trial and error. But once we identify the target, we concentrate on How we can stabilize our trajectory. Thus, we can say that our movement is first looking out for What and then once What is identified, we concentrate on How. Balancing is associated with both Control and Coordination. We balance our body to identify What and then to identify How. The words by William L. Megginson, Professor of Finance at University of Oklahoma may remind us of how our world is changing. He pointed out, “Those who can survive are not those who are wise or strong, but those who can adapt to the changes.” In the age of frequent, extensive, and unpredictable changes, Things need to collaborate to adapt to the changing environments and situations. And indeed, if we define Life this way, there is no distinction between humans and materials. All are Things and although their life may be different, they live their own life.

1.5 Vertebrate and Invertebrate Such big changes are expanding our research areas rapidly. Cecilia Laschi initiated Robosoft in 2018 [3, 4]. She insists that we should pay more attention to direct interaction with the outside world. She points out we should learn from the octopus. Octopus parents die soon after their baby is born. Thus, knowledge or experience is not transferred from parents to their children. The octopus must live on their instinct alone. But the octopus is called by another name, “Expert of Escape,” They can escape from any environments or situations. The most famous case is that they can escape from the screwed container. If a human is put into the screwed container, he or she would be panicked and may not be able to escape from there. But the octopus can. The octopus represents the invertebrate and humans belong to vertebrate. Figure 1.8 shows octopus and human baby. Unlike other invertebrates, the octopus has a large brain. But as knowledge is not transferred from generation to generation, this brain is used for manipulating their arms to interact directly with the outside world. The octopus has eight arms, so the brain is used to coordinate and balance them. Humans have large brains, but it is filled with tremendous amount of knowledge, which is the structured accumulation of experience and which is transferred from the previous generations. But we must be aware that knowledge does not represent the real-world. Betty Edwards, a famous sketch drawer in the US, pointed out [5] that until 7 years old, we draw sketches of the real-world. But after 7 years old, we draw sketches based on our concepts. For example, children under 7 years old draw a car from the front. So, we cannot tell it is a human face or the front of a car. But children after 7 years old

8

S. Fukuda

Fig. 1.8 Octopus and human

draw a car from side. So, we easily understand it is a car. Our eyes move horizontally so that we can easily recognize it is a car. Antoine de Saint Exupery published “The Little Prince” [6]. The Little Prince draws a picture of a snake swallowing an elephant, but the adults say “It is a hat.” Those after 7 years old do not see the real-world. They look at the world as concepts. We can demonstrate how concepts are dominating our brain. Give a seminar and after some time, tell the participants to get out of the room. And ask them if there was a clock in the room. Half of them do no remember, but other half answers yes. Then ask this half about the clock, its shape, its dial, etc. Surprisingly enough, only few people can answer exactly. The other people know that there was a clock in the room, but they recognized it as a time-telling machine, and do not recognize it as it is. If we consider that the space of our brain is limited, this is very natural and convincing. Further, we must be aware that these concepts vary from person to person. When we say knowledge, most of us think that it represents the real-world and that the same knowledge is shared among us. No, knowledge varies from person to person, because how we convert the real-world to concepts varies from person to person. We think that the digital world represents our concepts precisely. But this is not true. Let us consider music. Music is recorded by a musical score. This is a digital representation. If digital symbols define our world uniquely, then music will be played the same way, no matter who plays. But as we know well, music varies from conductor to conductor, from orchestra to orchestra and from player to player. And listeners enjoy the same music in different ways. Yes, music is real and analog, and we are enjoying the real-world. Invertebrate lives their life on their instinct alone. They interact directly with the outside, real-world and make decisions on how to adapt to the changing environments and situations. As they do not inherit knowledge, there is no other way. What Cecilia Laschi points out is that we need to interact directly with the realworld. Regrettably enough, we, humans, forgot to utilize our instincts. Let us take our movement for example. As described earlier in Bernstein’s cyclogram (Fig. 1.6), our

1 Instinct Engineering: What We Learn from Invertebrate

9

motion trajectories vary widely at first. This is because we coordinate and balance our body parts to adapt to the environment and situation. But we cannot develop any models for this portion. We need to coordinate and balance our body parts by trial and error. And this is done entirely by our instincts.

1.6 Increasing Importance of Communication Let us change the topic to understand how Teamworking and Communication become crucially important. As environments and situations change frequently, extensively, and unpredictably, we need to work together as a team. Two heads are better than one. This holds true with machines as well. Let us consider how communication becomes important in an age of teamworking, taking American football and soccer as an example. American football and soccer are both a team game with 11 players. Knute Rockne, American football player and coach, left the famous word “11 Best, Best 11.” What he pointed out is that even if we have 11 best players, we cannot make up the best team. To make up the best team, we need 11 players who understand the current game situation and take the necessary teamworking strategy. He demonstrated this by bringing up the University of Notre Dame from the bottom to ever-winning team. In soccer, Franz Beckenbauer, Der Kaiser, introduced Libero System. Until then, formation did not change during the game because the game did not change much. Thus, every player was expected to play best at his own position. This was truly the world of 11 Best. But Beckenbauer realized that games came to change very frequently and extensively. So, he introduced Libero system to adapt to the changes more flexibly and more adequately. He knew the position of a midfielder enables him to see and understand most accurately how the game is changing. So, he introduced playing manager system and now one of the midfielders plays this role. Until then, managers were outside of the pitch and gave instructions. But today, managers are on the pitch, playing together (Fig. 1.9). This management style is the same as in machine–human relation. Yesterday, humans are outside of the machine system and gave instructions. And machines responded to them. But today, humans work together with machines. Otherwise, they cannot understand the current changing situations.

Fig. 1.9 Outside of the system to inside the system

10

S. Fukuda

IoT is one step further. This team is composed of living and nonliving things and they work together on the same team. In short, today we need to interact directly with the frequently and extensively changing real-world.

1.7 Mind–Body–Brain In English, we say “Make up your mind,” but we do not say “Make up your brain.” Let us consider the problem of Mind–Body–Brain. Mind–Body–Brain can be illustrated as in Fig. 1.10 Our body interacts directly with the outside world. But this is the issue of Exteroception. We must remember the role of Interoception, too. In human movement control, we distinguish between motion control, which is observed from outside; and motor control, which relates to the inside of our body. Indeed, invertebrates interact with the outside world directly using their instinct, but this is Motion Control. We, humans, also interact with the outside world, but we use Motor Control and Motion Control in an integrated way. We work Exteroception and Interoception together. These exteroceptive and interoceptive senses constitute our Mind. When we talk about our instinct and try to utilize instinct more in engineering, we need to remember this. We need to pay attention not only to exteroceptive awareness but also to interoceptive awareness. Brain does not react immediately. It takes time to really communicate. In robotics, it is made clear that if robots react immediately, humans do not feel right. If their reaction takes some time, then humans feel satisfied and they rely on robots. This is considered due to the time our brain needs to process the stimuli. We often think that mind = brain, but this is not correct. We must remember some of our interactions with the outside world are immediate, such as the reflex, which is the response of our body to the outside world. And when we talk about brains, its digital processing attracts wide attention these days, but we must remember that analog elements play a very important role in our Fig. 1.10 Mind–Body– Brain

1 Instinct Engineering: What We Learn from Invertebrate

11

brain. Blood plays a very important role in our brain, which is analog. When our brain stops digital information processing, we call it “brain death.” But it is not really the death of our brain. Even when it cannot process digital information anymore, still it keeps other body elements alive by circulating blood. So, if we keep flowing blood in the brain, we can transplant other organs to somebody else, if within three hours after the brain death. We need to realize here again our real-world is analog.

References 1. 2. 3. 4. 5.

Bernstein NA (1967) The co-ordination and regulation of movements. Pergamon Press, Oxford https://en.wikipedia.org/wiki/Nikolai_Bernstein https://ieeexplore.ieee.org/xpl/conhome/8396915/proceeding https://www.youtube.com/watch?v=mPp3Ks7xfFs Edwards B (1979) Drawing on the right side of the brain: a course in enhancing creativity and artistic confidence. JP Tarcher, St. Martin’s Press, Los Angeles, New York 6. de Saint Exupery A (1943) The little prince. Reynal and Hitchcock, New York

Chapter 2

From Work for Others to Work for Yourself Shuichi Fukuda

Abstract The Industrial Revolution realized a great advance in engineering. But it pursued product value, and efficiency and mass production were evaluated. To achieve this goal, it introduced Division of Labor. Thus, we came to work for others. But if we look back, engineering started to make our dreams come true and we made efforts to realize it by trial and error. This is a challenge. Challenge is the core and mainspring of all human activities. We enjoy challenges. That is why we love games. However, the Industrial Revolution deprived us of such a joy of challenge. We work for others based on instructions. In a way, we are regarded as something like machines. Further, it consumes tremendous amount of resources (energy, etc.). Thus, how we can sustain development becomes crucially important. But if we start to work for ourselves and for pleasure, we can revive self-sufficient society in a new style. This will solve many problems facing us, such as lack of labor force (aging society), limited resources, too much energy consumption, etc. And this is nothing other than Self-Actualization and Self-Determination. It will enable us to enjoy the highest level of happiness and satisfaction.

2.1 Engineering Yesterday and Today Engineering yesterday and today are compared in Table 2.1. This is described in Chap. 1, but here let us discuss the issue again to grasp the whole picture more clearly. Yesterday, as environments and situations did not change appreciably, and even if they changed, the changes were smooth. So, we could differentiate them and predict the future. Therefore, engineers could set their goals precisely and focus on final products. In other words, industry or engineering yesterday was goal-fixed and product-focused. As the goal is clear, how we get to the goal, i.e., efficiency was regarded most important. Again, such a rational approach is possible because our world yesterday was explicit and verbal. We knew what parameters we should S. Fukuda (B) Keio University, System Design and Management Research Institute, Tokyo, Japan e-mail: [email protected] © Springer Nature Switzerland AG 2020 S. Fukuda (ed.), Emotional Engineering, Vol. 8, https://doi.org/10.1007/978-3-030-38360-2_2

13

14 Table 2.1 Engineering yesterday and today

S. Fukuda Yesterday

Today

Fixed goal

Goal finding

Product

Process

How

What

Efficiency

Adaptability

Explicit

Tacit

Verbal

Nonverbal

Rational

Trial and error

Control

Coordination decision-making

Knowledge

Wisdom

Motion control

Motor control + motion control

consider so we could control them. We noted motion part of our movement and controlled them. Yes, engineering yesterday was knowledge-centric. But as the twentieth century progressed, our world came to expand very rapidly and extensively. Our world became a tacit and nonverbal world. Therefore, what became most important was to find the goal. We must know what we should do. In other words, we need to adapt to the frequently, extensively and unpredictably changing environments and situations. Thus, adaptability became most important, but since our world is tacit and nonverbal, we have no other ways than to achieve that by trial and error. Yesterday, the goal was clear so that we did not need to make decisions on what actions we should take. We could concentrate on improving how. But now we need to find out what to do. And the number of tools for improving how was limited yesterday. But today, there are so many tools for many different purposes and applications, so we need to find out which works best and what combination of them works best. In other words, we need to coordinate many different tools to get the best result. This is what our body is doing. Our body mobilizes many different body parts, depending on environments and situations. Coordination is crucial in our body movement. This holds true with the case of selecting appropriate tools to adapt to the changing environments and situations. Therefore, the control of our body movement cannot be carried out by only motion control, but motion and motor control.

2.2 Engineering Before and After the Industrial Revolution Let us study here from another perspective how the Industrial Revolution changed our engineering or industry (Table 2.2). The Industrial Revolution from the eighteenth to the nineteenth century really revolutionized our world. Not only engineering advanced amazingly, but it changed

2 From Work for Others to Work for Yourself

15

Table 2.2 Engineering before and after the industrial revolution The Industrial Revolution

Before

After

Work for

Yourself

Others Division of Labor

Develop

Your product

Components parts

Product

One of a kind

Mass production

Basic idea

Holism

Reductionism

Motivation

Internal self

External

Human needs

Self-actualization highest needs in Maslow’s hierarchy

Bottom layers in Maslow’s hierarchy

Satisfaction

Mental

Material

the framework of our society. Until then, we worked for ourselves. We designed and manufactured products and exchanged products for products. At first glance, this does not seem different from the current product-based engineering. But at that time, we knew what we were designing and what we are producing. We developed what we have on our minds. But the Industrial Revolution brought us the Division of Labor. We came to work for others without much knowledge about the final products. As Table 2.2 shows, before the Industrial Revolution, we were developing products on our own initiative with full knowledge about what we are producing. So, it may be said our approach at that time was holistic. But after the Industrial Revolution, Division of Labor was introduced, and we came to manufacture components, not the whole product. Of course, we know what the final product will be, but we do not know exactly about it. We only have a rough idea of it. This is because the production system became reductionistic. If components or parts are produced correctly, then these components or parts can be assembled into a system or the whole product. As the reductionistic approach accelerates efficiency and uniformity, it led to mass production. Products came to be produced in mass. Not only components or parts but the final products came to be produced in mass. Although this industrial system looks very efficient and the quality of final products is guaranteed. This industrial system lacks flexibility. Such a reductionistic system is a linear system as shown in Fig. 2.1 and its structure is a tree (Fig. 2.2) because every member has his own task and by putting these tasks together, the final Fig. 2.1 Linear production system

16

S. Fukuda

Fig. 2.2 Tree

product is realized. And what final product they produce is fixed. A tree structure works best for such a purpose. It only has one output node and it divides labor into hierarchy. Although our world expanded rapidly, we expanded Rational World to Controllable World, by introducing the idea of System Identification as described in Chap. 1. But recently, the expansion of our world is accelerated incredibly, so we cannot apply System Identification approach anymore. We are getting out of the Rational World and now entering the Tacit World. To put it figuratively, we are no more on the bank, but we are now in the midst of a river. We need to swim against the water to get to our goal. But we cannot identify what parameters we should control, so we have no other choice than to swim by trial and error. As described in Chap. 1, to cope with today’s frequent, extensive and unpredictable changes, we must shift from 11 Best to Best 11. In other words, we need to shift from a tree structure to a network structure (Fig. 2.3). While a tree structure has only one output node, all nodes in a network can be an output node. So, if we shift to a network, we can be more flexible and adaptive. But to increase flexibility and adaptability, nodes are expected to have as many links as possible, so that they can change from one network to another adaptively in

2 From Work for Others to Work for Yourself

17

Fig. 2.3 Network

response to the changes of the environments and situations. In a nutshell, today is the age of Adaptive Network. As described in the next section, this Adaptive Network must be truly adaptive. It does not mean adaptive with a fixed network structure, but with a structure which varies largely from case to case. And it is our big challenge how we can develop such a truly adaptive network. This will be discussed in detail in the next section. Now, let us consider how such changes brought by the Industrial Revolution affect us psychologically. Before the Industrial Revolution, our activities are basically all intrinsic motivated and we made decisions on our own. But after the Industrial Revolution, we do the jobs by external instructions. American psychologists Edward Deci and Robert Ryan proposed SelfDetermination Theory [1, 2]. They made clear that if we make decisions ourselves and do the job on our intrinsic motivation, we feel happiest. No matter how much reward we may be provided from outside, we cannot get such amount of happiness and satisfaction. Before the Industrial Revolution, we made decisions and carried out activities by ourselves. Thus, we are most satisfied in the psychological sense. But after the Industrial Revolution, all jobs are given from outside, so we are not fully satisfied from the standpoint of psychology. Let us look at this shift from internal motivation to external motivation from another perspective. Abraham Maslow proposed Hierarchy of Human Needs [3] (Fig. 2.4). At the bottom level, we look for material satisfaction, but as we climb up the ladder, we come to look for mental satisfaction. And at the highest level, we look for Self-Actualization. We would like to demonstrate how capable we are. For example, mountain climbers choose a more difficult route and the more difficult the route is, the happier they feel. In fact, challenge is the core and mainspring of all human activities. When our challenge is successful, we feel very happy and satisfied. In fact, this is why we love games.

18

S. Fukuda

Fig. 2.4 Maslow’s hierarchy of human needs

We also must note that at the bottom level, the needs do not vary much from person to person. Thus, we can produce products in mass and satisfy people’s needs as mass. But as we go up the ladder, our needs become more and more individual-focused and at the top, it is the need of yourself, completely individual-based. What challenge or how we challenge varies from person to person. Thus, it should be emphasized again that Self-Actualization varies from person to person. In other words, our needs shifts from uniformity to diversity, as we go up. Thus, Self-Actualization not only satisfies our own needs, but it must also be emphasized that it satisfies the need as a human to expand our world of human species. If there are diverse challenges, then the possibility increases that there are cases which work to adapt to the environments and situations better than others and these cases will work as a trigger to expand our world of human species. This is Evolution. Thus, Self-Actualization is deeply associated with Evolution. This reminds us of the importance of “Work for Yourself.” Since the Industrial Revolution, our engineering or industry has developed based on “Work for Others” principle. But it looks it is time we need to develop the Industrial Revolution 2.0, where we can enjoy more as an individual and at the same time we can contribute to the evolution of human species.

2.3 Challenges of the Real IoT The Industrial Revolution indeed realized a surprising progress in engineering. But what has been lacking is the balancing or coordinating sense in industry and engineering. For example, AI (Artificial Intelligence) is getting wide attention these days. But the basic theories used in Deep Learning today were developed about 50 years ago. Then, why does AI come to be so effective in applications. The primary reason is not because of software, i.e., Deep Learning theories developed highly, but because of

2 From Work for Others to Work for Yourself

19

hardware, i.e., computers come to be produced with far less cost and with far more capabilities. But we must remember that even though the cost of running a computer becomes so much small today, the cost of applying Deep Learning approach to solve a problem costs 100 times more than human labor. Deep Learning is effective when the goal is fixed. But with the rapid expansion of our world, it becomes more important to find out What to do. Indeed, How to do is important. But it comes after What or the goal is fixed. Regarding the search, Deep Learning is very effective, but it does not help us make decisions on what to do or to find the goal. And we should also remember that our society is quickly shifting from 11 Best to Best 11. That is to say, the structure of our society is shifting from a tree to a network. We do not need many best players. We need diverse players with diverse capabilities so that they can tie up with other players in different formation in a very flexible and adaptive manner. The above discussion about teamworking can be illustrated as shown in Fig. 2.5. Although the structure of our industrial society shifted from a tree to a network, or to be precise, an adaptive network. In most cases, the current teamworking is sport type with the number of players fixed and with strict rules. Thus, it is a complete information game, so the current AI is very effective. But our world is quickly shifting to a society of variable number of players with no rules at all. We must invent a team truly adaptive to the changing environments and situations without being bounded by hard constraints. Every constraint is now changed to negotiable. This is the world of the real IoT, where even the wall between living and nonliving things disappear. So, our big challenge is how we can organize and manage such truly adaptive team or network. Fig. 2.5 Teamworking

20

S. Fukuda

2.4 Self-Sufficient Society Until the Industrial Revolution, our society had been Self-Sufficient. We designed and manufactured our own products. But we should note not only our production system was Self-Sufficient, but our ecosystem was. Earthworms strongly attract robotics researchers’ attention these days because it can move freely in a very soft, unstructured environments. But we must note that they contributed so much to the disposal of wastes, i.e., composts. Without them, our ecosystem could not have been maintained. But such great contributions of the earthworms are completely forgotten. We only regard them as a good model of a mobile robot. But the fact that SDGs are emerging, we need to pay more attention to how we can dispose of wastes and keep our ecosystem sustainable. We should learn from the earthworms about it, too. Then, can we return to our old Self-Sufficient Society before the Industrial Revolution? Yes, we can. If we change our society from product-centric and efficiencyfocused to process-centric and satisfaction-focused. We should pay more attention to psychology and we should pay more attention to how we can increase our satisfaction. As Maslow pointed out, we looked for material satisfaction yesterday. We looked out for better products with better functions. And these expectations did not vary much from person to person. So, products were produced in mass. But today, with its rapid expansion, our world becomes more and more diversified and personalized. We are now looking for mental satisfaction. But our society is still material- or product-focused. But if we note the highest need of humans at the top of Maslow’s hierarchy is Self-Actualization, we should develop engineering which help realize Self-Actualization. Frankly, we got used to the life of working for others too much. We forgot completely about the world of working for ourselves. But if we can revive the SelfSufficient Society, then we can reduce wastes to a great extent and can contribute to Sustainable Development very much.

2.5 Sustainable Development The word “Development” has been regarded as improvements in product functions since the Industrial Revolution. But we should note Development and Growth are closely associated in their meanings and in their etymologies as well. Deci and Ryan pointed out that in addition to the importance of Self-Determination, the need for Growth is very important for humans. If we can revive Self-Sustaining Society, we can realize such a society where the need for Self-Determination and Growth are satisfied. Then, we can enjoy life much more. Then, how can we realize such Self-Sustaining Society in the new framework? It is the new society for “Work for yourself.”

2 From Work for Others to Work for Yourself

21

2.6 Work for Yourself This can be realized if we change our engineering to much simpler one, which would allow everybody to participate in production. Take 3D Printing, for example. Many efforts are now being paid to utilize this technology to move the current industry more and further ahead. But if we remember Chris Anderson’s “Makers: The New Industrial Revolution” [4], we immediately realize this technology can be used for another purpose. It will enable us to enjoy producing what we want. In fact, our engineering education is focused on growing up excellent chefs. We develop excellent tools and excellent materials. So, our engineering products are restaurant dishes. But do we take restaurant dishes every day? Would this happiness continue? No. If we can produce good tasting dish from the leftovers in the refrigerator, that would give us more satisfaction and we can keep this happiness from day-to-day, by devising new dishes. Yes, restaurant dishes are very good. But we evaluate it as a product. But when we invent new fusion dishes out of the leftovers, we enjoy the processes. If we realize that the process value is increasing its importance, we can realize a new Self-Sustaining Society, which makes our life satisfactory and full of happiness, not with products, but with enjoyable processes.

2.7 Changing Industrial Framework Recently modularization is introduced to cope with the quickly increasing diversification and personalization. Therefore, the industrial framework shifted from a linear system to a network system as shown in Fig. 2.6. Yesterday, available materials were limited so that engineers had no choice but to select material among such limited variety, and final products are clearly defined so

Fig. 2.6 Traditional to recent industrial framework

22

S. Fukuda

that the system was linear. But now material engineering advanced so much that it can provide a very wide variety of materials. And by the introduction of modularization, the industry framework has shifted to a network. In fact, it is a neural network. Engineers can combine many different modules and come up with many different final products. Although at present, such jobs of combining modules into final products are left to experts, but if we can make the technology simple, everybody can enjoy putting different modules together and realize final products. Yesterday, a large factory and many people were needed to produce final products. And as a system is linear, how we can get to the goal, i.e., to the final product was very important. But in the network framework, jobs can be processed in parallel, so that the great reduction of manufacturing time can be achieved. In other words, today’s industry framework realized true concurrent engineering. As intermediate component companies do not need a large factory, they can produce their components or parts in a small factory with small number of people. And it should also be noted that we do not need many number of experts. A small number of experts who can take care of particular kinds of components or parts are sufficient. But they can produce their components or parts in mass because these components or parts can be utilized in many different final products. Thus, they can keep on mass production by expanding their markets. This is the same as we see in the electronics field. We buy parts and assemble them into a final system. Thus, if the assembly does not take too much expertise, we can all buy necessary intermediate components or parts and realize the final product. In fact, EV will be a typical example. Figure 2.7 shows Daihatsu Copen. This is not EV, but it is decomposable.

Fig. 2.7 Decomposable vehicle

2 From Work for Others to Work for Yourself

23

2.8 Work for Your Happiness: Its Benefits The words Labor and Work entails impression of difficulty, hardship or torture as their etymologies indicate. But if we can develop simple technologies, then the impression of the word Work would change drastically. It would mean our efforts to develop our happiness. And the word Labor Force will fade away because even seniors can enjoy assembling parts to realize what they want. And everybody can satisfy his or her own need or can pursue what he or she wants in his or her own way. This makes everybody happy and satisfied. The problem of aging is becoming very serious in advanced countries because Labor Force is rapidly decreasing. But if the industry framework changes from “Work for others” to “Work for yourself or for pleasure,” then everybody can produce what he or she needs or wants so that there will be no more problem of shortage of Labor Force. We will be realizing Self-Sufficient Living. Besides, large factories are no more needed so that large investments are no more necessary. And training many expertise is no more necessary. Only small number of experts in each field are sufficient. Thus, the cost or investment necessary to maintain our society drops down drastically. Therefore, we can realize Sustainable Development. It should also be emphasized that if we develop simple technologies, we can also revive our old style of doing design-manufacture all the way by ourselves. This is truly a revival of the old age, but in a different sense. Realizing a final product will not be our only pleasure, but we will be enjoying the processes themselves. We may be able to enjoy serendipity as well.

2.9 Enjoyable Engineering: From Needs to Joy We have discussed the importance of shifting our society from “Work for others” to “Work for yourself” or “Work for Pleasure.” Let us discuss here the same topic from another perspective. Our engineering started to make our dream come true. But as Maslow pointed out we needed to satisfy our needs first. We need food, housing, etc., to survive. This stage of satisfying our basic needs lasted so many years, but as we came to satisfy our basic needs, our needs quickly increased in sophistication or it might be better to say, our wants became quickly extravagant. This call for higher and better functions led to the Industrial Revolution. But we must remember that in the days when we designed and produced what we want, we could enjoy the processes, no matter what quality the final products may be. If the quality was very good, we were very happy, of course. But even if the quality was not so good and it betrayed our expectations, we were still happy because we enjoyed the time and processes to realize it. If the quality was not so good, we could try again and we could do better next time.

24

S. Fukuda

Another change of our world after the Industrial Revolution is we came to evaluate everything from the standpoint that the shorter time it consumes in production, the better. We would like to have our final product as early as possible because what came to count is the product value and the value of processes came to be forgotten. In short, efficiency became most important. So, we did not want to waste time and we wanted to identify the goal as early as possible and get there as fast as possible. To fulfill this, we introduced System Identification approach and we succeeded in establishing Controllable World. However, if we look at this Controllable World from a very different perspective, it is nothing else but putting ourselves in a cage. Although the world is rapidly moving into an open world without boundaries, we confine ourselves in a small cage. We needed a frame with boundaries to control the problem. In other words, we prepare a cage which meets our requirements, i.e., we introduce constraints to solve the problem easily and effectively. If we look back, our engineering was done in a wide, open world. We worked in nature. We have no boundaries and trials and errors themselves were a joy. So, what we should do is to revive such a world of joy. To satisfy our needs is fine, but we should not stick to just satisfying the needs. We should enjoy engineering. We should move away from just satisfying requirements. We should enjoy the wide, open world (Fig. 2.8). Finally, I would like to add one more. As I am Japanese, let me point out that Japanese culture is process-value-rich. In Japan, there are many do’s (pronounced dough). These do’s provides pleasure and satisfaction by doing (pronounced doo). For example, flower arrangement is very popular among Japanese ladies (Fig. 2.9). There are three famous do’s in Japan. Flower Arrangement (Ka-do), Tea Ceremony (Sa-do), and Incense Burning Ceremony (Kou-do (pronounced Kohdough)). Tea Ceremony and Incense Burning Ceremony are popular among men as well. These three are typical do’s in Japanese culture. What characterizes them is that they do not emphasize the outcomes, i.e., What you achieve. They emphasize How you do it. They emphasize Learning by Doing. Yes, they provide a title when

Fig. 2.8 Cage to nature

2 From Work for Others to Work for Yourself

25

Fig. 2.9 Flower arrangement (Ka-do)

you reach a certain level. But these titles are given for your efforts for doing. Not based on your achievements or outcomes. Indeed, if you would like to have flowers arranged beautifully, you can ask experts. It would be far less expensive, and it will be done in much shorter time. But in flower arrangement, getting the result in a short time with less cost is not encouraged at all. How you try and make errors and learn by yourself are evaluated. In Japan, there are many other do’s. The most famous one may be Ju-do. Although this is a sport, Ju-do emphasizes Learning by Doing. You are supposed to learn by yourself. They do not teach how to win in a straightforward manner. So, in Ju-do how you fight is more important than the result. In addition to many do’s, Japanese culture is rich with process values. For example, let us take up Kimono. Modularization is getting wide attention these days. But Kimono introduced the idea of modularization long time ago. Kimono is composed of four fabrics (Fig. 2.10). By making them over and by re-dyeing them, a grandchild wears the kimono her grandmother wore cheerfully because it is no more an old-fashioned Kimono. It is now a cutting-edge wear. And it should also be mentioned that Kimono is hanging clothing, while western wears are close-contact clothing. Thus, the same kimono can be worn by many different people, no matter what body build they are. Thus, Kimono is very much versatile. There are many other examples. I would like to emphasize that we can learn many things from traditional Japanese culture to develop Enjoyable Engineering.

26

S. Fukuda

Fig. 2.10 Kimono

References 1. Deci EL, Ryan RM (1985) Intrinsic motivation and self-determination in human behaviour. Plenum, New York 2. https://en.wikipedia.org/wiki/Self-determination_theory 3. http://psychclassics.yorku.ca/Maslow/motivation.htm 4. Anderson C (2012) Makers: the new industrial revolution. Crown Business, New York

Chapter 3

The Relationship Between Attributes of Objects and Phonemes of Naming Yuri Hamada and Hiroko Shoji

Abstract This chapter describes the trends in the phonemes used in naming by using the attributes of objects. The study used the names of characters in video games and automobiles as the targets of analysis. By categorizing the objects based on their attributes, the study sought to verify if there existed a difference in the used phonemes between the categories. As a result, the study clarified the phonemes used to illustrate smallness, lightness, and weakness in addition to the phonemes used to depict largeness, heaviness, and strength between two objects. In addition, the study also employed evaluation experiments to verify the trends in the phonemes used in naming, which was derived from the analyses of the objects’ attributes, and to verify whether the phonemes used in the naming process matched people’s impressions. Through this process, the study identified the objects’ specific naming trends and the characteristics of the phonemes. By performing the same analysis process for many objects, the study hopes to propose a supporting method for the naming process of specific objects.

3.1 Introduction Naming plays an important role in expressing the features and images of objects. Especially in products and services, it is necessary to provide product information and naming that stimulates consumers’ willingness to purchase. Therefore, there are great social needs for naming methodologies and support methods. Previous studies have analyzed what kind of image is recalled (big, strong, etc.) from the phonemes used to name objects, and clarified the characteristics of those phonemes. However, few studies have attempted to verify whether the phonemes pointed out in previous studies are used for objects with such images. The purpose of the present study is to clarify naming trends using object attributes. Specifically, size, Y. Hamada (B) · H. Shoji Chuo University, 1-13-27 Kasuga, Bunkyo-Ku, Tokyo 112-8551, Japan e-mail: [email protected] H. Shoji e-mail: [email protected] © Springer Nature Switzerland AG 2020 S. Fukuda (ed.), Emotional Engineering, Vol. 8, https://doi.org/10.1007/978-3-030-38360-2_3

27

28

Y. Hamada and H. Shoji

weight, and strength are identified as object attributes, and it is verified whether there is a difference in phonemes used between these categories. In addition, evaluation experiments were performed to verify the validity of phonological naming trends obtained by analysis using object attributes. Further, the analysis results are compared with the phonological features pointed out in previous studies. Section 3.2 describes related studies on naming and the status of this research. Section 3.3 details analysis methods, and Sects. 3.4 and 3.5 detail the analysis results for the two objects (video game characters and automobiles). Section 3.6 examines commonalities and differences between the two objects. From this, it is expected that object-specific phonemes and new phoneme trends will be discovered.

3.2 Related Studies 3.2.1 Study on Naming Phonemes Much of the literature describes the characteristics of naming phonemes. According to Iida et al. [1], the dull sounds ga (/g) and da (/d) are often used for male names to embrace images, whereas consonants such as na (/n) and ma (/m) are popular for female names because they convey a soft image. The image received from such a sound and the working of the subconscious are described as “sound impressions.” Kurokawa discusses the naming and the sound of words from various angles [1, 2]. He states that, as sounds have a subliminal effect, automobiles sell well if their names begin with C (Corolla, Crown, Cedric, etc.), women’s magazines sell well if their names include na (/n) or ma (/m) (Nonno, Anne, More, etc.), and names of popular monsters always have dull sounds (Godzilla, Gamera, King Ghidorah, etc.). The main words in all headings (even run-in headings) begin with a capital letter. Articles, conjunctions, and prepositions are the only words which should begin with a lower case letter. In addition, the consonant /n has a sweet, smooth, or wet and sticky image, whereas the consonant /m is a rich and happy sound that enhances static impressions [2]. Kidori et al. state that words consist of two elements: the “brightness (B value)” that captures the sound of the word from the aspect of brightness and darkness, and the “cervical (H value)” that captures its intensity and severity [3, 4]. Further, a BH value is calculated for each phoneme, and the characteristics of the phoneme are discussed numerically as a sound value table. For example, the consonants /k and /t are both explosive and unvoiced sounds. /k gives the impression of hard and dry, whereas /t gives a strong masculine impression. Many studies have quantitatively analyzed brand names (BN) and impressions [5– 8]. Yorkston et al. state that consumers guess the nature of a brand based on the sound image contained in that brand name [5]. Klink shows that, as characteristics of product attributes associated with the pronunciation of brand names by English speakers, the posterior tongue vowels /a,u,o/ where the tongue is positioned at the back to emit low

3 The Relationship Between Attributes of Objects …

29

sounds are associated with larger, heavier, and round product attributes, compared to the front tongue vowels /i,e/ where the tongue is positioned forward and emits high sounds [6]. Based on the findings of Klink’s research, Park et al. confirmed that the same phenomenon was observed in Japanese sound impressions from experiments with Japanese subjects [9], and also mentions the combined effects of consonants and vowels [10].

3.2.2 Studies on Naming Support As a study on naming support, Nagamachi has constructed a system (WIDIAS) for diagnosing the sound of words using fuzzy integration [11]. Although this system only evaluated each character composing BN, Nagamachi et al. succeeded in considering the influence of each character on other characters using a neural network [12]. However, the analysis target was limited because double consonants, contracted sounds, and prolonged sounds were not considered. Doizak et al. are developing a BN creation support system based on sound impressions [13]. This can evaluate all Japanese expressions used in BN, but the meaning of words is not taken into account. There is also a naming support software called “Namemaster 7Lite” [14]. This is a tool that hierarchically suggests synonyms or synonyms from words for input character strings by users. This system only gives a suggestion of meaning, and does not capture phonological images. Shimizu et al. quantified human impressions of onomatopoeia based on impression evaluation using adjectives [15]. They also constructed a system that quantitatively captures the impression given by phonemes that comprise the onomatopoeia. Fujisawa et al. quantified the effects of individual phonemes on the impression of onomatopoeia, and modeled the relationship between phonology and evaluation of impression by the linear sum of these quantities [16]. Nakabe et al. proposed a system that automatically calculates the impression and nuances that onomatopoeia gives to people, and presents them to the user [17]. There are many studies that quantitatively analyze and systemize the impression of onomatopoeia in this way. Miura et al. conducted a survey of phonetic symbols (phoneme images and features) using machine learning based on data that humans perceived as “strength” for the names of Pokemon (Pocket Monsters), and verified the degree of coincidence with phonetic symbols referred to by linguists [18]. Further, Pokemon names were automatically generated using SVM.

3.2.3 Approach of This Study As mentioned above, there are many case studies related to naming phonology and studies related to quantitative analysis and naming support systems. In these studies, the images of the phonemes were first investigated and followed by the relation to

30

Y. Hamada and H. Shoji

object attributes. However, few studies have verified whether or not the phonemes pointed out in previous studies are used for objects with such images. In this study, objects are categorized based on their attributes, and it is verified whether there are differences in the phonemes used between the categories. As a study using a similar approach, Kawahara et al. are analyzing the naming of Pokemon [19]. As a result, it has been shown that dull sounds and the number of mora in the name are positively correlated with size, weight, strength, and evolution level of individual Pokemon. Kawahara et al. focus only on dull sounds and the number of mora, but in the present study, Japanese expressions including vowels and consonants are analyzed. In addition, evaluation experiments were performed to verify the validity of phonological naming trends obtained by the analysis using object attributes. Furthermore, similarities and differences with the phonological characteristics pointed out in related research are also considered. As a result, it is expected that object-specific phonemes and new phoneme trends will be discovered.

3.3 Analysis Method In this study, two objects are analyzed, and the analysis results are compared. Cars and characters (Pocket Monsters) are used as the two objects. The reasons for selecting automobiles and characters are that there are many names for both, automobiles are industrial products, characters are sensitive content, and their domains are different. Selection of object attributes is detailed in Sects. 4.1 and 5.1. In this study, objects are categorized by analysis using attributes. Further, the features of naming phonemes are discussed by comparing namings belonging to categories. Next, the analysis procedure will be described. First, based on Table 3.1, the naming was broken down into syllable units in Japanese called mora. Next, Table 3.2 was generated classified into vowels (/a, /i, /u, /e, /o), consonants (/k, /s, /t, /n, /h(f), /m, /y, /r, /w, /g, /z(j), /d, /b, /p, /v) and special sounds (/N, /Q, /R) according to Daijirin [20], third edition. In addition, the difference in the ratio of phonemes between attribute categories was tested. The test methods used were the χ2 test and Fisher’s exact test. The χ2 test is a method used as a test of independence. When the number of samples was small, Fisher’s exact test was used [21–23].

3.4 Phonological Analysis of Character Naming 3.4.1 Outline of Data and Selection of Attributes In this chapter, as in the study by Kawahara et al. [19], the target of phonological analysis was a pocket monster character (hereinafter referred to as Pokemon). The

·

0

Total

·

·

0

0

Daigaku

Gakko

0

Chu-o

·

a

Naming

1

0

·

·

1

0

i

3

1

·

·

0

2

u

Table 3.1 Example mora segmentation table

…

…

…

…

…

…

…

0

0

·

·

0

0

chi

…

…

…

…

…

…

…

2

1

·

·

1

0

ga

…

…

…

…

…

…

…

1

0

·

·

0

1

chu

…

…

…

…

…

…

…

0

0

·

·

0

0

n

1

1

·

·

0

0

tsu

0

0

·

·

0

0

–

3 The Relationship Between Attributes of Objects … 31

32 Table 3.2 Phoneme score table by category

Y. Hamada and H. Shoji Sample no.

ni

nJ

nk

Category

A

B

C

Total value for /k

/k i

/k j

/k k

Total value for /s

/si

/S j

/sk

Total value for /t

/t i

/t j

/t k

…

…

…

…

Total value for /a

/ai

/aj

/ak

Total value for /i

/ii

/ij

/ik

…

…

…

…

Total value for /N

/N i

/N j

/N k

Total value for /Q

/Qi

/Qj

/Qk

Total value for /R

/Ri

/Rj

/Rk

Grand total

Ni

NJ

Nk

reason for selecting Pokemon as an analysis target is that the name of the character is composed of up to six letters. This is because the appearance frequency of alphanumeric characters and special characters is low. The attributes used for the analysis were numbers that indicate the height, weight, and ability of Pokemon [24]. Pokemon also has the concept of evolution. It is known that there is a positive correlation between Pokemon evolution, and height, weight, and ability. Therefore, if height and weight are “size” and ability is “strength,” it can be said that “size” and “strength” change by using the concept of evolution. Then in this study, it will be examined how the phonemes of the naming change as it evolves. Next, an overview of the data will be given. Pokemon’s evolution includes “a case where evolution occurs only once” and “a case where evolution occurs twice.” This analysis proceeds as “Evolution Process (I)” and “Evolution Process (II)” for those two cases. In the phonological analysis, special sounds and alphanumeric characters and their associated evolutionary Pokemon were excluded. In Evolution Process (II), there are three stages, so the initial stage Pokemon is given “Attribute S (Small),” Stage 1 evolution Pokemon is given “Attribute M (Middle),” and Pokemon evolved to the final stage is given “Attribute L (Large).” The Pokemon to be analyzed are Evolution Process (I), 172, and evolution process (II), 85. For Evolution Process (II), Attribute S and Attribute L are compared.

3.4.2 Analysis Results This section mainly refers to the analysis results of vowels and consonants that are often mentioned in previous studies. Table 3.3 shows the analysis results of “Evolution Process (I),” and Table 3.4 shows the analysis results of “Evolution Process (II).” Tables 3.3 and 3.4 show the

3 The Relationship Between Attributes of Objects …

33

Table 3.3 Significant difference in evolution process (I)

Table 3.4 Significant difference in evolution process (II)

Table 3.5 Analysis results for Pokemon

Small, Light Weak

Big, Heavy, Strong

/k, /t /m

/r, /g, /d, /i, /o, /N

p-values and phonemes obtained by the χ2 test and Fisher’s exact test. To phonemes that showed a significant difference at a significance level of 1%, (**) was assigned, and to phonemes that showed a significant difference at a significance level of 5%, (*) was assigned. After evolution, they are colored red, and before evolution, blue. In Tables 3.3 and 3.4, since no conflicting results were obtained, Table 3.5 summarizes the phonemes for which significant differences were found in Evolution Process (I) and (II). From Table 3.5, it is seen that before evolution, the consonants /k, /t, /m are often used, and after evolution, the consonant /r, dull sounds /g, /d, and vowels /i, /o are often used.

3.4.3 Discussion In this section, we will focus on a comparison with the findings obtained in previous studies. First, the phonemes that are consistent with the findings of previous studies will be described. Before evolution, the consonants /k and /t are both explosive and unvoiced sounds, and /k is said to be “sharp, light, small, and beautiful” with a hard, dry feeling [4]. It is stated that /k is a rhythmical sound [4], and /t is a rich and fun sound [25]. Therefore, by using the consonants /k, /t to name Pokemon before evolution, a “pop” impression can be given while keeping a small image. The consonant /m is a rich and happy sound that enhances static impressions [2], and it has been pointed out that it has a calm and feminine image [4]. Due to this, it is thought that it also conveys

34

Y. Hamada and H. Shoji

an impression of static softness and weakness for Pokemon before evolution. As for the dull sounds /g, /d, dull sounds give a strong impression when they are heard, so it is stated that they are masculine, heavy sounds [1, 26]. From this, it can be considered that by using the dull sounds /g, /d for naming evolved Pokemon, they convey an image of grandeur or greatness. This is consistent with the research results of Kawahara et al. [19]. Regarding the vowel /o, it was pointed out in previous studies that the back tongue vowels (/a, /u, /o) are heavy sounds [6], which is consistent with the result that they are frequently used in evolved Pokemon. Next, phonemes that do not agree with the findings of previous studies will be described. Regarding the consonant /r, Kurokawa states that it is a “beautiful sound, and gives a brilliant and dignified impression” [2]. Kidori also states that the consonant /r has a light image. Therefore, it is necessary to verify that /r is included in many evolved Pokemon, but the evolution position in Pokemon space is not only change in size, weight, and strength but also rarity and scarcity. This can also be interpreted as a high degree of formality. Therefore, adding /r may indicate a high level of rarity. The vowel /i is a front vowel (/i,/e), and is known to give the opposite impression to back tongue vowels [6]. Thus, a different result to that of previous studies was obtained.

3.5 Phoneme Analysis of Automobile Naming 3.5.1 Data Summary and Attribute Selection In this section, automobiles sold by domestic manufacturers will be analyzed. As analysis targets, we considered seven body types (light automobiles, compact automobiles, etc.) that were being sold by domestic automobile manufacturers (as of November 30, 2018). Data for 70 minivan wagons, sedans, coupes, SUVs, and station wagons were collected at random. Next, physical feature quantities were extracted from price and specifications tables as attributes, and the components obtained from the principal component analysis were used as an index for category classification. The 13 attributes used were price (yen), fuel consumption (km /L), number of passengers (people), total displacement (cc), overall length (mm), overall width (mm), overall height (mm), minimum turning radius (m), vehicle weight (kg), interior/length (mm), interior/width (mm), interior/height (mm), and minimum ground clearance (mm). In the phonological analysis, for the target data, the names of seven body types similar to the above were obtained from a used automobile sales site [27, 28]. As conditions, for Japanese automobiles, alphanumeric characters were converted to katakana, and naming of grade differences was also extracted. The data included 150 mini automobiles, 61 compact automobiles, 110 minivan wagons, 138 sedans, 53 coupes, 49 SUVs, and 48 station wagons. Next, the determination of category classification index will be described. To determine this index, the result of the principal component analysis was used. In this

3 The Relationship Between Attributes of Objects …

35

analysis, the cumulative contribution rate was 80% up to the third principal component, therefore, up to the third principal component was considered. Table 3.6 shows the principal component score graph with the eigenvectors of each principal component. It was determined that the first principal component was related to an index indicating the power of the automobile, such as fuel efficiency and fuel consumption, and elements indicating the length and breadth of the outer shape, which was interpreted as a component related to “strength.” The second principal component is related to elements showing the space in the automobile, such as the number of passengers and the length and height of the automobile, and price, so this was interpreted as a component related to “in-car space and price.” The third principal component is related to height from the ground, so this was interpreted as a component related to “height of the vehicle.” In this paper, for the purpose of comparison with Pokemon, the analysis results of the first principal component (PC1) representing “strength” will be described in detail. To perform the phonological analysis, it is necessary to divide objects into several categories. On this occasion, the body type was classified into groups as shown in Table 3.7 based on the principal component score calculated from the eigenvectors in Table 3.6 and the actual data used for principal component analysis. In the phonological analysis, body type naming data for two groups were added together, and analyzed using data for 297 automobiles in the positive direction and 211 automobiles in the negative direction. Table 3.6 Eigenvectors in main component analysis

36

Y. Hamada and H. Shoji

Table 3.7 Basic statistical quantities for each body type in PC1 Light vehicles

Dispersion

Standard deviation

Average

0.1961

0.4428

−4.4664

Compact cars

0.2270

0.4765

−2.0141

Minivan wagons

0.4984

0.7060

2.1270

Sedans

1.2742

1.1288

1.4982

SUV

2.3121

1.5205

1.5283

Coupes

4.0812

2.0202

0.9361

Station wagons

1.1040

1.0507

0.3910

3.5.2 Analysis Results This section mainly refers to the analysis results of vowels and consonants that are often mentioned in previous studies. Table 3.8 shows the p-values obtained by the χ2 test and Fisher’s exact test for the first principal component. Phonemes that showed a significant difference at a significance level of 1% are denoted by (**), and phonemes that showed a significant difference at 5% are denoted by (*). The positive direction is colored red, and the negative direction is colored blue. Table 3.9 summarizes the phonemes in which significant differences were found in the automobile analysis results. The first principal component is a component related to “strength;” the positive direction can be interpreted as denoting strength and largeness of the outline, whereas the negative direction represents weakness and smallness of the outline. It can be seen that the consonants /h, and dull sounds /g, /d are often used in the positive direction, whereas the consonants k, /t, /m, /w, /u are often used in the negative direction. Table 3.8 Significant difference in PC1

Table 3.9 Analysis results for automobiles

Small, Light Weak

Big, Heavy, Strong

/k, /t /m , /w , /u

/h, /g, /d

3 The Relationship Between Attributes of Objects …

37

3.5.3 Discussion In this section, the discussion will focus on a comparison with the findings obtained in previous studies. First, phonemes that are consistent with the findings of previous studies are described. It is seen that, in the positive direction, there are many dull sounds of consonant sounds of /g, /d. It is reported that dull sounds are masculine and heavy [1, 25]. Minivan wagons, SUVs, and sedans, which are considered to be big, powerful automobiles in the car space, are given a massive image of grandeur and awesomeness by adding dull sounds with a sense of weight and strength to their name. Next, there are many negative consonants /m, /k, /t. Kurokawa et al. stated that the consonant /m is a rich sound contained in a feeling of happiness, and enhances static impressions [2]. It can be seen that in the first principal component, good fuel efficiency, and small external length and breadth, are characteristic as elements indicating the negative direction. Further, body types that can be classified in the negative direction are light vehicles and compact cars. These are generally considered to be small vehicles, and by using the consonant /m having the abovementioned characteristics for small vehicles in the vehicle space, a static, soft feeling can be imparted. According to Kidori, the consonants /k and /t are both explosive sounds, giving an image of “sharp, light, small, and beautiful” [3]. Further, it is reported that /k is a rhythmical sound [3], and /t is a sound of richness and happiness [4]. By using these consonants /k, /t for small automobiles in the automobile space, as well as the abovementioned consonant /m, it is possible to give a “pop” impression while keeping a small image at the same time. /w is said to give a feminine impression [29], and might be used in a negative direction indicating smallness. Next, phonemes that do not agree with the findings of previous studies will be described. Concerning the vowel /u, /u is a back tongue vowel and is said to represent a feeling of weight. However, /u appears significantly in the negative direction in automobiles, which is inconsistent with previous findings. As for /h, Iwanaga points out that it is a phoneme indicating lightness [30]. Therefore, the results for /h did not agree with the findings of previous studies.

3.6 Comparison Study Between Objects In this study, for video game characters (Pokemon) and automobiles, the relationship between object attributes and naming phonemes was analyzed. In this chapter, the two objects are compared, and commonalities and differences are discussed. Table 3.10 compares the phonological trends of the two objects. For automobiles and Pokemon, phonemes with significant differences in each category are shown. Common phonemes are shown in red.

38

Y. Hamada and H. Shoji

Table 3.10 Comparison of two objects

First, the commonalities of the two objects will be discussed. Although the attributes of the two objects are different, using PC1 for automobiles and the Evolution Process (I) and (II) of Pokemon as indices, they can be classified as “big–small” and “strong–weak.” By examining the results of phonological analysis for these, in the category described as “small” and “weak,” the consonants /k, /t, /m were often used, and in the category described as “big” and “strong,” the dull sounds /g, /d were often used. It is reported that dull sounds are used for something big and strong like monsters [25]. Using the attributes of the objects, it was possible to show that there was a trend similar to the phonological trend pointed out in the naming literature. It was also found that the consonants /k, /t, /m were used for objects that exhibit characteristics such as smallness, weakness, and softness. Next, the differences between the two objects will be described. In the category of big, heavy, and strong, /h can be confirmed for automobiles, but not for Pokemon. Therefore, /h is considered to have phonological characteristics specific to automobiles. As car-specific naming trends, many English words are used for automobiles (75/508). In addition, many English alphabetical letters are included (184/508). Regarding the relationship between sound impressions and language, it has been reported that sound impressions are observed regardless of language, and that words representing smallness have the same tendency for vowels regardless of language [31]. However, based on the differences in pronunciation characteristics among languages, Park et al. conducted experiments on Japanese people based on findings revealed by Klink’s research, and verified that some results were different [32]. Thus, it cannot be said that phonological trends of other languages coincide with Japanese phonological trends, and it is necessary to gain knowledge about sound impressions of Japanese speakers. It is also necessary to investigate the differences in impressions between Japanese and English speakers. In the case of automobiles, /h was frequent in the positive direction of the first principal component. There were 66 names containing /h in the positive direction and 26 in the negative direction of the first principal component. As names including /h, 16 “hybrids” were found in the positive direction and 2 in the negative direction. In the positive direction, “Hiace” (4) and “Hilux” (3) were confirmed, which suggests that the meaning of the word as well as the phonemes has an effect. These are English words (Hiace is a compound word of “High” and “Ace,” Hilux is a compound word of “High” and “Luxury”). From this, it can be said that it is necessary to consider not only the impression of the phonemes but also the impression and meaning of the English words and alphabet.

3 The Relationship Between Attributes of Objects …

39

Also, in categories that describe size, weight, and strength, Pokemon contains many /r, /i, and /o. In previous studies, it was pointed out that /r and/i give an impression opposite to the feeling of weight, so that further investigation is required. As described above, this study clarified the phonological naming trends for two objects, automobiles and video game characters, and compared them. As a result, common trends were observed in the phonetics of naming automobiles and video game characters. These trends generally agreed with the phonological characteristics pointed out in related studies. However, there were differences between automobiles and video game characters, especially car-specific trends. It may be that features specific to the object were found because of detailed analysis using the object attributes. Previous studies do not necessarily mention differences due to objects, and analysis of different kinds of objects reveals important findings when considering the naming of specific objects.

3.7 Summary and Future Challenges In this study, statistical analysis was performed on two objects, automobiles and video game characters, using the attributes of the objects, and the relationship with naming phonology was clarified. The results of the comparison between the analysis results and phonological features in previous studies suggested that there were trends similar to the phonological features pointed out in previous work. By comparing the analysis results of the two objects, commonalities and differences were then considered. From this, it was found that there were naming trends and phonological characteristics specific to the object. In future, it will be required to clarify the quantitative and detailed trends of naming phonemes unique to the object by performing the same analysis for more objects. By collecting knowledge about phonological naming trends specific to an object, it is expected to propose a naming support technique corresponding to a specific object. Additionally, in this study, the analysis was done by dividing the naming into sound segments called mora. However, the combination of sounds is also important as an element of naming. In addition, the effect of the meaning of alphabetical letters and words on naming impressions can also be considered. Therefore, in future, it is hoped to study the characteristics of phonemes due to combinations of sounds, and the impressions given by alphabetical letters and words.

References 1. Iida A (2012) Naming says things. Chuo University Press 2. Kurokawa I (2012) Sensitive naming lab. People decide what is “good or bad” by their language sense. Kawade Shobo Shinsha

40

Y. Hamada and H. Shoji

3. Kidori T (2004) The essence of naming: the charm of Japanese is the creation of sound. Chikuma Shobo 4. Kidori T (1990) Sound. President Co. 5. Yorkston E, Menon G (2004) A sound idea: phonetic effects of brand names on consumer judgments. J Consumer Res. https://doi.org/10.1086/383422 6. Klink RR (2000) Creating brand names with meaning: the use of sound symbolism. Market Lett. https://doi.org/10.1023/A:1008184423824 7. Klink RR (2001) Creating meaningful new brand names: a study of semantics and sound symbolism. J Market Theor Prac. https://doi.org/10.1080/10696679.2001.11501889 8. Klink RR (2003) Creating meaningful brands: the relationship between brand name and brand mark. Market Lett. https://doi.org/10.1023/A:1027476132607 9. Jaewoo P, Ose Y (2009) Effects of pronunciation of brand name on brand evaluation: approach from sound symbolism, consumer behavior research. https://doi.org/10.11194/acs.16.1_23 10. Jaewoo P (2010) Basic analysis of product attribute associations by brand name pronunciation. Economics 245:80–99 11. Nagamachi M (1993) Kansei engineering on sound of words. Jpn J Acoust Soc. https://doi. org/10.20697/jasj.49.9_638 12. Nagamachi M, Matsubara Y, Maeda H, Ohgama T (2018) Brand name decision AI system. In: Proceedings of the 7th international conference on Kansei engineering and emotion research. https://doi.org/10.1007/978-981-10-8612-0_5 13. Doizaki R, Shimizu Y, Sakamoto M (2012) Image evaluation system based on the sound symbolism of brand names. In: IEEE soft computing and intelligent systems (SCIS) and 13th international symposium on advanced intelligent systems (ISIS). https://doi.org/10.1109/scisisis.2012.6505236 14. Namer 7Lite. http://www.bds.ne.jp. Accessed 26 Apr 2019 15. Shimizu Y, Dobasaki R, Sakamoto M (2014) A system for estimating a detailed impression of individual onomatopoeia. J Jpn Soc Artif Intell. https://doi.org/10.1527/tjsai.29.41 16. Fujisawa N, Obata F, Takada M, Iwamiya S (2006) Impression of sound imaged from onomatopoeia of 2 mora. J Acoust Soc Jpn. https://doi.org/10.20697/jasj.62.11_774 17. Nakabe F, Watanabe C (2009) Development of onomatopoeia learning system using Kansei information. 1st Forum Papers on Data Engineering and Information Management 18. Miura S, Murata M, Yasuda S, Miyabe M, Aramaki E (2012) Reproduction of phonetic symbols by machine learning: creation of the strongest Pokemon. The Association of Language Processing 19. Kawahara S, Noto A, Kumagai G (2018) Sound symbolic patterns in Pokémon names. Phonetica. https://doi.org/10.1159/000484938 20. Daijirin, Japanese sounds, 3rd edn. http://daijirin.dual-d.net/extra/nihongoon.html. Sanseido. Accessed 26 Apr 2019 21. Matsubara N, Nada K, Nakai K (1991) Introduction to statistics. The University of Tokyo Press 22. Crawley MJ, Nomaguchi K, Kikuchi Y (2016) Statistics: introduction using R, 2nd revision. Kyoritsu Publishing Co. Ltd. 23. Miyagawa M, Aok S (2018) Statistical library, statistical analysis of contingency tables—from 2D to multi-dimensional. Asakura Shoten 24. Official Pokemon Picture Book. http://www.pokemon.jp/zukan. Accessed 26 Apr 2019 25. Kurokawa I (2004) Why the name of the monster is Gagigugego. Shinchosha 26. Iwanaga Y (2002) This will really sell! Naming success law. PHP Institute 27. Car sensor. https://www.automobilesensor.net/. Recruit Marketing Partners Inc. Accessed 26 Apr 2019 28. Goonet. https://www.goo-net.com/. Proto Corporation. Accessed 26 Apr 2019 29. Kazuko S, Kawahara S (2013) The sound symbolic nature of Japanese maid names. In: Proceedings of the 13th annual meeting of the Japanese cognitive linguistics association 30. Iwanaga Y (1990) Naming to sell. Naming to buy. Dobunkan, pp 200–204

3 The Relationship Between Attributes of Objects …

41

31. Ohala J (1984) An ethological perspective on common cross-language utilization of FO of voice. Phonetica. https://doi.org/10.1159/000261706 32. Jaewoo P, Ose Y (2019) Effects of brand name pronunciation on product attribute association and brand attitude. Nikkei Advertising Institute

Chapter 4

Computational Handicraft Yuki Igarashi

Abstract The design of original handicrafts requires appropriate construction plans, since the creation of such plans is very difficult for children, they are typically restricted to the use of off-the-shelf projects. To provide alternatives, we created interactive systems that allow nonprofessional users to design their own original handicrafts, such as plush toys, knitted animals, beadwork, and stencils. For plush toys, the “Plushie” interactive design system uses a sketching interface for threedimensional (3D) modeling and provides various editing operations tailored for plush toy creation. Internally, the system constructs a two-dimensional (2D) cloth pattern that, in turn, is used to produce a simulation that matches the user’s input strokes. For knitted animal toys, we created the “Knitty” interactive design system, which also provides a production-assistant interface for novices, while the “Beady” system interactively assists users in the design and construction of customized 3D beadwork. The latter system provides a mesh modeling user interface specialized for beadwork models by combining gestural operations and physical simulations. It also provides a visual 3D step-by-step guide to facilitate the manual beadwork construction process. Finally, the “Holly” system lets users quickly design stencils from scratch by interactively drawing free-form strokes on a digital canvas, which the system then uses to generate the stencils automatically. The support systems mentioned above were successfully demonstrated in special workshops and user studies, the results of which show that even children can use them effectively to design original handicrafts.

4.1 Introduction Personal computers (PCs) can be powerful tools for designing real-world objects. For example, users can build virtual three-dimensional (3D) models on a PC using computer-aided design (CAD) and then use those models to run various simulations with computer-aided engineering (CAE) without the need to build or damage costly real-world objects. The benefits are evident in many areas, from architecture to Y. Igarashi (B) Meiji University, Tokyo, Japan e-mail: [email protected] © Springer Nature Switzerland AG 2020 S. Fukuda (ed.), Emotional Engineering, Vol. 8, https://doi.org/10.1007/978-3-030-38360-2_4

43

44

Y. Igarashi

automobile design. However, these tools are designed primarily for professional users and are not particularly accessible to ordinary persons. Furthermore, the construction of a 3D model using a standard CAD system is tedious, and running a physical simulation using a standard CAE system requires a certain level of expertise. Our goal is to put the benefits of CAD and CAE into the hands of nonprofessional users, including children. In this chapter, we introduce a handicraft design support system and show how children who have no specialized knowledge can experience the design process via a PC. Many people enjoy handicrafts, such as making plush toys, knitted animals, and beadwork designs. However, it is difficult for nonprofessional users to design corresponding construction plans (e.g., two-dimensional (2D) patterns from 3D models) because the relationship between a plan (2D pattern) and the results (3D model) is complicated, and intensive experience and knowledge are required to perform 2D flattening manually. With those points in mind, we have created and presented a number of interactive systems that allow nonprofessional users, including novice children, to design original handicraft toys. One key aspect of our work is the tight integration of physical simulations into the modeling process. In traditional applications, modeling and simulation are entirely separate. More specifically, a virtual model is typically created in modeling software without considering any physical constraints, and it is then passed to a simulation environment. If the simulation results reveal a problem, the user must return to the model to fix the problem. We made this process more efficient by running the simulation concurrently with the modeling in order to ensure it only creates physically realizable models. In this way, the user can more efficiently explore the design dimensions within realistic constraints. From the user’s point of view, the model generated by the system may not correspond precisely to the shape that was input, but it will be a physically realizable shape that reflects the input shape. Some modern systems have tried to incorporate fast physical simulations into interactive design processes. For example, Igarashi and Hughes developed a markbased interface for putting clothing on a virtual character [1], and Decaudin et al. proposed a system for designing an original garment via sketching [2]. Both used simple geometric simulations to represent the physical properties of cloth material. In another study, Masry and Lipson described a system in which the user can quickly build a CAD model by sketching and then immediately apply a finite element analysis to the model [3]. However, in these systems, the model construction is computed before the simulation, and no dynamic feedback loop exists between the simulation results and the original user input. In the following sections, we introduce our interactive systems, which allow novices to create a hand-crafted plush toy, knitted, beadwork, and stencil designs. We will also report on workshops in which novice users were encouraged to try our systems, the results of which demonstrated that they allowed nonprofessional users to efficiently design plush toy, knitted, beadwork, and stencil projects.

4 Computational Handicraft

45

4.2 Plush Toy Design This section introduces Plushie [4], an interactive system that allows nonprofessional users to design original plush toys (Fig. 4.1). As mentioned above, designing a plush toy requires the ability to construct an appropriate 2D pattern, which is a complicated process for nonprofessional users. Some recent systems can automatically generate a 2D pattern for a given 3D model, but constructing a 3D model is also a challenge. Furthermore, an arbitrarily created 3D model cannot necessarily be realized as an actual plush toy, and the final sewn result could be significantly different from the original 3D model. We avoid such mismatches by constructing appropriate 2D patterns and then applying simple physical simulations to them, on the fly, during 3D modeling. In this way, the model on the screen is always a good approximation of the final sewn result, which makes the design process much more efficient. To accomplish this, we use a sketching interface for 3D modeling and provide various editing operations tailored for plush toy design. Internally, the system constructs a 2D cloth pattern in a way that ensures the simulation result matches the user’s input strokes. Our goal is to show that when using Plushie, relatively simple algorithms can provide easy, fast, and satisfactory results, thereby successfully demonstrating that nonprofessional users can quickly design plush toys or balloons. However, it should be noted that the pursuit of optimal layout and simulation accuracy lies outside the scope of this paper.

Fig. 4.1 Plushie system design sequence [4]

46

Y. Igarashi

4.2.1 User Interface As shown in Fig. 4.1, the system consists of two windows: one shows the 3D plush toy model being constructed, and the other shows the corresponding 2D pattern. In the 3D view, the user works interactively with the system to build a 3D model via a sketching interface. When performing modeling operations, which are based on the gestural interface introduced by Igarashi et al. [5], the user interactively uses gestures to draw free-form strokes on the canvas, and the system performs corresponding operations. We also provide some special editing operations tailored for plush toy design. Creating a New Model: Starting with a blank digital canvas, the user creates a new plush toy model by drawing its silhouette with a closed free-form stroke. The system automatically generates two cloth patches corresponding to the stroke and visualizes the shape of the resulting plush toy by applying a simple physical simulation. Cut: Cutting operations are used to make relatively flat surfaces, such as those for a foot or belly. Cutting strokes should start outside of the model, cross it, and end outside of the model. The model is then cut at the intersection, and a flat patch is generated at the cross section. Part Creation: The user can add protruding parts such as ears and arms to the base model by drawing a single stroke that defines the part silhouette. The stroke should start and end on the base model (Fig. 4.2a). The system generates two candidate shapes and presents them to the user as suggestions [6] (Fig. 4.2b). One is for fat, rounded parts like the torso, arm, and leg that have bases connected to the base model with an open hole (Fig. 4.2c). The other candidate shape is for thin parts like ears and tails that have closed bases (Fig. 4.2d). In those cases, the user clicks the desired thumbnail, and the system updates the main model accordingly. We found that the ability to create thin parts with a single stroke is particularly useful because such

Fig. 4.2 Part creation user interface. a The user draws a stroke, and b the system suggests two different possibilities. The user chooses either c or d

4 Computational Handicraft

47

Fig. 4.3 User interface for the pull operation

parts are frequently seen on real toys and are challenging to design using standard modeling software. Pull: The user can also grab a seam line and pull it to modify the shape. For example, the user can pull on an ear to make it larger when it is smaller than the other (Fig. 4.3). The pulling operation begins when the user starts dragging on the background region near a seam line. Note here that the system changes the mouse cursor when it approaches a seam line to indicate an area where the user can start pulling. We use the peeling interface introduced by Igarashi et al. [7] to adjust the size of the region to be deformed. More specifically, the more the user pulls, the larger the area of the deformed region becomes. The system continuously updates the 2D cloth pattern during pulling and shows the simulation result in the 3D view. Insertion and Deletion of Seam Lines: The modeling operations performed thus far automatically generate 2D patches according to predefined algorithms and seam lines (patch boundaries) that appear on the 3D model surface without the user’s explicit control. However, for more precise control, it is sometimes desirable for knowledgeable users to design seam lines manually. This is especially important when using non-stretchy cloth, such as in the case of balloon models, because it is often necessary to divide a rounded surface into numerous almost-developable small patches. The user can also add a new seam in the seam line drawing mode by drawing a free-form stroke on the model surface (Fig. 4.4 left). The corresponding cloth patch is then automatically cut along the new seam line. If the stroke crosses the entire patch, the patch is divided into two separate patches. If the stroke starts or ends in the middle of a patch, it becomes a dart. Note that the 3D geometry does not change immediately after the insertion of these seam lines, but the user can pull the seam line afterward to modify the shape. This operation is handy for creating a salient feature in the middle of a flat patch. Deletion is achieved by tracing the target seam line in the erasure mode. This merges the separated patches and thus flattens the area (Fig. 4.4 right). 2D Pattern View Operations: The 2D pattern view is used primarily to preview the pattern to be printed for sewing, but it also works as an interface that allows advanced users to edit the pattern directly (Fig. 4.5). The preview helps the user to understand the relationship between the 3D model and the 2D patches, while also giving a sense of the labor required for assembling the patches.

48

Y. Igarashi

Fig. 4.4 Insertion and deletion of a seam line

Fig. 4.5 2D pattern operations

The system also allows the user to edit the patches directly by use of the pulling interface. The user can grab the boundary of a patch and pull it to deform the shape [12]. We again use a peeling interface to adjust the size of the area to be deformed. The effect of 2D deformation immediately appears in the 3D view because of the physical simulation.

4.2.2 Algorithm The resulting model is always associated with a 2D pattern and a 3D model result of a physical simulation that mimics the inflation effect caused by stuffing. Therefore, the model on the screen is always a reasonable estimation of the final sewn product.

4 Computational Handicraft

49

Fig. 4.6 Our simple model used to mimic the stuffing effect. Internal pressure pushes the vertices outwards (a) and the edge springs pull them back (b)

Internally, the system computes the geometry of the 2D pattern so that the simulation result matches the user’s input sketch. This is a nontrivial inverse problem that we address by using a simple iterative adjustment method. We show that a straightforward simulation method (Fig. 4.6), which involves just moving vertices in their normal directions to mimic pressure and pulling them back to maintain edge length, works well for our application and provides the unprecedented experience of designing a physical object on a PC.

4.2.3 Workshop We conducted a small workshop to give novice users opportunities to try our system. Children, 10–14 years old, joined the workshop accompanied by their parents. We began with a brief tutorial and then encouraged them to design plush toys using the system. The design process took about an hour, after which they printed out their designed patterns and sewed real toys in a process that took approximately three hours. Figure 4.7 shows several plush toys created during the workshop. The participants quickly learned how to use the system and, with some help from volunteers, successfully designed the 3D models they envisioned. Furthermore, they enjoyed the process. The resulting toys were their creations and one-of-a-kind designs.

4.3 Knitted Animal Design In this section, we introduce Knitty [8], an interactive design system for creating knitted animals with which users design 3D surface models using a sketching interface. The system automatically generates knitting patterns and then visualizes the shapes of the resulting 3D animal models by applying a simple physics simulation. This allows users to see the resulting shapes before they begin knitting. The system also provides a production assistant interface for novices with which they can easily understand how to knit each stitch and what to do in each step.

50

Y. Igarashi

Fig. 4.7 Snapshots of a small workshop held to introduce Plushie along with examples of original plush toys designed and created by children during the workshop

4 Computational Handicraft

51

Fig. 4.8 Designing an original knitted animal using Knitty [8]

4.3.1 User Interface Knitty consists of two windows: one shows the 3D knitted animal model being constructed, and the other shows the corresponding knitting pattern (Fig. 4.8). A knitting pattern is a figure arrangement for making animals that is equivalent to the 2D cloth patterns used for making stuffed animals. With this system, the user can design a 3D model in the 3D model window using a mouse or pen tablet from which the system automatically creates a knitting pattern based on the user’s input strokes. The system also visualizes the shape of the resulting animal by applying a simple physics simulation. The modeling operations are based on the gestural interface introduced by Igarashi et al. [5], and the system also provides special editing operations tailored to knitted animal design. A production assistant interface is also provided with this system. This mode teaches novices how to make knitted animals. Users can change the thickness of the yarn (thick, normal, or thin) and the size of the crochet needle (sizes 5–10), after which the system updates the number of stitches and indicates both the total knitting time and the total length of yarn required. The system automatically marks the current knitting position on the knitting pattern column when the user pushes the up/down key (Fig. 4.9 left). Clicking the right mouse button on the knitting pattern also shows the user how to knit (Fig. 4.9 right).

4.3.2 Algorithm In this system, we used a polygonal mesh representation similar to a sweep-based model [9, 10] or a generalized cylinder [10]. Here it should be noted that although actual knitted toys can contain many types of stitches, our system is limited to the

52

Y. Igarashi

Fig. 4.9 Knitty production assistant interface. Right-clicking the mouse button on the knitting pattern shows the user how to knit

three most basic stitches (normal, increase, and decrease) because it was designed for novice users. We create a closed planar polygon by connecting the start and endpoints of a stroke and then determine the spines or axes of the polygon using the chordal axis [11]. After extracting the longest axis, the system constructs vertices (vi ) on the axis and the plane (Pi ) that passes through the vertex vi and has a normal vector vi − 1 to vi . The system calculates radius r i from the user’s input stroke to the center vertex (vi ). The system then constructs circular vertices on the plane (Pi ). Next, the system creates stitches between the circular vertices on a plane Pi and the circular vertices on the next plane Pi+1 (Fig. 4.10). The system then calculates the number of vertices and constructs the circular vertices on the plane. The first plane (P0 ) and the last plane (Pn ) only have a center vertex, so the system constructs virtual edges. The system also constructs a knitting pattern using the number and type of stitches. Since the final shape of the actual knitted animal is determined by the physical interaction between the knitted “skin” and the inner cotton stuffing, a physical simulation

Fig. 4.10 Stitching between a plane Pi and the next plane Pi+1 . Length enforcement is only applied to solid edges and not to the dashed edges

4 Computational Handicraft

53

must be conducted to modify the input shape so that it is consistent with such physical conditions. As with the Plushie system [4], we used a simple static method for the physical simulation to achieve interactive modeling on a standard PC.

4.3.3 Workshop We conducted a small workshop to give novices opportunities to try our Knitty system. Ten children aged 10–14, accompanied by their parents, participated in the workshop, and created knitted animals using the system. Even though most participants did not have prior knitting experience, it only took them approximately one hour to design the animals, and approximately three hours to knit the actual toys. Examples of original knitted toy animals designed and created by children in our workshop are shown in Fig. 4.11.

4.4 Beadwork Design In this section, we introduce our interactive Beady system [12] which was devised to assist novices in the design and construction of customized 3D beadwork. Beadwork is the art of connecting beads together by wires. However, the design and construction of 3D beadwork is very difficult (Fig. 4.12) because the final shape is defined by complicated 3D interactions between beads and wires. The creator also needs to specify an appropriate wire path to hold the beads together and to manually insert

Fig. 4.11 Snapshots of Knitty workshop and examples of original knitted animals designed and created by children during the workshop

54

Y. Igarashi

Fig. 4.12 3D beadwork and the corresponding 2D pattern

the wire into each bead one-by-one, while following the designed path to construct the beadwork. Careful observation of existing beadwork structures shows several geometrically interesting problems that make beadwork design a technical challenge. Figure 4.13 shows the overall process. In this system, users first create a polygonal mesh model, called a design model, which represents the overall structure of the beadwork (Fig. 4.13a). A bead of the beadwork is represented as an edge (not a vertex) of the design model. The system runs a simulation to predict the resulting shape (Fig. 4.13b). The system then converts the design model into a beadwork model by placing beads on the edges with the appropriate wiring (Fig. 4.13c). Users specify the colors and shapes of individual beads in the beadwork model by using a painting interface after which the system computes the wire path (Fig. 4.13d). Finally, they manually construct the physical beadworks by following the step-by-step instructions generated by the system (Fig. 4.13e).

4.4.1 User Interface Users create the geometry of their beadwork models interactively with the system. Starting with basic primitives, they interactively edit the primitives by using gestural operations. The system runs a physical simulation after each editing operation and updates the mesh on the screen to enforce the constraint. As a result, users only need to specify the mesh topology from which the vertex positions are automatically decided. Users can modify the model by using the mesh editing operations shown in Fig. 4.14. These operations are designed to maintain the mesh as a manifold. Since the overall shape is already given as a combination of primitives, we expect that these editing operations will only be needed to adjust the shape locally. We implemented

4 Computational Handicraft

55

Fig. 4.13 Beady system overview. a The user creates a design model either by modeling from scratch or by converting an existing polygonal model. b The system runs a simulation to predict the resulting shape. c The beadwork model visualizes the expected result. d The system computes the wire path. e The system provides a step-by-step construction guide

Fig. 4.14 Example modeling sequence using the gestural interface. Users only need to specify the mesh topology by using the gestural interface. A physical simulation automatically provides the vertex positions

56

Y. Igarashi

Fig. 4.15 An example of the visual construction guide. a Initial state. b, c, f Blue wire passes through a newly added bead. d Red wire passes through a newly added bead. e Red wire passes through an existing bead

these editing operations as modeless context-sensitive gestural interactions in which the user either clicks on a component (face, edge, or vertex) or draws a short line using a dragging operation, after which the system takes the appropriate action. The physical simulation adjusts the overall geometry after each of these operations. The system also provides a step-by-step visual guide to assist the manual beadwork construction process. The construction guide provides 3D graphics that show which wire passes through which bead in each step. Figure 4.15 shows an example sequence. The user can view each step from an arbitrary viewing direction. The user presses then the “next” button to proceed to the next step or the “prev” button to return to the previous step. The construction process starts by placing a bead in the middle of a long wire (Fig. 4.15a). One side of the wire is blue, and the other side is red. The system shows the initial length of each side. The construction proceeds by inserting the blue or the red wire into an existing or new bead. A loop shows the wire that is used in that step (Figs. 4.15b through f), and an arrow indicates that the wire passes through a bead newly added to the beadwork (Figs. 4.15b–d and f). Otherwise, the wire passes through a bead already included in the beadwork (Fig. 4.15e).

4.4.2 Algorithm Users first create a polygonal mesh model, called the design model, which represents the overall structure of the beadwork in which each edge of the mesh model corresponds to a bead in the beadwork. The mesh topology is defined with gestural interactions, and the system continuously adjusts edge lengths by considering the physical constraints among neighboring beads. The system then converts the design model into a beadwork model with the appropriate wiring. Since the computation of an appropriate wiring path requires careful consideration, we present an algorithm based on mesh face stripification. A wire path defines the order in which a wire connects the beads to hold them in place. Based on a careful examination of existing beadwork designs, we selected the following three goals for the algorithm. First, the wires should pass through a bead for the minimum number of times necessary to hold it in the correct position. In our case that value is always two, except for chains. Second, the minimum number of

4 Computational Handicraft

57

Fig. 4.16 Details of the wire path-planning algorithm

wires should be used to reduce the need for wire cutting and tying. Third, the wire path should be designed so that each bead in the beadwork is as stable as possible during the construction process. We consider a bead to be stable when its position is fixed to a specific location by the wires. More specifically, when the bead belongs to a completed face. Given the design model (Fig. 4.16a), we start the process with the structure model (Fig. 4.16b) that defines the local wire connectivity. A global wire path (Fig. 4.16c) is given as a loop that meets all wire edges once, and all bead edges twice. This is the Eulerian cycle, with an additional constraint that a wire entering a bead should exit from the other side of the bead. This construction method guarantees the existence of a Eulerian cycle because each bead edge always has four wire edges. Various methods exist for obtaining such a Eulerian cycle. However, an arbitrary Eulerian cycle (e.g., Fig. 4.16d) can be inconvenient in the manual construction process because it can cause many unstable beads during the construction process (Fig. 4.16e). Accordingly, we carefully examined various existing beadwork designs and found that the production of unstable beads could be reduced by the use of a face strip (Fig. 4.16f). More specifically, the design model is covered by a face strip, and then a wire path is placed so that faces in the strip are completed one-by-one. This method ensures that all the beads in the previously visited faces are stable during construction (Fig. 4.16g).

4.5 Workshop We conducted a small workshop to provide novice junior high school student users with opportunities to use our system. We began by giving them a brief tutorial and encouraged them to design beadworks using the Beady system. The design process

58

Y. Igarashi

took about one hour, after which they constructed real beadworks in approximately two hours. Figure 4.17 shows a couple of beadworks created during the workshop. We found that, with some help from volunteers, the participants quickly learned how to use the system and could successfully design the 3D beadwork models they envisioned.

Fig. 4.17 Snapshot of Beady workshop and examples of original beadworks designed and created by participants

4 Computational Handicraft

59

4.6 Stencil Design As a final example, we will introduce our 2D stencil design system. Stenciling is a form of artistic expression in which users print images on target objects (for example, fabric or a postcard) by applying pigment over a plate with holes. A stencil contains two types of regions; a negative region is an empty space through which paint can pass and a positive region, which is the surface area surrounding the negative region. Creating stencils can be difficult because they must satisfy specific physical constraints. In a proper stencil, the template is a single, connected piece of material. However, islands (isolated unconnected positive regions) are sometimes created inadvertently and must be connected to the main part of the stencil by bridges. Therefore, stencil design typically requires knowledge, experience, and skill. Consequently, most people prefer to purchase ready-made stencil plates rather than make their own. In contrast, our Holly system [13] allows users to design proper stencils from scratch easily (Fig. 4.18). In use, the user interactively draws free-form strokes on the digital canvas, and Holly automatically generates the stencil. The user can print out the final image using a cutting plotter, such as Craft ROBO. Holly is so simple and straightforward that even children can use it.

4.6.1 User Interface In operation, the user designs a stencil using a combination of strokes, primarily employing the brush and fill tools. Each time the user completes a stroke, the system automatically generates a stencil image in which all positive regions are connected.

Fig. 4.18 Holly stencil design system

60

Y. Igarashi

Fig. 4.19 Results obtained using Holly: a the result of Holly’s painting operations, b the resulting outline dataset obtained using the cutting plotter, c the physical stencil plate created using the cutting plotter, and d decorated paper produced using the stencil

The user can erase a portion of an existing stroke using the brush-eraser or the fill-eraser tool. Figure 4.19 shows the results.

4.6.2 Algorithm Holly automatically detects any islands and adds a bridge connecting them to the main sheet. For example, suppose the user adds a fill primitive in the underwriting mode that creates an island (Fig. 4.20a). Holly then adds a bridge (Fig. 4.20b). When the user moves a stroke, Holly automatically updates the bridge’s stroke position (Fig. 4.20c) or deletes the bridge if it is no longer necessary (Fig. 4.20d). If the user dislikes the automatically created bridge, he/she can manually add a different one by using one of the eraser tools (Figs. 4.20e, f). The user can also change the bridge’s width to account for the stencil material’s rigidity.

4 Computational Handicraft

61

Fig. 4.20 Bridge creation. The user (a) adds a fill primitive in underwriting mode, thus creating an island (an isolated positive region). In the automatic mode, Holly (b) detects the island and connects it to the main sheet. If the user moves a stroke, the system either (c) updates or (d) deletes the bridge. Users can also (e) use one of the eraser tools to (f) manually add a bridge

4.6.3 Workshop We also conducted a small workshop to give novice users the chance to try Holly. The workshop was attended by ten children, ages 8–11, who were accompanied by their parents. After a brief tutorial, the children took about an hour to create their designs (Fig. 4.21a). They then printed their designed stencils using a cutting plotter and used them to decorate fabric bags in a process that also took roughly one hour (Fig. 4.21b). Figure 4.21c shows some stencil designs and finished bags created by the children. They quickly learned how to use the system and enjoyed the design process. They also gave us valuable feedback for future improvements. For example, they wanted to use stencil templates, and they wanted an easy-to-use input method to name files because keyboard-free tablet PCs were used in the workshop.

62

Y. Igarashi

Fig. 4.21 Photos from a workshop in which ten children used Holly: a designing stencils, b decorating fabric bags, and c some of the stencils and bags. The entire process took approximately two hours

4.7 Conclusion We introduced four different handicraft design systems as examples of our efforts to make CAD and CAE more accessible to end-users. These systems allow users to design original design handicrafts quickly and simply by combining simple sketching or drawing operations. In operation, users design their desired shapes on the digital canvas of their tablet PC, after which the system automatically generates a handicraft model and the corresponding construction plan. The systems run a simple physical

4 Computational Handicraft

63

simulation in the background so that the resulting model always provides a reasonable estimate of the final construction result. To demonstrate the effectiveness of the approach even further, we also developed a system for the creation of a cover design [14, 15], a weaving design [16], an iris folding pattern design [17], and a craft band design [18]. We also developed systems to convert existing 3D surface models into handicraft 3D models and corresponding patterns such as plush toys [19], knitted toys [20], and beadwork [12]. The handicraft design processes were supported by devising dedicated user interfaces. Internally, advanced computations are performed based on the discrete mathematics and physics introduced in the algorithm section. However, users can use these systems without concerning themselves with those issues. Interactive shape modeling, assisted by concurrent physical simulation, can be a powerful tool in many other application domains. For example, if a user could run an aerodynamic simulation during the interactive design of a paper airplane model, the entire geometry could be adjusted intelligently in response to the user’s simple deformation operations in order to produce a model that could fly [21]. This kind of interaction would make it easier for designers to pursue aesthetic goals while satisfying engineering constraints. Furthermore, while real-time simulation does require high-performance computing resources, some meaningful support should be possible by carefully limiting the target task and designing appropriate interfaces, as shown in this chapter. We hope that our efforts will inspire more work in this direction. Acknowledgements We are grateful to Moeko Nakajima for her support. We wish to express our appreciation to the test users for their cooperation. This work was supported in part by Japan Science and Technology, under PRESTO Grant JPMJPR16D1Japan.

References 1. Mori Y, Igarashi T (2007) Plushie: an interactive design system for plush toys. ACM Trans Graphics (Proceedings of SIGGRAPH 2007) 26(3):45, 1–8 2. Igarashi T, Matsuoka S, Tanaka H (1999) Teddy: a sketching interface for 3d freeform design. In: Proceedings of ACM SIGGRAPH 1999, pp 409–416 3. Igarashi T, Hughes JF (2001) A suggestive interface for 3d drawing. In: Proceedings of 14th annual symposium on user interface software and technology (ACM UIST 2001), pp 173–181 4. Igarashi T, Moscovich T, Hughes JF (2005) As-rigid-as-possible shape manipulation. ACM Trans Graphics (In: ACM SIGGRAPH 2005) 24(3):1134–1141 5. Igarashi T, Hughes JF (2002) Clothing manipulation. In: 15th annual symposium on user interface software and technology (ACM UIST 2002), pp 91–100 6. Decaudin P, Julius D, Wither J, Boissieux L, Sheffer A, Cani M-P (2006) Virtual garments: a fully geometric approach for clothing design. Comput Graphics Forum (In: Eurographics 2006 Proceedings) 25(3):625–634 7. Masry M, Lipson H (2005) A sketch based interface for iterative design and analysis of 3D objects. In: Proceedings of eurographics workshop on sketch-based interfaces, pp 109–118 8. Hyun DE, Yoon SH, Kim MS, Juttler B (2003) Modeling and deformation of arms and legs based on ellipsoidal sweeping. In: Computer graphics and applications, pp 204–212

64

Y. Igarashi

9. Yoon SH, Kim MS (2006) Sweep-based freeform deformations. Comput Graphics Forum 25(3):487–496 10. Shaff EB, Kuijlaars ABJ (1997) Distributing many points on a sphere, vol 19. Springer, New York, p 1 11. Prasad L (1997) Morphological analysis of shapes. In: CNLS Newsletter, vol 139, pp 1–18 12. Igarashi Y, Igarashi T, Mitani J (2012) Beady: interactive beadwork design and construction. ACM Trans Graphics (Proceedings of SIGGRAPH 2012) 31(4):49 13. Igarashi Y, Igarashi T (2010) Holly: a drawing editor for designing stencils. IEEE Comput Graphics Appl 30(4):8–14 14. Igarashi Y, Igarashi T, Suzuki H (2009) Interactive cover design considering physical constraints. Comput Graphics Forum (Proceedings of Pacific Graphics 2009) 28(7):1965–1973 15. Igarashi Y, Suzuki H (2011) Cover geometry design using multiple convex hulls. Comput Aided Des 43(9):1154–1162 16. Igarashi Y, Mitani J (2014) Weavy: interactive card-weaving design and construction. IEEE Comput Graphics Appl 34(4):22–29 17. Igarashi Y, Igarashi T, Mitani J (2016) Computational design of iris folding patterns. Comput Vis Media 2(4):321–327 18. Igarashi Y (2019) BandWeavy: interactive modeling for craft band design. IEEE Comput Graphics Appl 39(5):96–103 19. Igarashi Y, Igarashi T (2008) Pillow: interactive flattening of a 3D model for plush toy design. Lecture notes in computer science. Springer (In: Proceedings of SmartGraphics 2008), vol 5166/2008, pp 1–7 20. Igarashi Y, Igarashi T, Suzuki H (2008) Knitting a 3D model. Comput Graphics Forum (In: Proceedings of Pacific Graphics 2008) 27(7):1737–1743 21. Umetani N, Koyama Y, Schmidt R, Igarashi T (2014) Pteromys: interactive design and optimization of free-formed free-flight model airplanes. ACM Trans Graphics (Proceedings of SIGGRAPH2014) 33(4):65, 1–10

Chapter 5

Sustainable Behaviour: A Framework for the Design of Products for Behaviour Change Giulia Wally Scurati, Marina Carulli, Francesco Ferrise and Monica Bordegoni Abstract The increasing concern for sustainability-related issues leads to the rise of new fields in design research, dedicated to limit the negative impact of human activities on the environment and society. After addressing issues related to production, efficiency, recyclability and disassembly, designers start to recognize their responsibility in guiding users to behave in a more responsible and sustainable way. For this reason, designing products to support users’ behaviour change is becoming one of the most popular trends in design research at the moment. To achieve the desired results design for behaviour change, and in particular, Design for Sustainable Behaviour exploits a variety of approaches. In this Chapter, we explore the use of Design for Sustainable Behaviour techniques through a literature review of theories and case studies. Then, we defined a framework which describes the use of multisensory stimuli as elements to support different phases of interaction during the user experience with an interactive product. We relate this framework to previous works and then we discuss two case studies.

5.1 Introduction The increasing concern for sustainability-related issues leads to the rise of new fields in design research, dedicated to limit the negative impact of human activities on the environment and society. After addressing issues related to production, efficiency, recyclability and disassembly, designers start to recognize their responsibility in guiding users to behave in a more responsible and sustainable way. The idea that the way a particular design affects users’ behaviour is not new: Norman already discussed how a product invites us to perform certain actions on and through them, guiding and potentially changing our behaviour [1]. However, it is only more recently that we started reasoning about the rebound effects that users’ behaviour has on the main challenges we are facing on the global G. W. Scurati · M. Carulli · F. Ferrise · M. Bordegoni (B) Politecnico di Milano, Via La Masa 1, Milan, Italy e-mail: [email protected] © Springer Nature Switzerland AG 2020 S. Fukuda (ed.), Emotional Engineering, Vol. 8, https://doi.org/10.1007/978-3-030-38360-2_5

65

66

G. W. Scurati et al.

and local scales, including resources consumption, environmental damage, health and social issues. For this reason, designing products to support users’ behaviour change is becoming one of the most popular trends in design research at the moment. In fact, the field of design started integrating knowledge and theories from the field of behavioural psychology, proposing different models and frameworks. Some examples can be found in the work of Fogg [2], who analyses the factors affecting behaviour and relate them to different triggers, and Bhamra et al. [3], who map different design strategies according to the phases of behaviour change. To achieve the desired results, eliciting pro-environmental and pro-social behaviours, design for behaviour change, and in particular, Design for Sustainable Behaviour exploits a variety of approaches. In particular, the use of User Experience and gamification is getting more and more widespread. The power of similar approaches lays in their ability to raise immediate and spontaneous responses in users, involving also the less motivated one. This potential has been recognized by other research fields: the recent Nobel prize R.H. Thaler discusses some examples in his Nudge Theory [4]. “Nudges” are in fact cues to stimulate and simplify for humans the performance of desired behaviours. In particular, gamification elements can be effective nudges, as they address the human tendency to engage in entertaining, competitive and rewarding tasks. In this chapter, we explore the use of these techniques in Design for Sustainable Behaviour through a literature review of theories and case studies. Then we defined a framework for the design and development of applications and products to be used for behaviour change. We relate this framework to previous works and then we discuss two case studies.

5.2 Literature Review This section reports a critical analysis of the literature review on Design for Sustainable Behaviour and gamification integrated into product design to support behaviour change.

5.2.1 Design for Sustainable Behaviour Some research efforts regard the development of models, frameworks and tools to orient designers, supporting them in understanding the reason behind users’ behaviours and select the ideal solution. For instance, Fogg [2] describes a series of elements that affect behaviour, dividing them into elements of simplicity and motivation. The formers are related to the resources available for users, which can be of various types (e.g. mental, physical, temporal, economic). Those can indeed define the simplicity of a behaviour, or the users’ ability to accomplish it, regardless of their motivation.

5 Sustainable Behaviour: A Framework for the Design …

67

The latter describe the users’ motivation to accomplish a behaviour. In fact, when users have the ability to perform a desired behaviour, they often fail because it is not a priority for them. Powerful motivators are, for instance, social praise or exclusion, hope and fear: for example, the fear that the others could notice and disapprove the user’s actions, or the hope that by changing behaviour, some conditions in the user’s life or in the world will improve. In this sense, gamification can be a means to provide motivation to users: game elements as competition and rewards/penalties are elements related to social mechanisms and hopes/fears. Bhamra et al. [3] discuss different strategies for behaviour change mapping them to different phases. According to them, more freedom should be left to users at the beginning, to allow a conscious change, while the products should get control in a progressive way, ensuring the permanence of the change. Gamification cannot be part of the later stages, since it is not coercive. However, it does not seem the best option also considering the first phases. In fact, it often triggers spontaneous responses that leave small spaces for reflection and awareness, which are required to change habits in a durable way. Nevertheless, R.H. Thaler [4] points out that human choices and actions are often not dictated by rationality. Hence, he defines “nudges” as any aspect in the context of choice or behaviour that influence people decisions and actions in a predictable way without forcing the users or forbidding options. Gamification can be used in “nudges”, since it causes a change of behaviour by simply involving and amusing users. People are not aware that they have been “nudged”, but rather change behaviour in a spontaneous way which is not fully conscious. This represents a significant advantage when the behaviour to change is strongly habitual, and in public spaces when users have a limited time to interact with a product, since changes due to awareness and happening through a progressive improvement require time. This is particularly important in cases of bad habits. In fact, as we can notice in the next section, many examples of gamification address bad habits, such as littering, taking escalators to avoid physical exercise, and disposing of products without using recycling facilities.

5.2.2 Gamification for Behaviour Change Many companies, including Samsung, Siemens and Volkswagen, have proposed gamified experiences for various purposes from improving the customer experience and corporate promotion to discovering talents and hiring [5]. While the trend of serious games shows how games can be used to promote awareness and develop knowledge and skills, gamification can be used to modify an existing task, making it funnier to accomplish. In fact, game elements can be integrated into functional websites, applications and products. In particular, gamified products are defined and described in different ways. Dichev et al. [6] define gameful design as a kind of design in which “game elements

68

G. W. Scurati et al.

are used as design lenses to improve the overall experience of the task. […] the tasks themselves are supposed to be designed in a manner similar to game design”. In a similar way, Deterding et al. [7] define gamification as “the use of game design elements in non-game contexts”. They also distinguish playful and gameful design: the former presents more free-form and exploratory features, while the latter is characterized by goals and rules. There are many examples of how gameful and playful products can be used to stimulate healthier and more sustainable behaviour. FUMO [8] is a playful interactive ashtray that plays random music or sounds and displays light patterns when smokers throw cigarette butts inside it. This ashtray also makes users curious about trying new music tunes and lights, stimulating them to collect others’ cigarette butts. The Volkswagen car manufacturer, with its Fun Theory, proposed several case studies. An example of playful design is the Piano Stairs that stimulate the passersby to use stairs instead of escalators through playful stairs resembling a piano and emitting sounds. While users walk on them, each stair plays a different note. Instead, the Speed Camera Lottery takes a photo when a car passes by a crossing, measuring its speed. If the driver is above the speed limit, he/she will get a fine, as with any speed camera. However, in this case, there is also a lottery prize to which any driver respecting the speed can participate. This can be an example of gameful design, since users have to follow rules and accomplish a goal, even though very simple. The Bottle Bank Arcade is also gameful: it invites people to play an arcade game when collecting bottles [9]. Similarly, Coca-Cola proposed the Happiness Arcade, in which users have to insert plastic bottles to play a video game [10]. Also, the TetraBin consists of an urban trash bin displaying a Tetris game in response to the litters disposed of by users [11]. Public installations are also used to incite charity donations. In the case of [12], users can fix leaking tubes represented on screens, but to make the fix permanent they need to donate to an organization which promote accessibility to potable water. This can be related to the playful category, since there are no rules to follow, but it is rather based on the users’ exploration. However, the presence of a leader board, reporting the donation of different teams, introduces a competitive goal.

5.3 Gameful and Playful Products for Behaviour Change: A Framework This section presents the framework for the design and development of applications and products to be used for behaviour change. In the current scenario, the behaviour change of a great number of users is of fundamental importance, since only a collective change will allow to obtain effective and lasting results from the environmental sustainability point of view. This challenge can be faced through several approaches, dedicated to the awareness of new generations and also of adults to environmental problems and to the change of their bad habits. For this reason, products should be

5 Sustainable Behaviour: A Framework for the Design …

69

Fig. 5.1 Graphical representation of the framework

designed in such a way as to push users to more careful use of resources and to reduce their impact on the environment, and the gameful and playful approaches can be used in this direction. Gameful and playful products present some diversities and can be designed for different contexts, users and to trigger different behaviours. However, they share a similar kind of user experience and interaction scheme. Therefore, a framework has been developed to identify the main features of gameful and playful products, based on the analysis of the sensory and perceptual stimuli that can impact on user behaviour, and to define the strategies to integrate them into commercial products. Specifically, three main steps of the user–product interaction, each supported by different sensory stimuli, have been identified (see Fig. 5.1). First, the product attracts the user’s attention through features and signals, then it triggers the desired behaviour through cues and indicators and, finally, it provides a reward through feedback. All these steps of the framework should characterize gameful and playful products for behaviour change. Consequently, the authors used them to describe some of the products cited in the previous section (Table 5.1). Then, the framework has been used as a basis to develop new case studies, described in the next section.

5.4 Case Studies This section presents two case studies based on the framework described in the previous section. Both the case studies regard the use of gamification to support people in behaving in a more sustainable way, abstaining from performing a harmful behaviour in one case and reducing their impact in resource consumption in the other. In fact, the first case study consists of an ashtray to reduce cigarette littering on the university campus. Littering is indeed a habit that greatly damages the environment: it has been the focus of many previous works we described.

70

G. W. Scurati et al.

Table 5.1 Analysis of products cited in the previous section by using the framework Raise attention Features/signals

Trigger behaviour Cues/indicators

Reward Feedback

FUMO

The ashtray is a smokers pole modified with led lights

Cavities to throw away the cigarette butts resemble microphones

Random sounds/music and light patterns are played

Bottle Bank Arcade

The bottle bank resembles an arcade game

Lights placed on cavities indicate where to put the bottle

Points are displayed

Happiness Arcade

Text signals and screens indicate the presence of a game

Bottle-shaped cavity and text message

Users can play the game

Piano stairs

The stairs resemble a piano and emit sound as users walk on them

Each stair represents a piano key suggesting it plays a different note

A different sound is played as users step on each stair

TetraBin

A colourful Tetris game is visible on the bin surface

Cavities for litters, Tetris blocks moving on the bin surface

The blocks fall and pile up

Speed Camera Lottery

The camera reports text indicating the lottery game

The car speed is reported indicating if the users can take part in the lottery

Winners receive a monetary reward

Installation for donations

Screens representing leaking tubes inviting users to fix them

The fix is not permanent unless users make donations

The fix is permanent and water glasses are filled up

The second addresses hand-washing behaviour in a public space, to limit water consumption when users might not consider the economic value of the resources. In fact, despite automatic sinks using sensors are available in many buildings, it is still important to make users aware of the need to save energy and water. Hence, these cases address habitual behaviour in a public context, target young people and use a gameful design approach, since elements as challenges and goals are present. This kind of approach matches with the context and users, since public environments require designs that are able to change users’ behaviour immediately, as there is no time to create a conscious and long-lasting change of habit. In both cases, the authors have used the framework described in the previous section to define the concept of the product and the interaction modalities with the final users. Consequently, the prototype of each product has been developed, testing sessions with users have been performed and the collected data have been analysed. Moreover, a multisensory approach, combining visual and auditory stimuli, was applied to the case studies. In fact, they can be effective to accomplish every objective in the framework. Including sounds and images that are unusual in an everyday context easily raise people attention, while the fact that they are typical of a gaming

5 Sustainable Behaviour: A Framework for the Design …

71

one contributes to engage users and trigger the desired behaviour. Finally, expressing rewards through these stimuli has the double effect to provide satisfaction and attract the following users that may be present and assisting the scene. In fact, curiosity can cause imitation, in particular in the case of contexts where social activities are dominant as the one of smoking: this is described in the first case study hereafter presented.

5.4.1 AIM: An Interactive Ashtray Against Cigarette Littering AIM is a device to contrast cigarette littering in a university campus environment that stimulates users to throw their cigarette butts by aiming at the centre of the ashtray [13].

5.4.1.1

Interaction

AIM is designed to engage multiple users since the smoking activity in university campus is related to socialization and typically takes place in small groups during breaks, or before and after the classes. Similar to DropPit [14] users perform the gesture of throwing cigarette butts on the ground, since it is where the receptacle is placed. However, in this case a gamification approach is adopted, reminding the case of the Bottle Bank Arcade [9]. AIM is in fact inspired by the darts game: the smoker has to throw the cigarette butt aiming at the centre of the receptacle. AIM presents three different coloured areas which the cigarette butt can hit, and depending on this, the receptacle plays a different audio feedback. If the user hits the centre hole, a winning sound is provided. On the basis of the framework discussed in Sect. 5.3, AIM goes through the following steps: • Raise attention: The product attracts the user through the playful aesthetic and a signal laced aside which reports the sentence “PUT OUT and AIM!”. • Trigger behaviour: The product suggests where to through the cigarettes through the coloured circles as indicators, also letting the user imagine possible expected outcomes (the game recalls previous experiences with dart games). • Reward: The product returns a reward using different winning sounds (Fig. 5.2).

72

G. W. Scurati et al.

Fig. 5.2 Storyboard of the user–product interaction

5.4.1.2

Prototype

The bottom part of the prototype contains a pressure-sensitive conductive material called Velostat placed between two conductive layers of aluminium foil [15] to detect the cigarette butts. An Arduino Uno board [16] is used to read the data from the Velostat and send audio feedback through a speaker. The communication between the Arduino and the speaker is managed using a DF Player module [17] on which an SD card, containing the sounds to play, is installed. The prototype is powered by a 9 V battery. The Velostat material is cut to cover three areas, combined with three-inner outward-dipping surfaces, to decrease the number of bounced-off cigarettes, while a cover surrounding the whole structure avoids that cigarettes deflect outside. The three different areas in the bottom part are matched with three layers of yellow material, for the outer rings and red for the central one, to indicate where to aim. Finally, a small ashtray was mounted on a pole placed on the receptacle, with a signal reminding users to put off the cigarette before throwing it. Figure 5.3 represents the main components and illustrates where they are placed.

5.4.1.3

Test

The study consists of a 2 (two smoking areas on campus) × 2 (no intervention vs. with intervention) between-subjects test (Fig. 5.4). The main aim of the study was to test the effectiveness of our prototype in reducing the littering behaviours. However, it was interesting also to observe the cigarette butts littering among students smoking on campus for a given time span. This could allow to collect insights about the possible reasons, providing directions for future designs.

5 Sustainable Behaviour: A Framework for the Design …

73

Fig. 5.3 Architecture and components of the prototype

We hypothesized that the presence of AIM will reduce the littering behaviour among smokers compared to original conditions. However, we also expected that the behaviour would have been affected by other factors, especially those related to the context cleanliness, or to the availability and proximity of ashtrays. The study was performed by two observers recording data about people behaviours, taking care of not being noticed. At the same time, a third conducted interviews with random smokers. The same measurement was replicated in the four sessions (two locations and two conditions: with and without the prototype) to compare them afterwards.

5.4.1.4

Results and Discussion

Figure 5.5 represents the percentage of cigarette butts littering behaviour in two different campus locations considering total littering among all the observed student smokers in that session. The presence of the prototype effectively reduced the amount of littering in both locations. In fact, the number of littering behaviours falls greatly, especially in location A. This could be due to the different environmental cleanness in the two areas. In fact, the perceived cleanness level of the environment in location A without the prototype, (M = 7.45; SD = 0.65) together with the prototype (M

74

G. W. Scurati et al.

Fig. 5.4 The prototype placed in location A (on the left) and a user interacting with the prototype in location B

Fig. 5.5 Percentage of cigarette butts littering behaviour in two different campus locations without and with AIM

5 Sustainable Behaviour: A Framework for the Design …

75

= 7.95; SD = 1.01), is higher than in location B without the prototype (M = 5.18; SD = 1.18) and with the prototype (M = 5.90; SD = 0.80). This result is consistent with previous works that also suggest humans tend to litter more in places that are perceived as dirty, or in which there are signs of previous littering [18, 19]. Moreover, in location B there were about double the smokers than in location A: in the session with the prototype, they were 43 in location B and 23 in location A. Hence, in location A it was easier to be in proximity and exposed to the prototype. Furthermore, since location B was more crowded, this could have made it harder finding a bin. 16 subjects, equally distributed in the two locations and conditions (with–without AIM) with an average age of 21.75 (SD = 2.27) were interviewed. 87.5% of smokers confirmed that they usually smoke with classmates on campus, having conversations in the meanwhile. This was also happening during the experimental sessions and support the idea that, in a similar context, a gamified ashtray should be designed for multiple smokers. We also questioned smokers about potential reasons for littering their cigarette butts: they frequently mentioned the lack of availability and proximity of the ashtray, similarly to previous studies [18, 20, 21]. Moreover, 8 subjects, equally distributed in the two locations, reported their experience using AIM rating it through 12 pairs of adjectives. In Fig. 5.6, the purple circle represents the result in location A, and the yellow triangle is for location B. The prototype received similar average evaluations in the two locations. In particular, it was found easy to learn, friendly, pleasant and innovative by most of the users. All the participants agreed that they were willing to use the new type of interactive ashtray since it is interesting and engaging, especially when smoking with friends.

Fig. 5.6 Subjects’ rating of the AIM (purple circles for location A, yellow triangles for location B)

76

G. W. Scurati et al.

5.4.2 A Gamification on a Sink for Water Saving During Handwashing This case study is focused on the gamification of the handwashing experience, using a metaphor to create and the association between the running water and economic value. We developed a temporary installation designed for a public space used by young adults [22]. The prototype includes a screen mounted on a sink and playing an animation: coins are progressively dissolved in water splashes with the aim to reduce the consumption of water in public environments. The prototype sets an amount of necessary water for handwashing (1.5 gallons). In fact, USGS water science school [23] suggests that the appropriate amount of water for people to wash their hands in a hygienic way is about 1–1.5 Gallons, equals to 3.79–5.68 litres. We adopted it as a reference parameter for designing the interactive prototype. In order to win the game, no more than this amount should be used.

5.4.2.1

Interaction

With the aim to link the idea of money-consuming to the water use, as represented in Fig. 5.7, we defined three phases: 1. As a user opens the tap, ten coins will appear and while the water flows, the coins fade away one by one into water splashes; 2. if less than 1.5 gallons (5.68 litres) have been used when the user ends washing his or her hands, then a positive sound expressing success will be played; 3. if the user exceeds the established water consumption level, a negative sound, representing failure will be played. Also, in this case, on the basis of the framework discussed in Sect. 5.3, the interaction follows these steps:

Fig. 5.7 Storyboard of the user–product interaction

5 Sustainable Behaviour: A Framework for the Design …

77

• Raise attention: The product attracts the user with the presence of an unusual device close to the sink, raising attention when the user turns on the tap and images appear. • Trigger behaviour: The played animation allows us to associate water with money, encouraging the user to stop coins from fading away. • Reward: The product returns a reward with a winning sound or a losing sound.

5.4.2.2

Prototype

The prototype includes the components shown in Fig. 5.8: (1) a Water Flow Sensor (model: YF-S201) to measure the water flow rate and detect the water flow; (2) a LCD to show instant feedback considering the water consumption in litres; (3) a switch simulating the water tap; (4) a TransFlash memory card with a DFPlayer Mini, to store, decode and play MP3 files; (5) a Digital Speaker Module to provide audio feedback and (6) an Arduino board to control the system. All these items are connected in a circuit and arranged into a white box. Moreover, in the experimental tests, we used an Apple iPad display to show the animations.

Fig. 5.8 Architecture and components of the prototype

78

5.4.2.3

G. W. Scurati et al.

Test

A user study was performed to validate the effectiveness of the animation in conveying the water-saving message compared to a more traditional media, in this case, a poster (Fig. 5.9) and also to evaluate the whole user experience. In condition 1, a poster was attached close to the sink, and subjects were indicated which sink to use, to make sure that they were exposed to the message (left images in Fig. 5.9). Instead, in condition 2, subjects were invited to try the prototype that was placed above the sink. No instruction or information about the purpose where provided (right images in Fig. 5.9). Finally, we asked participants to answer a shortened version of the user experience questionnaire. We created the questionnaire selecting adjectives from the original edition by Martin Schrepp [24], according to the aspects we wanted to evaluate in our design. It contains some negative adjectives (e.g. annoying) and positive adjectives (e.g. interesting) related to the overall experience and appreciation (e.g. exciting) ability to trigger a change (e.g. motivating) and clarity in conveying the message and the product functioning (e.g. confusing). Those terms were listed in a random way, and participants had to rate them using a 7-point Likert scale from totally disagree (left) to totally agree (right). Demographic information was also collected, and we questioned participants about previous experience with water-saving devices, especially related to the use of sinks. Two experimental sessions were conducted, during which subjects’ handwashing behaviour was coded and commented by two different observers at the same time. The coding sheet for recording data was handed in two recorders in advance. 23 subjects were recruited among students in the school of design, and voluntarily participated in the tests. In condition 1, observers recorded the handwashing behaviour of 13 subjects (including 8 females), with a mean age of 24 (SD = 1.67). In condition 2, 10 subjects

Fig. 5.9 Material used in the user study: condition 1, with a poster attached close to the sink, and condition 2, with the prototype placed above the sink

5 Sustainable Behaviour: A Framework for the Design …

79

(gender evenly distributed), with a mean age of 21 (SD = 1.89), tested the prototype. The experiments took place in a bathroom located on the university campus, in the area in which washbasins are situated, separated from toilets.

5.4.2.4

Results and Discussion

Figure 5.10 illustrates the recorded handwashing time in two experimental conditions, reporting the average values of two observers’ coding data. Blue bars refer to condition 1 (poster), yellow ones indicate condition 2, using (prototype). As we can see, the use of the prototype had an effect on the handwashing time. In particular, there was a significant drop among male subjects, since the average handwashing time is almost halved. For the female group, the recorded time was lower in the beginning and this drop is less consistent. This difference could due to the fact that the female population is generally more sensitive to environmental issues and tends to behave in a pro-environmental way more than the male population [25]. We specify that the standard error in male subjects of condition 1 is considerably higher than the others, as one subject reported he had been distracted by talking to other people while using the sink. Considering the users’ evaluation of the designed prototype (Fig. 5.11), there was a significant agreement on positive adjectives and disagreement for the negative ones, while concerning the poster participants showed less strong opinions. Positive evaluations were made for the adjectives related to the overall appreciation and experience. For instance, subjects strongly agree (50%), agree (40%), or slightly agree (10%) that our design is interesting, while most of them slightly

Fig. 5.10 Recorded handwashing time in the two experimental conditions

80

G. W. Scurati et al.

Fig. 5.11 Users’ evaluation of the prototype

disagree (38%), disagree (23%), or strongly disagree (15%) regarding the poster. Similar results were shown for definitions as exciting and innovative. Similarly, participants agree (60%), strongly agree (10%), and slightly agree (20%) that the prototype was pleasing, while only the 8% strongly agree and the 8% slightly agree giving this definition to the poster. Considering adjectives more strictly related to the ability of triggering behaviour change, subjects strongly agree (40%), agree (20%) and slightly agree (20%) that the device was supportive, while considering the poster, results are contrasting: 31% disagree, while 38% slightly agree. In a similar way, the prototype was evaluated as motivating by most of the participants: 40% strongly agree, 30% agree and 20% slightly agree. Instead, the poster was rated as motivating by fewer users (23% slightly agree and 8% agree). Furthermore, the prototype was judged unpredictable (40% agree and 10% slightly agree) by more subjects compared to the poster which was assigned this adjective by few participants (15% strongly agree, and 8% agree). However, in both cases, a

5 Sustainable Behaviour: A Framework for the Design …

81

small percentage of subjects defined the design as confusing (10% strongly agree considering the prototype; 8% agree and 8% slightly agree for the poster). Analogue results are shown for the term complicated. It is interesting to note that the prototype was rated as more understandable (50% strongly agree, 20% agree and 20% slightly agree) than the poster, which results are contrasting (in particular 23% disagree, 23% slightly disagree and 23% strongly agree). In a similar way, 80% of the participants strongly disagree that the prototype was complicated, while 53% strongly disagree regarding the poster. Neither the poster nor the prototype was considered unattractive or annoying, but stronger statements were chosen for the prototype rather than for the poster. The results we presented show that the interactive device was able to involve users and trigger a behaviour change, while the effectiveness of the traditional approach was limited. This can be seen in the recorded handwashing time, and is reinforced by the fact that participants rated the device as supportive and motivating. Considering the final satisfaction with the product, we can also state that the rewarding approach was effective, as it is evident by the positive adjectives used to describe the product (e.g. interesting, exciting). Moreover, despite we could expect a traditional media to be perceived as clearer compared to a new interactive device, the prototype was able to provide the water-saving message in an understandable way. Furthermore, the overall interaction with the product was judged as simple and not confusing by the participants: it should be also considered that none of them reported previous experience with water-saving devices.

5.5 Conclusions The above-presented case studies show that it is possible to use the framework developed by the authors for the design of applications and products to be used for behaviour change. These case studies have been defined and developed on the basis of the three steps of the framework, by using the gamification approach to support people in behaving in a more sustainable way. In particular, both case studies focused on user interaction, and the products have been designed to attract the user’s attention, to trigger the desired behaviour and, finally, to provide rewards. Subsequently, the prototypes of the products have been developed and testing sessions with users have been conducted to verify the impact of products on users’ behaviours and to collect users’ feeling and responses. In both cases, from the analysis of the collected data, it is possible to state that products have successfully modified the users’ behaviours, although users have not been asked to modify their behaviours, nor have they received any information about the products. This was made possible by the first two steps of the framework, which actually attract the attention of the users and support them in understanding the function and the operation of the products. Furthermore, by inviting users to carry out an activity seen as playful, rather than as tiring or difficult, the products have been

82

G. W. Scurati et al.

accepted and appreciated by users. Finally, the third step of the framework, in which the products provide rewards, allows designers to communicate users that they are actually helping to reduce their impact on the environment. With regard to the benefits of the presented framework, its implementation can lead to the widespread of sustainable behaviours, also and above all in the case of small daily behaviours that can substantially contribute in reducing our impact on the environment. Even more, the use of this framework and of playful products could contribute at increasing the environmental awareness even of groups of the population currently little interested in the topic, or who believe that changing their behaviours may be too tiring. In fact, although these products can have a reduced direct impact on some deeply rooted bad behaviours, they can help improving the popular opinion regarding the importance of changing our lifestyles and becoming more sensitive to environmental sustainability.

References 1. Norman DA (1988) The psychology of everyday things. Basic books 2. Fogg BJ (2009) A behavior model for persuasive design. In: Proceedings of the 4th international conference on persuasive technology. ACM 3. Bhamra T, Lilley D, Tang T (2011) Design for sustainable behaviour: using products to change consumer behaviour. Des J 14(4):427–445 4. Sunstein C, Thaler R (2008) Nudge: improving decisions about health, wealth, and happiness. Yale University Press 5. Park HJ, Bae JH (2014) Study and research of gamification design. Int J Softw Eng Appl 8(8):19–28 6. Dichev C, Dicheva D, Angelova G, Agre G (2015) From gamification to gameful design and gameful experience in learning. Cybern Inf Technol 14(4):80–100. https://doi.org/10.1515/ cait-2014-0007 7. Deterding S, Sicart M, Nacke L, O’Hara K, Dixon D (2011) Gamification using game-design elements in non-gaming contexts. In: Proceedings of CHI’11 extended abstracts on human factors in computing systems Vancouver, Canada, 7–12th ACM May 2011, pp 2425–2428. https://doi.org/10.1145/1979742.1979575 8. https://www.fastcompany.com/3030479/meet-fumo-the-silliest-possible-way-to-fix-cigarette littering 9. Kim B (2015) Designing gamification in the right way. In: Kim B (ed) Library technology reports, ALA TechSource 51(2), 10–18 10. https://www.coca-colacompany.com/stories/coke-gamifies-recycling-with-happiness-arcade 11. http://www.tetrabin.com/ 12. Nguyen TA, Kodinsky D, Skelton W, Kaur P, Yin Y, Mathew A, Basapur S (2012) Interactive philanthropy: an interactive public installation to explore the use of gaming for charity. In: Conference on designing interactive systems, pp 482–485 13. Huang S, Scurati GW, Elzeney M, Li Y, Lin X, Ferrise F, Bordegoni M (2019) AIM: an interactive ashtray to support behavior change through gamification. In: Proceedings of the design society: international conference on engineering design, vol 1, no 1. Cambridge University Press, pp 3811–3820 14. https://www.thedroppit.eu/ 15. https://www.adafruit.com/product/1361

5 Sustainable Behaviour: A Framework for the Design …

83

16. https://www.arduino.cc/ 17. https://www.robotics.org.za/MP3-PLAY 18. Schultz P, Bator R, Large L, Bruni C, Tabanico J (2013) Littering in context personal and environmental predictors of littering behavior. Env Behav. 45:35–59. https://www.doi.org/10. 1177/0013916511412179. 19. Cialdini RB, Reno RR, Kallgren CA (1990) A focus theory of normative conduct: recycling the concept of norms to reduce littering in public places. J Pers Soc Psychol 58(6):1015–1026. https://doi.org/10.1037/0022-3514.58.6.1015 20. Bator RJ, Bryan AD, Schultz PW (2011) Who gives a hoot?: Intercept surveys of litterers and disposers. Environ Behav 43(3):295–315. https://doi.org/10.1177/0013916509356884 21. Finnie WC (1973) Field experiments in litter control. Environ Behav 5(2):123–144. https:// doi.org/10.1177/001391657300500201 22. Scurati GW, Huang S, Wu S, Chen T, Zhang Y, Graziosi S, Ferrise F, Bordegoni M (2019) Multisensory nudging: a design intervention for sustainable hand-washing behavior in public space. In: Proceedings of the design society: international conference on engineering design, vol 1, no 1. Cambridge University Press, pp 3341–3350 23. https://water.usgs.gov/edu/ 24. Laugwitz B, Held T, Schrepp M (2008) Construction and evaluation of a user experience questionnaire. In: Symposium of the Austrian HCI and usability engineering group. Springer, pp 63–76 25. Hunter LM, Hatch A, Johnson A (2004) Cross-national gender variation in environmental behaviors. Soc Sci Q 85(3):677–694. https://doi.org/10.1111/j.00384941.2004.00239

Chapter 6

Quantifying Trust Perception to Enable Design for Connectivity in Cyber-Physical-Social Systems Yan Wang

Abstract Cyber-physical systems possess highly integrated functions of data collection, data processing, communication, and control. Given their seamless integration with human society, they are also termed as cyber-physical-social systems (CPSS). The advanced capabilities and functions of CPSS rely on their highly networked working environment and deep interdependency. The effectiveness of their performance critically depends on what and how they share among each other. Designing a trustworthy CPSS network, which can work together collaboratively thus is important. To perform systems design, quantitative measures of trustworthiness are required. In this chapter, quantitative metrics of trustworthiness, including capability, benevolence, and integrity, are proposed based on a generic probabilistic graph model of CPSS networks. The proposed metrics can be calculated from either subjective perception or objective statistics of sensing, computing, and communication functions in CPSS networks. A design optimization framework based on the trustworthiness metrics is also demonstrated.

6.1 Introduction Cyber-physical systems (CPS) are integrated software–hardware systems which possess all four functions of data collection, data processing, communication, and control. That is, sensing, computing, communication, and actuation are highly integrated in CPS. The unprecedented values of CPS rely on the communication networks, which is also referred to as the Internet of Things (IoT). The original idea of IoT was to extend the capability of radio-frequency identification chips with Internet connectivity. Later, the concept was generalized to any physical objects with sensing, computation, communication, and actuation capabilities. We can envision that in the future any object we interact with in our daily lives would probably have the functions of data collection and exchange, be it thermostat, pen, car seat or traffic light. Given that human society has become dependent upon CPS for communication Y. Wang (B) Georgia Institute of Technology, 813 Ferst Drive NW, Atlanta, GA 30332-0405, USA e-mail: [email protected] © Springer Nature Switzerland AG 2020 S. Fukuda (ed.), Emotional Engineering, Vol. 8, https://doi.org/10.1007/978-3-030-38360-2_6

85

86

Y. Wang

and decision-making, and CPS has been seamlessly integrated with human society, cyber-physical-social systems (CPSS) need to be designed and engineered from a holistic perspective. Designing CPSS products is the major task of future engineers. CPSS products are meant to be Internet-ready. The functions of CPSS are the collected efforts from individuals. The consideration of CPSS network architecture should be part of the product design. Design for connectivity is the new design paradigm for future CPSS products. One product is not an isolated system. Rather, it is an open system that can be reconfigured and re-adapted into the evolution of the Internet itself. Therefore, the concept of open system design with robust and diverse connectivity becomes important for such products and systems. The confederated systems formed by IoT compatible CPSS products do not have centralized control and monitoring units. Ad hoc networks are formed by vastly different types of products. Good adaptability is important in designing the architecture of such systems. In addition, the advanced capabilities and functions of CPSS rely on their highly networked working environment and deep interdependency. The effectiveness of their performance critically depends on what and how they share among each other. Research issues such as resiliency, interoperability, sustainability, adaptability, usability, scalability, safety, privacy, security, and trust for CPSS need to be studied. Among different research issues of design for connectivity, trust is a new challenge for engineers to design CPSS. The functionality of CPSS relies on extensive information sharing between them through networks. One has to trust others on how the shared information is handled and used when it is given out. Human users who interact with CPSS also need to feel comfortable about personal information being collected and shared. Trustworthiness directly affects the information sharing policies that are adopted by CPSS products, which in turn influences the design of networks. Therefore, trustworthiness is an important research area in CPSS design. In the domain of systems engineering, there is a lack of formal methods to study how trustworthiness can affect system design. Designing a trustworthy network that CPSS can work together and collaboratively thus is essential. For a CPSS product, how to intelligently automate the process of choosing trustable partners and form a strategic network for the benefits of itself and its network is part of the design considerations. In this chapter, how to design trustworthy strategic networks of CPSS is studied. In human societies, networks and groups are naturally formed based on the interaction experiences and trust relationships. An individual tends to choose trustworthy partners to collaborate. This social behaviour in the context of CPSS is referred to as the formation of trustworthy strategic networks. In general, an individual CPSS node is willing to share more information within its trustworthy strategic network and relies on the information obtained within the network for its decision-making. As illustrated in Fig. 6.1, CPSS nodes form networks and share information. Based on the different trust levels, different societies can be formed with respect to a trustor. The trustworthiness metrics can help the trustor to identify its strategic networks at different trust levels. From the system perspective, strategic networks help organizations to achieve bigger goals than individual ones can. Users of CPSS can design information sharing policies according to their strategic relationships. Therefore,

6 Quantifying Trust Perception to Enable Design for Connectivity … Fig. 6.1 Societies or strategic networks with respect to Node 2 formed based on different trust levels

87

5

1 0

3

2

4

7

6

9 Society A

8

10

Society B

a formal design formalism of identifying trustworthy nodes to form the strategic network is necessary. Quantitative metrics are essential for engineering design, during which design objectives can be measured to evaluate design alternatives and design can be ultimately optimized. Here, trustworthiness is quantified in order to support strategic network identification and design. We develop quantitative metrics of trustworthiness for CPSS networks. Specifically, trustworthiness is measured by the quantities of ability, benevolence, and integrity. In a trustor–trustee relation, the ability of the trustee indicates how well it is able to perform. Benevolence shows whether the motivation of the trustee is for the benefit of the trustor or purely for the benefit of the trustee itself. The integrity of trustee measures if it does what it claims. The concepts of ability, benevolence, and integrity for trust were originally proposed by Mayer et al. [22] who identified a dozen of key behaviours that would affect organizational trust and grouped them into three major categories of ability (e.g. expertise, competency), benevolence (e.g. loyalty, openness, receptivity, availability) and integrity (e.g. consistency, discreetness, fairness, promise fulfilment, reliability). However, in the above as well as many other research efforts in organizational behaviour and human psychology, the studies remained at the conceptual level. There is a lack of quantitative metrics for ability, benevolence, and integrity, which are important for engineering design. In our new trustworthiness metrics for CPSS, the quantification is based on a generic probabilistic graph model of networks [31, 33], where information exchange and processing at nodes are modelled with prediction and reliance probabilities. Prediction probability measures how well a node can make sound judgement and decision, whereas reliance probabilities capture the extent of influence for one node to another via information exchange. In our A-B-I (ability-benevolence-integrity) model for trust quantification, the ability of a node is quantified with its sensing and prediction capability as well as its influence on other nodes in their decision-making. Benevolence is quantified by the extent that a node is willing to share information, as well as the motivation of sharing. Integrity is to measure how predictable, honest, and consistent the node is in terms of prediction and information sharing. With these quantitative metrics, network design and optimization can be performed. Note that the three metrics are independent of each other. Multi-objective design optimization procedures need to be performed if multiple metrics are considered simultaneously. In the rest of this chapter, the background of related work in trustworthiness study in computer science and CPS is first given. The probabilistic graph model is also

88

Y. Wang

introduced. Then the new trustworthiness metrics of the A-B-I model are described. The strategic network design and optimization based on the trustworthiness metrics are demonstrated with examples.

6.2 Background In this section, the existing studies of trustworthiness metrics in computer science and CPS domains are summarized. The recently proposed probabilistic graph model, as the foundation of the A-B-I model for trust quantification, is also introduced.

6.2.1 Trustworthiness Metrics In the domains of social sciences such as management and human organization, trust relationship has been studied extensively. Computer scientists traditionally treated trust as security policies for credential exchange, access control, and referral reputation [13, 15]. Recently, it was expanded to semantic web [11] and social networks [27]. Data security is critical for trust. However, security alone cannot guarantee the trustworthiness. Although security protocols can ensure data that are not intercepted during transmission, they provide no guarantee against the misuse by the receiving party or against fraud by the transmitting party. There is only limited work on how to quantify the levels of trust. Most researchers regarded trust as reputation in networked environments [14, 21, 28]. The levels of trust are calculated from users’ explicit ratings and recommendations. Trustworthiness has been quantified with the number of positive and negative experiences [5], scaled reputation ratings in social networks [38], the number of successful transactions [18], and users’ ratings in combination with similar opinions and preferences [24]. The qualitative A-B-I trust model [22] has also been adopted in the applications of e-commerce [17], online banking [37], information systems [1], and system integration [10]. Nevertheless, the perceptions of A-B-I trust levels were measured mostly through survey studies and remain at an abstract level. In communication networks, trust was quantified as quality of services [12] such as numbers of forwarded data packets, executed routing protocols, and modified packet addresses [29]. In sensor networks, trust has been quantified as average success rates of transactions [20], weighted average of consistency factors including consistency of individual nodes from their historical data [40] and consistency between nodes in local regions [8], as well as neighbours’ data forwarding behaviour [26]. To model the stochastic and subjective nature of trust, probabilistic approaches have been adopted. These include modelling trust as belief about information reliability [4], credibility within the framework of Dempster–Shafer evidence theory [39], expectation of fulfilled commitment [25], probability of resource availability [16], and average information entropy in data exchange [19].

6 Quantifying Trust Perception to Enable Design for Connectivity …

89

Recently trustworthiness was studied in the context of IoT. Different approaches have been used to quantify trustworthiness. Chen et al. [6] developed a fuzzy model of communication reputation factors including packet forwarding, package delivery, and energy efficiency. Saied et al. [30] quantified trustworthiness as user ratings and recommendations. Nitti et al. [23] combined quality of service and opinions of credibility to measure trustworthiness. Chen et al. [7] treated trust as the combination of an overall probabilistic assessment from direct interaction and the social similarity in a recommendation system. Al-Hamadi and Chen [2] calculated trust as aggregated user ratings. In the above approaches, the perception of trust in a social environment is not explicitly modelled. Only the communication function of IoT objects is focused on, whereas the functions of sensing and reasoning are not considered. For CPSS networks, trustworthiness needs to be measured from the perspectives of sensing, communication, reasoning, and execution. The quantitative metrics of ability, benevolence, and integrity in our A-B-I model are based on a generic probabilistic graph model as an abstraction of sensing, communication, and reasoning functions.

6.2.2 Probabilistic Graph Model In the probabilistic graph model [31, 33] for CPSS networks, each node has its own sensing, reasoning, and communication units. As illustrated in Fig. 6.2, there are probabilities associated with information gathering and exchange between nodes. For each node, there is a prediction probability indicating the performance of sensing and reasoning. For each directed edge for information exchange, there are associated two reliance probabilities which indicate the communication and level of information dependency. The three probabilities are defined as follows: The prediction probability that the i-th node detects the true state of world θ is P(xi = θ ) = pi Fig. 6.2 Probabilistic graph model of CPS systems

(6.1)

pi k

i

qji pji

pij qij j

pj

90

Y. Wang

where xi is the state variable. The information dependency between nodes is modelled by P-reliance probabilities, defined as   P x j = θ |xi = θ = pi j

(6.2)

which is the probability that the jth node predicts the true state of world given that the ith node predicts correctly. In addition to positive correlation, the negative correlation is captured by Q-reliance probabilities, defined as   P x j = θ |xi = θ = qi j

(6.3)

because the reasoning and prediction algorithms in a node may have inherent bias towards information shared by some other nodes, or miscommunication between nodes may exist. Therefore, different from the adjacency matrix in traditional graph model with binary ‘yes-or-no’ edge connection topology, there are reliance probabilities associated with each pair of nodes in the proposed model. If the communication channel from node i and node j is disrupted, both pi j and qi j are zeros. The random state variables with binary values (=θ or =θ ) can be extended to multiple values or continuous. For instance, instead of only providing the sense of ‘cold’ and ‘hot’, one sensor measures a temperature value that follows some distribution, as a prediction probability. When the state variable takes a discretized value θn out of {θ1 , . . . , θ N }, the prediction probability is P(xi = θn ). Similarly, the reliance probability is P(x j = θm |xi = θn ) where P-reliance and Q-reliance probabilities are generalized. The reliance probability can be used to model reliability of communication between nodes. For instance, in ad hoc wireless networks of moving vehicles, data packet loss is not uncommon. It captures the correlation of predictions between nodes in general. In this work, we focus on the simplified case of binary state values. The probabilistic graph model provides a mesoscale abstraction of large-scale CPSS networks, where information exchange and aggregation are captured. The aggregation of information can be performed at each node as the reasoning process. For instance, with a worst-case or pessimistic rule, a node predicts that the state variable is positive only if all information that it obtains from its neighbours shows a positive value. In contrast, with a best-case or optimistic rule, a node predicts that the state variable is positive if one of its neighbours would share a positive value. The prediction and reliance probabilities can be obtained from the historical statistics of sensing, computing, and communication. The sensing and prediction errors are the indictors of prediction probabilities. The number of packet loss is an indicator of communication reliability and can be used to infer P-reliance probabilities. Similarly, the history of computational results for the same quantities by neighbouring nodes can also be used to estimate P- and Q-reliance probabilities.

6 Quantifying Trust Perception to Enable Design for Connectivity …

91

6.3 Trustworthiness Metrics The estimation of the level of trust helps identify and assess the strategic partnerships between CPSS nodes. The three quantitative metrics used here to measure trustworthiness include: ability (general competence and expertise of the trusted party or trustee), benevolence (the extent to which the trusting party or trustor believes that the trustee acts for the welfare of the trusting party, rather than just maximizes its own profit), and integrity (the trusting party’s perception that the trusted partner will be honest and adhere to an acceptable set of principles). Trustable CPSS products are also evaluated from these three perspectives. When designing CPSS products and networks, these three criteria can be applied for optimization and decision-making. Based on the mesoscale probabilistic graph model of CPSS networks, the ability of a node is quantified as the perception of its capability to predict the true state of the world and its influence on other nodes in its neighbourhood. Benevolence is quantified by reciprocity which measures the willingness and effectiveness of information sharing in the networks as well as the motivation of sharing. Integrity is quantified as the perception of consistency for a node’s performance.

6.3.1 Ability The ability of a CPS product is evaluated by its capability of prediction and its influence on others’ prediction in the networks. In the probabilistic graph model, the capability of a node is the perceived probability that it predicts the true state of the world accurately based on its gathered information from its own sensing unit and its neighbouring nodes. The influence of a node is about the extent of leadership to influence its neighbours.

6.3.1.1

Capability

The perceived capability of node j for a correct prediction of the state of the world is    A j (θ ) = P P x j = θ

(6.4)

where P() denotes perception. Suppose that the perceptions follow Gaussian distributions. The expectation of the perception is   E A j (θ ) = p j

(6.5)

  V A j (θ ) = τ j−1

(6.6)

and variance is

92

Y. Wang

where τ j is usually regarded as the precision of prediction from node j. In a highly networked environment where exchanged information is regularly used to update the perception, the perception of capability for a node can be continuously updated once new information is available. Similarly, the perceptions about P- and Q-reliance probabilities for nodes i and j are    L i j = P P x j = θ |xi = θ

(6.7)

   L icj = P P x j = θ |xi = θ

(6.8)

and

respectively, and their expectations given in the prediction capability are   E L i j |A j = pi j

(6.9)

  E L icj |A j = qi j

(6.10)

The variances of the perceptions are   V L i j |A j = τi−1 j, p

(6.11)

  V L icj |A j = τi−1 j,q

(6.12)

The information update is associated with the neighbouring nodes. By neighbours, we mean the topologically adjacent nodes in the graph. Based on the direction of information flow, the neighbouring nodes are categorized as source nodes and destination nodes, as illustrated in Fig. 6.3. The set of source nodes with respect to node j is denoted as S j , whereas the set of destination nodes is denoted as D j . Node Fig. 6.3 Source and destination nodes with respect to node j

pi i

source

pij qij

pj j

pjk qjk

pk k

destination

6 Quantifying Trust Perception to Enable Design for Connectivity …

93

j receives information from its source nodes and sends information to its destination nodes. Based on Bayes’ rule, the perception of capability can be updated with additional reliance probability information. As the further simplification, the variances of the perceived reliance probabilities are assumed to be the same, as τi j, p = τs, p and τi j,q = τs,q for all i ∈ S j . To simplify the notation further, the complete set of Pand Q-reliance probabilities associated with the source nodes with respect to node j    c (+ j) = L i j |i ∈ S j ∪ L i j |i ∈ S j . With the new information from is denoted by L the source nodes, the updated perception of capability then has the expectation     τ j p j + τs, p i∈S j pi j + τs,q i∈S j qi j (+ j) E A j (θ |L ) = τ j + s j τs, p + s j τs,q

(6.13)

and variance −1    V A j (θ |L(+ j) ) = τ j + s j τs, p + s j τs,q

(6.14)

where s j = S j denotes the number of source nodes for node j. From the model in Eq. (6.13), it is seen that the perceived capability of a node is affected by not only its prediction capability but also its ability of absorbing information from other resources. The effectiveness of information absorption can be regarded as the capability of learning. The influences of information sources have different effects and are characterized by different weights. This Bayesian belief update procedure is a generic information assimilation process that has not only been applied in computational reasoning, but also been observed in human neurological activities for combining multisensory cues [3, 9].

6.3.1.2

Influence

The ability of an individual node is also measured by its influence on others in a society. If one has strong influences on others in their decision-making process, its ability thus trustworthiness is high. In the CPSS networks, the influence can be measured with the effectiveness of information sharing with others. set of P- and QFor node j with a set of destination nodes D j , the complete    reliance probabilities is denoted by L(− j) = L jk |k ∈ D j ∪ L cjk |k ∈ D j . With the assumptions of τ jk, p = τd, p and τ jk,q = τd,q for all k ∈ D j , the expected perception of the ability of node j given its prediction capability and its influence on the destination nodes can be estimated as     τ p + τ p + τ 1 − q   j j d, p jk d,q jk k∈D k∈D j j (6.15) E A j (θ |L(− j) ) = τ j + d j τd, p + d j τd,q

94

Y. Wang

and variance as −1    V A j (θ |L(− j) ) = τ j + d j τd, p + d j τd,q

(6.16)

where d j = D j is the number of destination nodes with respect to node j. The Bayesian reasoning approach is the same as the previous where the capability of learning or information absorption is considered.

6.3.1.3

Overall Ability

The overall ability of node j with the considerations of prediction and learning capabilities as well as influence is obtained as

   E A j θ |L(+ j) , L(− j) =

   τ j p j + τs, p i∈S j pi j + τs,q i∈S j qi j     +τd, p k∈D j p jk + τd,q k∈D j 1 − q jk τ j + s j τs, p + s j τs,q + d j τd, p + d j τd,q

(6.17)

and   −1   V A j θ |L(+ j) , L(− j) = τ j + s j τs, p + s j τs,q + d j τd, p + d j τd,q

(6.18)

The consideration of more factors leads to more precise perceptions of ability. The overall variance in Eq. (6.18) is smaller than the ones in Eqs. (6.6), (6.14) and (6.16). The expectation can be regarded as the weighted average from the perceived capabilities of prediction, learning and influence.

6.3.1.4

High-Order Perception

In a typical social environment, one’s perception will affect other’s perceptions. That is, one’s perception is determined by all available information, which includes other’s perception as well. Therefore, the perceived ability of node j in Eqs. (6.17) and (6.18) can be further updated with the additional perceptions about the immediate neighbours of node j, particularly its destination nodes.  perceptionin Eqs. (6.17) and (6.18) are simplified as  of first-order  If the notations E A j (θ |+, −) = E j and V A j (θ |+, −) = V j , then the second-order perception is

    V j−1 E j + τd, p k∈D j p jk Vk−1 E k + τd,q k∈D j 1 − q jk Vk−1 E k   E(2) A j (θ |+, −) =     V j−1 + τd, p k∈D j p jk Vk−1 + τd,q k∈D j 1 − q jk Vk−1

(6.19)

6 Quantifying Trust Perception to Enable Design for Connectivity …

95

and ⎤−1 ⎡      V(2) A j (θ |+, −) = ⎣V j−1 + τd, p 1 − q jk Vk−1 ⎦ p jk Vk−1 + τd,q k∈D j

k∈D j

(6.20) where reliance probabilities are part of the weights associated with the perceptions. That is, the ability of a node is also measured through the abilities of those nodes that are directly influenced by this node. If a ‘high achiever’ is included in the destination nodes of one node, the ability of this node is also perceived to be high. Similarly, the third-order perception of a node includes the perceptions about its neighbours as well as those of its neighbours’ neighbours. A higher order perception incorporates the lower order perceptions and can be defined recursively.

6.3.1.5

An Illustrative Example

Here we use the simple example in Fig. 6.1 to illustrate the ability perceptions. A total of 11 nodes are included in the network. Nodes share information and make decisions in two different modes. As a result, the perceived abilities of the nodes are different. In the first case, nodes work in a ‘collaborative’ mode. The mean values of prediction probabilities for the 11 nodes are assumed to be 0.5, and the variances are 0.3. The means of P- and Q-reliance probabilities for all edges are assumed to be 0.9 and 0.1, respectively. The variances for reliance probabilities are 0.1. This is a scenario where individual node’s sensing capabilities are limited and they need to work collaboratively. The nodes rely on the shared information in decision-making. The ability perceptions of all nodes, including the capability, influence, overall ability, and the second-order ability perception are shown in Fig. 6.4a, where the mean values are denoted by dots and standard deviations are denoted by error bars. It is seen that additional information from other nodes reduces the variance of prediction capability. When there are no source nodes such as node 4, the variance of prediction capability is large. Similarly, the variance of influence is not reduced when nodes do not share information with others, such as node 5. Incorporating both capability and influence, the overall abilities are expected to increase when nodes work in the collaborative mode. Here, the means of abilities are mostly greater than 0.5. The variances also reduce. When the perceptions are incorporated in the second-order abilities, the variances are further reduced. The perceived trustworthiness for nodes 4 and 5 can fluctuate drastically from first order to second order, because of the effect of information sharing and mutual influence. In the second case, nodes work in an ‘independent’ mode. The mean values of prediction probabilities are 0.9, whereas the variances are 0.1. The means of Pand Q-reliance probabilities are 0.5, and their variances are 0.3. The nodes rely much more on individuals’ predictions in comparison with the first case. The ability

96

Y. Wang

Fig. 6.4 The perceived abilities for the 11 nodes in Fig. 6.1 measured with learning capability, influence, overall ability, and the second-order ability. a Collaborative mode; b independent mode

perceptions are seen in Fig. 6.4b where the variance reduction of the overall ability from capability and influence is not as significant as in the previous collaborative scenario. The changes of mean values by incorporating more information are not as significant as in the first case.

6.3.2 Benevolence Benevolence in CPSS networks mainly measures how much information is exchanged between nodes for mutual benefits. The willingness to share information with others is deemed to be more trustworthy. Reciprocity is used to measure such willingness. In addition, motive is used to measure the motivation of such information sharing. Sharing ‘good’ or ‘true’ information is regarded as a trustworthy conduct, whereas sharing ‘bad’ or ‘fake’ one is not. In this section, two versions of reciprocity are discussed. The first one is a deterministic definition where only the connectivity or topology of networks is considered. The second one is the perceived reciprocity where the reliance probabilities are also considered.

6.3.2.1

Deterministic Reciprocity

The deterministic reciprocity of node j with respect to node i, denoted by ri, j , is defined by the number of hops in the network that node j shares information with node i in comparison with how node i shares information with node j reciprocally. It is defined as

6 Quantifying Trust Perception to Enable Design for Connectivity …

      ri, j = ex p −h j→i − ex p −h i→ j + ex p −h j→i − h i→ j

97

(6.21)

where h j→i is the minimum number of hops or the shortest topological distance when information flows from node j to node i. Notice that ri,i = 1 because h i→i = 0. When i = j, ∂ri, j /∂h j→i < 0 and ∂ri, j /∂h i→ j > 0. When the travel distance for inbound information from node j to node i increases, the reciprocity of node j reduces. When the travel distance for outbound information increases, then reciprocity of node j increases. Reciprocity captures this mutuality. As an illustration, in the example of Fig. 6.1, r0,1 = e−2 −e−1 +e−3 = −0.18276, r0,2 = e−1 − e−2 + e−3 = 0.28233 and r0,5 = e−∞ − e−∞ + e−∞ = 0. Notice that if there is no path from node i to node j, h i→ j = ∞. All pair-wise reciprocities of the 11 nodes are listed in Table 6.1. Based on the levels of reciprocity, the trustworthy levels in terms of reciprocity can be ranked and clustered. In the simple example of Fig. 6.1, with respect to a reference node, all other nodes can be categorized into the most trustworthy group with positive reciprocities, the neutral group with zero reciprocities, and the least trustworthy one with negative reciprocities. Figure 6.5 shows the trustworthy networks for nodes 0, 2, 3, 4 and 5 respectively, where the heavily shaded nodes are the most trustworthy groups, and the lightly shaded ones are neutral.

6.3.2.2

Perceived Reciprocity

In the probabilistic context, the expected reciprocity of node j perceived by node i is defined as       E Ri, j = DKL pi→ j || p j→i − DKL p j→i || pi→ j + b0

(6.22)

 where p j→i = i−1 k= j pk,k+1 is the product of all P-reliance probabilities pk,k+1 corresponding to the probability that information follows through the shortest path  P log(P from node j to node i, DKL (P||Q) = i /Q i ) is the Kullback–Leibler i i   divergence from probability Q to P, and b0 is a reference value such that E Ri, j > b0 when node j has a larger reciprocity with respect to nodej. Here,  b0 = 0.5 such that reciprocity has a value between 0 and 1. Furthermore, E Ri,i = b0 . The variance of reciprocity can be calculated from the variances of P-reliance probabilities. If the perceptions of P-reliance probabilities are assumed to be independent, the variance will be associated with the high-dimensional Gaussian distribution formed by these perceptions. If there are n hops in the path associated p j→i , the variance will be n-dimensional. High-dimensional Gaussian distributions, however, are expensive to compute and use. A simplification is made, and only one-dimensional Gaussian distribution is used here. A conservative estimation of the variance for the reciprocity perception is

1

0.000

0.000

0.000

0.135

0.050

0.018

5

6

7

8

9

10

0.002

0.007

0.018

0.000

0.000

0.007

0.018

0.050

0.000

0.000

0.000

−0.368

−0.050

−0.135

4

0.000

0.092

−0.079

0.135

3

0.050

0.135

0.368

0.000

0.000

0.000

−0.050

1.000

−0.079

0.092

1.000

0.282

−0.183

1.000

0.282

−0.183

0.135

3

0.282

2

2

−0.183

1

1.000

0

0

ri, j

0.135

0.018

0.050

0.368

0.368

0.368

1.000

0.050

0.368

0.050

0.135

4

0.000

0.000

0.000

0.000

0.000

0.000

0.000

0.000

0.000

1.000

−0.368

5

Table 6.1 Deterministic reciprocities for all node pairs of the network in Fig. 6.1

0.000

0.000

0.000

0.000

0.135

0.018

0.050

0.368

1.000

0.000

−0.368

6

0.000

0.000

0.000

0.000

8

9

10

−0.368 0.282

−0.050 −0.183 0.368

0.050

1.000 0.282

0.282 −0.183

1.000

−0.135 0.135

1.000

0.000

1.000

−0.183

−0.135

−0.018

−0.050

−0.135

−0.050

−0.007

−0.002

−0.018

−0.368

0.000

−0.018

−0.135

−0.018

−0.007

−0.050

0.000

−0.050

−0.368

−0.050

−0.018

−0.135

0.000

−0.368

7

98 Y. Wang

6 Quantifying Trust Perception to Enable Design for Connectivity …

0

4

2

3

5

1

5

1

0

6

7

7

6

8 10

8

10

(b)

(a)

4

2

5

1

5

1 3

4

2

3

9 8

0

99

0

6

7

4

2

3

7

6

9

9 8

10

8

10

(d)

(c ) 5

1 0

4

2

3

6

7

9 8

10

(e ) Fig. 6.5 Reciprocities in the network of Fig. 6.1 indicate the trust levels based on deterministic pair-wise reciprocities; a trustable clusters with respect to Node 0 based on pair-wise reciprocities (heavily shaded orange nodes are the most trustworthy ones, lightly shaded yellow nodes are the neutral ones and the blank ones without color are the least trustworthy nodes); b trustable clusters with respect to Node 2; c trustable clusters with respect to Node 3; d trustable clusters with respect to Node 4; e trustable clusters with respect to Node 5

⎛ ⎞     −1 −1 ⎠ V Ri, j = min⎝Vmax , τab + τcd j→i

(6.23)

i→ j

where τab ’s are the precisions of P-reliance probabilities along path j → i, τcd ’s are the ones along path i → j, and Vmax is a theoretical maximum of variance which is to ensure the validity of the value. Given that the range of probability is from 0 to 1, an upper bound of variance is about 0.5. Thus the theoretical limit can be Vmax = 1.0. When a path  not exist, the associated variance can be set as Vmax . Furthermore,  does we set V Ri,i = 0 since no uncertainty is involved in one’s own perception. The calculated reciprocity perceptions for all pairs of nodes in Fig. 6.1 are listed in Table 6.2, where all P-reliance probabilities have the value of 0.9, as in the collaborative modepreviously discussed. The nodes canbe grouped into the most trustable   group with E Ri, j > b0 , the neutral group with E Ri, j = b0 , and the least trustable one with E Ri, j < b0 . The trustable groups for nodes 0, 2, 3, 4 and 5 are illustrated in Fig. 6.6. It is seen that the results are consistent with the case of deterministic

0.5000

0.5086

0.4914

0.5000

0.4486

0.5000

0.5000

0.5000

0.5514

0.5127

0.5025

0

1

2

3

4

5

6

7

8

9

10

0

0.5000

0.5003

0.5025

0.5000

0.5000

0.5000

0.4873

0.4978

0.5086

0.5000

0.4914

1

0.5003

0.5025

0.5127

0.5000

0.5000

0.5000

0.2940

0.5022

0.5000

0.4914

0.5086

2

0.5127

0.5514

0.7060

0.5000

0.5000

0.5000

0.4873

0.5000

0.4978

0.5022

0.5000

3

0.5514

0.5025

0.5127

0.7060

0.7060

0.7060

0.5000

0.5127

0.7060

0.5127

0.5514

4

0.5000

0.5000

0.5000

0.5000

0.5000

0.5000

0.2940

0.5000

0.5000

0.5000

0.5000

5

0.5514

0.5025

0.5127

0.7060

0.5000

0.5000

0.2940

0.5000

0.5000

0.5000

0.5000

6

Table 6.2 Reciprocity perceptions for all node pairs of the network in Fig. 6.1 in the collaborative mode 7

0.7060

0.5127

0.5514

0.5000

0.2940

0.5000

0.2940

0.5000

0.5000

0.5000

0.5000

8

0.4914

0.5086

0.5000

0.4486

0.4873

0.5000

0.4873

0.2940

0.4873

0.4975

0.4486

9

0.5086

0.5000

0.4914

0.4873

0.4975

0.5000

0.4975

0.4486

0.4975

0.4997

0.4873

10

0.5000

0.4914

0.5086

0.2940

0.4486

0.5000

0.4486

0.4873

0.4997

0.5000

0.4975

100 Y. Wang

6 Quantifying Trust Perception to Enable Design for Connectivity …

0

4

2

3

5

1

5

1

0

6

7

7

6

8 8

10

10

(b)

(a)

4

2

5

1

5

1 3

4

2

3

9 8

0

101

0

6

7

4

2

3

7

6

9

9 8

8

10

10

(d)

(c) 5

1 0

4

2

3

7

6

9 8

10

(e ) Fig. 6.6 Reciprocity perceptions in the network of Fig. 6.1 indicate the trust levels based on Preliance probabilities; a trustable clusters with respect to Node 0 based on pair-wise reciprocities; b trustable clusters with respect to Node 2; c trustable clusters with respect to Node 3; d trustable clusters with respect to Node 4; e trustable clusters with respect to Node 5

reciprocity. The clusters are the same, except that node 3 is neutral with respect to node 0 in Fig. 6.6a in comparison with Fig. 6.5a, and node 0 is neutral with respect to node 3 in Fig. 6.6c in comparison with Fig. 6.5c. The consistency between the probabilistic and deterministic metrics is a further verification of the definitions.

6.3.2.3

Motive

Motive is further proposed to measure a node’s intention of sharing information. The expected value of the perceived motive for node j is defined as   d E Mj = pj j

(6.24)

where p j is the prediction probability of node j and d j = D j . The motive of a node is high when its prediction is accurate. Yet when the prediction is shared with too many neighbours, its motive of sharing could become questionable and is

102

Y. Wang

reduced. The motive of sharing inaccurate prediction or false information with many neighbours can be malicious and not trustworthy. The variance of motive is related to the precision of prediction is defined as   V M j = τ j−1

(6.25)

This definition is an empirical simplification without the consideration of the number of destination nodes. As the number of destination nodes increases, the motivation becomes clear and the variance is reduced. More precise and complex variances can be defined with these factors incorporated.

6.3.2.4

Overall Benevolence

With both probabilistic reciprocity and motive considered, we can obtain the overall benevolence of node j perceived by node i as          V−1 Ri, j E Ri, j + V−1 M j E M j      E Bi, j = V−1 Ri, j + V−1 M j

(6.26)

     −1  V Bi, j = V−1 Ri, j + V−1 M j

(6.27)

and

    Notice that V Bi,i = 0 and E Bi,i = b0 . The perception of own benevolence has no uncertainty. Therefore the variance of reciprocity is zero, and the  expectation of benevolence is dominated by the expectation of reciprocity E Ri,i .

6.3.3 Integrity Integrity is the traditional focus in the study of cyber security. The operations of CPS in networks need to be protected from malicious attacks. The risks of attacks such as deception attack and replay attack need to be considered in quantifying integrity. In deception attacks, adversary or compromised nodes send false or fake information, e.g. incorrect measurement, incorrect time of measurement, incorrect metadata such as who measured the data and what the purpose of data is, to other nodes to mislead their perceptions. In replay attacks, data transmitted between nodes are intercepted or delayed for some period of time so that the decisions of other nodes can be maliciously manipulated. The perception of integrity about a node can be obtained from historical performance and behavioural data. For instance, the statistics of how often the node share false and inconsistent information with others (i.e. sending ‘True’ to some nodes

6 Quantifying Trust Perception to Enable Design for Connectivity …

103

and ‘False’ to others simultaneously), how often the shared information at different time periods is inconsistent (i.e. sending both ‘True’ and ‘False’ to the same node at different periods of time), and how other parties or nodes rate the node, which is the second-order perception about the integrity. As an illustration, the perceived integrity of a node can be influenced by the consistency of sensing and communication. Suppose that the prior perception of integrity for node j is   E Ij = gj

(6.28)

  V I j = ω−1 j

(6.29)

with variance

The likelihood that node j maintains its integrity can be inferred by estimating the deviation of its prediction from the ones from its neighbours in the neighbourhood, which follows a Gaussian distribution, as ⎡   2 ⎤ 

xi − x j D i∈S j j ⎦   P x j |xS j  D j = g sj ∝ exp⎣− (6.30) s j + d j σs2 where s j = S j and d j = D j are number of source and destination nodes in the neighbourhood of node j, and σ 2s captures the random variation among the sensing units. Furthermore, the consistency of shared information from node j to others is another important measure of integrity. If the state value received by node i from node j is denoted by yij , the probability of genuine communication is 

P yD j |x j



⎡   2 ⎤ y − x i j j i∈D j ⎦ = g cj ∝ exp⎣− d j σc2

(6.31)

where σ 2c captures the natural random error associated with communication channels. The posterior perception of integrity about node j can be obtained as

g j ω j + g s σ −2 + g c σ −2 j s j c E I j |xS j  D j , yD j = ω j + σs−2 + σc−2

(6.32)

104

Y. Wang

6.3.4 Attacks on Trust Similar to the perception of risk, where bias in either public or private information channel can have major impacts to the formation of perception [32], trust perception can be easily manipulated by feeding biased information. Therefore, trust is susceptible to attacks. In a perception attack, attackers can broadcast their manipulated negative or positive perception and seek the ripple effect. For instance, in order to attack the ability of a node, its malicious source nodes can send false information to the node, and its malicious destination nodes can also damage its reputation and influence by exhibiting false prediction. Higher order perceptions of ability can also be influenced by malicious nodes outside the intermediate neighbourhood. To whitewash and quickly improve the perception of benevolence, malicious nodes can modify the existing network connectivity and send positive information frequently to a node in order to earn the trust. In the traditional reputation-based trust management system, the trust levels of individual nodes are maintained in a centralized fashion. This centralized system is susceptible to various attacks, such as self-promoting attack (providing good recommendations to selves), ballot-stuffing attack (providing good recommendations to bad nodes), bad-mouthing attack (providing bad recommendations to good ones), and whitewashing attack (disappear and rejoin the community with new identity). These recommendation-based or reputation-based measurement approaches are less reliable than the proposed perception-based approach. Although the perception-based trust management system is still susceptible to attacks, it is less vulnerable to sudden attacks than a reputation-based system. Because perception or belief update based on Bayes’ rule has been known as a gradual process, particularly with the involvement of variance or imprecision associated with perception. Whenever new evidence arrives, the change of perception is affected by the precision associated with new evidence. Imprecise evidence brings little change to the perception. Even with precise ones, it takes iterations of updates to sway the perception of ability, benevolence or integrity. Simulation studies [35] have shown that the trust perception exhibits robust dynamics while being under attacks. The perception measures are sensitive enough to detect the attacks and at the same time are steady during recovery.

6.4 Trust-Based Network Design The design of trustable network with respect to a node is to choose the optimum combination of nodes to form a society with the maximum trustworthiness. Since the trustworthiness is a multifacet concept, choosing different metrics of ability, benevolence, and integrity as the objective or utility function can lead to different

6 Quantifying Trust Perception to Enable Design for Connectivity …

105

trustworthy networks. In this section, design and optimization approaches with different criteria are demonstrated. In the first criterion, the utility function is the node’s average reciprocity with respect to the nodes in its network. The optimum design is found by maximizing the objective function. In the second criterion, the average benevolence is used as the design objective. In the third one, the objective function is the node’s ability. The trustworthy network with respect to a node is the one that the node can rely on for collaboration and information sharing, therefore is also called strategic network.

6.4.1 Average Reciprocity as the Design Objective Individual nodes’ reciprocities typically have conflict of interest. It is individual nodes’ interest to receive information as much as possible from others. At the same time, the willingness to share with others can be dampened without reciprocal treatment from others. In the optimum network of the reference node, nodes are selected to maximize the utility value. When the deterministic reciprocity is applied, the average reciprocity of node j within the society i formed by node i is calculated as r¯ j =



r j,k /|Ωi |

(6.33)

k∈Ωi

The utility function for society i then can be defined as U (i) =



w j r¯ j

(6.34)

j∈Ωi

 where j∈Ωi w j = 1. The utility is the weighted average of all average reciprocities for the nodes in the society. For a ‘selfish’ approach, only the average reciprocity of the reference node is considered. That is, wi = 1 and w j = 0(∀ j = i). For an ‘altruistic’ approach, only the reciprocities of nodes other than the reference node are considered, i.e. wi = 0  and j=i w j = 1. Between the above two extreme situations, the weighted average among all nodes is taken. To search all combinations to find the global optimum is computationally expensive. In this work, a breadth-first search (BFS) greedy algorithm [34, 36] is applied instead to find the optimum. The algorithm works as follows. Starting from the reference node, all destination nodes in its neighbourhood are taken as candidates to be potentially added into the network. If the utility value does not decrease by adding one candidate, the candidate node is then included in the network. Once the candidate node is included, its neighbouring destination nodes also become candidates for further exploration. The search continues until there are no candidates.

106

Y. Wang

The search results of the BFS algorithm for the strategic networks for all 11 nodes in the simple example of Fig. 6.1 are listed in Table 6.3. When the utility function is based on the perceived reciprocities, P-reliance probabilities need to be considered. As an illustration, all P-reliance probabilities are assumed to be 0.5 as in the independent mode. The strategic networks for the 11 nodes based on the BFS algorithm are listed in Table 6.4. The results are different from the deterministic ones in Table 6.3. Table 6.3 Strategic networks for all 11 nodes in the example of Fig. 6.1, where different weights for deterministic average reciprocities are chosen Ref. node

Strategic network (wi = 1)

Strategic network (wi = 0.5)

Strategic network (wi = 0)

0

0, 2

0, 1, 2, 3

0, 1

1

0, 1

0, 1, 2, 3

1, 2

2

1, 2, 4

0, 1, 2, 4

0, 2

3

0, 2, 3

0, 3

0, 3

4

4

2, 4

2, 4

5

4, 5

2, 4, 5

5

6

4, 6

0, 1, 2, 3, 4, 5, 6, 7

6, 7

7

4, 6, 7

4, 6, 7, 10

7, 10

8

3, 8, 10

0, 3, 8, 9, 10

8, 9

9

8, 9

8, 9, 10

9, 10

10

7, 9, 10

7, 8, 9, 10

8, 10

Table 6.4 Strategic networks for all 11 nodes in the example of Fig. 6.1, where different weights for perceived average reciprocities are chosen and all P-reliance probabilities are 0.5 in the independent mode Ref. node

Strategic network (wi = 1)

Strategic network (wi = 0.5)

Strategic Network (wi = 0)

0

0, 1, 2, 3, 4

0, 1, 2, 3, 4

0, 1, 2

1

0, 1, 2, 4

0, 1, 2, 4

0, 1, 2, 3, 8, 9, 10

2

0, 1, 2, 3, 4

0, 1, 2, 3, 4

0, 1, 2, 4, 5, 6

3

0, 2, 3, 4, 8

0, 2, 3, 4, 8

0, 1, 3, 8, 9, 10

4

1, 2, 4, 5, 6, 7

1, 2, 4, 5, 6, 7

0, 1, 2, 3, 4, 5, 6, 7, 8, 9

5

0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

6

0, 1, 2, 3, 4, 5, 6, 7, 8, 9

4, 5, 6, 7

2, 4, 5, 6, 7, 8, 9, 10

7

0, 1, 2, 3, 4, 5, 6, 7, 9, 10

0, 1, 2, 3, 4, 5, 6, 7, 9, 10

4, 7, 8, 9, 10

8

0, 1, 2, 3, 4, 6, 7, 8, 9, 10

0, 1, 2, 3, 4, 6, 7, 8, 9, 10

3, 8, 9, 10

9

0, 1, 2, 3, 4, 6, 7, 8, 9, 10

0, 1, 2, 3, 4, 6, 7, 8, 9, 10

8, 9, 10

10

0, 1, 2, 3, 7, 8, 9, 10

0, 1, 2, 3, 4, 8, 9, 10

8, 9, 10

6 Quantifying Trust Perception to Enable Design for Connectivity …

107

6.4.2 Average Benevolence as the Design Objective The average benevolence which includes both reciprocity and motive can also be used as the utility for design optimization. In the previous 11-node simple network example in the independent mode with all P-reliance probabilities as 0.5, the optimum networks can be obtained similarly with the average benevolence, as shown in Table 6.5. It is seen that the results are slightly different from the ones in Table 6.4 with only reciprocity considered. A second example is a randomly generated probabilistic graph, where the nodes are randomly connected. The prediction and reliance probabilities are also randomly generated. The number of nodes is 100. The probability that an edge exists between two nodes is set to be 0.05. The randomly generated graph is shown in Fig. 6.7, where the number of directed edges is 495. When wi = 1 and the average benevolence is applied as the utility, the optimum network for node 0 and the change of utility value along iterations are shown in Fig. 6.8. The optimal network includes 17 nodes. The utility value monotonically increases during the search with the greedy algorithm. When wi = 0 and the average benevolence is applied, the optimum network for node 0 and the evolution of utility value during iterations are shown in Fig. 6.9. The strategic network also includes a set of 17 nodes, but the nodes are different from the previous case when wi = 1. In general, the numbers of nodes in the two cases will be different with the randomly generated prediction and reliance probabilities. Table 6.5 Strategic networks for all 11 nodes in the example of Fig. 6.1, where different weights for perceived average benevolence are chosen and all P-reliance probabilities are 0.5 in the independent mode Ref. node

Strategic network (wi = 1)

Strategic network (wi = 0.5)

Strategic network (wi = 0)

0

0, 1, 2

0, 1, 2

0, 1, 2

1

0, 1, 2, 4, 5, 6

0, 1, 2

0, 1, 2

2

0, 1, 2

0, 1, 2

0, 1, 2

3

0, 1, 2, 3, 4, 5, 6

0, 3

0, 3

4

0, 2, 4, 5, 6

1, 2, 4, 5, 6

2, 4, 5, 6

5

0, 2, 4, 5, 6

0, 2, 4, 5, 6

1, 2, 4, 5, 6

6

0, 2, 4, 5, 6, 7

0, 2, 4, 5, 6

1, 2, 4, 5, 6

7

3, 7, 8, 10

1, 2, 3, 4, 5, 7, 8, 9, 10

0, 2, 4, 5, 7, 8, 9, 10

8

8, 9, 10

8, 9, 10

8, 9, 10

9

0, 2, 4, 5, 7, 8, 9, 10

8, 9, 10

8, 9, 10

10

0, 1, 2, 3, 4, 8, 9, 10

8, 9, 10

8, 9, 10

108

Y. Wang

Fig. 6.7 A randomly generated probabilistic graph with 100 nodes and 495 edges

Fig. 6.8 The strategic network for Node 0 when average benevolence is used as utility and wi = 1

Fig. 6.9 The strategic network for Node 0 when average benevolence is used as utility and wi = 0

6 Quantifying Trust Perception to Enable Design for Connectivity …

109

Fig. 6.10 The strategic network for Node 0 with 59 nodes from the random network in Fig. 6.7 when ability is used as the utility

6.4.3 Ability as the Design Objective Ability is another criterion to be used to find the trustable strategic network. Ability incorporates both prediction capability and societal influence of nodes. Therefore, a network that maximizes the capability and influence of the reference node can be regarded as the most trustable network that the node can rely on. The utility for the network design problem can be based on the first- or secondorder ability. The trustable strategic networks for Node 0 in the random network with 100 nodes in Fig. 6.7 can be obtained to maximize the ability of Node 0. Based on the BFS algorithm, the optimal networks can be found. When the first-order ability is used as the design objective, the optimal network and the utility values during the iterations are shown in Fig. 6.10. The trustworthy network for Node 0 includes 58 other nodes. When the second-order ability is applied as the utility, the optimum network with respect to Node 0 is shown in Fig. 6.11, where a total of 46 nodes are included.

6.5 Summary Designing cyber-physical-social systems is becoming the major task for design engineers. Given that the highly interconnected products and devices rely on information exchange for functioning, the consideration of trust becomes important, in addition to security and privacy issues. In this chapter, quantitative metrics of trustworthiness are described to quantify the trust relationship for engineering design purposes. The new multidimensional metrics of ability, benevolence, and integrity are used to measure trustworthiness. Based on a generic mesoscale probabilistic graph model, the quantities of ability, benevolence, and integrity can be defined and calculated. The quantitative measures are essential to include trust in networked system design. A

110

Y. Wang

Fig. 6.11 The strategic network for Node 0 with 46 nodes from the random network in Fig. 6.7 when second-order ability is used as the utility

trustworthy network formed by CPSS nodes, which share sufficient information and use such information effectively in decision-making, can result in the assurance of expected system behaviour. On the other hand, a network which lacks mutual trust between nodes can bring the detrimental effect on the functions of the system. The proposed quantitative metrics can be either subjective perceptions or detailed measurements such as topological information of networks. The study shows that the perception metrics can become more precise with additional information included. If more information about the behaviours of nodes is available, the metrics should be extended to incorporate such information for more precise evaluation for certain applications. The current probabilistic graph model is simple but generic enough to capture networks with information flow. Although the probabilistic graph model was proposed for CPSS networks, it can be generalized to other types of networks. Information, energy, and material flows can be modelled similarly. For instance, in supply chains or transportation networks, a prediction probability can correspond to the probability that goods or supplies satisfy the demand at a node, the distribution of demand or the distribution of inventory levels at a node, whereas reliance probabilities characterize the correlations between demands at different nodes (percentage of supply from one node goes to another), percentage of transport capacities being employed, or probability that transportation is not interrupted. The feasibility of the proposed metrics for trustable network design is demonstrated with individual metrics as design objectives. Further extension of how they can be applied for multi-objective design optimization is needed. In addition, in large-scale and complex networks, a global view of network topology is impossible to obtain. The direct knowledge of networks used in design and optimization mostly is from local neighbourhoods, whereas the one beyond local neighbourhoods has to be inferred from others indirectly. This is where transferrable metrics can be brought

6 Quantifying Trust Perception to Enable Design for Connectivity …

111

in. Quantifiable trustworthiness measure from neighbours’ neighbours can be propagated into the local domain of consideration. This propagation scheme needs to be studied in the future. The limitation of the proposed metrics, particularly ability, is that they rely on the availability of prediction and reliance probabilities, which requires the collection and analysis of historical data. If no such data are available, subjective elicitation may be required from experts. Elimination of inconsistency during subjective elicitation requires meticulous cares. Acknowledgements This work is supported in part by National Science Foundation under grant CMMI-1663227.

References 1. Akter S, D’Ambra J, Ray P (2011) Trustworthiness in mHealth information services: an assessment of a hierarchical model with mediating and moderating effects using partial least squares (PLS). J Assoc Inf Sci Technol (Wiley Online Library) 62(1):100–116 2. Al-Hamadi H, Chen R (2017) Trust-based decision making for health IoT systems. IEEE Internet Things J IEEE 4(5):1408–1419 3. Angelaki DE, Gu Y, DeAngelis GC (2009) Multisensory integration: psychophysics, neurophysiology, and computation. Curr Opin Neurobiol (Elsevier) 19(4):452–458 4. Barber KS, Kim J (2001) Belief revision process based on trust: agents evaluating reputation of information sources. In: Falcone R, Singh M, Tan Y-H (eds) Trust in Cyber-societies. Springer, pp 73–82 5. Beth T, Borcherding M, Klein B (1994) Valuation of trust in open networks. Computer security—ESORICS 94. Springer, pp 1–18 6. Chen D et al (2011) TRM-IoT: a trust management model based on fuzzy reputation for internet of things. Comput Sci Inf Syst 8(4):1207–1228 7. Chen R, Bao F, Guo J (2016) Trust-based service management for social internet of things systems. IEEE Trans Dependable Secure Comput (IEEE) 13(6):684–696 8. Chen Z, Tian L, Lin C (2017) Trust model of wireless sensor networks and its application in data fusion. Sensors (Multidisciplinary Digital Publishing Institute) 17(4):703 9. Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature (Nature Publishing Group) 415(6870):429 10. Fuller MA et al (2010) Clarifying the integration of trust and TAM in e-commerce environments: implications for systems design and management. IEEE Trans Eng Manag 57(3):380–393 (IEEE) 11. Golbeck J, others (2008) Trust on the world wide web: a survey. Foundations and trends® in web science, vol 1, no 2. Now Publishers, Inc., pp 131–197 12. Govindan K, Mohapatra P (2012) Trust computations and trust dynamics in mobile adhoc networks: a survey. IEEE Commun Surv Tutor 14(2):279–298 (IEEE) 13. Grabner-Kräuter S, Kaluscha EA (2003) Empirical research in on-line trust: a review and critical assessment. Int J Hum-Comput Stud 58(6):783–812 (Elsevier) 14. Huynh TD, Jennings NR, Shadbolt NR (2006) An integrated trust and reputation model for open multi-agent systems. Autonomous agents and multi-agent systems 13(2):119–154 (Springer) 15. Keung SNLC, Griffiths N (2010) Trust and reputation. In: Agent-based service-oriented computing. Springer, London, pp 189–224. https://doi.org/10.1007/978-1-84996-041-0_8 16. Kim H et al (2010) A trust evaluation model for QoS guarantee in cloud systems. Int J Grid Distrib Comput 3(1):1–10

112

Y. Wang

17. Lee MKO, Turban E (2001) A trust model for consumer internet shopping. Int J Electron Commer (Taylor & Francis) 6(1):75–91 18. Lee S, Sherwood R, Bhattacharjee B (2003) Cooperative peer groups in NICE. In: INFOCOM 2003. twenty-second annual joint conference of the IEEE computer and communications. IEEE Societies, pp 1272–1282 19. Li X et al (2015) ‘Service operator-aware trust scheme for resource matchmaking across multiple clouds. IEEE Trans Parallel Distrib Syst 26(5):1419–1429 IEEE 20. Li X, Zhou F, Du J (2013) ‘LDTS: A lightweight and dependable trust system for clustered wireless sensor networks. IEEE Trans Inf Forensics Secur 8(6):924–935 IEEE 21. Ma X, Lu H, Gan Z (2014) Improving recommendation accuracy by combining trust communities and collaborative filtering. In: Proceedings of the 23rd ACM international conference on conference on information and knowledge management, pp 1951–1954 22. Mayer RC, Davis JH, Schoorman FD (1995) An integrative model of organizational trust. Acad Manag Rev 20(3):709–734 Academy of Management 23. Nitti M, Girau R, Atzori L (2014) Trustworthiness management in the social internet of things. IEEE Trans Knowl Data Eng 26(5):1253–1266 IEEE 24. O’Doherty D, Jouili S, Van Roy P (2012) Towards trust inference from bipartite social networks. In: Proceedings of the 2nd ACM SIGMOD workshop on databases and social networks, pp 13–18 25. Patel J et al (2005) A probabilistic trust model for handling inaccurate reputation sources. In: Herrmann P, Issarny V, Shiu S (eds) Trust management. Springer, Berlin, Heidelberg, pp 193–209 26. Reddy VB, Venkataraman S, Negi A (2017) Communication and data trust for wireless sensor networks using D-S theory. IEEE Sens J 17(12):3921–3929 IEEE 27. Ruan Y, Durresi A (2016) ‘A survey of trust management systems for online social communities—Trust modeling, trust inference and attacks. Knowl-Based Syst 106:150–163 Elsevier 28. Sabater J, Sierra C (2002) Reputation and social network analysis in multi-agent systems. In: Proceedings of the first international joint conference on autonomous agents and multiagent systems: part 1, pp 475–482 29. Sahoo RR et al (2016) A bio inspired and trust based approach for clustering in WSN. Nat Comput 15(3):423–434 Springer 30. Saied YB et al (2013) Trust management system design for the internet of things: a contextaware and multi-service approach. Comput Secur 39:351–365 Elsevier 31. Wang Y (2016) System resilience quantification for probabilistic design of internet-of-things architecture. In: ASME 2016 international design engineering technical conferences and computers and information in engineering conference, p V01BT02A011 32. Wang Y (2017) On social value of risk information in risk communication. ASCE-ASME J Risk Uncertain Eng Syst Part B: Mech Eng (American Society of Mechanical Engineers) 3(4):41009 33. Wang Y (2018a) Resilience Quantification for probabilistic design of cyber-physical system networks. ASCE-ASME J Risk Uncertain Eng Syst Part B (ASME) 4(3):31006 34. Wang Y (2018b) Trust based cyber-physical systems network design. In: ASME 2018 international design engineering technical conferences and computers and information in engineering conference, p V01AT02A037–V01AT02A037 35. Wang Y (2018) Trust quantification for networked cyber-physical systems. IEEE Internet Things J 5(3):2055–2070 IEEE 36. Wang Y (2018d) Trustworthiness in designing cyber-physical systems. In: Proceedings of 12th international symposium on tools and methods of competitive engineering (TMCE2018). Las Palmas, Gran Canaria 37. Yousafzai SY et al (2005) Strategies for building and communicating trust in electronic banking: a field experiment. Psychol Mark 22(2):181–201 Wiley Online Library 38. Yu B, Singh M (2000) A social mechanism of reputation management in electronic communities. In: Klusch M, Kerschberg L (eds) Cooperative information agents IV-the future of information agents in cyberspace. Springer, Berlin, Heidelberg, pp 154–165

6 Quantifying Trust Perception to Enable Design for Connectivity …

113

39. Yu B, Singh MP (2002) Distributed reputation management for electronic commerce. Comput Intell 18(4):535–549 Wiley Online Library 40. Zhou P et al (2015) Toward energy-efficient trust system through watchdog optimization for WSNs. IEEE Trans Inf Forensics Secur (IEEE) 10(3):613–625

Chapter 7

A Study of “Waku-Waku” at Work Ryotaro Inoue and Takashi Maeno

Abstract Taking the upcoming “Society 5.0” and “100 years of lifetime” into account, what should each and every one of us, as a worker, do to confront our work positively and realize a fulfilling career? Moreover, in economic circumstances that are insecure and uncertain for businesses, what should we do to develop human resources capable of creating new values and enhancing labor productivity? In this study, in order to meet personal and organizational demands, the author defines the fundamental problem as “What should we do to get each employee to work with more initiative?” If you are in a comfortable, active state in which you find your work rewarding, and you work with initiative, then you have high work engagement. This study focuses on the mediation effect of work engagement and positive emotion. We also give heed to “Waku-Waku,” an onomatopoeia expressing positive emotion that is unique to the Japanese language. Originally, onomatopoeia has rarely been used in academic papers as it does not have a solid conceptual definition. It, however, is a linguistic expression that is excellent in conveying our intention with complexity and precision. Although “Waku-Waku” is often translated into “excitement” in English, it has been confirmed in a prior study that its nuance is much closer to concepts such as “well-being” and “flow”. These concepts are considered from the standpoint of new intervention to make work more satisfying and fulfilling in light of human resource management. This study focuses on the positive emotion of “Waku-Waku,” and builds up a statistic structure of the main factor that can evoke this motion in working scenes, through qualitative and quantitative analysis. Among others, reported here is a study of analysis on factors of feeling “Waku-Waku” at work that many working individuals in Japan are aware of. As a result, four factors were confirmed: 1. Challenge to creativity and the unknown; 2. Fortune and pleasure; 3. Interest in others at work and; 4. Passion in terms of sensitivity. Since it was shown that working individuals feel “Waku-Waku” when they face these things, the concept of “Waku-Waku” was tentatively defined in conclusion.

R. Inoue (B) · T. Maeno Keio University, SDM, 4-1-1, Hiyoshi, Kohoku-ku, Yokohama 223-8526, Japan e-mail: [email protected] © Springer Nature Switzerland AG 2020 S. Fukuda (ed.), Emotional Engineering, Vol. 8, https://doi.org/10.1007/978-3-030-38360-2_7

115

116

R. Inoue and T. Maeno

7.1 Background of the Study Recently, Japanese businesses have been confronting major environmental changes. IT (Information Technology) has made remarkable progress and borderless international competition has expanded in a way that various factors intricately influence each other. This situation has become known as the age of VUCA (an unforeseeable, chaotic situation) and uncertainty has been increasing. In addition, Japan is one of the nations with the longest lifespans. As its society has aged, the birth rate has remained low at about 1.4%. A decline in total population is predicted, suggesting that a significant fall in labor population is also expected. Under these unprecedented circumstances, a domestic industrial structure is obliged to make drastic changes. The organization management system, that is based on the practice of conventional Japanese employment, also needs a structural shake-up. Japan’s low labor productivity stands out compared to other developed countries [1], and “work-style reform” is being led by the government, focusing on “the correction of long work hours.” As such, measures to reduce overtime have been similarly implemented in many Japanese companies, such as: introduction of no-overtime day (37.2%); setting an upper limit of overtime (29.2%); and prior application of overtime (24.8%). On the other hand, some reports indicate that measures to rev up production by boosting employee work motivation have not been noticeably implemented [1]. Other reports point out that pushing forward similar measures to restrain overtime without changing work or office environment increases overtime work at home and management stress. Measures to improve productivity led by the government may cause a decrease in employee motivation by conversely keeping them from concentrating on work they have a desire for. It makes no sense if this causes lowered productivity. Companies need to ensure they understand their management systems and find the leverage point in which time delay is taken into consideration, so as not to let measures intended to increase productivity conversely decrease it. At least in terms of the proposition to improve productivity, it seems important to take a look at measures to boost and keep employees’ motivation as well as those to restrain overtime. Upon doing so, it seems difficult to expect further improvement in work productivity without considering the essence of what working individuals in today’s Japan want from work. So, what do working individuals want from work now? According to a survey conducted by Sanno University, where the author used to work, it was “reward in work (38.8%)” [2]; employees of Japanese companies most wanted their work to have reward. Reward here means to get the due results for their work. Therefore, “reward in work” can be defined as a state of mind where you are able to get your due effects, compensation, and satisfaction for your hard-work. Additionally, what the prior survey suggests is that they expected not only money but also certain kinds of psychological remuneration. In fact, income and other monetary demands cannot be ignored. Nevertheless, the fact that “reward in work” showed the highest percentage in an anonymous Internet survey seems to be a noteworthy result from the viewpoint of a work-motivation study in Japan today.

7 A Study of “Waku-Waku” at Work

117

However, “reward in work” is subjective and depends on the diverse outlook on labor. It is easy to assume that there is an influence of various factors including their occupations, titles, cultural and historical backgrounds, and family situations. This is regarding “meaning,” “mastery,” “motivation,” and “flow” of life and work, which are themes that have been dealt with in psychology, business administration, philosophy, and other domains of study. Thus, based on the aforementioned viewpoints, this study defines a state of mind where people work as they feel rewarded as “a positive activated state in which they volunteer to do their current job, not because they are forced to or have to.” Taking the upcoming “Society 5.0” and “100 years of lifetime” into account, what should each and every one of us, as a worker, do to confront our work positively and realize a fulfilling career? Moreover, in economic circumstances that are insecure and uncertain for businesses, what should we do to develop human resources capable of creating new values and enhancing labor productivity? In this study, in order to meet personal and organizational demands, the author defines the fundamental problem as “What should we do to get each employee to work with more initiative?”

7.2 Prior Study An individual’s state of activity that is elevated not voluntarily but because something forces them to work or is made to feel they have to work, is called workaholism or being a workaholic in the study domains of clinical psychology and mental health. On the contrary, high activity level when they work comfortably on their own initiative and feel rewarded is called a state of high work engagement [3]. It is exactly the concept representing an activated state when an individual volunteers to work, not because they are forced to, as mentioned above. According to its proponent, Schaufeli, work engagement is defined as “a positive state related to work, characterized by enthusiasm, flow and vitality, not a temporary state but general emotion and acknowledgment.” Furthermore, it is also considered that “a high work engagement state is preparation for boosting work motivation.” [4]. Interest in this concept has been increasing recently in the field of business administration and human resource management that require not only the needs of health management but also motivation, productivity, and innovation. How can work engagement be heightened and maintained then? In a prior study, it was confirmed that the reports show that the more the psychological capital (PsyCap) of an individual increases, the more work engagement heightens. PsyCap is a concept composed of four factors: a sense of efficacy, optimism, hope, and resilience [5]. Some studies suggested that work engagement is further promoted if PsyCap comes with a positive feeling (emotion) [6]. This study focuses on the mediation effects between work engagement and positive emotion, which will become an important concept in future corporate management and human resource management (HRM). Next, let us confirm what positive emotions and feelings are focused on in this study. Study on positive emotion and feeling is one of the important study domains

118

R. Inoue and T. Maeno

in positive psychology. Since 2000, when positive psychology was proposed by Seligman, there has been remarkable progress in studies on positive emotion (feeling) and study results have especially accumulated in Europe. However, many of these studies have only been conducted in Europe and North America, and Japan is still in the process of development. In addition, events at work that triggers positive emotions are considered to be affected by many elements, such as memories and beliefs (outlook on labor and humanity) of working individuals, characteristics of organizational human resource management, and cultural characteristics of the region and organization. In this respect, there have been few studies that focus on typically Japanese positive emotions while considering Japanese cultural and management characteristics and personal conviction. As a starting point of the way for working individuals to become positively active at work, this study focuses on relatively short-term positive emotion and state of mind.

7.2.1 Focusing on the Feeling of “Waku-Waku” For people living in Japan, everyone from children to adults understand the word “Waku-Waku”. It is a common instance of onomatopoeia with positive nuance. In Europe and North America, onomatopoeia like “clap” (pachi-pachi: the sound of clapping hands, sound of burst) and “flip-flop” (pata-pata: the sound when the laundry flaps, for example) are commonly used. Kakei et al. use the example of “how to walk” to show the linguistic characteristics of Japanese onomatopoeia and discuss as follows [7]: In Japanese, the neutral verb, “Aruku (walk)”, has several onomatopoeia like “FuraFura”, “Bura-Bura” and “Pura-Pura”. We differentiate ways of walking by changing these onomatopoeia like exchanging parts. Most importantly, the core of expression lies in onomatopoeia…Moreover, in Japanese, the difference of unvoiced consonants, voiced consonants and consonants in onomatopoeia connotes subtle differences in meaning… [7]

As this sentence shows, onomatopoeia can convey the subtle and ambiguous nuances of things to others with a direct image and is also suitable for transmitting things in a complex and inoffensive manner. Thus, it is a very typical Japanese expression. Next, let us confirm the definition of this instance of onomatopoeia, “Waku-Waku,” in Japanese dictionaries and prior studies in Japan. According to Japan’s most prestigious dictionary, Kojien sixth edition, it is explained as “A restless state of leaping heart in expectation, joy or excitement.” (Iwanami Shoten) Another Japanese dictionary published by Gakken says “A restless state of heart in expectation or anxiety. A state of heart beating.” (Gakushu Kenkyusha) These dictionaries commonly have the meaning of “expectation,” “leap,” “excitement,” and “restless.” Some dictionaries uniquely have the meaning of “joy,” “anxiety,” and “heart-beating.” Looking back at the derivation, the word “Wakusekito” can be found in Edo period. This means “excited and the heartbeats like “Doki-Doki” (Edo Word Dictionary (new format popular edition) Tokyodo Publishing). “Doki-Doki”

7 A Study of “Waku-Waku” at Work

119

here indicates heartbeats of anxiety. (Note that Edo Word Dictionary is merely one of the opinions) It is confirmed that from general definition by the dictionary and derivation of the word, “Waku-Waku” shows complicated emotion; it has positive nuance mixed with a negative one. How is “Waku-Waku” defined in academic study? Even CiNii and Google scholar search could not find many usage examples of onomatopoeia in academic papers. This seems to be because onomatopoeia is regarded as a childish expression as it does not have a clear definition. Against this backdrop, several studies that focused on the nuances of the onomatopoeia, “Waku-Waku,” were confirmed; however, even more limited studies clearly showed the definition of “Waku-Waku.” The definitions we confirmed are shown below. In a study of the sensitivity engineering domain that performed physiological measurement of sensitivity when riding a bicycle, we found the definition was “to learn about some new discoveries in your area of interest or to expect and feel fun when things turn into favorable state” [8]. Also, in the domain of linguistics on onomatopoeia, we found studies on the meaning of “Waku-Waku.” Nakazato stated, “In the Meiji and Taisho periods, “Waku-Waku” included fear and anxiety. But the scope of the meaning of the word shrank as time passed and only positive emotion was left.” [9]. As Nakazato said, linguistically, the word is no longer used to express negative emotion today. It was found that “Waku-Waku” is a typical example of a shrunken word in terms of meaning. There were some cases where “Waku-Waku” was translated as “excitement” in English. In other cases, it was translated as “thrilled” or “wowed.” Part of the element is indeed similar, so there is no discrepancy. However, the onomatopoeia that has been widely used in Japan today is considered as a concept closer to “flow” than “excitement,” according to the prior study in Japan. “Flow” is a concept proposed by Csikszentmihalyi, indicating “the best experience,” in which your challenge level and your ability level match. Csikszentmihalyi defines the best experience as “A state of being so into one activity that anything else is not perceived as a problem. A state of sparing a lot of time and energy for the experience because the experience itself is very fun.” [10]. “Flow is a concept indicating a comfortable state you are in when getting the best experience and being into your current activity. It is believed to be an effective approach to improve “quality of life” and pursue “happiness”. As mentioned above, “Waku-Waku” with its unique Japanese nuance could be a positive emotion that would lead to “flow.” Against the backdrop of this, this study focuses on a positive sense, “Waku-Waku,” as one of the viewpoints to break through the social background (feeling of hopelessness) in the Japanese industry.

7.3 Purpose of This Study and Hypothesis As stated in the previous section, we focused on a work engagement study as a concept capable of inducing performance in which working individuals can work vigorously, and have their creativity and productivity enhanced. We also focused

120

R. Inoue and T. Maeno

on “Waku-Waku” as a concept showing positive emotion in Japan. In this study, it is expected that identifying the structure and influential factors of “Waku-Waku” at work will be a foothold of a new intervention method in a future human resource management area. It would be significantly meaningful to seek a new ideal management method focusing on positive emotion, if working individuals and businesses seek the opportunity to nurture lively human sense like enjoyment, joy, and impression through work. The following two points are the novelty of this study. First, it is an attempt to clarify factors of Japanese positive emotion, “Waku-Waku,” at work as a statistically significant structure. “Motivation” and “commitment” at work have been discussed in work-motivation and other study domains, so there are already many prior studies. However, as for positive emotions and intuitive senses that have been hard to visualize are relatively new areas, and they seem to have high potential in a work-motivation study. In addition, there are not enough studies on positive emotions even in classical psychology. Therefore, we could not find any prior studies that defined “WakuWaku,” an instance of onomatopoeia unique to Japanese, nor “A Sense of WakuWaku” at work or a study that demonstrated it as a positive psychological state. Second, it is an attempt to statistically verify a self-sustaining structure: “A sense of Waku-Waku at work” enhances “work engagement,” and those who have high “work engagement” take action with initiative, and then this “action with initiative” induces a new “sense of Waku-Waku.” In prior studies, although this circulation structure was partially mentioned as a direct pass analysis and a theoretical model, there were no studies that applied it for different study purposes and clarified it with multiple population analysis. Here we report excerpts from results of analysis on an overt sense of Waku-Waku, which was performed as the study herein.

7.4 Hypothesis in This Study Based on prior studies, we formulated hypothesis on “a sense of “Waku-Waku” at work as follows: Hypothesis 1. “A sense of “Waku-Waku” at work is a positive emotion. Hypothesis 2. What makes you “Waku-Waku” at work (overt Waku-Waku factors) and events that make you Waku-Waku at daily work (potential Waku-Waku factors) are different. Hypothesis 3. Initiative, a sense of self-efficacy and intellectual curiosity can become prior factors of a sense of Waku-Waku at work. Hypothesis 4. “A sense of Waku-Waku” at work, work engagement and action with initiative form an interactive circulation structure (Fig. 7.1). The circulation model as in hypothesis 4 was established based on the following three theories. 1. “Social learning theory” [11] that showed the fact that whether something inducing some action or not is influenced by “efficacy expectation” and “outcome

7 A Study of “Waku-Waku” at Work

121

Fig. 7.1 Hypothesis 4 “A model to improve the initiative level of Waku-Waku factors at work”

expectation.” 2. The broaden-and-build theory [12] showed the following fact: Positive emotions like “joy,” “interest,” and “pride” momentarily expand the repertoire of thoughts at a certain time, meaning they work as enhancers of an individual’s various resources. As a result, the feedback obtained from these actions continuously form resources of the individual and promotes spiral of changes and growth. 3. “Study on the relationship between PsyCap/positive emotion and behavior/action of employees” [6] that showed the direct model: PsyCap influences employees’ behavior/action via positive emotions. Using these theories in prior studies, hypothesis 4 can be explained and become tenable.

7.5 Study Methods 7.5.1 Structure and Purpose of the Study This study is roughly composed of a qualitative and a quantitative survey. The purpose of the qualitative survey was to collect events that induce a sense of Waku-Waku in people through work, and break them down by category to convert them into codes. The purpose of the quantitative survey was to design survey slips based on qualitative survey’s results, extract statistic factors of Waku-Waku events at work, and create a structure of relationship between the extracted factors and similar concepts.

122

R. Inoue and T. Maeno

7.5.2 Subjects of the Qualitative Survey In order to collect as many events as possible where working individuals feel “WakuWaku” at work, we explained the outline of the survey to the following three organizations and asked for their cooperation with our quantitative survey. We received agreement of cooperation from three organizations. Survey 1 consisted of 10 subjects in total: 5 adult students who belong to the author’s graduate school; 3 adult researchers and; 2 teachers (date of survey: February 9, 2018). Survey 2 was of the general staff of the company R, which consisted of 12 workers engaged in different job areas such as sales, general office work, accounting and general affairs (date of survey: February 2, 2018). Survey 3 was for general staff of the cosmetics manufacturing & sales company C, which consisted of 6 workers engaged in different job areas such as sales, general affairs, product development, and overseas sales (date of survey: July 4, 2018). In survey 4, we were kindly provided data of 950 people related to “Business person’s sense of “Waku-Waku” survey” [13] conducted by a survey agency where the author used to belong, with the approval of the department in charge (survey period: August 3–6, 2012).

7.5.3 Methods of the Qualitative Survey For the qualitative survey, we wrote out the “Waku-Waku events” on sticky notes using one of the methods to come up with ideas, the brainstorming method (survey 1, 2, and 3). We also wrote out free answer data in the prior survey on sticky notes (survey 4). Then we categorized all sticky notes using an affinity diagram and named the categories with two adult doctoral course students in our lab who specialized in psychology. We repeated the process and converted it into codes. Through this encoding process, we qualitatively formulated “the model of factors to get a sense of “Waku-Waku” at work.” In survey 1, we asked subjects “What are the things that make you feel Waku-Waku at work?” In survey 2 and 3, not limiting to work scenes, we asked “What are the things that make you feel Waku-Waku?” In survey 4, we extracted text data from the question (free answer) “What makes you feel Waku-Waku?” which is an additional question to 111 responders who answered “I often/sometimes/rarely feel Waku-Waku at work,” in a similar question in the prior survey.

7.5.4 Subjects of the Quantitative Survey For the quantitative survey, we carried out a questionnaire survey on the Internet, because we could collect a wide range of employment status data of employed people in Japan without bias in region and age, and analyze collected results quantitatively.

7 A Study of “Waku-Waku” at Work

123

We used Macromill Inc., which has about 1.2 million high-quality original monitors in Japan and is evaluated for its academic survey, to choose 1000 employed people all over Japan. In order to exclude as much bias as possible and grasp the employment situation in Japan, we implemented a survey on the following panel extraction conditions: 1. Must be employed in Japan; 2. In order to exclude residential bias, allot the number of respondents in line with demographics of each prefecture; 3. In order to exclude age bias, secure a certain number of samples from each generation from 20s to 60s. Because our subjects were employed people from the 20s to 60s, there would be no problem to regard these people as a bracket of people who use the Internet on a daily basis. The survey period was from October 11 (Thu) to 12 (Fri), 2018.

7.5.5 Creation of Survey Slips As preparation for creating a questionnaire of this survey, we decided the contents of it. Items we adopted are as follows: 1. Items to ask on personal attributes were “sex,” “age,” “prefecture,” “married/unmarried,” “having children or not,” “personal annual income,” “household annual income,” “educational background,” and “experience of having studied abroad;” 2. Items to ask on the personal employment situation were “occupation,” “type of industry,” “scale of employees,” “type of occupation,” “title,” “years of service,” and “the number of job-change;” 3. Items to ask on personal attribute were “Big 5: the Japanese version of the Ten Item Personality Inventory (TIPI-J) [14] and “Value scale” (Schwartz) [15]. The item to measure relatively longterm stable happiness was “SWLS” (Diener) [16]. The item to measure relatively middle-term happiness was “Four Happiness Factors” [17]. To measure self-efficacy, we used “the Scale Measuring a Sense of Generalized Self-Efficacy” [18, 19], the “Scale of Diffusive Intellectual Curiosity,” and the “Scale of Special Intellectual Curiosity” [20]. To measure how much action an individual takes with initiative, we used the “Initiative Scale” [21]; 4. Items to ask values concerning “feeling WakuWaku” were,” I think it is important to feel Waku-Waku at work,” “I think it is important to feel Waku-Waku at home,” “I think it is important to feel Waku-Waku in private life (other than at work and at home).” And items to ask if the subjects presently feel Waku-Waku at work were “I feel Waku-Waku at work;” 5. For scale items to get “overt Waku-Waku factors,” which is a subjective Waku-Waku event at work, we created questions from 41 events obtained from a qualitative survey. In order to check the “Waku-Waku” level of respondents, we asked, “How much do you feel Waku-Waku regarding the following items at work?” and gave scenes like “When are you convinced of realizing what you wanted to do” and “When are you eager to know the results.” Answers were made based on a seven-level Likert scale: 1. I don’t feel Waku-Waku at all; 2. I don’t feel Waku-Waku; 3. I would rather not feel Waku-Waku; 4. I can’t say; 5. I would rather feel Waku-Waku; 6. I feel Waku-Waku; 7. I feel Waku-Waku very much. 6. For an existing scale to check the validity of the

124

R. Inoue and T. Maeno

facts that were obtained concerning overt Waku-Waku factors of positive emotion, we adopted the “Japanese version of PANAS scale” [22, 23].

7.6 Survey Results 7.6.1 Quantitative Survey Results We collected 389 Waku-Waku events at work from qualitative surveys, such as “talking with the president,” “a new PC is delivered,” “moving to the new office,” “when the office is clean,” “when the work report is better than expected,” “when going to a seat with a good view,” “when the deal is made,” “a party after pulling off an event,” “when a business partner has high motivation.” Subsequently, we looked into 389 Waku-Waku events collected from survey 1, 3, 4, and 5 by repeating an affinity diagram. As a result, we qualitatively obtained 41 Waku-Waku events at work in 6 categories. The result is shown in Fig. 7.2.

7.6.2 Quantitative Survey Results Valid answers were from 1,034 employed people (answer rate 103.4%). We got answers from all across Japan in keeping with the allotted sample numbers based on demographics, as we had expected. Breakdown of the respondents were 650 males (62.9%) and 384 females (37.1), so we had more males than females. Age brackets of respondents were: 20s (14%); 30s (23%); 40s (26%); 50s (26%); and 60s or above (10%). This indicates that we collected a reliable number of samples from each generation.

7.6.3 Result of Simple Tabulation and Discussion First, we checked the question, “I think it is important to feel Waku-Waku at work,” to know how important respondents think feeling “Waku-Waku” is. We used a sevenlevel Likert scale for all questions (1. I don’t think so at all; 2. I don’t think so; 3. I would rather not think so; 4. I can’t say; 5. I would rather think so; 6. I think so; 7. I strongly think so.) The results were: “7. I strongly think so” 10.9%; “6. I think so” 21.6%; “5. I would rather think so” 30.5%. This indicates that we confirmed 63% of respondents thought feeling Waku-Waku at work was important. We also confirmed that 9.3% of respondents did not think feeling Waku-Waku was important (“1. I don’t think so at all” 5.0% and “2. I don’t think so” 4.3%, see Table 7.1).

7 A Study of “Waku-Waku” at Work

Fig. 7.2 The result of qualitative survey

125

126

R. Inoue and T. Maeno

Table 7.1 Values of feeling Waku-Waku at work Q. 1 think it is important to feel Waku-Waku at work Male

%

Female

%

N

%

1 don’t think SO at all

30

0.05

22

0.06

52

0.05

1 don’t think so

27

04

17

04

44

0.04

1 would rather not think so

56

0.09

34

0.09

90

0.09

1 can’t say

140

0.22

57

0.15

197

0.19

1 would rather think so

185

0.28

130

0.34

315

0.30

1 think so

137

0.21

86

0.22

223

0.22

1 strongly think so

75

12

38

0.10

113

0.11

650

1.00

384

1.00

1034

1.00

Next, as women’s social progress has been permeating the workplace, we examined gender differences in values of Waku-Waku events at work. We also looked into the cases of “at home” and “private life (other than at work or home)” at the same time (Table 7.2). The result was that no significant difference in values of Waku-Waku events at work was found. However, women had high scores for values of WakuWaku events at home and private life (other than at work and home), meaning there was a significant gender difference. This suggests that men and women have similar levels of values of Waku-Waku events at work, because women’s social progress has become common in Japan, too. Next, we examined the situation of feeling Waku-Waku at work. In order to know how much working individuals subjectively feel Waku-Waku at work, we asked for an answer to the question “I feel Waku-Waku at work” (Table 7.3). As a whole, the results were: “7. I strongly think so” 3.9%; “6. I think so” 6.9%. This means 10.8% of respondents answered they clearly “feel Waku-Waku” at work. On top of this, if we add 17% of “5. I would rather think so,” it totals 27.8%. On the other hand, “1. I don’t think so at all” was 13.9% and together with 15.1% of “2. I don’t think so,” people who do not feel Waku-Waku accounted for 29%. We found that if 16.4% of “3. I would rather not think so” is included, 45.4% of respondents do not feel Waku-Waku at work. Moreover, 3.8% of males were “7. I strongly think so” and 5.8% were “6. I Table 7.2 Analysis of gender difference in Waku-Waku events Analysis of gender difference in Waku-Waku events (Mean value, SD and t-test) Male

Female

Mean

SD

Mean

SD

t store

1 think it is important to feel Waku-Waku at work

4.74

1.51

4.73

1.54

0.11

1 think it is important to feel Waku-Waku at home

4.86

1.41

5.16

1.40

−3.31**

1 think it is important to feel Waku-Waku in private life (other than at work and at home)

4.76

1.46

5.21

1.38

−4.93*

*p < 0.05, **p < 0.01

7 A Study of “Waku-Waku” at Work

127

Table 7.3 Question “I feel Waku-Waku at work” Q. I feel Waku-Waku at work Male

%

Female

%

N

%

1 don’t think so at all

83

0.13

61

0.16

144

1 don’t think so

97

0.15

59

0.15

156

0.15

113

0.17

57

0.15

170

0.16

1 would rather not think so

0.14

1 can’t .say

184

0.28

93

0.24

277

0.27

1 would rather think so

110

0.17

66

0.17

176

0.17

38

0.06

33

0.09

71

0.07

25

0.04

15

0.04

40

0.04

650

1.00

33

1.00

1034

1.00

1 think so 1 strongly think so

think so,” indicating that 9.7% of male respondents feel Waku-Waku at work. And if we add 16.9% of “5. I would rather think so,” it becomes 26.6%. On the other hand, 3.9% of females were “7. I strongly think so” and 8.6% were “6. I think so.” This means 12.5% of respondents feel Waku-Waku at work, which was slightly more than that of males. And if we add 17.2% of “5. I would rather think so,” it becomes 29.7%. The percentage of respondents who answered “I don’t think so at all” was 12.8% for males and 15.9% for females, indicating the percentage of females was slightly higher than that of males. However, the difference in the number of male and female samples requires us to carry out further survey for interpretation of this result. Because of this, we carried out a t-test to examine gender difference in feeling Waku-Waku at work. Though females showed a higher score, the result was t(758.4) = 0.17, n.s., which indicates that gender difference was not significant. These findings suggest that there was no gender difference as to how they subjectively feel WakuWaku at work. In Japan today, we found that it was unable to find the features of “those who feel Waku-Waku at work” by gender.

7.6.4 Finding Overt Waku-Waku Factors at Work and the Analysis of Them What events at work may induce, in working individuals, the psychological state that the word Waku-Waku can be applied to? Even though individual difference is presumed, we think that clarifying universal events will provide a useful viewpoint when an individual would want to feel Waku-Waku at work. So, we clarified the factor structure of working individuals’ subjective to Waku-Waku events at work. First, we calculated a missing value, a mean value, and a standard deviation for answers to the questions (41 items) designed according to the result of the qualitative survey. As we checked the ceiling effect and the floor effect, there was no missing value in the main data of 1034 people, suggesting that both effects were not found.

128

R. Inoue and T. Maeno

Thus, whole number was subjected to analysis. Next, we performed an exploratory factor analysis with the maximum likelihood method for the 41 items using IBM’s SPSS(24). The result was 19.88, 2.43, 1.74, 1.17, 0.86… and though the first factor was extremely high, we thought 4-factor structure was plausible due to the Guttman– Kaiser rule and the possibility of interpretation of the factors. So, we omitted items whose load is less than 0.35 on any factors. Considering what we are going to treat is on a psychological scale, we performed a factor analysis with Promax rotation of the maximum likelihood method. As a result, we found a factor structure that consists of 4 factors and 35 items. Correlation between the final factor pattern after Promax rotation and factors are shown in Table 7.4. The cumulative contribution ratio (a ratio to explain dispersion) of 4 factors and 35 items after rotation was 68.4%.

7.6.5 Naming Overt Work Waku-Waku Factors We named the obtained overt work Waku-Waku factors based on traits of lower scale items as follows (Table 7.5). The first factor is composed of 16 items, and vigorous items where awareness is toward challenges to the unknown or creation showed high load, such as “when thinking about new ideas or plots,” “when making a challenge in a new area,” “when working on different work or tasks from the usual ones” and “when working on uncertain and vague tasks.” This prompted us to name it the factor of “creation and challenges to the unknown.” The second factor is composed of 10 items, and the items where awareness is toward events of getting rewards or leaping at good luck showed high load, such as “when receiving monetary remuneration like salary or bonus,” “when there is something fun after work,” “when coming across lucky events out of the blue,” “when eating something good,” and “when encountering the opposite sex of my type.” So, we named it the factor of “happiness and pleasure.” The third factor is composed of 5 items, and these items where awareness is toward interpersonal interests at work showed high load, such as “when learning about colleague’s new aspects,” “when learning about colleague’s private secrets,” and “when meeting new people.” As such, we named it the factor of “interpersonal interests at work.” The fourth factor is composed of 4 items, and these items where awareness is toward external stimuli through the five senses showed high load, such as “when smelling a favorite fragrance at work,” “when hearing one’s favorite sound or music at work,” “when the office looks neat and tidy,” and “when eating one’s favorites during work.” So, we named it the factor of “sensitive pickiness.”

7 A Study of “Waku-Waku” at Work Table 7.4 The factor analysis result of the work Waku-Waku scale

129

130

R. Inoue and T. Maeno

Table 7.5 Naming overt work Waku-Waku factors Overt “work Waku-Waku factors” I

Creation and challanges of the unknown

Vigorous items where awareness is toward challanges to the unknown or creation

II

Happiness and pleasure

Items where awareness is toward events of getting rewards or leaping at good luck

III

Interpersonal interests at work

Items where awareness is toward interpersonal interests at work

IV

Sensitive pickiness

Items where awareness is toward external stimuli through the five senses

7.6.6 Verification of Internal Consistency of Overt Work Waku-Waku Factors We calculated the mean value of items that correspond to four lower scales of the overt sense of Waku-Waku scale at work, then we calculate an α-coefficient of each lower scale in order to consider internal consistency. The results were: I. “Creation and challenges to the unknown” α = 0.97; II. “Happiness and pleasure” α = 0.95; III. “Interpersonal interests at work” α = 0.87; IV. “Sensitive pickiness” α = 0.91. These results gave us sufficient values in each lower scale. Lower scale correlation of overt “sense of Waku-Waku” at work is shown in Table 7.6. Four lower scales showed a significant positive correlation. Table 7.6 Correlation between lower scales, the mean value, SD and α-coefficient of overt “work Waku-Waku factor” I

Creation and challenges of the unknown

II

Happiness and pleasure

III

Interpersonal interests at work

IV

Sensitive pickiness

*p < 0.05, **p < 0.01

Fl

F2

F3

F4

Mean

SD

α-coefficient

–

0.69

0.73**

0.04**

4.31

1.21

0.97

–

0.54**

0.58**

4.84

1.27

0.95

–

0.62**

3.88

1.10

0.87

–

4.08

1.28

0.91

7 A Study of “Waku-Waku” at Work

131

Table 7.7 Scores of overt “work Waku-Waku factor” scales and PANAS correlation analysis Correlation

NA (negative)

PA (positive)

I

Creation and challenges of the unknown

0.01

0.46**

II

Excitement at rewards or pleasure

0.15**

0.34**

III

Interpersonal interests at work

0.03

0.42**

IV

Sensitive pickiness

0.08**

0.40**

0.08**

0.48**

Total number of overt Waku-Waku factors *p < 0.05, **p < 0.01

7.6.7 Validation of an Overt Sense of Waku-Waku Factors at Work First, we conducted validation of overt work Waku-Waku factors, which was obtained by exploratory factor analysis using SPSS (24), as to whether it can be considered as hypothesized positive emotion. The result is shown in Table 7.7. As with potential work Waku-Waku factors, we adopted concurrent validation with the PANAS scale, whose reliability and validity have been already proved. This resulted in moderate to slightly low positive correlation with positive emotion for all factors. Furthermore, it is suggested that they have just about no correlation with negative emotions. Therefore, the overt work Waku-Waku factors we obtained to be a scale of a positive psychological state have been proven valid.

7.6.8 Verification by Correlation with “People Who Feel Waku-Waku at Work” In order to confirm whether overt work Waku-Waku factors obtained by exploratory factor analysis indeed represent “a sense of Waku-Waku at work,” we conducted correlation analysis with the question, “I feel Waku-Waku at work” and verified their scales. The result is shown in Table 7.8. We found that every factor from I to IV showed moderate correlation. Table 7.8 Scores of overt Waku-Waku factors and correlation analysis with “I feel Waku-Waku at work”

Correlation

N = 1034

I

Creation and challenges of the unknown

0.058*

II

Happiness and pleasure

0.31*

III

Interpersonal interests at work

0.49*

IV

Sensitive pickiness

Total number of overt Waku-Waku factors *p 0. We compared any two information gain functions

196

H. Yanagisawa

of δ using formula (12.10) with constant external noise between different degrees of uncertainty. If the two functions of different uncertainties have an intersection, then the information gains change as δ increases. We then assumed two information gain functions with different uncertainties, G 1 = α1 δ 2 + β1 and G 2 = α2 δ 2 + β2 . A condition where the two functions have an intersection is α1 δ 2 +β1 = α2 δ 2 +β2 . We derived δ 2 (α1 − α2 ) + (β1 − β2 ) = 0 under β1 = β2 . Therefore, (α1 − α2 )(β1 − β2 ) < 0 is the condition. We found that this condition applies when the relationship between different uncertainties sp1 and sp2 and constant external noise sl is as follows: s p1 s p2 > sl2 .

(12.13)

Because the uncertainty of prediction is likely to exceed the external noise (i.e., the uncertainty of sensory stimuli), the condition in question is likely to occur. Given formula (12.12), the greater the uncertainty, the greater the intercept of the information gain function. As the prediction error increases, the difference in information gains between the two functions changes such that lower uncertainty tends to mean greater information gain. Figure 12.2 shows two functions of information gain with respect to different uncertainties at constant external noise. The two information gain functions have an intersection point. The information gain as an index of arousal (in this case, surprise) increases as the prediction error increases. The prediction error and uncertainty have an interaction effect on information gain. The greater the uncertainty, the greater the information gain for zero or small prediction errors. The smaller the uncertainty, the greater the information gain for larger prediction errors. We term this intersectionrelated phenomenon the arousal crossover effect. 3 Uncertainty level 0.2

Information gain: G

Fig. 12.2 Information gain or surprise as a function of prediction error. The intercept and the slope change for uncertainty level

1.0

2

1

0

0

0.5

1

Prediction error: δ

1.5

12 A Mathematical Model of Emotions for Novelty

197

12.2.5 Valence Dimension We next investigated how novelty affects the valence dimensions of positivity and negativity. Berlyne [2] proposed collative variables that consist of stimulus factors, such as novelty, complexity, uncertainty, and conflict. Each collative variable has the quality of arousal potential (i.e., the ability to affect the intensity of arousal). Highly novel stimuli can increase arousal. Berlyne [3, 4] assumed that the hedonic qualities of stimuli arise from separate biological incentivization systems. The first system, the reward system, generates positive effect whenever arousal potential increases. The second system, the aversion system, generates negative effect whenever arousal potential increases. The aversion system has a higher absolute activation threshold than the reward system does. Thus, the joint operation of these two systems creates an inverted U-shaped curve, as shown in Fig. 12.3. The valence of a stimulus changes from neutral to positive as the arousal potential increases but shifts from positive to negative after the arousal potential passes the peak positive valence. This inverse U shape is reasonable. One may feel safe and experience boredom if stimuli are too familiar (i.e., not novel). Conversely, one may feel uncomfortable if stimuli are extremely unfamiliar and novel. However, in Bayesian models, repeated exposure to the same stimulus decreases both prediction errors and uncertainty. Thus, the iterative information gain for each update decreases. The decreasing information gain and the inverse U-shaped function may explain emotional desensitization, which is the Positive

Reward function

hr

Valence

Valence function

0

Gr

Ga

Information gain (Arousal level)

G

Aversion function

ha Negative

Fig. 12.3 Valence, emotion positivity and negativity, is an inverse U-shaped function as a summation of reward and aversion functions of arousal

198

H. Yanagisawa

psychological phenomenon of emotional responsiveness to a negative, aversive, or positive stimulus diminishing after repeated exposure to it [17]. The positive hedonic response to a stimulus is diminished by decreasing information gain after repeated exposure to it, and a negative hedonic response to an extremely novel stimulus is shifted to a positive or neutral response by decreasing information gain after repeated exposure. As noted in Sect. 12.2.1, we formalized the arousal level as information gain from an event. If an event does not provide any information, then the valence can be neutral. At the opposite extreme, if an event provides excessive information that is difficult for the brain to process, then the valence can become negative. We can reasonably assume that between these two extremes there lies a “sweet spot” at which an optimum information gain maximizes a positive valence. We formalized valence as a summation of the reward and aversion systems and used sigmoid functions [15] to model information gain for each system: Valence = Reward(G) + Aversion(G) −h a hr + = 1 + exp(−cr G + G r ) 1 + exp(−ca G + G a )

(12.14)

In these formulae, Gr and Ga represent the thresholds of information gain that activate reward and aversion systems, respectively. The variables hr and ha are the maxima of positive and negative valence levels, respectively, and cr and ca represent the respective gradients. The condition G r < G a must always be satisfied because the threshold of the reward system is lower than that of the aversion system. If an extreme information gain occurs, then the condition h r < h a must be satisfied to obtain a negative valence. Figure 12.3 shows the valence, reward, and aversion functions of formula (12.14). We can observe that the valence function is an inverse U-shaped curve.

12.3 Model Summary Figure 12.4 shows a schematic of our emotion model. We formalized emotional arousal using information gain from an event, which we represented as the KL divergence from the prior to the posterior. We derived the information gain as a function of three parameters: uncertainty, the prediction error, and external noise, which are represented as the variance of the prior (or entropy), the difference between the prior expectation and the peak of the likelihood function, and the variance of the likelihood function, respectively. We formulated valence (i.e., positivity or negativity) as a summation of reward and aversion systems represented as information gain functions. In our model, the information gain is a key parameter to explain the emotional dimensions of arousal and valence. The information gain increases as the prediction

Bayesian update

12 A Mathematical Model of Emotions for Novelty

Posterior

Prior

KL divergence

Information gain: G

Reward + Aversion

Uncertainty Prediction error

Likelihood

199

Noise

Arousal (surprise)

Valence (acceptance)

Emotion dimensions

Fig. 12.4 Summary of emotion model [22]

error increases. Recent neurological studies have shown that dopaminergic neurons encode the prediction error signal of reward [1, 16]. Our model explains a reward system as a function of information gain affected by prediction errors. From a mathematical analysis, we found that uncertainty and prediction errors have interaction effects on information gain. Prediction errors increase information gain. The greater the uncertainty, the more the information gain for zero or small prediction errors. In contrast, the smaller the uncertainty, the more the information gain for large prediction errors. Uncertainty represents the degree of belief in the prior expectation. The familiarity of an event or target and one’s knowledge and experience of a target affect uncertainty. For example, if a product is so familiar that everyone knows it well, then uncertainty about the product is small. In contrast, if a product is unfamiliar, then uncertainty about the product should be considerable. Thus, uncertainty represents prior information before experiencing a target event. Indeed, uncertainty is proportional to the information entropy of the prior, as in formula (12.9). This model suggests that emotion is influenced by prior information, discrepancies between expectations and reality, and stimulus attributes.

12.4 Conclusion The mathematical model formulated the intensity of emotion arousal such as surprise using information gain from an event, which can be calculated using KL divergence between the Bayesian prior and the posterior. We derived the information gain as a function of prediction errors, uncertainty, and noise. Prediction errors, which are differences between prior expectations and likelihood function peaks, increase information gain and surprise. We derived an interaction effect of prediction errors and uncertainty, which we termed the arousal crossover effect: uncertainty, represented as variance of the prior, increases information gain when prediction errors are zero or small, but uncertainty decreases information gain when prediction errors are large. The authors experimentally validated the arousal crossover effect using subjective reports and event-related potential P300 as indicators of arousal [22].

200

H. Yanagisawa

Uncertainty of the prior depends on an individual’s knowledge and prior experiences as well as the familiarity of an event. Prior knowledge and experience produce certainty of expectations. This implies that the model can explain individual differences in emotional responses to an identical novel event as resulting from differences in knowledge and prior experience. For example, an expert’s expectations should be more certain than those of a novice. Using our model, we can therefore predict that novices are more surprised than experts are when an event differs marginally from prior expectations but that experts are more surprised than novices are when an event greatly differs from prior expectations. We formalized emotional valence as a function of arousal levels. The functional model forms an inverse U-shaped curve that has a positive valence peak at a certain arousal level. Therefore, we can predict that variable uncertainty levels related to an individual’s knowledge and experience and the familiarity of an event modulate the effect of prediction errors on valence responses.

References 1. Bayer HM, Glimcher PW (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47(1):129–141 2. Berlyne DE (1970) Novelty, complexity, and hedonic value. Percept Psychophys 8(5):279–286 3. Berlyne DE (1967) Arousal and reinforcement. Nebr Symp Motiv 15:1–110 4. Berlyne DE (1971) Aesthetics and psychobiology. Appleton-Century-Crofts, New York 5. Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429. https://doi.org/10.1038/415429a 6. Itti L, Baldi P (2009) Bayesian surprise attracts human attention. Vision Res 49(10):1295–1306. https://doi.org/10.1016/j.visres.2008.09.007 7. Kersten D, Mamassian P, Yuille A (2004) Object perception as bayesian inference. Annu Rev Psychol 55(1):271–304. https://doi.org/10.1146/annurev.psych.55.090902.142005 8. Kullback S, Leibler RA (1951) On information and sufficiency. Ann Math Stat 22(1):79–86 9. Körding KP, Wolpert DM (2004) Bayesian integration in sensorimotor learning. Nature 427(6971):244–247 10. Lang PJ (1995) The emotion probe: studies of motivation and attention. Am Psychol 50(5):372 11. Loewy R (1951) Never leave well enough alone: the personal record of an industrial designer from lipsticks to locomotives, Simon & Schuster 12. Ma WJ, Beck JM, Latham PE, Pouget A (2006) Bayesian inference with probabilistic population codes. Nat Neurosci 9:1432. https://doi.org/10.1038/nn1790 13. Mauss IB, Robinson MD (2009) Measures of emotion: a review. Cogn Emot 23(2):209–237. https://doi.org/10.1080/02699930802204677 14. Russell JA (1980) A circumplex model of affect. J Personal Soc Psychol 39(6):1161 15. Saunders R (2012) Towards autonomous creative systems: a computational approach. Cogn Comput 4(3):216–225. https://doi.org/10.1007/s12559-012-9131-x 16. Schultz W, Dayan P, Montague PR (1997) A Neural substrate of prediction and reward. Science 275(5306):1593–1599. https://doi.org/10.1126/science.275.5306.1593 17. Sekoguchi T, Sakai Y, Yanagisawa H (2019) Mathematical model of emotional habituation to novelty: modeling with Bayesian update and information theory. In: Proceedings of the IEEE international conference on systems, man & cybernetics, SMC2019, Bari, Italy 18. Shannon CE, Weaver W, Blahut RE, Hajek B (1949) The mathematical theory of communication, vol 117. University of Illinois press Urbana

12 A Mathematical Model of Emotions for Novelty

201

19. Stocker AA, Simoncelli EP (2006) Noise characteristics and prior expectations in human visual speed perception. Nature Neuroscience 9(4):578–585 20. Yanagisawa H (2016a) A computational model of perceptual expectation effect based on neural coding principles. Journal of Sensory Studies 31(5):430–439. https://doi.org/10.1111/joss. 12233 21. Yanagisawa H (2016b) Expectation effect theory and its modeling. In: Fukuda S (eds) Emotional engineering, vol 4. Springer, Cham 22. Yanagisawa H, Kawamata O, Ueda K (2019) Modeling emotions associated with novelty at variable uncertainty levels: a Bayesian approach. Front Comput Neurosci 13(2). https://doi. org/10.3389/fncom.2019.00002

Chapter 13

The Relation Between Characteristics of Forest Sounds and Psychological Impression Terutaka Yana and Takashi Maeno

Abstract In recent years, the effects of forests on the psychological and physiological aspects of human beings have been identified, and effects such as relaxation, stress reduction, and health promotion have been clarified. On the other hand, there are still many unclear points about what elements of the forest environment have a relaxing effect and a stress-reducing effect on humans. Therefore, as one of the elements of them, this study focused on the sounds of the forest environment, extracted characteristics from the measurement data of forest sounds, and investigated the psychological impression of forest sounds. As a result of this study, we found that the sound of the forest is quiet; the sound sources contain high-frequency components, and these sources are fluctuating. Moreover, sounds with this tendency were found to be comforting, open/active, and related to a sense of relaxation. This result suggests that the sound of a forest environment is one of the factors that have a relaxing effect on humans.

13.1 Introduction In life in the city, humans might have forgotten a valuable treasure. What can we hear in the hustle and bustle of the city when we close our eyes and listen carefully? The driving sounds generated by cars and motorcycles, the roaring sound of airplanes that cut through the air, automatic announcements that are repeatedly played in stations, the stagnant noise of air conditioners, and background music that is loud enough to drown out the sound of people talking. All of these sounds are produced by artifacts. A question arises when we try to capture sounds closely related to our daily lives. Are we able to adapt to the modern urban environment filled with artifacts?

T. Yana (B) · T. Maeno Graduate School of System Design and Management, Keio University, 4-1-1, Hiyoshi, Kohoku-ku, Yokohama City, Kanagawa Prefecture 223-8526, Japan e-mail: [email protected] © Springer Nature Switzerland AG 2020 S. Fukuda (ed.), Emotional Engineering, Vol. 8, https://doi.org/10.1007/978-3-030-38360-2_13

203

204

T. Yana and T. Maeno

13.2 Immigration to Urban Areas The revised UN World Cities Population Forecast 2018 reported that 55% of the world population lives in urban areas as of 2018 [1]. The urban population, which was only 30% in 1950, is expected to make up 68% of the world population in 2050, as people move to the city in search of jobs and new opportunities. Urban areas are blessed with access to information, food, education, medical care, etc., and the benefits of people living efficiently and comfortably are significant. So the urban population rate will inevitably continue to increase. On the other hand, some studies have reported the adverse effects of urban environments on human health and psychology. There are reports that people living in urban areas are 21% more likely to develop anxiety disorders and 39% more likely to develop mood disorders than those living in rural areas [2], and the prevalence of schizophrenia was doubled [3]. Also, a report shows that the brains of people living in the city tend to be more susceptible to social stress than people living in the countryside [4]. In an artificial environment created for convenience, humans can now live a comfortable life. However, if human beings stay only in the environment that they have created, is there a possibility that they will eventually meet contradictions that cause mental and physical disorders and modulations?

13.3 Environmental Sounds of Urban Life Focusing on the environmental sounds of urban life, we notice that we are living in a structure surrounded by sounds produced by artifacts or in structures isolated from sounds. However, humans easily get used to the sounds around them, therefore there is still little interest in the effects of environmental sounds on our health and psychology. In February 2019, The World Health Organization (WHO) reported that listening to loud sounds for a long time on acoustic devices such as smartphones and audio players could cause hearing loss [5]. About 1.1 billion people, nearly half of the world’s 12–35-year-olds, are at risk of noise-induced hearing loss. The WHO has recommended a guideline for safe sound levels that do not cause hearing impairment. For adults, the sound level is about 80 dB, which is about the same level of noise as a running train, and for children, it is suppressed to 40 h per week at a volume of 75 dB. This warning is mainly for intentional input to the ear. When it comes to urban life, we are unknowingly surrounded by loud noises, and urban environmental sounds such as car and airplane sounds, explosive background music, etc. which may be louder than the WHO warning level.

13 The Relation Between Characteristics of Forest Sounds …

205

13.4 Unconscious Stress Miyazaki states that modern people are unconsciously stressed out by an urban environment. “We live in a man-made urban environment since we were born, so it is hard to notice, but modern people are always tense and are too sympathetic [6].” Sounds from an urban environment may be an influencer of unconscious stress on humans beings. Therefore, we are focusing on a natural environment that is opposite to an urban environment, especially the sound of a forest environment.

13.5 Forest Research In recent years, there has been a boom in getting in touch with nature, such as hiking, mountain climbing, glamping, seeking health-consciousness, and healing. There is also news that Shinrin-Yoku, which was advocated by Japan as the latest fitness trend, is a hit in the US [7]. In the field of research, scientific elucidation of the physiological and psychological effects of staying in the forest has advanced in the twenty-first century. In particular, research on the relations between forests and humans, including Shinrin-Yoku, is gaining attention [8]. For example, previous studies have shown that salivary amylase, which increases in proportion to stress, decreases day to day by staying in a forest for several days [9]. Also, a study reported that when sitting in urban and forest environments for 15 min each, parasympathetic nerve activity was significantly increased in the former, and sympathetic nerve activity was significantly decreased in the latter [10]. Another study reported that creativity was increased by 50% after hiking in nature for 4– 6 days by cutting off all electronic devices [11]. In this way, research on the impact of forests on human beings has progressed from the psychological and physiological aspects, and it has gradually become clear that humans can obtain relaxation and stress reduction effects when staying in forests.

13.6 The Purpose of Researching Forest Sounds So, what do forest environments have for humans to relax and reduce stress? What is the input from the forests to the human senses? Focusing on the elements of forest environments, research on the psychological and physiological effects of these elements are still insufficient. Although there are some reports on the effects of bird sounds and river sounds derived from the natural environment on human psychology and physiology, there are few studies focusing on the relations between acoustic parameters and human psychology and physiology from an approach that captures the entire forest environment. Therefore, in this study, instead of focusing on specific

206

T. Yana and T. Maeno

natural sounds, we measured sounds at various locations in the forest and extracted the characteristics of the sound of the entire forest. We aimed to investigate the relations between the characteristics and psychological impressions.

13.7 Characteristic Extraction of Forest Sounds To extract the characteristics of forest sounds, we measured noise level and 1/3 octave band frequency analysis in forests and urban environments. We selected even forests as survey sites for the forest environment in the suburbs away from the city center of Tokyo, and four of them are recognized as forest therapy bases [12]. The urban environment was classified into outdoor and indoor areas, and the outdoor was measured at station platforms, high-traffic roads, and forests and parks representative of Tokyo. Also, measurements were taken in cafes, restaurants, libraries, and general households, which are generally considered to be quiet. We measured at each survey site during the day and acquired the data using a precision sound level meter (LA-7500 Ono Sokki).

13.7.1 Comparison of Suburban Forests and Urban Environments First, we indicated the primeval forest in the suburbs (Fig. 13.1 Kannami-Genseirin) as a measure of a forest environment, and as an example of the urban environment, measurement data of urban station platforms (Fig. 13.2) and stores such as cafes and restaurants.

Fig. 13.1 Kannami-Genseirin (Shizuoka prefecture in Japan)

13 The Relation Between Characteristics of Forest Sounds …

207

Fig. 13.2 Urban station platform in the morning commute

Figure 13.3 shows the measurement results of equivalent noise level (LAeq), minimum noise level (LAmin), and maximum noise level (LAmax) at each measurement point. This result shows that the noise level of suburban forests is about 25 dB lower than urban station platforms, cafes, and restaurants. Figure 13.4 shows the results of the 1/3 octave band frequency analysis of the equivalent noise level at each survey site. This result shows that the sound pressure of suburban forests is lower than the sound of the urban environment by about 20 dB or more over the whole frequency band. Also, the sound pressure level of suburban forest sounds tends to increase around 4 kHz. This tendency is due to the influence of sound sources, including high-frequency components present in the forest environment, such as bird sound, insect sounds, and water flowing sounds.

Fig. 13.3 Comparison of noise levels in urban and forest environments

208

T. Yana and T. Maeno

Fig. 13.4 Comparison of 1/3 octave band analysis of urban and forest environments

13.7.2 Comparison of Suburban Forests and Urban Forests The sound of the suburban forest environment was shown to have a lower noise level than that of the urban area. So we compared the measured data of the next seven suburban forest environments and five urban forests and parks with similar environments. Figure 13.5 shows the measurement results of the noise level (LAeq), minimum noise level (LAmin), and maximum noise level (LAmax) at seven suburban forest environments and five urban forests and parks. This result shows that the noise level of the suburban forest environment is about 8 dB lower than the urban forest environment. Figure 13.6 shows the results of the 1/3 octave band frequency analysis at each survey site.

Fig. 13.5 Comparison of noise levels in urban and suburban forests

13 The Relation Between Characteristics of Forest Sounds …

209

Fig. 13.6 Comparison of 1/3 octave band analysis of urban and suburban forests

By comparing the measurement data between the forest and urban environment, the following are the indicators of the characteristics of forest sounds: 1. The equivalent noise level of the forest environment is less than 50 dB, and below 40 dB in the suburban forest environment. 2. The forest environment has sound sources that contain high-frequency components such as birds, insects, and rubbing leaves. Another characteristic is that the sounds of birds and insects containing high-frequency components and the rubbing sound of leaves are not constant and are variable.

13.8 Impression Evaluation of Sounds To extract elements that affect the psychological impression of forest sounds, we evaluated the feeling by hearing using the SD method. The evaluation sound sources were the sounds of the forest, urban life, a quiet cedar forest, a quiet room, sea waves, and a riverside. We selected 14 groups of adjective pairs by referencing previous research on the impression scale composition for environmental sounds [13] and the previous study on forest therapy [14]. Also, the evaluation scale was set to seven levels. The total number of participants in the experiment was 28 (twenty males and eight females) between the ages of 20–60. We instructed the participants to evaluate the sound source played from the speakers during a playback time of about 1 min without telling. After completing the evaluation of one sound source, we played the next sound source.

210

T. Yana and T. Maeno

13.8.1 Factor Analysis of Impression Evaluation Results Table 13.1 shows the results of the factor analysis of responses obtained from impression evaluations for a total of six sound sources, including forest sounds. As a result of factor analysis, we extracted two factors. The first factor was named “relaxation factor” because of the significant factor load of adjective pairs related to peace of mind and rest, such as “loose–busy” and “quiet–noisy.” Also, the second factor was named “open/active factor” because of the significant factor load of adjective pairs related to open spaces and stimuli such as “open–closed” and “bright–blurred.” Each factor contribution rate is 55.3% and 16.2% in the first and second, respectively, and the cumulative contribution rate is 71.6%. Next, Fig. 13.7 shows the average factor scores of the six sound sources for the two factors extracted by factor analysis. Forest sounds scored higher than other sound sources, with a score of two factors for “comfort factor” and “open/active factor,” while urban sounds scored lower for both factors. Table 13.1 Adjective pairs and the results of factor analysis of responses obtained from impression evaluations

Adjective pairs

Comfort

Relaxing–Busy

1.00

−0.29

Quiet–Noisy

0.97

−0.26

Rounded–Rough

0.95

−0.31

Calm–Frustrated

0.89

0.07

Refreshing–Annoying

0.76

0.23

Safe–Uneasy

0.75

0.21

Like–Dislike

0.71

0.32

Comfortable–Uncomfortable

0.68

0.37

Beautiful–Ugly

0.59

0.39

Open–Closed

0.08

0.83

Vivid–Blurred Vibrant–Stagnant Bright–Dark Dull–Sharp

Open & Active

0.03

0.79

−0.28

0.78

0.20

0.72

0.53

−0.61

Contribution (%)

55.3

16.2

Cumulo–Contribution (%)

55.3

71.6

13 The Relation Between Characteristics of Forest Sounds …

211

Fig. 13.7 The average factor scores of the six sound sources for the two factors

13.9 Relations Between the Analysis Value of Sound Sources and the Impression Evaluation Results The six sound sources used in the impression evaluation were analyzed using five indicators: SPL, loudness, fluctuation intensity, sharpness, and roughness. We investigated the relations between the value extracted by the analysis and the “comfort factor” and “open/active factor” derived from the result of impression evaluation by factor analysis. Also, in the impression evaluation of the sound source, we got an answer about “relaxed-not relaxed,” and this value was used as a “relaxation index” to examine the relations with the acoustic analysis value.

13.9.1 Multiple Regression Analysis of the Impression Evaluation Results Multiple regression analysis was performed using the analysis values of five indicators of SPL, loudness, fluctuation strength, sharpness, and roughness as independent variables, and two factors from factor analysis as dependent variables. Table 13.2 shows the results of multiple regression analysis of “comfort factor” and analysis values, and Table 13.3 shows the results of multiple regression analysis of “open/active factor” and analysis values. The commonality between the two factors shows that the sound level was significantly affected by “low sound pressure level,” “variable,” and “the center of frequency energy balance is in the high-frequency band.” Next, the results of the multiple regression analysis of the “relaxation index” are shown in Table 13.4.

212

T. Yana and T. Maeno

Table 13.2 The results of multiple regression analysis of “comfort factor” and analysis values Model

B

(Constant) Sound pressure level

SE B

β

p-value

4.231

0.678

−0.074

0.013

−0.835

0.013

0.009

0.243

0.135

16.951

4.222

0.221

0.000

Loudness Fluctuation strength

0.000 0.000

Sharpness

0.347

0.067

0.303

0.000

Roughness

−24.415

10.215

−0.318

0.018

R2

0.631***

***p < 0.001

Table 13.3 The results of multiple regression analysis of “open/active factor” and analysis values Model

B

(Constant) Sound pressure level

SE B 3.328

0.805

−0.096

0.015

Loudness

β

p-value

−1.104

0.000

0.000

0.036

0.011

0.664

0.001

25.287

5.014

0.337

0.000

Sharpness

0.667

0.079

0.596

0.000

Roughness

0.705

12.133

0.009

0.954

R2

0.454***

Fluctuation strength

***p < 0.001

Table 13.4 The results of the multiple regression analysis of the “relaxation index” and analysis values Model

B

SE B

(Constant)

12.223

1.429

Sound pressure level

−0.161

0.027

Loudness

β

p-value 0.000

−0.927

0.000

0.030

0.019

0.282

0.108

37.009

8.898

0.247

0.000

Sharpness

0.823

0.141

0.369

0.000

Roughness

−31.732

21.531

−0.212

0.142

Fluctuation strength

R2

0.569***

***p < 0.001

In common with the two factors mentioned above, the relaxation index was also significantly affected by “low sound pressure level,” “variable,” and “frequency energy balance centered in a high-frequency band.”

13 The Relation Between Characteristics of Forest Sounds …

213

13.10 Conclusion The results of this study are summarized as follows together with the results of forest sound characteristics extraction and impression evaluation. The characteristics of forest sounds are quiet, and there are fluctuating sound sources, including high-frequency components. Moreover, it became clear that sounds that have these tendencies were related to comfort, openness and activeness, and relaxation. The results of this study suggest that the psychological impression due to the characteristics of forest sounds is one of the factors, regarding the previous research in which urban dwellers get a relaxing effect by staying in the forest.

13.11 Epilogue What are the treasures that have been forgotten by humans in modern urban life? Is it not “silence?” The result of this study suggests what “silence” is and suggests the answer. This is because it seems that the three conditions characteristic of forest sounds: (1) quiet, (2) sound sources containing high-frequency components, and (3) the sound source is not constant but fluctuates, are related to silence. Silence is entirely different from noiselessness. As Alan Corban wrote in his book: Silence is not simply the absence of noise. We have almost forgotten what it is [15].

Silence may mean that “because it is tranquil, we can hear the sounds of nature well.” This condition is what I feel every time when looking at the sound pressure level of the sound level meter in a quiet suburban forest. Silence means that there are sounds, and it is quiet but never noiseless. This “silence” is a valuable treasure that has been forgotten by people living in urban environments. Because in the silence of nature, humans can relax while gaining a sense of comfort and feel open/active.

References 1. World Urbanization Prospects (2018) United Nations Population Division. https://population. un.org/wup/. Accessed 11 June 2019 2. Peen J, Schoevers RA, Beekman AT, Dekker J (2010) The current status of urban-rural differences in psychiatric disorders. Acta Psychiatr Scand. https://doi.org/10.1111/j.1600-0447. 2009.01438.x 3. Krabbendam L, van Os Jim (2005) Schizophrenia and urbanicity: a major environmental influence—conditional on genetic risk. Schizophr Bull. https://doi.org/10.1093/schbul/sbi060 4. Lederbogen F et al (2011) City living and urban upbringing affect neural social stress processing in humans. Nature. https://doi.org/10.1038/nature10190 5. Deafness and hearing loss. World Health Organization. https://www.who.int/news-room/factsheets/detail/deafness-and-hearing-loss. Accessed 21 June 2019 6. Miyazaki Y (2003) Why Shinrin-Yoku is good for our health?. Bunshun Shinsho

214

T. Yana and T. Maeno

7. Chillag A (2017) Why you should be forest bathing (and we don’t mean shampoo). CNN. https://edition.cnn.com/2017/08/10/health/forest-bathing/index.html. Accessed 21 June 2019 8. Tsunetsugu Y, Park BJ, Miyazaki Y (2010) Trends in research related to “Shinrin-Yoku” (taking in the forest atmosphere or forest bathing) in Japan. Environ Health Prev Med. https://doi.org/ 10.1007/s12199-009-0091-z 9. Takayama N et al (2007) Influence of five-day suburban forest stay on stress coping, resilience, and mood states. J Environ Inf Sci. https://doi.org/10.11492/ceispapersen.2017.2_49 10. Park BJ et al (2007) Physiological effects of Shinrin-Yoku (Taking in the Atmosphere of the Forest)—Using salivary cortisol and cerebral activity as indicators. J Physiol Anthropol. https:// doi.org/10.2114/jpa2.26.123 11. Atchley RA, Strayer DL, Atchley P (2012) Creativity in the wild: improving creative reasoning through immersion in natural settings. PLoS ONE. https://doi.org/10.1371/journal.pone. 0051474 12. Forest Therapy. https://www.fo-society.jp/quarter/cn49/62forest_across_japan.html. Accessed 22 June 2019 13. Miyakawa M, Aono S (2001) Reexamination of rating scales on the impressions of environ mental sounds. Acoust Soc Jpn 58(3):151–164 14. Park BJ et al (2011) Relationship between psychological responses and physical environments in forest settings. Landsc Urban Plan. https://doi.org/10.1016/j.landurbplan.2011.03.005 15. Corbin A (2018) A history of silence: from the renaissance to the present day. Polity Press, Medford

Chapter 14

Artificial Intelligence and Virtual Reality-Based Kansei, Emotional, and Cognitive Science and Engineering Keiichi Watanuki

Abstract In modern Japanese society, managing a super-aged society is an urgent task, and high demands exist for ensuring safety and comfort. Hence, the elucidation of human cognition, judgment, and behavior is important. This paper introduces a new trend of approaches, including the engineering approach to kansei and cognition. Various technologies such as noninvasive living body information measurement, artificial intelligence, internet of things, virtual reality, and human–machine interface have been integrated, and the new kansei, emotional, and cognitive science and engineering have become a basic technology for improving quality of life.

14.1 Variety of Color Perception The “discourse on method” (“Discourse on the method of rightly conducting one’s reason and of seeking truth in the sciences: attempt in refractive optics, meteorology, and geometry”) published by philosopher René Descartes (1596–1650) reveals that the rainbow is a white light of the sun refracted in fine water droplets in the air. This suggests that if colored light is collected, it becomes white light. Sir Isaac Newton (1642–1727) combined prisms, convex lenses, and slits, says, “The white light of the sun could be broken down into a lot of light like a rainbow by a glass prism.” This suggests that light brings a sense of color to humans. Physicist Thomas Young (1773–1829) advocated the “three primary colors of light,” and physiologist/physicist Hermann Ludwig Ferdinand von Helmholtz (1821–1894) developed the “three primary colors of light,” and clarified that people can recognize colors. Three elements of color vision exists: red, green, and blue; when these are stimulated at the same rate, white is felt. Different colors are generated according to the ratio of three-element stimuli. Later, the three most sensitive types of red, green, and blue were identified in a cone, which is the color vision receptor of the retina. The color vision of organisms K. Watanuki (B) Department of Mechanical Engineering, Graduate School of Science and Engineering, Saitama University, Saitama, Japan e-mail: [email protected] Advanced Institute of Innovative Technology, Saitama University, Saitama, Japan © Springer Nature Switzerland AG 2020 S. Fukuda (ed.), Emotional Engineering, Vol. 8, https://doi.org/10.1007/978-3-030-38360-2_14

215

216

K. Watanuki

with three types of color vision receptors, such as humans, is called trichromacy. Furthermore, light stimuli are perceived by three types of cones and processed as three-dimensional sensory information, in which the light color is recognized as the mixing ratio of the three primary colors. Organisms with different numbers of types of color vision receptors feel color with different numbers of primary colors. Meanwhile, organisms with tetrachromacy have four types of color vision receptors and recognizes colors by combining the four primary colors. Humans could detect wavelengths from 800 nm (red) to 400 nm (purple), and tetrachromacy organisms could detect ultraviolet rays of wavelengths from 300 to 330 nm. Many birds and reptiles have tetrachromacy, as do some people. In addition, the wavelength to which human color vision receptors respond varies by person. The spectral absorbance and photosynthesis rate of chlorophyll are large at short wavelengths (blue) and long wavelengths (red), and small at medium wavelengths (green). Blue and red light are necessary for photosynthesis, but green is unnecessary; green leaves transmit and reflect unnecessary medium-wavelength light. Comparing the spectral intensity distribution of sunlight with the standard spectral luminous efficiency of humans, the distribution and maximum value of the spectral intensity of sunlight are almost the same as the distribution of human photosensitivity. Green light overflows on earth, and living organisms have high sensitivity to green light in the medium-wavelength range. From the viewpoint of light energy utilization, the natural environment and living organisms coexist well at the wavelength of light.

14.2 Product Design Considering Human Perception and Sensitivity 14.2.1 Perceptual Characteristics Humans are sensitive to color vision and have excellent ability to discriminate between colors. The color of an object is a sensation determined by the physical property of spectral intensity distribution, and the sensation varies depending on the observer. In other words, for color expression, it is necessary to consider the light source, object, and human senses. Many cases exist where visual information is managed in product design. Among them, “color” is often managed quantitatively, and many viewpoints exist such as physical and psychological aspects. In human vision, light is captured by photocells on the retina and then converted into electrical signals by retinal neurons. Additionally, the visual cortex of the cerebral occipital lobe is activated via the chiasma and lateral knee nucleus. In response, activity occurs in various brain regions. The photoreceptor cells include highly sensitive pyramidal cells in light and highly sensitive rods in the dark. Three types of cones exist depending on their spectral sensitivity; they behave differently in various wavelength regions. In addition, the human eye exhibits chromatic aberration that is

14 Artificial Intelligence and Virtual Reality-Based Kansei …

217

out of focus owing to short and long wavelengths, rendering it difficult to focus on different types of light. Furthermore, the visual system adapts by changing the sensitivity according to the brightness level. It is necessary to consider eye characteristics such as light adaptation from dark to light levels and dark adaptation from light to dark levels. In the color system that expresses object colors, the modified Munsell renotation system uses a three-dimensional color space based on the three attributes of color: hue, saturation, and lightness.

14.3 Kansei and Emotion Kansei is an intuitive ability for ambiguous and backhanded information contained in a subject, and is a human’s perceptual ability. When the mind changes, the brain’s reaction “to be aware of something (cognition)” occurs, and emotions arise depending on the perceived object. In other words, emotions related to Kansei exist. Emotion comprises four aspects: cognition, independent experience (feeling), expression behavior (facial expression, body movement), and physiological changes. Although all emotions are not stimulated, emotional stimuli are perceived, the cognitive process is activated, and emotional responses such as subjective experience, expression behavior, and physiological changes occur. These emotional responses are fed back as internal stimuli, and an interaction occurs between cognition and emotional responses. In recent years, noninvasive brain function measurement technology has progressed, and research on the brain internal mechanism of emotion generation has become active primarily in the field of neuroscience. Independent experience is a state of awareness and consciousness of emotions. This is a state in which the activity of the emotion center is projected to the inner world of consciousness, and the awarded and conscious emotion is a psychological state that can be confirmed by the person experiencing the emotion, although the measurement is becoming possible by noninvasive brain function and physiological measurements. Behaviors such as body movements, facial expressions, and voice expressions can be measured by motion and image measurements. Physiological changes caused by slight changes in emotions are often not noticed, although recently it has been measured by noninvasive brain function and physiological measurements, which are measurements of internal organs and body reactions.

14.4 Noninvasive Brain Function Measurement 14.4.1 Brain Activity Measurement Approximately 100 billion neurons exist in the entire human brain, and approximately 1000 trillion synapses connect them. If these connections can be elucidated, the

218

K. Watanuki

functions and operations of the brain can be clarified, as well as, the connectome, which is a three-dimensional map of the neural circuit. Along with neural activity, which is the primary signal of the brain, metabolic activity and blood flow increases, which is a secondary signal. This series of processes is collectively referred to as neurovascular coupling. Noninvasive brain function measurement is the measurement of the primary or secondary signal generated when a stimulus or task is given without causing reversible changes to the brain; furthermore, from the spatial or temporal pattern of the signal, the brain part and time from which the neural activity occurred are estimated, and the correspondence with the function and mental activity of the part is examined from the characteristics of the given stimulus and task and the correspondence with the behavior of the subject. The main brain function measurement methods are as follows: (1) Primary signal: electroencephalography (EEG), magnetoencephalography (2) Secondary signal: functional magnetic resonance imaging (fMRI), positron emission tomography (PET), single-photon emission computed tomography, near-infrared spectroscopy (NIRS).

14.5 NIRS Measurement Principle When a person decides on information transmission, information processing, behavior, and reaction, neural activity occurs in the brain. When nerve activity occurs, blood flow and blood volume increase in tissues near the active nerve, and the concentration ratio of oxygenated hemoglobin (oxyHb) to deoxygenated hemoglobin (deoxyHb) in the blood changes. In addition, in hemoglobin, the near-infrared absorption from 700 to 900 nm varies depending on oxygen concentration. Therefore, it is possible to measure changes in oxyHb concentration in the blood using near-infrared radiation [1]. This method using near-infrared is called NIRS. Compared with PET and fMRI, the equipment for NIRS is small, portable, and less restrictive for measurement; hence, the posture and movement of a subject will not affect measurements significantly. In addition, as it does not generate a strong magnetic field, it can be used in combination with various devices. Therefore, it is often used to measure brain activation response during exercise. In NIRS, a dedicated mounting holder is fixed such that it is in close contact with the subject’s head, and near-infrared light of three wavelengths is incident and received in the subject’s brain. Hence, it can measure noninvasively and is less restricted by posture and movement. It captures changes in hemoglobin on the surface of the cerebrum, and based on its oxygen concentration, the brain activity in the measurement area (brain activation response) is displayed as color mapping in real time with the device. A light-transmitting, light-receiving probe are installed on the scalp at a distance of 30 mm, and the near-infrared light that passes through the living body 20–30 mm

14 Artificial Intelligence and Virtual Reality-Based Kansei …

219

above the scalp is captured. Typically, visible light perceived by the human eye is strongly scattered by living tissues and strongly absorbed by chromoproteins (such as hemoglobin) contained in blood. Therefore, light does not reach the cerebrum, rendering it difficult to measure the cerebrum located inside the skull from above the scalp. However, near-infrared light (700–900 nm) has relatively high permeability to the skin and skull, and light reaches the surface of the cerebrum from above the scalp. Near-infrared light radiates from the light-transmitting probe and then penetrates the skin and skull to enter the body, reflecting and absorbing into the body tissue while drawing a banana-like arch reaching a distance of 20–30 mm from the probe. The reflected light that has not been absorbed by the brain returns to the surface of the scalp and is detected by the light-receiving probe. Hence, near-infrared light is emitted to the living body and returns to the surface of the scalp. Furthermore, the hemoglobin concentration is measured from the difference in the amount of reflected light detected by the light-receiving probe. It is noteworthy that the data measured by NIRS are a relative value because the optical path length at this time is unknown. Generally, in brain function measurement by light, the concentration changes of oxyHb and deoxyHb are measured primarily using the light absorption characteristics of blood hemoglobin. These oxyHb and deoxyHb have different light absorption spectra. This spectral difference is intuitively understandable because arterial blood rich in oxygen is bright red and venous blood after supplying oxygen to tissues is dark red. OxyHb and deoxyHb can be obtained using two different measured wavelengths and by solving simultaneous equations based on the modified Lambert Beer law because the respective molar molecular absorption coefficients are known. In general, this two-wavelength method is used. In fact, it is affected by noise components. By measuring brain activities in the visual cortex, auditory cortex, and motor cortex using noninvasive brain function measurement technologies such as NIRS and fMRI, parts responsible for brain functions such as visual recognition, language understanding, and motor control can be mapped. By adding multiple brain parts and interactions related to cognition, semantic understanding, creativity, sensibility, and stress, a brain function model including brain dynamic models can be built. This enables not only visual, auditory, tactile, taste, and olfactory measures to be obtained, but also the quantitative evaluation of cognitive status, emotion estimation, and semantic understanding.

14.6 Interface Design Considering Emotion 14.6.1 Human–Machine Interface In product development, user requirements are accurately grasped and a product is designed according to the requirements. If the design plan is confirmed to satisfy the desired user requirements, the product is then handed over to the user. This process is called a human-centered design process and is shown as ISO 13407/JIS Z 8530

220

K. Watanuki

“Ergonomics of human–system interaction—Human-centered design for interactive systems.” For a person to operate a machine, an operation unit is generally provided on the equipment side. The operation section includes a push button switch, dial knob, lever handle, and meter. The part that becomes the contact point that reflects the operator’s intention to the machine or the contact point that notifies the operator of the state of the machine is called a human–machine interface (HMI). For example, in automobiles, many HMIs are required for driving such as an accelerator pedal, brake pedal, shift lever, steering wheel, speedometer, and tachometer. Additionally mechanical parts, graphical user interface (GUI), touch panel, voice recognition, and image recognition are HMIs. Recently, as shown in Fig. 14.1, an HMI is regarded as not only the interaction between machines and people, or the communication between people via machines, but also an integration with technologies such as virtual reality (VR), internet of things (IoT), and artificial intelligence (AI) that is used as an interface that considers human perception, cognition, thinking, judgment, behavior, and emotions. In a human-friendly HMI design, it is necessary to consider the universal design (UD) proposed by Ronald L. Mace. A UD is a design of equipment, products, and information that can be used regardless of differences in culture, language, nationality, age, gender, disability, or difference in ability. The seven principles of a UD are as follows: (1) fair use, (2) flexibility in use, (3) simple and intuitive use, (4) recognizable information, (5) generosity to failure, (6) less physical effort, and (7) size and space for access and use.

Fig. 14.1 AI/IoT/VR/HMI techniques

14 Artificial Intelligence and Virtual Reality-Based Kansei …

221

As human intentions and emotions are processed in the brain, the brain–machine interface (BMI), which operates devices by measuring and recognizing brain activity, has attracted attention. Two types of BMI exists: invasive and noninvasive. Furthermore, two types of invasive methods exist: placing electrodes in a relatively safe place such as under the dura (partial invasive method) and implanting electrodes directly in the brain. The noninvasive method measures human brain activity through EGG, fMRI, positron decay tomography, magnetoencephalography, and NIRS. Currently, systems that move robots and other machines based on brain signals are popular. Furthermore, although muscles can be directly controlled by brain signals, it can be applied to assistive devices such as prosthetic hands and artificial legs.

14.7 Affective Computing Affective computing is a concept advocated by Professor Rosalind R. Picard of the Massachusetts Institute of Technology, in 1997. It develops IT systems and devices that understand and express human emotions [2]. Affective computing is related to a wide range of fields such as VR, AI, consumer electronics, cognition, health, HMI, music, wearable computing, social science, behavioral science, and social robots. In a previous study regarding affective computing, the facial action coding system (FACS) theory was devised by Paul Ekman in the 1970s. The FACS is a theory that encodes various facial movements by combining basic movements. It is used to express a CG or animation character with a realistic expression, construct a camera that reads various expressions, and develop robots that express emotions and sensing. Affdex, an emotion-recognition AI developed by Affectiva, founded by Rana el Kaliouby and Rosalind R. Picard, uses a web camera that captures slight movements of the target face muscles in real time; emotions are converted into data and then analyzed. It is used in products such as mobile applications, games and robots, education, and healthcare. The Microsoft Cognitive Services Application Programming Interface (API) allows one to instantly obtain images of people who are surprised, pleased, or sad from many images using multiple APIs; search for video frames; and extract the moment of a certain emotion. By fusing these APIs and systems, a system that can recognize emotions can be developed. In addition, Empath developed a system that can analyze emotions using AI technology from voice and perform various analyses from mental care to presentation analysis.

14.8 Application to HMI The authors are studying HMI technology to realize an ambient society, in which information and communication technology blend naturally into society such that people can live safely and comfortably without being conscious of it. For example, in a car-driving environment, not only the human side acts to access the machine or

222

K. Watanuki

the environment side, but also the machine or environment side senses humans with an advanced sensor interface, and the machine or environment side also supports humans autonomously. We are developing an ambient mobility interface that creates a safe and comfortable environment required for next-generation vehicles. Based on brain science and engineering, BMI technology that connects people and machines and interacts with them has been developed. The authors are studying the application of BMI combined with human-friendly robot technology based on technology that measures human brain functions noninvasively and technology that reads and analyzes human intentions, for medical welfare equipment such as surgical robots and wheelchairs. In addition, we are researching a system that can easily obtain the knowledge that users require from a vast amount of knowledge through an appropriate method. The system should be able to analyze the conversations of manufacturing engineers and technicians, extract design and manufacturing knowledge, organize them, build a knowledge database (explicit knowledge), and merge manufacturing processes (tacit knowledge) using multimedia technology. Furthermore, we are developing a system that can effectively convey skills that are difficult to convey in words by presenting information related to the five senses, such as sight and touch, using VR technology and robot technology. An environment is realized where multiple engineers and technicians can enter the VR space; additionally, VR and augmented reality technology enable an effective design review of the design and manufacturing processes through communication and collaboration. Furthermore, research on new marketing, such as methods for the quantitative evaluation of product impressions and methods for presenting sensory information for promoting communication, are performed based on brain science and engineering.

14.9 Skills Training Using AI/IoT/VR/HMI Technology The authors developed a skills training system for the transfer of basic manufacturing technology and skilled skills to create design and manufacturing knowledge, which is a new human resource development method for manufacturing engineers and technicians [3–5]. By analyzing manufacturing engineers’ and technicians’ conversations, extracting design/manufacturing knowledge, systematizing them, creating a knowledge database (explicit knowledge), and appropriately combining manufacturing process videos (tacit knowledge) using multimedia technology, it is a system that can easily obtain the knowledge that users require from a vast amount of knowledge. We have developed a virtual training system that combines a system that acquires manufacturing knowledge using multimedia and one that transfers skilled skills using VR technology. Based on the knowledge acquired from the skills training system, the user replaces it with a process of making it his own through experience in the VR space. In the process of expression and connection, a skills training system is used; meanwhile, in the process of internalization, a VR space is used and while knowledge is becoming a user’s own, new knowledge creation may be possible.

14 Artificial Intelligence and Virtual Reality-Based Kansei …

223

Perception, Cognition, Thinking, Judgment, Behavior, and Emotion Skills Transfer System

VR-Based Training System

AI/IoT/VR/HMI -Based System

Visual, tactile, and auditory information Communication

Novice On-the-job training

Skilled Worker Sharing a place

Skill Acquisition, Value creation

Fig. 14.2 AI/IoT/VR/HMI-based virtual training system

By experiencing not only visual but also tactile sensation, force sensation, and hearing of images displayed in the VR environment, the internalization of knowledge related to manufacturing and design is promoted. By utilizing this system to acquire design/manufacturing knowledge and provide visual, haptic, and auditory “places” for skill acquisition, problems in the acquisition of conventional design/manufacturing knowledge and skills can be partially overcome. In addition, by combining with OJT, skills training can be performed more effectively. As shown in Fig. 14.2, designers can use this system with a brain–machine–brain interface training system that combines AI/IoT/VR/HMI technologies. By conducting manufacturing process experiences and design reviews, it is possible to reduce manufacturing costs, improve product usability, and provide new values and services. The experience of various technologies/ skills, problem solving, idea creation, new technologies/skills and production methods, and new products and services offer unprecedented value to customers, thus creating innovations.

14.10 Brain Science Evaluation of Human Support System The authors have developed a model that displays the relationship among visual, auditory, tactile, and human behavior with the goal of scientifically elucidating human– machine interaction and improving quality of life (QOL). However, the goal is to realize a system that supports humans by recognizing human actions and emotions from IoT sensor information [6]. We are conducting research on advanced human-friendly support systems and their HMIs to realize safety and comfort for consumers. To realize human interactions in which the machine can change the form of support according to the person’s

224

K. Watanuki

situation, it is indispensable for the machine to know the person. Hence, it is necessary to estimate the psychological state and intention of the person. Therefore, cerebral blood flow dynamics and cranial nerve activity are measured noninvasively with high accuracy, brain function measurement, facial expression, and biological signals are measured simultaneously, and machine learning is performed based on such information such that sensibility and cognitive state can be estimated. We aim to establish a method to quantitatively evaluate human psychological state and intention. In the driving support environment, we are studying the support system for preventive safety by estimating the Kansei and cognitive state after elucidating human cognition, judgment, and behavioral processes. Furthermore, we have developed a device that assists the behavior of elderly people with movement disorders and are conducting research on BMI that can support behavior as desired based on brain and biological signals.

14.11 Application to Human-Friendly Automobile HMI With the spread of noninvasive brain function measurement technology, not only basic research on brain function, but also brain activity when people are thinking and acting can be measured. Kansei evaluation, which typically relies on subjective evaluation such as sensory evaluation, can now be objectively evaluated using brain function measurement. In addition, brain activation caused by motor images can be captured and devices can be operated. In the automobile driving environment, the environment and system not only require the human side to act to access the machine or environment side, but also the machine or environment side senses humans with an advanced sensor interface, and the machine or environment side works autonomously. Therefore, the authors presented easily understood information based on the driver’s cognition and behavior, evaluated the arousal level, quantitatively evaluated the ride comfort of the car [7], and controlled the air-conditioning in the passenger compartment based on the thermal comfort evaluation of the occupant. Furthermore, we are researching advanced HMI technology to realize safe, comfortable, and eco-friendly spaces and movements through technology development [8]. In addition, research and development of driving support technology that compensates for the delay in cognition and judgment owing to aging and the deterioration of operation is required such that even an elderly person can drive a car without anxiety. The authors analyzed the location information of surrounding cars, bicycles, and pedestrians in real time and predicted the next movement to inform the driver such that driving support technology can prevent pedestrians from jumping out and encountering traffic accidents can be developed. In addition, as shown in Fig. 14.3, the driver’s brain function, biological information such as pulse, image information such as facial expressions, and behavior information of driving operations are measured, and AI technology such as deep learning is used to estimate the visibility of

14 Artificial Intelligence and Virtual Reality-Based Kansei …

Visual, tactile and auditory information display, noninvasive biological information measurement, body motion information measurement

225

Visibility and fatigue analysis by deep learning

Fig. 14.3 LED system for automobile interior lighting considering sensitivity value

the LED display system and the fatigue level of the driver such that technology to realize preventive safety can be developed. Affectiva’s Automotive AI (SDK) collects facial expression data and audio data using a camera and microphone installed in the car and analyzes passenger emotions in real time. With deep learning, the world’s largest database with improved accuracy, and Affectiva’s unique algorithm, it is possible to grasp not only the driver but also the passengers’ emotions and reactions in real time. Thus, the next-generation driver monitoring system can be constructed, a self-driving vehicle developed, and a comfortable driving environment realized, thereby resulting in improved safety.

14.12 Application to Healthcare Equipment As a new healthcare industry that encompasses the fields of health, medical care, and welfare, the development of systems and related services that enable “natural” support based on human cognitive and Kansei mechanisms is important. In the life support system, humans do not adjust to machines; henceforth, a natural support system is necessitated, in which the machine reads the human mind and of the sense incongruity in each person would disappear. We have developed an advanced healthcare system using AI/IoT/VR/HMI technology, as shown in Fig. 14.4, and a walking support system that considers human characteristics [9]. To develop such a system, it is important to understand people well using IoT and AI technology. In addition, the development of healthcare products requires various knowledge, skills, and product creation from the consumer’s perspective. Additionally, it is useful to incorporate information from not only doctors but also users and their families, nurses, occupational therapists, and engineers, while sharing product information through IoT and VR technology and experiencing product usability.

226

Noninvasive biological information measurement, body motion information measurement using advanced healthcare system

K. Watanuki

Biological information and body motion analysis using deep learning

AI/IoT/VR/HMI-based walking support system

Fig. 14.4 AI/IoT/VR/HMI-based advanced healthcare analysis system

14.13 Application to Music Therapy Music has been reported to affect people’s emotions. In response to this effect, music therapy, in which music is used to relieve illness, has begun to spread in Japan in recent years. Music therapy is a therapeutic and educational technique that uses music to recover mental and emotional disorders. In Japan, where a super-aged society has already occurred and awareness of disabilities and health is gradually changing, the demand for music therapy is expected to increase. However, while such an expectation exists, the relationship between music and emotions has not been elucidated in many areas. A scientific elucidation of the relationship will enable a more efficient music therapy. Previously, many studies have elucidated the emotions experienced by people according to different music genres and famous songs based on brain function. However, these studies involve existing songs and are not applicable to music in general. By focusing on harmonies and rhythms, which are the basis of music, the fundamental knowledge of music therapy can be obtained by elucidating the emotions that people recall from the root of music. Previous studies have revealed a close relationship between emotion and the brain. We aim to elucidate the relationship between sound and emotion from the viewpoint of brain function by measuring the human brain function with NIRS when listening to various harmonies and rhythms, which are the basis of music. By measuring the brain activation response when listening to sound, brain activation has been observed in the prefrontal cortex. In addition, various responses are expected by combining harmonies and rhythms, which may result in a more efficient music therapy.

14 Artificial Intelligence and Virtual Reality-Based Kansei …

227

14.14 Conclusion In modern Japanese society, managing a super-aged society is an urgent task, and high demands exist for ensuring safety and comfort. Hence, the elucidation of human cognition, judgment, and behavior is important. A wide range of approaches, including the engineering approach to Kansei and cognition introduced herein, has become a new trend. Various technologies such as noninvasive living body information measurement, AI, IoT, VR, and HMI have been integrated, and the new Kansei cognitive engineering has become a basic technology for improving QOL; thus, new social issues can be solved.

References 1. Watanuki K (2018) Noninvasive measurement technique of brain activity and its application to Kansei design and human-machine interfaces, J Jpn Soc Des Eng 53(9):668–677 (in Japanese) 2. Picard RW (1997) Affective computing. MIT Press, Cambridge 3. Watanuki K (2011) A mixed reality-based emotional interactions and communications for manufacturing skills training. Springer, Emotional Engineering, pp 39–61 4. Hou L, Watanuki K (2012) Measurement of brain activity under virtual reality skills training using near-infrared spectroscopy. J Adv Mech Des Syst Manuf 6(1):168–178 5. Hou L, Watanuki K (2013) Measurement of brain activity during lathe operation using nearinfrared spectroscopy NIRS. Trans Jpn Soc Mech Eng Ser C 79(800):1124–1133 (in Japanese) 6. Fujimura Y, Watanuki K, Kaede K, Hou L, Kudou M, Miyake H (2016) Psychological verification based on positive and negative affect and measurement of brain activities while watching emotional film scenes. Trans Jpn Soc Mech Eng 82(842):16–00156 (in Japanese) 7. Hirayama K, Watanuki K, Kaede K (2012) Brain activation analysis of voluntary movement and passive movement using near-infrared spectroscopy. Trans Jpn Soc Mechan Eng Ser C 78(795):3803–3811 (in Japanese) 8. Kondo Y, Hou L, Watanuki K (2013) Evaluation of thermal comfort for room air conditioner using near-infrared spectroscopy (Brain activity analysis during changing room temperature). Trans Jpn Soc Mech Eng Ser C 79(807):4075–4083 (in Japanese) 9. Osawa Y, Watanuki K, Kaede K, Muramatsu K (2019) Study on gait discrimination method by deep learning for biofeedback training optimized for individuals. Adv Intell Syst Comput (Springer) 903:155–161

Chapter 15

Development of an LED Lighting System Through Evaluation of Legibility and Visual Fatigue Keiichi Muramatsu, Keiichi Watanuki, Naoya Mashiko, Yoichiro Watanabe and Masutsugu Tasaki Abstract Light-emitting diodes (LEDs) are commonly used to backlight text indicators in vehicles, among other uses. It is desirable that such text indicators are both easily legible and have low fatigue-inducing qualities. Suitable spectral distributions for text presentation in vehicles have not yet been clarified. This study examined the legibility and fatigue experienced by participants when reading sentences presented through backlight emission for 10 min in a dark room. The backlight indicators employed both colored and white LEDs. The participants were asked to answer questionnaires to evaluate the LEDs’ legibility and fatigue. Moreover, the participants’ critical fusion frequency and brain activity were measured to objectively evaluate fatigue. The questionnaire results show that all of the colored backlighting displayed an almost equal legibility. The overall results suggest that participants were most (least) fatigued when viewing text illuminated by high color-rending (greenish white backlighting). Thus, fatigue can be reduced by using LEDs with different spectral distributions.

K. Muramatsu (B) · K. Watanuki Graduate School of Science and Engineering, Saitama University, 255 Shimo-Okubo, Sakura-ku, Saitama-shi 338-8570, Japan e-mail: [email protected] K. Watanuki e-mail: [email protected] N. Mashiko · Y. Watanabe · M. Tasaki Asahi Rubber Inc., 2-7-2 Dote-cho, Omiya-ku, Saitama-shi 330-0801, Japan e-mail: [email protected] Y. Watanabe e-mail: [email protected] M. Tasaki e-mail: [email protected] © Springer Nature Switzerland AG 2020 S. Fukuda (ed.), Emotional Engineering, Vol. 8, https://doi.org/10.1007/978-3-030-38360-2_15

229

230

K. Muramatsu et al.

15.1 Introduction The light-emitting diode (LED) was invented by Nick Holonyak in 1962. Subsequently, blue LEDs were first commercialized in 1993, and white LEDs were then commercialized in 1997. Currently, basic research is being actively conducted into both LED elements and their application research and development. The luminous efficiency of LEDs has progressed, and they now exhibit higher reliability and higher efficiency than white bulbs and fluorescent lamps. The lifetime of LEDs exceeds 50,000 h, and a stable quality can be assured. As a next-generation light source that requires little maintenance, they are attracting attention in various fields. In particular, their range of practical uses centering on white light is steadily expanding. The structure of a white LED is most commonly a combination of an LED element that emits blue light (Fig. 15.1) and a phosphor that emits yellow light upon receiving the blue light. The blue light emitted from the LED element and the yellow light emitted from the phosphor are mixed, and are recognized as being white by the human eye. The element used for an LED is manufactured by growing a semiconductor crystal on a single crystal substrate. Due to the nonuniformity of the crystal growth process, slight variations in the dominant wavelength can occur. In particular, a white LED that uses a blue LED element with high light energy has a two-dimensional distribution of color variations on the chromaticity coordinates, due to variations in the amount of phosphor. The “ASA COLOR LED” was developed with the aim of providing an LED light source that can solve problems, such as color variations, and can provide stable colored light in a wide variety of roles. The structure of the ASA COLOR LED is different from that of a general white LED, as shown in Fig. 15.2. It is composed of a cap made of silicone rubber, which uniformly contains a blue LED; a phosphor is placed on top of it. Based on the optical specifications required at the product design

Fig. 15.1 Spectral power distribution of a blue LED (left-hand side), and chromaticity distribution depending both on LED dominant wavelength and phosphor amount (right-hand side)

15 Development of an LED Lighting System …

231

Fig. 15.2 Structure of the ASA COLOR LED

Fig. 15.3 A selection of the infinite range of colors that can be reproduced by the ASA COLOR LED

stage, the allowable range of the optical characteristics of the blue LED being used need to be calculated. By adjusting the type and number of phosphors according to the purpose of use, it is possible to create LEDs that provide infinite colors of light, as shown in Fig. 15.3. In fields, such as interior lighting and general lighting, there is a strong demand for lighting options that guarantee safety and comfort. The ASA COLOR LED reproduces more than 10,000 colors of light that meet these requirements, and its range of applications is constantly expanding. LED light sources are gaining attention as familiar light sources; they are both replacing conventional white light bulbs and expanding to provide new applications. As LED light sources have become more widespread in our society, a strong demand has developed for LED light sources suitable for environments in which LEDs are used, such as color rendering and blue light reduction in general lighting applications. Therefore, functional and high-quality design and manufacturing technology is required. For example, in the case of an LED light source being used for the backlight of an automobile indicator or an instrument panel, both high visibility and low fatigue induction are desirable for the driver. In this article, we will introduce research pertaining to the visibility assessment and fatigue assessment of LED light sources using physiological measurements. This research has focused on both colored [1] and white LEDs [2]. The authors used both subjective (questionnaire survey) and objective (physiological information) evaluations to examine LED’s visibility and fatigue when luminescent letters with different spectral distributions were presented.

232

K. Muramatsu et al.

15.2 Evaluation of Colored LED Lights 15.2.1 Bioinstrumentation First, we describe the bioinstrumentation used to measure fatigue and sympathetic nervous system activity. We used both critical fusion frequency (CFF) and brain activity to evaluate fatigue. The numerical value of CFF decreases when the participants are fatigued [3]. We measured CFF using a CFF measurement instrument, and used the amount of change in CFF before and after the experiments, based on the average of four measurements, as the evaluation index. Moreover, brain activity decreases when participants are fatigued [4]. Therefore, we used brain activity as another evaluation index of fatigue. We used near-infrared spectroscopy (NIRS) to measure brain function. NIRS can measure oxyhemoglobin (oxyHb), deoxyhemoglobin (deoxyHb), and total hemoglobin (totalHb) by exploiting the high biological permeability of nearinfrared light. It has been reported that there is a strong correlation between the blood oxygenation level-dependent signal in functional magnetic resonance imaging, which can measure deep brain activity, and oxyHb measured by NIRS [5]. We used the amount of change in oxyHb before and after the experiment as an evaluation index.

15.2.2 Physiological Measurements We used amylase activity, electrocardiography, respiration rate, and body temperature to assess sympathetic nervous system activity. Amylase activity increases when participants are under stress. We measured amylase activity by means of a salivary amylase monitor, and used the amount of change in amylase activity before and after the experiments as an evaluation index. The electrocardiogram includes various evaluation indexes. Through spectral analysis of the heart rate variability (HRV), the power of both the low frequency (LF) and high frequency (HF) were calculated. When individuals are under stress, the ratio of LF-to-HF increases. Therefore, we used the LF-to-HF ratio as an evaluation index. The respiration rate increases when individuals are under stress, so we also used the amount of change in respiration rate before and after the experiments as an evaluation index. We converted the time per cycle of respiration into the respiration rate. We measured respirations by determining the amount of change in the circumference of the chest. Body temperature increases when individuals are under stress. Therefore, we used the amount of change in body temperature before and after the experiment as a further evaluation index. A body temperature measurement instrument was attached to the back of the participants’ bodies for this purpose.

15 Development of an LED Lighting System …

233

15.2.3 Questionnaire We used the number of words that participants read over 1 min, and the results of the questionnaire, as evaluations of legibility. The participants’ evaluated legibility was measured on a five-point rating scale, ranging from “disagree completely” (1 point) to “agree strongly” (5 points), after the experiment. In the evaluation of fatigue, we used two questionnaires to achieve a subjective evaluation. First, we used the evaluation index for seeing burdens, taken from the “Shinso Sangyo Hiro handbook” [6]. This questionnaire consists of 15 items. The participants responded to each item using a seven-point rating scale, from “disagree completely” (1 point) to “agree strongly” (7 points). We evaluated fatigue by using the sum of the 15 items, analyzing the change in this value before and after the experiments. Second, we used “Jikaku-sho shirabe,” which was produced by the Research Group of Industrial Fatigue [7]. This questionnaire consists of 25 items that are classified into five factors, “Feeling of drowsiness,” “Feeling of instability,” “Feeling of uneasiness,” “Feeling of local pain or dullness,” and “Feeling of eyestrain.” The participants responded using a five-point rating scale, from “disagree completely” (1 point) to “agree strongly” (5 points). We evaluated fatigue by using the mean change in the points before and after the experiments. We also used CFF and amylase activity as objective evaluation indexes. We evaluated fatigue by using the mean change before and after the experiments. We used brain activity, LF-to-HF ratio, respiration rate, and body temperature as physiological indexes. Regarding bioinstrumentation, we evaluated participants’ sympathetic nervous system activity based on the mean change between their base-line values and those exhibited during the experiment.

15.2.4 Experiment Conditions and Procedure We used English sentences that were presented by backlight emission. The participants read these English sentences for 10 min, after resting for 1 min. We measured CFF, amylase, and questionnaire activity before and after the experiments. We measured brain activity, electrocardiogram, respiration rate, and body temperature during the experiments. We used six color LEDs: NSSC063AT [Blue], SM L Z14F4T [Green], SML Z14Y4T [Yellow], SML Z14D4T [Orange], SML Z14U4T [Red], and NSSC063AT + CAP [White]. In advance, we measured luminance using a spectral radiance meter; the luminance of each LED was 30 cd/m (±5%). The participants sat in a dark room, which was kept at a temperature of 25 °C, and the LED was placed on a desk at a distance of 0.60 m from the participants. The participants were ten people who were healthy and were in their twenties.

234

K. Muramatsu et al.

15.2.5 Results and Discussion The number of words that the participants read per minute and the results of the questionnaire about legibility are summarized in Figs. 15.4 and 15.5, respectively. The number of words read per minute did not differ significantly for the different colors. Based on the questionnaire about legibility, the white backlight resulted in high legibility, whereas the blue backlight led to low legibility. We assume that the white backlight resulted in high legibility because the surrounding color was black. Second, the results of the two questionnaires related to the subjective evaluation of fatigue are shown in Figs. 15.6 and 15.7. Blue and yellow backlights led to higher fatigue scores, whereas green and white backlights led to lower fatigue scores. In the Jikaku-sho shirabe, blue and yellow backlights produced a high score for “Feeling of uneasiness” and “Feeling of eyestrain,” whereas green and white backlights produced Fig. 15.4 The number of words that participants read per minute for colored LEDs

Fig. 15.5 Evaluation of legibility score for colored LEDs

15 Development of an LED Lighting System …

235

Fig. 15.6 Evaluation of fatigue for colored LEDs

Fig. 15.7 Results of Jikaku-sho shirabe for colored LEDs

low scores for these categories. Third, the results of the CFF and amylase activity are shown in Figs. 15.8 and 15.9, respectively. According to these results, yellow and white backlights led to objective indications of increased fatigue and increased sympathetic nervous system activity. In terms of the physiological indexes, the LFto-HF ratio, indicating sympathetic nervous activity, is shown in Fig. 15.10. These results indicated that green, orange, and white backlights induced sympathetic nervous system activity. These results indicate that the subjective and objective fatigue differed for the participants. We assume that the participants were affected by color impression; there is therefore a need to investigate the relationship between fatigue and color impression.

236 Fig. 15.8 Amylase activity for colored LEDs

Fig. 15.9 CFF for colored LEDs

Fig. 15.10 LF-to-HF ratio for colored LEDs

K. Muramatsu et al.

15 Development of an LED Lighting System …

237

15.3 Evaluation of White LED Lights 15.3.1 Experimental Conditions and Procedure We used Japanese sentences presented with a white backlight emission. We measured the luminance of each LED, using a spectral radiance meter, as being 30 cd/m2 (±5%). The physiological measurements and questionnaire were the same as for the colored LED lights. Ten healthy participants, who were all in their twenties participated in this study. They were asked to sit in a dark room and read Japanese sentences for 10 min, after having rested for 1 min. We measured CFF before and after the experiments and evaluated the questionnaires. In addition, we determined brain activity during the experiments. Four types of white LEDs were used for backlighting: general white (chromaticity value: 0.3299, 0.3298), high color-rending white (chromaticity value: 0.331, 0.3299), greenish white (chromaticity value: 0.3092, 0.3505), and reddish white (chromaticity value: 0.3555, 0.3091). We examined the general white backlight along with another type of white backlight on the same day, so as to take into consideration the participants’ conditions, which can change from day-to-day.

15.3.2 Results and Discussion This section describes average results of the 10 participants. The results of the questionnaire concerning legibility are summarized in Fig. 15.11. Based on the questionnaire results, all of the colored backlighting showed almost the same legibility. We assume that the chromaticity values of the backlighting were almost equal, and thus the legibility of these backlighting options were almost identical. Figures 15.12 and 15.13 display the results of the two questionnaires related to the subjective evaluation of fatigue. The two questionnaires showed that fatigue is more likely to be caused by the high color-rending white backlighting than by the general 5

5

5

4

4

4

3

3

3

2

2

2

1

General white color LED

High color rendering white color LED

1

General white color LED

Greenish white color LED

1

General white color LED

Fig. 15.11 Results of the questionnaire regarding the legibility of the white LEDs

Reddish white color LED

238

K. Muramatsu et al.

15

15

15

10

10

10

5

5

5

0

General white color LED

High color rendering white color LED

0

General white color LED

Greenish white color LED

0

General white color LED

Reddish white color LED

Fig. 15.12 Results of the questionnaire form regarding fatigue of the eyes for the white LEDs

5

5

5

4

4

4

3

3

3

2

2

2

1

1

1

0

General white color LED

High color rendering white color LED

0

General white color LED

Greenish white color LED

0

General white color LED

Reddish white color LED

Fig. 15.13 Results of “Feeling of eyestrain” for the white LEDs

white backlighting. In addition, fatigue is not more likely to be caused by greenish and reddish white backlighting, compared with the general white backlighting. Figures 15.14 and 15.15 show the results of the CFF and brain activity analysis, respectively. Similar to the previous results, the CFF values obtained through these results are more likely to be caused by high color-rending white backlighting than by general white backlighting. In addition, CFF is not more likely to be caused by greenish white than by general white backlighting. However, reddish white backlighting showed an almost equal CFF tendency to that of general white backlighting. Regarding brain activity, the high color-rending and reddish white backlighting seemed to decrease brain activity more than the general white backlighting, and thus caused more fatigue. In contrast, the greenish white backlighting increased brain activity more than the general white backlighting, and thus did not result in fatigue. These results show that fatigue is more likely to be caused by high color-rending and reddish white backlighting than by general white backlighting; however, it is not more likely to occur because of greenish white backlighting, compared with general white backlighting. The overall results suggested that participants were the most (least) fatigued when viewing text indicators illuminated by high color-rending (greenish white backlighting), respectively. Therefore, we suggest that fatigue can be reduced by using LEDs with different spectral distributions. We assume that the spectral distribution of an

239

0.0

0.0

-0.5

-0.5

Critical fusion frequency [Hz]

Critical fusion frequency [Hz]

15 Development of an LED Lighting System …

-1.0 -1.5 -2.0 -2.5 -3.0

General white color LED

High color rendering white color LED

-1.0 -1.5 -2.0 * : p