Effect of Display Response Time on Brain Activity in Human–Machine Interface Commander Operation 9781665417143

Citation preview

2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) September 27 - October 1, 2021. Prague, Czech Republic

2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) | 978-1-6654-1714-3/21/$31.00 ©2021 IEEE | DOI: 10.1109/IROS51168.2021.9636452

Effect of Display Response Time on Brain Activity in Human–Machine Interface Commander Operation Kentaro Oshima, Toru Tsumugiwa, Member, IEEE, Ryuichi Yokogawa, Member, IEEE, Mitsuhiro Narusue, Hiroto Nishimura, Yusaku Takeda, and Toshihiro Hara Abstract—With the recent diversification of operating devices, the demand for input operations that require confirmation of the effect of differences in display response on operability has increased. Regarding display response, previous studies have investigated the threshold time and sense of agency for a delayed response during device operation. However, these studies only focused on subjective evaluations. Therefore, this study aims to clarify the human motor characteristics and activated brain regions based on the differences in display response time during device operation. The target motion is the rotational operation of the cylindrical rotary controller using the index finger and thumb. The experimental conditions involve four types of display response times (the duration from the operation to the indicated response). We measured the brain activity using near-infrared spectroscopy, the muscle activity from a surface myoelectric potential measurement device, and the force data of the index finger and thumb tip obtained from two independent six-axis force/torque sensors. Although the experimental results showed no significant difference in the muscle activity and gripping force, a significant difference was observed in the brain activity and the questionnaire survey by the difference in display response time. This investigation reveals that the difference in display response time affects brain activity and subjective information, clarifying the relationship between brain activity and subjective information. I. INTRODUCTION Recently, human–machine interface (HMI) devices, such as touch screens and video presentation devices that transmit information between humans and machines owing to the development of virtual reality technology, have become more complex and diversified. Thus, the development of HMI devices that can be easily operated by humans is required. Furthermore, the number of input operations that require display response confirmation of the operations is increasing. Various studies have focused on the display responses of devices. For example, during device operations in This work was partially supported by JSPS KAKENHI Grant-in-Aid for Scientific Research (C) Grant Number 20K04388, 2020 and 21K03973, 2021. K. Oshima is with the Graduate School of Life and Medical Sciences, Doshisha University, Kyoto, Japan. T. Tsumugiwa is with the Department of Biomedical Engineering, Faculty of Life and Medical Sciences, Doshisha University, Kyoto, Japan (e-mail: [email protected]). R. Yokogawa is with the Department of Biomedical Engineering, Faculty of Life and Medical Sciences, Doshisha University, Kyoto, Japan (e-mail: [email protected]). M. Narusue is with the Technical Research Center, Mazda Motor Corporation, Hiroshima, Japan. H. Nishimura is with the Technical Research Center, Mazda Motor Corporation, Hiroshima, Japan. Y. Takeda is with the Technical Research Center, Mazda Motor Corporation, Hiroshima, Japan. T. Hara is with the Technical Research Center, Mazda Motor Corporation, Hiroshima, Japan.

978-1-6654-1714-3/21/$31.00 ©2021 IEEE

environments that assume an augmented reality or virtual reality space, delays in display responses lead to discomfort in humans, reducing immersiveness [1,2]. Therefore, a lower display delay response characteristic is required during operation in a highly immersive environment. Numerous studies have also investigated the sense of agency (SoA), which is the sense that “I am the one who is causing or generating an action” [3] while operating the device [4]. These studies have reported that the SoA is reduced in response without delay [5,6]. Other studies have investigated the delay response that humans can perceive [7]. In experiments concerning delay recognition of self-body images, a response delay of 230 ms or more is recognizable [8]; in touch screen operation, humans can recognize a delay of 10 ms or less [9]. According to the results of these studies, the delay recognition and SoA to the response of the device differ based on the operation target and environment; a lower delayed response to human operation sensation is not optimal. Previous studies have also investigated brain activity, SoA, and delayed recognition [10,11]. However, several human operation sensations to the response of device operation have been qualitatively evaluated through subjective information, and few studies have examined quantitative evaluations based on objective indicators. In this study, we evaluated the effect of the difference in display response time during device operation on human subjective information and brain activity. Evaluating brain activity during exercise is a useful quantitative evaluation method because exercises, such as device operation, are performed by processing sensory information in the brain and transmitting commands to the muscles [12]. We quantitatively evaluated the human operation sensation in response to device operation by analyzing the activity state of the cerebral cortex, subjective information on operability by a questionnaire, muscle activity, and gripping force. The purpose of this study was to clarify human motor characteristics and brain region to be activated, resulting from the difference in display response time. We aimed to elucidate human perceptual characteristics and establish device design indicators related to operational sensations; therefore, quantitative evaluation of operational sensations is useful as basic information in device development. As the target of the operation device, we focus on the grip rotation operation by the index finger and thumb of the cylindrical rotary commander used to adjust the temperature and volume in the car and select the menu of the car navigation display. Previous studies have evaluated the steering operation [13] and invehicle HMI using a driving simulator [14,15]. However, studies regarding the response to car navigation displays have not been conducted. In-vehicle devices, such as car navigation systems, require complicated input operations because of the

6204

Authorized licensed use limited to: Zhejiang University. Downloaded on October 06,2023 at 05:41:43 UTC from IEEE Xplore. Restrictions apply.

increasing functions, and commander operations during driving impose a heavy load on the driver. Therefore, confirming the display response for intuitive operation is necessary, and the design of an appropriate display response time is required. In this study, we focus on the display response when operating an in-vehicle HMI commander. We used near-infrared spectroscopy (NIRS), which is a noninvasive method for measuring brain activity. Its usefulness has been suggested by real-time measurements of brain activity and noise tolerance [16]. Muscle activity by surface myoelectric potential measurement and gripping force was measured using a six-axis force sensor. Operability was subjectively evaluated using a questionnaire using the semantic differential method [17]. We evaluated the brain region activated for the difference in display response by comparing the amount of change in oxygen–hemoglobin (oxy-Hb) concentration under each condition and analyzing the brain activation area to analyze time-series changes. II. EXPERIMENTAL OUTLINE Twelve healthy right-handed men, with a mean age of 21.8 (standard deviation [SD]: 0.83) years, participated in this study. The target motion was a rotational operation of the cylindrical rotary commander, in which the index finger and thumb were used for operating a car navigation system (Fig. 1). The experiment was conducted by changing the display response time to respond to the operation. Moreover, for the environment of the experiment, we assumed an actual driving position because the study targets the commander mounted on the automobile. The subjects placed their right arm from the elbow to the wrist on the armrest while seated, imitating the position of the driver in an automobile. They performed a commander rotation operation with two fingers (index finger and thumb). The armrest was placed in a position where the index finger and thumb made a horizontal plane when the subject grasped the commander and rotated it. The commander’s diameter was 50 mm, and the distance between the force sensor and commander was 15 mm. Two independent force sensors were used to measure the gripping forces of the index finger and thumb. The display was placed in front of the seated subject, and the lighting position of the

LED

Probe

Display

NIRS Probe

Display response

1 2 3 4 Two force sensors

light-emitting diode (LED) marker on the display was set to move up and down when the commander was rotated. The commander produced a click sensation every 12° in the rotation direction and one click in a clockwise direction (initial rotation direction), moving the LED marker downward by one square; and on the other hand, the commander moved the LED marker upward during a counterclockwise operation. The click angle was set according to the off-the-shelf products of Mazda Motor Corporation, which was used as the actual HMI commander. The subjects were asked to operate the commander at a tempo of 100 beats per minute (BPM) (clicks per minute) while gazing at the display and were asked to repeat the operation of moving the LED marker lighting from positions 1 to 4 in Fig. 1 (the operation of rotating the commander by three clicks in each rotation direction). The experimental conditions involved four types of display response time (the duration from the operation to the indication response): Condition 1 had no display response delay, and Conditions 2, 3, and 4 had a display response time of 50, 100, and 150 ms, respectively. The maximum device intrinsic delay in this experiment was approximately 1.392 ms. Therefore, the actual delay time for each condition was added to a maximum of approximately 1.392 ms. Because the intrinsic delay time was short, we assumed that it did not have any effect on human response time and operation in this experiment. The condition order was random, and the order effect was eliminated. The experimental time was 30 s for task execution time, 5 s for task start time (pre-time), 20 s for recovery time after task completion (recovery time), 5 s for post-recovery time (post-time), and 30 s for pre-time, post-time, and recovery time (rest time). The sum of the task and rest time was set as one trial. The subjects performed three consecutive trials of one set (a total of 180 s) in each condition. The subjects grasped the commander with the index finger and thumb simultaneously to signal the start of the task time; they operated the commander while gazing at the LED marker on the display. All instructions regarding the experiment were provided verbally. To prevent differences in the subject’s proficiency level in commander operation from affecting brain and muscle activity, the subjects practiced commander operation in advance using the same equipment one week before the experiment.

NIRS

HMI rotary commander

4

3 Display

2 Attachment

φ50

1

Fig. 1 Schematic diagram of the experiment. The subjects operated the commander with two fingers (index finger and thumb) while gazing at the LED marker on display and repeated the operation of moving the LED marker lighting from positions 1 to 4.

6205 Authorized licensed use limited to: Zhejiang University. Downloaded on October 06,2023 at 05:41:43 UTC from IEEE Xplore. Restrictions apply.

Muscle activity was evaluated using a surface myoelectric potential measurement device. Three measurement sites affect the rotation operation: first dorsal interosseous (FDI), extensor digitorum (EDC), and flexor digitorum superficialis (FDS). Brain activity was measured using NIRS (ETG-7100; Hitachi Medical Corporation, Tokyo, Japan). Fig. 2 shows the positions of the brain function area and measurement Ch. The NIRS measurement probe was attached based on the international 10–20 system [18], which is widely used for site identification. In the experiment, to ensure that the subjects were not affected by the external environment, we paid attention to noise, smells, and sights in the room. We also investigated the threshold of display response time that humans can recognize to identify the relationship between the difference in display response delay and brain activity. The subjects randomly performed the operation at each condition five times (20 times in total). We instructed them to verbally answer whether a delay was present. Similar to the above experiment, the speed was 100 BPM, and the subjects were instructed to operate three clicks in each rotation direction. Nose Left ear

1

Fp1

5

2 6

10 10

Probe5

14

3 7

11 15

19

12 16

20

3

14 10

6 2

17 13

9 5

1

4 24

16

3 23

15 15 11

12

2 22 1

C3

7

3

B. Surface Myoelectric Potential Signal To remove the noise of the surface myoelectric potential signals, we applied a 50-Hz Butterworth filter [26] to the fast Fourier transform (FFT) analysis results. We then performed root mean square (RMS) processing to represent the average amplitude of the surface myoelectric signals. In the RMS process, the surface myoelectric potential was squared at a certain time period, the average value within that time range was calculated, and the square root was evaluated. Furthermore, the average value of the data from the maximum value to the 10th among the task time data during all trials in each subject and each muscle was set to 100% because the amplitude of the surface myoelectric potential differed between subjects. The task time data were standardized as the %RMS data. Tukey’s test was used to compare muscle activity between experimental conditions at the 0.05 significance level (null hypothesis: the average values of muscle activities between different conditions are equal). The task time of 30 s was divided into 10-s intervals, and Tukey’s test was used in the same method for %RMS in each time zone.

16 20 20

15 15

24 19

4 8

12

23 20

10

11 15

19

16

5

Probe2

18 22

21

11 11 6

18

22 17

7

19

Probe4

18 22

18 13

8

20

12 8

4

21

17

Right ear

Fp2

9 13 13

21

9 7

4 8

9

17 17 21

1 5

13

2 6

10 14

3 7

14

Cz

14

10

21 17

C4

Probe1

24

Prefrontal (PFC) 6 cortex 13 20 2 9 16 23 Premotor area (PMA) 5

12

19

1 8 15 Primary motor cortex22(PNC) 4

11

measurement data were standardized based on the baseline for such problems [20]. In these studies, standardization was performed for all NIRS measurement data measured at each time point. The data were tested at a significance level of 0.05, to identify the areas of significant cortical activity. A t-test was performed for the brain activity in each condition (null hypothesis: the average values of oxy-Hb concentration during rest time and task time were equal). The Tukey’s test was used to compare brain activity between experimental conditions (null hypothesis: the average values of oxy-Hb concentration of task time between each condition were equal). We also calculated the brain activation area value by focusing on specific brain regions related to the experimental task and exercise. The brain regions evaluated in the analysis of brain activation area were the prefrontal cortex, which has the function of making appropriate decisions and attention [21,22], premotor area, which is responsible for the composition and execution of exercise [23], primary motor cortex, which has the function to output exercise [24], and primary somatosensory cortex, which processes sensory information [25]. Brain activation area values were calculated based on topography imagery using a t-test. To analyze the time-series changes in brain activity in each brain region, we evaluated the changes in brain activity over time during the task by dividing the 30-s task time into 5-s intervals.

18

Primary somatosensory cortex (PSC) Landmark of the 10–20 system Fig. 2 Location for detecting brain activation under the international 10–20 system.

III. ANALYSIS METHOD A. NIRS Data In this study, we used NIRS to measure the brain activity. Previous studies have considered oxy-Hb to be the most reflected brain activity in NIRS [19]. Therefore, in our study, the analysis was performed using the amount of change in oxy-Hb concentration. In addition, an integral analysis was performed by setting an approximate straight line connecting the pre- and post-time measurement data as the baseline for analysis and performing arithmetic averages. The data obtained from NIRS measurements were obtained by multiplying the amount of change in Hb concentration and the optical path length. However, the optical path length differed based on the measurement of Ch and the subject. Therefore, brain activity cannot be directly compared between the subjects and parts. In previous studies concerning NIRS, the

C. Force Data The gripping force of the index finger and thumb was obtained using two independent force sensors during the task time. The gripping force is the sum of the absolute values of the forces applied to the commander in the normal vector direction, whereas the commander is gripped by the index finger and thumb. The Tukey’s test was used to compare the gripping force between experimental conditions at a significance level of 0.05 (null hypothesis: the average value of the gripping force between conditions is equal). The task time of 30 s was divided into 10-s intervals, and Tukey’s test was used in the same method for the gripping force in each time zone.

6206 Authorized licensed use limited to: Zhejiang University. Downloaded on October 06,2023 at 05:41:43 UTC from IEEE Xplore. Restrictions apply.

D. Questionnaire To investigate the effect of the difference in display response time on the operational sensation, we conducted a questionnaire evaluation using seven scales after each set using the semantic differential method. The survey items were “ease of aligning the LED marker lighting position” and “the degree of operating naturally.” Tukey’s test was used to compare the questionnaire results between experimental conditions at the 0.05 significance level (null hypothesis: the average value of the questionnaire results between conditions is equal). Furthermore, the mean value of the delay recognition rate was compared between the conditions using Tukey’s test (null hypothesis: the mean value of the delay recognition rate under each condition was equal). IV. EXPERIMENTAL RESULTS AND CONSIDERATION A. Questionnaire Fig. 3 shows the questionnaire results. According to the average value of each condition for the “ease of aligning the LED marker lighting position” and “degree of operating naturally,” the average value of Condition 2 was the highest, and the average value of Condition 4 was the lowest. Comparing the conditions showed that a significant difference existed in the comparison between Conditions 2 and 4 in the “ease of aligning the LED marker lighting position.” Previous studies that investigated SoA for differences in response time reported a reduction in SoA for non-delayed response [5,6]. Therefore, a similar tendency was confirmed for Condition 2, which has a response time of 50 ms, showing a higher score than Condition 1, which has no delay. The reason for the high score for Condition 2 was not clarified, and will be a future issue. : Condition 1 4

: Condition 2

: Condition 3 : Condition 4 T : S.D. * : p < 0.05

*

Score [–]

3 2

1 0

Ease of aligning the LED marker lighting position

between 50 ms and 100 ms. In addition, when comparing this experiment with the previous studies that investigated the delay recognition rate [7,9], we confirmed that the threshold time differs depending on the experimental environment and the operation target. TABLE Ⅰ. DELAY RECOGNITION RATE RESULTS. Condition 1 [%] Condition 2 [%] Condition 3 [%] Condition 4 [%] 5.0 10.0 51.7 100.0 (SD: 12.4) (SD: 15.6) (SD: 23.3) (SD: 0.0)

B. Surface Myoelectric Potential Signal Table 2 shows the p-values of Tukey’s test for each condition in each muscle. As a result of testing under each condition, no significant difference was observed between the conditions in any of the muscles. To confirm the significant difference between the conditions over time, we divided the 30-s task time into 10 s and performed the same test in each time zone. No significant differences were observed. Therefore, this analysis revealed no significant differences in muscle activity in the HMI rotary commander operation because of the differences in the display response time. TABLE Ⅱ. THE P-VALUE OF TUKEY’S TESTS OF SURFACE MYOELECTRIC POTENTIAL SIGNALS. Condition FDI EDC FDS

1–2 0.94 0.99 0.94

1–3 0.87 0.99 0.94

1–4 0.99 0.99 0.91

2–3 0.99 0.99 0.99

2–4 0.98 0.99 0.99

3–4 0.93 0.99 0.99

C. Force Data Table 3 shows the p-value of Tukey’s test for each condition in the gripping force obtained from the force sensors. As a result of testing in each condition, no significant difference was observed between the conditions in the gripping force. To confirm the significant difference between the conditions over time, we divided the 30-s task time into 10 s and performed the same test in each time zone. No significant differences were found. Therefore, this analysis revealed no significant differences in the gripping force in the HMI rotary commander operation because of the display response time differences. TABLE Ⅲ. THE P-VALUE OF TUKEY’S TESTS OF GRIPPING FORCE. Condition

Degree of operating naturally

Fig. 3 Questionnaire results.

Gripping force

Table 1 shows the results of the investigation of the delay recognition rate for the display response. In the experiment, the delay recognition rate was 100% for Condition 4, confirming that the display delay of 150 ms was a response time recognized by all subjects during the commander operation. As a result of the comparison between conditions, a significant difference was noted in all combinations, except for the comparison between Conditions 1 and 2. For Condition 1, the delay recognition rate was 5.0%, confirming that some subjects observed a delay in response without delay. In a previous study, a response with a delay recognition rate of >50% was set as the threshold time for delay recognition [8]. In our study, the delay recognition rate of Condition 2 was 10.0% and that of Condition 3 was 51.7%, confirming that the delay threshold time that the subject could recognize was

1–2 0.99

1–3 0.91

1–4 0.99

2–3 0.84

2–4 0.99

3–4 0.98

D. NIRS Data Fig. 4 shows a brain activation image created using oxyHb as an index of brain activity status. The colors and numerical values of the color bars shown in the lower part of the figure correspond to the t-value of the t-test performed to create the brain activation image. When the brain was activated during the task time, the amount of change in the oxy-Hb concentration increased compared with that during rest time because of an increase in cerebral blood flow. The tvalue was positive, and the red side is shown on the color bar. As a result of the analysis, Ch showing significant activity (activated Ch) was confirmed under all conditions, as shown in Fig. 4. The number of activated Ch on the left-side brain was greater than that on the right side. Previous studies have reported that when a subject exercises with the right hand, the

6207 Authorized licensed use limited to: Zhejiang University. Downloaded on October 06,2023 at 05:41:43 UTC from IEEE Xplore. Restrictions apply.

contralateral side of the left brain is activated [27]. Therefore, in this experiment, the left brain was activated because of commander operation using the right hand.

(a) Condition 1

-4

-3

(b) Condition 2

(c) Condition 3 -1 -2 0 -2.201

1

2

(d) Condition 4 3 4 Activated Ch

2.201

t-value

Activated

Inactivated

Fig. 4 The topography imagery of oxy-Hb.

1) Prefrontal cortex: Activated Ch was confirmed in the prefrontal cortex under conditions other than Condition 2. According to the results of the two questionnaires, the prefrontal cortex’s activity probably decreased under the conditions of high scores of “ease of aligning the LED marker lighting position” and “degree of operating naturally.” In the dorsolateral prefrontal cortex, significant activation was observed in Conditions 3 and 4 compared to those in Conditions 1 and 2. The prefrontal cortex has a working memory function, and the dorsolateral part is particularly involved in memory and concentration [21], which indicates that the subject was concentrating on and paying attention to the display when the commander was operated with a display response time of 100 ms (threshold time for delay recognition) or more. The experimental results of investigation of the delay recognition rate showed that the display response time increased with the increase in delay recognition rate, suggesting that the discrimination of the display delay corresponded to the prefrontal cortex activity. Fig. 5 shows the time-series change graph in the activated area of the prefrontal cortex. Under Condition 4, the activated area tended to decrease over time. Previous studies have reported that prefrontal cortex activity declines during more practical tasks [21] and is activated during complicated tasks. : Condition 2

: Condition 3

2) Premotor area: In the premotor area, activated Ch was confirmed under conditions other than Condition 2, similar to the prefrontal cortex, suggesting that the premotor area’s activity decreased when the score of the questionnaire was high. As a result of comparing the conditions, the premotor area’s activity in Condition 1 was significantly activated compared to other conditions; it was more significantly activated in Condition 3 than in Condition 2. The premotor area is considered important in the preparatory stage of exercise and is particularly involved in the execution and planning of exercise. In addition, a function exists to select an appropriate motion by associating sensory information with motion and linking motion information, such as how to move a hand with a target movement [23]. A significant difference was found in the comparison between Conditions 1 and 2, suggesting that the difference in display response time, in which no significant difference was found in the subjective information based on the questionnaire survey, can be evaluated by the activity of the premotor area. Fig. 6 shows the time-series change graph in the activated area of the left premotor area. In Conditions 1 and 4, the brain activity was continuously activated during the task time, the activation area increased over time in Condition 3, and activation was not confirmed in Condition 2. Based on the questionnaire results, the left premotor area was considered to be activated when the operability expressed as subjective information was low. Furthermore, Condition 1 without display delay showed the same tendency as Condition 4, in which the subject felt a delay; thus, we concluded that the response without delay reduces the ease of operation. This result confirms the same tendency, that is, the response without delay reduces the SoA, as in previous studies [5,6]. Therefore, from the function of the premotor area, the two questionnaire results, and the result of the delay recognition rate for the display response time, we observed that the activity of the left premotor area reflects the human operation sense for the difference in the display response time. It is suggested that when the activation area of the left premotor area is large, the subject finds it difficult to operate the commander.

: Condition 4

: Condition 1 100 Activation area ratio [%]

Activation area ratio [%]

: Condition 1 50

Therefore, in this experiment, the tendency of deactivation was confirmed over time under Condition 4, which had the longest response time. In commander operations with display delay, complicated operations are assumed to be concentrated at the start of the operation. In contrast, it was suggested that the practice of the task reduced dependence on the prefrontal cortex, and the activated area decreased over time.

40 30 20

10 0 0

10

20

30

: Condition 2

: Condition 3

: Condition 4

80

60 40 20 0 0

10

20

30

Time [s]

Time [s]

Fig. 5 Activation area ratio in the prefrontal cortex in time-series change.

Fig. 6 Activation area ratio in the left premotor area in time-series change.

6208 Authorized licensed use limited to: Zhejiang University. Downloaded on October 06,2023 at 05:41:43 UTC from IEEE Xplore. Restrictions apply.

3) Primary motor cortex: Fig. 4 shows that in the primary motor cortex, activated Ch was confirmed under all conditions. The primary motor cortex functions in limb motion perception and motion execution [24,28]. These functions suggest that the activation of the primary motor cortex is due to the finger exercise in the commander operation under all conditions. As a result of the comparison between conditions, a significant difference was observed between Conditions 1 and 2. However, a significant difference was not found in the comparison of subjective information based on the questionnaire survey. Therefore, it is suggested that the difference in display response time, for which no significant difference was observed in the subjective information, can be evaluated by the activity of the primary motor cortex as in the premotor area.

[10] [11] [12] [13]

[14] [15]

V. CONCLUSIONS In this study, we evaluated the effect of display response time on brain activity and subjective information in commander operation and its relationship with the display response time. The experimental results revealed no significant difference in muscle activity and gripping force; however, a significant difference was observed in the brain activity and the questionnaire survey with the difference in display response time during commander operation. In addition, the activity of the left premotor area clearly reflected human subjective information regarding the difference in display response time during commander operation.

This work was partially supported by JSPS KAKENHI Grant-in-Aid for Scientific Research (C) Grant Number 20K04388, 2020 and 21K03973, 2021.

[3] [4]

[5]

[6] [7]

[8] [9]

[18]

[20] [21]

REFERENCES

[2]

[17]

[19]

ACKNOWLEDGMENT

[1]

[16]

R. S. Allison, L. R. Harris, M. Jenkin, U. Jasiobedzka, and J. E. Zacher, “Tolerance of temporal delay in virtual environments,” Proceedings IEEE Virtual Reality, 2001, pp. 247–254. J. D. Moss, J. Austin, J. Salley, J. Coats, K. Williams, and E. R. Muth, “The effects of display delay on simulator sickness,” Displays, vol. 32, no. 4, 2011, pp. 159–168. S. Gallagher, “Philosophical conceptions of the self: implications for cognitive science,” Trends in Cognitive Sciences, vol. 4, no. 1, 2000, pp. 14–21. M. Osumi, S. Nobusako, T. Zama, N. Yokotani, S. Shimada, T. Maeda, and S. Morioka, “The relationship and difference between delay detection ability and judgment of sense of agency,” PLoS ONE, vol. 14, no. 7, 2019, doi: 10.1371/journal. pone.0219222. T. Maeda, K. Takahata, T. Muramatsu, T. Okimura, A. Koreki, S. Iwashita, M. Mimura, and M. Kato, “Reduced sense of agency in chronic schizophrenia with predominant negative symptoms,” Psychiatry Research, vol. 209, no. 3, 2013, pp. 386–392. C. Stetson, X. Cui, P. R. Montague, and D. M. Eagleman, “Motor– sensory recalibration leads to an illusory reversal of action and sensation,” Neuron, vol. 51, no. 5, 2006, pp. 651–659. S. Nobusako, R. Ishibashi, Y. Takamura, E. Oda, Y. Tanigashira, M. Kouno, T. Tominaga, Y. Ishibashi, H. Okuno, K. Nobusako, T. Zama, M. Osumi, S. Shimada, and S. Morioka, “Distortion of visuo–motor temporal integration in apraxia: evidence from delayed visual feedback detection tasks and voxel–based lesion–symptom mapping,” Frontiers in Neurology, vol. 9, no. 709, 2018, doi: 10.3389/fneur.2018.00709. S. Shimada, Y. Qi, and K. Hiraki, “Detection of visual feedback delay in active and passive self–body movements,” Experimental Brain Research, vol. 201, no. 2, 2010, pp. 359–364. A. Ng, J. Lepinski, D. Wigdor, S. Sanders, and P. Dietz, “Designing for low–latency direct–touch input,” Proceedings of the 25th Annual ACM

[22] [23]

[24]

[25] [26]

[27] [28]

Symposium on User Interface Software and Technology, 2012, pp. 453– 464. M. Tsakiris, M. R. Longo, and P. Haggard, “Having a body versus moving your body: neural signatures of agency and body–ownership,” Neuropsychologia, vol. 48, no. 9, 2010, pp. 2740–2749. K. Toida, K. Ueno, and S. Shimada, “Neural basis of the time window for subjective motor–auditory integration,” Frontiers in Human Neuroscience, vol. 9, no. 688, 2015, doi: 10.3389/fnhum.2015.00688. N. R. Carlson, “Physiology of behavior,” Pearson, 11th ed. London, United Kingdom, 2013, pp. 235–287. T. Sakajiri, Y. Tanaka, and A. Sano, “Relation between gravitational and arm–movement direction in the mechanism of perception in bimanual steering,” Experimental Brain Research, vol. 231, no. 2, 2013, pp. 129–138. J. Ma, Z. Gong, J. Tan, Q. Zhang, and Y. Zuo, “Assessing the driving distraction effect of vehicle HMI displays using data mining techniques,” Transportarion Research Part F, vol. 69, 2020, pp. 235–250. N. T. Richardson, C. Lehmer, and M. Lienkamp, “Conceptual design and evaluation of a human machine interface for highly automated truck driving,” IEEE Intelligent Vehicles Symposium (IV), 2018, pp. 2072– 2077. H. Koizumi, A. Maki, T. Yamamoto, H. Sato, Y. Yamamoto, and H. Kawaguchi, “Non–invasive brain–function imaging by optical topography,” TrAC Trends in Analytical Chemistry, vol. 24, no. 2, 2005, pp. 147–156. D. Welch, D. Shepherd, K. Dirks, M. Y. Tan, and G. Coad, “Use of creative writing to develop a semantic differential tool for assessing soundscapes,” Frontiers in Psychology, vol. 9, no. 2698, 2019, doi: 10.3389/fpsyg.2018.02698. M. Okamoto, H. Dan, K. Sakamoto, K. Takeo, K. Shimizu, S. Kohno, I. Oda, S. Isobe, T. Suzuki, K. Kohyama, and I. Dan, “Three– dimensional probabilistic anatomical cranio–cerebral correlation via the international 10–20 system oriented for transcranial functional brain mapping,” NeuroImage, vol. 21, no. 1, 2004, pp. 99–111. Y. Hoshi, N. Kobayashi, and M. Tamura, “Interpretation of near– infrared spectroscopy signals: a study with a newly developed perfused rat brain model,” Journal of Applied Physiology, vol. 90, no. 5, 2001, pp. 1657–1662. G. Matsuda, and K. Hiraki, “Sustained decrease in oxygenated hemoglobin during video games in the dorsal prefrontal cortex: A NIRS study of children,” NeuroImage, vol. 29, no. 3, 2006, pp. 706–711. E. K. Miller, and J. D. Cohen, “An integrative theory of prefrontal cortex function,” Annual Review of Neuroscience, vol. 24, 2001, pp. 167–202. C. E. Curtis, and M. D’Esposito, “Persistent activity in the prefrontal cortex during working memory,” Trends in Cognitive Sciences, vol. 7, no. 9, 2003, pp. 415–423. P. Cisek, D. J. Crammond, and J. F. Kalaska, “Neural activity in primary motor and dorsal premotor cortex in reaching tasks with the contralateral versus ipsilateral arm,” Journal of Neurophysiology, vol. 89, no. 2, 2003, pp. 922–942. C. S. R. Li, C. Padoa–Schioppa, and E. Bizzi, “Neuronal correlates of motor performance and motor learning in the primary motor cortex of monkeys adapting to an external force field,” Neuron, vol. 30, no. 2, 2001, pp. 593–607. T. Umeda, T. Isa, and Y. Nishimura, “The somatosensory cortex receives information about motor output,” Science Advance, vol. 5, no. 7, 2019, doi: 10.1126/sciadv.aaw5388. H. C. Dörge, T. B. Andersen, H. Sørensen, E. B. Simonsen, H. Aagaard, P. Dyhre-Poulsen, and K. Klausen, “EMG activity of the iliopsoas muscle and leg kinetics during the soccer place kick,” Scandinavian Journal of Medicine and Science in Sports, vol. 9, no. 4, 1999, pp. 195– 200. A. Li, F. Z. Yetkin, R. Cox, and V. M. Haughton, “Ipsilateral hemisphere activation during motor and sensory tasks,” American Journal of Neuroradiology, vol. 17, no. 4, 1996, pp. 651–655. J. N. Sanes, and J. P. Donoghue, “Plasticity and primary motor cortex,” Annual Review of Neuroscience, vol. 23, no. 1, 2000, pp. 393–415.

6209 Authorized licensed use limited to: Zhejiang University. Downloaded on October 06,2023 at 05:41:43 UTC from IEEE Xplore. Restrictions apply.