Note Taking Activities in E-Learning Environments (Behaviormetrics: Quantitative Approaches to Human Behavior, 11) 9811661030, 9789811661037

Citation preview

Behaviormetrics: Quantitative Approaches to Human Behavior 11

Minoru Nakayama   Editor

Note Taking Activities in E-Learning Environments

Behaviormetrics: Quantitative Approaches to Human Behavior Volume 11

Series Editor Akinori Okada, Professor Emeritus, Rikkyo University, Tokyo, Japan

This series covers in their entirety the elements of behaviormetrics, a term that encompasses all quantitative approaches of research to disclose and understand human behavior in the broadest sense. The term includes the concept, theory, model, algorithm, method, and application of quantitative approaches from theoretical or conceptual studies to empirical or practical application studies to comprehend human behavior. The Behaviormetrics series deals with a wide range of topics of data analysis and of developing new models, algorithms, and methods to analyze these data. The characteristics featured in the series have four aspects. The first is the variety of the methods utilized in data analysis and a newly developed method that includes not only standard or general statistical methods or psychometric methods traditionally used in data analysis, but also includes cluster analysis, multidimensional scaling, machine learning, corresponding analysis, biplot, network analysis and graph theory, conjoint measurement, biclustering, visualization, and data and web mining. The second aspect is the variety of types of data including ranking, categorical, preference, functional, angle, contextual, nominal, multi-mode multi-way, contextual, continuous, discrete, high-dimensional, and sparse data. The third comprises the varied procedures by which the data are collected: by survey, experiment, sensor devices, and purchase records, and other means. The fourth aspect of the Behaviormetrics series is the diversity of fields from which the data are derived, including marketing and consumer behavior, sociology, psychology, education, archaeology, medicine, economics, political and policy science, cognitive science, public administration, pharmacy, engineering, urban planning, agriculture and forestry science, and brain science. In essence, the purpose of this series is to describe the new horizons opening up in behaviormetrics — approaches to understanding and disclosing human behaviors both in the analyses of diverse data by a wide range of methods and in the development of new methods to analyze these data. Editor in Chief Akinori Okada (Rikkyo University) Managing Editors Daniel Baier (University of Bayreuth) Giuseppe Bove (Roma Tre University) Takahiro Hoshino (Keio University)

More information about this series at http://www.springer.com/series/16001

Minoru Nakayama Editor

Note Taking Activities in E-Learning Environments

Editor Professor Minoru Nakayama Department of Information and Communications Engineering Tokyo Institute of Technology Tokyo, Japan

ISSN 2524-4027 ISSN 2524-4035 (electronic) Behaviormetrics: Quantitative Approaches to Human Behavior ISBN 978-981-16-6103-7 ISBN 978-981-16-6104-4 (eBook) https://doi.org/10.1007/978-981-16-6104-4 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore

Preface

Recent innovations have made distance learning one of the most widely used teaching methodologies used in training and educational programmes. Some forms of distance learning based on information technology and enhanced communications are known as e-learning. More advanced forms called MOOCs (Massive Open Online Courses), which are supported by recent innovations in information and communications technologies, and can manage very large numbers of online participants. MOOCs are used worldwide by numerous institutions of higher education. The history of e-learning is brief, but the use of MOOCs is becoming more widely accepted in business and educational sectors. The current pandemic has necessitated greater participation in online learning, which has therefore made consideration of e-learning best practices even more important to course leaders and developers. Although online learning is convenient for everyone, its effectiveness was challenged during the early stages of evolution. Sometimes the problem was reported to be hesitation of participants to join online courses, boring course materials or quitting courses before they were completed. In addition, support or encouragement by organisers is problematic when many participants study remotely, and thus mutual support by participants is difficult to establish. Co-authors, Dr. Hiroh Yamamoto, Professor Emeritus at Shinshu University, and I surveyed participant’s learning activity and studied the methodologies for supporting students based on individual characteristics and academic records. Simple analyses can extract factors regarding participant’s learning progress, but it is not easy to conduct interviews intended to support study habits without interfering with individual learning activity. Another co-author, Dr. Kouichi Mutsuura, Professor Emeritus at Shinshu University, introduced observation of participant’s note-taking skills into blended and fully online learning courses. Note taking is a conventional learning tool, and can be adapted in response to the content and style of the course. The function of note taking has been discussed as a key learning skill, used in conventional learning instruction and evaluation methodologies. Figure 1 shows the role of note taking. Also, notes can be recognised as a record of individual learning.

v

vi

Preface Lecturer

Student

(Blended / Fully online) Verbal / Non-verbal

Oral Instraction Add

ition

al in form iewin ation g co nten ts

Representation

Rev

Board writing Presentation slide

Information transfer Lexical comparison

Self Learning Summarising Emotional activity Reviewing

Note Taking

Fig. 1 The function of note taking in a class

In order to synthesise these activities, all three authors have been studying the effectiveness of note-taking activities in consideration of the effect of individual characteristics on e-learning course achievement. Note-taking activity also reflects individual learning progress and the emotional factors of learning. These relationships were analysed step by step. The analytical techniques used involved the lecturer rating the contents of notes taken, including student’s summarising techniques and skills, comparing the text content of student’s notes and lecturer’s presentations, using text lexical analysis techniques, and using causal analysis techniques to study the relationships between learning performance and note taking activities. Some improvements in learning procedures were tried out during several practical sessions of the course. The practices used will be presented in detail in the following chapters. Here, the editor wishes to thank all of the collaborators, particularly Professors Mutsuura and Yamamoto. The authors would like to emphasise that conventional techniques can be effective, even in a modern learning environment, and have the potential to improve learning activity. Of course, other techniques exist which may also enhance learning effectiveness. Flexible practices are required to improve current learning systems, using trial and error to determine the best practices. The authors hope that our trials have stimulated the challenges faced by educational practitioners. The editor would like to thank Dr. Akinori Okada, the editorial supervisor of the monograph: “Behaviormetrics: Quantitative Approaches to Human Behavior”, who invited me to summarise this monograph. Also, it is necessary to thank Mr. Yutaka Hirachi of the editorial management division of Springer Japan. This monograph consists of published journal papers. The authors wish to thank all of the publishers and journal organisations, such as The International Journal of Distance Education Technologies, IGI Global, The Electronic Journal of e-Learning, Academic Conferences and Publishing International Limited, The International Journal of Information and Education Technology, The International Journal of New Computer Architectures and their Applications, The Society of Digital Information and Wireless Communications and The International Journal of Educational Technology in Higher Education, Springer. Tokyo, Japan May 2021

Minoru Nakayama

Contents

Evaluation of Student’s Notes in a Blended Learning Course . . . . . . . . . . Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto Impact of Learner’s Characteristics and Learning Behaviour on Learning Performance During a Fully Online Course . . . . . . . . . . . . . . Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto Lexical Analysis of Students’ Learning Activities During the Giving of Instructions for Note-Taking in a Blended Learning Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto Note-Taking Evaluation Using Network Illustrations Based on Term Co-occurrence in a Blended Learning Environment . . . . . . . . . . Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto Effectiveness of Students’ Note-Taking Activities and Characteristics of Their Learning Performance in Two Types of Online Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto The Possibility of Predicting Learning Performance Using Features of Note-Taking Activities and Instructions in a Blended Learning Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto

1

15

37

51

67

89

Student’s Reflections on Their Learning and Note-Taking Activities in a Blended Learning Course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto How Note-Taking Instruction Changes Student’s Reflections Upon Their Learning Activity During a Blended Learning Course . . . . . . . . . . . 121 Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto

vii

Evaluation of Student’s Notes in a Blended Learning Course Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto

Abstract Student’s notes are evaluated to trace their learning process in a blended learning course, and the factors affecting the quality of these notes are discussed. As individual note-taking performance may be based on student’s characteristics, these contributions are also examined. Some factors about personality and the learning experience are significant and positively affect the grades given to notes. Lexical features of notes taken were extracted using a text analysis technique, and these features were compared with the grades given. The good note-takers constantly recorded terms independently of the number of terms which was presented during the class. Conceptual mapping of the contents of notes was conducted, and it suggests that the deviation in the features of notes can be explained by the number of terms in a lesson. Keywords Note-taking · Blended learning · Learning activity · Learning evaluation · Text analytics

1 Introduction Information technology supports various types of educational practices in university teaching. Some black board writing has changed to PC slide presentations and the Internet environment has also given students opportunities to take courses elsewhere, both inside and outside of lecture rooms. Furthermore, a blended learning course, which is a hybrid of face to face lectures and learning using online learning materials, can provide a flexible method of teaching university courses. Though student’s learning activities are monitored using automated evaluations such as online test scores and access logs for the online learning content, some limitations for students of conducting courses this way have been suggested. Therefore, the actual learning activities of students should be measured to trace their learning process outside of

Originally published in the International Journal on New Computer Architectures and Their Applications (IJNCAA), Vol. 1, No. 4, pp. 1056–1065, 2011. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Nakayama (ed.), Note Taking Activities in E-Learning Environments, Behaviormetrics: Quantitative Approaches to Human Behavior 11, https://doi.org/10.1007/978-981-16-6104-4_1

1

2

Evaluation of Student’s Notes in a Blended Learning Course

the relationship between course materials and final performance, such as final exams or test scores. “Note-taking” is a commonly used and time-honoured skill, employed in all types of learning situations; even in higher education [2]. The functions and effectiveness of note-taking have already been reviewed and discussed [3, 4]. In particular, “notetaking” requires the summarisation and understanding of the context of the notes [5]. Learning performance of note-takers has been previously confirmed at the university level [6–8]. Therefore, notes taken may exhibit the process of learning in a class, including in a blended learning session. Blended learning environments are advancing and do provide paperless settings for learning so that students receive fewer printed materials and also encounter fewer opportunities to take notes. This problem has been pointed out [9], though some of the influence of the blended learning environment on note-taking activities has also been confirmed using a previous survey of results of learning activities of ordinary students [8]. According to these phenomena, the information technology environment may sometimes influence and veil students’ learning activities. In a sense, introducing note-taking assessments into a blended learning course is useful to trace student’s learning activity and their progress. For Japanese students, there are few instructions for taking good notes and learning using online learning materials. Some improvement in student’s courses and appropriate learning support are required for Japanese universities to improve students’ note-taking skills. The contents of students’ notes taken should be morphologically analysed to examine the actual situation. Additionally, note-taking activities may depend on the setting, and also on the characteristics of students, because some linguistic information processing abilities are required to take notes. The effectiveness of these factors should be evaluated. The contents of notes taken require the quantification of the features of the notes in order to develop support systems and to improve the teaching methodology. In this paper, the following topics were addressed in response to the problems mentioned above: • The relationship between the assessment of the contents of notes taken and student’s characteristics was measured to extract factors used in note-taking activities in a blended learning environment. • The relationship between lexical features of the contents of students’ notes and the lecturer’s notes which were provided during classes were analysed to extract the features of the contents of notes.

2 Method 2.1 Blended Learning Courses Note-taking activity was surveyed during an information networking system course, which was a blended distance education course in a bachelor-level program at a

2 Method

3

Japanese university. All participants were bachelor students in the Faculty of Economics, and they have already had some experience studying in a blended learning environment. In this blended learning course, face-to-face sessions with students were conducted every week, and students from the course who participated in this survey gathered in a lecture room. Being a good note-taker was expected of all participants during face-to-face sessions. Students were encouraged to take an online test for each session of study outside the class room, as a function of the learning management system (LMS). To do well in an online test, good notes are preferred. These online test scores were referred to when final grades were determined and students could take the tests repeatedly until they were satisfied with their results. The LMS recorded the final scores. This course required participants to write three essay reports and a final exam. In the essay reports, participants have to synthesise their own knowledge regarding three topics taught during the course. These activities are an important part of the constructivism learning paradigm and may influence comprehension during the course [10]. As the course included face-to-face sessions, the lecturer could talk freely about course topics, and students could take notes of the impressions they received there.

2.2 Note-Taking Assessment All participants were asked to take their own notes, and to present their notebooks to the lecturer every week. The lecturer quickly reviewed and assessed each student’s notes after the weekly sessions, then returned the notebooks to students as soon as possible. Prior to doing so, all notes were scanned by the lecturer and stored as images in a PC. The contents of the notes created by the lecturer when the course sessions were designed can be used as a standard for the notes taken in every class. Therefore, these notes can be used as the criterion for evaluating students’ lecture notes. The individual content of students’ notes was evaluated using a 5-point scale (0–4), 4: Good, 3: Fair, 2: Poor, 1: Delayed, 0: Not presented. If a student reproduced the same information in his or her notebook, the note-taking was rated as “Fair”. “Fair” note-taking is the reproduction of transmitted information given as instructions. If any information was omitted, the rating given was “Poor”. In a sense, “Poor” notetakers failed to reproduce the information transmitted. When students wrote down additional information from the lecture, the note-taking was rated as “Good”. The “Good” note-takers included those who integrated this knowledge with relevant prior knowledge [11], as some pieces of knowledge are related to each other, and some pieces of knowledge are related to relevant prior knowledge. At this point, several kinds of constructivistic learning activities were occurring. The number of valid subjects was 20. As mentioned above, the students did not have any special training with regard to note-taking skills. The note-taking skills they exhibited were their own.

4

Evaluation of Student’s Notes in a Blended Learning Course

2.3 Characteristics of Students To analyse note-taking performance in this study, the student’s characteristics were focused on, since precise note-taking may be concerned with some personality traits. The summarising process may be affected by the student’s level of information literacy, while a degree of familiarity with the blended learning environment also enhances their academic performance. According to the hypothesis, the student’s characteristics were measured using three constructs. These constructs were personality [12, 13], information literacy [14] and learning experience [15]. 1. Personality: For the first construct, the International Personality Item Pool (IPIP) inventory [13] was used. Goldberg [12] lists five personality factors which are called the “Big 5” and for this construct, there are five component scores: “Extroversion”, “Agreeableness”, “Conscientiousness”, “Neuroticism” and “Openness to Experience”. 2. Information literacy: Fujii [16] defined and developed inventories for measuring information literacy. For this construct, the survey had 32 question items and 8 factors were extracted: interest and motivation, fundamental operation ability, information collecting ability, mathematical thinking ability, information control ability, applied operation ability, attitude and knowledge and understanding. The overall mean of factor scores was used to indicate each student’s information literacy level. This inventory was originally developed to measure information literacy among high school students in many countries. It can also be used to measure the information literacy level of university students [16]. 3. Learning experience: Students’ learning experiences in a blended learning environment were measured using a ten-item Likert-type questionnaire. This construct measures students’ attitudes to blended learning environments. As in previous studies, three factors were extracted: Factor 1 (F1): overall evaluation of the elearning experience, Factor 2 (F2): learning habits and Factor 3 (F3): learning strategies [15].

2.4 Text Analysis of the Notes Taken As already mentioned, notes taken by all 20 subjects in every weekly session were scanned and stored as image files. All images of notes taken were converted into digital text files by reading and manual data entry, because some written characters needed to be read by the human eye to be entered correctly. Figures were excluded. The texts of student’s and lecturer’s notes were classified into noun and adjective terms using a Japanese morphological term analysis tool [17], then by synthesising a term–document matrix (X ), term frequency vectors were created for each class session. Latent semantic indexing (LSI) was used to classify documents according to the degree of similarity between them [18]. The features of the terms and the course sessions were extracted from the term–document matrix of the notes taken

2 Method

5

in each class using SVD (singular value decomposition analysis) [18] as follows: X = T S D  . The term feature matrix (T ) contains feature vectors of each term. The feature vectors of note documents (N ) for each class session (i) were calculated for the lecturer and for each student using a summation of term feature vectors (T ) which was weighted by term frequency (Fi ) in a class (i), therefore, Ni = T Fi .

3 Results 3.1 Note-Taking Assessment 3.1.1

Grades of Notes Assessed

The assessment scores of each set of notes taken were gathered and summarised in Fig. 1. According to the figure, percentages for “Fair” are almost always higher than the other ratings during almost all weeks. The percentage of note-takers rated “Good” in the first 5 weeks of the course is relatively higher than the percentages for the remaining weeks. This suggests that students cannot create “Good” notes as the course progresses. Also, the percentages for “Poor” ratings are almost always the lowest.

3.1.2

Effectiveness of Student’s Characteristics

One of the hypotheses of this paper, that the influence of student’s characteristics may affect the quality of notes taken, is examined in this section. The sums of

100

80 Percentage (%)

Fig. 1 Grades of notes assessed across weeks (n=20)

60

NT: Fair

40 NT: Good 20 NT: Poor 0 1

2

3

4 5 6 7 8 9 10 11 12 13 Course sessions (weeks)

6

Evaluation of Student’s Notes in a Blended Learning Course

Table 1 Learning performance between two groups of note assessment scores Note assessment score High (n = 11) Low (n = 9) Report score Online test score Final exam score

0.59(0.27) 289.7(85.0) 49.5(7.3)

0.67(0.18) 254.3(79.7) 44.9(7.3)

() indicates SD

assessment scores for note-taking during the course are calculated using a 5-point scale, to evaluate each student’s note-taking ability. To emphasise the differences in the note-taking abilities, the students are then divided into two groups consisting of high and low scores. The scores of the two groups are compared, to measure the contribution of student’s attributes and characteristics to note-taking activities. First, students’ learning performance from report scores, online test scores and final exam scores of the two groups was summarised in Table 1. Though there are no significant differences between the two groups, scores of online tests and final exams for the group of note-takers with high scores are higher than those for the group with low scores, while report scores for the low-score group are higher than those for the high-score group. As a result, note-taking scores did not contribute to learning performance in this course. The contribution of learning experiences such as subjective evaluation was examined, and three factor scores are summarised in Table 2 using the same format as in Table 1. The third factor, learning strategies, is significantly higher for the low-score group than it is for the high-score group. This consists of two question items: “I have my own method and way of learning” and “I have my own strategies on how to pass a course”. Therefore, the purpose of note-taking may affect the scores of note-takers. The relationship between a student’s personality and note-taking performance was measured. The note-taking scores are summarised across five personality factors in Table 3. As shown in the table, the scores for the high-score group are significantly higher than those for the low-score group across all personality factors except the “Openness to Experience” factor. This result suggests that learning personal-

Table 2 Learning experience between two groups of note assessment scores Factors of learning experience Note assessment score High (n = 11) Low (n = 9) Overall evaluation of the e-Learning experience Learning habits Learning strategies 5-point scale, () indicates SD significance level *: p < 0.05

2.82(0.58)

3.17(0.65)

2.09(0.63) 3.00(0.71)

2.50(0.97) 3.67(0.61)∗

3 Results

7

Table 3 Scores of personality factors between two groups of note assessment scores Personality factors Note assessment score High(n = 11) Low (n = 9) Extroversion Agreeableness Conscientiousness Neuroticism Openness to Experience

3.29(0.61)∗ 3.58(0.35)∗ 3.37(0.47)∗ 3.34(0.60)∗∗ 3.74(0.76)

2.52(0.73) 3.10(0.63) 2.88(0.51) 2.38(0.75) 3.23(0.64)

5-point scale, () indicates SD Significance level *: p < 0.05, **: p < 0.01 Table 4 Scores of information literacy between two groups of note assessment scores Information literacy factors Note assessment score High (n = 11) Low (n = 9) Interest and motivation Fundamental operation ability Information collecting ability Mathematical thinking ability Information control ability Applied operation ability Attitude Knowledge and understanding Grand total

3.84(1.00) 4.52(0.49)# 3.57(0.98) 2.86(0.85) 3.09(1.10) 3.11(1.06) 3.00(0.85) 3.68(0.79)# 3.46(0.61)

4.22(0.49) 3.92(0.88) 3.11(0.93) 3.17(1.31) 2.86(0.79) 2.89(1.11) 2.69(0.69) 2.97(0.81) 3.23(0.65)

5-point scale, () indicates SD Significance level #: p < 0.10

ity positively affects note-taking activity. These aspects should be considered when out-of-class assistance is provided to students to improve their note-taking skills. The contribution of information literacy is also measured using the same format. In Table 4, the scores of note-takers are summarised across 8 factors of information literacy. Additionally, the scores are summarised in response to the two secondary factors and a summation of information literacy is given. There are significant differences in note-taking scores for “Fundamental operation ability” and “Knowledge and understanding” at a 10% level of significance. All factor scores for information literacy for the high-score group are higher than the scores for the low-score group, except for the factor “Interest and motivation”. Study of the contribution of information literacy to note taking should be continued using larger samples.

Fig. 2 Relationship between the number of terms in notes of lecturer and students (The number of students = 20)

Evaluation of Student’s Notes in a Blended Learning Course

300

Number of terms (Students)

8

250 7

8

200 11

150

9

6

2

3 10 4

100

5 12

50

1

Good

13

Fair

0 0

50

100

150

200

250

300

Number of terms (Lecture)

3.2 Features of Notes In order to compare the number of terms in notes between the lecturer and students, mean numbers of terms are summarised in Fig. 2. Because the numbers of terms which appear in lectures is different across course sessions, the volume of notes taken by students may depend on the lecturer’s notes. In Fig. 2, the horizontal axis shows the number of terms the lecturer has presented in each course session, and the vertical axis shows means of terms students have written down. Therefore, the diagonal line in the figure shows the same numbers of terms which the lecturer and note-takers have written. The error bar shows the standard error of the mean. The number indicates a series of course sessions. The means are calculated for “Good” and “Fair” note-takers, respectively. Both the numbers and means of terms for “Good” note-takers are always higher than the numbers and means for “Fair” note-takers, while the number of terms for “Fair” note-takers is almost the same as the number of terms written by the lecturer, because these plots are distributed on a diagonal line. In particular, the number of terms for “Good” note takers stays high even when the number of terms in the lecturer’s notes is low. According to Fig. 2, the “Good” note-takers record more terms. The number of terms these students record is almost always higher than the number of terms provided by the lecturer. The next question is whether the terms students recorded actually covered the terms which the lecturer presented. To evaluate this degree of coverage, a coverage ratio was calculated as a percentage of terms which were recorded by students. The coverage ratios of terms for each session are summarised in Fig. 3. As the figure shows, there are some differences in coverage rates between “Good” and “Fair” note-takers. The differences are almost always small, while the ratios are almost always over 70%. The possible reasons why coverage rates are relatively low

3 Results 1

Term coverage rate (%)

Fig. 3 Coverage ratio of terms in student’s notes compared to lecturer’s notes (n = 20)

9

0.9

NT: Good

0.8

0.7 NT: Fair 0.6

0.5 1

2

3

4 5 6 7 8 9 10 11 12 13 Course sessions (weeks)

in the fourth and ninth sessions may be because it is not easy for students to pick up additional terms, while the lecturer displays figures. The above analyses suggest that all subjects record almost all terms the lecturer presents during the class. This means that all students can reproduce the conceptual contents of the lecture. Also, choosing to record the rest of the terms presented by the lecturer may affect the grades of notes taken. To confirm this phenomenon, features of each note are calculated using the term frequency of the session and feature vectors of terms which are extracted from the LSI model mentioned above. The features of all notes of all 20 students in 13 course sessions are illustrated two-dimensionally in Fig. 4, using two principal components of the feature vectors. The horizontal axis shows the first component, and the vertical axis shows the second component. The features of notes for “Good”, and “Fair” note-takers and for the lecturer are displayed

0.6 Second component of feature

Fig. 4 Conceptual map for students and lecture notes for all sessions (n = 20)

10

0.4 0.2

5

9 3

4 8

1

11

Good Fair Lecturer

2

0 12 -0.2 -0.4 -0.6 0.2

13

6 7

0.4 0.6 0.8 First component of feature

Fig. 5 Conceptual map for students and lecture notes for typical sessions (n = 20)

Evaluation of Student’s Notes in a Blended Learning Course 0.6 Second component of feature

10

0.4

10

Good Fair Lecturer

0.2 0

12

-0.2 -0.4 -0.6 0.2

13 7

0.4 0.6 0.8 First component of feature

separately. Also, the number of sessions is indicated. The distances between the features of notes show a degree of similarity between them. The degree of separation of the plots of these session notes represents certain aspects of note-taking. As the figure shows, there are clusters for each class session. For some sessions, features of notes for students and features of the lecturer’s notes overlapped each other. To make the relationship clear, four typical sessions are extracted in Fig. 5. The four sessions consist of sessions 7 and 10, which have the highest number of terms taken, and sessions 12 and 13, which have the lowest number of terms taken, as shown in Fig. 2. In this figure, plots for sessions 7 and 10 produce a small cluster, which means that all notes are similar to each other. This suggests that all students can reproduce the lecturer’s notes, though their notes are classified into two grades. On the other hand, plots for sessions 12 and 13 are distributed in the same plane, but plots for the student’s notes are disparate from the lecturer’s notes. From a detailed inspection of the clusters for sessions 12 and 13, we can observe that the plots for “Fair” notes surround the lecturer’s notes in close proximity, but the plots for “Good” notes are more widely distributed around both the lecturer’s and the “Fair” notes. When the number of terms is large, most notes include almost all major features, so that all plots are mapped around a common point. The rest of the terms and the structure of student’s note-taking may affect their note-taking grade, however, because the lecturer had classified their notes as “Good” and “Fair”. When the lecturer presents a small number of terms, the plots of “Good” note-takers are disparate from the plots of the lecturer’s notes, since they wrote down some additional, related terms, and some of these were to explain the terms presented by the lecturer i.e. they recorded more terms than the lecturer presented, as shown in Fig. 2. The “Fair” note-takers skipped some terms presented by the lecturer and added inappropriate terms to their notes in place of missed words, because they recorded the same number of terms presented by the lecturer in Fig. 2. However, the coverage ratios of the terms were not as high as in Fig. 3, since their plot positions are different from those of the lecturer.

3 Results

11

Again, both plots of “Good” and “Fair” note-takers are disparate from the lecturer’s notes, but the reasons for this are different between the two grades. This point should be taken into consideration to ensure that the development of student’s note-taking skills is supported and improved. To compensate for this learning condition, designs for the structure of classes and student support programs should be considered carefully. These points will be subjects of our further study.

4 Discussion Since their note-taking performance depends on the content of the weekly courses, students’ note-taking abilities were evaluated using the sums of their assessment scores graded on a 5-point scale across the 13 weeks of the course. Twenty students were divided into two groups using the scores. To determine the contribution of each student’s characteristic metrics, the scores of metrics are compared between the two groups. In the results, there are significant differences in learning strategies depending on the kind of learning experience, and significant differences in the four factors about personality. There are also some differences in the two factors which concern the information literacy of students between the two groups. Therefore, note-taking ability may be affected by these factors. When note-taking skills are taught to students or students are supported in the improvement of their skills, their characteristics should be considered in order to achieve better performance, since student’s aptitude often affects their learning performance. According to the lexical analysis of student’s notes, “Good” note-takers constantly record the same number of terms, even if the lecturer presented a small number of terms. “Fair” note-takers often record the same number of terms as the lecturer presented, even when the lecturer has presented many terms or only a small number of terms. However, if the coverage ratios of lecture’s terms are not high, then the lexical concepts are slightly different from the ones the lecturer presented, particularly when the number of terms the lecturer presented is small. As the number of terms “Fair” note-takers recorded increased with the number of terms the lecturer presented, it may be that the lecturer presented as many terms as possible when trying to explain a concept. Also, the lecturer should ask students to take notes carefully in blended learning sessions or in lectures using information technology aids. These results are based on Japanese university students who have almost no previous note-taking skills or experience. Also the results suggest that the information technology environment affects student’s note-taking performance in spite of their insufficient skills for note-taking during lectures. To improve students’ performance and to help them adapt a more positive attitude toward studies in a blended learning environment, appropriate instruction and support to enable students to take sufficient notes during lectures is required. The methodology for this should be discussed with a wide range of university faculty and staff.

12

Evaluation of Student’s Notes in a Blended Learning Course

5 Conclusion To improve the learning progress and to develop a support program for use in a blended learning environment, the assessment of note-taking ability as an important learning activity was conducted during a blended learning course. The relationships between note-taking activity and student characteristics were examined. According to the results of a comparison of scores between “Good” and “Fair” notes taken, student’s characteristics, such as personality and one of three factors regarding the learning experience affect the contents of their notes. Some lexical features of the notes taken were extracted, and the relationship between these features and the grades of the note-takers were discussed. The “Good” note-takers constantly record terms for their own notes independent of the terms presented by the lecturer. From the results, a set of possible instructions should be proposed. The development of a supporting methodology will also be a subject of our further study. Acknowledgements This research was partially supported by the Japan Society for the Promotion of Science (JSPS), Grant-in-Aid for Scientific Research (B-22300281: 2010–2012). A portion of this paper were presented at the IEETel2011/DICTAP2011 Conference.

References 1. Nakayama M, Mutsuura K, Yamamoto H (2011) Evaluation of student’s notes in a blended learning course. Int J New Comput Arch Appl 1(4):1080–1089 2. Weener P (1974) Note taking and student verbalization as instrumental learning activities. Instr Sci 3:51–74 3. Kiewra KA (1989) A review of note-taking: the encoding-storage paradigm and beyond. Educ Psychol Rev 1(2):147–172 4. Trafton GJ, Trickett SB (2001) Note-taking for self-explanation and problem solving. HumComput Interact 16:1–38 5. Piolat A, Olive T, Kellogg RT (2005) Cognitive effort during note taking. Appl Cogn Psychol 19:291–312 6. Nye PA, Crooks TJ, Powley M, Tripp G (1984) Student note-taking related to university examination performance. High Educ 13:85–97 7. Kiewra KA, Benton SL, Kim SI, Risch N, Christensen M (1995) Effects of note-taking format and study technique on recall and relational performance. Contemp Educ Psychol 20:172–187 8. Nakayama M, Mutsuura K, Yamamoto H (2010) Effectiveness of note taking activity in a blended learning environment. In: Proceedings of 9th European conference on e-learning, pp 387–393. Porto, Portugal 9. Kiewra KA (1985) Students’ note-taking behaviors and the efficacy of providing the instructor’s notes for review. Contemp Educ Psychol 10:378–386 10. Tynajä P (1999) Towards expert knowledge? a comparison between a constructivist and a traditional learning environment in the university. Int J Educ Res 31:357–442 11. Mayer RE, Moreno R, Boire M, Vagge S (1990) Maximizing constructivist learning from multimedia communications by minimizing cognitive load. J Educ Psychol 91:638–643 12. Goldberg L (1999) A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personal Psychol Eur 7:7–28

References

13

13. IPIP (2001) A scientific collaboratory for the development of advanced measures of personality traits and other individual differences. http://ipip.ori.org 14. Nakayama M, Yamamoto H, Santiago R (2008) Impact of information literacy and learner characteristics on learning behavior of Japanese students in on line courses. Int J Case Method Res Appl XX(4):403–415 15. Nakayama M, Yamamoto H, Santiago R (2007) The impact of learner characteristics on learning performance in hybrid courses among Japanese students. Electron J e-Learn 5(3):195–206 16. Fujii Y (2007) Development of a scale to evaluate the information literacy level of young people -comparison of junior high school students in japan and northern europe. Jpn J Educ Technol 30(4):387–395 17. MeCab: Yet another part-of-speech and morphological analyzer. http://mecab.sourceforge.net 18. Deerwester S, Dumais ST, Furnas GW, Landauer TK, Harshman R (1990) Indexing by latent semantic analysis. J Am Soc Inf Sci 41:391–407

Impact of Learner’s Characteristics and Learning Behaviour on Learning Performance During a Fully Online Course Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto

Abstract A fully online learning environment requires effective learning management in order to promote pro-active education. Since student’s notes are a reflection of the progress of their education, analysis of notes taken can be used to track the learning process of students who participate in fully online courses. This paper presents the causal relationships between student’s characteristics, note-taking behaviour, learning experience, note assessment and test scores while the relationships between these metrics is examined. A fully online course for undergraduate students in Economics was conducted. Participants were asked to study each course module and present their notes to the lecturer every week. The student’s learning performance was then measured using online tests, weekly confirmation tests and a final exam. The total number of valid participants in the courses was 53. Three factors of note-taking behaviour were extracted according to the survey and their relationships with other metrics were calculated. A structural equation modelling technique was used to track student’s learning activity as note-taking occurred, using the scores of their metrics. The results of this modellingtechniquesuggestthatkeyfactorsandtheircontributionstotestscorescanbe measured. Also, the factors which contribute to note-taking behaviour were examined. Keywords Note-taking · Fully online course · Learning assessment · Causal analysis

1 Introduction The online learning environment is expanding throughout educational institutions around the world, including universities. In particular, this is occurring at universities which offer courses internationally. Online learning mostly consists of blended learning and fully online courses. Blended learning primarily employs face-to-face sessions, including distance learning/lecturing sessions and online materials are also

Originally published in the Electronic Journal of e-Learning, Vol.12, Issue 4, pp. 394-408, 2014 [1]. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Nakayama (ed.), Note Taking Activities in E-Learning Environments, Behaviormetrics: Quantitative Approaches to Human Behavior 11, https://doi.org/10.1007/978-981-16-6104-4_2

15

16

Impact of Learner’s Characteristics and Learning Behaviour on Learning …

provided to students. Fully online learning has no face-to-face sessions, and most learning processes are provided through an online environment. Therefore, this type of instruction can present students with freedom from learning restrictions. Also, as the online learning environment supports student’s studies, some benefits and improvements in learning activities have been reported [2]. They are recognised as significant learning tools. Recently, Open Educational Resources (OERs) which are online learning materials developed for online courses have come to be used widely by students and others studying online. This trend is spreading worldwide [3]. Currently, a new concept for online educational delivery known as Massive Open Online Courses (MOOCs) has emerged. These are available for both blended and fully online courses, and are attracting the interest of both educators and students [4]. Though institutes of higher education recognise some potential benefits, the impact on teaching and learning is still being discussed [5]. On the other hand, it has often been suggested that a great deal of these participants have difficulty with continuing their education online, exacerbating drop-out rates [6]. Since the evaluation of the learning process is restricted, online tests and access logs in the case of online courses have enabled analysis up until now [7]. While the worldwide use of MOOCs as fully online courses has increased rapidly, the course completion rate is still one of the most serious problems regarding their success [4]. Therefore, accessing the behaviour of participants for MOOCs has been widely analysed [8, 9]. However, when instruction is provided, even in a face-to-face blended learning environment, students’ evaluations of their attitudes and learning processes can be readily observed. During a fully online course, which consists of the use of online course materials only and no face-to-face instruction, the restrictions on analysis are much tougher than in a blended learning course. It then becomes difficult to track and evaluate students enrolled in courses such as these, where the course materials are limited to textbooks and online materials. In particular, the question of how participants learn the course content during a fully online course arises. An effective solution to this problem is therefore needed. To track each student’s learning process, we asked all participants to take notes during the course and present them for examination periodically. Since note-taking reflects learning activity [10–13] and also constructivistic learning [14], this information can be used as a significant index of the learning process. Additionally it is well known that learner’s attitude, literacy and learning strategies affect their learning performance [15]. Some previous studies have identified the causal relationships between these constructs and indices of learning performance, such as test scores [16]. Note-taking behaviour in a blended learning course have been evaluated in a previous survey [17]. The factors of note-taking behaviour may be related to student’s characteristics and test scores [18]. This suggests that student’s characteristics affect note-taking behaviour and test scores. To quantitatively determine the relationships, a structural equation modelling technique was used and possible causal analysis was conducted throughout the course.

1 Introduction

17

Our research questions were: 1. Are student’s note-taking performance and note-taking behaviour related to metrics of learning progress, such as test scores, in a fully online university course? 2. Using causal analysis, are there any relationships between the factors of notetaking behaviour and student’s personality or information literacy, and the relationships between the factors of note-taking behaviour and learning experience or test scores? 3. Is it possible to extract a reasonable causal model based on the extracted submodels and develop a learning model using the effects of student’s characteristics on test scores? 4. What kinds of support are significant for students who participate fully online courses?

2 Background 2.1 Obstacles to Online Learning Online learning or e-Learning is a style of instruction using information technology (ICT) which includes distance learning and conventional teaching/learning activities that use the Internet. The above learning styles use various learning materials, such as web sites, and offer some freedom of study, such as the ability to learn anytime and anywhere. Currently, open educational resources such as Open Course Ware [19], which are freely available to learners not enrolled in online courses, are very popular at many universities, and are not only for registered students [2]. Fully online courses are becoming popular due to the proliferation of online learning. Since these course can maximise cost-benefits for all [20], both universities and students have an interest in taking advantage of this. Though most online courses have been well designed using an instructional design methodology, the failure of participants to complete courses is often discussed [6]. Various factors regarding online learning, such as learning styles and student’s characteristics have been discussed in order to provide better courses [21, 22]. In particular, internal factors regarding the courses, such as course design, mental factors, such as personality and literacy, and support services, are often focused on [21, 23]. One approach analyses behavioural data during online learning. An online course management system can easily gather individual data about the learning process, using access logs and online test scores, as these courses are integrated with a learning management system (LMS). Using this data, the prediction of likely drop-outs [7], and appropriate advice providing systems have been developed [24]. However, data about participant’s behaviour can not explain the actual problems of online learning courses or online learning systems.

18

Impact of Learner’s Characteristics and Learning Behaviour on Learning …

2.2 Learner Characteristics Learner’s characteristics can be defined as individual mental factors which may affect learning activity [25]. This factor is recognised as a major one, and a significant source of problems related to online learning. Since some characteristics affect online course completion rates and their evaluations [21, 22], these influences should be examined. Of course, this has been carefully considered in the design of traditional courses, as conventional text books provide discussion [26, 27]. Recently, a wider range of characteristics, such as motivation, efficacy, thinking style, learning skills and socio-cultural factors, have been introduced to improve online learning [23, 28–30]. Learner’s fundamental characteristics that may affect learning activity are personality, which is recognised as consisting of 5 factors, information literacy, which is related to the ability to use information technology, and the proper attitude to deal with the content of the information provided. Sometimes, attitudes towards or impressions of the learning environment can be included in the characteristics because they influence the learner’s performance. As some student’s characteristics, such as scores of final exams affect learning performance, in a formal course, the effectiveness of the factors is evaluated, and the causal relationships between student’s characteristics and scores of tests have been analysed [15, 16].

2.3 Note-Taking Behaviour “Note-taking” is a popular and conventional skill for all types of learning activities [31]. The effectiveness of note-taking has been confirmed at universities and also in primary and secondary schools. The functions and effectiveness of “note-taking” have already been reviewed and discussed [11, 31, 32]. In particular, “note-taking” requires cognitive effort because this activity is based on summarising and understanding of the context [33, 34]. This process is recognised as constructivistic learning [14] while its major effect is called the “Coding effect” [13]. The relationship between some factors of note-taking and learning performance in university courses has been established previously [12, 13, 35]. Also, note-taking styles have been systematically classified [34]. As note-taking is a common activity for students, many universities provide students with instructions about the functions of note-taking [36]. In addition, some practical aspects have been investigated and discussed, such as the effectiveness of examining test scores and student’s note-taking strategies have been discussed [32, 37]. Since educational technologies have introduced overhead and digital slides into lectures, note-taking behaviour has also been affected. When these slides show the critical points of the lecture, students’ recording performance is equivalent to a condition using “guided notes” which are a modified version of the instructor’s notes or slides [38]. Learning using guided notes is effective for quizzes during study class

2 Background

19

sessions and other learning activities [39]. Therefore showing slides in classes affects students summarisation of the content to be learned. Recently, many digital note-taking systems or digital writing systems have been developed to support flexible new methods of learning [40]. Most information communication technology (ICT) applications for education promote the minimisation of the cognitive load and the transformation of course content into knowledge [41, 42]. These systems provide paper-less learning settings, allowing people to learn using digital files [19], which are now commonly provided. With these styles of learning, the effectiveness of note-taking was not clearly apparent [43]. This means that many student’s ability to take notes may have declined, as they prefer using online methods [44]. Some researchers bear this point in mind, noting that no evidence has been provided to prove the phenomena [10]. This point should be re-evaluated in the current online learning environment.

3 Method A formal credit course, Information Systems Network, was conducted as a fully online course. The participants were junior and senior undergraduate students in Economics at a Japanese national university. The total number of valid participants was 53. The online course consisted of modules with slides, audio files and online tests. An example of a slide is shown in Fig. 1. When a student starts the slide show, he or she can join the ordinary lecture, which consists of slides and audio instruction. Of course, students can stop and replay the slide show, and rejoin repeatedly. The only printed material was a textbook, and there were no face-to-face sessions during the course. Participants were asked to take notes freely throughout the course, without any specialised instructions. They know well that good note-taking behaviour may con-

Fig. 1 An example of a Slide of a Fully Online Course (in Japanese)

20

Impact of Learner’s Characteristics and Learning Behaviour on Learning …

tribute to their own learning performance and they also have developed their own note-taking habits. All students were provided with a conventional notebook for this survey. They were asked to present their paper-based notes for review and survey by the professor. Students were not informed of the purpose of surveying their notes. Therefore, the survey investigated their voluntary note-taking activity. They were also asked to study one module per week, and they were encouraged to take online tests to verify that they had mastered the contents of the modules. These online tests functioned as part of the learning management system (LMS). Students could evaluate test scores themselves, and take online tests repeatedly until they were satisfied with their scores. Online test scores were recorded for 13 out of 15 weeks. Also, paper-based weekly confirmation tests were conducted to monitor student’s progress using online learning materials. Due to the tests, all participants had to gather in a lecture room every week for 12 out of the 15 weeks of the course. This condition simulated ordinary classes. The learning pace was set by the lecturer, and the final scores of the online tests and the weekly confirmation test scores were recorded [17]. Students were permitted to refer to their notes during these online and reviews before the weekly test sessions.

3.1 Characteristics of Students Student’s fundamental characteristics were measured using two constructs: personality [45, 46] and information literacy [44, 47]. The metrics of their learning behaviour were surveyed at the beginning of the term, using questionnaires. An abstract of these constructs is as follows.

3.1.1

Personality

The personalities of students were estimated using the International Personality Item Pool (IPIP) inventory [46]. This inventory consists of a five factor personality model which was proposed by Goldberg [45]. The five factor components are “Extroversion”, “Agreeableness”, “Conscientiousness”, “Neuroticism” and “Openness to Experience”. These factors are explained as follows [48]: Extraversion encompasses specific traits such as talkativeness, being energetic, and assertiveness. Agreeableness includes traits like sympathy, kindness, and affection. Conscientiousness includes traits like organisation, thoroughness, and planning ability. Neuroticism includes traits like tension, mood, and anxiety. Openness to Experience includes traits like having wide interests, and being imaginative and insightful. These factor scores were calculated using the results of factor analysis.

3 Method

3.1.2

21

Information Literacy

Fujii has developed a set of inventory for information literacy surveys which consists of 32 question items regarding 8 factors, as follows: interest and motivation, fundamental operational ability, information collecting ability, mathematical thinking (reasoning) ability, information controlling ability, applied operational ability, attitude, and knowledge and understanding [47]. Two secondary factors were extracted from these 8 factors: operational skills and attitudes toward information literacy [44].

3.2 Metrics of Learning Behaviour Students acquire various study habits in school. Their habits can be measured using two metrics: note-taking behaviour [18] and learning experience [15].

3.2.1

Note-Taking Behaviour

The note-taking behaviour of students and their behaviour may reflect not only notetaking performance but also learning achievement [32, 35]. Therefore, the construct for note-taking behaviour may be a key to the evaluation of the learning process. Note-taking behaviour is sometimes discussed, but has not been identified as a major technique for studying at Japanese universities, however. To observe student’s notetaking abilities, attitudes and techniques, 20 original inventories were developed by the authors. The inventories were created using question items from Cornell style notes [36] and items from other previous studies. The inventories are displayed in Table 1 [18]. Further details will be discussed in the Results section.

3.2.2

Learning Experience

Student’s learning experiences were surveyed using a set of 10 question items which had been previously developed to evaluate an online university course [15]. This construct consisted of three factors, as follows: Factor 1 (LE-F1): “Overall evaluation of the e-learning experience”, Factor 2 (LE-F2): “Learning habits”, and Factor 3 (LEF3): “Learning strategies” [15].

3.3 Note-Taking Assessment All participants were required to present their session notes on a weekly basis. The lecturer reviewed and graded these. The contents of each session are defined using the slide presentations of the online materials. The contents of the slides are references

22

Impact of Learner’s Characteristics and Learning Behaviour on Learning …

Table 1 Factor loading matrix for Note-taking behaviour (Promax rotation) No. Question item 1st Factor 2nd Factor 1 2

0.87 0.82

-.11 0.02

0.02 -.01

0.79

0.11

-.06

0.79

0.09

0.05

0.63 0.56

-.09 -.04

0.13 0.17

0.53

0.29

0.21

0.52 -0.10

0.18 0.92

-0.24 0.04

-0.08 0.18

0.89 0.77

-0.01 0.05

-0.02

0.71

0.17

-0.02 0.33

0.62 0.58

0.24 -0.14

0.20

0.55

-0.13

0.08

-0.27

0.88

-0.03 0.12

0.14 0.03

0.73 0.72

0.05 -0.17

0.12 0.23

0.71 0.67

F1: Recognising note-taking functions F2: Methodology of utilising notes F3: Presentation of notes

1.00 0.47 0.31

Correlation 1.00 0.39 1.00

Contribution ratio

0.30

0.31

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

NT during sessions to clarify the contents NT is for understanding the whole course not only the session topics NT during sessions to understand the course contents NT consists of what teacher presented and talked about I understand the syllabus summary of this course I use a colored pen or marker to highlight important points I think about the meaning and importance of words during NT NT during sessions to review the contents later I use NT to write some additional information in the notes taken I use NT to revise the notes taken after the session I think about relationships between contents of the notes taken Notes of surveyed contents are added to notes taken I have an original writing format for NT I use NT to review the notes taken after the session I use notes taken to review the contents of a session in advance of a test Notes are taken so that even non-participants can understand the contents My NT techniques have improved Notes are taken so that other participants can understand the contents Classmates are considered when notes are taken I have NT skills

3rd Factor

0.23

NT: Note-taking

for assessing notes. The professor who is the designer and lecturer of this course assessed all notes for 11 of its 15 weeks. The assessment was used in the final grades of participants as part of their course credit. Therefore, the assessment was evaluated

3 Method

23

as a reliable measure. Even if other professors joined in the assessments, certain aspects of their evaluations may not be consistent with those of the original professor. The contents of student’s notes were evaluated using a 5-point scale (0–4), 4: Good, 3: Fair, 2: Poor, 1: Delayed, 0: Not presented [17]. “Fair” note-taking is the thorough reproduction of information given. If any information was omitted, a “Poor” rating was given. The “Good” note-takers included those who integrated additional relevant prior knowledge with the slides and audio materials [41]. This behaviour is sometimes explained as constructivistic learning [14, 42].

3.4 Causal Analysis To determine the hypothetical relationship between student’s characteristics, notetaking behaviour, learning experience and test scores, a causal analysis has been used to explain the contribution of note-taking. To design a model and evaluate the fitness of the model, the structural equation modelling technique (SEM) has been introduced. The actual calculations were conducted using the AMOS package [49]. The causal models were designed with consideration given to the factors of notetaking behaviour, as follows: 1. Impact of student’s characteristics on note-taking behaviour Student’s characteristics, in particular personality and information literacy, may be related to note-taking ability. A simple model is hypothesised that these factors affect student’s note-taking behaviour. 2. Note-taking skills affect note-taking performance and test scores As note-taking behaviour may be related to overall note-taking performance and weekly test scores [18], a causal model was designed. The contribution factor and the direction of learning experience factor should be carefully considered. As the learning experience in this paper is an online one, it is hypothesised that in a causal model note-taking behaviour affects both the learning experience and course performance, such as scores in online and weekly tests and the final exam. 3. An integrated model The two above models are merged to form a single integrated model which includes all factors mentioned above. All paths between variables were assessed using trial and error while the path coefficients were evaluated. An optimised model was extracted by maximising the Goodness-of-Fit (GFI) and Adaptive Goodness-of-Fit (AGFI) indices.

24

Impact of Learner’s Characteristics and Learning Behaviour on Learning …

4 Results 4.1 Note-Taking Behavioural Factors Valid responses from 53 participants were summarised. To extract note-taking factors for this survey, factor analysis was conducted using Promax rotation. Table 1 shows the factor loading matrix and the correlation coefficients across three factor axes, and the contribution ratio of each factor while ignoring the other factors is also illustrated. As a result, three factors were extracted and each factor was given labels such as F1: Recognising note-taking functions, F2: Methodology of utilising notes, and F3: Presentation of notes. These factors emphasise three aspects of notetaking. According to the correlation coefficients, all factor axes correlate with each other; therefore the three factors are related to each other though they are identified separately.

4.2 Note-Taking Assessment The percentages of note assessment levels across the weeks of the course are summarised in Fig. 2. According to the figure, percentages for “Fair” are almost always higher than those for the other assessment levels through the course period. The percentage rates of note-takers rated “Good” is very low. This suggests that students are unable to create “Good” notes as the course progresses. Also, the percentages of “Poor” assessment levels are almost always the lowest. This result suggests that most students have simply reproduced slide contents in their notes. All assessments are summed up as ratings for all weeks of the course, and note assessment scores

100 NT: Good

80 Percentage (%)

Fig. 2 Grade Percentages of Note-taking Assessments

NT: Fair 60

40

20 NT: Poor

0 2

3

4

5 6 7 8 9 Course session (weeks)

10

11

12

4 Results

25

are calculated for each participant. The scores indicate a kind of note-taking performance ability. For the following analysis, all participants were divided into two groups: high assessment scores and low assessment scores, using the average. The high and low groups consist of 30 and 23 students, respectively.

4.3 Effectiveness of Student’s Characteristics for Note-Taking Assessment Various factors of student’s characteristics may affect note-taking performance. Some relationships between them have been confirmed in the results for a blended learning environment [50]. In this study, the effectiveness of note-taking in a fully online learning environment is investigated. First, mean factor scores of personality are compared between the high and low note-taking assessment groups. The results are summarised in Table 2. There are significant differences in these scores for “Agreeableness” and “Conscientiousness”. Two of the five personality factors may affect note-taking behaviour. Next, mean second factors of information literacy are compared. The second factors are “operational skills” and “attitude” which are extracted using secondary factor analysis [44]. There is a significant difference in the second factor between high and low note-taking assessment groups. Again, the secondary factor consists of several factors, and there are some significant differences in the original factor scores. Mean factor scores of learning experience are also compared. Although there is no significant difference in the first factor (Overall evaluation of e-learning experience), there are significant differences in the two remaining factor scores (Learning habits and Learning strategies). Mean scores of the high group are higher than means of the low group. Students in the high group possess both good learning habits and effective learning strategies, such as the right tactics to master the academic requirements of the course. The factor scores of note-taking behaviour between the two groups of notetaking assessments are also summarised and compared in Table 2. For the first factor, “Recognising note-taking functions”, the score of the high group reaches 4 out of 5 points, as they know the functions of taking notes well, while the score of the low group stays in the middle of the scale. There is a significant difference between the two groups. For the second factor, “Methodology of utilising notes”, the score of the high group stays in the middle of the scale, but the score is significantly higher than the score of the low group. There is no significant difference in scores for the third factor, “Presentation of notes”. Both scores remain in the lower part of the scale. The students did not object to the presentation and collective review of their notes by their classmates. Finally, test scores, which are online test scores, weekly test scores and final exam scores, are compared between the two groups. These test scores are calculated as

26

Impact of Learner’s Characteristics and Learning Behaviour on Learning …

Table 2 Comparing means of metrics between high and low levels of Note-taking score [Mean(STD)] Note-taking score Metrics Personality Extroversion (IPIP-1) Agreeableness (IPIP-2) Conscientiousness (IPIP-3) Neutroticism (IPIP-4) Openness to experience (IPIP-5) Information Literacy Operational skills (IL-1) Attitude (IL-2) Learning Experience Overall evaluation of e-learning (LE-F1) Learning habits (LE-F2) Learning strategies (LE-F3) Note-taking skills Recognising note-taking functions (NT-F1) Methodology of utilising notes (NT-F2) Presentation of notes (NT-F3) Test scores Online tests (OT) Weekly tests (WT) Final exam (FE)

High(30)

Low(23)

Sign.

2.70(0.70) 3.43(0.53)

2.68(0.51) 3.06(0.50)

n.s. p < 0.05

3.31(0.59)

2.83(0.64)

p < 0.01

2.83(0.87) 3.17(0.62)

2.75(0.87) 3.00(0.46)

n.s. n.s.

3.43(0.53)

3.15(0.69)

n.s.

3.04(0.74)

2.64(0.60)

p < 0.05

3.36(0.74)

3.05(0.61)

n.s.

2.80(0.93)

2.13(0.79)

p < 0.01

3.43(0.80)

2.91(0.76)

p < 0.05

4.05(0.54)

3.31(0.83)

p < 0.01

2.96(0.88)

2.19(0.73)

p < 0.01

2.49(0.84)

2.41(0.82)

n.s.

98.25(4.93) 62.65(11.09) 64.61(9.39)

96.40(4.44) 48.73(14.22) 54.58(11.01)

n.s. p < 0.01 p < 0.01

individual averages across the course. There are significant differences in test scores between high and low groups, except with online test scores. As mentioned above, online test scores are the final scores students record when they are satisfied with their scores after repeated trials. Therefore, there are no differences. The differences in scores of weekly tests and final exams show that note-taking behaviour contributes to learning performance.

4 Results

27

Table 3 Correlation coefficients between note-taking behaviour, learner characteristics and test scores Metrics NT-F1 NT-F2 NT-F3 Note score IPIP-F1 IPIP-F2 IPIP-F3 IPIP-F4 IPIP-F5 IL-Skills IL-Attitude Experience-F1 Experience-F2 Experience-F3 Online test Weekly test Final exam

– 0.45 0.40 – 0.27 0.55 0.46 0.29 0.51 0.44 0.33 –

– 0.29 0.47 – 0.35 0.47 0.54 0.28 0.55 0.42 0.36 – –

– 0.33 0.53 0.30 – 0.47 0.55 0.42 – 0.40 0.38 – –

– – 0.28 – – – – – – – 0.31 0.58 0.46

Bold: p < 0.01, others: p < 0.05, -:n.s

4.4 Effectiveness of Note-Taking To confirm the detailed relationship between student’s learning performance and scores related to note-taking performance, a correlation analysis is conducted and the correlation coefficients are summarised in Table 3. Significant coefficients are displayed in bold font. For personality, these three factors of note-taking behaviour strongly correlate with “Agreeableness” and “Conscientiousness”, as Table 2 confirms. Additionally, “Neuroticism” correlates with the NT-F3 score, and “Openness to Experience” correlates with both NT-F1 and NT-F2. For information literacy scale scores, all three factors of note-taking behaviour strongly correlate with both secondary factors of information literacy, skills and attitude. For learning experience, both NT-F1 and NT-F2 correlate with factor scores of the learning experience. The sum of assessment scores for note-taking correlates with mean scores of online tests (r = 0.31), weekly confirmation test scores (r = 0.58) and with final exam scores (r =0.46). Since note assessments originate with note-taking performance, these results suggest that student’s note-taking performance affects their learning performance, while their personal characteristics are related to their note-taking behaviour. As good note-taking performance may help test scores, this suggests that the contents of the notes contribute to the test scores. Three factors of note-taking behaviour, (NT-F1) Recognising note-taking functions, (NT-F2) Methodology of utilising notes and (NT-F3) Presentation of notes are examined to determine their correlational relationships with student’s characteristics. Also, the sum of assessment scores for note-

28

Impact of Learner’s Characteristics and Learning Behaviour on Learning …

taking correlates with NT-F1 (r = 0.34; p < 0.05) and NT-F2 (r = 0.30; p < 0.05) except NT-F3 (r = 0.02).

4.5 Impact of Student’s Characteristics on Note-Taking Behaviour Using Causal Analysis The causal relationships between student’s characteristics and note-taking behaviour are determined using a structural equation modelling technique. To build a model, results of the above correlational analysis were used. The results of correlation analysis suggest that note-taking behaviour has significant correlations with “Agreeableness”, “Conscientiousness”, and “Openness to Experience”. Also, both secondary factors of information literacy correlate with note-taking behaviour, and are employed in the model. A causal model with correlational paths shows the results in Fig. 3. In this diagram, arches represent correlations and arrows represent directional paths. An “e” indicates an error term. The independence of personality factors is widely recognised, but

IPIP-2

GFI=0.95, AGFI=0.80

0.29

IPIP-3

IPIP-5

(0.13)

(0.23)

IL-1

IL-2

(0.21) (0.50) (0.65)

0.30 (0.26) (-.15)

NT-F1

e

(-.27)

NT-F2

e

NT-F3

e

Fig. 3 Paths from Student’s characteristics to Note-taking behaviour (IPIP-2: Agreeableness, IPIP3: Conscientiousness, IPIP-5: Openness to experience, IL-1: Information literacy - Operational skills, IL-2: Information literacy - Attitude, NT-F1: Recognising note-taking functions, NT-F2: Methodology of utilising notes, NT-F3: Presentation of notes, ( ): Path coefficient is not significant)

4 Results

29

sometimes there are correlations between other factors [15]. These secondary factors of information literacy have been extracted using Promax rotation, to confirm that there are correlational relationships. For this model, path settings are repeatedly adjusted by trial and error until the above-mentioned paths have been optimised (GFI=0.95; AGFI=0.80). The coefficients for the correlations and the paths are summarised in the top left corner of Table 4. There are correlational relationships between 5 student characteristics, and the coefficients of these varied between 0.19 and 0.94. In particular, the coefficient between the two secondary factors of information literacy is the largest. The larger coefficients appear in paths between information literacy operational skills and both NT-F2 “Methodology of utilising notes” and NT-F3 “Presentation of notes”. Three personality factors affect factors of note-taking behaviour. Attitude as a factor of information literacy negatively influences NT-F2 and NT-F3, and positively influences NT-F1 “Recognising note-taking functions”.

4.6 Impact of Note-Taking Behaviour on Note Assessments and Test Scores The causal model in Fig. 4 shows the relationships between note-taking behaviour, note assessments and test score factors. Factors of note-taking behaviour shown in Table 1 were extracted using Promax rotation, and are correlated with each other. Therefore, correlational paths are established between factors of note-taking behaviour in Fig. 4. According to our observations of student’s learning activities, notes are used as a reference before and during online tests, weekly confirmation tests and final exams. The directional paths from factors of note-taking behaviour to test scores results may be logical. Also, the directional paths between tests occur logically. As a result, the index of this model’s Goodness-of-Fit is the highest in this paper, because it is simple and logical (GFI = 0.98; AGFI = 0.93). Both note-taking behaviour factors NT-F1 and NT-F2 positively affect the assessment of notes, and all test scores through NT-F3 negatively influence all of these variables except online test scores. The path model can be used to merge factors affecting the learning experience, and this modified model is optimised. The order of the two constructs mentioned in the modelling procedure is as follows: The course is taken at a university, though student’s note-taking behaviour has been developing since primary and secondary school. Therefore, note-taking behaviour also affects the learning experience. The index of Goodness-of-Fit decreased slightly from the GFI index for Fig. 4 (GFI = 0.94; AGFI = 0.76). By comparing path coefficients for test scores with factors of note-taking behaviour or factors of learning experiences, we see that coefficients for learning experience factors are larger than coefficients for note-taking behavioural factors. By comparing path coefficients of note-taking behaviour to test scores between two previously diagrams mentioned, most coefficients decrease when learning experience

–

(8)NT-F3

–

–

–

(13)OT

(14)WT

(15)FE

–

–

–

–

–

–

–

–

–

–

0.31

0.42

–

(0.26)

(0.19)

(3)

–

–

–

–

–

–

–

–

–

–

0.94

–

0.42

0.55

0.38

(4)

–

–

–

–

–

–

–

–

–

–

–

0.94

0.31

0.46

0.33

(5)

–

–

–

–

–

–

–

–

–

–

(0.26)

(0.21)

–

–

0.29

(6)

–

–

–

–

–

–

–

–

–

–

(–0.26)

(0.65)

0.13

(0.23)

–

(7)

–

–

–

–

–

–

–

–

–

–

(–0.27)

(0.65)

–

0.30

–

(8)

–

–

–

–

–

–

–

–

(–0.15)

–

(–0.54)

1.14

–

(0.25)

(–0.16)

(9)

(1)IPIP-2: Agreeableness, (2)IPIP-3: Conscientiousness, (3)IPIP-5: Openness to experience, (4)IL-1: Information literacy - Operational skills, (5)IL-2: Information literacy - Attitude, (6)NT-F1: Recognising note-taking functions, (7)NT-F2: Methodology of utilising notes, (8)NT-F3: Presentation of notes, (9)LE-F1: Overall evaluation of e-Learning experience, (10)LE-F2: Learning habits, (11)LE-F3: Learning strategies, (12)NT-A: Note-taking assessments, (13)OT: Online tests, (14)WT: Weekly tests, (15)FE: Final exams ( ): Path coefficient is not significant, -: No signicant path exists

–

–

–

–

(12)NT-A –

–

–

–

–

–

–

(7)NT-F2

–

-

–

(6)NT-F1

0.46

0.55

(11)LEF3

0.33

(5)IL-2

–

0.38

(4)IL-1

(0.26)

–

(0.19)

(3)IPIP-5

–

(10)LEF2

0.54

(2)IPIP-3

0.54

(2)

(9)LE-F1

–

(1)IPIP-2

(1)

Table 4 Standardised path coefficients using all variables

–

–

–

–

–

–

–

–0.26

0.24

(0.17)

(–0.44)

0.76

(0.18)

–

–

(10)

–

–

–

–

–

–

–

–

–

0.28

(0.58)

(–0.44)

0.33

(0.19)

(–0.17)

(11)

–

–

–

–

–

(0.18)

(0.26)

–0.33

0.40

0.33

–

–

–

–

–

(12)

–

–

–

0.31

–

–

–

–

–

–

–

–

–

–

–

(13)

–

–

0.22

0.46

–

0.29

(0.18)

(–0.16)

(–0.16)

–

–

–

–

–

–

(14)

–

0.60

–

–

0.23

–

–

(–0.16)

–

–

–

–

–

–

–

(15)

30 Impact of Learner’s Characteristics and Learning Behaviour on Learning …

4 Results

31 FE (0.13)

e (-.13)

0.62

e

GFI=0.98, AGFI=0.93

0.46

NT-A

WT (0.15)

e 0.25 (-.13)

(0.13)

OT

e

(-.23)

0.30

(0.18)

0.31 0.33

NT-F1

NT-F2 0.57

NT-F3 0.45

0.38

Fig. 4 Paths from Note-taking behaviour to Note assessments and Test scores (NT-F1: Recognising note-taking functions, NT-F2: Methodology of utilising notes, NT-F3: Presentation of notes, NT-A: Note-taking assessments, OT: Online tests, WT: Weekly tests, FE: Final exams, ( ): Path coefficient is not significant)

factors are introduced. This suggests that the contribution of learning experience to test scores is larger than the contribution of note-taking behaviour.

4.7 Unified Model Some causal relationships are discussed in the above sections, though an overall model is needed to explain the causal relationship between student’s characteristics of note-taking behaviour, learning experience, note assessments and test scores by merging all of the partial models together. The optimised final model is shown in Fig. 5, where some paths have been removed to aid optimisation. The index of Goodness-of-Fit gets worse (GFI = 0.84; AGFI = 0.65), but its occurrence remains possible. Path coefficients are summarised in Table 4. These results suggest the following: Student’s characteristics affect note-taking behavioural factors and learning experience factors, but they do not affect note-taking assessments and test scores. Note-taking behavioural factors merely affect test scores while learning experience factors positively affect test scores. Note assessments are affected by both note-taking behaviour and learning experience factors, as are

32

Impact of Learner’s Characteristics and Learning Behaviour on Learning … e LE-F1

e

e

LE-F2

LE-F3

FE

IPIP-2

e IPIP-3

e

IPIP-5

NT-A

WT

e

IL-1

IL-2

OT

e

GFI=0.84, AGFI=0.65 NT-F1

NT-F2

e

e

NT-F3

e

Fig. 5 IPIP-3: Conscientiousness, IPIP-5: Openness to experience, IL-1: Information literacy— Operational skills, IL-2: Information literacy—Attitude, NT-F1: Recognising note-taking functions, NT-F2: Methodology of utilising notes, NT-F3: Presentation of notes, LE-F1: Overall evaluation of e-Learning experience, LE-F2: Learning habits, LE-F3: Learning strategies, NT-A: Note-taking assessments, OT: Online tests, WT: Weekly tests, FE: Final exams)

tests scores. Therefore, many factors positively affect test scores via note-taking assessments. According to the results, as certain factors of student’s characteristics also affect test scores, these characteristics should be considered when a study support system is developed. Also, the improvement of note-taking behaviour may positively affect the learning experience and test scores due to the assessment of student’s notes. A set of instructions to enhance student’s note-taking behaviour should be developed and introduced. The development of this procedure will be a subject of our further study.

5 Discussion In this study, note-taking was introduced into a fully online course to permit detailed examination of student’s learning activities. First, note-taking performance during a fully online course was insufficient and the level of note-taking that was rated as “Fair” notes was the highest. This suggests that most students simply reproduced the contents of slides in their notes. Every time students studied, they used downloaded slide files. This condition may be equivalent

5 Discussion

33

to one using “guided notes”, which promote note-taking [38]. The survey results showed that note-taking was not promoted by these materials. The possible reasons for this may be a lack of note-taking behaviour or a dependency on using digital files and not taking notes. To measure note-taking behaviour, survey inventories were developed and three factors were extracted, as follows. Factor 1: NT-F1 “Recognising note-taking functions”, Factor 2: NT-F2 “Methodology of utilising notes”, and Factor 3: NT-F3 “Presentation of notes”. As participants’ factor scores were relatively low, their consciousness of their note-taking behaviour might have been insufficient. In the causal analysis regarding note-taking, the coefficients of causal paths from these factor scores to note-taking assessments were significant, while the coefficients of causal paths from note-taking assessments to test scores of both online and weekly tests were insignificant, however. The Goodness-of-fit index (GFI) of the causal paths in Fig. 4 is high, so that the above relationship may be stable while the results agree with the previous studies regarding insisting importance of note-taking, even when an ICT environment is used. Also, the causal paths illustrate effect-spreading from note-taking behaviour to note-taking assessment and scores of weekly tests and final exams. This suggests that learning performance such as test scores may be improved when better note-taking behaviour is developed, as a means of raising factor scores and taking “Good” notes. Note-taking assessments did not directly affect final exam results, though coefficients of causal paths from some factors and scores of weekly test to scores of the final exam exist. Figure 5 show that scores of the final exam are directly affected by factor scores of note-taking behaviour and the learning experience. Again, these factors may be key to overall performance. Figure 3 indicates that both factors of note-taking behaviour and learning experience were affected by some factors of personality and information literacy. This causal model is also stable regarding GFI values, and the contribution of some factors to student’s characteristics has been confirmed in regards to note-taking activities. These results confirm the following points. In this study, the above-mentioned possible causal paths were extracted, and the above-mentioned factors need to be taken into account in order to improve learning performance in fully online courses. To maximise learning performance in a fully online learning environment, the development of better note-taking behaviour and a better learning environment for university courses are important. Characteristics of individual students also need to be considered. Other psychological factors such as learning efficacy may affect note-taking behaviour, of course. These relationships should be examined in a future survey. The effectiveness of these programs should be confirmed, and will be a subject of our further study. According to these results, a support system and educational improvements are required to improve learning activity during a fully online course. These issues will also be subjects of our further study. Acknowledgements This research was partially supported by the Japan Society for the Promotion of Science (JSPS) Grant-in-Aid for Scientific Research (B-22300281: 2010–2012).

34

Impact of Learner’s Characteristics and Learning Behaviour on Learning …

Part of this study (Nakayama, Mutsuura, & Yamamoto, 2012) was presented at IEETel 2012: 3rd International Workshop on Interactive Environments and Emerging Technologies for eLearning, as a part of the 11th International Conference on Trust, Security and Privacy in Computing and Communication in Liverpool, UK, June 25–27, 2012. The authors would like to thank the reviewers for their comments.

References 1. Nakayama M, Mutsuura K, Yamamoto H (2014) Impact of learner’s characteristics and learning behaviour on learning performance during a fully online course. Electr J e-Learn 12(4):394–408 2. NIME (2005) National Institute of Multimedia Education: E-learning in higher education: conditions for success. National Institute of Multimedia Education, Chiba, Japan 3. OECD-CERI (2007) Giving knowledge for free: the emergence of open educational resources. OECD, Paris, France 4. Hill P (2012) Online educational delivery models: a descriptive view. Educ Rev November/Dcember:85–97 5. Gaebel M (2014) Moocs: massive open online courses. http://www.eua.be/Libraries/ Publication/MOOCs_Update_January_2014.sflb.ashx 6. Tyler-Smith K (2006) Early attrition among first time e-learners: a review of factors that contribute to drop-out, withdrawal and non-completion rates of adult learners undertaking elearning programmes. MERLOT J Online Learn Teach 2(2):76–85 7. Nakayama M, Kanazawa H, Yamamoto H (2009) Detecting incomplete learners in a blended learning environment among Japanese university students. Int J Emerg Technol Learn 4(1):47– 51 8. Seaton DT, Nesterko S, Mullaney T, Reich J, Ho A (2014) Characterizing video use in the catalogue of MITx MOOCs. eLearning Pap (37):33–41 9. Seaton DT, Bergner Y, Chuang I, Mitros P, Pritchard DE (2014) Who does what in a massive open online course? Commun ACM 57(4):58–65 10. Kiewra KA (1985) Students’ note-taking behaviors and the efficacy of providing the instructor’s notes for review. Contemp Educ Psychol 10:378–386 11. Kiewra KA (1989) A review of note-taking: the encoding-storage paradigm and beyond. Educ Psychol Rev 1(2):147–172 12. Kiewra KA, Benton SL, Kim SI, Risch N, Christensen M (1995) Effects of note-taking format and study technique on recall and relational performance. Contemp Educ Psychol 20:172–187 13. Kobayashi K (2005) What limits the encoding effect of note-taking? a meta-analytic examination. Contemp Educ Psychol 30:242–262 14. Tynajä P (1999) Towards expert knowledge? a comparison between a constructivist and a traditional learning environment in the university. Int J Educ Res 31:357–442 15. Nakayama M, Yamamoto H, Santiago R (2007) The impact of learner characteristics on learning performance in hybrid courses among japanese students. The Electronic Journal of e-Learning 5(3):195–206 16. Nakayama, M., Yamamoto, H., Santiago, R.: Online learning management and learners’ behavior: A case study of online learning in Japan. In: F. Lazarinis, S. Green, E. Pearson (eds.) Developing and Utilizing E-Learning Applications, chap. 9, pp. 155–174. Information Science Reference, Hershey, PA, USA (2011) 17. Nakayama, M., Mutsuura, K., Yamamoto, H.: Effectiveness of note taking activity in a blended learning environment. In: Proceedings of 9th European Conference on E-Learning, pp. 387– 393. Porto, Portugal (2010) 18. Nakayama, M., Mutsuura, K., Yamamoto, H.: Student’s characteristics for note taking activity in a fully online course. In: Proceedings of 10th European Conference of E-Learning, pp. 550–557. Brighton, UK (2011)

References

35

19. Dinevski, D.: Open educational resources and lifelong learning. In: Proceedings of the ITI 2008 30th International Confernece on Information Technology Interfaces, pp. 117–122. IEEE, Cavtat, Croatia (2008) 20. Bates A (2000) Managing Technological Change: Strategies for College and University Leaders. Jessey-Bass Publishers, San Francisco, CA, USA 21. Park JH, Choi HJ (2009) Factor influencing adult learners’ decision to drop out or persist in online learning. Educational Technology & Society 12:207–217 22. Cercone K (2008) Characteristics of adult learners with implications for online learning design. AACE J 16:137–159 23. Song L, Singleton ES, Hill JR, Koh MH (2004) Improving online learning: Student perceptions of useful and challenging characteristics. Internet and Higher Education 7:59–70 24. Ueno, M.: Data mining and text mining technologies for collaborative learning in an ilms "samurai". In: ICALT, p. 0. IEEE, Joensuu, Finland (2004) 25. Nakayama, M., Santiago, R.: Learner Characteristics and Online Learning, pp. 1745–1747. Encyclopedia of the Sciences of Learning, Part 12. Springer Sciences+Business Media (2012) 26. Cronbach LJ, Snow RE (1977) Aptitudes and Instructional Methods -A Handbook for Research on Interactions-. Irvington Publishers, Inc., New York, USA 27. Dick W, Carey L, Carey JO (2005) The Systematic Design of Instruction. Allyn and Bacon, Boston, MA 28. Lim DH, Kim H (2003) Motivation and learner characteristics affecting online learning and learning application. Journal of Educational Technology System 31:423–429 29. Dabbagh N (2007) The online learner: Characteristics and pedagogical implimentations. Contemporary Issues in Technology and Teacher Education 7:217–226 30. Prinsen F, Volman ML, Terwel J (2007) The influence of learner characteristics on degree and type of participation in a cscl environment. Br J Edu Technol 38:1037–1055 31. Weener P (1974) Note taking and student verbalization as instrumental learning activities. Instr Sci 3:51–74 32. Meter P, Yokoi L, Pressley M (1994) College students’ theory of note-taking derived from their perceptions of note-taking. J Educ Psychol 86:323–338 33. Piolat A, Olive T, Kellogg RT (2005) Cognitive effort during note taking. Appl Cogn Psychol 19:291–312 34. Makany T, Kemp J, Dror IE (2009) Optimising the use of note-taking as an external cognitive aid for increasing learning. Br J Edu Technol 40:619–635 35. Nye PA, Crooks TJ, Powley M, Tripp G (1984) Student note-taking related to university examination performance. High Educ 13:85–97 36. Penn State Learning: Listening and note taking survey. http://penstatelearning.psu.edu/ resources/study-tips/note-taking/survey 37. Tran TAT, Lawson M (2001) Students’ procedures for reviewing lecture notes. Int Electron J 2:278–293 38. Austin JL, Lee M, Carr JP (2004) The effects of guided notes on undergraduates students’ recording of lecture content. J Instr Psychol 31:314–320 39. Austin JL, Lee M, Thibeault M, Carr JE, Bailey J (2002) Effects of guided notes on university student’s responding and recall of information. J Behav Educ 11:243–354 40. Trafton GJ, Trickett SB (2001) Note-taking for self-explanation and problem solving. HumanComputer Interaction 16:1–38 41. Mayer RE, Moreno R, Boire M, Vagge S (1990) Maximizing constructivist learning from multimedia communications by minizimizing cogmitive load. Journal of Edcational Psychology 91:638–643 42. Tam M (2000) Constructivism, instructional design, and technology: Implications for transforming distance learning. Educational Technology & Society 3(2):50–60 43. Moos DC (2009) Note-taking while learning hypermedia: Cognitive and motivational considearations. Comput Hum Behav 25:1120–1128 44. Nakayama, M., Yamamoto, H., Santiago, R.: Impact of information literacy and learner characteristics on learning behavior of japanese students in on line courses. International Journal of Case Method Research & Application XX(4), 403–415 (2008)

36

Impact of Learner’s Characteristics and Learning Behaviour on Learning …

45. Goldberg L (1999) A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality Psychology in Europe 7:7–28 46. IPIP: A scientific collaboratory for the development of advanced measures of personality traits and other individual differences (2001). http://ipip.ori.org 47. Fujii Y (2007) Development of a scale to evaluate the information literacy level of young people -comparison of junior high school students in japan and northern europe. Japan Journal of Educational Technology 30(4):387–395 48. Srivastava, S.: Measuring the big five personality factors (2013). http://psdlab.uoregon.edu/ bigfive.html 49. Toyoda H (2007) KYO BUNSAN KOUZOU BUNSEKI [AMOS HEN]. Tokyo Syoseki, Tokyo, Japan 50. Nakayama M, Mutsuura K, Yamamoto H (2011) Evaluation of student’s notes in a blended learning course. International Journal of New Computer Architectures and their Applications 1(4):1080–1089

Lexical Analysis of Students’ Learning Activities During the Giving of Instructions for Note-Taking in a Blended Learning Environment Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto

Abstract Note-taking activity affects students’ learning performances in blended learning courses, which consist of face-to-face sessions and online learning materials. To promote the effectiveness of note-taking, a lecturer gave students instructions during the course. According to the results of a lexical analysis of the contents of notes taken by students, the lecturer’s instructions had a significant effect on some of the indices of features of notes taken. This effectiveness can be observed when the lecturer gives instructions as opposed to not giving them. Also, the relationships between students’ characteristics and indices of features of the content of notes taken were analysed. Keywords Note-taking · Blended learning · Lexical analysis · Learning performance · Students’ characteristics

1 Introduction The modern learning environment using information communication technologies (ICT) offers various types of learning experiences for university-level education and for informal methods of learning [2]. Those environments are easy for students to learn, and their learning processes are not simple. To maintain learning effectiveness in the online learning environment, the analysis of students’ actual learning activities is absolutely necessary. For monitoring the learning situation, students’ access logs of online materials are often analysed; these approaches are sometimes effective [3] and sometimes not [4]. This shows that an effective extraction of data features is required.

Originally published in International Journal of Information and Education Technology, Vol. 6, No. 1, pp. 1–6, 2016 [1]. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Nakayama (ed.), Note Taking Activities in E-Learning Environments, Behaviormetrics: Quantitative Approaches to Human Behavior 11, https://doi.org/10.1007/978-981-16-6104-4_3

37

38

Lexical Analysis of Students’ Learning Activities During the Giving of Instructions …

Though the online learning environment is currently a requirement for many universities worldwide, the student behaviour through the environment needs to be evaluated, since it implies to present their learning performance. These analyses are currently being conducted in the Massive Open Online Course (MOOC) environment [5, 6]. Regarding this analytical trend, note-taking activity has always been analysed, as even in the online learning environment note-taking is a key learning activity [7–10], since it stimulates constructive learning [11]. In addition, the relationship between some factors of note-taking and learning performance at universities worldwide has been identified in previous studies [9, 10, 12]. Again, note-taking is a time-honoured and commonly used skill [13]. The results of several surveys which the authors have conducted presented notetaking behaviour contribution to learning performance. In addition to those results, the effectiveness of instructions about note-taking skills given was measured [14]. According to the results, both learning performance and the contents of notes may be improved when students recognise good note-taking skills, as explained by a competent instructor. To confirm this hypothesis, two experimental surveys were conducted. The following topics are addressed in this paper: • How does instruction regarding note-taking improve the contents of notes? • How do students summarise the contents of their notes are according to these instructions? • How do students’ individual characteristics contribute to their note-taking activities? To respond to these questions, a lexical analysis of the contents of notes taken by students was conducted.

2 Method 2.1 Courses The surveys were conducted over 2 years during Blended Learning courses at a Japanese university. The subject was Information Network and the courses were Bachelor-level credit courses. The course consisted of weekly face-to-face sessions for 15 weeks [15]. Participants were also encouraged to take an online test (OT) for each session outside of the lecture room, as a function of the learning management system (LMS). Also, a final exam (FE) was given at the end of the course.

2 Method

39

2.2 Note-Taking Instructions All the participants were required to present their notebooks in order to track the progress of their learning. In most sessions, the lecturer reviewed and assessed the contents of notes for 14 weeks. The contents were evaluated using a scale, and the sum of the scores is defined as the individual note-taking assessment score (NT-A). To determine the possibility of improving note-taking activities by having the lecturer give instructions, two survey conditions were developed: with and without instructions. The first-year course was conducted without any instructions being given or feedback about notes taken, and this condition is defined as “without instruction”. The second-year course was conducted twice, with instructions concerning notetaking techniques and the display of examples of good notes shown at the beginning and mid-point of the course. This condition is defined as “with instruction”. The valid number of participants is 32 for without instruction and 24 for with instruction.

2.3 Characteristics of Students Students’ characteristics have an impact on their learning activities. Some indices were surveyed previously, during our prior studies. The constructs are Personality [16, 17], Information Literacy [18] and Learning Experience [19]. Additionally, an inventory of note-taking skills was surveyed to extract three factor scores. Personality: The personalities of students were measured using a public domain item pool, the International Personality Item Pool (IPIP) inventory [17]. This questionnaire can produce a five-factor personality model [16], and measure scores of the five components: “Extroversion” (IPIP-1), “Agreeableness” (IPIP-2), “Conscientiousness” (IPIP-3), “Neuroticism” (IPIP-4) and “Openness to Experience” (IPIP-5). Information Literacy: Information literacy inventories were defined and developed by Fujii [18]. The survey consists of 32 question items, and 8 factors are extracted. These 8 factors can be summarised as two secondary factors: Operational Skills (IL-1), and Attitudes towards Information Literacy (IL-2) [20]. Learning experience: Students’ online learning experiences were measured using a set of questions and three factors identified. Three factors are as follows: Factor 1 (LE-F1): Overall Evaluation of the e-learning Experience; Factor 2 (LE-F2): Learning Habits; Factor 3 (LE-F3): Learning Strategies [19]. Note-taking skills: A set of constructs for measuring the note-taking skills of students was surveyed. These note-taking skills may consist of note-taking abilities, attitudes and techniques, and original inventories of these have been developed by the authors [15, 21]. This construct consists of the following three factors: F1: Recognising note-taking functions; F2: Methodology of utilising notes; F3: Presentation of notes.

40

Lexical Analysis of Students’ Learning Activities During the Giving of Instructions …

2.4 Contents of Notes Taken Students’ notes were scanned and stored as image files, and the contents were read and recorded manually as electronic text files. Figures and tables were extracted. The lecturer’s hand-written notes to be presented to students during face-to-face sessions were also transformed into electronic text files. The texts of the students’ and the lecturer’s notes were lexically analysed using a Japanese morphological term analysis tool [22]. In this paper, the frequency of nouns in each session is calculated. A comparison of the frequencies in students’ notes and the lecturer’s presentation produces two indices, defined as follows [15, 23]. • Word ratio: the ratio between the number of terms written and the number of terms given (the number of terms students recorded vs. the number of terms the lecturer presented). • Coverage: the coverage ratio was calculated as a percentage of the number of terms recorded by students. As an additional lexical analysis, co-occurring noun terms were surveyed using the following procedure: For example, noun strings A-B and B-C were extracted from the string A-B-C. A pair of nouns appearing consecutively in a sentence is defined as a consequential noun, or a 2-gram.

3 Results 3.1 Relationship Between Word Ratio and Coverage Both word ratios and coverages were calculated for every session in which students took notes. They correlate with each other, as high coverage requires a higher word ratio in comparison with the previous survey [15, 23]. The relationships between the two experimental conditions, with and without note-taking instructions being given, are summarised in Fig. 1. The horizontal axis represents the word ratio, and the horizontal axis represents the coverage. Though there are some deviations in the scatter grams, an overall correlation relationship can be observed. The correlation coefficients are r=0.38 for with instruction and r=0.18 for without instruction. Notetaking instructions that can promote better note-taking behaviour have already been reported in previous surveys by the authors. To determine the effectiveness of giving instructions, the ratios between the first and the second halves of the sessions were compared since the lecturer advised students at the beginning and mid-point of the courses. The change in ratios is summarised in Fig. 2. The horizontal axis represents two groups of sessions in sequence, the horizontal axis represents mean ratios. The coverage remains at the same levels across the two session conditions regardless of whether or not instructions are given. The word ratio increases during the courses, with the increasing rates for with

3 Results

41

Fig. 1 Relationship between word ratio and coverage of students’ notes

1.0

Coverage

0.8

0.6

0.4

0.2 w/o instrcution with instrction 0 0

Fig. 2 Comparison of mean ratios of word ratio and coverage between the first and second halves of the course

2

4 6 Word ratio

8

10

2.5

Mean ratio

2.0

w/o instruction with instruction

1.5 Word ratio 1.0 Coverage

0.5

0.0 First half

Second half Sessions

instructions given higher than the ones for without instructions given. If the differences were influenced by the experimental conditions, the instructions affected the word ratios in students’ notes.

3.2 Comparison of Adjacency Matrices Both students’ notes and the lecturer’s presentation were converted to an adjacency matrix, which indicates the connections between nouns. Figure 3 contains an example of a matrix of a lecturer’s presentation (Session 13). Generally, students do not record all terms the lecturer presents, though they do record some related, original terms. Therefore, the relationship between lecturers and

42

Lexical Analysis of Students’ Learning Activities During the Giving of Instructions …

Fig. 3 An example of an adjacency matrix

Fig. 4 Relationship between two adjacency matrices

L

S

students in two adjacency matrices can be illustrated in Fig. 4. When students made notes using terms which were not mentioned by the lecturer, the number of terms was larger than the number of terms in the lecturer’s notes. The difference between the two matrices represents the behaviour of the note-takers. The differences can be mathematically measured as edit distances, otherwise known as Levenshtein distance. The distances for sub-matrices L and S were evaluated separately [24]. As a result, the two indices are defined as follows: • Additional distance means the sum of the number of additional nodes or edges in a matrix. • Insufficient distance means the sum of the number of reduced nodes or edges in a student’s matrix in comparison with the lecturer’s matrix. Both distances are influenced by the total number of terms in the lecturer’s presentation, so that the relative distances are calculated using the number of terms the lecture presented in each session. The relationship between the two distances is summarised in Fig. 5. The horizontal axis represents relative insufficient distance, and the vertical axis represents relative additional distance. The two lines illustrate the regression relationship between the two distances. Regarding these calculations, the correlation coefficients are r=0.45 for with instruction and r=0.04 for without instruction. The correlation can illustrate new terms written as replacements as additional distances instead of omitting these

3 Results

43

Fig. 5 Relationship between insufficient and additional distances Relative additional distance

6 w/o instrcution with instrction 4

2

0 0.2

Fig. 6 Comparison of mean distances across sessions Mean relative distance

3

0.4 0.6 0.8 Relative insufficient distance

1

w/o instruction with instruction Additional distance

2

1 Insufficient distance 0 First half

Second half Sessions

terms as being insufficient distances when the concurrent appearance of terms is considered. Both word rate and coverage show the number of terms the lecturer presented during the session which students took. The distance metric suggests that connections between terms create concepts, and term replacement activities may reflect better understanding of the contents of the information provided by the lecturer. To determine the effectiveness of the instructions given, mean distances across the sessions were compared. The results are summarised in Fig. 6, in the same format as in Fig. 2. When instructions were given to students, mean additional distances increased significantly in the second halves of the sessions. Again, the additional distance shows an additional recording of their own new terms, when the connection with the terms the lecturer presented is considered.

Lexical Analysis of Students’ Learning Activities During the Giving of Instructions …

Fig. 7 Relationship between coverage and insufficient distance

1.0 Relative insufficient distance

44

0.8

0.6

0.4 w/o instrcution with instrction 0.2 0

0.2

0.4 0.6 Coverage

0.8

1

3.3 Relationship Between Term Ratio and Distances of Adjacency Matrices Both word ratio and coverage are easily calculated using term frequency, without considering the co-appearance of other terms. To confirm the degree of occurrence, the relationships between them were analysed. First, the relationship between coverage and insufficient distance was evaluated. When students correctly recorded terms in their own notes or recorded term networks, the insufficient distances are shorter, though the coverage increases. But both coverage and insufficient distance increase when students chose to replace the presented terms with others. These relationships are illustrated in Fig. 7. The horizontal axis represents coverage and the vertical axis represents insufficient distance. The plots illustrate the total number of negative correlation relationships. The correlation coefficients are calculated as r=-0.51 for with instruction and r=-0.33 for without instruction. The absolute value for with instruction is larger than the one for without instruction. Instructions being given can make participants pay greater attention to recording term network terms. Second, the relationship between word rate and additional distance is evaluated. The word ratio reflects students’ writing activities. The additional distances also increase when these terms are linked with other terms. The relationship is illustrated in Fig. 8. The horizontal axis represents word rate and the vertical axis represents additional distance. As the regression lines in the figure illustrate, they are linearly correlated, and the correlation coefficients are r=0.97 for with instruction and r=0.90 for without instruction. Regarding the slopes of the regression lines, recording new words contributes to increases in additional distances when instructions have been given. These results show that note-taking instructions for students can encourage them to create term networks.

3 Results

45

Fig. 8 Relationship between word rate and additional distance Relative additional distance

25 w/o instrcution with instrction

20

15

10

5

0 0

2

4 6 Word ratio

8

10

3.4 Causal Relationship Between Learning Performance and Students’ characteristics The above analyses confirm that giving note-taking instructions improves the effectiveness of students’ note-taking activities. Previous studies suggest that a participant’s characteristics may also affect their performance. To examine the effectiveness of the lecturer’s instructions and the participant’s characteristics, a causal analysis using a structural equation modelling technique was introduced. There are many indices of students’ characteristics, and some factors were extracted step by step, using correlation analysis. All the parameters were estimated using structural equation modelling (SEM) software (AMOS). As a result, a causal path can be created, as in Fig. 9. The paths consist of some factors of personality, information literacy, note-taking skills, learning experience and learning performance, including note-taking scores. Regarding the indices of fitness of the model (GFI: Goodness of Fitting index) (GFI=0.80, AGFI=0.64, RMSEA=0.08), a possible relationship is exhibited. The arrows indicate paths between variables, and path coefficients with and without note-taking instructions being given are indicated. Non-significant coefficients are indicated using parentheses. By comparing the coefficients between with and without instruction, significant differences between the two paths are revealed: IPIP3 (Conscientiousness) to NT-F2 (Methodology of utilising notes), and LE-F2 (Learning habits) to NT-A (Note-taking assessment). Though the coefficients are significant for the condition without instruction, some coefficients changed to being not significant for the condition with instruction. The instructions given about note-taking techniques may affect the note-taking activity to a greater extent than the students’ own characteristics. This result is evidence of improvement in students’ note-taking activities due to the lecturer’s instructions. There are no significant changes in the coefficients for the final exams (FE), though there are some significant changes in

46

Lexical Analysis of Students’ Learning Activities During the Giving of Instructions … e

LE-F1

LE-F2

LE-F3 (-.08) / (0.17)

0.47

)

.10

(0 .18) /

0.3

(0

0.32 / 0.45

2)

(0.3

6) /

(0.2

FE

0.29 / 0.63

** 6) .2 (0

(0 6/

.33)

IPIP-2

e

)/ 25 (-.

2) /

(-.0

e

e

e -.40 / -.40

0.47 / 0.58 (0.21) / (0.12)

IPIP-3

(0.2

8) /

55

)

/(

0.

IL-2

4)

1 (0.

/(

0.

05

NT-A

(0.26) / (0.18)

OT

0.34 / (0.05) 12

)

e

0.33 / (0.30)

)**

(0.13) / 0.41

0.40 / (0.07) (0. 00 )/ (0. 33 )

w/o instruction / with instruction

)

.10

/ (0

(0.16) / (0.02)

38

(-.01

0.

0.61 / 0.45

0.

NT-F1

NT-F2

NT-F3

e

e

e

GFI=0.80, AGFI=0.64, RMSEA=0.06

Fig. 9 Causal relationships between students’ characteristics and learning performance Table 1 Correlation coefficients between students’ characteristics and indices of text features of notes taken Word rate Coverage Insufficient D Additional D IPIP-3 IL-f IL-S NT-F1 NT-F2 NT-F3 LE-F1 LE-F2 LE-F3 NT-A

– – – 0.43 0.57 0.35 0.76 0.70

Upper column line: without instruction Lower column line: with instruction

– – – 0.53 -.41 0.52 0.36

0.43 – -.53 0.35 0.44 0.48 -

– – 0.41 0.56 0.79 0.56

3 Results

47

coefficients in relation to note-taking assessment (NT-A). How to give more effective instructions and suggestions related to this will be the subject of our further study. To determine the relationship between students’ characteristics and note-taking activity, correlation coefficients between these were calculated. The significant coefficients are summarised in Table 1. As mentioned in the above section, there are significant correlations between coverage and insufficient distance, and between word ratio and additional distance. Therefore, significant coefficients regarding those relationships appear simultaneously when instructions are provided. The results of correlation analysis suggest that students’ characteristics affect the indices of text features of notes taken. Regarding this result, a possible causal relationship should be identified, and effective instructions for better note-taking should be developed. These will be subjects of our further study.

4 Conclusion To improve students’ learning performance in a blended learning course, the effectiveness of the lecturer’s note-taking instructions was evaluated using lexical analysis of the contents of students’ notes. Regarding the results of our analysis, the following points have been extracted: 1. The results of lexical analysis of students’ notes suggest that word ratio, which means the ratio of the number of terms students recorded against the number of terms the lecturer presented, increased while the additional distances, which consider the co-appearance of terms, also increased in the with instruction condition. 2. The lecturer’s instructions had significant effect on the correlation relationships with the indices of note-taking features in compared with no instructions were given. 3. The causal paths between students’ characteristics and learning performance were analysed using the SEM technique. The lecturer’s instructions affected the causal relationship between students’ characteristics and note-taking. Also, students’ characteristics correlated with indices of the content of features of notes taken. More effective instructions and additional suggestions about improving learning activities will be subjects of our further study. Those relationships should be confirmed in other types of online learning environments, such as fully online courses and courses using social media. Also, the development of improving instruction using these results will be a subject of our further study. Acknowledgements This research was partially supported by the Japan Society for the Promotion of Science (JSPS), Grant-in-Aid for Scientific Research (B-26282046: 2014–2016).

48

Lexical Analysis of Students’ Learning Activities During the Giving of Instructions …

References 1. Nakayama M, Mutsuura K, Yamamoto H (2016) Lexical analysis of student’s learning activities during the giving of instructions for note-taking in a blended learning environment. International Journal of Information and Educational Technology 6(1):1–6 2. Nakayama M, Santiago R (2004) Two categories of e-learning in japan. Educational Technology Research & Development 53(2):100–111 3. Ueno, M.: Online outlier detection for e-learning time data. IEICE Trans. J90-D, 40–51 (2007) 4. Nakayama M, Kanazawa H, Yamamoto H (2009) Detecting incomplete learners in a blended learning environment among japanese university students. International Journal of Emerging Technology in Learning 4(1):47–51 5. Seaton, D.T., Nesterko, S., Mullaney, T., Reich, J., Ho, A.: Characterizing video use in the catalogue of MITx MOOCs. eLearning Papers (37), 33–41 (2014) 6. Seaton DT, Bergner Y, Chuang I, Mitros P, Pritchard DE (2014) Who does what in a massive open online course? Communication of the ACM 57(4):58–65 7. Kiewra KA (1985) Students’ note-taking behaviors and the efficacy of providing the instructor’s notes for review. Contemp Educ Psychol 10:378–386 8. Kiewra KA (1989) A review of note-taking: The encoding-storage paradigm and beyond. Educ Psychol Rev 1(2):147–172 9. Kiewra KA, Benton SL, Kim SI, Risch N, Christensen M (1995) Effects of note-taking format and study technique on recall and relational performance. Contemp Educ Psychol 20:172–187 10. Kobayashi K (2005) What limits the encoding effect of note-taking? a meta-analytic examination. Contemp Educ Psychol 30:242–262 11. Tynajä P (1999) Towards expert knowledge? a comparison between a constructivist and a traditional learning environment in the university. Int J Educ Res 31:357–442 12. Nye PA, Crooks TJ, Powley M, Tripp G (1984) Student note-taking related to university examination performance. High Educ 13:85–97 13. Weener P (1974) Note taking and student verbalization as instrumental learning activities. Instr Sci 3:51–74 14. Nakayama, M., Mutsuura, K., Yamamoto, H.: Effectiveness of instructional suggestions for note-taking skills in a blended learning environment. In: Proceedings of 12th European Conference on E-Learning, pp. 333–339. Nice, France (2013) 15. Nakayama M, Mutsuura K, Yamamoto H (2011) Evaluation of student’s notes in a blended learning course. International Journal of New Computer Architectures and their Applications 1(4):1080–1089 16. Goldberg L (1999) A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality Psychology in Europe 7:7–28 17. IPIP: A scientific collaboratory for the development of advanced measures of personality traits and other individual differences (2001). http://ipip.ori.org 18. Fujii Y (2007) Development of a scale to evaluate the information literacy level of young people -comparison of junior high school students in japan and northern europe. Japan Journal of Educational Technology 30(4):387–395 19. Nakayama M, Yamamoto H, Santiago R (2007) The impact of learner characteristics on learning performance in hybrid courses among japanese students. The Electronic Journal of e-Learning 5(3):195–206 20. Nakayama, M., Yamamoto, H., Santiago, R.: Impact of information literacy and learner characteristics on learning behavior of japanese students in on line courses. International Journal of Case Method Research & Application XX(4), 403–415 (2008) 21. Nakayama, M., Mutsuura, K., Yamamoto, H.: Causal analysis of student’s characteristics of note-taking activities and learning performance during a fully online course. In: Proceedings of 2012 IEEE 11th International Conference on Trust, Security and Privacy in Computing and Communication, pp. 1924–1929. Liverpool, UK (2012) 22. MeCab: Yet another part-of-speech and morphological analyzer. http://mecab.sourceforge.net

References

49

23. Nakayama, M., Mutsuura, K., Yamamoto, H.: Visualization analysis of student’s notes taken in a fully online learning environment. In: Proceedings of 16th International Conference of Information Visualisation, pp. 434–439. Monperier, France (2012) 24. Nakayama, M., Mutsuura, K., Yamamoto, H.: A note taking evaluation index using term networks in a blended learning environment. In: Proceedings of eighth international conference on complex, intelligent and software intensive systems, pp. 486–490. Birmingham, UK (2014)

Note-Taking Evaluation Using Network Illustrations Based on Term Co-occurrence in a Blended Learning Environment Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto

Abstract Note contents taken by students during a blended learning course were evaluated to improve the quality of university instruction. To conduct a quantitative comparison of the contents of all notes for effective instruction from lecturer to students to occur, the contents were mathematically compared and evaluated using two ways of summarizing the frequency of term co-occurrences. In order to evaluate visually the differences between the contents of notes taken, an adjacency matrix and Levenshtein distance metrics were employed to represent noun co-occurrence in the notes. In comparing notes between the lecturer and students, insufficient distance and additional distance are defined and measured for all participants during the course. In the results, students recorded additional nouns to replace the nouns given by the lecturer, and introduced new nouns of their own as substitutes as well. There are significant correlational relationships between the above mentioned distances and the ratios, such as the word ratio and coverage, which have been defined in previous studies. Also, summarizing ways of co-occurrence frequency did not affect any of the relationships. Finally, possible applications for visualizing learning activities of the lecturer and students are discussed. Keywords Blended learning · Note assessment · Term co-occurrence · Adjacency matrix · Distance

1 Introduction Educational methodology using Information Communication Technology (ICT) is spreading to various teaching styles, and in particular, this trend is becoming significant in courses at higher education institutions [2]. Regarding this development, education assessment procedures are required to improve the educational effective-

Originally published as International Journal of Distance Education Technology, Vol. 14, No. 1, pp.77–91, 2016. [1]. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Nakayama (ed.), Note Taking Activities in E-Learning Environments, Behaviormetrics: Quantitative Approaches to Human Behavior 11, https://doi.org/10.1007/978-981-16-6104-4_4

51

52

Note-Taking Evaluation Using Network Illustrations …

ness of ICT. Even in the most modern learning environment, known as Massive Open Online Courses (MOOCs), student’s learning activities have been surveyed in order to improve learning performance [3, 4]. Conventionally, note-taking activity can be used as a significant index of the learning process [5, 6] since the activity promotes constructivist learning [7]. There is also a relationship between note-taking activity and learning performance [8, 9]. as reviews of the contents of student’s notes taken to affect their autonomous learning [5, 10]. In an assessment of the contents of student’s notes taken during online learning courses, various factors were analyzed, including the contribution of student’s characteristics, the contribution of note-taking toward test scores, and the terms recorded in student’s notes [11–14]. In particular, text analysis of student’s notes has revealed how much information students can extract from the contents of a lecture presented during one session of a blended learning course [12, 13]. These analyses show the degree of content transformation between the lecturer’s presentation and the student’s notes, with regards to the assessment of student’s notes. The term frequencies in notes taken by students may represent their note-taking activity. The information about concurrent term frequencies can indicate meanings recorded in notes since most descriptions consist of short sentences. To extract semantic information such as the connections between terms, network analysis is often used [15]. During the analysis, the network can be illustrated as a graph which is a method of presentation similar to the concept mapping method [16]. The visualized information can be used for efficient evaluation because graph illustrations contain quantitative information in addition to textual analysis of the notes. This procedure can provide a technique for the comparison of the contents of notes, including the comparison of semantic structures [17]. However, there are still some questions, such as the analysis of the contents of notes taken and a comparison of the methodologies used to study term co-occurrences [15]. It is not easy to determine whether the concepts the lecturer has presented are recorded in student’s notes. Though the contents of notes taken can be compared using features of texts, it is not clear whether the same benefit of analysis for texts can be obtained when two major approaches are employed, such as bi-directional term cooccurrence or sequential co-occurrence. These detailed approaches are discussed in a later section. Therefore, the analytical methodologies used to compare the contents of notes taken should be studied. In this paper, the analytical visualization procedures used for concurrent occurrences of terms in the lecturer’s and student’s notes are compared using sequential terms such as two-word patterns and all two-word connections between terms. Also, the development of evaluation methodologies using the term co-occurrences is assessed in order to extract concepts and linked information, which are presented by the lecturer and recorded in notes taken by students. The purpose of this paper is to determine the effectiveness of illustrating term networks visually using two methods to evaluate metrics of graph presentations. The following topics are addressed in this paper:

1 Introduction

53

• The procedures used to illustrate a network of terms which represent co-occurrences of concepts in notes taken are compared between the two procedures, namely, two sequential terms and all connections between terms in a sentence. Visualization and comparison techniques are also developed. • The assessment of the degree of reproduction of term networks as concepts using graph metrics, and the evaluation of the relationships between text features and graph metrics are conducted. For these purposes, the contents of notes which have been previously reported are reanalyzed. The detailed methodologies used are described in the following section.

2 Method 2.1 Blended Learning Courses A note-taking survey was conducted during an information networking system course, which was a blended learning course using a distance education system in a bachelor level program at a Japanese university. The course consisted of weekly face-to-face sessions with students and online tests of student’s self-directed learning across 15 weeks. The content of the course has been reported in studies published previously [12, 13]. The valid number of participants who have presented all of their notes is 20 students.

2.2 Contents of Notes Taken The fundamental procedure for extracting descriptions of notes has been reported in one of our previous studies [12]. All students who participated in the course surveyed were asked to present their notes taken during the face-to-face sessions every week. Students took notes voluntarily in their own handwriting, without instruction or feedback. Student’s notes were collected each week, scanned, and stored in a PC as image files. The contents of notes taken by students were manually converted into electronic text from these images in terms of human reading. Figures and nontextual information were excluded. Therefore, some texts are sentences, and the rest are words and symbols. Also, the lecturer reviewed and assessed each student’s notes. The individual content of students’ notes was graded and classified manually by the lecturer as to whether the notes were “Good” or “Fair”. The lecturer’s handwritten notes, which were displayed on a screen during faceto-face sessions, were also transformed into electronic text. In total, all of the lecturer’s notes, and those recorded by 20 participants across 13 weeks of sessions, were collected [12].

54

Note-Taking Evaluation Using Network Illustrations …

2.3 Text Analysis of Notes Taken All texts mentioned above were analyzed to extract every co-occurring technical and non-technical noun term, including proper nouns, using a Japanese morphological term analysis tool (MeCab) [19]. The part of the grammar of every term was labeled by the tool. The features of the number of terms extracted, the course sessions and the content assessments of notes taken were previously summarized in Fig. 1 [12, 18]. In the figure, the overall number of terms in notes was compared between lecturer and students across course sessions. The horizontal axis shows the number of terms the lecturer presented in each course session, and the vertical axis shows the means of terms students have written down. The error bar shows the standard error of the mean. The number indicates a series of course sessions. The means are calculated for “Good” and “Fair” note takers, respectively. Also, two types of ratios have been introduced to evaluate note-taking performance [13] as follows: • Word ratio: the ratio of the overall number of terms students wrote in their notes to the number of terms the lecturer presented in a session. • Coverage: the percentage of terms students actually recorded that coincide with terms which the lecturer presented. In this paper, meaningless terms were excluded from analysis in order to extract the co-occurrence relationships between noun terms. Term co-occurrence in nouns is summarized in two ways, which are illustrated in Fig. 2. Here, A to D are nouns in the order extracted from a text. Jin [15] suggested two ways of doing this: the sequential bi-grams are recognized as two-word patterns [20], which are indicated as “2-grams” in Fig. 2, and all combinations of two extracted

Fig. 1 Comparison of number of terms presented by lecturer and in notes taken by students [12, 18]

300

Number of terms (Students)

250 10

200

7

8

150

9

100

11 2 6

3 4

5 12

50 13

1

Good Fair

0 0

50

100 150 200 250 Number of terms (Lecturer)

300

2 Method Fig. 2 Two ways of 2-grams from a term sequence

55

2-gram

A-B-C-D multi 2-gram nouns are recognized and are indicated as “Multi 2-grams”. The procedural details of a “2-gram” are as follows: the number of nouns in a text is noted as N (i = 1, . . . , n), and the ith noun is noted as pi and a set of nouns as P. Here, 2-grams are uniquely noted as ( pi , pi+1 ), and multi 2-grams as a set of ( pi , pi+1 ), . . . , ( pi , pn ). This can be extended to all notes taken, where the total number of sets of nouns is Q, and the number of nouns is M, and the co-occurrence of nouns (q) can be noted as a relationship (qi , q j ), i, j = 1, . . . , M. All relationships between the two nouns are summarized as a matrix. This matrix is known as an adjacency matrix, which shows nodes and edges as defined in graph theory. Using this matrix, a graph can be produced using an appropriate software tool such as an “iGraph” [21]. Though there are two ways to present the term co-occurrence, it is not clear which way is better to illustrate the relationships. The effectiveness of illustrations using the two ways is compared in this paper.

3 Results 3.1 Comparison of Graphs of Note-Descriptions First, the visualization procedure for the information contained in notes taken is confirmed using term co-occurrence data. The term co-occurrence frequencies were counted using the procedure mentioned in the above section, thus two sets of adjacency matrices were generated for each session. Figure 3 shows an example of an adjacency matrix of the lecture presented in the 12th course session, where the frequencies were summarized according to the frequency of two-word patterns. If the use of the term was terminated, all components in the line are 0. The size of the matrices depends on the number of terms. In the matrix, “InternetJ” means Internet in Japanese, since the course is taught in Japanese. Also, as the connections were extracted from handwritten images, some phrases with no meaning were extracted. The adjacency matrix can be converted into a visualized graph using the component patterns. The content of a lecture presented in the same session was converted

56

Note-Taking Evaluation Using Network Illustrations … T1

T2

T3

T4

T5

T6

T7

T8

T9

T 10

T 11

T 12

T 13

T 14

T 15

T 16

T 17

T 18

T 19

T 20

T 21

T 22

T 23

T 24

T 25

T 26

T 27

T 28

T 29

T 30

T 31

T 1 : Example

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 2 : Onself

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

T 3 : Access

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 4 : Address

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

T 5 : Application

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

T 6 : InternetJ

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 7 : DN S

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 8 : Domain

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 9 : Hotmail

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 10 : IM AP

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 11 : IP

0

0

0

1

0

T 12 : Internet

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 13 : Layer

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 14 : M ail

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 15 : N ame

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

T 16 : Of f ice

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 17 : P OP

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

T 18 : P ost

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 19 : System

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 20 : Server

0

1

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 21 : W eb

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

T 22 : M ailbox

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 23 : bit

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 24 : com

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

T 25 : Electro

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 26 : Human

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 27 : N ame

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

T 28 : N et

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T 29 : P rotocol

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

T 30 : www

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

T 31 : yahoo

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

Fig. 3 An adjacency matrix of the lecturer presentation (12th session) Oneself Layer

Mailbox

Server

IMAP

Internet

Name

Office

Mail Post

Access

Protocol POP Application Network bit

Name

Domain

DNS

address

System

IP Example

Hotmail

Electro

Human com

Web www

Fig. 4 An example of a lecturer’s map (2-gram)

yahoo

Internet

3 Results

57 address www com IP yahoo bit Name Domain

human Internet

System Access

DNS Mail

IMAP

Network Internet

Electro Hotmail Web

Protocol

Application

Example Layer Server Mailbox

Name

Post Oneself Office POP

Fig. 5 Another example of a lecturer’s map (Multi 2-gram)

into an adjacency matrix in two ways, 2-gram and Multi 2-gram. Visualized graphs are shown in Fig. 4 for 2-gram and Fig. 5 for Multi 2-gram. Figure 4 shows some chain networks of terms and Fig. 5 shows some clusters of terms since they connect to each other. Both figures illustrate term networks of nouns which have been presented by the lecturer in a session. The figures reflect the procedures used to count term frequency. For the illustration of term networks, the features used to count term frequency should be considered for the purpose of analyzing the content of lecturer’s and student’s notes. When the graph illustrating procedure is applied to students notes, two adjacency matrices and graphs are generated using 2-grams and Multi 2-grams. Student’s notes were classified into two grades, “Good” and “Fair”, as mentioned above. These typical graphs are compared in Figs. 6 and 7 in response to Fig. 4, which represents a lecturer’s presentation using 2-grams. Figure 6 shows a good example, which includes a lot of additional writing. Figure 7 shows a fair example, which includes insufficient writing.

58

Note-Taking Evaluation Using Network Illustrations … bit

Hotmail

Address

Example

Access

Cache

Electro

Mail

Web to

IP

Internet Network

Peer IMAP

P

Content Protocol System Name Domain

Oneself

Name Server

Application

Mailbox Mirror

Data

Trafic

Layer com yahoo www

Local Route RFC

Port

DNS

POP Authority

Internet

Post Network Office

Fig. 6 An example of a student’s map using 2-gram way (Insufficient distance = 23, Additional distance = 49)

3.2 Two Metrics of Distance Between Two Graphs Both Figs. 6 and 7 summarize the contents of student’s notes. Figure 6 shows many additional terms which are supplemental or replacement terms instead of terms the lecturer presented. Figure 7 shows some omissions of terms the lecturer presented. These differences can be indicated as differences in components of the adjacency matrices, and are shown in Fig. 8. Figure 8 shows a merged matrix as a direct sum of two matrices (L, S) and it illustrates the relationship between the adjacency matrices of terms in notes taken by students (S) and those presented by the lecturer (L). Here, matrix A is defined as a logical disjunction matrix converted from matrices L and S, and illustrated in Fig. 8. All matrices are square matrices, and sizes are Sl for L, Ss for S, and Sa for A. When students made notes using terms which were not mentioned by the lecturer, the number of terms was larger than the number of terms in the lecturer’s notes. Therefore, in Fig. 8, size of the adjacency matrix (S) for student is larger than size of the matrix (L) for lecturer with regards to the number of terms. However, if students omitted some information presented by the lecturer, co-occurrences of terms the lecturer used were not observed, though the size of S may be comparable to the size of L. Here, the size differences of the matrices is not important; the differences between the matrices may show some of the problems with this method of transferring knowledge.

3 Results

59 Address

IP

bit

Human

Mailbox

Web oneself

com Electro yahoo

www server

Example

Internet

Yahoo

Access

P DNS

Mail Network Internet

System

Content

Name IMAP Domain POP

Post

Office

Fig. 7 Another example of a student’s map using 2-gram way (Insufficient distance = 17, Additional distance = 13) Fig. 8 Relationship between adjacency matrices of lecturer (L) and students (S)

L

S

The differences can be measured as edit distances, otherwise known as Levenshtein distances. The distances for sub-matrices L and S were evaluated separately. The distances which are based on sub-matrix S suggest that students recorded additional terms, so these distances represent a degree of positive improvement. On the other hand, the distances which are based on the sub-matrix L indicate insufficient quantities of terms recorded, so these distances represent the degree of failure to transfer terms. The two indices are defined using matrix A, and components ai j and matrix L, and components li j , as follows:

60

Note-Taking Evaluation Using Network Illustrations …

• Additional distance means the number of additional nodes or edges shown on a student’s graph (i,Slj=1 (ai j |li j = 0) + i,Saj=l+1 ai j ). Additional distances appear in two cases. In the case of Figure 3, which represents a lecturer’s presentation such as L, additional distance can be accumulated when a student has written down “POP server”, which produces an additional connection between T18 and T20. The other case is the creation of a new connection when a student originally writes down “cache memory” which consists of two terms in matrix S that are not included in matrix L. • Insufficient distance means the sum of the number of reduced nodes or edges on a student’s graph in comparison with the lecturer’s graph (i,Slj=1 (li j − ai j )). If a student omitted “IP address” in Fig. 3, there is no connection between T11 and T4 even though a student has recorded them separately. As a result, the length of the edge of matrix A was reduced. If a student omitted the term “IP”, the number of nodes was also reduced. Insufficient distance causes these reductions to accumulate. For example, the distances for Figs. 6 and 7 were calculated. The distances for Figs. 4 and 6 are “insufficient” = 23, “additional” = 49, and the distances between Figs. 4 and 7 are “insufficient” = 17, “additional” = 13. Regarding these results, student’s additional descriptions are sometimes based on their own terms instead of the terms presented by the lecturer, in addition to the terms which are written in the lecturer’s notes. Therefore, an increase in insufficient distance indicates selfdirected writing of notes and promotes the writing of different terms which produce an additional distance. Examples in Figs. 6 and 7 show the differences. The two types of distances are related to each other [17], and the relationships for the two calculation procedures are confirmed, as shown in Fig. 9. Two results for 2gram and Multi 2-gram overlap in Fig. 9. The horizontal axis represents insufficient distance and the vertical axis represents additional distance. As the figure shows, there are correlational relationships between the two distances (Pearson correlation coefficient: 2-gram: r = 0.60; Multi 2-gram : r = 0.63). Both results illustrate the same relationship. These results confirm a tendency for additional descriptions to replace the terms presented by the lecturer. In the next analysis, the relationship between term frequencies and co-occurrence frequencies was examined. The relationship between the insufficient distance and the coverage rates was analyzed to confirm whether the insufficient distance is related to the coverage of student’s notes in comparison with the lecturer’s presentation. To produce good notes, which means building a closed term network with many terms, students should be encouraged to take notes which include comprehensive terms instead of terms the lecturer presented. Figure 9 provides confirmation that insufficient distance produces in an increase in additional distance. The relationship is shown in Fig. 10. The horizontal axis represents coverage and the vertical axis represents the relative values of insufficient distance since the distance depends on the total number of terms the lecturer presented during a session. The plots show negative correlation (2-gram: r = −0.50, Multi: r = −0.49). Most plots overlap in Fig. 10. Since the insufficient distances are shorter when the coverage

3 Results

61

Fig. 9 Relationships between insufficient distance and additional distance in two ways

Fig. 10 Relationships between coverage and insufficient distance in two ways

increases, this causes negative correlation relationships to become more prominent. This means that the adjacency matrix for notes taken by students approaches the concepts presented by the lecturer. In this analysis, there is little difference in the summarizing procedures between 2-gram and Multi 2-gram. To examine the activity of making additional notes, the relationship between the additional distance and the total number of terms students recorded, such as word ratios, is summarized. This scattergram is shown in Fig. 11. The horizontal axis represents the word ratio, the vertical axis represents the relative additional distance, which has been standardized using the total number of terms the lecturer presented. As shown in Fig. 11, most plots are centered around 1.0 for word ratio, since the word ratio represents a relative number of terms taken by students. The relative additional distance monotonically increases with the word ratio. Therefore, there are strong correlation relationships (2-gram: r = 0.94, Multi: r = 0.86).

62

Note-Taking Evaluation Using Network Illustrations …

Fig. 11 Relationships between word ratio and additional distance in two ways

The word ratio shows a high value when the number of terms the lecturer presents is small, since some students take notes proactively. This activity affects additional distances. Therefore, deviations of both word ratio and additional distance occur when their values increase, although these plots are sparse. As a result, correlational coefficients are relatively high. In regards to the correlational relationships, the taking of additional notes may be based on the co-occurrences of nouns. This result shows that the terms students recorded in their notes are not taken down as separate terms; they are linked with other terms.

4 Discussion As the text analyzing procedure can extract features of the text structure, two types of term relationships, such as 2-gram and Multi 2-gram, may show the characteristics of the presentations. These procedures have been applied to the term analysis of notes taken, and the differences were then examined. For the procedure using Multi 2-gram, term co-occurrences were summarized as term connections with each other. Therefore, these connections are represented as clusters, as shown in Fig. 5. For the procedure using 2-gram, term relationships were illustrated as chain networks, and directional relationships are emphasized. The sequential relationships between terms may depend on the grammatical order of terms as a characteristic of the specific language used, and these points should be considered in advance. The above mentioned points are based on the selection of two text analytical procedures, so the benefits of using analysis should be taken into account. As these note-taking metrics can portray student’s learning activity, any means of visualizing this information is useful. Again, these analyses visually represent concepts and content clusters in notes taken. The visualized information may help students to reflect upon their own

4 Discussion

63

note-taking and learning activities in addition to better understanding the lecture’s presentations. Also, the lecturer can easily evaluate every student’s learning progress using this visualized information. In order to promote these evaluation and reflection techniques, an appropriate software application may be required. This development will be a subject of our further study. In regards to the relationship between term co-occurrence and term frequency, all frequencies using the Multi 2-gram procedure are larger than the frequencies using the sequential 2-gram technique. For the frequency base analysis, there is no difference between the two procedures. The information about the contents of notes taken by students using text analyzing techniques shows characteristics of note-taking activity and also of learning performance. A more effective method of indexing note-taking activity should be the development of a procedure to indicate student learning activity. Also, these techniques can be applied to text messages used in social media communication [22]. The possibility of utilizing this type of analysis for texts should be confirmed. These topics will also be subjects of our further study.

5 Conclusion The contents of notes were lexically analyzed and compared using two summarizing procedures in order to determine whether concepts presented by a lecturer could be transferred to students during a blended learning course which was a credit course at a Japanese university. The following points were examined in this paper: 1. The noun co-occurrences in students’ notes were illustrated mathematically in graph form and the distances between the lecturer’s presentation and the student’s notes were visually compared using graph mapping. In particular, co-occurrences were summarized as sequential 2-gram and Multiple combination 2-gram, and these features were then extracted. The difference in co-occurrence frequencies was quantitatively measured as Levenshtein distances and these distances were also analyzed. 2. The results of distance analysis showed that in addition to their having recorded some of terms from the lecturer’s presentation, additional descriptions were added by students and also often replaced the lecturer’s original descriptions with student’s own words. The summarizing procedures of term co-occurrence affected frequency or distance, but the relationships between the indices were not influenced. Therefore, the procedure should be accepted for the purpose of analysis. The development of methods of improving instruction in blended learning courses using these results will be a subject of our further study.

64

Note-Taking Evaluation Using Network Illustrations …

Acknowledgements This research was partially supported by the Japan Society for the Promotion of Science (JSPS), Grant-in-Aid for Scientific Research (KAKEN, B-26282046: 2014–2016). This chapter is reprinted by permission of the IGI Global.

References 1. Nakayama M, Mutsuura K, Yamamoto H (2016) Note-taking evaluation using network illustrations based on term co-occurence in a blended learning environment. Int J Distance Educ Technol 14:77–91 2. Twigg C (2005) Course redesign improves learning and reduces cost. Technical report, The National Center for Public Policy and Higher Education. http://www.highereducation.org/ reports/pacore/core.pdf 3. Seaton DT, Nesterko S, Mullaney T, Reich J, Ho A (2014) Characterizing video use in the catalogue of MITx MOOCs. eLearning Pap (37):33–41 4. Seaton DT, Bergner Y, Chuang I, Mitros P, Pritchard DE (2014) Who does what in a massive open online course? Commun ACM 57(4):58–65 5. Kiewra KA (1985) Students’ note-taking behaviors and the efficacy of providing the instructor’s notes for review. Contemp Educ Psychol 10:378–386 6. Kobayashi K (2005) What limits the encoding effect of note-taking? A meta-analytic examination. Contemp Educ Psychol 30:242–262 7. Piolat A, Olive T, Kellogg RT (2005) Cognitive effort during note taking. Appl Cogn Psychol 19:291–312 8. Nye PA, Crooks TJ, Powley M, Tripp G (1984) Student note-taking related to university examination performance. High Educ 13:85–97 9. Kiewra KA, Benton SL, Kim SI, Risch N, Christensen M (1995) Effects of note-taking format and study technique on recall and relational performance. Contemp Educ Psychol 20:172–187 10. Kiewra KA (1989) A review of note-taking: the encoding-storage paradigm and beyond. Educ Psychol Rev 1(2):147–172 11. Nakayama M, Mutsuura K, Yamamoto H (2010) Effectiveness of note taking activity in a blended learning environment. In: Proceedings of 9th European conference on e-learning. Porto, Portugal, pp 387–393 12. Nakayama M, Mutsuura K, Yamamoto H (2011) Evaluation of student’s notes in a blended learning course. Int J New Comput Archit Appl 1(4):1080–1089 13. Nakayama M, Mutsuura K, Yamamoto H (2013) Effectiveness of note-taking skills and student’s characteristics on learning performance in online courses. In: Ivanova M, Nakayama M (eds) Proceedings of 4th international workshop, intaractive environments and emerging technologies for elearning (IEETeL 2013). CEUR, vol 991, pp 13–21. http://ceur-ws.org 14. Nakayama M, Mutsuura K, Yamamoto H (2013) Effectiveness of notes-taking content features on test scores in online courses. In: Proceedings of 2013 17th international conference on information visualisation, pp 451–456 15. Jin M (2009) Tekisuto deta no tokei kagaku nyumon. Iwanami shoten, Tokyo, Japan 16. Novak JD, Canas AJ (2008) The theory underlying concept maps and how to construct and use them. Technical report ihmc camp tools 2006-01 rev 01-2008, Florida Institute for Human and Machine Cognition. http://cmap.ihmc.us/Publications/ResearchPapers/ TheoryUnderlyingConceptMaps.pdf 17. Nakayama M, Mutsuura K, Yamamoto H (2014) A note taking evaluation index using term networks in a blended learning environment. In: Proceedings of eighth international conference on complex, intelligent and software intensive systems. Birmingham, UK, pp 486–490 18. Nakayama M, Mutsuura K, Yamamoto H (2012) Relationship between feature of note-taking contents and test scores in a blended learning environment. Jpn J Educ Technol 36(Suppl.):21– 24

References

65

19. MeCab: Yet another part-of-speech and morphological analyzer. http://mecab.sourceforge.net 20. Jackson P, Moulinier I (2002) Natural language processing for online applications - text retrieval. Extraction and categorization. John Benjamins Publishing, Amsterdam, Netherlands 21. iGraph: network analysis and visualization. http://cran.r-project.org/web/packages/igraph/ index.html 22. Nakayama M, Leh A, Santiago R (2014) A case study of the impact of instructional design on blogging and terms networks in a teacher-training course. In: Proceedings of ECSM2014, pp 328–334

Effectiveness of Students’ Note-Taking Activities and Characteristics of Their Learning Performance in Two Types of Online Learning Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto

Abstract Aspects of learning behavior during two types of university courses, a blended learning course and a fully online course, were examined using a note-taking activity. The contribution of students’ characteristics and styles of learning to the note-taking activity and learning performance were analyzed, and the relationships between the two types of courses were compared using causal analysis techniques. In addition, lexical analysis of the contents of notes taken was introduced. Features of notes taken, such as the number of terms, the word ratios of students’ notes and the degree of coverage of the lecturer’s notes were compared. The results of the evaluation of the two types of learning styles were summarized by determining the relationships between students’ characteristics and metrics of the contents of notes taken. The metrics were significantly different between the two learning styles. The contributions of students’ characteristics to learning performance were also different. These results provide points to consider for the design and organization of the two types of learning. Keywords Causal analysis · Learning performance · Lexical analysis · Note-taking · Online learning · Students’ characteristics

1 Introduction The Internet has made various new types of learning possible, and the flexibility of these types of learning is supported by Information Communication Technology (ICT). The most popular and frequently used style is e-learning, which solves the problems of time and distance. E-learning can be defined using two types of learning, known as “online learning or technology-enhanced learning (TEL)” which adheres to the basic tenets of face-to-face teaching [2]. In this paper, the focus is on one of the styles, which is known as “fully online learning” and does not involve face-to-face learning sessions. The other, known as “blended learning”, is a combination of online

Originally published as International Journal of Distance Education Technology Vol. 15, No. 3, pp. 47–64, 2017 [1]. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Nakayama (ed.), Note Taking Activities in E-Learning Environments, Behaviormetrics: Quantitative Approaches to Human Behavior 11, https://doi.org/10.1007/978-981-16-6104-4_5

67

68

Effectiveness of Students’ Note-Taking Activities and Characteristics …

learning and face-to-face instruction. Both learning styles are types of “e-learning”. E-learning permits the use of modern variations of learning activity, such as Massive Open Online Courses (MOOCs) [3, 4] and Flipped classrooms, which are based on blended learning [5]. However, learning performance in e-learning courses requires examination, as serious claims have been made about both their actual effectiveness and their return on investment. As the participant’s learning activities were analyzed to determine the level of achievement, improvements were not easy to carry out [6]. To maximize learning performance, learner’s participation in the online system has been monitored and analyzed as a topic of research in modern learning analytics [7]. Another approach is to focus on the learning process of participants, such as analyzing various records regarding a participant’s learning, or using text mining techniques to examine the note-taking activity [8]. This analysis is frequently used to assess their authenticity [9]. Therefore, the authors looked at participants’ note-taking activities [10] as well as the previous studies [11–13]. Note-taking is well known as a conventional index of learning progress [14–16] and learning performance [16–18]. Various aspects of the relationships between these two activities have been discussed, and the details will be summarized in the following section. Note-taking surveys were introduced to both courses studied, one a blended learning course and the other a fully online course. Both types are defined as “e-learning” in this paper, and are typical learning formats which use an online learning environment. Since the learning format of the two styles is significantly different, the relationships between the note-taking activity and learning performance differ, due to various factors which concern the individual participants. As the quantitative investigation of these points may help improve the design and organization of the courses, data surveys and modeling analysis of the two courses were conducted. In a previous study conducted by the authors, students’ characteristics affected the note-taking activity and learning performance during a fully online course [18], and the dependency of these relationships on learning style should be examined. In this paper, the following topics are addressed using surveys of participants in both blended and fully online courses: • Note-taking skill factors are created to compare participants’ performance between the two courses. • The causal relationships between participants’ characteristics and learning achievement are measured and compared between the two courses. • Features of notes participants took during the two courses are compared, to extract the differences in learning behavior between the two courses. • Relationships between learning performance and note-taking behavior are analyzed by considering participants’ characteristics, in order to recognize patterns in the learning behavior of participants.

2 Related Works

69

2 Related Works 2.1 Note-Taking “Note-taking” is conventionally believed to provide some positive benefits to students in terms of test scores. Note-taking is a common skill used in all types of learning activities [19]. In particular, contributions to examination performance have been widely studied [17]. The effectiveness of note-taking for various types of learning has been discussed [15, 20], since the note-taking activity is a summarization of the knowledge we have acquired [21, 22]. Also, the functions of note-taking and the cognitive efforts required during notetaking have been investigated in order to improve students’ scholastic achievement [15, 21]. In addition to the note-taking activity, as a part of the learning process the process of reviewing the contents of notes taken may help the recall and recognition of learned content. These activities may contribute to constructivist learning [23, 24]. While in practice note-taking skills are generally intended to be used by university students [25], the techniques are applicable to learning across a variety of subjects. In addition, some practical aspects of note-taking have been investigated and discussed [20, 26] and note-taking styles have been systematically classified [22]. Though the relationships between note-taking activities and learning performance have been emphasized [16, 17, 27], some studies have noted the limitations of the note-taking activity [28] and the necessity of providing appropriate instructions to students [11]. These studies suggest the need to consider conditional factors, such as student’s individual aptitudes or characteristics, and learning environment factors such as the use of face-to-face learning or technology-enhanced learning. For example, when the showing of slides is introduced to a lecture, students record the content as well as “guide notes”, which are a modified version of the instructor’s slides [29]. Using guide notes is an effective learning exercise [30]. Therefore, the method of instruction used may affect students’ note- taking behavior. Currently, some electronic note-taking systems which make the extraction of the written contents of notes easier have been developed to track participants’ learning progress [31]. Some improvements are still required [32].

2.2 Effectiveness of Learning Environment Modern learning systems provide some technological enhancements, such as multimedia or hyper-media learning resources, to the online learning environment. The environment influences learning in some areas, such as learner’s attitude, emotional factors, and cognitive workload for example. In the former, the relationships between self-regulated learning (SRL) and media-enhanced learning have been discussed [33]. Also, regarding the latter issue, the effectiveness of reducing the cognitive load of learning activities has been discussed [24, 33], and various design techniques have been developed and are being used in current e-learning systems which take

70

Effectiveness of Students’ Note-Taking Activities and Characteristics …

into consideration the multi-modality characteristics of human communications [34, 35]. Environmental factors of the two types of e-learning, blended learning and fully online learning, also affect the above-mentioned learning process. Blended learning uses face-to-face sessions which are conducted by instructors, and student’s selfpaced learning using web-based materials such as online tests. As a form of fully self-regulated learning, fully online learning proceeds at the learner’s pace. Therefore, the difference is not only in the learning environment but also in the degree of self-regulated learning. Here, note-taking is a key activity for both self-regulated learning and for minimizing the cognitive workload. The note-taking activity involves the process of selfreflection during the reviewing of the contents of notes [36], and the taking of notes reduces the cognitive load during learning and memorization [15, 21, 22]. Therefore, these phenomena are related to the above- mentioned factors. Since learning is a developmental process, an analysis of the details used in the process for distance learning should be analyzed [37]. The analysis of the contents of notes taken is often employed to track the progress of learning [13, 33, 38]. In particular, quantitative analysis of the textual content of notes taken is possible [12]. Though the influence of the learning environment has been assessed through experimentation, observation in natural settings is preferred, as various environmental factors contribute to learning behavior.

3 Method 3.1 Cohorts of Online Learning Courses Data collection was conducted during two online Bachelor-level information technology courses at a Japanese university. The teaching styles were a blended learning course and a fully online course; the subject was Information System. The same professor taught both as credit courses, as follows. Scores of all tests were recorded for individual evaluation. (A) Blended learning course (BL) This was a conventional distance learning course across 15 weeks which was supported using a two-way audio-visual communication system, as well as ordinary face-to-face sessions [39]. The lecturer displayed hand-written lecture content using an overhead projector, and no slide presentations. The style is the traditional lecture style, where the lecturer can freely respond to participants’ questions. The online learning environment, incorporating a learning management system (LMS), provided all participants of online tests with the ability to assess their answers immediately. The lecturer asked students to join online test sessions in order to encourage their self-directed learning outside of the class. Tests could be repeated, and only the final scores were used for final grades. After all the sessions were completed, final exams were conducted using conventional pencil and paper tests in a classroom. The total number of valid participants in the course was 40.

3 Method

71

(B) Fully online course (FO) This course was designed to emphasize the benefits of online learning. Its style offered participants self-paced learning outside of a classroom, using online materials. The content was completely different from a blended learning course, as all materials were designed for a fully online course and were available online. Therefore, participants could go online at any time and from anywhere, using a slide show lecture and the audio track which accompanies it [18]. The slide show can be stopped and any part repeated at any time. To manage their learning progress, participants were asked to join weekly test sessions in a classroom, in addition to taking online tests outside of the classroom. The total number of valid survey participants in the course was 53.

3.2 Characteristics of Participants As student’s characteristics impact their learning activity and learning performance [40, 41], several indices of these have been referred to, in order to understand behaviors such as personality [42, 43], Information Literacy [44] and the degree of Learning Experience [40]. These were introduced to both courses using common surveys. A survey was conducted at the beginning of the course. (A) Personality A five-factor model of personality, the “Big Five”, has been frequently used to recognize human behavior [42]. The scores were measured using an established questionnaire which is popularly known as the International Personality Item Pool (IPIP) inventory [43]. The five components are labeled “Extroversion” (IPIP-1), “Agreeableness” (IPIP-2), “Conscientiousness” (IPIP-3), “Neuroticism” (IPIP-4) and “Openness to Experience” (IPIP-5). (B) Information Literacy Activities using information technology are concerned with information processing abilities [44]. Overall performance may depend on individual information literacy. Fujii [44] developed a survey inventory to measure information literacy. Regarding the validation analysis of the inventory, the 32 question items were summarized as 8 factors such as interest and motivation, fundamental operational ability, information collecting ability, mathematical thinking (reasoning) ability, information control ability, applied operational ability, attitude, and knowledge and understanding. In addition, these 8 factors can be condensed into two major factors: operational skills (IL-1) and attitude toward information literacy (IL-2) [41]. (C) Learning Experience The degree of maturity of the learning activity for university-level education including online courses was measured using a set of 10 questions evaluated using the 5-point Likert scale. Three factors were extracted: Factor 1 (LE-F1)—overall evaluation of the e-learning experience, Factor 2 (LE-F2)—learning habits and Factor 3 (LE-F3)—learning strategies [40].

72

Effectiveness of Students’ Note-Taking Activities and Characteristics …

(D) Note-Taking Skills Students’ knowledge concerning note-taking skills was measured using a set of 5-point Likert scale inventories [18]. The inventories were created using question items from Cornell style notes [25] and questions from other previous studies. For the fully online course, three factors were extracted: NT-F1—Recognizing note-taking functions, NT-F2—Methodology of utilizing notes and NT-F3—Presentation of notes. This construct was applied to both the blended and fully online courses to determine the factor structure of note-taking skills.

3.3 Note-Taking Assessment To monitor individual learning progress, notes of all participants in both courses were gathered to evaluate their activity every week. The lecturer checked the notes and graded them using Good, Fair or Poor as levels of the overall assessment. Good and Fair were focused upon in the following analyses. The assessment guideline was the degree of sufficiency in comparison with the lecturer’s presentations [39]; all the participants were not informed of how the contents of notes were evaluated. Participants may have taken notes in their own usual styles. As the lecturer rated these notes subjectively, the grades were simplified into two levels. For example, if students reproduced the same information that was in the lecturer’s notes, a rating of “Fair” was given. If some additional information was written down or if the descriptions were well structured, a rating of “Good” was given. The sum of the weekly rating was calculated as note-taking assessment (NTA) in order to evaluate students’ note-taking performance. The individual note-taking assessment (NT-A) was defined using a sum of the rates, such as 2 points for “Good” notes and 1 point for “Fair” notes. Participants were divided into groups with high (High) or low (Low) levels of note-taking performance according to the sums of their assessment scores. According to NT-A, students were also classified into two groups consisting of high or low note-taking ratings, using the sums of the weekly ratings of notes taken. In addition to their overall assessment, note contents were lexically analyzed. Images of notes presented were scanned and stored on a PC, and all texts and terms were converted into computer-readable text. Lists of terms were extracted using a Japanese morphological analysis tool (MeCab) [39, 45]. The source of students’ notes, the lecturer’s hand-written presentation notes for face-to-face sessions in the blended learning course and the presentation notes for slides in the fully online course were also analyzed. Therefore, the number of terms students recorded in their notes may be related to the number of terms the lecturer presented. Two types of ratios were then defined, as follows [45]:

3 Method

73

• Word ratio: the ratio of written terms in comparison with the number of lecturer’s terms (the number of terms students recorded/the number of terms the lecturer presented). • Coverage: the coverage ratio was calculated as a percentage of the number of terms recorded by students. In these analyses, the coding of text for the lexical analysis of note-taking activities was limited by the number of participants, such that due to time restrictions 20 out of 40 were selected for the blended learning course and 39 out of 53 were selected for the fully online course.

3.4 Causal Analysis The measures in the above-mentioned sections are related to each other, in particular the effect of students’ characteristics on learning performance and note-taking assessment. The causal model was designed as a causal structure which shows the relationships between the above- mentioned measures in a fully online course [18], which was applied to both courses at the same time, using a simultaneous analysis in multi-population technique [46]. The causal model is based on path analysis using SEM [47]. First, correlation coefficients for all variables of the metrics mentioned above were calculated using correlation analysis. In regards to the magnitudes of the correlation coefficients, some relationships were able to be extracted. Second, the causality between the metrics is introduced. For example, personality affects note-taking activity, but the activity may not have an impact on personality. Therefore, path directions are given to illustrate connections between the relationships. In most cases, if there are no directional relationships, such as between one factor of personality and another, then the relationships are noted as correlations. Also, the model was created step by step. There are some indices to measure the degree of fitness of models, such as GFI, AGFI, RMSEA and others [47]. If some paths were inappropriate, these indices of fitness would worsen. When appropriate paths were implemented and the model could be explained reasonably using the significant values of the indices of fitness, the calculations would be terminated. If the causal direction was inappropriate, this influences the indices of overall fitness in the model. The causality may be assessed during the calculation process. The strengths of path relationships are noted as standardized path coefficients. The path coefficients are tested for their significance. Some paths may contribute to the fitness of the model for relationships overall, even if they were not significant. Therefore, all causal relationships should be examined using both path structures and their coefficients. The calculations of model validity were conducted using AMOS structural equation modeling software [46, 48].

74

Effectiveness of Students’ Note-Taking Activities and Characteristics …

a

b

100

100 Blended

Fully online NT: Good

80

60

NT: Fair

40

Percentage (%)

Percentage (%)

80

NT: Fair 60

40

NT: Good 20

NT: Poor

0

20 NT: Poor

0 2

3

4

5 6 7 8 9 10 11 12 13 Course session (weeks)

2

3

4

5

6

7

8

9

10

11 12

Course session (weeks)

Fig. 1 Grade percentages of note-taking assessments in a a Blended course [39], and b a Fully online course [18]

It is clear that various factors influence the learning activity. The metrics surveyed in this study are limited. Therefore, the results show some aspects of learning activity when any significant causal relationships were present.

4 Results 4.1 Note-Taking Assessment Overall note-taking assessment surveys were conducted every week during the two courses. The rates for “Good” and “Fair” notes are summarized in Fig. 1a for the blended learning course and Fig. 1b for the fully online course. The horizontal axis indicates survey weeks, and the vertical axis indicates the grade of the notes as a percentage. In total, the percentage of “Good” is highest in the blended learning course, while the percentage of “Fair” is highest in the fully online course. During the blended learning course, the lecturer suggested that students write down some of the terms received aurally, as well as note other indirectly given points through the use of hand gestures or repeated underlining. However, students in the fully online course were not encouraged to take notes. The differences in assessments of notes suggest that course environment variations exist between the two courses [39].

4.2 Confirmatory Factor Analysis of Note-Taking Skills The questionnaires about note-taking skills were originally developed to measure participants’ skills for the fully online course since their note-taking activities were

4 Results

75

Table 1 Questionnaire about note-taking skills and factor loading [49] No. Question item F1 1 2 3 4 5

I understand the syllabus summary of this course NT during sessions to understand the course contents NT during sessions to clarify the contents NT during sessions to review the contents later NT is for understanding the whole course not only the session topics 6 I understand well the contents of items in my NT 7 NT consists of what the teacher presented and talked about 8 I think about the meaning and importance of words during NT 9 I think about the relationship between items presented during NT 10 I use NT to revise the notes taken after the session 11 I use NT to write some additional information in the notes taken 12 I think about relationships between the content of the notes taken 13 Notes of surveyed contents are added to notes taken 14 Notes are taken so that other participants can understand the contents 15 Notes are taken so that even non-participants can understand the contents 16 Classmates are considered when notes are taken 17 I have NT skills F1: Recognizing note-taking functions F2: Methodology of utilizing notes F3: Presentation of notes Contribution ratio

F2

F3

0.55 0.79 0.73 0.53 0.81

−0.09 −0.05 −0.04 −0.01 −0.12

0.00 0.02 0.09 0.02 0.00

0.54 0.61

0.11 0.17

0.08 −0.17

0.58

0.18

0.07

0.57

0.24

−0.06

−0.04 −0.05

0.91 0.91

0.00 0.00

0.14

0.80

−0.02

0.02 0.13

0.59 −0.11

0.18 0.70

−0.01

−0.03

0.79

−0.06 −0.05 1.00 0.40 0.22 0.25

0.15 0.20

0.67 0.55

1.00 0.31 0.20

1.00 0.14

NT Note-taking

almost always at a low level [18]. The survey was then extended to the blended learning course, where it was confirmed that the factor structure of the fully online course was the same as that for blended learning. The merged data (N = 93) from the questionnaires for both courses was analyzed using Promax rotation, and the factor loading matrix for the three factors was extracted, as shown in Table 1 [18]. The three question items from the two courses were excluded. Regarding the structure of the three factors and the question items from which they were derived, three factor labels were selected: NT-F1—Recognizing notetaking functions, NT-F2—Methodology of utilizing notes and NT-F3—Presentation of notes. Correlational coefficients are summarized across the factors in Table 1.

76

Effectiveness of Students’ Note-Taking Activities and Characteristics …

Table 2 ANOVA of note-taking factor 1 (NT-F1: Recognizing note-taking functions) Source df SS MS F Prob. (A) Note group: High-Low 1 (B) Blended/Fully Online 1 Interaction (A × B) 1

5.90 0.73 1.29

5.90 0.73 1.29

13.60 1.70 3.00

p < 0.01 n.s. p < 0.10

4.3 Comparison of Factors Between Courses These factor scores, which were averages of scale values, were compared between the blended and the fully online course, and between the high and the low note-taking groups. The averaged factor scores are summarized in Fig. 2. As the means of first factor scores (NT-F1) are higher than the median in both note-taking groups, most participants understand the note-taking functions well. To confirm the differences between the factors mentioned above, two-way analysis of variance (ANOVA) was conducted. For example, the results for NT-F1 are summarized in Table 2. The factors of the high and low note-taking groups are significant ( p < 0.01), while the factors of the courses (blended and fully online) are not significant. For the second factor (NT-F2), the factors of note-taking groups are significant. Other factors are not significant across the three factors, however. These results suggest that there are no significant differences, such as in the mode of learning between blended and fully online courses. Thus, note-taking skills are common between the two courses. For all variables of constructs of student’s characteristics, identical analysis was conducted. In the results, similar statistical effects, such as the significance of the factor of the group (whether High or Low) was confirmed for the following variables: IPIP-3 (Conscientiousness), Information Literacy (IL-2: attitude), Note-Taking Skills

High

High

4.01

NT-F1

Low

3.74

4.07 3.32

Low

3.08

2.81 NT-F2

2.30

2.55 Blended

Fully online 2.49

2.78 NT-F3

2.41

2.27 5

4

3 Factor score

2

1

1

2

3 Factor score

4

5

Fig. 2 Factor scores for note-taking skills between two note-taking assessment groups in both a blended and a fully online course [49]

4 Results

77

(NT-F2: Methodology of utilizing notes), Learning Experience (LE-F2: Learning Habits), Online Test Scores, and Final Exam Scores [18]. These results suggest that most constructs were not influenced by course factors, i.e. whether blended learning or fully online, but the factor for the group was affected by the differences in the scores of the constructs.

4.4 Causal Analysis of Note-Taking Activity Between Blended and Fully Online Courses Regarding the relationships between student’s characteristics and their learning performance, a causal analysis was conducted for the fully online course [18]. The type of learning environment (blended or fully online course) may affect the relationship, as some types of behavior were influenced by participants’ characteristics. To confirm this, causal analysis of the sets of data from the blended and fully online courses was conducted. The variables are almost the same as in the previous analysis [18], and are concerned with participants’ characteristics (IPIP) including information literacy (IL), note-taking skill factors (NT), learning experience (LE), note-taking assessment (NT-A) and test scores (OT: Online test; FE: Final exams) [18]. The final results are summarized in Fig. 3. All the metrics are indicated to emphasize the relationships between them. The arc mean correlations with correlation coefficients and arrow lines indicate causal paths and path coefficients. The coefficients on the left side are for the blended course, and the coefficients on the right side are for the fully online course. Coefficients with ( ) mean no significant correlation.

e

e

e

LE-F1

LE-F2

LE-F3 (-.21) / 0.33*

(0.11) / (0.15)

0.35 / 0.26

0.32 / 0.38

0.05 / 0.33 0.61 / 0.57

(0.14) / 0.31 0.38 / 0.30

e -.35 / (-.15)

(0.23) / 0.26 NT-A

0.35 / (0.23)

0.14 / 0.46 IL-1

(0.24) / 0.41 e

0.34 / (0.02)

IPIP-3

FE

(-.28) / (0.14)

(0.15) / 0.42

IPIP-2 0.47 / 0.54

0.54 / 0.67

(-.20) / (-.14)

0.25 / 0.37

(0.21) / 0.40 0.34 / 0.39

0.32 / (0.21)

OT

(0.20) / -.32* e (0.08) / 0.28

0.56 / 0.47

IL-2 0.33 / 0.37 0.30 / 0.48

Blended / Fully Online NT-F1

NT-F2

NT-F3

e

e

e

GFI=0.80, AGFI=0.64, RMSEA=0.08

Fig. 3 Results of causal path analysis using both a blended and a fully online course

78

Effectiveness of Students’ Note-Taking Activities and Characteristics …

The overall fitness of this model is not good, but it may be acceptable (GFI = 0.80, AGFI = 0.64 and RMSEA = 0.08). Bold paths show that there are significant differences ( p < 0.05) between coefficients in the blended and fully online courses. They are NT-F3 (Presentation of Notes) to NT-A (note-taking assessment) and LE-F2 (Learning Habits) to FE (Final Exam Scores). In an overview of the causal relationship, students’ characteristics affect test scores by way of note-taking assessments. Therefore, note-taking is a key activity of both of the courses. For the fully online course, these path coefficients are significant, and the relationships are clearly recognizable. Examples are the relationship between learning habits and final exam scores, and the negative relationship between note presentation skills and note assessments, as the lecturer has focused on the volume of notes taken. In addition to these, coefficients for the fully online course are higher than are those for the blended course, along several paths. In the individual fully online learning environment, students’ characteristics may affect learning performance. The coefficients for the blended course are definitely higher than the ones for the fully online course along other paths, but the causal relationships are different between the two courses. These differences should have been considered when both courses were organized.

4.5 Features of Contents of Notes Taken Text analysis was applied to all note contents, and terms and the frequencies of terms were summarized, as mentioned in the above-mentioned section. The number of participants selected was 20 for the blended learning course, and 39 for the fully online course, and the participants’ notes were evaluated throughout the sessions. First, the number of terms students recorded was compared between the blended and the fully online course. Figure 4 displays these results, with the left side showing

300

300

Fully online

250 7

8

200 11

150

9

6

2

Number of terms (Students)

Number of terms (Students)

Blended

3 10 4

100

5 12

50

1

Good

13

250 14

200 13

150 9

100

5

50

4

Fair

7

12 10 8 3 2

11

5

Good Fair

0

0 0

50

100

150

200

250

Number of terms (Lecturer)

300

0

50 100 150 200 250 Number of terms (Lecturer)

300

Fig. 4 The number of lecturer-presented and student-recorded terms in a blended and a fully online course

4 Results

79

results for the blended course and the right side showing results for the fully online course. Figure 4 summarizes the relationships between the number of terms students took on the vertical axis and the number the lecturer presented on the horizontal axis. The number of terms the lecturer presented refers to the hand-written terms recorded on paper for the blended course, and in the lecturer’s slide presentation for the fully online course. The plots show every session for both Good and Fair note-taking groups. Error bars indicate standard errors. Most plots for the blended course are above the diagonal line, while most plots for the fully online course are below the diagonal line. This means that students recorded more terms in the blended course and fewer terms in the fully online course, in comparison with the number of terms the lecturer presented. The deviation in the number of terms of the blended course is larger than the deviation for the fully online course. There are few differences between the Good and Fair groups in the fully online course, but in the blended course the Good note takers remain at the same levels of the number of terms (around 150 words), while the Fair note takers write the same number of terms as presented by the lecturer. The relationships between the number of terms students took and the number of terms the lecturer presented were examined statistically, using analysis of covariance (ANCOVA), which tests the contribution of the three factors: the number of terms presented by the lecturer, the course (blended/fully online) and the students’ notetaking grade (Good/Fair). In the results, two factors, the course (F(1, 601) = 90.3, p < 0.01) and the number of terms (F(1, 601) = 382.6, p < 0.01), are significant, while the note- taking assessment is not a significant factor (F(1, 601) = 0.1, p = 0.73). The factor for note-taking grades is not significant for either course (Blended: F(1, 222) = 0.34; Fully online: F(1, 376) = 0.04). The interaction between courses and note-taking grades is also significant (F(1, 601) = 20.1, p < 0.01), however. The number of terms students recorded is significantly different between the two courses. Though the factor for note-taking assessments is not significant, the contribution of note taking assessments depends on the course. Both Word ratio and Coverage have been defined as indices of note-taking activity [45]. The relationships between word ratios and coverage between blended and fully online courses are compared in Fig. 5. The plot formats are the same as those shown in Fig. 4. The error bars show standard errors. Regarding the figures, the coverage for the fully online course is higher than the coverage for the blended course, though the word ratios are evenly distributed around 1.0, except in some sessions of the blended course. Also, note-taking grades affect the distribution of word ratios. These results suggest that participants reproduced the contents of slides in their notes during the fully online course, though participants took notes during the blended learning course in response to the lecturer’s emphasis. During the fully online course, the note-taking activity may have been restricted by the contents of the slide presentations, therefore, participants made an effort to reproduce the contents in their own notes. The contributions of the factors for coverage were statistically tested using ANCOVA, and are shown in Fig. 4. In the results, the factors for the course

80

Effectiveness of Students’ Note-Taking Activities and Characteristics … 1.0

1.0

Fully online

0.9

0.9

0.8

0.8 Coverage

Coverage

Blended

0.7

0.7 0.6

0.6

Good

Good 0.5

0.5

Fair

0.4 0

1

2 Word ratios

3

4

Fair

0.4 0.5

1

1.5 Word ratios

2

Fig. 5 Relationships between word ratios and coverage in a blended and a fully online course

(blended/fully online) (F(1, 599) = 41.1, p < 0.01), the note-taking assessment grade (Good/Fair) (F(1, 599) = 9.4, p + b

3.2.1

Performance Comparisons Between the First and Second Halves of the Courses

First, a simple multiple regression analysis using liner models for G(x) was introduced, in order to determine key variables and their contributions to the scores of final exams. During the analysis, a step wise method of selection was introduced to choose the variables which were significant. This procedure can optimize the model in order to calculate the rate of contribution, which is expressed as an R-squares. The results are summarized in Table 4. After taking into consideration the stimulation of note-taking instruction and the differences in the number of terms the lecturer presented in Fig. 1, four note-taking metrics were summarized as three conditions: means of the first and the second halves of the course, and the overall course. In the results, the effective variables and their contributions (partial R-squares) are summarized and compared between two conditions, such as data with instruction and data without instruction. In a comparison of an index of overall performance of the regressions between the two conditions, the R-squares for the condition with instructions are much higher than for the other condition. In regards to these results, the contributions of final exam scores can be explained using individual sets of variables during the course with instruction. Also, performance is better when variables from the first half of the course are employed. The contributions using multiple regression models are higher than are the ones for single regression models. Therefore, every index can contribute to the relationships. The lists of variables indicate that some variables of student’s characteristics were selected from the course without instruction, while note-taking features were included when instruction was given. In particular, four metrics of note-taking activity in the first session, and NT3 (presentation of notes) were selected for regression analyses of the first half of the course, and the first and second halves of the course.

The Possibility of Predicting Learning Performance using Features …

Fig. 2 R-square and RMSE of regression models between final exam scores and mean NT features of course sessions

1.0 With Instruction

Without Instruction

0.6

0.4

with

0.2

0.0 1

3.2.2

2

3

4

5

8 4 0

RMSE

R-square

0.8

without

98

6 7 8 9 10 11 12 13 14 Course session

Performance in Regards to the Progress of the Course

Since participants learn content and make improvements to their notes during each session, learning performance may be based on cumulative learning behavior. For example, the learning performance of the i th session may reflect overall learning activities from the first to i th session. As mentioned in the previous section, the four note-taking metrics and one of the note-taking skills (NT-F3) contributed to final exams scores (FE) such as x ∈ {W R, C V, AD, I D, N T − F 3 }, and temporal prediction performance in regards to the evaluation of cumulative indices. The contribution ratios of the variables were calculated as R-squares, and the accuracy of prediction was indicated as root mean square errors (RMSE). The temporal changes are summarized in Fig. 2. The contribution ratios stay around 0.2–0.4 without instruction. When instruction was made, the ratios increased with the number of sessions. During sessions 5–12, the ratios were distributed around 0.8, which was quite a high rate. In the last two sessions, the ratios decreased, though they were above 0.6. In addition to these improvements, after the fourth session RMSEs with instructions tended to be smaller than those for without instruction. These results suggest that the effectiveness of note-taking activity appears at an early stage in the course, and can be maintained until the end of the course.

3.3 Possibility of Prediction of Final Exams Scores In regards to the results of multi regression analysis, the scores of final exams reflect metrics of note-taking activity with instruction. The results of analysis show some significant relationships between the scores of final exams (FE) and the four metrics of note-taking activities and the one factor score for note-taking skills (NT-F3). The accuracy of prediction for FE scores was not evaluated precisely. Here, prediction accuracy means precision of estimation of scores of a novel participant using his

3 Results

99

Table 5 R-squares and RMSE between final exam scores and predictions across sets of selected feature variables Feature set R-squares RMSE Without Inst. With Inst. Without Inst. With Inst. NT-f∗ + 13 variables1 NT-f∗∗ + 2 variables2 NT-f∗ + 2 variables3 NT-f(first half means) NT-f(second half means) 3 variables4

0.06 0.17 0.18 0.04 0.02 0.04

0.07 0.64 0.72 0.36 0.02 0.08

6.6 5.9 5.9 6.7 6.9 6.6

4.4 2.8 2.5 3.6 4.9 5.1

NT-f ∗ Means for four features of NT in the first half sessions (1–7) 1 Four features and other 13 features of characteristics. 2 Selected features of NT-f, IPIP1, and NT3 3 NT-f, NT3, and a feature of NT in the second sessions (8–14) 4 IPIP2, LE-1, and a feature of NT in the first sessions

or her scores of metrics of note-taking behavior as an index of the capability to generalize. To estimate scores accurately, a support vector regression (SVR) technique was introduced as a more robust model. A Gaussian kernel was employed as the function (G) using the constant b. Prediction accuracy was evaluated using a leave-one-out procedure which estimated the scores of each individual using a model trained with the rest of the data. With this, the model training process and prediction process can be evaluated independently of each other.

3.3.1

Prediction Performance Using SVR

The computed function G(x) can provide estimates of the scores of final exams ( F˜E) after optimization training using data of participant that does not target any one specific individual. The actual calculation was conducted using a LIBSVM package [34]. To evaluate prediction performance, R-squares and prediction error RMSEs were calculated. The performance was tested under several conditions using selected variables. The results of some trials suggest that the four features of note-taking activity contribute to performance prediction. The results of several feature sets are summarized in Table 5. In comparing the performance of estimations between with and without note-taking instruction, the predictions were more accurate when note-taking instruction was given. The results of R-squares show that some note-taking instruction metrics can contribute to the prediction of FE (R 2 > 0.6), though the contribution of these metrics without instruction is low. Both RMSEs for the two sets of data are comparable with those which use multiple regression analysis, as Fig. 2 shows.

The Possibility of Predicting Learning Performance using Features …

Fig. 3 R-square and RMSE of prediction models with SVR between final exam scores and mean NT features of course sessions

1.0 With Instruction

Without Instruction

0.6

0.4

0.2

8 4 0

0.0 1

3.3.2

2

3

4

5

RMSE

R-square

0.8

with without

100

6 7 8 9 10 11 12 13 14 Course session

Prediction Performance of Course Progress

The possibility of predicting final learning performance provides useful information to improve student’s learning activities during the course. In particular, generalization and the period of evaluation are important issues. Prediction performance using SVR and the cumulative information about participant’s learning features is calculated in this section, and in Sect. 3.2.2 as well. Here, prediction was also conducted using cumulative values of four notetaking features and the note-taking factor score (NT-F3) and comparing them with Sect. 3.2.2. Performance is displayed in Fig. 3 using the same format as in Fig. 2. R-squares remain around 0.6 after the fourth session, except for the last two sessions when instruction was provided. When instruction was not provided, R-squares remained around 0.2. In order to evaluate prediction accuracy, a leave-one-out procedure was introduced. Though overall performance is lower than for multiple regression analysis, the accuracy of with instruction shows the possibility of estimating individual final exams scores as the course progress.

4 Discussion As mentioned in the introduction, student’s characteristics, including note-taking scores (NTS), have been confirmed to have an affect on the scores of final exams. Also, the effectiveness of note-taking instruction was introduced, though the detailed relationships between these variables were not specified. This paper tries to examine the relationships mathematically in order to improve the learning process. Fundamentally, the relationships between the scores of final exams and features of note-taking activities were analyzed. Though the metrics for note-taking activity for courses with and without instruction are comparable in Table 1, the correlation relationships between the scores of final exams and features of note-taking activities

4 Discussion

101

have changed due to the note-taking instruction that was given. All correlation coefficients between variables were significant during the course with instruction, while the coefficients were not significant for the course without instruction. In comparing correlation relationships across metrics of note-taking activity between Tables 2 and 3, the contents of notes participants took may have changed. Generally, students replaced the terms presented with their own words in their notes. When note-taking instruction was given, students recorded their own words in addition to recording the words presented. Therefore, the metric of additional words written down increased during the course with instruction, as is shown in Table 1. In the next step, the process of increasing the effectiveness of note-taking metrics was measured as a formative assessment, using two approaches. First, the relationship between the scores of final exams and variables of student’s individual behaviors were evaluated using multiple regression analysis and a step wise method of selecting variables. In the results, the overall contribution of most note-taking activity metrics which were selected was high when instruction was given. However, the contribution of some variables of student’s characteristics which were selected is small for the no instruction condition. The contribution of a set of metrics of note-taking activity increased with the number of sessions as the course progressed. The contribution, when measured as an R-square, remained at a high level between the 4th and 12th sessions of the course, as is shown in Fig. 2. However, the contribution decreased in the last two sessions. In regards to the change in the number of terms the lecturer presented in Fig. 1, the number of terms in the last two session was the smallest of all sessions during the course. The lecturer explained the contents using mainly a textbook, and so the number of terms presented was small. Therefore, this information may influence the metrics of cumulative values. When the relationships were validated using multiple regression analysis, the possibility of prediction of final exam scores during the progress of the course was confirmed. To do this, SVR was introduced as a robust prediction procedure to test note-taking activity metrics. Also, the contribution to predicting final exam scores remained at around 0.6 between the 4th and 12th sessions of the course, as is shown in Fig. 3. These results suggest that it is possible to improve participant’s final exam scores during the course. Since the prediction function is based on metrics of note-taking activity, it may be possible to provide each participant with appropriate instruction regarding their individual note-taking abilities. However, the results are based on the case of one course at a single Japanese university, and the number of participants was not large. The validity of this approach should be investigated with care, and the validation for these points will be the subject of our further study.

5 Conclusion The possibility of developing a diagnostic procedure to improve learning performance using participant’s characteristics and features of note-taking activity during a blended learning was confirmed. The relationships between these indices and the

102

The Possibility of Predicting Learning Performance using Features …

procedure used to predict the scores of final exams were examined. The following results were obtained. 1. Some metrics of note-taking activity were defined using lexical analysis of the contents of notes taken by students. The statistics of two groups of students in courses with and without note-taking instruction were compared. The effectiveness of note-taking with instruction was observed in correlational relationships between these metrics. 2. The relationships between final exam scores and the metrics of note-taking activity were analyzed, and the contribution of these metrics during the course with instruction was confirmed. Additionally, the relationships were established after only several sessions of the course, using formative analysis. 3. The possibility of predicting the scores of the final exams was confirmed using support vector regression (SVR) functions. Also, the contribution of the metrics of note-taking activity was confirmed. The sessions available for predicting scores were examined using formative analysis. These techniques can be applied to improving participant’s learning activity through the use of better note-taking methods. A detailed procedure and confirmation of the effectiveness of this using various other courses will be a subject of our further study. Acknowledgements This work was supported by JSPS KAKENHI Grant Number B-26282046, 2014–2016. This paper is an extended version which is based on reports at ESANN2015 [35] and a JSET short letter [22]. The authors would like to thank those who provided useful comments regarding both of these papers.

References 1. Nakayama M, Mutsuura K, Yamamoto H (2017) The possibility of predicting learning performance using features of note taking activities and instructions in a blended learning environment. Int J Educ Technol Higher Educ 14(6):1–14 2. Seaton DT, Bergner Y, Chuang I, Mitros P, Pritchard DE (2014) Who does what in a massive open online course? Commun ACM 57(4):58–65 3. Seaton DT, Nesterko S, Mullaney T, Reich J, Ho A (2014) Characterizing video use in the catalogue of MITx MOOCs. ELearning Papers (37):33–41 4. Ueno M (2007) Online outlier detection for e-learning time data. IEICE Trans J90-D, 40–51 (2007) 5. Nakayama M, Kanazawa H, Yamamoto H (2009) Detecting incomplete learners in a blended learning environment among japanese university students. International Journal of Emerging Technology in Learning 4(1):47–51 6. Bates A (2000) Managing Technological Change: Strategies for College and University Leaders. Jessey-Bass Publishers, San Francisco, CA, USA 7. Gulikers JTM, Bastiaens TJ, Kirschner PA (2004) A five-dimensional framework for authentic assessment. Educ Technol Res & Devel 52(3):67–86 8. Bloom BS, Hastings JT, Madaus GF (1971) Handbook on formative and summative evaluation of student learning. McGraw-Hill Inc., New York, USA 9. Bennett RE (2015) The changing nature of educational assessment. Rev Res Educ 39:370–407

References

103

10. Bell C, Jones N, Lewis J, Qi Y, Kirui D, Stickler L, Liu S (2015) Understanding consequential assessment systems of teaching: year 2 final report to Los Angles United school district. Educational Testing Service. Princeton, NJ, USA 11. Cronbach LJ, Snow RE (1977) Aptitudes and Instructional Methods -A Handbook for Research on Interactions-. Irvington Publishers, Inc., New York, USA 12. Kiewra KA (1985) Students’ note-taking behaviors and the efficacy of providing the instructor’s notes for review. Contem Educ Psychol 10:378–386 13. Kiewra KA (1989) A review of note-taking: the encoding-storage paradigm and beyond. Educ Psychol Rev 1(2):147–172 14. Kiewra KA, Benton SL, Kim SI, Risch N, Christensen M (1995) Effects of note-taking format and study technique on recall and relational performance. Contem Educ Psychol 20:172–187 15. Piolat A, Olive T, Kellogg RT (2005) Cognitive effort during note taking. Appl Cognit Psychol 19:291–312 16. Kobayashi K (2005) What limits the encoding effect of note-taking? a meta-analytic examination. Contem Educ Psychol 30:242–262 17. Weener P (1974) Note taking and student verbalization as instrumental learning activities. Instruc Sci 3:51–74 18. Nye PA, Crooks TJ, Powley M, Tripp G (1984) Student note-taking related to university examination performance. Higher Educ 13:85–97 19. Nakayama M, Mutsuura K, Yamamoto H (2014) Impact of learner’s characteristics and learning behaviour on learning performance during a fully online course. Electric J E-Learn 12(Issue 4):394–408 20. Nakayama M, Mutsuura K, Yamamoto H (2016) Lexical analysis of student’s learning activities during the giving of instructions for note-taking in a blended learning environment. Int J Inf Educ Technol 6(1):1–6 21. Nakayama M, Mutsuura K, Yamamoto H (2017) Effectiveness of student’s note-taking activities and characteristics of their learning performance in two types of online learning. Int J Distance Educ Technol 15:47–64 22. Nakayama M, Mutsuura K, Yamamoto H (2015) Relationship between the final test scores and indices of note-taking activity. Jpn J Educ Technol 39(Suppl.):53–56 23. Nakayama M, Mutsuura K, Yamamoto H (2011) Evaluation of student’s notes in a blended learning course. Int J Comput Archit Appl 1(4):1080–1089 24. Goldberg L (1999) A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Person Psychol Eur 7:7–28 25. IPIP (2001) A scientific collaboratory for the development of advanced measures of personality traits and other individual differences. http://ipip.ori.org 26. Fujii Y (2007) Development of a scale to evaluate the information literacy level of young people -comparison of junior high school students in japan and northern europe. Jpn J Educ Technol 30(4):387–395 27. Nakayama M, Yamamoto H, Santiago R (2007) The impact of learner characteristics on learning performance in hybrid courses among japanese students. Electron J E-Learn 5(3):195–206 28. Nakayama M, Yamamoto H, Santiago R (2008) Impact of information literacy and learner characteristics on learning behavior of japanese students in on line courses. Int J Case Method Res & Appl XX(4):403–415 29. Penn State Learning: Listening and note taking survey. http://penstatelearning.psu.edu/ resources/study-tips/note-taking/survey 30. Nakayama M, Mutsuura K, Yamamoto H (2012) Causal analysis of student’s characteristics of note-taking activities and learning performance during a fully online course. In: Proceedings of 2012 IEEE 11th international conference on trust, security and privacy in computing and communication, pp 1924–1929. Liverpool, UK 31. MeCab: Yet another part-of-speech and morphological analyzer. http://mecab.sourceforge.net 32. Nakayama M, Mutsuura K, Yamamoto H (2014) A note taking evaluation index using term networks in a blended learning environment. In: Proceedings of eighth international conference on complex, intelligent and software intensive systems, pp 486–490. Birmingham, UK

104

The Possibility of Predicting Learning Performance using Features …

33. Nakayama M, Mutsuura K, Yamamoto H (2016) Note-taking evaluation using network illustrations based on term co-occurence in a blended learning environment. Int J Distance Educ Technol 14:77–91 34. Chang CC, Lin CJ (2011) Libsvm: a library for support vector machines. ACM Trans Intell Syst Technol 2:27:1–27:27. http://www.csie.ntu.edu.tw/~cjlin/libsvm 35. Nakayama M, Mutsuura K, Yamamoto H (2015) The prediction of learning performance using features of note taking activities. In: Proceedings of 23rd European symposium on artificial neural networks, computational intelligence and machine learning (ESANN), pp 325–330. Brugge, Belgium

Student’s Reflections on Their Learning and Note-Taking Activities in a Blended Learning Course Minoru Nakayama, Kouichi Mutsuura, and Hiroh Yamamoto

Abstract Student’s emotional aspects are often discussed in order to promote better learning activity in blended learning courses. To observe these factors, course participant’s self efficacy and reflections upon their studies were surveyed, in addition to the surveying of the metrics of student’s characteristics during a Bachelor level credit course. Regarding the causal relationships between these factors, the contributions of the factors of self efficacy and other characteristics were evaluated. The contents of notes students took during the course were lexically evaluated to determine whether this activity promoted reflection. Four indices of note-taking activities were extracted from the lexical analysis. Correlation analysis was conducted, and according to the provisional results of the correlation analysis between the four indices of note-taking and student’s characteristics of their own degree of self efficacy, there were some significant relationships between note-taking indices and some of the self assessment indices, such as word rates in notes and the degree of out of course study, and between the content coverage of notes taken and self understanding. Keywords Note-taking · Blended learning · Student’s reflection · Student’s characteristics · Causal analysis · Text analysis

1 Introduction Various types of e-learning are proliferating, and becoming more widely used due to their beneficial educational aspects [2]. In particular, blended learning, which consists of face-to-face sessions and learning materials that are supported by information communication technologies (ICT), is the easiest way to use modern educational media which is familiar to both students and lecturers [3]. Conventionally, when the relationship between student’s learning activities and achievement has been discussed, learning performance in an e-learning environment was always a more important topic than participant’s satisfaction. Since encouraging student’s learning activities

Originally published in the Electronic Journal of e-Learning, Vol. 14, Issue 1, pp. 43–53, 2016 [1]. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Nakayama (ed.), Note Taking Activities in E-Learning Environments, Behaviormetrics: Quantitative Approaches to Human Behavior 11, https://doi.org/10.1007/978-981-16-6104-4_7

105

106

Student’s Reflections on Their Learning and Note-Taking Activities …

has been shown to improve performance, the e-learning environment, which uses learning materials together with ICT, is considered to contribute positively to their results. Recently, flipped classrooms, which are a kind of a blended learning, are believed to promote student’s self directed learning outside of classroom and to help collaborative learning in face-to-face sessions [3]. To observe student’s learning activity during courses, two types of learning evaluation have conventionally been used: emotional, and cognitive evaluation [4]. In addition to the use of learning motivation studies [5] to evaluate emotional aspects, evaluation of student’s preferences and their level of satisfaction, such as through the use of the ARCS model [6], is discussed. Regarding students’ attitudes towards learning, their learning efficacy and information which is based on self reflection are often considered [7]. For cognitive assessment, note taking activity has been observed in order to track student’s learning progress [8–11]. In addition to recording the content of notes taken, a lexical illustration of notes enables the use of conceptual mapping representation [12]. As a result, the effectiveness of learning performance can be confirmed, even during online courses [13, 14]. In the past few years, more detailed analysis of participant’s learning activity has been promoted, in order to improve and enhance learning performance. This is often referred to as learning analytics, and it can provide significant feedback to stakeholders [15]. Some of the techniques have been developed using information based on conventional sources, such as student’s characteristics and learning activities [16]. In addition to improving learning performance, learning analytics provides a design process for learning, using an instructional design technique to optimise learning activity [16]. Regarding the above discussion, a record of note-taking activity can be a source of learning analysis. Also, the emotional effectiveness of note taking activity in a blended learning environment can be observed. The purpose of this paper is to examine the relationship between participant’s evaluations of their own self-efficacy and their note-taking activities. Towards this aim, two sets of questionnaires were used to survey student’s self reflection, and their note taking activities were lexically analysed. The following topics are addressed in this paper: 1. Student’s self efficacy and the ways in which they reflect upon their own learning activities were measured and evaluated. 2. The relationship between student’s self efficacy and student’s characteristics were causally analysed. 3. Note-taking activities were evaluated using metrics based on lexical analysis of both the lecturer’s presentations and the contents of student’s notes. 4. The relationships between measurements of participant’s self assessments and metrics of note-taking activity were examined in order to confirm the effectiveness of note-taking behaviour.

2 Method

107

2 Method 2.1 Blended Learning Course as a Survey Course The surveys were conducted during a blended learning course at a Japanese university. The course consisted of 15 weeks of face-to-face sessions. The course was a Bachelor level Information System Network credit course [17]. For this cohort, several conventional surveys were conducted, to evaluate participant’s characteristics at the beginning of the course. Some metrics were also surveyed in the middle of the course. In addition to these, two types of surveys of student’s reflections were employed for this study. To monitor participant’s learning progress, all participants were asked to present the notes they took during each session of the course. These notes were scanned, and the images were stored on a PC. The textual content of the notes was lexically analysed. To encourage participants to take notes, some note-taking techniques were provided, using examples of well-taken notes. Instructions were given twice during the course, once early in the course and again at the midpoint. The valid number of participants for surveys was 40, but the number of valid participants for note content analysis was 27.

2.2 Characteristics of Students In this study, student’s characteristics, such as Personality [18, 19], Information Literacy [20], Note taking skills [13] and Learning Experience [21] were continuously surveyed [14, 21, 22]. These metrics are introduced here. Personality: Five factor scores were extracted using a public domain item pool, the International Personality Item Pool (IPIP) inventory [19]. The five components are “Extroversion” (IPIP-1), “Agreeableness” (IPIP-2), “Conscientiousness” (IPIP-3), “Neuroticism” (IPIP-4) and “Openness to Experience” (IPIP-5). Information Literacy: Information literacy inventories (32 items) were defined and developed by [20]. Originally, 8 factors were extracted, and they can be summarised as two secondary factors: Operational Skills (IL-1), and Attitudes towards Information Literacy (IL-2) [23]. Learning experience: Students’ online learning experiences were measured using a set of questions, and three factors were identified, as follows. Factor 1 (LE-F1): Overall Evaluation of the e-learning experience, Factor 2 (LE-F2): Learning Habits, and Factor 3 (LE-F3): Learning Strategies [21]. Note-taking skills: Student’s note taking skills were measured using the following three factors [13] They are NT-F1: Recognition of functions of note taking, NT-F2: Methodology of utilising notes, and NT-F3: Presentation of notes.

108

Student’s Reflections on Their Learning and Note-Taking Activities …

2.3 Participant’s Reflections Upon Learning Activity Participant’s emotional factors and impressions of their leaning attitudes during the course were surveyed, using two sets of questionnaires. The first one is a self efficacy metric consisting 9 question items which were developed by Pintrich (1990) [7], as shown in Table 1. The second one is a metric of participants’ level of satisfaction regarding their experience during the course, such as the participant’s self directed effort. Student’s self assessment of the degree of effort, self satisfaction, and study hours are frequently included in course assessments. These types of questions are often used in participant’s assessments of courses. Some of them were measured using a 5 point scale, and others were measured using a 10 point scale.

Table 1 Questionnaire items regarding self efficacy Question item Mean

Factor1

Factor2

I’m certain I can understand the contents taught in this course I expect to do very well in this class Compared with others in this class, I think I’m a good student I think I’ll receive a good grade in this class

3.75

0.86

−0.03

3.73

0.79

−0.17

3.28

0.74

0.14

3.08

0.70

0.21

6

I am sure I can do an excellent job on the problems and tasks assigned for this class

3.60

0.51

0.20

8

Compared with other students in this class I think I know a great deal about the subject

2.65

−0.27

0.82

0.18

0.59

0.25

0.48

0.13

0.42

0.31 0.33

0.17 0.19

2

1 5

4

7

My study skills are 2.85 excellent compared with others in this class 3 Compared with other 3.10 students in this class I expect to do well 9 I know that I will be able to 3.38 learn the material for this class Contribution ratio (of each factor with other factors eliminated Contribution ratio (of each factor with other factors ignored)

2 Method

109

2.4 Lexical Comparison of Lecturer’s Presentations and Student’s Notes The contents of participant’s notes were read and recorded manually, as computer readable text. The lecturer’s hand-written notes to be presented to participants during face-to-face sessions were also transformed into computer readable text. Notes of both the participants and the lecturer were lexically analysed using the Japanese morphological term analysis tool MeCab. Nouns were extracted from the texts of the notes. From these, term-session matrices, such as frequency of nouns across sessions, were generated. The term frequencies in the contents of notes of both the lecturer and the participants were evaluated as follows [13, 22]. 1. Word ratio: the ratio between the number of terms written and the number of terms given (the number of terms participants recorded vs. the number of terms the lecturer presented). 2. Coverage: the coverage ratio was calculated as a percentage of the number of terms recorded by participants. To extract the semantic structures in the contents of note taken, a social network analytical technique was used on the texts of notes [24, 25]. In comparing the contents of the lecturer’s presentations with the student’s notes, co-occurring nouns were analysed using a previously reported methodology [13, 22]. Term co-occurrence shows the structure of the conceptual meanings using a lexical representation of the term connection patterns. Noun transitions in the notes were extracted from phrases, such as A-B and B-C extracted from the text A-B-C. The relationship between the two terms, known as 2-gram nouns, is summarised using an adjacency matrix. An example of the lecturer’s presentation in Session 13 is shown in Fig. 1. The matrix of the participants should coincide precisely with the lecturer’s matrix when all contents have been transferred to the participants. The adjacency matrix can be illustrated as a networked graph, such as a conceptual map [12, 25]. The difference between the two maps shows the distinctness in processing information between the lecturer and the participants. Therefore, the differences indicate the

L

S

Fig. 1 Example of adjacency matrix (Left) and the relationship between two adjacency matrices (Right)

110

Student’s Reflections on Their Learning and Note-Taking Activities …

degree of transformation of the lecturer’s contents. The differences between the two matrices can be calculated as a distance measure. The distance between the lecturer’s presentations and student’s notes is defined using two metrics, as follows: • Additional Distance means the sum of the number of additional nodes or edges in a matrix. • Insufficient Distance means the sum of the number of reduced nodes or edges in a participant’s matrix, in comparison with the lecturer’s matrix. Both distances are influenced by the total number of terms in the lecturer’s presentation, so that the relative distances are calculated using the number of terms the lecturer presented in each session. As a result, note-taking activity was evaluated using four indices of note-taking, then overall averages across all sessions and partial averages for the first and the second halves of sessions were calculated, respectively.

2.5 Causal Relationships Analysis Across the Indices The relationships among the indices mentioned in the above sections were examined using a structural equation modelling technique (SEM). The possible causal relationships and the parameters of the models were estimated using structural equation modelling software (AMOS) [26], and the validity of the models was tested using indices of the fitness of the model (the GFI: Goodness of Fitting index).

3 Results 3.1 Responses of Participant’s Self Reflection The means of participant’s responses for the two sets of questionnaires are summarised in two tables. Table 1 represents self efficacy and Table 2 represents their own reflections during the course. All means for questions concerning self efficacy inventories are above the middle value of a 5 point scale, indicating that participants have responded positively. As these means are at the same levels, the latent factors are extracted using factor analysis with Promax rotation. The factor loading values for the two factor structures are also summarised in Table 1. As the table shows, the first factor contributes over 30% of the total. Therefore, regarding the contents of the question items, the label for the first factor (SE-1) is “self confidence in student’s own attitude”, and the label for the second factor (SE-2) is “self confidence in student’s own level of competence”. The mean scores for the two factors are displayed in Table 2, where the score of the first factor (SE-1) is higher than the score of the second factor (SE-2), while their

3 Results

111

Table 2 Question items and means for self evaluation Label Question Items SE-1

Factor1: self confidence in student’s own attitude SE-2 Factor2: self confidence in student’s own level of competence 1 Syllabus reading Read the syllabus of this course carefully in advance 2 Out-of-class study Study out of class for this course 7 Learning hours Grade a level of learning hours for this course out of the class 3 Self directed effort Self assessment of own attitude towards this course 4 Self understanding Self assessment of level of understanding for contents of this course 5 Self achievement Self assessment of level of achievement for this course 6 Self satisfaction Self assessment of level of overall satisfaction for this course Bold means are based on a 10 point scale

Mean 3.49 2.87 3.65 2.83 2.38 7.26 6.60 6.53 7.60

correlation coefficient is small (r = 0.13). The participants have confidence in their own attitudes rather than in their levels of competence. The levels for self evaluative responses are also relatively high, as shown in Table 2. These results suggest that most students participate sincerely, and they satisfy the requirements of the course. The exception is participant’s “learning hours”, a question regarding their own learning opportunities outside of the classroom. This point is often noted by researchers of higher education [27]. The responses mentioned in the two tables correlate with each other, and these responses may also be related to the student’s characteristics which were mentioned in the above sections.

3.2 Causal Relationships Across the Indices 3.2.1

Learning Activity Outside the Classroom

In regards to the above discussion, learning hours and frequency of study outside the classroom should be measured, in order to evaluate student’s self directed learning activity. A possible causal relationship is displayed in Fig. 2. Significant path coefficients are indicated using bold characters, and non-significant values are represented by ( ).

112

Student’s Reflections on Their Learning and Note-Taking Activities …

IL-1

0.62

0.65 Syllabus reading

(0.29) (-.20)

0.23

0.47

e Out-of-class study

IL-2

Self Directed Effort

0.37

0.28

0.41

Learning hours

e

e

0.34

e LE-F3

GFI=0.96, AGFI=0.89, RMSEA=0.00

Fig. 2 Relationships between some student’s characteristics and learning opportunity outside the classroom

Some student’s characteristics, such as information literacy (IL-1 and IL-2), learning strategy (LE-F3), and some self evaluation inventories were selected. Since the subject of this course concerns information systems, the information literacy of operational skills (IL-1) affects the extent of syllabus reading, while student’s learning strategy experience affects the degree of self directed effort. Both contribute to student’s level of attention towards learning outside the classroom, which is shown in Fig. 2 as causal paths. This confirms the importance of both lecture syllabi, and the degree of effort students make, as reflected in their number of hours of study.

3.2.2

Relationship Between Self Efficacy and Reflections

In the previous section, two types of metrics were introduced to evaluate participant’s emotional factors, such as self efficacy and reflections. The means are summarised in Table 2. As mentioned in Fig. 2, information literacy affects participant’s attitude, thus it has also been employed in this analysis. The contribution of information literacy and the relationships between self efficacy and reflections are summarised in Fig. 3, using a path diagram. Though some path coefficients are not significant, the goodness of fit index (GFI) of the structure of this model is significant (GFI = 0.87). The factor of information literacy (IL-1) affects the factor of self efficacy, such as confidence in one’s own attitude (SE-F1) and self directed effort. Both self efficacy (SE-F1) and self directed effort significantly affect self satisfaction, while self directed effort also affects student’s own impression of their level of self understanding and indirectly affects self achievement. This causal path suggests that participant’s confidence in their attitude and self directed effort have an effect on their self evaluation, such as the level of satisfaction, and the degree of understanding of the course contents.

3 Results

113

0.30

SE-F1

(0.13) Self Understanding

(0.13)

e

IL-1 (0.30)

0.61

(-.17)

0.51

e (0.16)

0.37

e

Self Achievement

SE-F2 (-.22)

0.28 (0.19)

IL-2

(0.23)

e Self Directed Effort

e

0.56 Self Satisfaction

0.46

e

GFI=0.87, AGFI=0.72, RMSEA=0.05 Fig. 3 Contribution of information literacy to self efficacy and self assessment

3.2.3

Contribution of Note-Taking Skills and Learning Experience

As mentioned in the Introduction section, note taking activity may have some effect on participant’s emotional factors. As factors of student’s learning experience have contributed to learning activity outside of the classroom, some additional factors may also affect their self evaluations. To validate the hypothesis, factors for note taking skills and factors for learning experience were mapped in Fig. 3 using causal paths. The contribution of note taking skills was confirmed, as Fig. 4 shows. Though possible paths which concern note taking skills have been indicated, most are not significant. Additionally, factors for learning experience were introduced into the causal path diagram, as shown in Fig. 5. Though some of the factors, such as LE-F1 (e-Learning experience) and LE-F3 (Learning strategies) significantly influence participant’s self evaluation, the GFI of this model is not significant (GFI = 0.79, AGFI = 0.64, RMSEA = 0.10). Therefore, the contribution of participant’s learning activities, such as note taking skills and student’s learning experience towards their own self-assessment was not confirmed. To verify the hypothesis, a more detailed analysis was employed, such as making changes to participant’s activities as the course progressed, and in the metrics of the contents of notes taken.

3.3 Note Taking Activity Four types of indices for note taking activity were measured, using both the lecturer’s presentations and student’s notes from each classroom session across the 14 weeks of the course. As the number of terms depends on the contents of the lecturer’s

114

Student’s Reflections on Their Learning and Note-Taking Activities … e

GFI=0.87, AGFI=0.72, RMSEA=0.05 SE-F1 (0.19) (0.30)

(0.18)

IL-1 (-.17)

(0.31)

0.62

Self Understanding

(0.17)

(0.26)

0.51

0.53

0.37

e

Self Achievement

e (0.16) (-.22)

SE-F2

(0.21)

IL-2

e

0.27

(-.18)

0.34

Self Satisfaction

0.35 0.49 0.34

(0.24)

e

(0.17)

Self Directed Effort

NT-F1

NT-F2

NT-F3

e

e

e

e

Fig. 4 Causal relationships while note-taking skills were being introduced e

e

e

LE-F1

LE-F2

LE-F3

e SE-F1 0.42 (0.30)

(0.21)

(0.27)

Self Understanding

(0.29)

0.30

IL-1

0.31

(0.32) (0.31)

0.62

0.35

(0.22)

0.33

0.39 0.35

0.37

(0.26)

e

0.38

e

Self Achievement

(0.19)

IL-2

(0.21)

-.43

SE-F2 0.32

0.41

e

(0.20)

-.34

(0.28)

0.34

(0.24)

Self Directed Effort

e

Self Satisfaction

0.44

e

NT-F1

NT-F2

NT-F3

e

e

e

GFI=0.79, AGFI=0.64, RMSEA=0.10

Fig. 5 Causal relationships between self efficacy and reflection using factors of student’s characteristics

presentation, word rates and distances are normalised using the lecturer’s metrics. In order to consider student’s experience, the metrics between the first and second halves of the course were compared, such as the means of the first 7 sessions and the means of the second 7 sessions. Mean ratios are summarised in Fig. 6, which compares these two ratios. Though word rates in the second half decreased in comparison with sessions in the first half, both means of lengths for sessions in the second half are higher than for means in the

3 Results Fig. 6 Mean metrics of note-taking activity across the course 1st and 2nd halves

115

1st survey

Word ratio

2nd survey Coverage

Insufficient Distance

Additional Distance

Rate

Fig. 7 Mean factor scores of note-taking skills in the two surveys

NT-F1:

3.84

1st survey

Recognizing NT functions

p