Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education: Proceedings of ‘The Learning Ideas Conference’ 2023 (Lecture Notes in Networks and Systems, 767) [1st ed. 2023] 3031416368, 9783031416361

Citation preview

Lecture Notes in Networks and Systems 767

David Guralnick Michael E. Auer Antonella Poce   Editors

Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education Proceedings of ‘The Learning Ideas Conference’ 2023

Lecture Notes in Networks and Systems Volume 767

Series Editor Janusz Kacprzyk , Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland Advisory Editors Fernando Gomide, Department of Computer Engineering and Automation—DCA, School of Electrical and Computer Engineering—FEEC, University of Campinas— UNICAMP, São Paulo, Brazil Okyay Kaynak, Department of Electrical and Electronic Engineering, Bogazici University, Istanbul, Türkiye Derong Liu, Department of Electrical and Computer Engineering, University of Illinois at Chicago, Chicago, USA Institute of Automation, Chinese Academy of Sciences, Beijing, China Witold Pedrycz, Department of Electrical and Computer Engineering, University of Alberta, Alberta, Canada Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland Marios M. Polycarpou, Department of Electrical and Computer Engineering, KIOS Research Center for Intelligent Systems and Networks, University of Cyprus, Nicosia, Cyprus Imre J. Rudas, Óbuda University, Budapest, Hungary Jun Wang, Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong

The series “Lecture Notes in Networks and Systems” publishes the latest developments in Networks and Systems—quickly, informally and with high quality. Original research reported in proceedings and post-proceedings represents the core of LNNS. Volumes published in LNNS embrace all aspects and subfields of, as well as new challenges in, Networks and Systems. The series contains proceedings and edited volumes in systems and networks, spanning the areas of Cyber-Physical Systems, Autonomous Systems, Sensor Networks, Control Systems, Energy Systems, Automotive Systems, Biological Systems, Vehicular Networking and Connected Vehicles, Aerospace Systems, Automation, Manufacturing, Smart Grids, Nonlinear Systems, Power Systems, Robotics, Social Systems, Economic Systems and other. Of particular value to both the contributors and the readership are the short publication timeframe and the world-wide distribution and exposure which enable both a wide and rapid dissemination of research output. The series covers the theory, applications, and perspectives on the state of the art and future developments relevant to systems and networks, decision making, control, complex processes and related areas, as embedded in the fields of interdisciplinary and applied sciences, engineering, computer science, physics, economics, social, and life sciences, as well as the paradigms and methodologies behind them. Indexed by SCOPUS, INSPEC, WTI Frankfurt eG, zbMATH, SCImago. All books published in the series are submitted for consideration in Web of Science. For proposals from Asia please contact Aninda Bose ([email protected]).

David Guralnick · Michael E. Auer · Antonella Poce Editors

Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education Proceedings of ‘The Learning Ideas Conference’ 2023

Editors David Guralnick Kaleidoscope Learning New York, NY, USA

Michael E. Auer CTI Global Frankfurt, Germany

Antonella Poce University of Rome Tor Vergata Rome, Italy

ISSN 2367-3370 ISSN 2367-3389 (electronic) Lecture Notes in Networks and Systems ISBN 978-3-031-41636-1 ISBN 978-3-031-41637-8 (eBook) https://doi.org/10.1007/978-3-031-41637-8 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

The Learning Ideas Conference looks to bring together people from around the world to help reimagine what learning can be, particularly using, and inventing, new technologies. The Learning Ideas Conference 2023 was held as a hybrid event, with significant participation from in-person and virtual participants. Our goal was for conference participation to be easily accessible to all, and for in-person and virtual participants to have the opportunity to interact, thus creating an experience that was truly hybrid. The conference featured four wonderful keynote speakers: • Dr. Lydia Liu, Principal Research Director, ETS, Princeton, New Jersey, USA. “Power Skills: Unlocking Potential for the Future of Work.” • Dr. Tony O’Driscoll, Adjunct Professor, Duke University’s Fuqua School of Business & Pratt School of Engineering; Research Fellow, Duke Corporate Education, Duke University, Durham, North Carolina, USA. “Learning in Context: How Emerging Technologies Will Change the Game in Generative Learning.” • Dr. Maria Rosaria Re, Research Fellow, Dept. of Education, University Roma Tre and Assistant Professor, University of Modena and Reggio Emilia, Rome Italy. “Museum Education for Professional Development: How to Use Heritage to Create Training Experiences for both Hard and Soft Skills.” • Dr. Joiselle Cunningham Smith, CEO, Pathways to Creative Industries, New York, New York, USA. “Higher Education & Transitions to Careers.” The conference also featured a panel discussion on “Artificial Intelligence and the Future of Higher Education and Workplace Learning” and a total of over 110 sessions. All papers were double-blind peer-reviewed. I very much appreciate all of the work it took to make The Learning Ideas Conference 2023 a success, from our keynotes, our Executive Committee and Program Committee members, our reviewers, and of course our conference organizing team, led by Tongtong Huang.

v

vi

Preface

I am looking forward to The Learning Ideas Conference 2024, to be held again as a hybrid event, in both New York and online, from June 12 to 14, 2024.

David Guralnick Conference Chair, The Learning Ideas Conference 2023

Committees

Conference Chair David Guralnick

Kaleidoscope Learning and Columbia University, New York, New York, USA

Executive Committee Chairs Michael E. Auer Antonella Poce

CTI, Frankfurt, Germany University of Rome Tor Vergata, Rome, Italy

Publication Chair Lara Ramsey

Kaleidoscope Learning, New York, New York, USA

Executive Committee Abdallah Al Zoubi Mohammed Ali Akour Kostas Apostolou Sharon Bailin Ryan Baker

Princess Sumaya University for Technology (PSUT), Amman, Jordan A’Sharqiyah University, Ibra, Oman McMaster University, Hamilton, Ontario, Canada Simon Fraser University, Vancouver, British Columbia, Canada University of Pennsylvania, Philadelphia, Pennsylvania, USA

vii

viii

Patricia Behar John Black Patrick Blum Angela Bullock Santi Caballé Nicola Capuano Imogen Casebourne Manuel Castro Veronica Chehtman Hal Christensen Gary J. Dickelman Samir El-Seoud Kai Erenli Matthias Gottlieb Christian Guetl Alexander Kist Gila Kurtz Mark J. W. Lee Christy Levy Matthea Marquart Bruce McLaren Jorge Membrillo Hernández Dominik May Christina Merl

Committees

Federal University of Rio Grande do Sul, Brazil Teachers College, Columbia University, New York, New York, USA Blum consulting, Aachen, Germany University of the District of Columbia, Washington, DC, USA Open University of Catalonia, Barcelona, Spain University of Basilicata, Potenza, Italy University of Oxford, Oxford, UK Universidad Nacional de Educacion a Distancia (UNED), Madrid, Spain AySA Water and Sanitation Argentina, Buenos Aires, Argentina QuickCompetence, New York, New York, USA EPSScentral, Boynton Beach, Florida, USA The British University in Egypt (BUE), Egypt University of Applied Sciences BFI Vienna, Vienna, Austria Technical University of Munich, Munich, Germany Graz University of Technology, Graz, Austria University of Southern Queensland, Queensland, Australia Holon Institute of Technology, Holon, Israel Charles Sturt University, Bathurst, Australia Kaleidoscope Learning, Chicago, Illinois, USA Columbia University, New York, New York, USA Carnegie Mellon University, Pittsburgh, Pennsylvania, USA Tecnológico de Monterrey, México University of Georgia, Athens, Georgia, USA TalkShop/2CG®, Vienna, Austria

Committees

Gary Natriello Barbara Oakley Andreas Pester Kinga Petrovai Robert Pucher Teresa Restivo Fernando Salvetti Alicia Sanchez Sabine Seufert Thrasyvoulos Tsiatsos James Uhomoibhi Matthias Utesch Ellen Wagner

Sarah Wang Xiao-Guang Yue

ix

Teachers College, Columbia University, New York, New York, USA Oakland University, Oakland, Michigan, USA Carinthia Tech Institute, Villach, Austria The Art & Science of Learning, Ottawa, Ontario, Canada University of Applied Sciences Technikum Wien, Vienna, Austria University of Porto, Porto, Portugal Logosnet, Turin, Italy Czarina Games, Alexandria, Virginia, USA Universität St. Gallen (HSG), St. Gallen, Switzerland Aristotle University of Thessaloniki, Thessaloniki, Greece Ulster University, Newtownabbey, UK Technical University of Munich, Munich, Germany North Coast EduVisory Services, Sonoma, California; and University of Central Florida, Orlando, Florida, USA Xi’an Jiaotong-Liverpool University, Suzhou, Jiangsu, China European University Cyprus, Nicosia, Cyprus

Program Committee Carme Anguera Iglesias Fahriye Altinay Aksal Zehra Altinay Gazi Sarah Appleby Estefanía Avilés Anabel Bugallo Martha Burkle

Open University of Catalonia, Barcelona, Spain Near East University, Nicosia, Cyprus Near East University, Nicosia, Cyprus Online Learning International, New York, New York, USA Complutense University of Madrid, Madrid Spain ADP, New York, New York, USA Southern Alberta Institute of Technology, Calgary, Canada

x

Mihai Caramihai Nunzio Casalino Mark Cassetta Hui Soo Chae Elisabeth Counselman-Carpenter Covadonga Diez-Sanmartin David Foster Sarah Frame Marga Franco i Casamitjana Eran Gal Genevieve Gallant Abel Henry Manir Abdullahi Kamba Okba Kazar Adamantios Koumpis Molly Koenen Maria Lambrou Stacy Lindenberg Allison Littlejohn Chee Ken Nee Luis Ochoa Siguencia Grace O’Malley Michael Paraskevas

Committees

University Politehnica Bucharest, Bucharest, Romania Guglielmo Marconi University and LUISS Business School, Rome, Italy Pfizer, New York, New York, USA New York University, New York, New York, USA Adelphi University, Norwalk, Connecticut, USA Complutense University of Madrid, Madrid, Spain ExecOnline, Chapel Hill, North Carolina, USA University of East London, London, England Open University of Catalonia, Barcelona, Spain Holon Institute of Technology, Holon, Israel GG Consultants Limited, St. John’s, Newfoundland, Canada United Nations Development Programme, Copenhagen, Denmark Bayero University, Kano, Nigeria Biskra University, Biskra, Algeria University Hospital Cologne, Cologne Germany Pioneer Management Consulting, Minneapolis, Minnesota, USA University of the Aegean, Greece Talent Seed Consulting, Columbia, South Carolina, USA Glasgow Caledonian University, Glasgow, Scotland Universiti Pendidikan Sultan Idris, Perak, Malaysia Jerzy Kukuczka Academy of Physical Education, Katowice, Poland National College of Ireland, Dublin, Ireland University of Peloponnese, Patras, Greece

Committees

Iina Paarma Stefanie Quade Maria Rosaria Re Laura Ricci Gina Ann Richter Andree Roy John Sandler Steven Schmidt Barbara Schwartz-Bechet Julie-Ann Sime Amando P. Singun, Jr. Anelise Spyer Christian Stracke Qianhui Sun Kyla L. Tennin Terrie Lynn Thompson Caryn Tilton Leyla Y. Tokman Christos Troussas Chris Turner Karin Tweddell Levinsen Maggie M. Wang Steve Wheeler Annika Wiklund-Engblom Rusen Yamacli Xiaofei Zhou

xi

United Nations Development Programme, Copenhagen, Denmark Berlin School of Economics and Law, Berlin, Germany Roma Tre University, Rome, Italy MIA Digital University, Barcelona, Spain St. Charles Consulting Group, New York, New York, USA University of Moncton, Moncton, Canada Telstra Corporation, Melbourne, Australia East Carolina University, Greenville, North Carolina, USA Misericordia University, Elkins Park, Pennsylvania, USA Lancaster University, Lancashire, UK Higher College of Technology, Muscat Sultanate of Oman Docta, São Paulo, Brazil eLC Institute for eLearning, Bonn, Germany Ernst & Young, California, USA University of Phoenix, Tempe, Arizona, USA University of Alberta, Edmonton, Alberta, Canada MyPlacetoLearn, Welches, Oregon, USA Anadolu University, Eskisehir, Turkey University of West Attica, Athens, Greece University of Winchester, Winchester, UK Danish University of Education, Denmark The University of Hong Kong, Hong Kong University of Plymouth, Plymouth, UK Abo Akademi University, Vasa, Finland Anadolu University, Eskisehir, Turkey University of Rochester, Rochester, New York, USA

xii

Committees

ALICE Special Track Co-chairs Santi Cabellé Nicola Capuano

Open University of Catalonia, Barcelona, Spain University of Basilicata, Potenza, Italy

ALICE Special Track Committee Joan Casas Jordi Conesa Thanasis Daradoumis Sara De Freitas Christian Gütl Giuseppina Rita Mangione Agathe Merceron Anna Pierri Antonio Sarasa Marco Temperini Daniele Toti

Open University of Catalonia, Spain Open University of Catalonia, Spain University of the Aegean, Greece Coventry University, UK Graz University of Technology, Austria Institute of Educational Documentation, Innovation and Research, Italy Beuth University of Applied Sciences, Berlin, Germany University of Salerno, Italy Universidad Complutense de Madrid, Spain Sapienza University of Rome, Italy Catholic University of the Sacred Heart, Italy

Contents

Main Conference Cybersecurity Awareness, Education, and Workplace Training Using Socially Enabled Intelligent Chatbots . . . . . . . . . . . . . . . . . . . . . . . . . . Sherif Abdelhamid, Tanner Mallari, and Mona Aly Creating Binge-Worthy e-Learning Experiences . . . . . . . . . . . . . . . . . . . . . . Nafiza Akter Thinking and Chatting Deontically—Novel Support of Communication for Learning and Training with Time Travel Prevention Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Oksana Arnold, Ronny Franke, Klaus P. Jantke, Rainer Knauf, Tanja Schramm, and Hans-Holger Wache Corrective vs. Nurturing Feedback in Design Education: Alternative Models of Critique that Positively Impact Students’ Sense of Self-efficacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zinka Bejtic Improving Sonographer-Patient Communication in a Diverse and Multicultural Environment Through Role-Plays with Digital Humans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Barbara Bertagni, Linda Zanin, Fernando Salvetti, and Ianna Contardo Psychology and STEM Education: From the Classroom to Society . . . . . Evi Botsari, Konstantina Sdravopoulou, and Sarantos Psycharis

3 17

25

39

55 61

Creating Cloud Experts: The Power of an Innovative, Hybrid Learning Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Natalie Brooks Powell

71

A Pedagogy for Engineering Concepts Focusing on Experiential Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kanmani Buddhi

83

xiii

xiv

Contents

Exploring Differences in Work Environment and Work Engagement as Moderated by Psychological Capital . . . . . . . . . . . . . . . . . . 105 Rebekah L. Clarke What’s Behind the Learning Management System: Algorithmic Design in Online Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Simone C. O. Conceição and Lilian H. Hill A Participatory Museum for Intercultural Development: Innovative Fruition of South-Asian Art Collections in Italy . . . . . . . . . . . . 129 Luca Contardi Promoting Positive Ageing Lifestyles and Wellbeing Through the Use of Social Media to Facilitate and Enhance Creative Decision-Making . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Ron Corso and Charlie-Helen Robinson Learning and Performance Technology for the Performing Arts . . . . . . . 151 Gary J. Dickelman Fuel Your Top Five: Using What You Know to Make Real Change . . . . . 179 Erin Donovan Online Education Innovation Strategies to Gain Support and Accomplish Team Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 Joseph Evanick Overcoming Learning Gaps and Building Transferability Skills in a Higher Education Math Course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Subhadra Ganguli Evaluating the Effectiveness of a New Programming Teaching Methodology Using CodeRunner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 Siba Haidar, Antoun Yaacoub, and Felicia Ionascu Institutional Effectiveness of Innovative Learning Experiences: How MOOCs Transform and Encourage Lifelong Learning . . . . . . . . . . . 225 Ryan Hamilton Project DOCE as a Pedagogical Experience for Innovative Teaching . . . . 235 Susana Jorge, Paula Tavares, Manuel Albino, and Marta Madureira A Comparison of Online and In-Person MBI Classes on Self-Compassion and Creativity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 Young Min Jung and Eunmi Kim Learning in Encounter: Collaborative and Project-Based Strategies for Learning in Culturally and Religiously Diverse Contexts in the Higher Education Sector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 Christoph Knoblauch and Gökcen Sara Tamer-Uzun

Contents

xv

New Perspectives for Internationalization in Higher Education: Collaborative Formats in Project-Based and Blended Learning Contexts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 Christoph Knoblauch Student Satisfaction and Graduation Rates in Finnish Master of Engineering Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 Matti Koivisto Ensuring Optimal Performance in Online Learning of STEM Subjects: An Autoethnographic Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 Gopala Krishna Koneru The Development of a Learning Arrangement in a Characteristic Curve Remote Laboratory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 Ingrid Krumphals, Thomas Benedikt Steinmetz, Christian Kreiter, and Thomas Klinger How’s Identity Being Learned in City Museums? An Identity Education Approach to Museum Education . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Zhichao Lei Motivational Factors for College Success: A Focus on First-Generation and Immigrant Students . . . . . . . . . . . . . . . . . . . . . . . . 343 Meitong Lu Perceptions of Online Strategies and Digital Readiness in the COVID-19 Environment: An Instrumental Case Study . . . . . . . . . . 355 Pamela McCray, Valentina Rada, Steven A. Szeszko II, Norman St. Clair, and Kevin B. Vichcales Using Polyphonic Storytelling Techniques for Skills Development . . . . . . 369 Christina Merl The 50 + 10 Concept for the Development of Future Skills: A Pedagogical Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 Ana Francisca Monteiro, Soraia Gonçalves, and António H. J. Moreira Loose Parts: Creating Learning Opportunities Beyond the LMS . . . . . . . 393 Gary Natriello and Hui Soo Chae ChatGPT: Should It Have a Role in Education? . . . . . . . . . . . . . . . . . . . . . . 405 Birgit Oberer and Alptekin Erkollar Education 5.0: Design Thinking Goes ICT . . . . . . . . . . . . . . . . . . . . . . . . . . . 417 Birgit Oberer and Alptekin Erkollar Redesigning Education Using Serious Games . . . . . . . . . . . . . . . . . . . . . . . . 427 Jenny Pange, Liudmila Rupsiene, and Agostino Marengo

xvi

Contents

Gamifying Diabetes — An Education Game Teaching People with Diabetes About Physical Activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439 Maria Anna Rauchensteiner, Tim Colsman, Albina Fatykhova, Nilüfer Deniz Faizan, Matthias Christoph Utesch, Holger Wittges, and Helmut Krcmar Intelligent Digital Humans for Bias-Free Recruitment Interviews: A Diversity & Inclusion Training Program . . . . . . . . . . . . . . . . . . . . . . . . . . . 455 Fernando Salvetti, Barbara Bertagni, and Ianna Contardo Extended Reality and Medical Simulation: Cooperation into the Metaverse to Boost Diversity and Inclusion . . . . . . . . . . . . . . . . . . . 463 Fernando Salvetti, Roxane Gardner, Jenny Rudolph, Rebecca Minehart, Barbara Bertagni, and Ianna Contardo Using a Podcast to Foster Success Among Computer Science Students . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475 Sigrid Schefer-Wenzl and Igor Miladinovic Research on Mental Health Training for Pre-service Teachers to Address Pupil Issues in Schools: An International Virtual Learning Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485 Barbara Schwartz-Bechet Towards Inclusive Excellence: A Case Study in Engineering . . . . . . . . . . . 495 Hong Shaddy Massive Open Online Courses at Ukrainian Agrarian Universities: To Be or not to Be . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505 Bohdan Shunevych Participative Learning Experience Design Through Group Concept Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513 Slavi Stoyanov Using Design Thinking to Understand Student (Dis)engagement in Higher Education: Involving Students in the Co-creation of Their Own Learning Experiences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527 Elaine Tan So You Want to Be a Tutor? Professional Development and Scenario-Based Training for Adult Tutors . . . . . . . . . . . . . . . . . . . . . . . 537 Danielle R. Thomas, Shivang Gupta, Erin Gatz, Cindy Tipper, and Kenneth R. Koedinger Fostering Inclusion and Well-Being Through Digital Language Learning in Museum Contexts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549 Maria Tolaini

Contents

xvii

Proposing a Hybrid Campus: A Community Engagement Framework for Online Learners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561 Roxana Toma and Matthew Berge Narrating the Museum: Enhancing Cultural Heritage Through User Profiling and Individualized Content . . . . . . . . . . . . . . . . . . . . . . . . . . . 575 Eliana Maria Torre Closing the Gender Gap in STEM MOOCs Through Brief, Novel Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 589 Alexandra D. Urban Peer Assessment in the University Context for the Development of Transversal and Digital Skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607 Mara Valente The Coherence Between Innovative Teaching Methods and Formative Assessment in Higher Education . . . . . . . . . . . . . . . . . . . . . . 621 Vilmos Vass The Potential of Tele-Assessment and Virtual Treatment for the Future of Lifelong Learning: A Literature Review . . . . . . . . . . . . . 629 A. Jordan Wright ALICE (Adaptive Learning Via Interactive, Collaborative and Emotional Approaches) Special Track Explainable Prediction of Student Performance in Online Courses . . . . . 639 Nicola Capuano, Diego Rossi, Victor Ströele, and Santi Caballé The Magic of Games: Creating a Pull-Based Learning System Through Serious Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653 Ritika Datta, Ajay Gupta, and Bob Philips Predicting Students’ Academic Success Based on Various Course Activities: A Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 665 ˇ c Poturi´c, Sanja Candrli´ ˇ Vanja Coti´ c, and Ivan Draži´c Augmented Reality in STEM Education: Mapping Out the Future . . . . . 677 Sarantos Psycharis, Konstantina Sdravopoulou, and Evi Botsari Certainty-Based Self-Assessment: A Chance for Enhanced Learning Engagement in Higher Education. An Experience at the University of Barcelona . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 689 Ana Remesal, María José Corral, Patricio García-Mínguez, Judit Domínguez, Iria SanMiguel, Tomas Macsotay, and Ernesto Suárez Curated Recommendations of Teaching and Learning Videos on YouTube with the Help of a Chatbot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 701 Theresa Zobel, Hendrik Steinbeck, and Christoph Meinel

Main Conference

Cybersecurity Awareness, Education, and Workplace Training Using Socially Enabled Intelligent Chatbots Sherif Abdelhamid , Tanner Mallari , and Mona Aly

Abstract Over the past ten years, chatbot technology has become a burgeoning field within AI, finding practical applications in sectors such as banking, customer service, medicine, education, and e-commerce. Nevertheless, the use of intelligent agents in cybersecurity needs to be more extensive. Recently, most research in cybersecurity has focused more and more on detecting malicious activities automatically. Unfortunately, few works focused on how the human factor can result in such attacks. Humaninduced attacks and errors are found to be either unintentional or intentional. The first type, our focus, can result from lacking training and cyber awareness. Attackers use different tactics to lure victims into unsafe behaviors, leading to potential personal blackmail, ransomware attacks, or data breaches through phishing emails. Usually, to address the problem mentioned above, universities, employers, and organizations provide cyber awareness education. However, the problem is that the training is a onetime process and does not provide the necessary continuing support and education. As a result, people usually find themselves in a situation where they need specific advice before engaging in a particular activity. Sometimes, they might need someone to alert them to a suspicious email, fishy link, or abnormal attachment. In response, in this study, we discuss the integration of chatbots with social networking for cybersecurity public awareness, education, and workplace training. We focus on utilizing chatbots as social, collaborative agents to protect users against cyber-attacks. The chatbots provide continuous support to users by providing immediate answers and advice on what to do in various threatening situations. In addition, admin users add training material remotely into chatbots, and users can view and answer interactive questions to ensure they are up-to-date regarding cyber awareness. Keywords Chatbot · Artificial Intelligence · Cybersecurity Awareness · Workplace Training · Social Network Analysis

S. Abdelhamid (B) · T. Mallari · M. Aly Virginia Military Institute, Lexington, VA 24450, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_1

3

4

S. Abdelhamid et al.

1 Introduction 1.1 Background In recent years, there has been an increase in data breaches, security issues, vulnerabilities, and exploits. According to a recent 2022 report [21], healthcare institutions in the United States experienced an average of 1,410 cyberattacks weekly per institution. Additionally, cyberattacks increased by 38% in the same year. To mitigate this issue, institutions have outsourced cybersecurity training to approved vendors, who offer typical training media that may not engage all employees. These training programs assume that all employees possess the same knowledge level and technical background. Unfortunately, research has indicated that employees may not engage with cybersecurity training material. In a recent study [22], the authors reported that employees find cybersecurity training programs unengaging. In 2021, it was estimated that more than 18.28 million individuals work for federal and state governments in the United States. Unfortunately, the currently available training programs do not offer significant benefits, and the number of security breaches continues to rise. This situation underscores the need for a more engaging, effective, and continuous training programs. This paper proposes developing a chatbot system that serves as employees’ personal trainers and go-to security experts. The intelligent agents will customize training based on their organization’s needs and knowledge levels, and users will engage with them during the training process.

1.2 Innovation and Approach Chatbots offer a personalized and engaging approach to cybersecurity training that mimic human–human interaction. Traditional training methods, often lead to poor learning outcomes due to low engagement. Chatbots can provide personalized feedback, guide learners through complex topics, and adapt to the organization’s needs and learners’ knowledge levels. The paper also highlights the potential benefits of integrating chatbots with social network analysis for better cybersecurity protection, including early threat detection, improved risk assessment, and enhanced incident response.

1.3 Research Goal and Objectives The research aims to advance state-of-the-art cybersecurity education and training by providing a more interactive and collaborative learning environment that engages users naturally and effectively. This approach can mitigate possible attacks due to the human factor and address some of the current limitations and shortcomings

Cybersecurity Awareness, Education, and Workplace Training Using …

5

of existing cybersecurity training solutions, particularly regarding user engagement and effectiveness. To achieve this goal, we conducted a literature review to understand the research landscape related to this work. We designed a chatbot system to engage users and improve their learning outcomes through personalized and adaptive training based on their organizational needs, knowledge, skills, and learning preferences. Furthermore, we implemented various chatbot backend services for risk assessment and enhanced cybersecurity protection. The team behind this research work is planning to assess the usability and user experience of the chatbot system and identify opportunities for improvement and refinement.

1.4 Research Questions The following questions provide a framework and serve as a vital tool to plan our research effectively and produce useful findings for the research community. RQ[1] What is the status of the research landscape related to chatbots and cybersecurity? We conducted a sequential multi-method approach involving quantitative and qualitative publication data collection and analysis to address this question. RQ[2] How can chatbots be used for cybersecurity protection and threat assessment? To answer this question, we explored the benefits of empowering chatbots with structural and dynamic network analysis capabilities. Illustrative examples were provided, and a supporting simulation experiment with results was presented. RQ[3] How to accommodate different organizations’ cybersecurity needs? We proposed a multi-tier system design for higher scalability, allowing a flexible exchange of information between the chatbot and various services. In addition, new components can be added over time without affecting the end-user experience. RQ[4] How do users perceive using chatbots for cybersecurity awareness, education, and workplace training? We conducted a survey to collect the users’ beliefs and opinions on using chatbots for cybersecurity training and protection. Understanding users’ perceptions will lead to designing an effective and efficient chatbot system.

6

S. Abdelhamid et al.

2 System Features and Overview 2.1 System Architecture The proposed platform is a comprehensive web-based application system that aims to address the increasing threat of cyberattacks. The system is designed to be highly intelligent and adaptive, utilizing artificially intelligent chatbots, a cloud-based database, web services, and a responsive web-based interface. The main components are shown in Fig. 1. One of the system’s key features is its use of social network analysis (SNA) to monitor and analyze the network structure, record user/device behavior, and predict the outcomes at the organizational level. This service provides a powerful tool for predicting and mitigating virus or attack outbreaks. The system, with the help of security analysts and IT professionals, can take appropriate counterdefenses to prevent the spread of malicious software or other security threats. Furthermore, the chatbot system can learn and adapt to new threats and vulnerabilities over time, improving its ability to respond to security threats. In addition, the chatbot is equipped with speech recognition utilities that allow it to communicate with users naturally and intuitively, making the system more accessible and user-friendly. The system’s overall design is intended to be highly customizable, allowing it to satisfy the various needs of different institutions and organizations. In addition, the chatbot can be adapted to various network topologies and infrastructures. This architecture design makes the proposed platform flexible and versatile to many cybersecurity problems.

Fig. 1 Proposed system main components and services

Cybersecurity Awareness, Education, and Workplace Training Using …

7

Fig. 2 Sample human-chatbot conversations depicting three chatbot use cases: a testing a suspicious link for safety, b supporting cyber awareness by suggesting strong passwords, and c keeping user training up to date

2.2 Artificially Intelligent (Smart) Chatbot The chatbot is the primary interface between users and the system. The rule-based chatbot utilizes natural language processing (NLP) to understand user messages and reply with appropriate advice and guidance. The chatbot has access to a cloud-based database that contains relevant cybersecurity information and training resources. The training materials are organization-specific and provided by the organization staff and administration. The system has a user-friendly and responsive web-based interface that helps users interact with the chatbot from different devices with various screen sizes. Since the system is a cloud-based environment, users can view their resources and monitor their progress and performance in cybersecurity training from any device only if they provide the correct credentials for authentication. Figure 2 illustrates three possible scenarios of interactions between users and the chatbot. The chatbot provides a natural way for users to seek help regarding suspicious links or behaviors. Additionally, it can ensure that users are updated on cybersecurity training and provide continuous reminders.

2.3 Social Network Analysis (SNA) Service The SNA service collects information about each device’s status, user behavior, and underlying network structure and connectivity. The service uses this information to predict the virus spread or attack outbreak, assess the situation, and suggest the appropriate counter-defenses. The system can proactively perform multiple strategies to protect the network by sending warning emails to other users within the same network (e.g., direct contacts), sending reports to the IT staff and security analysts, providing advice on blocking certain communication links, or shutdown/stop selected devices, or inform users to take specific actions. The service’s inner workings utilize graph dynamical system (GDS) and network structural analyses, as shown in Fig. 3.

8

S. Abdelhamid et al.

Fig. 3 The organization network is mapped into an abstract graph form. Each node represents a device/user, and edges in the graph represent the pairwise interactions between each device/user. Once the graph is constructed, the chatbot can conduct both graph dynamical and network structural analyses

The service uses the GDS framework [25–28] to anticipate the magnitude of the virus, malware, or attack spread on the network, given the information about each device/user state, the network connectivity, the state update scheme, and each device/ user behavior. The GDS consists of four main components, including a network of vertices and edges, a set of vertex functions, a set of vertex states, and a predetermined order for executing vertex functions. Graph G represents the interaction network for a population of network users/devices whose dynamics are modeled. Each vertex i corresponds to a device/user, and edges in G represent the pairwise interactions between devices/users. Each node or vertex i has a state xi , an element of K, where K can be assigned 0 (respectively, 1) and typically represents a healthy (respectively, infected) state. The system state x = (x1 , x2 , …, xn ) is a vector of vertex states. Each device/user i has a function fi that describes how the node transitions from one state to another from time t to t + 1 based on the vector of states of vertex i and all its distance-1(direct contact) neighbors at time t. All vertex functions fi execute simultaneously at each time step. This framework has proven applications in various social, biological, and technological systems. It has provided insights into the collective behaviors that emerge from the interactions of individual agents.

2.4 GDS Illustrative Example Consider the organization network, illustrated as a six-vertex graph G in Fig. 4. The system considers each device/user as a vertex, with two states (0,1) representing healthy and infected states, respectively. For simplicity, each vertex is assigned a 2-threshold function that changes a vertex’s state to 1 at time t + 1 if it has at least two direct contacts in state 1 at time t. The update scheme is synchronous, and the initial state of the system shows two vertices in state 1 and the others in state 0. As time progresses, infected vertices spread the infection to their direct contacts, leading to the infection of all devices in the network by time t = 4.

Cybersecurity Awareness, Education, and Workplace Training Using …

9

Fig. 4 This GDS example demonstrates threshold dynamics on a graph, with the system state x = (×1, × 2, × 3, × 4, × 5, × 6) given at each time 0 ≤ t ≤ 4

The example shown in Fig. 4 illustrates the mechanism used by the chatbot for risk assessment. The system can suggest solutions for security analysts or IT personnel. A possible solution to mitigate the impact of the initial infections is demonstrated in Fig. 5. We demonstrate the GDS concept using a 2-threshold model; however, other behavior models can be utilized based on the analyst’s decision. In addition to the network dynamical analysis using GDS, the service collects additional network structure properties. The structure properties can help security specialists and IT professionals in reasoning about how viruses, malware, or other attacks can propagate through the network based on the importance and centrality of users/devices within the same network. For example, we can identify clusters of nodes most vulnerable to infection and the potential pathways through which the virus or malware will likely spread. Analysts can use this information to design strategies for containing and preventing the spread of viruses, malware, or phishing emails. Moreover, network structure analysis can help us identify users/devices (nodes) with high degrees of connectivity (i.e., nodes connected to many other nodes), which are more likely to be involved in spreading infections. The model presented in Fig. 5 demonstrates some of these use cases.

Fig. 5 a The chatbot suggests a counter-defense approach to protect the rest of the network from virus spread after two devices (1 and 2) were infected. b The analyst responded by securing or blocking the suggested device (in blue) from infection. Device or vertex three cannot change its state now. Since vertex 4 has only one neighbor (device 2) in state 1 (infected), its state cannot change to infected at time 2, based on the 2-threshold model. No other vertices can change their state and the system turns into a fixed state. Blocking device three controlled the outbreak since none of the healthy devices (in green) could have two or more infected direct contacts

10

S. Abdelhamid et al.

Fig. 6 Chatbot steps to verify suspicious or phishing links. The workflow adapts the human-in-the-loop methodology where the human analyst is an integral part of the cybersecurity process

2.5 Cloud-Based Database The database contains various cybersecurity resources, such as best practices, tips, guidelines, and training materials, such as videos and interactive quizzes. In addition, the database is accessible to the chatbot and other system components to provide continuous real-time information to users. These are examples of the database’s main entities: user information, records of previous interactions, organization-specific policies and resources, analytics data, configuration settings, and previous phishing/scam email attempts that include the sender’s email address, subject line, the message content, and any links in the email. A workflow of the steps taken by the chatbot to detect possible threats is shown in Fig. 6.

3 Related Work We adopted a multi-method approach in our related work review, where we started with a quantitative bibliographic analysis to get a high-level view and identify the main research areas and trends related to this topic. Finally, we conducted a qualitative literature review and grouped sources based on common themes or topics identified in phase one. This approach allowed for an organized, justified, and cohesive review. In addition, it resulted in a transparent process that showed the rationale behind including and excluding the existing sources [29].

3.1 Bibliographic Analysis To explore the research landscape of using chatbots for cybersecurity education and workplace training, we conducted a bibliometric analysis using OpenAlex, a free and open catalog of online scholarly publications. This analysis aimed to identify research

Cybersecurity Awareness, Education, and Workplace Training Using …

11

Fig. 7 In the co-occurrence network, each node is a key term, and each link is a co-occurrence relationship. Nodes that belong to the same cluster are assigned the same color. The red cluster contains key terms related to the design and development of chatbots. The blue cluster has keywords related to chatbot-human interactions and personalization. The yellow cluster focuses on ethics and concerns related to new trends in chatbots, including ChatGPT. Applications of chatbots, especially in the medical and health sciences, were the central theme of the green cluster. This figure illustrates how bibliographic analysis combined with clustering and visualization can provide a high-level wide view lens of the current themes related to chatbots

themes related to our topic of interest. We retrieved all publications that contain the keyword “chatbot” in the title or abstract sections. Then we extracted the terms with a minimum occurrence of fifteen. Finally, based on the co-occurrence frequency, we constructed the network of key terms, as shown in Fig. 7. This approach helped to quantitively identify the major research themes related to chatbots and provide a comprehensive understanding of the current state of research in this field.

3.2 Literature Review We used the identified themes and clusters from the previous phase (bibliographic analysis) to sample sources and guide our qualitative literature review. Chatbot Design and Development. This research theme focuses on developing chatbots that respond to security threats and integrate with other cybersecurity tools. A study [1] proposed chatbot framework that can detect humans accurately in realtime and use Facebook messenger. Others proposed chatbots that respond accurately to user queries using pre-processed data [2] and automate web-based communication [3]. Additionally, researchers have conducted technical reviews of modern chatbot systems to evaluate their architectural design and implementation process [23]. User Interaction and Engagement. This research theme delves into designing chatbots to improve user engagement and motivation [4]. Studies have found that users’

12

S. Abdelhamid et al.

conversational expectations and behavior with chatbots are like those in humanto-human interactions [5], but there is greater diversity in information shared with chatbots. Furthermore, people use discursive strategies to create “shared meaning” and identity for themselves when interacting with chatbots [6]. Anthropomorphic chatbot profiles have been found to enhance consumer engagement outcomes. A mobile application-based chatbot named Alpha was also proposed as an intelligent teaching assistant to enhance the students’ learning processes and engagement [7]. Personalization and Customization. This research theme is centered around developing chatbots that can provide customized and personalized training to individual users based on their roles, responsibilities, and learning needs. Developing chatbots is relatively straightforward; however, determining the appropriate chatbot behavior is a complex problem. In a research work [8], a pre-study was conducted to identify the most critical security behavioral problems for users. Based on the survey results, three topics were identified: passwords, privacy, and secure browsing. Another study [9] designed a chatbot system to improve social skills training for security guards. A pilot experiment and survey were conducted to evaluate the acceptance and impact of the system, with results suggesting that the chatbot system is an effective tool for enhancing social skills training for security guards. SecBot [10] is another chatbot for supporting cybersecurity planning and management. SecBot can identify cyberattacks, suggest solutions, and provide information for decision-making on cybersecurity risks. Security and Privacy. This theme focuses on identifying potential concerns, risks, and vulnerabilities associated with chatbots. In one research work [11], researchers studied the problems related to data storage and communication between users and chatbots. In another work [12], the authors pointed out the potential vulnerabilities in existing chatbot architectures. They found that attackers can exploit these vulnerabilities to access sensitive information or disrupt the chatbot’s functioning. Another research study [13] found that 6.29% of chatbots use insecure communication protocols to user chats. Another research study [14] indicated that chatbots with human-like features are perceived as more anthropomorphic, resulting in more trust and lower privacy concerns from users. Evaluation and Effectiveness. This research theme examines the effectiveness of chatbots in education and training. One study [15] found that a chatbot system for Object-Oriented Programming Languages improved memory retention and learning outcomes compared to using a search engine. Another paper [16] suggested evaluating chatbots through linguistically-motivated comparisons of human and chatbot dialogues, as well as open-ended trials. Unfortunately, a review [17] of chatbot evaluation techniques based found no standardized way of measuring chatbot quality. Ethical and Legal Considerations. This research theme focuses on the ethical and legal considerations of chatbot use. A literature analysis [18] identified ethical challenges in chatbots and proposed a conceptual model to address them. Concerns were raised about chatbots learning offensive behavior from large unlabeled datasets, and

Cybersecurity Awareness, Education, and Workplace Training Using …

13

safer techniques were proposed [19]. A study [24] found that 10–50% of interactions with chatbots were abusive, and there is a need to understand and address this problem through the design of chatbots that can stop such behavior. Summary. The study of related works revealed the increased research on chatbots and their applications in various domains. However, only a few studies have addressed their applications in cybersecurity awareness or protection. Additionally, none of these studies considered integrating social network analysis for risk assessment. Chatbots can have computational capabilities in addition to their linguistic ones.

4 User Perception Survey We conducted a survey to identify the users’ perception of using chatbots for cybersecurity protection and awareness. The survey included twelve questions that focused on how users perceive the chatbots’ effectiveness and their role in improving training engagement and motivation. Results showed that 33% of the respondents think cyber awareness education should be provided continuously. More than 66% think that integrating chatbots for cybersecurity awareness and workplace training would be beneficial. When asked about the most needed feature they would like to see in the chatbot, the top selection was “Immediate answers and advice on what to do in various threatening situations,” followed by “Continuous support and education.” When asked how they feel about using chatbots for training in terms of user engagement? (On scale from 0 to 10, where 10 means very engaging),” the average score was 7.2. Additionally, when asked the same question in terms of motivation (On scale from 0 to 10, where 10 means very motivating),” the average score was 6.5. Additional results from the survey are shown in Fig. 8. Overall, there was a positive perception of chatbots use in this domain.

(a)

(b)

(c)

Fig. 8 User responses to selected questions a “Which of the following sectors do you think chatbot technology has practical applications in?”, b “How do you think human-induced attacks and errors can be prevented?”, and c “In your opinion, what are the best effective approaches to protect ourselves from cyberattacks?”

14

S. Abdelhamid et al.

5 Cyberattack Simulation To demonstrate the chatbot’s ability to conduct risk assessments and anticipate virus or malicious behavior outbreaks, we created a simulation model of an organization network based on the GDS framework. The chatbot system, specifically the network analysis service (SNA), reads the network status and each device/user behavior and predicts the outcomes. Each user/device is represented as a node, and each link represents the pairwise communication between two nodes. In our model, each device starts in the state healthy (H) and moves to the state infected (I) with probability γ when infected by direct contacts in state (I). Devices in the state (I) can move back to the state (H) with probability π. If a device stays in state (I) for more than a period (t), then there is a probability α that it will stop working or drop out of the network. We used fixed values for π = 0.8, γ = 1.5, α = 0.3, and set the initially infected devices to five for our experiment setup.

5.1 Experiment 1 (Effect of the Average Number of Direct Contacts) The model defined the infection threshold as 50% of the devices. The time to exceed the threshold for various network topologies was recorded. The average number of direct contacts “d” per device was set to 1, 3, 5, 10, 20, and 30. The chatbot using the SNA service showed that d has an effect only at a value of five or more. As d increased, reaching the threshold decreased from 170 to 5 days only. The outcome was alarming as it showed how fast the malware can diffuse through a network in a very short time.

5.2 Experiment 2 (Different Intervention Forms) We conducted three types of blocking strategies: 1) block devices with the highest number of direct connections, 2) block devices with the highest betweenness centrality, and 3) block a random set of devices. The goal was to simulate situations where the organization network is under attack, show how the chatbot will conduct a risk assessment, and propose a solution to control the outbreak. After running the simulation several times and calculating the average values, the chatbot found that blocking devices with the highest number of direct contacts provided the best results with the lowest number of infected devices. These simulations illustrate how the chatbot system can aid IT analysts in decision-making.

Cybersecurity Awareness, Education, and Workplace Training Using …

15

6 Conclusion and Future Work The rising number of data breaches and security issues highlight the importance of effective cybersecurity training programs. Unfortunately, traditional training methods are unengaging and do not provide immediate support. In this paper, we developed a chatbot system to provide a more personalized approach to cybersecurity training. The proposed chatbot system provides immediate feedback and active engagement. The design is innovative as it is a multi-tier extensible system and the first to explore the potential benefits of integrating chatbots with social network analysis for better cybersecurity protection and risk assessment. The results from the user survey support the need for continuous security support and cyber awareness training. There was a positive perception among respondents regarding the use of chatbots in this domain. Additionally, the simulation results demonstrated selected chatbot use cases. Our future work includes assessing usability and identifying opportunities for system improvement. Acknowledgements This work was supported in part by the Commonwealth Cyber Initiative, an investment in the advancement of cyber R&D, innovation, and workforce development. For more information about CCI, visit https://cyberinitiative.org/.

References 1. Van Cuong, T., Tan, T.M.: Design and implementation of chatbot framework for network security cameras. In: 2019 International Conference on System Science and Engineering (ICSSE), pp. 324–328. IEEE (2019) 2. Pavitha, N., Bhatele, P., Desai, S., Pande H.: Design and implementation of multipurpose Chatbot. In: 2022 4th International Conference on Smart Systems and Inventive Technology (ICSSIT), pp. 1332–1337. IEEE (2022) 3. Lakshmi, K.N., Kishore Reddy, Y., Kireeti, M. Swathi, T., Ismail, M.: Design and implementation of student chatbot using AIML and LSA. Int. J. Innov. Technol. Explor. Eng. 8, 1742–1746 (2019) 4. Portela, M., Granell-Canut, C.: A new friend in our smartphone? Observing interactions with chatbots in the search of emotional engagement. In: Proceedings of the XVIII International Conference on Human Computer Interaction, pp. 1–7 (2017) 5. Dev, J., Jean Camp, L.: User engagement with chatbots: a discursive psychology approach. In: The 2nd Conference on Conversational User Interfaces, pp. 1–4 (2020) 6. Tsai, W.-H., Liu, Yu., Chuan, C.-H.: How chatbots’ social presence communication enhances consumer engagement: the mediating role of parasocial interaction and dialogue. J. Res. Interact. Mark. 15, 460–482 (2021) 7. Abdelhamid, S., Katz, A.: Using Chatbots as smart teaching assistants for first-year engineering students. In: 2020 First-Year Engineering Experience (2020) 8. Gulenko, I.: Chatbot for IT security training: using motivational interviewing to improve security behavior. In: AIST (supplement), pp. 7–16 (2014) 9. de Bever, S., Formolo, D., Wang, S., Bosse, T.: A multimodal chatbot system for enhancing social skills training for security guards. In: Human-Computer Interaction. Perspectives on Design: Thematic Area, HCII 2019, Orlando, FL, USA, July 26–31, 2019, Proceedings, pp. 499–513. Springer International Publishing (2019)

16

S. Abdelhamid et al.

10. Franco, M.F., et al.: SecBot: a business-driven conversational agent for cybersecurity planning and management. In: 2020 16th International Conference on Network and Service Management (CNSM), pp. 1–7 (2020) 11. Hasal, M., Nowaková, J., Saghair, K.A., Abdulla, H., Snášel, V., Ogiela, L.: Chatbots: security, privacy, data protection, and social aspects. Concurr. Comput. Pract. Exp. 33(19) (2021) 12. Ye, W., Qun, L.: Chatbot security and privacy in the age of personal assistants. In: 2020 IEEE/ ACM Symposium on Edge Computing (SEC), pp. 388–393. IEEE (2020) 13. Waheed, N., Ikram, M., Hashmi, S.S., He, X., Nanda, P.: An empirical assessment of security and privacy risks of web-based Chatbots. In: 23rd International Conference, Biarritz, France, pp. 325–339, Springer Publishing (2022) 14. Ischen, C., Araujo, T., Voorveld, H., van Noort, G., Smit, E.: Privacy concerns in chatbot interactions. In: Følstad, A., et al. (eds.) Chatbot Research and Design. CONVERSATIONS 2019. Lecture Notes in Computer Science, vol. 11970. Springer, Cham (2019). https://doi.org/ 10.1007/978-3-030-39540-7_3 15. Abbasi, S., Kazi, H.: Measuring effectiveness of learning chatbot systems on student’s learning outcome and memory retention. Asian J. Appl. Sci. Eng. 3(2), 251–260 (2014) 16. Abu Shawar, B., Atwell, E.:Evaluation of chatbot information system. In: The 8th Maghrebian Conference on Software Engineering and Artificial Intelligence (2004) 17. Casas, J., Tricot, M.O., Khaled, O.A., Mugellini, E., Cudré-Mauroux, P.: Trends & methods in chatbot evaluation. In: the International Conference on Multimodal Interaction, pp. 280–286 (2020) 18. Murtarelli, G., Gregory, A., Romenti, S.: A conversation-based perspective for shaping ethical human–machine interactions: the particular challenge of chatbots. J. Bus. Res. 130, 927–935 (2021) 19. Xu, J., Da, J., Margaret, L., Boureau, Y-L., Weston, J., Dinan, E.: Recipes for safety in opendomain chatbots. arXiv preprint arXiv:2010.07079 (2020) 20. Checkpoint research reports. https://blog.checkpoint.com/2023/01/05/38-increase-in-2022global-cyberattacks/. Accessed 2 Dec 2022 21. Reeves, A., Calic, D., Delfabbro P.: Get a red-hot poker and open up my eyes, it’s so boring: employee perceptions of cybersecurity training. Comput. Secur. 106 (2021) 22. Cahn, J.: CHATBOT: Architecture, Design, & Development. University of Pennsylvania School of Engineering and Applied Science (2017) 23. Lokman, A.S., Ameedeen, M.A.: Modern chatbot systems: a technical review. In: Proceedings of the Future Technologies Conference (FTC) 2018: Vol. 2, pp. 1012–1023. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-02683-7_75 24. Siegel, J.: The Ethical Implications of the Chatbot User Experience. Bentley University.:/ /www.bentley.edu/centers/user-experience-center/ethical-implications-chatbot-user-experi ence. Accessed 2 Dec 2022 25. Mortveit, H., Reidys, C.: An Introduction to Sequential Dynamical Systems. Springer, New York (2007) 26. Abdelhamid, S., Kuhlman, C.J., Korkmaz, G., Marathe, M.V., Ravi, S.S.: Edison: a web application for computational health informatics at scale. In: Proceedings of the 6th ACM Conference on Bioinformatics, Computational Biology and Health Informatics, pp. 413–422 (2015) 27. Abdelhamid, S., Kuhlman, C.J., Marathe, M.V., Ravi, S.S., Reid, K.: Agent-based modeling and simulation of depression and its impact on student success and academic retention. In: ASEE Annual Conference & Exposition (2016) 28. Abdelhamid, S.: Providing high performance computing based models as a service: architecture and services for modeling contagions on large networked populations. Ph.D. diss., Virginia Tech, (2017) 29. Brocke, J.V., Simons, A., Niehaves, B., Niehaves, B., Reimer, K., Plattfaut, R., Cleven, A.:Reconstructing the giant: on the importance of rigour in documenting the literature search process (2009)

Creating Binge-Worthy e-Learning Experiences Nafiza Akter

Abstract Engaging employees in learning activities has always been challenging and is becoming increasing more difficult as employees navigate through their complex workdays. Although the research clearly shows that narratives, conversational language, and emotions help learners better retain learning materials, there are few instances in which those elements make it into formal training opportunities. Learning and development professionals can draw inspiration from phenomenon like binge-watching television to transform learning experiences and creating meaningful, memorable, and relatable learning experience. This not only simulates a reallife activity that feels natural to learners, but it also invites them into a learning experience that can be delivered using emotions and conversational language. Keywords Adult learners · E-learning · Narratives · Storytelling

1 Introduction The average employee regularly navigates complex challenges on a daily basis. Just imagine: you finally have a break after a series of back-to-back meetings and have a half an hour to get through some tasks you set aside for the day. Although you’re getting hungry, you decide you just need to get through the tasks and open up your email to get started with follow-ups — but at the top of your inbox is an urgent email about an audit that needs to be addressed right away. Then you see your instant messenger is lighting up: you have messages from colleagues about the audit, you have an employee that is unwell and needs help, you have some messages about things colleagues need for upcoming meetings, and you have two corporate-wide news announcements. Oh, and you missed a call from your doctor’s office about your lab results that can only be disclosed during a phone call. N. Akter (B) Pfizer, New York, NY, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_2

17

18

N. Akter

The last thing on your mind at the moment? The training you have due by end of day today.

1.1 The State of Existing Training For the busy employee, required training is often coming to them at a time they are trying to meet many other demands. They certainly will complete the training to meet compliance needs; however, are they actually engaged? Are they in a mindset to learn? Is the training designed in a way that leverages adult learning principles that helps them learn? The challenge that exists is that many trainings are designed in such a way that they use technical language and have limited interactions, which are mostly presented as checks for understanding quizzes. Although this gives employees the technical information they need, it may not be presented in memorable and conversational language that they can easily connect to [5]. Furthermore, science also backs that memory is influenced by emotions and leveraging strong emotions can help with better remembering the learning materials presented [7]. So the typically “neutral” language used in technical information does not necessarily help learners better recall the information presented. This, coupled with the issue of many trainings being the typical page-turner experiences with limited interactivity [4], makes it hard to really feel excited about completing that training before the end of your already busy day. There is also a clear disconnect between the complexity of interconnected, ambiguous, real-life challenges that employees navigate daily and the traditional training language provided.

2 A Glimmer of Hope: Opportunity for Change A training opportunity surfaced where we were working with stakeholders that were genuinely invested in pushing the boundaries of traditional training to better engage learners. This is a training that goes out to employees on an annual basis and reaches a large number of individuals. The training is centered around good clinical practices — more specifically how upholding quality in the day-to-day work is central to these practices. Knowing that the stakeholders are invested in seeing a change, this seemed like an opportunity to creatively experiment and, ultimately, create a more engaging learning experience. As the training was so closely tied to day-to-day work, it seemed ideal to use a narrative approach, which would allow the use of conversational language and emotional connections. Narratives are also ideal for working with a global audience as they appeal to individuals across cultures and age groups, while still delivering crucial information that builds knowledge and abilities [2].

Creating Binge-Worthy e-Learning Experiences

19

Additionally, narratives are ideal for capturing the complexity that exists within healthcare, where it is not only important to understand protocols and processes but to also learn ethics, culture, values, and sound judgement. In order to help learners connect training materials to their practice, we had to move beyond the traditional approach of providing theoretical information that is abstracted from practice. We wanted to use the idea of a narrative pedagogical approach, which “goes beyond acquisition of knowledge and skills” and fosters critical thinking [3]. We also know that our subset of learners wanted to see more narrative pedagogy, which we discovered through analyzing a series of focus groups (3 total) on different educational interventions. The challenge we face when trying to engage subject matter experts in achieving this is simply that they do not have time or capacity to help deliver this synchronously or asynchronously (video, podcast, or even written vignettes); however, through using a first-person narrative, we could essentially simulate narrative pedagogy. Ultimately, this approach would help connect the practical real-life situations of our healthcare professionals experience with the underlying theories of good clinical practices. The challenge that existed was simply that the language we had to work with was the traditional technical language written for a broad audience. The message stayed the same: quality and good clinical practices are essential to our success and we have to work together to ensure we uphold our standards. The first task was to reimagine the language and construct a narrative around the key message that we knew we wanted to deliver.

2.1 Shifting the Tone, Look, and Feel of Training I set up sessions with my colleagues for bursts of time (2–3 h each) to work together and rewrite the language as a story that followed a colleague through her work-fromhome day. The main character would be going through the experience of discovering the good clinical practices quality standards, talking to other colleagues that are experts on this, and eventually referring to them when incidences come up in her daily work. It was important to write in the first-person narrative, as we wanted to simulate a form of narrative pedagogy where learners believe they are learning from the experiences of their colleague—which we know to be effective in healthcare and from focus group findings of our subset of learners. Specifically in healthcare, “using first person narratives were more than twice as likely to find an effect as those using third-person narratives (43 versus 20%, respectively)” [8]. We went through several sessions together of writing and rewriting, then sharing the drafts with our stakeholders, and integrating their feedback into the final revision.

20

N. Akter

2.2 Making Learners Feel Connected to Training The writing process was fun because it leveraged and fused the divergent ideas and creativity we each brought to the sessions. We also recognized that in order to really connect to individuals on an emotional level, we needed to spend as much time writing for the storytelling elements as we did the content for the course. Furthermore, it was vital that the narrative was as authentic to the lived experiences of learners, as we were trying to draw in elements from the narrative pedagogical approach that we knew our learners craved. So we spent time messaging colleagues who were in the roles of the actors to make sure we understood their interactions with other actors and the processes as deeply as we could. For example, we often asked things like, “When would you engage another colleague in x role within y situation?” Although this seems minor, our facial expressions, along with our tone and language change when interacting with individuals depending on our level of familiarity with them. Therefore, even if the answers did not make it into the script, it was important for actors to know when they’d be reaching out to someone they had an existing relationship with versus a person they were meeting for the first time. The writing process also forced us to reflect on ourselves as employees and our daily habits working from home. How do we start our day? How do we work with others? What happens when something urgent comes up? What are some quirks or roadblocks we face when doing daily tasks? As context is critical to a successful narrative [2], we integrated these storytelling elements into the narrative so that the learner could feel like they relate to the main character on an emotional level and felt as though they identified with the character. We included details such as the main character starting her work-from-home day after a jog, logging onto her computer for the day while her coffee was brewing, and she gets up to get coffee all the while ignoring the distant sound of her neighbor’s dog occasionally barking. The videos also show the main character using company devices, navigating through company webpages to find artifacts related to the topic, and getting emails and notifications from applications used by the company. We finally came up with a series of three short videos that delivered the core messaging we wanted to convey to learners taking the training. We could definitely take the three videos and put them in a traditional e-learning format where they live on a slide and learners click through slides to get the videos; however, we wanted to push this a bit further.

2.3 Creating a Binge-Worthy Learning Experience: Simulating Binge-Watching Phenomenon in e-Learning It is clear that binge-watching is a social phenomenon [6], and many colleagues can relate to sitting down to binge-watch something on their favorite streaming applications. What if we leveraged this component of an employee’s daily life to

Creating Binge-Worthy e-Learning Experiences

21

better connect to them? Could we put the videos into a container that simulates a video streaming application? A recent study found that binge-watching is most common on laptops and tends to occur more frequently during weekdays [1]; this made simulating a binge-watching experience even more ideal as those conditions would be typically present for most individuals during working hours. Starting a training that looks like a video streaming application creates a different feeling from the traditional experience. As emotion is key, seeing a streaming application pop-up immediately creates a different feeling, surprising the learner but also drawing on their familiarity with these apps. It may also help connect the topic of the e-learning to daily life activities, in addition to the training actually simulating the day-to-day life of a colleague. Ultimately, we created an interface that resembled the user-experience in popular streaming applications (see Fig. 1). We integrated popular features, such as displaying the upcoming episode on the lower-right corner towards the end of an episode and auto playing the next episode after a certain number of seconds. Ensuring Accessibility for a Broad, Global, Audience. One of the key reasons we wanted to keep the episodes short was to ensure that the video experience was accessible for our global audience, especially in lower bandwidth areas. During the writing process, we paid special attention to using global English to ensure greater understanding of the content. We also wanted to ensure diversity and inclusion was integrated throughout. We selected actors and character names that represented diversity. In the spirit of inclusion, we also paid special attention to captions by including descriptive captions (e.g., dogs barking in the background, or the coffee maker sound dispensing) and enabled closed-captioning by default.

Fig. 1 Simulation of a video streaming platform to bring together three short episodes

22

N. Akter

3 Learner Reception and Feedback Overall, there was a 98% positive rating on the feedback provided (2% margin of error at a 95% confidence level). The qualitative feedback highlighted that learner recognized and appreciated the “a real case ‘day-to-day’ scenario” presented, as well as the overall design and functionality. Feedback also revealed that learners perceived this as an “innovative approach” and understood the use of the “storytelling” approach. Although we have limited access to LMS data, the feedback data also revealed that learners were willing to provide their contact information to provide further feedback/clarify their feedback. Out of approximately 95,000 feedback entries provided since March 2022, an average of 4% of participants were willing to provide their contact information for further discussion. In this course specifically, approximately 9% of participants were willing to provide further feedback. This was an interesting finding for us, and we will leverage this as an opportunity to reach out to gather deeper feedback on learner experience and preferences through focus groups and surveys. This subset of learners, along with previous focus group participants, can help us better understand whether this training intervention provided the components of narrative pedagogy that learners have requested, and ultimately fostered critical thinking (ethics, culture, values, sound judgement). Furthermore, they can also provide critical insights on what could be done to improve future offerings. Some learners with a learning and development background also left comments asking that insights from the design and development process be shared with other groups. Learners wanted to see improvements in terms of better access to the related materials. The related materials popped-up at the end as a “You may also be interested in…” but some learners (approximately 0.2%) reported missing them. The interface can definitely be improved to better highlight these materials or automatically redirect learners to them in future iterations of the course. Through open-ended qualitative feedback, learners said that the most effective components were: the format (video approach), the short length, the storytelling approach using a real-life/day-to-day case study, and the quality of the overall product. As fewer participants opted to provide qualitative feedback, the margin of error is 4% at a confidence level of 95%.

4 Conclusion Employees are working in a highly complex environment, managing various responsibilities within and outside of work. Training opportunities often end up feeling like yet another task on the list of to-dos, which undermines the importance of learning. This not only limits the individual employee in their growth, but it also hinders organizational growth as individuals do not have the time or bandwidth to learn. Through creating learning experiences as immersive experiences that learners can easily connect to and understand, we can transform the look, feel, and tone of

Creating Binge-Worthy e-Learning Experiences

23

training opportunities. Leveraging conversational language and emotion, as well as simulating daily life, helped us achieve this goal. I challenge you to create your own binge-worthy learning experiences, whether it comes in the form of following an employee throughout their workday, solving mysteries behind defective products, or crossword puzzles that raise awareness on various company initiatives. Acknowledgements I am deeply thankful to my colleagues that enabled this work to come to life: Anthony Loria, Napoleon Juaniza, Peter Sorenson, Petcharat Jones, and Joana Brenner. A special thanks to Martin Warters for supporting these endeavors, and Fran Bahn for working with stakeholders to ensure all the components launch successfully.

References 1. Castro, D., Rigby, J.M., Cabral, D., Nisi, V.: The Binge-Watcher’s journey: investigating motivations, contexts, and affective states surrounding Netflix viewing. Converg. Int. J. Res. New Media Technol. 27(1), 3–20 (2021). https://doi.org/10.1177/1354856519890856 2. Dettori, G., Morselli, F.: Accessing knowledge through narrative context. In: Kendall, M., Samways, B. (eds.) IFIP – The International Federation for Information Processing, vol. 281, pp. 253–260. Springer, Boston (2008). https://doi.org/10.1007/978-0-387-09729-9_39. 3. Gilkison, A., Giddings, L., Smythe, L.: Real life narratives enhance learning about the ‘art and science’ of midwifery practice. Adv. Health Sci. Educ. 21, 19–32 (2016) 4. Guralnick, D., Larson, D.: The cultural impact of e-learning and intranets on corporate employees. Connected minds, emerging cultures. In: Wheeler, S. (ed.) Cybercultures in Online Learning, pp. 251–252, IAP (2009) 5. Mayer, R.E.: Multimedia learning. Psychol. Learn. Motiv. 41, 85–139 (2002) 6. Starosta, J.A., Izydorczyk, B.: Understanding the phenomenon of binge-watching—a systematic review. Int. J. Environ. Res. Public Health 17(12), 4469 (2020) 7. Tyng, C.M., Amin, H.U., Saad, M.N., Malik, A.S.: The influences of emotion on learning and memory. Front. Psychol. 8 (2017) 8. Winterbottom, A., Bekker, H.L., Conner, M., Mooney, A.: Does narrative information bias individual’s decision making? A systematic review. Soc. Sci. Med. 67(12), 2079–2088 (2008)

Thinking and Chatting Deontically—Novel Support of Communication for Learning and Training with Time Travel Prevention Games Oksana Arnold, Ronny Franke, Klaus P. Jantke , Rainer Knauf, Tanja Schramm, and Hans-Holger Wache

Abstract The authors’ key area of application is training for the prevention of accidents in the process technology industries. They run a professional training center with own 3D virtual environments. At TLIC 2021, four of the present authors delivered a contribution advocating planning of human training experiences as dynamically as managing some severely disturbed technical system back into a normal operation—such as an out of control chemical reactor—and enabling human trainees who failed to complete a risky task—thereby possibly ruining a (fortunately only virtual) technical installation—to virtually travel back in time to make good the damage. At TLIC 2022, they introduced cascades of gradually more intricate categories of time travel games. With every step from one category to the next, the deployed AI gets more powerful and effective in providing adaptive guidance of human trainees. The most advanced time travel games are those that allow for the dynamic modification of events experienced in the virtual past. In this way, the game system evolves over time and adapts to the needs of human trainees with emphasis on guidance for trainees who fail repeatedly. The extended team of authors presents a novel perspective at time travel prevention games that leads to a more human-centered adaptive guidance. O. Arnold Erfurt University of Applied Sciences, Altonaer Str. 25, 99085 Erfurt, Germany R. Franke Fraunhofer Institut für Fabrikbetrieb und -Automatisierung IFF, Sandtorstr. 22, 39106 Magdeburg, Germany K. P. Jantke (B) ADICOM Software, Frauentorstr. 11, 99423 Weimar, Germany e-mail: [email protected] R. Knauf · T. Schramm Ilmenau University of Technology, P.O. Box 10 05 65, 98684 Ilmenau, Germany H.-H. Wache Institution for Statutory Accidents Insurance and Prevention for Raw Materials and Chemical Industry, Prevention Center Berlin, Innsbrucker Str. 26/27, 10825 Berlin, Germany © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_3

25

26

O. Arnold et al.

Training is seen through the lens of deontic modal logic. The focus is on undesired events such as explosions, fire, health hazards due to toxic vapors, and the like. The game system’s AI is reasoning about necessity and possibility of such events. It offers to human trainees/players helpful chats about modalities of decisive events of training. Keywords Technology enhanced learning · Game based training · Time travel prevention games · Accident prevention · Storyboarding · Adaptivity · Human-system communication · Deontic Logic

1 Introduction In this short introductory section, the authors sketch the practical focus of their work, briefly describe their methodological approach, mention key technologies invoked, and relate all this to modal logic, in general, and to deontic thinking, in particular.

1.1 Training for Industrial Accident Prevention—The Focus of Applications Some years ago, the authors have been engaged in large-scale training applications in the area of civil protection and disaster management [12, 13]. Their emphasis has been shifted toward industrial accident prevention [9–11]. More than motor skills and craft, cognitive competencies and skills are in focus. This requires a high level of adaptivity to the human trainee’s needs and desires.

1.2 Learning and Training with Time Travel Prevention Games The concept of time travel exploratory games has been introduced and studied in some depth in refs. [7] and [8], respectively. Time travel prevention games form a more intriguing subclass of exploratory games; the term has been coined a few years earlier by the present third author on the conference and expo named German Prevention Day [32]. In time travel prevention games, trainees get offered opportunities to impact the fate. The authors’ put it earlier; nothing is more effective than a self-induced accident being memorable and sometimes worth telling. With the option to impact the fate, human trainees get opportunities to finally succeed based on their own competencies. Affective training becomes effective.

Thinking and Chatting Deontically—Novel Support of Communication …

27

1.3 Plan Generation, Storyboarding and Time Travel Gamification According to the practical goals and to the didactic approach to game-based training, a key task is the design—game design—of affective and effective human experiences. The underlying design methodology is digital storyboarding as developed in ref. [34] and successfully demonstrated in largely varying application areas [7–13, 30, 31, 36, 37, 45]. In fact, storyboarding is planning that expands upon dynamic plan generation [3] using expressive graph-theoretic concepts [4, 5]. Every storyboard is a collection of graphs hierarchically structured with respect to graph expansion by node substitution. The use of the technologies will be exemplified throughout the main part of this contribution. The overall process is time travel gamification [36].

1.4 Artificial Intelligence for Adaptive Trainee Guidance In complex environments performing risky operations, it is hardly surprising that trainees may fail, even repeatedly. Nevertheless, the ultimate goal of playful training is the experience of self-reliant mastery. Adaptive care leads the trainee to a success. For this purpose, the digital game system needs to understand the human trainee— it hypothesizes theories of mind [16, 17, 20, 33, 35] to provide adaptive guidance [11]. Reasoning about necessities, possibilities, and the needs to act is essential.

2 Training with Time Travel Prevention Games Exemplified The present section is intended to demonstrate the authors’ gamification approach— published very recently [36]—and to set the stage for an investigation of the novelty of modal logic applications to training by means of time travel prevention games. This section’s content reflects the state of the art prior to the present contribution. Accident prevention training is important in complex and risky environments such as chemical industry installations. Prevention training is of societal relevance. In a virtual world such as the one on display in Fig. 1, human trainees get assigned certain tasks. It rather frequently happens that a training session suddenly ends with an undesired event such as an explosion with fire as on display of Fig. 2. Notice that—from a didactical point of view—those undesired events are desirable. As the authors put it already earlier in several places, nothing is more affective and, thus, effective than the experience of a self-induced accident.

28

O. Arnold et al.

Fig. 1 Virtual factory by Fraunhofer IFF deployed by the German Institution for Statutory Accident Insurance and Prevention for Raw Materials and Chemical Industry for the purpose of accident prevention training Fig. 2 A trainee’s self-induced accident: Unforgettable experience, motivation for time travel, and origin of a successful completion experienced as the trainee’s own achievement

But for many good reasons, the authors prefer experiences like that only virtually. In time travel prevention games, trainees get a second chance, perhaps even a third one, a fourth one, and so on. There are offered opportunities to impact the fate. Digital storyboarding is the organization of experience [34, p. 25]. Teams of designers representing different areas of competence such as the application domain, didactics, learning psychology, digital game design, augmented and virtual reality, and the like negotiate variants of anticipated training experience. Storyboard graphs specify intended human-system interactions. Time travel gamification is bringing in cycles into the designed experience (Fig. 3). For selection of the concrete destination of time travel, the authors prefer a rather intuitive time tunnel (see Fig. 4). Players may zoom in or out and select a scene from the past to travel to. The time tunnel has three buttons on top. With the right button, one dives forward into the past, and with the left button, one turns back coming stepwise closer to the present. A click to the button in the middle does select the central object on display and brings the player to the past scene inside of the core game based training episode where this object has been used.

Thinking and Chatting Deontically—Novel Support of Communication …

29

Fig. 3 A top-level storyboard graph according to the technology adopted and adapted from [11] and applied within the process of time travel gamification [36] Fig. 4 A Time tunnel specifying destinations by means of objects that represent scenes of interaction from the past—shown are scenes within the episode of Core Game Base Training (Fig. 3)

The crux is that repeatedly trying does not necessarily lead to an ultimate success. Effective, sustainable training, however, shall be experienced as a finally successful. Toward this very end, the authors developed what they call cascades of intelligent user guidance [11]. The system’s adaptivity is key to the human’s success of training.

3 Modalities and Deontic Reasoning in a Nutshell Within the limits of the present publication, this section about the underlying formal reasoning concepts must be kept short; no severe problem due to rich sources such as [1, 2, 14, 15, 21–24, 27–29, 40, 41, 46, 47].

3.1 The Rise of Modalities in Prevention Training In training sessions as exemplified in the preceding section, it is essential to ponder the inevitability of events like the one illustrated in Fig. 2. According to Garson [23], “modal logic is, strictly speaking, the study of the deductive behavior of the expressions ‘it is necessary that’ and ‘it is possible that.’”. In modal logic, variables p, q, and so on represent statements similar to variables of conventional propositional logic. From a logical point of view, time travel aims at the change of modalities. Trainees get offered opportunities to modify the virtual

30

O. Arnold et al.

past (e.g., to impact the fate such that—by way of illustration—“it is necessary that p” turns into “it is possible that ¬p”). The formal terms of modal logic are “□p” and “ ¬p”, respectively. For modeling causality in application domains, this approach might be appropriate. In an educational context, the interest shifts from the domain to the human trainees. Training goals are “oughts” [18, 29]. Educators want their trainees to acquire domain-specific normative concepts. About a century ago, the Austrian philosopher Mally developed “fundamental principles of the logic of ought” [40, 29, p. 5]. This developed into what we nowadays call deontic logic.

3.2 Practical Deontic Reasoning and the Role of Axiomatization In deontic logic writings, the authors abandon the modal logic operators □ and . Instead, they prefer operators like O for “ought to,” OB for “obligatory that,” PE for “permissible that,” and so on. At this point, it is hard to suppress an interesting reference to Johan van Benthem who reports a student’s request to formalize the idea “nothing is absolutely relative” [46, p. 12]. He came up with a proposal: ¬□ ( p∧ ¬p). Interestingly, there is no need for novel operators. The crux is that there are as many interpretations of deontic operators as authors writing about. As soon as practitioners come in, the space of interpretations explodes. There is an overload of bias. Authors are sneaking in what they have in mind. The only way out seems to be a strict axiomatization, i.e. to express clearly what relationships are intended and to exclude those that contradict the intended meaning. Ruzsa provides an excellent case study [41]. By way of illustration, Kurt Gödel was interested in a modal relationship such as “□p → p” [26]. We do not want an axiom like this, because whatever modal statement is made about an event p, it should not necessarily imply p. Very loosely speaking, the insight that an accident p appears unavoidable does not mean that p takes place. The authors’ axiomatic opinions, so to speak, will be exemplified throughout the case study within the next section demonstrating deontic communication at work. Following [35], learner modeling becomes theory of mind induction, but deontically.

4 Time Travel Prevention Games with Deontic Communication The authors closed the preceding section with a remark on theory of mind induction. Inspired by work in the behavioral sciences such as [16, 17, 20], they developed an approach to computerized theory of mind modeling and induction as an advanced

Thinking and Chatting Deontically—Novel Support of Communication …

31

form of learner/player/trainee modeling [6, 35] and provided implementations [43, 44]. Interested readers may get some impression of the touch and feel of computerized learning of theories of mind and may see how well it can really work in practice [33]. Due to its sophistication, theory of mind induction cannot be discussed in much detail throughout this section, but it will be shining through.

4.1 Location of a Case Study for the Paint and Coatings Industry The running case study of the main section in this paper is a training application for accident prevention by means of one of the authors’ time travel prevention games applied in the paint and coatings industry. Related content may be found in ref. [11] (Fig. 5). In contrast to the introductory case study, training sessions do not always end in such a disaster like the one visualized by means of Fig. 2. Results may be more subtle and, consequently, the episode of data analysis (Fig. 3) gains greater importance. The more subtle the results, the finer distinguished are the human trainees’ individual strengths and weaknesses and the more involved is learner/trainee/player modeling. The history of game play is a key source for hypothesizing trainee characteristics. Notice that the system’s analysis of human–computer interaction data hardly results in definite insights. Modeling—especially theory of mind modeling—results in hypotheses only [6]. All models are wrong, but some are useful (ascribed to George E.P. Box). Unfortunately, this discussion goes beyond the limits of the present contribution.

Fig. 5 The factory and its receiving store by Fraunhofer IFF deployed by the German Institution for Statutory Accident Insurance and Prevention for Raw Materials and Chemical Industry

32

O. Arnold et al.

Fig. 6 The factory’s intermediate bearing and the basket mills workspace by Fraunhofer IFF

Figure 6 shows virtual locations where the real training is experienced. Instead of going through all the details of a full training session, the authors prefer to focus selected issues of adaptive trainee guidance that relate to deontic thinking. Some authors like Ruzsa [41] clearly explicate that variables of deontic logic play a role different from what we are used to in the conventional propositional calculus. To keep it short, think of variables for actions and for events. By way of illustration, imagine a variable [vap] that refers to vaporization and formulas such as □[vap], for example.

4.2 Deontic Thinking and Chatting Within a Training Session Exemplified When a trainee is embarking on a very first time travel and selects a destination from the time tunnel, the target scenery appears unchanged and the trainee may act freely. Repeated time travel is an indicator of the need for more adaptive guidance. Repeated time travel results in hypothetical entries in the theory of mind that accuse, so to speak, the trainee of some thoughts or beliefs. Alternatively, the absence of some theory of mind entry may be relevant, e.g., missing any knowledge about [vap]. Relying on the readers’ imagination, Fig. 7 illustrates three subsequent stages of adaptive guidance; consult ref. [11] for the concept of cascades introduced by the authors. On a first level of adaptivity, the trainee revisiting the hall of basket mills suddenly hears a voice from the off—visualized by a big speech bubble—that speaks about pollution (left screenshot in Fig. 7). On a second level, when the system according to the hypothesized theory of mind believes that the trainee has no clue, the

Thinking and Chatting Deontically—Novel Support of Communication …

33

Fig. 7 Revisiting the basket mills hall in the past that changes dynamically to guide the trainee

acoustic warning is followed by a visualization (screenshot in the middle). On a third level, in addition to the warning before, the system suggests a question (rightmost screenshot). Pondering this question shall trigger the emergence of certain ideas that might relate the pollution to vaporization. It is an issue of analytical thinking to identify reasons for significant vaporization. In case it appears necessary, the training system tries to get the trainee engaged in some dialogue. In intricate situations, it may be possible that the clarification of background problems—disciplinary questions for properties of chemicals, for resilience of materials, for the complexity of algorithms, and the like—leads beyond the limits of the training environment. Web resources and disciplinary chatbots may be invoked, including the necessity to communicate deontically. This exceeds the present paper. Instead, the authors are going to investigate simpler situations of current practice illustrated by means of Fig. 8. Chats are limited to the human trainee and the system. The cases on display in Fig. 8 belong to the currently most advanced approaches [11] of dynamically changing the past to impact the future that, in fact, is the present time where the time traveler comes from. If p is an event or action variable, p means that the event takes place or the action is executed, respectively. The formulas □p and p are modalities of p. By way of illustration, [buck] is the action variable of using buckets for transportation and [vap] is the event variable of vaporization taking place. In the left screenshot of Fig. 8, the avatar talks to the human trainee aiming at the awareness of a relationship that may be represented in a form like [buck] →

Fig. 8 Chatting on the basis of deontic knowledge and a hypothetical deontic theory of mind

34

O. Arnold et al.

[vap]. When this does not work—i.e., the trainee is using the buckets nonetheless— pollution will be too high once more, another time travel is needed, and the game system is undertaking a much stronger intervention represented by Fig. 8’s right screenshot. Buckets are removed—the past is no longer what it used to be. Abduction is essential to deontic reasoning [39]. The authors’ approach is a form of brute force abduction that assures the avoidance of undesirable events or actions.

5 Summary, Conclusions and Outlook On the one hand, the focus of the authors’ contribution is rather narrow on training of industrial accident prevention based on the single core concept of virtual time travel in time travel prevention games. On the other hand, the background is heterogeneous and interdisciplinary based upon behavioral sciences, storyboarding, and modal logic, to mention a few. This interdisciplinarity is operational—adaptivity is storyboarded, theory of mind modeling and induction is implemented, and a system’s utterances based on deontic reasoning are generated. The devil is in the details. Lewis developed a hierarchy of modal logics [38] of which he considered system S2 the right one [23]. The crux is that every author has preferred meanings of modal operators in mind such that certain axioms appear contradictory. This applies to deontic ideas, in particular, because those are frequently having a moral overtone [19]. The result are more or less lengthy debates that hardly resolve the problems [42]. It doesn’t matter whether or not an operator attached to an event/action variable p is read that “p is obligatory” or a trainee “ought to p.” In the process of game design, the interdisciplinary team of designers including domain experts and other specialists has to negotiate the cognitive states training is aiming at. Instead of introducing allusive operator names, the present authors advocate to stick with □ and as long as possible. Axioms, not notations, shall specify meaning. The only axiom required under all circumstances is “□p → p.” Some consider this a Kantian principle. It makes a logic deontical and, therefore, is named axiom D [23]. As said above, we do not adopt Gödel’s axiom “□p → p.” And we can neither accept nor reject “p → □p.” If the deontic meaning of □ comes close to the awareness of oughts, this axiom relates to the perceivableness of events. And this, in turn, relates to issues of interactive digital storytelling [30, 31]. It turns out that axioms we want characterize the systems we build and the applications we enable. Another aspect of meanings close to awareness is reflection expressed by axioms such as “□□p → □p” and “□ p → p.” Both are adopted. A further pragmatic aspect is that these two axioms, under the inference rule of detachment, reduce operators. This may simplify chatting deontically with trainees—from axiomatic to pragmatics. Further fundamental studies and discussions of variants of deontic axiomatization exceed the present contribution and seem worth a separate treatment.

Thinking and Chatting Deontically—Novel Support of Communication …

35

There are paramount challenges for extensions of the present work. Let us take as a launch pad, so to speak, an (in an engineering publication, at least) untypical source— Gregg’s recent book “If Nietzsche Were a Narwhal” [25]. From the viewpoint of Gregg’s research into animal behavior and cognition, he characterizes humankind as “the why specialists”—the only animals that always ask back. Recall the dialogue possibly initiated in a scenery like the rightmost one of Fig. 8. When we install, say, a button for the trainee to ask back, this may trigger a dialogue in which the game system is scrutinized to explain its own behavior—explainable AI. An explanation dialogue leads to background discussions as touched below Fig. 8 above, with modalities carried over. Acknowledgements The German Federal Ministry of Labor and Social Affairs (BMAS) has supported part of this work and publication. The BMAS—through its Civic Innovation Platform— awarded a prize to the authors’ proposal “AI on the Fly” that aims at the low-threshold dissemination of knowledge about human-centered AI concepts, technologies, and applications (German pitch of 3 minutes under the link https://www.youtube.com/watch?v=cfkjdteN3vw). The BMAS awarded another prize to the first and the third author’s proposal on explainable AI entitled “Hyper KI” (the German acronym for “Hypothesenbildende erklärbare Künstliche Intelligenz”) that aims at a wider understanding of the power and the limitations of AI. The BMAS’s support may initiate a widening of the horizon of the present work toward the AI explanation of oughts—possibly another novelty. Last but not least, an anonymous reviewer provided a pleasant review and directed the authors’ attention to the choice and syntactic representation of deontic operators. In addition to the earlier discussion in Sect. 3.2, the authors took up the challenge and extended Sect. 5 by a few extra paragraphs dedicated to the issue.

References 1. Alchourron, C.E.: Von Wright on deontic logic and the philosophy of law (1973/89). In: Bulygin, E., Bernal, C., Huerta, C., Mazzarese, T., Moreso, J. J., Navarro, P. E., Paulson, S.L. (eds.) Essays in Legal Philosophy, pp. 88–116. Oxford University Press, Oxford (2015) 2. Anderson, A.R.: The logic of norms. Logique et Anal. (N.S.) 2, 84–91 (1958) 3. Arnold, O.: Die Therapiesteuerungskomponente einer wissensbasierten Systemarchitektur für Aufgaben der Prozeßführung, DISKI, vol. 130. infix, St. Augustin (1996) 4. Arnold, O., Jantke, K.P.: Therapy plan generation in complex dynamic environments. ICSI report TR-94–054. International Computer Science Institute, Berkeley (1994) 5. Arnold, O., Jantke, K.P.: Therapy plans as hierarchically structured graphs. In: Fifth International Workshop on Graph Grammars and their Application to Computer Science, Williamsburg, VA, USA (1994) 6. Arnold, O., Jantke, K.P.: Mining HCI data for theory of mind induction In: Thomas, C. (ed.) Data Mining, chap. 4, pp. 47–68. IntechOpen (2018) 7. Arnold, O., Jantke, K.P.: AI planning for unique learning experiences: the time travel exploratory games approach. In: Csapo, B., Uhomoibhi, J. (eds.) 13th International Conference on Computer Supported Education, CSEDU 2021, vol. 1, pp. 124–132. SCITEPRESS (2021) 8. Arnold, O., Jantke, K.P.: The time travel exploratory games approach: an artificial intelligence perspective. In: Csapo, B., Uhomoibhi, J. (eds.) Computer Supported Education, 13th International Conference, CSEDU 2021, Virtual Event, 23–25 April 2021, Revised Selected Papers, CCIS, vol. 1624. pp. 40–54. Springer Nature (2021)

36

O. Arnold et al.

9. Arnold, O., Franke, R., Jantke, K.P., Wache, H.-H.: Dynamic plan generation and digital storyboarding for the professional training of accident prevention with time travel games. In: Guralnick, D., Auer, M.E., Poce, A. (eds.) Innovations in Learning and Technology for the Workplace and Higher Education: Proceedings of The Learning Ideas Conference 2021. LNNS, vol. 349, pp. 3–18. Springer Nature (2022) 10. Arnold, O., Franke, R., Jantke, K.P., Wache, H.-H.: Professional training for industrial accident prevention with time travel games. Int. J. Adv. Corp. Learn. 15(1), 20–34 (2022) 11. Arnold, O., Franke, R., Jantke, K.P., Wache, H.-H.: Cascades of concepts of virtual time travel games for the training of industrial accident prevention. In: Guralnick, D., Auer, M.E., Poce, A. (eds.) Innovative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education: Proceedings of the Learning Ideas Conference 2022. LNNS, vol. 581, pp. 53–64. Springer Nature (2023) 12. Arnold, S., Fujima, J., Jantke, K.P.: Storyboarding serious games for large-scale training applications. In: Foley, O., Restivo, M.T., Uhomoibhi, J., Helfert, M. (eds.) Proceedings of the 5th International Conference on Computer Supported Education, CSEDU 2013, Aachen, Germany, 6–8 May 2013, pp. 651–655 (2013) 13. Arnold, S., Fujima, J., Jantke, K.P., Karsten, A., Simeit, H.: Game-based training for executive staff of professional disaster management: Storyboarding adaptivity of game play. In: Tan, D. (ed.) Proceedings of the Intl. Conference on Advanced Information and Communication Technology for Education, ICAICTE 2013, Hainan, China, 20–22 September 2013, pp. 42–53. Atlantis Press (2013) 14. Becker, O.: Zur Logik der Modalitäten. Jahrbuch für Philosophie und Phänomenologische Forschung 11, 497–548 (1930) 15. Blackburn, P., de Rijke, M., Venema, Y.: Modal Logic. Cambridge Tracts in Theoretical Computer Science, vol. 55. Cambridge University Press, Cambridge (2001) 16. Call, J., Tomasello, M.: Does the chimpanzee have a theory of mind? 30 years later. Trends Cogn. Sci. 12(5), 187–192 (2008) 17. Carruthers, P., Smith, P. K.: Theories of Theories of Mind. Cambridge University Press, Cambridge (1996) 18. Castaneda, H.-N.: On the semantics of the ought-to-do. Synthese 21(3/4), 449–468 (1970) 19. Chisholm, R.M.: Contrary-to-duty imperatives and deontic logic. Analysis 24(2), 33–36 (1963) 20. Emery, N.J., Clayton, N.S.: Comparative social cognition. Annu. Rev. Psychol. 60, 87–113 (2009) 21. Gabbay, D.M., Maksimova, L.: Interpolation and Definability: Modal and Intuitionistic Logics. Oxford Logic Guides. Clarendo Press, Oxford (2006) 22. Gabbay, D.M., Horty, J., Parent, X., van der Meyden, R., van der Torre, L. (eds.): Handbook of Deontic Logic and Normative Systems. College Publishing Ltd., Rickmansworth (2013) 23. Garson, J.: Modal logic. In: Zalta, E.N., Nodelman, U. (eds.) The Stanford Encyclopedia of Philosophy (2021). https://plato.stanford.edu/archives/sum2021/entries/logicmodal/ 24. Goldblatt, R.: Mathematical modal logic: a view of its evolution. In: Gabbay, D.M., Woods, J. (eds.) Handbook of the History of Logic, vol. 7, pp. 1–98. Elsevier, Amsterdam (2006) 25. Gregg, J.: If Nietzsche Were a Narwhal: What Animal Intelligence Reveals About Human Stupidity. Little, Brown and Company (2022) 26. Gödel, K.: Eine Interpretation des intuitionistischen Aussagenkalküls. Ergebnisse eines mathematischen Kolloquiums 4(1931/32), 39–40 (1933) 27. Hansen, J., Pigozzi, G., van der Torre, L.: Ten philosophical problems in deontic logic. In: Boella, G., van der Torre, L., Verhagen, H. (eds.) Normative Multiagent Systems. Dagstuhl Seminar Proceedings, vol. 7122, pp. 1–26. Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2007) 28. Hilpinen, R. (ed.): Deontic Logic: Introductory and Systematic Readings. Reidel, Dordrecht (1971) 29. Hilpinen, R., McNamara, P.: Deontic logic: a historical survey and introduction. In: Gabbay, D.M., Horty, J., Parent, X., van der Meyden, R., van der Torre, L. (eds.) Handbook of Deontic Logic and Normative Systems, pp. 3–136. College Publishing Ltd., Rickmansworth (2013)

Thinking and Chatting Deontically—Novel Support of Communication …

37

30. Jantke, K.P.: Dramaturgical design of the narrative in digital games: AI planning of conflicts in non-linear spaces of time. In: IEEE Symposium on Computational Intelligence and Games, Sept. 7–10, 2009, Milano, Italy, pp. 88–95. IEEE Press (2009) 31. Jantke, K.P.: The evolution of story spaces of digital games beyond the limits of linearity and monotonicity. In: Iurgel, I., Zagalo, N., Petta, P. (eds.) Proceedings of the 2nd International Conference on Digital Storytelling, 9–11 December 2009, Erfurt, Germany. LNCS, vol. 5915. pp. 308–311. Springer, Heidelberg (2009) 32. Jantke, K.P.: Time Travel Prevention Games (2015). https://www.praeventionstag.de/nano.cms/ vortraege/begriff/Time-Travel-Prevention-Games. Accessed 10 May 2023 33. Jantke, K.P.: Theory of mind modeling and induction: Touch & feel. ADISY Technical report 01/2016, ADISY Consulting, Weimar, Germany (2016) 34. Jantke, K.P., Knauf, R.: Didactic design through storyboarding: standard concepts for standard tools. In: Proceedings of 4th International Symposium on Information and Communication Technologies, Cape Town, South Africa, 3–6 January 2005, pp. 20–25. Computer Science Press, Trinity College Dublin (2005) 35. Jantke, K.P., Schmidt, B., Schnappauf, R.: Next generation learner modeling by theory of mind model induction. In: Proceedings of 8th International Conference on Computer Supported Education, CSEDU, Rome, Italy, 21–23 April 2016, vol. 1. pp. 499–506. SCITEPRESS (2016) 36. Jantke, K.P., Wache, H.-H., Franke, R.: Time travel gamification of learning and training: from theoretical concepts to practical applications. In: Sobota, B. (ed.) Game Theory – From Idea to Practice, chap. 8, pp. 141–163. IntechOpen, Rijeka (2023) 37. Knauf, R., Sakurai, Y., Tsuruta, S., Jantke, K.P.: Modeling didactic knowledge by storyboarding. J. Educ. Comput. Res. 42(4), 355–383 (2010) 38. Lewis, C.I., Langford, C.H.: Symbolic Logic. Century Company (1932) 39. Magnani, L.: Abduction, Reason and Science: Processes of Discovery and Explanation. Springer, Heidelberg (2001) 40. Mally, E.: Grundgesetze des Sollens: Elemente der Logik des Willens. UniversitätsBuchhandlung, Graz, Leuschner und Lubensky (1926) 41. Ruzsa, I.: Axiomatischer Aufbau eines Systems der deontischen Logik. Acta Sci. Math. Szeged 26, 253–267 (1965) 42. Saint Croix, C., Thomason, R.H.: Chisholm’s paradox and conditional oughts. In: Cariani, F., Grossi, D., Meheus, J., Parent, X. (eds.) Proceedings of DEON 2014, Deontic Logic and Normative Systems. LNAI, vol. 8554, pp. 192–207. Springer, Cham (2014) 43. Schmidt, B.: Theory of mind player modeling: Konzeptentwicklung, Implementierung und Erprobung mit logischer Programmierung. Bachelor’s thesis, Erfurt University of Applied Sciences, Applied Computer Science (Angewandte Informatik) (2014) 44. Schmidt, B.: Theory of mind modeling and induction: Eine praktische Anwendung. Master’s thesis, Erfurt University of Applied Sciences, Applied Computer Science (Angewandte Informatik) (2017) 45. Tsuruta, S., Knauf, R., Dohi, S., Kawabe, T., Sakurai, Y.: An intelligent system for modeling and supporting academic educational processes. In: Penã Alaya, A. (ed.) Intelligent and Adaptive Educational-Learning Systems: Smart Innovation, Systems and Technologies. KES Springer Book Series, vol. 17, pp. 469–496 (2013) 46. van Benthem, J.: Modal Logic for Open Minds. (2010). CSLI Lecture Notes, vol. 199. CSLI Publications, Stanford (2010) 47. von Wright, G.H.: An Essay in Modal Logic. North Holland, Amsterdam (1951). How to use it for the sake of adaptivity

Corrective vs. Nurturing Feedback in Design Education: Alternative Models of Critique that Positively Impact Students’ Sense of Self-efficacy Zinka Bejtic

Abstract Design students’ attitudes, thoughts, and beliefs are essential to producing innovative, creative work. Self-efficacy, a construct of the cognitive theory that defines one’s belief in their capacity to reach a specific goal, is a critical variable that predicts students’ motivation and ability to perform a particular task successfully. This paper critically examines the quality of feedback in design education, arguing that it has a direct positive or negative impact on students’ sense of self-efficacy. Solely corrective feedback given in a poorly established learning milieu that does not allow a relationship of trust to flourish has a direct negative impact on students’ thoughts and their self-esteem. It is hypothesized that nurturing feedback has the potential to positively impact students’ self-efficacy and increase their overall effort and perseverance, prompting them to embrace a challenge as they face a difficult task. This allows students to take on a more active role as constructors of their knowledge. A case study was conducted in a senior design studio course with 16 students of the Multimedia Design program and one instructor in the Department of Art and Design at the American University of Sharjah (UAE). The paper reports on problems created by a lack of appropriate feedback and proposes strategies to transform feedback into a tool for nurturing students’ sense of self-efficacy. It further provides insight for all instructors who teach in educational environments where feedback plays an essential role. Keywords Creative pedagogy · Self-efficacy · Nurturing feedback

Z. Bejtic (B) American University of Sharjah, Sharjah, United Arab Emirates e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_4

39

40

Z. Bejtic

1 Introduction The notion of self-efficacy in cognitive theory was initially introduced in the 1970s. In recent years, the significance of self-efficacy as a determinant of academic achievement among students has gained increasing recognition. The extent of selfefficacy varies among students and is known to impact their ability to persevere through challenging tasks and their overall academic performance. This study aims to examine how educators in creative disciplines can enhance students’ self-efficacy through effective feedback. The findings indicate that modifying critique sessions and enhancing the quality of teacher-student interactions in the classroom can have a positive influence on self-efficacy, ultimately resulting in improved academic outcomes for students.

2 Common Framework and Definitions 2.1 The Studio Classroom The studio classroom is a prevalent learning environment within creative disciplines. In a typical studio class, there are no more than sixteen students, allowing the instructor to provide personalized attention and feedback to each student [1]. The pedagogy of creative disciplines is based on a student-centered teaching approach that excludes traditional lecture-based content delivery. Instead, it prioritizes handson demonstrations, process-oriented work, analysis-synthesis organization, articulation of ideas, and individual feedback given to students. The studio education is rooted in a “hidden curriculum” that acknowledges the importance of both the experience and social context, in addition to the actual content being taught [2]. Given that studio education is experiential and inseparable from its instructional setting, studio instructors play a crucial role in creating effective learning experiences through content delivery, management techniques, and most importantly, through individualized critique sessions that students receive throughout their education [3].

2.2 The Construct of Academic Self-efficacy In the academic literature, intrinsic motivation and academic self-efficacy have been identified as critical predictors of student achievement [4]. This suggests that successful students are not solely driven by extrinsic motivators such as grades or recognition but, instead, possess a natural interest in learning stemming from their passion for the discipline. Such passion can be fostered by caring instructors who provide a supportive learning environment. Students’ beliefs about academic tasks and their confidence in their ability to perform them are derived from the following

Corrective vs. Nurturing Feedback in Design Education: Alternative …

41

four sources (a) positive mastery experiences related to performing the task, (b) vicarious learning (observation of peers who are successful in performing the task), (c) social persuasion (encouragement students receive from teachers, parents or peers), and (d) emotional and physiological states (emotional reactions to academic tasks can predict an expected success or failure while experiencing anxiety or stress can have a negative impact on academic self-efficacy. This means that educators, parents, and other students can play a key role in promoting the development of self-efficacy in others. Within the academic setting it’s the social persuasion aspect of self-efficacy that has the most impact on students’ sense of self-efficacy [5]. The importance of recognizing the impact of studio instructors on the learning experience in the studio classroom cannot be overstated. Failure to do so can result in a range of problems, such as miscommunication, missed opportunities, finger-pointing, unfulfilled expectations, poor quality of creative output, and demoralized students with low self-efficacy. However, it is unfortunate that many studio instructors lack formal training in pedagogy, which includes critiquing work and providing feedback. Educational institutions often prioritize hiring instructors based on their research and scholarship achievements, rather than their teaching abilities, which is a significant issue as it suggests that institutions are not keeping pace with contemporary demands and practices [6]. Several factors shape the relationship between studio instructors and their students, including the instructors’ personalities, teaching styles, levels of emotional intelligence, empathy, and cultural backgrounds, which can lead to a relaxed or tense environment [7]. When students evaluate a course, they primarily consider the instructor and determine whether the class is interesting or dull [8]. Thus, the pedagogy of creative disciplines encompasses not only teaching, learning, curriculum, and assessment but also the quality of relationships between instructors and students and the type of feedback students receive. Providing corrective feedback solely based on opinions (using “I” statements, value judgments, or advice) has a direct negative impact on students’ self-efficacy. However, providing nurturing feedback helps to increase students’ self-efficacy, enhance their creative performance, and develop their critical thinking abilities. This study aims to demonstrate the importance of nurturing feedback and explore how it can personalize each student’s learning experience in meaningful ways. This perspective does not negate the necessity of corrective feedback, but rather it argues that it must be complemented with nurturing feedback to create a safe and supportive environment for students. Self-efficacy, a critical component in human motivation, behavior, and achievement, is defined as the belief an individual has in their ability to successfully perform a particular task or achieve a specific goal. With a robust sense of self-efficacy, individuals are more likely to set challenging goals, persist through difficulties, and ultimately attain success. It is important to distinguish self-efficacy from self-esteem, which pertains to an individual’s global self-appraisal of their worth and competence. While self-esteem is a broad, generalized construct, self-efficacy is context-specific, applied to a particular task or situation. “Students may experience higher self-efficacy when they are told they are capable of learning by a trustworthy source, whereas they may discount the advice of less

42

Z. Bejtic

credible sources” [9]. Past experiences, social support, and personal characteristics also influence this belief. Those with high self-efficacy are more likely to set challenging goals, persevere through obstacles, and take proactive steps to achieve their desired outcomes. Contrary to that, people with low self-efficacy tend to become demotivated, avoid challenges, and often struggle to perform at their best. It is essential to differentiate between generalized self-efficacy and academic selfefficacy. Generally, no relationship has been found between generalized self-efficacy and academic success. In contrast, research on academic self-efficacy has “consistently shown to predict grades and persistence in college” [10]. Students learn through observation, imitation, and modeling. The interaction between the individual, the environment, and the behavior shapes that behavior [11]. Bandura emphasizes the value of self-efficacy in various aspects of human functioning, including cognition, emotion, motivation, and behavior. He argues that self-efficacy beliefs are critical in determining how people approach and respond to challenging situations. “Successes build a robust belief in one’s personal efficacy. Failures undermine it, especially if failures occur before a sense of efficacy is firmly established” [12]. Students who feel capable and confident generally experience positive emotions such as joy, happiness, and satisfaction. In contrast, students with low self-efficacy tend to feel anxious, stressed, and overwhelmed. These negative emotions often interfere with their academic performance, relationships, and overall quality of life. Numerous empirical studies published over the last two decades found that of several psychosocial constructs, academic self-efficacy was the most prominent predictor of students’ academic achievement and performance [13]. A high sense of self-efficacy fosters a growth mindset—a belief that one’s abilities and intelligence can be developed through dedication and hard work. Students with a growth mindset readily embrace challenges, view mistakes as opportunities for learning and develop resilience in the face of setbacks. This mindset is especially beneficial in design disciplines, where experimentation, risk-taking, and failure are integral to creative discovery. Measuring self-efficacy can be challenging, as it is a subjective construct that may vary across tasks and situations. However, various scales and measures have been developed to assess self-efficacy in different domains, such as academic achievement, sports performance, and job-related tasks. These measures typically ask individuals to rate their confidence in their ability to perform specific tasks or overcome specific obstacles [14].

2.3 Role of Feedback in Design Education As most of the learning in design studios happens during critique sessions, it is clear why feedback is the most critical aspect of design education and why it directly impacts students’ motivation, engagement, and their self-efficacy. The type of feedback, the tone, and the context in which it is provided have an impact on its effect on

Corrective vs. Nurturing Feedback in Design Education: Alternative …

43

students’ self-efficacy [15]. Actionable, precise, and nurturing feedback acknowledging one’s efforts enhances self-efficacy, allowing the student to set more challenging goals, persist through difficulties, and take on new tasks. In contrast, negative feedback, or criticism lower self-efficacy, create doubt in one’s abilities and cause students to feel demotivated and discouraged, negatively impacting their performance. It is important for design education institutions to focus not only on the development of student’s skills and competencies but also on building their selfefficacy. In the following study, we address the potency of social persuasion as a source of self-efficacy established by social cognitive theory and the answer to the research question: does the feedback provided by the instructor within studio disciplines have impact the self-efficacy of students? Desk critiques, often referred to as “desk crits,” are a fundamental aspect of design education. As described by Schön [16], design critiques or feedback sessions play a central role in the learning process. The studio instructor draws upon their existing repertoire of knowledge, experience, and examples to provide comments. Feedback is an essential component of the instructor-student relationship in a studio environment. Carol Dweck’s [17] research has shown that providing the appropriate type of corrective feedback, which acknowledges effort rather than achievement and encourages challenging tasks, is a vital strategy for increasing students’ sense of self-efficacy. During desk crits, the instructor spends time with a group of students or individually with each student, typically reviewing their work at their desks and providing feedback. For this reason, studio classes have longer contact hours to allow the instructor adequate time to carefully review each student’s work. This process enables the instructor to follow up on each student’s progress and, through process-oriented formative feedback, address issues and allow students to ask questions in an informal setting. It is through desk critiques that a personal relationship is formed between the instructor and the student. Group critiques or reviews are performed during a specific milestone. They consist of a small group of students pinning their work on the board and publicly discussing it with the instructor. During this process, students discuss design problems and share opinions, views, and ideas. This experience strengthens the relationship among students, while the instructor can steer the discussions in a way, he or she finds beneficial for students in terms of providing successful learning experiences for a particular given design problem. This process, furthermore, builds students’ self-confidence as they learn to articulate their ideas, express opinions, and engage in public speaking. In the design disciplines, feedback is given throughout the various stages of the design process, starting with the research and analysis, developing a design problem, generating ideas, making design choices, iterative stage, building communication and presentation skills, writing compelling design proposals, and applying critical thinking. Design education involves the development of creative and critical thinking skills, as well as technical proficiency in various design software and tools. Effective feedback is critical in helping design students enhance their work and develop their skills. It provides learners with information about their progress, helps them identify areas

44

Z. Bejtic

for improvement, and directly impacts learners’ sense of self-efficacy [18]. However, not all types of feedback are equal concerning their impact on students’ self-efficacy. Critical Corrective Feedback. Corrective feedback is concerned with identifying and rectifying errors in students’ work. Although it can be advantageous in enhancing students’ performance, it may also be discouraging if perceived as excessively critical or punitive. Corrective feedback, when accompanied by supportive guidance on how to improve and framed in a constructive manner, can promote self-efficacy and a growth mindset. Additionally, it can assist students in seeing their mistakes as opportunities for learning and growth. Constructive Feedback. Constructive feedback concentrates on recognizing areas that need improvement and providing precise directions for making progress. Typically, this kind of feedback is presented positively, emphasizing what was accomplished satisfactorily and providing detailed recommendations for enhancement. Constructive feedback fosters a growth mindset, which allows students to view mistakes as opportunities for learning and growth, rather than setbacks. When provided appropriately, constructive feedback has a positive influence on selfefficacy. Moreover, it can be empowering as it enables students to take ownership of their learning and progress. Nurturing Feedback. Nurturing Feedback focuses on encouraging and reinforcing positive approaches and behaviors, attitudes, and mindsets in students. It involves providing meaningful support and recognition of efforts to allow students to gradually build confidence in their skills, and motivation to engage with challenging tasks. Constructive criticism coupled with nurturing feedback highlights the areas for improvement while simultaneously acknowledging students’ strengths and their effort. It’s a powerful tool for building positive relationships, enhancing students’ performance and creating a supportive and empowering work environment. Positive Feedback. Positive feedback serves to recognize and endorse the strengths and achievements of students. By doing so, it has the potential to cultivate a sense of worth and assurance in their competencies, thereby enhancing their motivation and involvement. Nonetheless, it is imperative to ensure that such feedback is both authentic and precise, as opposed to being generic or disingenuous. Negative Feedback. Negative feedback entails identifying inadequacies or errors in the work of students. While it can facilitate the identification of areas that require enhancement, it may also have a demotivating effect if perceived as excessively critical or unaccompanied by constructive guidance on how to improve. This can lead to a reduction in students’ self-efficacy. Additionally, if the negative feedback fails to acknowledge or appreciate students’ efforts, it may further contribute to their demotivation. Adaptive Feedback. Adaptive feedback is characterized by its customization and alignment with the learners’ competencies and requirements, which renders it highly efficacious in fostering self-efficacy. By providing a personalized outlook, adaptive

Corrective vs. Nurturing Feedback in Design Education: Alternative …

45

feedback facilitates the perception of progress and development over time, thereby reinforcing a growth-oriented mindset. In essence, feedback that is characterized by specificity, personalization, and a positive tone is likely to exert the most substantial influence on learners’ self-efficacy and achievements. Technical Feedback. Technical feedback pertains to the technical facets of design, encompassing craftsmanship, materials, and techniques. Positive technical feedback has the potential to enhance design students’ confidence in their technical acumen, thereby augmenting their self-efficacy in executing design tasks with proficiency. Conversely, negative or insufficient technical feedback may have an adverse impact on students’ self-efficacy, given that it may imply inadequacies or limitations in their technical competencies. Conceptual Feedback. Conceptual feedback directs attention to the conceptual dimensions of design, encompassing the clarity, coherence, aesthetics, and overall design vision. Affirmative conceptual feedback that authenticates and reinforces students’ creative notions and concepts can substantially bolster their self-efficacy by attesting to their capacity to generate inventive and captivating design concepts. In contrast, critical or dismissive conceptual feedback may impede students’ self-efficacy, instilling doubt in their ability to develop robust design concepts. Process Feedback. Process feedback pertains to the design process, encompassing the aptitude to efficiently plan, organize, and manage the various phases of the design process. Insufficient or erratic process feedback may result in bewilderment and selfquestioning, potentially leading to a decline in students’ self-efficacy. Conversely, constructive process feedback that steers students in enhancing their design process can augment their self-efficacy by enabling them to formulate effective approaches for tackling design tasks. Peer Feedback. Peer feedback encompasses feedback from fellow students or peers in a design program. Affirmative and constructive peer feedback can considerably augment design students’ self-efficacy, given that it offers diverse perspectives and insights, authenticates their design choices, and nurtures a sense of community and collaboration. Nonetheless, peer feedback that is negative or unhelpful, characterized by excessive criticism or an absence of constructive suggestions, may have an adverse effect on students’ self-efficacy, leading to self-doubt and reduced confidence in their design competencies. Self-reflection Feedback. Self-reflection feedback that promotes self-awareness, critical thinking, and self-improvement has the potential to enhance self-efficacy. Self-reflection feedback requires students to introspectively scrutinize their design work and provide self-assessment. It enhances students’ capacity to evaluate their work and identify their strengths and areas of improvement. Nevertheless, selfreflection feedback that is excessively self-critical or biased may have an adverse effect on students’ self-efficacy, especially if they overemphasize their perceived weaknesses or overlook their strengths.

46

Z. Bejtic

Overall, the impact of different types of feedback on design students’ sense of selfefficacy can depend on the nature, quality, and timing of the feedback. In addition, students’ perceptions of feedback are directly related to their levels of self-regulation, self-efficacy, and academic achievement. Specifically, students who view feedback as primarily evaluative or critical have low levels of self-regulation and self-efficacy, as well as lower academic achievement. In contrast, students who view feedback as primarily informative or helpful have higher levels of self-regulation, self-efficacy, and academic achievement. The way the feedback is perceived by a student has a significant impact on their academic outcomes. It highlights the importance of providing feedback that is informative and supportive, rather than solely evaluative or critical [19]. While corrective feedback can be beneficial in improving performance, it should be provided in a supportive and constructive manner to avoid undermining students’ confidence. Constructive, positive, and adaptive feedback can be particularly effective in enhancing self-efficacy, promoting a growth mindset, and motivating students to persist in their learning. Finally, feedback that is specific, personalized, and framed positively is likely to have the most significant impact on students’ self-efficacy and success.

3 Methodology This study aims to examine the impact of critical vs. nurturing feedback on design students’ sense of self-efficacy. Sixteen students attending a fourth year of the Bachelor of Science Multimedia Design undergraduate program at the American University of Sharjah participated in the study.

3.1 Pre-screening Instrument A pre-screening tool for assessing academic self-efficacy, comprising eight questions, was utilized to evaluate the self-efficacy levels of thirty students before the commencement of the study. The tool was intended to select only those students who scored high in self-efficacy. Upon completion of the survey, the collected data were numerically organized, and the high self-efficacy belief average score section was scrutinized to select sixteen students for participation in the study.

3.2 Data Collection and Simulation This study sought to discover further information about the impact of corrective and nurturing feedback on students’ self-efficacy. The study was conducted through

Corrective vs. Nurturing Feedback in Design Education: Alternative …

47

phenomenological, explorative qualitative focus group interviews that applied an inductive approach in data analysis to understand participants’ perspectives, views and practices identified through codes and themes. Data were collected during Fall semester of 2022, through a focus group interview with 16 senior students of the Multimedia Design program at the American University of Sharjah who scored high for self-efficacy. The qualitative analysis was transacted through systematic text condensation, during which the categories were derived from data. During the Fall 2022 semester, a group of sixteen pre-screened students identified as having high self-efficacy were presented with a creative project brief. These students were subsequently divided into two groups: Group A consisting of eight students, who received critical corrective and negative feedback, and Group B, also comprising eight students, who received constructive and nurturing feedback, as defined in this paper. The instructor who executed the simulation was unfamiliar to the students and was only introduced for the purpose of the study. The project was introduced only for the purpose of this study, was not a part of the course, and was not assessed. During the four weeks, each student met with the instructor once a week for an individual critique session. Each student in the group had four critique sessions with the same instructor and received feedback for their work. Subsequently, the researcher interviewed all sixteen students using a phenomenology protocol. Data were generated through individual video recordings of in-depth qualitative interviews. The general techniques employed were aimed at being reproducible, systematic, credible, and transparent. Interview questions for participants in both Group A and Group B were as follows: 1. 2. 3. 4. 5. 6. 7. 8. 9.

Did you find the feedback sessions helpful in developing your work? How did you feel during and after the feedback session? Was the feedback easy to understand and apply? Did the instructor make you believe that you can improve your project? Did you feel motivated to make further attempts and iterations? Did you feel relaxed and supported? Did you feel encouraged to ask questions and engage in dialogue? Did the feedback you received affect your perceived confidence level? Any additional comments?

The interviews lasted 45–60 min and were audio-recorded. A semi-structured interview guide was used. The interviews have been transcribed, and the transcripts were read several times to get a better understanding of the findings, following which, the preliminary themes were identified and meaning units connected to the themes as examples of specific experiences. The emergent structure among the coded material was noted and was then classified into an attitude or experiences that were condensed and coded while preserving the essence. Bracketing and epoché processes were applied to prevent and block biases and allow the phenomenon to be explained in its own classification of meaning.

48

Z. Bejtic

3.3 Data Analysis The qualitative research data were analyzed using inductive approach and following the systematic steps. The initial step includes transcription of the interviews, followed by the familiarization of the researcher with the data and identification of initial patterns or themes and then coding appropriately through both descriptive and interpretive approaches to allow for new themes to emerge during the analysis; an inductive approach is utilized in this step. The second step is categorizing identified codes not assessed based on any given categories that capture the broader themes or concepts that are emerging from the data. An assessment of the reliability of the codes follows this process. The categories were examined for patterns or relationships between them. Similar categories were combined, and categories that are not relevant are eliminated to reduce the data to a more manageable size. Finally, the data were interpreted by developing a narrative that explains the relationships between the categories. This narrative is supported by quotes or examples from the data to illustrate the key themes and concepts that are emerging. The entire process was subjected to an evaluation of the interview participants to increase the validity of the findings. The researcher also kept a reflective journal with the comments from participating students as they relate to the transcribed interviews. The qualitative synthesis stage in the data analysis process in qualitative research represents the most debated segment since it depends on the researcher’s judgment [20]. The researcher’s bias has been bracketed to identify previous notions about the research subject. This process was followed by an analysis of the transcripts to identify themes that emerged through applying tags, coding, and developing categories. All the critical statements were clustered to avoid redundancies and then categorized into meaningful categories. Quirkos software for qualitative analysis was used to code and explore data once it had been coded. The qualitative analysis relied on recording thoughts and reflecting on the procedures through journaling, memos, and summaries. The software enabled the researcher to visually present the information, obtain themes, and visualize the findings. The researcher was able to obtain reports that are customizable and that graphically represent the text data.

3.4 Ethical Considerations Students who agreed to participate in the study were given oral and written information about the project and were guaranteed anonymity. They were informed that participation was voluntary and that they could withdraw participation at any time without negative consequences. Written informed consent was obtained.

Corrective vs. Nurturing Feedback in Design Education: Alternative …

49

4 Findings 4.1 Impact of Critical Corrective and Negative Feedback The data analysis revealed four main themes that emerged from the group A participants who received critical corrective feedback: (1) increased emotional distress, (2) Reduced self-esteem, (3) lowered intrinsic motivation, and (4) impaired future performance. Critical corrective feedback has been found to have a direct negative impact on students’ self-efficacy through: Increased Emotional Distress. Significant proportion of the participants reported experiencing heightened emotional distress subsequent to the feedback session. Specifically, most of the respondents expressed feelings of excessive worry, shame, and often guilt. The predominant emotions cited by the participants were anxiety, frustration, and shame, which they experienced in relation to their own work. While the participants acknowledged the relative value of the feedback they received, a notable subset reported feeling discouraged and doubtful of their capacity to improve. The participants attributed this sense of inadequacy to feelings of embarrassment and humiliation about themselves. Reduced Self-esteem. The present study highlights the diverse ways in which critical corrective feedback influenced the participants’ self-esteem. Specifically, a considerable number of respondents revealed experiencing apprehension towards being evaluated or critiqued, resulting in negative self-talk that further reinforced their sense of inadequacy. This psychological reaction resulted in a tendency to withdraw from class discussions and activities, leading to feelings of isolation and disconnection from the group. Additionally, some participants reported imposing unattainable standards for themselves, leading to a hypercritical approach towards their work, thereby exacerbating feelings of failure and inadequacy. Lowered Intrinsic Motivation. The present study has identified that critical corrective feedback has a direct impact on the motivation levels of students. Specifically, a considerable number of participants reported a diminished interest in their academic work, leading to procrastination and a reduced effort towards completing assignments. In addition, several respondents expressed a heightened focus on external rewards, such as instructor praise and grades, as opposed to intrinsic motivation. Furthermore, some participants noted a lack of perceived autonomy and ownership over their work, resulting in a diminished inclination to engage creatively with their assignments. Impaired Academic Performance. The findings of the present study indicate that most participants observed that when feedback was excessively critical or perceived as unfair, it resulted in resistance to receiving future feedback, which impeded their ability to learn and improve. Moreover, the respondents expressed a sense of resentment towards the instructor, which influenced their reception of information and

50

Z. Bejtic

learning. This negative attitude towards the course material was not limited to the specific feedback session but extended to the students’ overall perception of the educational experience, resulting in incomplete assignments, increased absences, and an overall decline in academic performance.

4.2 Impact of Constructive and Nurturing Feedback The following themes emerged from the data collected from the students in group B who have received nurturing feedback: (1) increased self-confidence and creativity, (2) increased intrinsic motivation, (3) adopting a growth mindset, (4) positive learning environment and relationships, (5) improved project outcomes, and (6) enhanced physical and emotional wellbeing. Specifically, nurturing feedback has had a direct positive impact on students’ self-efficacy through the following areas. Increased Self-confidence and Creativity. The present study reveals that a notable proportion of the participants expressed a willingness to take greater risks in their design work after the feedback session. The feedback served as a source of encouragement, leading the respondents to experiment with new techniques, explore unconventional ideas and push the boundaries of their work. Additionally, some participants noted an increased interest in presenting their work and engaging in discussions with their instructor, peers, or outside reviewers. This newfound confidence enabled them to articulate their ideas more effectively and incorporate feedback constructively into their designs. Furthermore, several participants expressed a sense of satisfaction that their work was innovative and enjoyed challenging themselves creatively. Increased Intrinsic Motivation. Participants, overall, have experienced an increase in their level of enthusiasm towards their work. The respondents reported a heightened sense of curiosity and excitement during the development of their concepts, which, in turn, motivated them to invest the requisite effort to produce high-quality work. Additionally, some participants felt encouraged by the perceived autonomy and the opportunity to self-direct their projects, which further enhanced their eagerness to seek additional resources to deepen their knowledge and skill. Finally, all participants expressed a sense of engagement and passion towards their work, highlighting the positive impact that constructive feedback can have on students’ overall motivation levels. Expanded Growth Mindset. The present study reveals that a considerable number of participants experienced a positive impact on their motivation levels subsequent to receiving nurturing feedback. The respondents reported feeling motivated to embrace challenges, viewing them as opportunities for growth and learning. Additionally, participants demonstrated a proactive approach to seeking feedback on their work from both their instructor and peers. Some students reported that the criticism offered during the feedback session served as a valuable learning experience, providing

Corrective vs. Nurturing Feedback in Design Education: Alternative …

51

them with useful insights to improve their work. Finally, all participants expressed a willingness to invest more effort in their work as they worked towards achieving their goals. These findings underscore the significant role that constructive feedback can play in facilitating students’ motivation and growth. Positive Learning Environment and Stronger Relationships. The present study highlights the positive impact of nurturing feedback provided by the studio instructor on creating a supportive learning environment and fostering strong relationships among participants. The respondents reported that the entire studio space evolved into a collaborative workspace, promoting a sense of community that enabled them to engage in conversations, share their ideas and collaborate on projects. Many participants noted feeling a strong connection with the instructor, developing a personal rapport that allowed them to share their interests and passions outside of the classroom. The instructor was perceived as approachable, supportive, and invested in students’ success, significantly contributing to the positive atmosphere. The respondents expressed comfort in approaching the instructor with questions or concerns, feeling supported and engaged in their learning. This, in turn, led to increased studentled initiatives, promoting a sense of ownership and pride among design students. These findings underscore the significant role that nurturing feedback and supportive instructor-student relationships play in creating an optimal learning environment. Improved Project Outcomes. The provision of nurturing feedback has been observed to yield positive academic outcomes. Feedback that emphasizes students’ strengths and progress creates a conducive environment for them to continue building on those areas and improve overall. In addition, participants reported increased clarity in defining project goals and objectives, leading to a heightened sense of focus and motivation throughout the project development process. Nurturing feedback was also noted to have a positive effect on participants’ ability to stay on track and adjust as required. Students were further encouraged to develop iterations and prototypes, enabling them to test and refine their designs, identify weaknesses, and ultimately improve project outcomes. Enhanced Physical and Emotional Wellbeing. The provision of nurturing feedback establishes the foundation for the overall studio experience and places an emphasis on the emotional and physical wellbeing of students. As reported by participants, a sense of ease was prevalent within the studio space, with students feeling supported and connected to the community. The opportunity for open communication and peer support was regularly provided, leading to a sense of comfort around the instructor and within the environment. The instructor was perceived as an individual who sincerely cared about the welfare of the students, as evidenced by the provision of resources and support and the cultivation of a growth-oriented mindset.

52

Z. Bejtic

5 Conclusion This study aimed to investigate the relationship between feedback type and its impact on design students’ academic self-efficacy. The study’s findings showed a direct correlation between feedback and self-efficacy. Specifically, critical corrective feedback, when provided alone, was found to decrease students’ academic self-efficacy. In contrast, nurturing feedback delivered in conjunction with corrective feedback in a positive tone was found to increase academic self-efficacy. The study highlights the significant role of instructors and the learning environment in shaping design students’ self-efficacy. According to Bandura’s social cognitive theory [21], instructors are agents of change that intentionally influence their students’ functioning and life circumstances. The triadic relationship among personal, behavioral, and environmental determinants affects each other to determine behavior [21]. The belief system and skills are directly linked to behavior, and positive experiences in class increase self-efficacy, while stress and anxiety decrease it [22]. The findings of this study suggest that instructors can effectively promote students’ self-efficacy by integrating nurturing feedback into their pedagogical practices. The nurturing feedback outlined in this study can serve as a starting point for instructors and applies to any studio-based learning environment. In the realm of teaching and learning, the goal is to achieve transformational growth. In design studios, critiques are commonly employed to provide constructive feedback and identify areas for improvement in students’ work. While nurturing feedback does not neglect this critical aspect, it strongly emphasizes the student’s transformational experience, inspiring them to explore their potential and facilitating discovery. This approach suggests that feedback should be centered on the whole student, including their perspectives and experiences, rather than solely on a narrow aspect of their creative work. Instructors who effectively promote students’ selfefficacy take into account their capabilities and worldviews as they define success for each student and help them develop the skills necessary to navigate the world as it is while enabling them to make it what they want it to be [23]. Investigating the direct impact of feedback on the development and execution of creative projects and the relationship between self-efficacy, creativity, and experimentation is crucial in comprehending students’ motivation in creative disciplines. Self-efficacy, though not equating to talent, is integral in cultivating creative abilities. Favorable perceptions of creative capacity can inspire students to persevere through the demanding process of building creative skills. Many studies reveal that self-efficacy is a psychological factor that correlates positively with design performance [24–27]. Designers with confidence appear to invest more effort, take more risks, and set more ambitious goals for themselves [28], all contributing to their success. Since self-efficacy is a notable motivational variable in creative education, further investigation is necessary to explore how educators can enhance it. For now, the findings of this study demonstrate that nurturing feedback in studio classrooms contributes directly to increased self-efficacy.

Corrective vs. Nurturing Feedback in Design Education: Alternative …

53

Educational institutions that teach design disciplines have the potential to actively foster self-efficacy in their students by providing training opportunities for their instructors. These opportunities enable instructors to develop their skills in building and maintaining nurturing studio environments and positively impact their students’ self-efficacy through regular self-reflection and peer assessment. Ultimately, transformative teaching calls for adopting more inclusive and humane educational approaches that intentionally shift from purely critical to inspirational practices across all disciplines. As we collectively strive to overcome conditions that constrain our personal freedom, we must recognize the value of transformative pedagogy and its nurturing approaches in cultivating creativity and curiosity among new generations of learners.

6 Implications and Future Directions The current study details an educational intervention aimed at investigating effective methods for enhancing students’ self-efficacy within creative studio disciplines. The findings of the study demonstrate a positive correlation between the provision of constructive and nurturing feedback and increased self-efficacy among students. These promising results suggest that this intervention may have broad applicability across diverse design studio settings. Furthermore, the potential impact of this intervention may be strengthened through additional research employing larger sample sizes and more extended timeframes to further validate its effectiveness.

References 1. Ochsner, J.K.: Behind the mask: a psychoanalytic perspective on interaction in the design studio. J. Archit. Educ. 53(4), 194–206 (2000) 2. Dutton, T.A.: Voices in Architectural Education: Cultural Politics and Pedagogy. Bergin & Garvey, New York (1991) 3. Kolb, D.A.: Experiential Learning: Experience as the Source of Learning and Development. Prentice-Hall, Hoboken (1984) 4. Deci, E.L., Ryan, R.M. (eds.): The Handbook of Self-Determination Research. University of Rochester Press, New York (2006) 5. Bergen, A.: Self-efficacy, special education. InSight: Rivier Acad. J. 9(2) (2013). https://www. rivier.edu/journal/ROAJ-Fall-2013/J783-Bergen.pdf. Accessed 19 Feb 2023 6. Biggs, J.: Teaching for Quality Learning at University – What the Student Does, 2nd edn. Open University Press, Milton Keynes (2003) 7. Gregory, M.V.: Teaching Excellence in Higher Education. Palgrave Macmillan, New York (2013) 8. Small, R.V.: Dimensions of interest and boredom in instructional situations. Paper presented at the National Convention of the Association for Educational Communication and Technology, Indianapolis, IN (1996) 9. Schunk, D.H.: Self-efficacy and academic motivation. Educ. Psychol. 26, 207–231 (1991)

54

Z. Bejtic

10. Zajacova, A., Lynch, S.M., Espenshade, T.J.: Self-efficacy and stress. Res. High. Educ. 46(6) (2005). http://www.princeton.edu/~tje/files/Self%20Efficacy%20and%20Stress%20Z ajacova%20Lynch%20Espenshade%20Sept%202005.pdf 11. Bandura, A.: Self-efficacy: toward a unifying theory of behavioral change. Psychol. Rev. 84, 191–215 (1977) 12. Bandura, A.: Self-Efficacy. Academic Press, San Diego (1994) 13. Artino, A.R.: Academic self-efficacy: from educational theory to instructional practice. Perspect. Med. Educ. 1(2), 76–85 (2012). http://www.ncbi.nlm.nih.gov/pmc/articles/PMC354 0350/ 14. Heslin, P.A., Klehe, U.C.: Self-Efficacy. Encyclopedia of Industrial/Organizational Psychology, edited by SG Rogelberg, vol. 2, pp. 705–708 (2006) 15. Gielen, S., et al.: Improving the effectiveness of peer feedback for learning. Learn. Instr. 20(4), 265–348 (2010) 16. Schön, D.A.: The Design Studio. RIBA Publications, London (1985) 17. Dweck, C.: Motivational processes affecting learning. Am. Psychol. 41(10), 1040–1048 (1986) 18. Ruegg, R.: The effect of peer and teacher feedback on changes in EFL students’ writing self-efficacy. Lang. Learn. J. 46(2), 87–102 (2018) 19. Brown, G.T.L., Peterson, E., Yao, E.S.: Student conceptions of feedback: impact on selfregulation, self-efficacy, and academic achievement. Br. J. Educ. Psychol. 86(4), 606–629 (2016) 20. Boyatzis, R.E.: Transforming Qualitative Information. Sage, Thousand Oaks (1998) 21. Bandura, A.: Adolescent development from an agentic perspective. In: Pajares, F., Urdan, T. (eds.) Self-Efficacy Beliefs of Adolescents, pp. 1–43. Information Age Publishing, Charlotte (2006) 22. Bandura, A.: Social cognitive theory: an agentic perspective. Annu. Rev. Psychol. 52, 1–26 (2001) 23. Hibbs, D.F.: An investigation of the self-efficacy beliefs of black and hispanic students that have experienced success or failure in mathematics. Ph.D. thesis, Seton Hall University, South Orange (2012). http://scholarship.shu.edu/cgi/viewcontent.cgi?article=2843&context= dissertations. Accessed 10 Mar 2023 24. White, E.: Elizabeth White on Teaching for the ‘Whole Student’. Eye on Design. AIGA (2019). https://eyeondesign.aiga.org/higher-education-is-not-a-level-playing-fieldeliz abeth-white-on-teaching-for-the-whole-student/. Accessed Mar 2023 25. Metin-Orta, I., Kiygi-Calli, M.: The role of self-efficacy in the development of design thinking skills. Think. Skills Creat. 25, 98–109 (2017) 26. Jung, E.C., Jung-Ja, K.: The relationship between self-efficacy and creative performance in industrial design. Int. J. Ind. Ergon. 44, 533–538 (2014) 27. Zhang, R., Xiaojuan, M.: Self-efficacy and design performance: an exploratory study. Int. J. Des. Creat. Innov. 7(4), 245–258 (2019) 28. Kim, S., Wood, K.: Self-efficacy in design education: exploring the relationship between selfefficacy, design performance, and learning outcomes. Des. Stud. 49, 1–17 (2017)

Improving Sonographer-Patient Communication in a Diverse and Multicultural Environment Through Role-Plays with Digital Humans Barbara Bertagni, Linda Zanin, Fernando Salvetti, and Ianna Contardo

Abstract Effective communication skills are essential for sonographers to build trust, to explain examination procedures to the patient in non-technical terms, to alleviate anxiety and gain patient consent and collaboration, and to provide information at a pace suitable for the patient. In order to communicate effectively, the sonographer needs to be able to communicate empathetically, adjusting their communication style to meet the needs of different audiences. This is particularly challenging when working with a diverse and multicultural group of patients where the risk of misinterpretation is higher. Students are provided with the opportunity to practice dialogues with virtual patients that are able to interact as real human beings, communicating concerns, emotions, and moods both at a verbal and non-verbal level. Coaching through digital humans accelerates learning from experience without the risks associated with learning in the field. Keywords Sonographer-patient interviews · Cultural intelligence · Diversity & inclusion · Digital humans

B. Bertagni (B) · F. Salvetti · I. Contardo e-REAL Labs at Logosnet, 10014 Turin, Italy e-mail: [email protected] e-REAL Labs at Logosnet, 6900 Lugano, Switzerland e-REAL Labs at Logosnet, Houston, TX 77008, USA e-REAL Labs at Logosnet, New York, NY 10013, USA L. Zanin Montgomery College, 7600 Takoma Ave, Takoma Park, MD 20912, USA © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_5

55

56

B. Bertagni et al.

1 Introduction Sonographers play a critical role in the healthcare industry, especially in diagnostic imaging. They use ultrasound equipment to create images of internal organs and tissues for medical diagnoses. Sonographer-patient communication is essential for a successful ultrasound examination, as the sonographer needs to obtain accurate information from the patient about their medical history, symptoms, and other relevant details. In a diverse and multicultural environment, communication barriers may exist, affecting the accuracy and quality of the ultrasound examination. This article explores the potential of using role-plays with digital humans to improve sonographer-patient communication in a diverse and multicultural environment.

2 Effective Communication Skills for Sonographers: The Digital Humans as a Key Component of the Training Effective communication skills are essential for sonographers to build trust, to explain examination procedures to the patient in non-technical terms, to alleviate anxiety and gain patient consent and collaboration, and to provide information at a pace suitable for the patient. In order to communicate effectively, the sonographer needs to be able to communicate empathetically, adjusting their communication style to meet the needs of different audiences. This is particularly challenging when working with a diverse and multicultural group of patients where the risk of misinterpretation is higher [1–14]. At Montgomery College in Takoma Park, Maryland, USA, we developed a bespoke module of training that aims to boost the sonographer’s communication skills. This module is a component of the Diagnostic Medical Sonography program that connects educational technology with face-to-face instruction/scanning to enhance and personalize an innovative curriculum. Diagnostic medical sonography is one of the fastest growing professions in the health care industry, and it is one of the most challenging and rewarding fields in the digital age, requiring professional judgment and problem solving skills [15]. Students are provided with the opportunity to practice dialogues with virtual patients that are able to interact as real human beings, communicating concerns, emotions, and moods both at a verbal and non-verbal level. We are using a solution known as e-REAL® [16, 17] to deliver immersive, glasses-free experiences, both online and in a “phygital” classroom setting, that allow students to deal with different situations and different patients. Each student is trained to communicate in a realistic scenario, with patients of different ages, gender, culture, and ethnicity (Figs. 1 and 2). Communication barriers in a diverse and multicultural environment can affect the accuracy and quality of the ultrasound examination. These barriers can include language differences, cultural beliefs, and misunderstandings. For example, a patient may not understand the instructions provided by the sonographer due to a language

Improving Sonographer-Patient Communication in a Diverse …

57

Fig. 1 The program’s structure

Fig. 2 Representative avatars or digital humans

barrier, leading to inaccurate information being collected and potentially inaccurate diagnoses. Misunderstandings may also arise due to cultural differences, such as the interpretation of pain, which may differ depending on cultural background. These barriers can lead to miscommunication, mistrust, and a lack of patient engagement, ultimately affecting the quality of the ultrasound examination. Role-plays with digital humans are an effective solution to overcome communication barriers in a diverse and multicultural environment. Digital humans are computer-generated characters that can simulate human behavior and interactions. They allow sonographers to interact with patients in a culturally sensitive manner and provide feedback to sonographers [18]. By using digital humans, sonographers can practice communication skills in a safe and controlled environment, allowing them to develop the most appropriate skills to communicate effectively with patients. Digital humans can also simulate diverse patient populations, including those with different cultural backgrounds and languages. This allows sonographers to practice communication with patients who may have different beliefs, values, and languages. The benefits of using role-plays with digital humans to improve sonographerpatient communication in a diverse and multicultural environment are significant. The use of digital humans can help sonographers overcome communication barriers,

58

B. Bertagni et al.

promote cultural sensitivity, and increase patient engagement. Improved communication can lead to more accurate information being collected, resulting in more accurate diagnoses and improved patient outcomes [19]. Digital humans can also provide immediate feedback to sonographers, allowing them to adjust their communication style and approach to better suit the patient’s needs. This can help to build trust and rapport between the sonographer and patient, leading to improved patient satisfaction. At the end of each interview, timely feedback is provided highlighting the communication style, the quality of the listening, and possible hidden bias in conversations. Coaching through digital humans accelerates learning from experience without the risks associated with learning in the field.

3 Conclusion Early findings show that the program enhanced communication skills and selfawareness regarding their own relational styles depending on the diversity of their patients. Students showed a strong involvement in the training, appreciating the availability to practice at their own pace, and the opportunity to deal with very different patients while learning how to adapt their communication style. Sonographer-patient communication is essential for a successful ultrasound examination. Communication barriers in a diverse and multicultural environment can affect the accuracy and quality of the ultrasound examination. Role-plays with digital humans are a potential solution to overcome these communication barriers. Digital humans can simulate diverse patient populations and provide feedback to sonographers, promoting cultural sensitivity and improving communication. The use of digital humans can lead to more accurate diagnoses, better patient outcomes, and improved patient satisfaction. Further research is needed to explore the potential of role-plays with digital humans in improving sonographer-patient communication in a diverse and multicultural environment.

References 1. Rossi, A., Baldisserotto, M., Del Sette, P.: La comunicazione medico-paziente in ecografia vascolare: il punto di vista dell’ecografista. Med. Lav. 107(5), 341–347 (2016) 2. Vanni, D., Zamparelli, R.: La comunicazione medico-paziente in diagnostica ecografica. G. Ital. Med. Lav. Ergon. 37(2), 99–102 (2015) 3. Aldea, A., Madrigal, E.: The importance of patient communication in sonography. J. Diagn. Med. Sonogr. 33(4), 300–305 (2017) 4. Allen, L., Hartman, T.: Communication strategies for sonographers. J. Diagn. Med. Sonogr. 31(2), 130–136 (2015) 5. Amini, R., Cullum, C.M.: Communication in diagnostic medical sonography: a literature review. J. Diagn. Med. Sonogr. 34(1), 43–49 (2018)

Improving Sonographer-Patient Communication in a Diverse …

59

6. Baraldi, E., Tassoni, G.: La comunicazione medico-paziente nella diagnostica ecografica. Recenti Prog. Med. 107(1), 11–16 (2016) 7. Dantas, L.F., de Souza, C.S., Santos, R.A., de Souza, C.A., Nunes, V.C.: The importance of communication in the sonographer-patient relationship. Radiol. Bras. 52(1), 33–37 (2019) 8. Lippi, G., Plebani, M.: La comunicazione medico-paziente in diagnostica per immagini. Med. Lav. 109(3), 227–232 (2018) 9. Luthy, C., Kaul, S., Rothenberg, E., Kachura, J.R.: A pilot study of patient communication in sonography: Is the language we use really patient centered? J. Diagn. Med. Sonogr. 32(5), 283–289 (2016) 10. McNulty, J.A., Sonntag, V.K., Hogg, J.P.: Patient-centered communication in diagnostic medical sonography. J. Diagn. Med. Sonogr. 34(2), 101–105 (2018) 11. Mercer, J., Reynolds, L.: Exploring the role of sonographers in patient communication. J. Diagn. Med. Sonogr. 31(3), 152–157 (2015) 12. Smith, S.N., Yock, K.I.: Effective communication skills for the sonographer: a practical guide to patient-centered care. J. Diagn. Med. Sonogr. 32(6), 348–353 (2016) 13. Verma, S., Mohanty, M.: Communication in sonography: a review of the literature. J. Diagn. Med. Sonogr. 32(1), 14–20 (2016) 14. Wilson, L.B., Pabico, R.C., Enright, P.L.: The impact of communication skills training on sonographer-patient communication: a pilot study. J. Diagn. Med. Sonogr. 33(1), 10–15 (2017) 15. https://www.montgomerycollege.edu/academics/programs/diagnostic-medical-sonography/ index.html 16. www.e-real.net 17. Salvetti, F., Gardner, R., Minehart, R., Bertagni, B.: Enhanced reality for healthcare simulation. In: Brooks, A.L., Brenham, S., Kapralos, B., Nakajima, A., Tyerman, J., Jain, L. (eds.) Recent Advances in Technologies for Inclusive Well-Being: Virtual Patients, Gamification and Simulation. Springer, Heidelberg (2021) 18. Lewis, B.A., Brenham, S., Kapralos, B., Nakajima, A., Tyerman, J., Jain, L. (eds.): Recent Advances in Technologies for Inclusive Well-Being: Virtual Patients, Gamification and Simulation. Springer, Heidelberg (2021) 19. Salvetti, F., Gardner, R., Minehart, R., Bertagni, B.: Effective extended reality: a mixed-reality simulation demonstration with digitized and holographic tools and intelligent avatars. In: Guralnick, D., Auer, M., Poce, A. (eds.) Innovative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education. Proceedings of the Learning Ideas Conference 2022. Springer, Cham (2023)

Psychology and STEM Education: From the Classroom to Society Evi Botsari , Konstantina Sdravopoulou , and Sarantos Psycharis

Abstract The aim of this paper is to explore the role of psychology in STEM education and, by combining literature from not well connected strands, to eventually offer a conceptual framework encompassing some important aspects of psychology that relate to STEM education. Psychology is indispensable in educational research, but the participation of psychologists and the incorporation of psychological research in STEM education has been rather weak or overlooked. Why and how this happened might prompt for a deeper examination, eventually leading us to address issues relating psychology not only to individual learners, but also to society. Most commonly, the contribution of psychology is exemplified by focusing on cognitive issues such as how students learn and reason in science, technology, mathematics and engineering classes; how their creativity can be enhanced; and how they interact with technology. For instance, it has shown to be particularly useful in the exploration of gaming experiences in education (e.g., by using mobile phones and videogames). Further, social psychology can elucidate how people form attitudes towards scientific practices and particular domains of technology and engineering. In short, most of the branches of psychology (i.e., with the exception of psychanalysis) enter in STEM education (cognitive psychology, social psychology, psychology of emotions, etc.) either directly (in the STEM class) or indirectly (formation of attitudes towards STEM-related subjects). With these considerations, a typology of actual and potential contributions of psychology to STEM education is proposed here, which can be useful as a vision for orienting future psychological research in STEM education. Keywords STEM Psychology · STEM Education · Psychology · Bibliometric

E. Botsari · K. Sdravopoulou (B) · S. Psycharis School of Pedagogical and Technological Education (ASPETE), Athens, Greece e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_6

61

62

E. Botsari et al.

1 Introduction Psychology occupies a central place in the literature of STEM education [1, 2]. Some of the characteristic areas in which psychological research has turned out to be particularly useful in STEM education are in studies examining: a) the role of gender in STEM education [3–6]; b) cognitive aspects of STEM education, with the examination of how STEM may enhance spatial ability prominent among them [7– 11]; c) how STEM increases creativity through games in education [12]; d) students’ and tutors’ beliefs [13] and attitudes [14]; and e) motivation in STEM education [15]. These are certainly not the only areas, since, expectedly, the contribution of Psychology in particular components of STEM education varies according to each STEM component. For instance, psychological research has turned out to be useful in exploring also some difficult circumstances relating to students’ anxiety during courses of mathematics [16, 17]. However, Psychology does not contribute only in the field of how STEM students learn; it also proved indispensable in exploring problem-solving processes during some STEM educational process [18, 19] while its involvement extends, besides STEM, well into STREAM [20]. Partially, the rationale behind the usefulness and utmost significance of Psychology in STEM education has also been presented at professional websites, i.e. at the site of the American Psychological Association [21]. Considering the breadth and diversity of the involvement of Psychology in STEM education [22–24], it appears necessary to carry out an in-depth analysis of the published scientific literature, in order to a) identify spatial and thematic domains in which Psychology is important for STEM education and b) assess its potential for contributing to the advancement of STEM education for each component of STEM education.

2 Methods The published scientific literature (from OpenAlex using Google Scholar, Scopus, Web of Science) that concerns the role of Psychology in education was analyzed geographically, quantitatively and thematically, from its onset until the end of 2022. The keywords used were “Psychology STEM” and “STEM Education.” The former pair of keywords yields results on the role of Psychology in STEM education, while the second pair was used to analyze the literature of STEM education regardless of Psychology. Thus, while “Psychology STEM” is expected to reveal the role of Psychology only in STEM, the second analysis aimed at identifying all important fields and concepts from within the entire literature of STEM education. For Psychology STEM, the search yielded 61 scientific documents, while for STEM education, as many as 6257 documents. The scientific literature was counted and analyzed by using the Vosviewer software which yields network representations of authors, countries, citations, keywords and other bibliometric data by linking the

Psychology and STEM Education: From the Classroom to Society

63

links among publication documents [25–28]. The network analysis of these documents aims at discovering correlations between researchers, countries or concepts. The node sizes represent the number of papers (or number of times a concept appears in the literature) that were published (compared to the sizes of the other nodes) and link thickness reflects the strength of interaction between two nodes (the higher the number of documents that relate the two nodes, the thicker the link is). Consequently, and on the basis of these analyses, a framework is proposed, portraying some characteristic fields in which psychological research is important for each STEM component.

3 Results 3.1 Psychology in STEM The cumulative number of papers that have the term “psychology” in their title has increased in the last years; in fact, this growth is modelled by a cubic polynomial (with correlation coefficient equal to 0.996) with respect to time t, where t = 0 for the year 2003 (Fig. 1). P = −0.0203213t 3 + 0.668221t 2 − 2.20759t + 1.0181 The geographical distribution reveals that the highest numbers of papers were published by authors from institutions American, British, Canadian, Australian and Swedish institutions (the ten prominent of them are shown in Fig. 2). The analysis by keywords suggests that the most important keywords in the literature of STEM Psychology are (beyond psychology itself) engineering, engineering ethics, political science, sociology and special psychology (Fig. 3). Fig. 1 Cumulative number of published papers on psychology and STEM in the scientific literature. Evidently, this field attracts the interest of increasingly more researchers worldwide. A cubic polynomial models the number of papers per 5-year interval

64

E. Botsari et al.

Fig. 2 The ten prevailing institutions worldwide from which publications on psychology and STEM have originated

Fig. 3 The 24 most important concepts in the literature on psychology in STEM: applications in physics and medicine appear more frequent in the last years. The time scale indicates the average publication year that applies to nodes and links

Psychology and STEM Education: From the Classroom to Society

65

Fig. 4 Cumulative numbers of papers that have been published on STEM education in the scientific literature, plotted on the basis of 5-year interval data and the quintic polynomial model that best fits the data

3.2 The Role of Psychology in the Literature of STEM Education The literature of STEM education also expands rapidly: its growth can be modeled (Fig. 4) by a quintic polynomial model (corr.coeff = 1), plotted on the basis of 5year interval data of documents (the time t = 0 corresponds to the b-year interval 1991–95): P = −0.00519733t 5 + 0.289267t 4 − 5.15767t 3 + 36.7683t 2 − 86.81t + 1 The country that participated most in the literature is the United States, but if the changes with time are considered, they indicate a marked shift to Asian countries in the last years (China, Taiwan, Indonesia, Turkey, Vietnam, and other countries; see Fig. 5). As concerns the key concepts in the field of STEM education, psychology ranks first and most prominent among all the concepts (Fig. 6).

66

E. Botsari et al.

Fig. 5 Cumulative number of papers that have been published on STEM education in the scientific literature, by country of origin

4 Discussion The ten most important concepts encountered in the literature of “STEM education” are significantly correlated with those of the literature on “Psychology STEM” (Table 1). From this table, it can be seen that the four components of STEM are represented by the ten most important keywords in both literature datasets, with the exception of a keyword relating to S (sciences) and, thus, future research in STEM education may focus more on psychological research in the sciences component of STEM education. The Spearman rank coefficient between the two columns is 0.5515, therefore indicating a statistically significant correlation between the two rankings: the ten most important keywords associated with the role of psychology in the literature of STEM psychology are correlated with the ten most important keywords of STEM education.

Psychology and STEM Education: From the Classroom to Society

67

Fig. 6 The most frequently occurring concepts in the literature of STEM education: psychology ranks first, followed by mathematics education. The time scale indicates the average publication year that applies to nodes and links

In view of these findings and considering that a) psychologists have been concerned with spatial differences across scales [29–32], b) some of the most frequent keywords encountered in the literature are sociology and political science, and c) the expansion of psychological research in STEM education worldwide, and, eventually, pursuing a global assessment of the role of Psychology in STEM education for all STEM components, a vision of the role of how Psychology might possibly contribute in advancing STEM education across social-geographic scales can be derived (Fig. 7), with the aim of fostering a more effective STEM education worldwide.

68

E. Botsari et al.

Table 1 Ranks of the ten most important keywords in the literature of Psychology STEM and STEM Education Keywords

Rank in the literature of: Psychology STEM

STEM Education

Psychology

1

1

Engineering

2

5

Political Science

3

4

Computer Science

4

6

Pedagogy

5

3

Sociology

6

7

Law

7

8

Philosophy

8

10

Mathematical Education

9

2

Medicine

10

9

Fig. 7 The various cases of relevance of psychology to the theory and practice of STEM education according to the geographical/social scale and the complexity of the intervention

5 Conclusions After this examination of the role of Psychology in the literature of STEM and STEM education, the following conclusions are derived: a) the number of papers with psychological research in STEM education has grown rapidly during the last years and this increase is associated with a geographical shift since increasingly more papers originate from Asian countries; b) the ten most important common concepts to which the literatures of both STEM Psychology and STEM education

Psychology and STEM Education: From the Classroom to Society

69

refer to are correlated; c) more emphasis should be placed on the “S” component of STEM in future psychological research in STEM, since most papers appear to concern psychological research in the other three components of STEM; and d) a typology of the role of Psychology in STEM is proposed, highlighting the fields in which Psychological research and applications can be useful in learning and in exploring emotions, behaviors, beliefs, attitudes, and decision-making processes in the context of STEM education.

References 1. Li, Y.: Journal for STEM education research–promoting the development of interdisciplinary research in STEM education. J. STEM Educ. Res. 1, 1–6 (2018) 2. Gil-Doménech, D., Berbegal-Mirabent, J., Merigó, J.M.: STEM education: a bibliometric overview. In: Modelling and Simulation in Management Sciences: Proceedings of the International Conference on Modelling and Simulation in Management Sciences (MS-18), pp. 193–205. Springer, Cham (2020) 3. Saucerman, J., Vasquez, K.: Psychological barriers to STEM participation for women over the course of development. Adultspan J. 13(1), 46–64 (2014) 4. Seyranian, V., Madva, A., Duong, N., Abramzon, N., Tibbetts, Y., Harackiewicz, J.M.: The longitudinal effects of STEM identity and gender on flourishing and achievement in college physics. Int. J. STEM Educ. 5(1), 1–14 (2018) 5. Kizilcec, R.F., Saltarelli, A.J.: Psychologically inclusive design: cues impact women’s participation in STEM education. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, Glasgow, Scotland, UK, pp. 1–10 (2019) 6. Ansong, D., Okumu, M., Albritton, T.J., Bahnuk, E.P., Small, E.: The role of social support and psychological well-being in STEM performance trends across gender and locality: evidence from Ghana. Child Indic. Res. 13, 1655–1673 (2020) 7. Wai, J., Lubinski, D., Benbow, C.P.: Spatial ability for STEM domains: aligning over 50 years of cumulative psychological knowledge solidifies its importance. J. Educ. Psychol. 101(4), 817 (2009) 8. Lubinski, D.: Spatial ability and STEM: a sleeping giant for talent identification and development. Personality Individ. Differ. 49(4), 344–351 (2010) 9. Uttal, D.H., Cohen, C.A.: Spatial thinking and STEM education: when, why, and how? In: Psychology of Learning and Motivation, vol. 57, pp. 147–181 (2012) 10. Buckley, J., Seery, N., Canty, D.: A heuristic framework of spatial ability: a review and synthesis of spatial factor literature to support its translation into STEM education. Educ. Psychol. Rev. 30(3), 947–972 (2018) 11. Harris, D., Lowrie, T.: The distinction between mathematics and spatial reasoning in assessment: do STEM educators and cognitive psychologists agree? Mathematics education research group of Australasia. In: Proceedings of the 41st Annual Conference of the Mathematics Education Research Group of Australasia, pp. 376–383. MERGA, Auckland (2018) 12. Gabdulchakov, V.F., Shishova, E.O.: Psychology of children’s play, imagination, creativity and playful pedagogies in early childhood education in Russia. In: Play and STEM Education in the Early Years: International Policies and Practices, pp. 65–83. Springer, Cham (2022) 13. LaCosse, J., Murphy, M.C., Garcia, J.A., Zirkel, S.: The role of STEM professors’ mindset beliefs on students’ anticipated psychological experiences and course interest. J. Educ. Psychol. 113(5), 949 (2021) 14. Vennix, J., den Brok, P., Taconis, R.: Do outreach activities in secondary STEM education motivate students and improve their attitudes towards STEM? Int. J. Sci. Educ. 40(11), 1263– 1283 (2018)

70

E. Botsari et al.

15. Richardson, D.S., Bledsoe, R.S., Cortez, Z.: Mindset, motivation, and teaching practice: psychology applied to understanding teaching and learning in STEM disciplines. CBE Life Sci. Educ. 19(3), ar46, 1–7 (2020) 16. Rozgonjuk, D., Kraav, T., Mikkor, K., Orav-Puurand, K., Täht, K.: Mathematics anxiety among STEM and social sciences students: the roles of mathematics self-efficacy, and deep and surface approach to learning. Int. J. STEM Educ. 7(1), 1–11 (2020) 17. Hall, S.S., McGill, R.M., Puttick, S., Maltby, J.: Resilience, science, technology, engineering, and mathematics (STEM), and anger: a linguistic inquiry into the psychological processes associated with resilience in secondary school STEM learning. Br. J. Educ. Psychol. 92(3), 1215–1238 (2022) 18. Kim, N.J., Belland, B.R., Walker, A.E.: Effectiveness of computer-based scaffolding in the context of problem-based learning for STEM education: Bayesian meta-analysis. Educ. Psychol. Rev. 30, 397–429 (2018) 19. Kim, N.J., Belland, B.R., Lefler, M., Andreasen, L., Walker, A., Axelrod, D.: Computerbased scaffolding targeting individual versus groups in problem-centered instruction for STEM education: meta-analysis. Educ. Psychol. Rev. 32, 415–461 (2020) 20. Yoh, T., Kim, J., Chung, S., Chung, W.: STREAM: a new paradigm for STEM education. J. STEM Educ. Innov. Res. 22(1), 46–51 (2021) 21. American Psychological Association. Advocacy for Psychology as a STEM Discipline. Resolution adopted by Council (2011). https://www.apa.org/about/policy/stem-advocacy. Accessed 1 Mar 2023 22. Mark, S.L.: Psychology of working narratives of STEM career exploration for non-dominant youth. J. Sci. Educ. Technol. 25, 976–993 (2016) 23. Master, A., Meltzoff, A.N.: Building bridges between psychological science and education: cultural stereotypes, STEM, and equity. Prospects 46(2), 215–234 (2016) 24. Merolla, D.M., Serpe, R.T.: STEM enrichment programs and graduate school matriculation: the role of science identity salience. Soc. Psychol. Educ. 16(4), 575–597 (2013) 25. Waltman, L., Van Eck, N.J., Noyons, E.C.M.: A unified approach to mapping and clustering of bibliometric networks. J. Informet. 4(4), 629–635 (2010) 26. Van Eck, N.J., Waltman, L.: Software survey: vosviewer, a computer program for bibliometric mapping. Scientometrics 84(2), 523–538 (2010) 27. Van Eck, N.J., Waltman, L., Noyons, E.C.M., Buter, R.K.: Automatic term identification for bibliometric mapping. Scientometrics 82(3), 581–596 (2010) 28. Van Eck, N.J., Waltman, L.: Citation-based clustering of publications using CitNetExplorer and Vosviewer. Scientometrics 111(2), 1053–1070 (2017) 29. Carr, S.C., Sloan, T.S. (eds.) Poverty and Psychology: From Global Perspective to Local Practice. Springer, New York (2003) 30. Weil, M.M., Rosen, L.D.: The psychological impact of technology from a global perspective: a study of technological sophistication and technophobia in university students from twenty-three countries. Comput. Hum. Behav. 11(1), 95–133 (1995) 31. Rentfrow, P.J., Jokela, M.: Geographical psychology: the spatial organization of psychological phenomena. Curr. Dir. Psychol. Sci. 25(6), 393–398 (2016) 32. Rentfrow, P.J.: Geographical psychology. Curr. Opin. Psychol. 32, 165–170 (2020)

Creating Cloud Experts: The Power of an Innovative, Hybrid Learning Approach Natalie Brooks Powell

Abstract In a business that changes as quickly as technology, learning and development (L&D) is—or should be—indispensable for both customers and employees. But when an L&D program is a small start-up within a major corporation that requires quantifiable results to prove the value of its operations, what’s an L&D to do? For the IBM Center for Cloud Training (ICCT), the market challenger, the answer was to operate grass-roots marketing and communication programs that enable it to move like a shark—pivoting quickly when survival required a new direction and never stopping in its quest to achieve success with both its internal and external constituents. Keywords IBM · Cloud · Training · Certification · IBM center for cloud training · ICCT · Learning · Development · Learning and development · L&D · Learners · Managers · Management · C-level · Mid-level · Line-level

1 Starting on the Road to Cloud Success It began with the discovery of a gap. After launching its suite of cloud technologies, IBM noticed a significant shortfall in the number of trained professionals worldwide necessary to implement and manage cloud solutions. Along with ongoing digital transformation, employment environments are also changing. Traditional jobs are being replaced with new roles and specialties unimaginable only a few years ago. IBM tapped Jani Byrne Saliga, Ph.D., a longtime IBM professional working to lead a workstream, with the charge of addressing the need for a learning and development effort supporting IBM Cloud. She then developed a business case that examined: N. B. Powell (B) IBM Center for Cloud Training, Armonk, USA e-mail: [email protected] Evanston, USA © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_7

71

72

• • • •

N. B. Powell

Market opportunity/demand for cloud certification Cloud skills gap internally and externally Competitive landscape in cloud computing Reasons to invest in an end-to-end certification program that IBM could offer to clients, business partners, and employees

This examination produced an overview of an organization that would become ICCT, how to operationalize the division, and the level of investment/ funding required. With these findings and conclusions in hand, Dr. Saliga then found a sponsor in Obed Louissaint, then Senior VP of IBM Talent, who, like her, understood the need for high-quality IBM Cloud training. With his sponsorship, they took the idea for a program that would offer training and professional certifications in cloud technologies to IBM’s CEO, Arvind Krishna, who agreed to this new initiative in late 2019. Realizing that cloud would continue to transform the way companies work—at an ever-increasing pace—IBM leadership decided to invest in this initiative, creating the IBM Center for Cloud Training (ICCT). ICCT launched a role-based, multi-level certification program. The concept and processes are innovative both within IBM and the industry. The program’s hybrid learning strategy is primarily online learning but bolstered with supplemental performance support experiences, such as Study jams, virtual online instructor-led sessions. ICCT learning is available to external clients, business partners, global systems integrators, individual learners, IBM employees, and more (Fig. 1). To further IBM’s goal to deliver a secure, hybrid cloud platform, ICCT delivers engaging, high-quality curricula. The training is based on the most sought-after Cloud job roles—Developer, Architect, Site Reliability Engineer—and through its associated test of expertise of these skills (in other words, certifications and badges). Fig. 1 Start your journey at the IBM center for cloud training video

Creating Cloud Experts: The Power of an Innovative, Hybrid Learning …

73

2 Developing a Portfolio that Creates Cloud Experts 2.1 Preparing to Launch the Program With a business strategy that seeks to make the company an essential provider of cloud-based solutions, ICCT offers a portfolio of professional certifications in cloud functions, grouped into four levels: Associate, Professional, Advanced, and Specialty. Candidates also can earn badges as they complete stages of training curricula. IBM conducts significant research, continuously evaluating needed roles and how to train learners to fill roles. In preparation for the 2020 launch, ICCT commissioned an extensive Human Resources study to anticipate learning needs by identifying the most sought-after cloud jobs worldwide. From this data, ICCT determined its mission would be to focus on specific job roles to form the basis of the program. To meet the challenges of training in a rapidly evolving technology environment— where research has shown that significant amounts of training material must be updated annually—ICCT employs a team of learning and talent specialists, subject matter experts, instructional designers, and course builders. To date, ICCT has developed and now offers 14 certifications. These range from foundational programs for those new to cloud to role-based programs for those in specific job functions, to specialty certifications for those with industry-specific needs.

2.2 Designing in an Ever-Changing Landscape In evolving and updating its program offerings, ICCT follows a formal “ADDIE” model (analysis, design, development, implementation, evaluation). An IBM internal team defines a desired outcome, then establishes the deliverables that will be most effective to achieve the intended outcomes. Additionally, when it comes to designing and implementing its training offerings, IBM follows an innovative deployment model that is predicated on the premise of “build once; deploy twice,” so that everyone (clients, partners, learners, and IBMers) receives the same industry-standard education on the internal and external platform. This ensures a more efficient roll-out of new courses and certifications and has the added benefit of creating familiarity for learners, who can follow a logical progression of curricula from entry-level through advanced and specialized certifications. Four additional design points play noteworthy roles in helping ICCT achieve its business objectives. • Enable rapid course development. Rapid development techniques used to meet initial delivery goals were helpful when the team was asked to add new learning paths and certifications that were not in the initial business plan.

74

N. B. Powell

• Synchronize learning paths and certification exams. This design process is a rigorous, innovative, best-in-class approach that is markedly different from the way most organizations create their certification and training programs. Instead of building a curriculum and then writing tests to determine whether participants have learned the information taught in class, ICCT begins with what is traditionally the end of the learning cycle—acquired professional skills. • Leverage IBM subject matter experts effectively and efficiently. Technical experts who are willing to assist with training are a valuable, indispensable, and finite resource. It is important that experts share their expertise and shape the next generation of cloud leaders. • Build in diversity. ICCT creates expert teams that have globally diverse backgrounds to ensure diversity in perspectives. This practice reduces bias in certification exam development and extends to building learning paths that appeal to wider, global audiences.

3 Meaningful Metrics 3.1 Recognizing a Global Need ICCT is built on the understanding that cloud computing has enormous potential for growth, with market demand that IBM and research have projected to hit $1 trillion USD by 2026 [1]. As opportunities increase for individual professionals, they also increase for the company, both in terms of market share and in the overall growth of cloud adoption. This growth will be led by trained people who can understand, use, share, and advise businesses regarding this crucial technology. IBM’s commitment to meeting this need is demonstrated by the company’s global plan [2] to provide 30 million people of all ages with new skills needed for the jobs of tomorrow by 2030. To achieve this goal, IBM has announced a roadmap with more than 170 new academic and industry partnerships.

3.2 Evolving Skills With this in mind, ICCT offers its training programs in ways that can reach the largest number possible of internal and external learners. To take full advantage of market changes, cloud enthusiasts must learn new cloud skills and competencies. Historically, this learning was accomplished by providing in-classroom training, and, later, digital learning. Educators tracked metrics such as the number of enrollments and the number of course completions. However, businesses soon found that the desired return on investment in training was not materializing and, as a result, training was often seen as an unnecessary

Creating Cloud Experts: The Power of an Innovative, Hybrid Learning …

75

expense. Education budgets were often slashed in economically difficult times. More critically, businesses began asking about the effectiveness of learning. For this reason, there is a movement afoot to focus less on traditional learning metrics, and instead on whether or not learners can demonstrate a mastery of the information, as demonstrated by successful completion of high-quality, third-party proctored exams. This skill mastery, in the form of certifications, is becoming the “currency” that many clients, business partners, and employees are demanding.

3.3 Understanding the Value of Credentials Studies show that professionals agree that certification makes them more successful. With training, more employees will opt to remain, their productivity will increase, and the company will sell more of its products. One recent study [3] documented learners’ satisfaction, noting that 75% of those surveyed said certification was a key factor in receiving a pay increase – and 78% experienced increased demand for their skills. Internally, IBM found that 92% of surveyed participants in its badge program said an IBM digital credential improved their employability [4]. The same study found that technical sales professionals who have earned digital badges are more likely to achieve sales quotas than employees who have not earned badges [4].

4 Achieving Big Results, Right Away ICCT’s program of training and certification is ambitious, and its results have been immediate. ICCT was launched in 4Q-2020, and the number of ICCT certified learners surpassed expectations. Notably, one senior manager stated that the ICCT initiative helped her group achieve learning scalability. Her group of 200 earned over 100 IBM Cloud certifications during the fourth quarter. By 2021, within ICCT at large, total certifications earned rose by 1,047% yearover-year, underscoring the demand for IBM Cloud training. And by 2022, ICCT’s goal was to increase credentialing of learners by an additional 40%. Momentum was strong, as evident in the fact that one vice president not only required all 300+ of his team to earn at least one certification, but he also led by example and passed a certification himself. IBM ultimately exceeded its target number of new credentials.

76

N. B. Powell

5 Giving Professionals Innovative Ways to Learn To support outstanding results in training, ICCT continuously monitors and recognizes market opportunities and trends while listening closely to clients’ challenges. It pivots and adapts its training and credentials to meet changing requirements. Another key has been to act like a startup. With an “adapt fast” culture, ICCT experienced early breakthroughs, resulting in a fresh, modern, learner-centric program that eclipsed traditional learning approaches. The result: ICCT has demonstrated real results in record time. ICCT employs a rich mix of innovative delivery and instructional methods so that each program participant can benefit from the style that suits the learner best. It is ICCT’s philosophy that learners should lead their own learning—that is, the learner is in control. ICCT content is delivered via technology, specifically through a web-based platform. Some learners prefer an in-depth, solo learning experience, taking one course at a time. Others thrive on social interaction and prefer intense, short-term study sessions with like-minded learners. Others look for a hybrid training solution—a mix of instructor-led training and digital self-study. Using technology, ICCT delivers multiple learning styles to suit all of these individual learners.

5.1 Supporting Engagement with a Helpful Avatar One of the most innovative and unique features of ICCT’s program is Ingrid, an engagement tool and visual guide, or as she describes herself, a “helper.” Ingrid’s “job” is to lead learners through the various stages of obtaining their certifications. She is designed to act as a friend and ease participants’ fears as they navigate IBM Cloud Certification. This experiment with artificial intelligence and augmented reality helps to better engage learners. Ingrid is a creative expression of ICCT who serves learners by motivating them, informing them, and clarifying various aspects of the ICCT learning experience.

5.2 Learning Together Over an Intensive Three Days ICCT’s learning audiences are busy professionals who were isolated due to the pandemic. Some needed short, instructor-led sessions at dedicated times. They also benefitted from the camaraderie of others going through the same experience. ICCT introduced Study jams and the result was an increase in certifications. Study jams are an innovative approach to learning and, eventually, to certification – built on robust and engaging study sessions that bring together IBM thought leaders, certified professionals, and learners from all over the world. These in-depth, online

Creating Cloud Experts: The Power of an Innovative, Hybrid Learning …

77

events incorporate seminars, one-on-one answer sessions with IBM experts, videos, and a dedicated Slack study channel, where an enthusiastic cloud community is ready to assist learners through collaboration and answering questions. The Study jam experience was valuable as it led to other innovations and validated the importance of using a variety of strategies to reach the audience. This information also helped validate the demand for IBM Cloud training and confirm the ICCT curricula and client advocacy strategies for this year.

5.3 Offering Performance Support Experiences for Learners To ensure it is meeting the complete range of participants’ learning styles and needs, ICCT also offers additional programs including: Study support: • Study jam: In-depth, online sessions featuring IBM experts, videos, and more, streamlined to help learners prepare for their certification exams. • IBM Cloud Prep App: A progressive web application and all-in-one study-on-thego resource featuring flashcards, detailed study guides, and practice quizzes. • Cloud Compass: An interactive, training evaluation tool designed to help learners find the certification that’s right for their career path. Motivational elements: • Badges: Introduced in 2022, displayable Credly badges are earned as learners work through cloud training curricula. • Second Chance promotion: An opportunity for learners to retake ICCT Cloud certification exams for free. • Competitions: A gamified method to increase credential outcomes. Information and knowledge sharing: • ICCT Cloud Training News: A regularly published newsletter that informs learners of the latest opportunities in ICCT certifications. • ICCT on Tech TV: A monthly broadcast, featuring the latest technical talks and up-to-date information from IBM experts. • IBM Cloud Certification FAQs: Answers to typical learner questions, including a certification exam checklist. Community: • IBM Cloud Community: An online source where collaborative experts share new training certifications, cloud discussions, training webinars, and blogs. • IBM Cloud blog: Online articles containing the latest information, research, and case studies related to IBM Cloud. • Amplify website: Communications that complement the IBM Cloud blog with success stories and best practice tips for training.

78

N. B. Powell

Additionally, as it recognized the value of community in learner success, ICCT piloted a unique initiative in 2022 for its Associate Site Reliability Engineer (ASRE) program, which recruited learners to earn certifications and achieved over 400% of the anticipated project enrollment. Those professionals who joined were engaged in a programmatic approach as a community experience. Participants received weekly email check-ins, marking progress along the ASRE learning path. Additional support to provide encouragement was made available via the IBM Cloud Prep App, which included study guides, quizzes, flashcards, and a dedicated Slack channel.

6 Fitting Success into the Budget Squeeze Budgets are constantly cut in today’s marketplace, and L&D organizations often are structured—or at least viewed—as a part of the business that does not make a profit. And when they are a small unit in a large, profit-centered organization, challenges to funding and staffing can be omnipresent. So, like most L&Ds, ICCT needed to reliably secure the resources that are necessary for it not only to survive but to thrive. It needed ways not only to meet its educational mission but to attract and maintain the attention and support of all audiences. ICCT garnered the attention of stakeholders, managers, executives, clients, and individual learners by designing programs that answered this common L&D dilemma. ICCT consistently broadcast its efforts and the value of its programs with a steady stream of grass-roots marketing efforts – blogs, speaking engagements, streaming videos, entries into competitions run by professional training organizations, ads on social media sites for The Weather Channel (an IBM Business)—that were designed for maximum exposure within its budget.

6.1 Gaining Support from Upper-Level and C-level Leadership C-level leaders who see shrinking budgets are challenged to maximize their investments, which they may attempt to achieve by increasing efficiency or reducing costs. One way to approach upper-level and C-level leadership is to demonstrate the “scalability” of L&D programming, that is, how minimal administrative effort can still support the education of increasing numbers of learners.

Creating Cloud Experts: The Power of an Innovative, Hybrid Learning …

79

Besides efficiency, C-level leadership often also responds to opportunities to increase revenue with education initiatives that expand and enhance the skills of their employees and the quality of service to customers. L&Ds, in short, can communicate to C-level leaders that investing in L&D programming offers the organization a proven way to increase revenue, retain customers and employees, and upskill their workforce to compete in their respective industries. One targeted—and particularly successful example—produced quickly and easily at ICCT is an e-newsletter titled “Inside Edition” that ICCT’s Chief Learning Officer sends to C-level and mid-level executives at IBM, highlighting recent accomplishments and new projects. A particular focus is news that benefit internal and external business audiences, such as the recent addition of two new IBM Cloud certifications for business-focused technologies: VMware and SAP.

6.2 Getting Support from Mid-Level and Line-Level Management Mid-level and line-level management have more at stake than others when it comes to supporting—or competing with—L&D programs. There may be competition, for example, for limited resources; however, businesses are designed to succeed through collaboration with a reliance on smooth interactions. One way to overcome competition is to prove the value of L&D’s programming, especially as it pertains to other divisions. L&Ds can offer proof that when teams are certified, they perform better, they are more cost-efficient, and they raise employee confidence. Each of these translates into the ability to take on challenging tasks. In fact, despite historical perceptions that ROI does not keep pace with training, a recent study of one company’s training programs found that clients who used its offerings realized an average three-year return on investment of 365% [5]. The enhanced ability to hold on to key employees is another incentive for midlevel and line-level managers who want to retain the best personnel. To that end, L&Ds can show that when employees are offered the opportunity to expand their skills and knowledge, they tend to repay the organization with a longer length of tenure. In IBM’s own experience, certified staff differentiate themselves from other employees in terms of both skills and organizational value. For managers, this means having greater confidence that their certified team members can handle more challenging and complex tasks.

80

N. B. Powell

7 Attracting and Engaging Learners with Demonstrated Value Candidates for certification often have varying degrees of education, skills, and knowledge. They likely have job-related responsibilities that leave them little time to dedicate to continuing their education. And they may be spread out globally, or dispersed in alternative work environments, including at home, far from the centralized office space. Each of these characteristics presents a challenge for meeting learning needs. L&D programs typically address an organization’s employees, independent professionals, employees of other companies, and students for whom certifications can help boost their skills, knowledge, and careers. While upper-level and mid-level leaders seek business advantages via a certified workforce, the source of benefits for learners is different. Continuing education can be fulfilling in a variety of ways, from increased professional opportunities and greater responsibilities to the simple joy of learning new skills. To meet learners’ needs, ICCT has created an ecosystem that keeps learners engaged—and learning—as they undertake their journeys to upskill in cloud technologies. This ecosystem leverages a number of insights based on program experience that have established ICCT as a leader in the L&D space. Over its three-year existence, ICCT has discovered that learners: • • • •

Do not have a lot of time for time-intensive credentialing programs May be globally dispersed, leading to cultural and language challenges Come to training and certification with varying degrees of knowledge and skills Cannot always afford to pay for training and certification programs

ICCT addresses these issues through a number of tactics. Each can be easily replicated by any L&D program. • • • •

Let time-constrained learners lead their own learning Provide an array of support to assist a globally dispersed audience Ensure that learners begin at the right level for certification Support learner training financially or offer meaningful incentives

8 Conclusion: Enjoy the Cloud Journey and Come Out Ahead Is there a recipe for success in L&D? That depends on the organization, its mission, and its people. For many companies, it may be possible to combine the most useful or pertinent ingredients from a successful organization like ICCT and find success. ICCT has succeeded using multiple techniques. It provides and maintains multiple forms of communication to learners as well as upper-level and mid-level leadership.

Creating Cloud Experts: The Power of an Innovative, Hybrid Learning …

81

It is committed to introducing new programs and upgrading current certifications. It maintains an alertness to the needs of the industry and its learners. Embrace and extend new modalities and methods of imparting knowledge, whether by enhancing nonconsecutive learning, new delivery methods and modalities (for example, entry into the Metaverse), and re-imaging the art of the possible to ensure that learners learn. In L&D, the quest for success never ends. But the effort to upskill and educate people is worthwhile and profitable. Acknowledgements I would like to thank Dr. Jani Byrne Saliga and the IBM Center for Cloud Training Advocacy, Curriculum, and Certification Teams.

References 1. Forrester: Public Cloud Is Poised to Surpass $1 Trillion By 2026 – But Not Without Enduring Several Global Challenges. Forrester Research, Inc. (2022). https://investor.forrester.com/newsreleases/news-release-details/forrester-public-cloud-poised-surpass-1-trillion-2026-not 2. IBM Corp. IBM Commits to Skill 30 Million People Globally by 2030. Press Release (2021). https://newsroom.ibm.com/2021-10-13-IBM-Commits-to-Skill-30-Million-People-Globallyby-2030#:~:text=Today%2C%20IBM%20commits%20to%20providing%2030%20million% 20people,build%20a%20better%20future%20for%20themselves%20and%20society.%22 3. Cooper, C.: Why Get IT Certified? The Value of IT Certification, IT Certification Council (2021). https://itcertcouncil.org/value-of-it-certification-white-paper/ 4. Leaser, D.: Do Digital Badges Really Provide Value to Business? IBM Corp. (2019). https:// www.ibm.com/blogs/ibm-training/do-digital-badges-really-provide-value-to-businesses/ 5. Goetz, K.: IDC study shows Red Hat training delivers ROI of 365%. Red Hat, Inc. (2021). https://www.redhat.com/en/blog/idc-study-shows-red-hat-training-delivers-roi-365#:~:text= For%20this%20study%2C%20IDC%20explored%20the%20value%20and,different%20largescale%20organizations%2C%20spanning%20multiple%20industries%20and%20countries.

A Pedagogy for Engineering Concepts Focusing on Experiential Learning Kanmani Buddhi

Abstract In this work, we attempt to present pedagogy for engineering courses. The proposed pedagogy can be applied to a large number of engineering concepts in various courses. The proposed pedagogy involves the faculty designing a set of experiments, for the proposed engineering concept. Students implement the experiment, designed by the faculty, observe the results and attempt to summarize the observations. The summary statement by the student is the actual statement of the intended engineering concept. In this pedagogy, since the student has gone through the experience of the engineering concept, and also made the formal statement of the concept, the student comprehends and appreciates the concept, and hence, enhances student learning. The suggested pedagogy is demonstrated through experiential design for concepts like: Fourier series, Central Limit Theorem, and Sampling Theorem. The usual pedagogy adopted involved five major stages: (i) the formal statement of the concept, (ii) the mathematical equation of the concept, (iii) the mathematical proof of the concept, (iv) numerical examples to apply the concept, and (v) implementation of experiments in laboratory to prove the concept. This five-step process for every concept was spread over one or two weeks, as found essential. While students were able to acquire conceptual knowledge, not many were able to answer assessments that involved true comprehension of the concept. The revised pedagogy required students first go through the laboratory experience, related to the intended concept. At the end of the laboratory sessions, every student records the observation. The revised pedagogy also involves five steps, but with the laboratory experience first. This changed pedagogy resulted in enhanced student learning. Keywords Experiential learning · Fourier series · Sampling theorem · Central limit theorem · Pedagogy

K. Buddhi (B) Department of Electronics and Telecommunication Engineering, BMS College of Engineering, Bengaluru 560019, India e-mail: [email protected] URL: https://bmsce.ac.in/ © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_8

83

84

K. Buddhi

1 Introduction The quality of graduating engineers is assured through the signatories of the Washington Accord, originally signed in 1989, and with more than twenty countries having full rights of participation in the accord, and represented by the respective accrediting body [1]. For example, the United States is represented by Accreditation Board for Engineering and Technology (ABET) [2], while India is represented by the National Board of Accreditation (NBA) [3]. The Washington Accord, through the accrediting bodies, ensures global acceptance of engineering graduates from accredited programs. Accordingly, the Student Outcomes [4] and Program Outcomes [5], ensure every student graduating from the engineering program have the essential domain knowledge together with professional skills and attitude. The need to develop the defined Outcomes through the program has resulted in the need to effectively implement Outcomes Based Education (OBE). This requirement has necessitated the need to bring in transformation in the teaching methodology, focused on engaging the students through the learning process: through including activity based learning, problem based learning, project based learning, one-minute paper, and think-pair-share [6–8]. In this work, we attempt to present pedagogy for engineering concepts, through laboratory experience. The proposed pedagogy can be applied to a large number of engineering concepts in various courses. The proposed pedagogy involves the faculty designing a set of experiments, for the proposed engineering concept. Students implement the experiment, designed by the faculty, observe the results and attempt to summarize the observations. In this work, we have considered the experiential learning through laboratory experience for Fourier series, Sampling theorem and the Central limit theorem. To help explore further, we have included access to the Python code developed to comprehend Fourier series and the Central limit theorem, and access to the Multisim Live circuit for the Fourier series. The laboratory experience for Sampling theorem is through use of discrete components, and the circuit is included. Hence, the pedagogy used to demonstrate experiential learning can be reproduced and explored further. In all the experiments, the summary statement by the student is the actual statement of the intended engineering concept. In this pedagogy, since the student has gone through the experience of the engineering concept, and also made the formal statement of the concept, the student comprehends and appreciates the concept, and hence, enhances student learning. The improvement in student learning is measured through suitable assessments. As a faculty in Electrical Sciences, who has taught under graduate courses related to Analog/ Digital Signal processing, and Analog/Digital Communication, which involve mathematical formulae, and abstract concepts, it has been disappointing to note that despite dedicated teaching efforts, the percentage of students truly comprehending the concepts was small. Through practice of the numerical examples, most students were able to solve the given problem, obtain desired results. However, the actual appreciation of the concept was not observed. The online experience during the pandemic, collapsed the usual border that existed between the theory

A Pedagogy for Engineering Concepts Focusing on Experiential Learning

85

session and the laboratory session, and there emerged the necessity to explore open source tools for laboratory sessions, so that student learning was not affected. The collapse in the boundary between the theory and laboratory sessions also resulted in the course faculty designing and handling the labs. This requirement during the pandemic provided the opportunity to explore new pedagogical initiatives. In this work, we are sharing the pedagogy that emerged during the pandemic, forcing the need to explore online platforms for conduction of laboratory sessions. The ultimate goal is to see that the curriculum has courses, and courses can have theory/tutorial/ laboratory sessions, as found essential by the course faculty, and hence the collapse of the existing rigid boundaries between the theory component, the tutorial component and the laboratory component of courses in the overall curricular structure. In the next section, we consider the pedagogy for three engineering concepts: Fourier series, Sampling theorem and the Central limit theorem.

2 Pedagogy for Engineering Concepts 2.1 Pedagogy for Fourier Series Fourier series is an essential concept for students from all branches of engineering, and especially for students from the Electronics and allied disciplines. Students usually get introduced to Fourier series in Mathematics, and further in courses on Signal processing, and also in course on Computer Networks. The notation used by various authors is different, and very few students truly comprehend and appreciate Fourier series [9–13]. The usual pedagogy adopted involved five major stages: (i) the formal statement of the concept, (ii) the mathematical equation of the concept, (iii) the mathematical proof of the concept, (iv) numerical examples to apply the concept, and (v) implementation of experiments in laboratory to prove the concept. This five step process for every concept was spread over one or two weeks, as found essential. To test student learning, suitable assessments were designed. This usual pedagogy applied for Fourier series is shown in Fig. 1. While students were able to obtain the Fourier series of any given periodic signal, not many were able to answer assessments that involved true comprehension of the concept. In most academic years, this course included one of the internal assessments based on the ‘Signals and Systems Concepts Inventory,’ which is a set of 25 questions to test students in signal processing concepts, and made available on request [14]. One of the questions in the internal assessment for the third semester course that includes Fourier series as the topic in course content is as given in Table 1. Question 1 is with the usual pedagogy, while Question 2, is with revised pedagogy. The similarity in both the type of questions can be observed. Subparts (a) and (b), involve direct application of the Fourier series formulae, and more than 60% of the class (with strength of 60 students), were able to answer the questions. However, in part (c)—which involves comprehending the concept of Fourier series, as indicated

86

K. Buddhi

Fig. 1 The usual Pedagogy adopted for the concept on Fourier Series

in Fig. 1—less than 5% of the class was able answer, when the usual pedagogy was used. On revising the pedagogy, as indicated later in Fig. 11, the percentage of students who could correctly answer part (c), enhanced to 60% of the class. The pedagogy was changed. In the proposed pedagogy, students first go through the laboratory experience, related to Fourier series. The first step here involved students implementing a set of experiments related to the Fourier series representation of continuous time periodic signal using the electronic circuit simulation tool (Multisim Live, an open source Tool, with limited features of Multisim by National Instruments). Figure 2, has the four term Operational Amplifier adder implementation of the Fourier series representation of the periodic square wave [15]. The students have knowledge of the use of operational amplifier as an adder, and have no knowledge about Fourier series. Students observe the output of the adder with one term, two term, three term, and four terms. They are asked to guess the fifth term, and implement the fifth term and observe the output. They are then asked to predict the output, in the case of having infinite number of terms. The whole class guesses the series tends to a square wave. Figure 3 has the Fourier series representation of the periodic saw-tooth waveform [16]. Once again, students need to predict the next term in the series, implement and observe the output. Here, we need to recognize, that it is not possible to implement this experiment using discrete components, as there is need to have three to five signal generators, all with phase coherence—hence, the essential need to use simulation tools for this engineering concept. In addition, to truly represent the Fourier series with large number of terms, students implement the code for a given equation (using open source Python from Google). Figures 4, 5, and 6 have examples of the code and the corresponding output for summation of sinusoidal harmonics for the periodic square wave, the saw-tooth wave and the triangular waveform [17]. The advantage of the programming environment is the ease of converting the mathematical equation to a suitable code, and observing the output for any desired number of harmonics. Students implement the representation of periodic signals like the half-wave rectified, full-wave rectified,

A Pedagogy for Engineering Concepts Focusing on Experiential Learning

87

Table 1 Sample question to assess Fourier series

saw-tooth wave, rectangular wave and the impulse train. Here, the focus is on developing the Python for a given mathematical equation (which is the Fourier series representation of the periodic wave), without any knowledge in Fourier series. This programming experience results in student realizing that periodic signals can be generated by the addition of sinusoidal harmonics. To further strengthen the Fourier series concept, the laboratory experience includes passing the periodic signal through a fourth order Butterworth filter of cut-off frequency 1 kHz and gain 2.53, as shown in Fig. 7. The equipment used is the

88

K. Buddhi

(a)

(b)

Fig. 2 The Four Term Fourier Series representation of square wave on Multisim Live: a the OPAMP adder, and b the corresponding output

(a)

(b)

Fig. 3 The Four Term Fourier Series representation of saw-tooth wave on Multisim Live: a the OP-AMP adder, and b the corresponding output

(a)

(b)

Fig. 4 a The Python Code for the Fourier Series representation of square wave, considered up to the 11th harmonic; and b the corresponding output.

Lab Volt Power Supply 9401-05 along with Lab Volt Dual Function Generator 940200. The signals are observed on the Digital Storage Oscilloscope, Tektronix TDS 2004B; Four Channel 60 MHz. Figure 8a, has the input and output of the filter for input triangular frequency being 900 Hz. In this case, it can be observed that the

A Pedagogy for Engineering Concepts Focusing on Experiential Learning

(a)

89

(b)

Fig. 5 a The Python Code for the Fourier Series representation of saw-tooth waveform, considered up to the 10th harmonic; and b the corresponding output.

(a)

(b)

Fig. 6 a The Python Code for the Fourier Series representation of the triangular waveform, considered up to the 10th harmonic; and b the corresponding output.

output has only the fundamental component since the filter cut-off is 1 kHz. On the other hand, Fig. 8b has the input–output when the input triangular wave is of frequency 50 Hz. In this case, close 20 harmonics as passed by the filter, and hence the output resembles the input. Figures 9 and 10 have similar experimental results for the periodic saw-tooth waveform and the square-wave. Hence, for the Fourier series concept, students have been introduced to the laboratory experience using the circuit simulation of Multisim Live, the python code and the implementation using discrete components. After this laboratory experience, the students are then introduced to the mathematic equation, the proof and numerical examples. Hence, the revised pedagogy is shown in Fig. 11, and once again involves

90

K. Buddhi

(a)

(b)

Fig. 7 a The Experimental set-up to pass the periodic signal through the Fourth order Butterworth Filter; b the circuit implementation of the Fourth order Butterworth Filter with cut-off 1 kHz, and gain 2.53.

(a)

(b)

Fig. 8 The input triangular wave (Channel-1, in Yellow), and the corresponding output waveform (Channel-2 Blue) of the Fourth order Low-Pass Butterworth filter with cut-off frequency 1 kHz, and dc-gain of 2.53 for: a the input triangular wave of frequency 900 Hz and b the input triangular wave of frequency 50 Hz

five steps, but with the laboratory experience first. This changed pedagogy, resulted in enhanced student learning. In the example we have considered, the typical pedagogy resulted in only 5% of the students comprehending the concept of Fourier Series, while with the proposed pedagogy of experiential learning, the percentage of students comprehending the concept enhanced to 60%.

2.2 Pedagogy for Sampling Theorem The sampling theorem is another concept that is abstract, highly mathematical and few students appreciating and comprehending the theorem. The sampling theorem

A Pedagogy for Engineering Concepts Focusing on Experiential Learning

(a)

91

(b)

Fig. 9 The input saw-tooth wave (Channel-1, in Yellow), and the corresponding output waveform (Channel-2 Blue) of the Fourth order Low-Pass Butterworth filter with cut-off frequency 1 kHz, and dc-gain of 2.53 for: a the input saw-tooth wave of frequency 900 Hz and b the input saw-tooth wave of frequency 50 Hz

(a)

(b)

Fig. 10 The input square wave (Channel-1, in Yellow), and the corresponding output waveform (Channel-2 Blue) of the Fourth order Low-Pass Butterworth filter with cut-off frequency 1 kHz, and dc-gain of 2.53 for: a the input square wave of frequency 900 Hz and b the input square-wave of frequency 50 Hz

Fig. 11 The revised Pedagogy adopted for the concept on Fourier Series

92

K. Buddhi

states that, ‘A signal band-limited to W radians/second, can be exactly recovered from its samples, provided the rate is greater than 2W radians/second’ [12, 13]. It is this theorem that forms the basis for converting an analog continuous time signal to a discrete time signal. The Sampling Theorem is another essential concept for all communication engineers, and required concept for all engineers, as all physical signals are today represented by the equivalent digital sequence, through the initial step of sampling. The process of ideal sampling is given by Fig. 12, which involves multiplying the analog signal by the periodic impulse train. We can get back the original signal, by passing the sampled signal through a low-pass filter of suitable cut-off frequency. The analog continuous time signal has to be band-limited, and hence, prior to multiplying by a periodic signal with sampling frequency, we first pass the continuous time signal through a LPF, also known as the anti-alias filter. When the sampling frequency is correctly chosen, this entire process of sampling and reconstruction does not introduce any error. This concept cannot be explained with time domain representation. There is a need to have knowledge of Fourier series and Fourier Transform, and students are likely to have been introduced to these concepts in earlier courses, with different notations. The ideal sampling requires the periodic impulse train, which does not exist physically, and hence, all practical sampling use rectangular pulse train instead, as shown in Fig. 13. We can once again exactly recover the original signal from its samples, even in the case of natural sampling. The usual pedagogy for sampling theorem, involved five major stages: (i) the formal statement of the concept, (ii) the mathematical equation of the concept, (iii) the mathematical proof of the concept, (iv) numerical examples to apply the concept, and (v) implementation of experiments in laboratory to prove the concept, as five step process for every concept was spread over one or two weeks, as found essential. To test student learning, suitable assessments were designed. While students were shown in Fig. 14. This able to state and prove the sampling theorem, not many were able to answer assessments that involved true comprehension of the concept. Hence, there was a need to explore change in pedagogy for the concept.

Fig. 12 The process of ideal sampling, where in the sampled signal is the product of the analog signal with the continuous time periodic impulse train, and the process of recovering the original signal from its samples by passing through a Low-Pass Filter

A Pedagogy for Engineering Concepts Focusing on Experiential Learning

93

Fig. 13 The process of natural sampling, where in the sampled signal is the product of the analog signal with the continuous time periodic pulse train, and the process of recovering the original signal from its samples by passing through a Low-Pass Filter

Fig. 14 The usual pedagogy for sampling theorem

To enhance student learning, there was a need to change the pedagogy. In the changed pedagogy, students first go through the laboratory experience, related to sampling theorem. The laboratory experience included process of sampling using the two channel analog switch, as shown in Fig. 15, followed by a Butterworth filter. The message is a sinusoidal signal of 1 kHz, and hence the low-pass filter is also of cut-off 1 kHz. The equipment used is Aplab 3 MHz Multi-waveform Signal Generator (MSG3M), the Aplab Source regulated DC Power Supply LD3202S and the Digital storage oscilloscope Tektronix TDS 2004B Four Channel 60 MHz, as shown in Fig. 16. Students are not introduced to sampling theorem. They need to generate the sampled signal using the LF398, and pass the sampled signal through the Butterworth low-pass filter of various orders. For each order of the filter, students need to obtain the minimum sampling frequency at which there is good recovery of the original message signal, and also observe a poor reconstruction due to an incorrect sampling frequency.

94

K. Buddhi

Fig. 15 The process of sampling using the LF398, a two-channel analog switch, followed by a low-pass filter for recovery of the original message from its samples

(a)

(b)

Fig. 16 a Laboratory implementation of the circuit for testing sampling theorem with the signal generators, the power supply, the four channel Digital storage oscilloscope and the breadboard implementation of the b the breadboard with the components for sampling using the LF398 together with the reconstruction Low-Pass Butterworth Filter using LM741.

The results of using a first order reconstruction filter with cut-off 1 kHz and gain 2, is given in Fig. 17. In this case, it can be observed that the sampling frequency needs to be as high as 70 kHz, for reasonable recovery of the message signal. It can also be observed that when the sampling frequency is 20 kHz, there is huge error in the recovered signal. The results of using a second order reconstruction filter with cut-off 1 kHz and gain 1.56 is given in Fig. 18. In this case, it can be observed that the sampling frequency has reduced to 20 kHz. It can also be observed that when the sampling frequency is 8 kHz, there is huge error in the recovered signal. The results of using a third order reconstruction filter with cut-off 1 kHz and gain 4 is given in Fig. 19. In this case, it can be observed that the sampling frequency has further reduced to 12 kHz. It can also be observed that, when the sampling frequency is 5 kHz, there is huge error in the recovered signal. The results of using a fourth order reconstruction filter with cut-off 1 kHz and gain 2.53 is given in Fig. 20. In this case, it can be observed that the sampling frequency has further reduced to 5 kHz. It can also be observed that when the sampling frequency is 2.8 kHz, there is error in the recovered signal.

A Pedagogy for Engineering Concepts Focusing on Experiential Learning

95

(a)

(b)

(c)

Fig. 17 a The First order Butterworth filter with cut-off frequency of 1 kHz and dc gain of 2. The message signal of 1 kHz (Channel-1, in Yellow), the sampled signal (Channel-3 in Pink) and the corresponding Low-Pass recovered output (Channel-2, in Green), with: b the recovered message with the sampling frequency of 70 kHz; and c a poor reconstruction with the sampling frequency of 20 kHz.

Hence, in the revised pedagogy for sampling theorem, students are taken through the laboratory experience of engaging in sampling process, and recovery of the message from its samples. Additional experiments to observe the effect of the duty cycle of the square-wave are also included. Later, in the course on Digital Signal processing, the sample experiment is repeated using digital filters, and the Python programming environment. The revised pedagogy is shown in Fig. 21. After this laboratory experience, the students are then introduced to the mathematical equation, the proof and numerical examples. This changed pedagogy, resulted in enhanced student learning. In the example we have considered, the typical pedagogy resulted in only 10% of the students comprehending the concept of sampling theorem, while with the proposed pedagogy of experiential learning, the percentage of students comprehending the concept enhanced to 75%.

96

K. Buddhi

(a)

(b)

(c)

Fig. 18 a The Second order Butterworth filter with cut-off frequency of 1 kHz and dc gain of 1.56. The message signal of 1 kHz (Channel-1, in Yellow), the sampled signal (Channel-3 in Pink) and the corresponding Low-Pass recovered output (Channel-2, in Green), with: b the recovered message with the sampling frequency of 20 kHz; and (c) a poor reconstruction with the sampling frequency of 8 kHz.

2.3 Pedagogy for Central Limit Theorem The pedagogy for introducing Central limit theorem, includes students implementing repeated convolution of any probability distribution function, and observing that it tends to Gaussian. The code is implemented using the open source Python from Google Colaboratory [18]. In Fig. 22, we have the results for Uniform distribution. Similarly, we have the results for exponential distribution in Fig. 23. Students are then encouraged to generate any arbitrary probability distribution function, and repeatedly convolve and observe the results, which tends to Gaussian. This is then followed by students being asked to generate sequence, whose repeated convolution does not lead to Gaussian. One such example is provided in Fig. 24. After this laboratory experience, students are then introduced to the Central limit theorem.

A Pedagogy for Engineering Concepts Focusing on Experiential Learning

97

(a)

(b)

(c)

Fig. 19 a The Third order Butterworth filter with cut-off frequency of 1 kHz and dc gain of 4. The message signal of 1 kHz (Channel-1, in Yellow), the sampled signal (Channel-3 in Pink) and the corresponding Low-Pass recovered output (Channel-2, in Green), with: b the recovered message with the sampling frequency of 12 kHz; and (b) a poor reconstruction with the sampling frequency of 5 kHz.

3 Results of Student Survey Students are the beneficiaries of any improved pedagogy. Hence, their opinion becomes an important component in arriving at a decision. With this in view, a survey was launched on April 20, 2023, and was open to all students of the second, third and final year. The survey was through the Google form, titled, ‘A comparison of Teaching methodology for Engineering Concepts,’ and was open for two days (https://forms.gle/74QnMHX7qmU92YRd6). With the class strength being close to 60, the survey was shared with close to 180 students. However, only 102 students took up the survey, as indicated in Fig. 25. The survey included details about the student (name and Roll number), courses taken, and programming tools introduced. It is interesting to note that 88% of the students expressed the opinion that the theory and the laboratory component shall be handled by the same faculty. One of the questions was with the specific example pertaining to the method to be adopted for Fourier series, as indicated in Fig. 26. It can be observed that close to 64% of the students preferred the experiential learning,

98

K. Buddhi

(a)

(b)

(c)

Fig. 20 a The Fourth order Butterworth filter with cut-off frequency of 1 kHz and dc gain of 2.53. The message signal of 1 kHz (Channel-1, in Yellow), the sampled signal (Channel-3 in Pink) and the corresponding Low-Pass recovered output (Channel-2, in Green), with: b the recovered message with the sampling frequency of 5 kHz; and c a poor reconstruction with the sampling frequency of 2.8 kHz.

Fig. 21 The revised pedagogy for sampling theorem

A Pedagogy for Engineering Concepts Focusing on Experiential Learning

(a)

99

(b)

Fig. 22 a The Python code to generate and repeatedly convolve the Uniform distribution, and b the corresponding output that gradually tends to Gaussian distribution.

(a)

(b)

Fig. 23 a The Python code to generate and repeatedly convolve the Exponential distribution, and b the corresponding output that gradually tends to Gaussian distribution.

wherein the faculty conducts the experiment to help comprehend the concept, while only 34% of the students preferred the conventional approach of handling the theory. Another specific question in the survey was regarding the faculty designing the experiment, with students observing and making a conclusion, which leads to the intended engineering concept; or, the usual method where the theory is introduced and then supported by the laboratory experience either by the same faculty or a different

100

K. Buddhi

(a)

(b)

Fig. 24 a The Python code to repeatedly convolve an arbitrary sequence, and b the corresponding output that does not tend to Gaussian distribution.

Fig. 25 Student participation statistics for the survey on: A comparison of Teaching methodology for Engineering Concepts

faculty. It is interesting to observe that, 90% of the students agreed to the proposed method of pedagogy for engineering concepts, as shown in Fig. 27. Hence, the proposed method of experiential learning can be explored for engineering concepts. In addition, few questions included the option for students to suggest teaching styles. Few of the suggestions are: (i) to avoid the use of PPTs for teaching; (ii) to develop concept videos and make the same available on YouTube; (iii) teaching without bringing text-books to the classroom; (iv) theory and practical should be handled by the same faculty, and should be conducted on the same day; and (v) teaching by giving real examples or live examples.

A Pedagogy for Engineering Concepts Focusing on Experiential Learning

101

Fig. 26 Response to the survey question on experiential learning or the conventional approach

Fig. 27 It can be observed that 90% of the students preferred the faculty to take them through an inducting learning experience

4 Conclusions The proposed pedagogy for engineering concepts requires the faculty to design set of experiments that take the students through the process of comprehending the concept, leading students to summarize the observations that are actually the intended engineering concept. In this work we have focused on the concept of Fourier series, Sampling Theorem and the Central limit theorem. We have included access to the Python code and circuits simulated on the Multisim live platform.

102

K. Buddhi

We have implemented the proposed method for other engineering concepts like: Frequency Response and Impulse Response/Step response of a circuit. The proposed method does not require any additional infrastructure or facility. It requires the faculty handling the course to design and conduct the experiment. Since the small change in pedagogy has resulted in enhanced student learning, we feel that this method of pedagogy for engineering concepts can be explored and implemented wherever possible. The proposed pedagogy has resulted in the collapse of the rigid boundary between the theory and laboratory sessions, and requires course faculty designing and handling the labs. The ultimate goal is to see that the curriculum has courses, and courses can have theory/tutorial/laboratory sessions, as found essential by the course faculty, and hence the collapse of the existing rigid boundaries between the theory component, the tutorial component and the laboratory component. This curricular structure enables the implementation of experiential learning for most engineering concepts of courses in the curriculum. This pedagogy when adopted for most engineering concepts in the curriculum, shall lead to effective implementation of OBE, and hence the development of Student Outcomes/Program Outcomes defined by ABET and NBA.

References 1. International Engineering Alliance, Washington Accord. https://www.ieagreements.org/acc ords/washington/. Accessed 08 June 2023 2. Accreditation Board for Engineering and Technology (ABET), ISO 9001 non-profit organization that accredits Universities and programs. https://www.abet.org/. Accessed 08 June 2023 3. National Board of Accreditation (NBA), for promoting International quality standards for Technical Education in India. https://www.nbaind.org/. Accessed 08 June 2023 4. Criteria for accrediting Engineering Programs (Academic year 2023–24), by ABET Engineering Accreditation Commission. https://www.abet.org/wp-content/uploads/2023/03/23-24EAC-Criteria_FINAL2.pdf. Accessed 08 June 2023 5. Manual for Accreditation of Undergraduate Engineering Programs (Tier-I Institutions), by NBA. https://www.nbaind.org/files/NBA_UGEngg_Tier_I_Manual.pdf. Accessed 08 June 2023 6. Felder, R.M.: Learning and teaching styles in engineering education. Eng. Educ. 78(7), 674–681 (1988) 7. Felder, R.M., Brent, R.: Designing and teaching courses to satisfy the ABET engineering criteria. J. Eng. Educ. 92(1) (2003). https://doi.org/10.1002/j.2168-9830.2003.tb00734.x 8. Felder, R.M., Brent, R.: Cooperative Learning. ACS Symposium Series, vol. 970, pp. 34–53 (2007). Active Learning. https://doi.org/10.1021/bk-2007-0970.ch004 9. Kanmani, B.: Introducing signals and systems concepts through analog signal processing first. In: IEEE Signal processing society: 14th DSP Workshop & 6th SPE Workshop, Enchantment Resort, Sedona, Arizona, 4th –7th January 2011, pp. 84–89 (2011). https://doi.org/10.1109/ DSP-SPE.2011.5739191 10. Haykin, S.: An Introduction to Analog and Digital Communications. Wiley (2001) 11. Haykin, S.: Communication Systems. Wiley (1983) 12. Taub, H., Schilling, D.L.: Principles of Communication Systems. McGraw-Hill Book Company (1986) 13. Bruce, C.A.: Communication Systems. McGraw-Hill Book Company (1986)

A Pedagogy for Engineering Concepts Focusing on Experiential Learning

103

14. Wage, K.E., Buck, J.R., Wright, C.H.G., Welch, T.B.: The signals and systems concept inventory. IEEE Trans. Educ. 48(3), 448–461 (2005). https://doi.org/10.1109/TE.2005.849746 15. Fourier series circuit simulation for square-wave shared during Pre-conference Workshop (online) on, ‘Analog Signal Processing concepts using Python and Multisim Live’, by Kanmani Buddhi, during International Conference on Remote Engineering and Virtual Instrumentation, REV 2022 Egypt, Cairo. https://www.multisim.com/content/ceWqGPWBmXV6HCif7S GoJY/rev-2022-fourier-series-square-wave/open/. Accessed 08 June 2023 16. Fourier series circuit simulation for saw-tooth wave shared during Pre-conference Workshop (online) on, ‘Analog Signal Processing concepts using Python and Multisim Live’, by Kanmani Buddhi, during International Conference on Remote Engineering and Virtual Instrumentation, REV 2022 Egypt, Cairo. https://www.multisim.com/content/vXxcUHEsid79foyrZo6Sr9/rev2022-saw-tooth-wave/open/. Accessed 08 June 2023 17. Python code for generating Fourier series of continuous time periodic signals using the Google colaboratory. https://colab.research.google.com/drive/124hC-FRVNmohxbYks SSkE0S2ELX2asaK?usp=sharing. Accessed 07 June 2023 18. Python code for performing Repeated convolution of signals, using the Google colaboratory. https://colab.research.google.com/drive/1RXAfF-lrglNUJkB7F-W8eeod5lYrN4FM? usp=sharing. Accessed 07 June 2023

Exploring Differences in Work Environment and Work Engagement as Moderated by Psychological Capital Rebekah L. Clarke

Abstract The primary purpose of this quantitative prospective causal-comparative study was to determine if and to what extent there is a difference in the overall work engagement of instructional designers who are either working predominantly at home or predominantly in the office in the United States, and the secondary purpose of this study was to examine the moderating effect of psychological capital on the predictive relationship between work environment and work engagement. The theoretical foundations were work engagement and psychological capital. Based on a sample size of 345, the results illustrate that the work engagement scores for those who work predominantly in the office (mean rank = 221.89) were statistically significantly higher than those who work predominantly at home (mean rank = 122.67, U = 23,431.50, z = 9.25, p < .001) but did not indicate that the interaction effect between types of work environment and psychological capital on work engagement was statistically significant. (B = −0.04, se[HC3] = 0.07, p = 0.58). The results extend research on work engagement by providing evidence that there is a statistically significant difference in mean ranks of work engagement scores between those who worked predominantly at home and those who worked predominantly in the office. Keywords Work environment · Work engagement · Psychological capital · Instructional designers

R. L. Clarke (B) Holman/Clarke Group, Dayton, OH 45458, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_9

105

106

R. L. Clarke

1 Introduction 1.1 Background of the Study With the growth of the information age and advancements in technology, virtual work is becoming more pervasive in the business world. According to Gallup [16], there are 100 million people working full-time, but only one-third of those workers are engaged at work. As part of that workforce, there are approximately 4.7 million people who telecommute, which is a number that has increased 44% during the last five years [29]. As a result of the coronavirus pandemic which began in 2020, 35.2% of employees in the United States transitioned to remote working, thereby, increasing the total percentage of employees working from home to approximately 50% [9]. In 2018, a Gallup [17] report indicated that organizational leaders need to understand how workers respond to the changes and demands of developing business realities, including learning more about the psychological needs of workers. With more of the workforce moving to a virtual work environment, it is critical that these employees are effectively engaged. The coronavirus pandemic created a demand for organizations to speed up the adoption of virtual work. Stay-at-home orders and social distancing recommendations created changes in how technology was used, increasing the number of virtual conferences and teams collaborating virtually [28]. The pandemic also prompted shifts from traditional office-based work to working from home with virtual interactions, making face-to-face communication less of the norm in many organizations [28]. Belzunegui-Eraso and Erro-Garcés [8] posited that virtual work may become more of a norm after the coronavirus pandemic risk has lessened, as organizational leaders begin to fully understand the effects of virtual working on the workplace. Research has shown that little is known about what impact the extent of time spent working virtually has on employee outcomes [18]. The increased number of employees working virtually due to the global pandemic intensifies the need to gain a better understanding of the relationship between work environment and employee outcomes, such as work engagement.

1.2 Review of Literature The role of the instructional designer within the learning and development function has increased in importance with the growth of digital technologies and the onset of the coronavirus pandemic in 2020. Engaging instructional designers is critical, as new job demands remain high and their personal and psychological resources are taxed through managing work-life balance and adjusting to new workplace policies [10]. The first goal of this study was to assess the difference in work engagement in instructional designers who are either working predominantly at home or predominantly in the office in the United States. As the coronavirus pandemic continues

Exploring Differences in Work Environment and Work Engagement …

107

into 2021, heavy demands still weigh down on instructional designers, potentially impacting their work engagement. Work engagement was shown to have a positive influence on work outcomes. Previous studies on work engagement indicated that when an employee is engaged, their performance will increase [4, 24]. Engagement was also shown to have a positive influence on organizational outcomes and employee well-being [14, 40]. Alessandri et al. [2] posited that work engagement is the motivational process whereby latent qualities, like psychological capital, are transformed into useful and positive organizational behaviors. Psychological capital has a positive relationship with engagement and strengthens engagement levels [14]. Engaged employees were shown to use positive organizational behaviors, like psychological capital, as coping tools to help them manage a dynamic virtual work environment [13] and demanding situations [27]. There is an urgency to understand the differences in work engagement of instructional designers based on their work environment. Due to the global pandemic and recent shutdowns, the expanded number of employees working virtually increases the urgency to understand what impact the extent of time spent working virtually has on employee outcomes [18]. To ensure a work environment conducive to engaging employees, organizational leaders should develop work policies with an eye on the future direction of work [33]. Bakker [3] suggested that human resources leaders should assess which job demands and resources need to be addressed to provide deeper enrichment to employees, and then policies and practices should be developed around those findings. One such personal resource is psychological capital. Psychological capital, as a personal resource, could be a powerful tool in helping employees manage stressful situations and develop work engagement. Psychological capital could be drawn upon to help one cope and focus in difficult and demanding circumstances or environments [1]. Some personal resources were shown to positively relate to work engagement and are vital for maintaining success in changing work environments. Further research was recommended to understand how other aspects of one’s personal resources, specifically psychological capital, relate to work engagement in a sample of knowledge workers [39]. The relationship between work engagement and psychological capital is valuable to understand in the changing world of work. As a result of changes in contemporary work, research on employee engagement has begun to focus more on the value of one’s personal resources as a key mechanism for increasing work engagement in knowledge workers [39]. As the context of the team or work environment shifts to virtual roles and technology is used to mediate those roles, employees are faced with new challenges for remaining engaged [32]. Technological changes also have an impact on the psychology of employees, as these changes affect how work is done, where work is done, how teams collaborate, and how virtual teams are organized [17]. As changes occur in the workforce, the engagement and psychology of workers is impacted, so it is critical to understand what employee engagement looks like in a virtual work environment and to learn more about what motivates that engagement [32]. By working to understand differences in work engagement of instructional designers based on their work environment (working predominantly at home or predominantly in the office) and whether

108

R. L. Clarke

that relationship is moderated by psychological capital, this study could provide practical direction for leaders to consider when designing human resource practices for different work environments, conceptualizing training programs for developing psychological capital, and creating avenues to increase work engagement.

1.3 Identification of the Problem Space An identification of the problem space evolved, providing evidentiary support that this study is warranted. First, Shaik and Makhecha [32] noted that the number of people moving to the virtual workplace is increasing; however, they also stated that it is not known how the drivers of engagement change when working virtually. Second, Lazauskaite-Zabielske et al. [24] indicated that engaged employees have improved performance. However, Golden and Gajendran [18] remarked that it was not known how much time spent working virtually impacts that level of performance. Carnevale and Hatak [10] indicated that the demands placed on instructional designers have increased with the onset of the coronavirus pandemic, but there is no known research that delineates how their work engagement may be affected based on these changes. Therefore, the first research question for this study was: If and to what extent does a difference exist between types of work environment and work engagement for instructional designers?

The problem space that evolved from the review of literature also emphasized a need to understand the effect of psychological capital on work engagement in knowledge workers. Toth et al. [39] ascertained that self-efficacy, a sub-construct of psychological capital, is positively associated with work engagement in knowledge workers; however, it is not known how psychological capital, as a whole construct made up of self-efficacy, optimism, hope, and resilience, relates to work engagement in knowledge workers. Van Steenbergen et al. [40] advised that employees with a larger number of personal resources, such as psychological capital, are better equipped to cope with shifting demands and that their engagement levels remained stable, but there is no known research outlining how those outcomes are manifested in different work environments. Therefore, the second research question for this study was: To what extent is there a moderation effect by psychological capital on the predictive relationship between types of work environment and work engagement for instructional designers?

1.4 Theoretical Framework The intent of this study was to expand the empirical evidence regarding the differences in the overall work engagement of instructional designers based on their work

Exploring Differences in Work Environment and Work Engagement …

109

Fig. 1 Theoretical Framework

environment and the moderation effect that psychological capital has on that relationship, supporting the concepts and models in this section. Therefore, two compelling theoretical foundations were identified to support this study. The first is Bakker et al. [5], with their work engagement theory, and the second is Luthans et al. [25], with their psychological capital theory. Figure 1 illustrates how these theoretical foundations are entwined and are most appropriate to support this study. As displayed, the hypotheses for this study suggest that the type of work environment an individual works in may affect their work engagement and that relationship may be moderated by the psychological capital of that individual. Work Engagement Theory. The first model providing theoretical support for this study is work engagement theory. Work engagement is described as “a positive, fulfilling, work-related state of mind that is characterized by vigor, dedication, and absorption” [30, p. 74]. Vigor is described as working hard, where an individual is willing to put effort into work and stay focused, even when faced with challenges [30]. Dedication extends beyond the idea of involvement and is described as having a sense of meaning, eagerness, inventiveness, enjoyment, and being challenged through one’s work [30]. Absorption is characterized by being focused and occupied in the work being done that one loses track of time and finds it difficult to disconnect from the work to leave and do other things [30]. Schaufeli et al. [30] argued that, while engagement is the positive antipode to burnout, it is a separate, distinct concept, and therefore cannot be measured on a burnout scale, leading to the development of the Utrecht Work Engagement Scale (UWES). Prior research pertaining to work engagement has demonstrated that maintaining work engagement requires a balance of job demands and resources. Bakker et al. [7] stated that engaged individuals often feel happy and enthusiastic, tend to have better health, and build their own job and personal resources. The ideas of “job” and one’s “personal resources” come from the job demands-resources model of work engagement [5], which identifies two sets of working conditions that impact an employee’s work engagement: job demands and job resources [6]. Job resources are “those physical, psychological, social, and organizational aspects” that could be used to address job demands, attain work goals, and encourage professional development

110

R. L. Clarke

[6, p. 275]. According to Bakker et al. [6], job resources, such as the ability to choose one’s own work environment, enabled work engagement and were better predictors of engagement than job demands. Psychological Capital Theory. The second theory providing foundational support for this research is the theory of psychological capital. Psychological capital is rooted in positive organizational behavior. Luthans et al. [26] defined psychological capital as: An individual’s positive psychological state of development that is characterized by: (1) having confidence (efficacy) to take on and put in the necessary effort to succeed at challenging tasks; (2) making a positive attribution (optimism) about succeeding now and in the future; (3) persevering toward goals and, when necessary, redirecting paths to goals (hope) in order to succeed; and (4) when beset by problems and adversity, sustaining and bouncing back and even beyond (resiliency) to attain success [26, p. 2]. One’s personal resources, such as psychological capital, could have a positive impact on engagement levels. Work engagement was described as the positive state an employee is in while working while psychological capital is identified as the dynamic and positive personal resource of that individual used to maintain an engaged state [19]. Du Plessis and Boshoff [14] found that employees with higher levels of work engagement often had more personal resources, such as self-efficacy, optimism, and resilience. Van Steenbergen et al. [40] suggested that employees with a larger number of personal resources, such as psychological capital, may be better equipped to cope while staying engaged, which is particularly pertinent as the 2020 global pandemic has brought on massive organizational changes. The theory of psychological capital provides theoretical support for the moderating variable of psychological capital.

2 Results The appropriate statistical tests were confirmed for use in answering the research questions for this study through the testing of the assumptions. To confirm the data were an appropriate match for each proposed statistical test, a series of assumptions for each test were conducted. Once the appropriate tests were determined and conducted, the next step was to interpret and report the results. The results reported in the following paragraphs were used to inform the discussion on the findings of the study as well as practical implications and future recommendations. The first research question was designed to measure the difference in overall work engagement scores between instructional designers who work predominantly at home or predominantly in the office. A Mann–Whitney U test was run to answer research question one. The distributions of work engagement scores for those who work predominantly at home and those who work predominantly in the office were not similar, as shown in Fig. 2. Due to the distributions not having the same shape, the data was described using mean ranks instead of median values.

Exploring Differences in Work Environment and Work Engagement …

111

Fig. 2 Population Pyramid Chart for Work Engagement

The results of the Mann–Whitney U test are contained in Table 1. The results illustrate that there was a statistically significant difference in work engagement scores between groups. The work engagement scores for those who work predominantly in the office (mean rank = 221.89) were statistically significantly higher than those who work predominantly at home (mean rank = 122.67, U = 23,431.50, z = 9.25, p < 0.001). Therefore, the null hypothesis was rejected. The second question was designed to assess the moderating effect that psychological capital could have on the predictive relationship between types of work environment and work engagement for instructional designers who work predominantly at home or predominantly in the office. Despite issues of heteroscedasticity and multicollinearity, moderated multiple regression test using the Hayes PROCESS macro was sufficiently robust to yield valid results to answer research question two, as shown in Table 2. While interpreted with caution, the results of the moderated regression analysis were significant — F (3, 336) = 1042.64, p = 0.000, R2 = 0.82 — indicating that approximately 82% of the variance in work engagement is explainable by type of work environment and psychological capital. Type of work environment significantly predicted work engagement (B = 0.47, se[HC3] = 0.09, p < 0.001), Table 1 Independent-Samples Mann–Whitney U Test Summary

Statistic Name

Statistic

Total N

345

Mann–Whitney U

23,431.50

Wilcoxon W

38,831.50

Test Statistic

23,431.50

Standard Error

925.40

Standardized Test Statistic

9.25

Asymptotic Sig. (2-sided test)

0.000

112

R. L. Clarke

Table 2 Moderated Multiple Regression Analysis (Psychological Capital Moderating the Relationship Between Type of Work Environment and Work Engagement) Variables

B

se(HC3)

t

p

CI

Work Engagement (Y)

4.13

0.05

91.00

0.000

[4.04, 4.22]

Type of Work Environment (X)

0.47

0.09

5.18

0.000

[0.29, 0.65]

Psychological Capital (W)

1.02

0.04

28.83

0.000

[0.95, 1.09]

Intercept (XW)

−0.04

0.07

−0.56

0.580

[−0.18, 0.10]

Note. CI is at the 95% confidence level

conditional on psychological capital = 0. The conditional effect of psychological capital on work engagement was also positive and significant (B = 1.02, se[HC3] = 0.04, p < 0.001), conditional on type of work environment = 0. The interaction between type of work environment and psychological capital was not statistically significant (B = −0.04, se[HC3] = 0.07, p = 0.58) in the model, suggesting that psychological capital was not a significant moderator on the predictive relationship of type of work environment and work engagement, accounting for less than 1% of the variation in work engagement. Therefore, the null hypothesis was failed to be rejected.

3 Discussion In conducting this study, the defined problem space in the literature was addressed by examining the differences between instructional designers who work predominantly at home or predominantly in the office in the United States and whether that relationship is moderated by psychological capital. There are two fundamental conclusions drawn from the results of this study. The first is that there is a difference in work engagement between those instructional designers who work predominantly at home and those who work predominantly in the office. Second, psychological capital was not a significant moderator of the effect of type of work environment on work engagement, accounting for less than 1% of the variation in work engagement. The first research question was designed to measure the difference in overall work engagement scores between instructional designers who work predominantly at home or predominantly in the office. The primary conclusion that can be drawn from these results is that there is a difference in work engagement between those who work predominantly at home and those who work predominantly in the office. This finding is supported by Hayes et al. [20], who noted that engagement is affected differently by those who work in virtual roles, and ten Brummelhuis et al. [38], who found a positive association between new ways of working and engagement. This finding conflicts with more recent work by de Vries et al. [12] and ter Hoeven and van Zoonen [37] indicating that teleworking has a negative effect on work engagement. de Vries et al. [12] specifically reported that neither working full-time or part-time

Exploring Differences in Work Environment and Work Engagement …

113

from home was related to work engagement in a quantitative study of 61 Dutch teleworkers, which was not supported by the findings in this study. In addition to there being a difference in work engagement between those working predominantly at home and those working predominantly in the office, the mean ranks between the two groups were quite different. The results of this study also show that work engagement was significantly higher for those working predominantly in the office (mean rank = 221.89) versus those working predominantly at home (mean rank = 122.67). As shown in Fig. 2, starting at an aggregate work engagement score of 2.0 and down, there is heavy output from those working predominantly from home (g1) and very little for those working predominantly in the office (g2). However, for aggregate work engagement scores of 5.0 and up, the opposite is true with a heavier population for those working predominantly in the office (g2) and a lesser population reflecting high work engagement scores for those working predominantly from home (g1). Essentially, this means that the higher the work engagement score, the greater the difference grew between the two groups, with a larger number of high work engagement scores for those working predominantly in the office. This finding was unanticipated but could be explained by Kang and Busser’s [21] suggestion that there are numerous contextual factors to consider even though work environment may nurture work engagement. Furthermore, Kulikowski [23] posited that work engagement may not be an all-inclusive concept but suggested instead that varied work contexts may influence the makeup of work engagement. At the time of data collection for this study, many organizations were transitioning back to working in the office after coronavirus pandemic restrictions were lifted in mid-May 2021 [38]. This could have been a motivator in terms of how the sample responded to the survey questions. The second question was designed to assess the moderating effect that psychological capital could have on the predictive relationship between types of work environment and work engagement for instructional designers who work predominantly at home or predominantly in the office. The primary findings did not show a statistically significant interaction effect between type of work environment and psychological capital on work engagement (B = −0.04, se[HC3] = 0.07, p = 0.58). This outcome conflicts with similar findings in previous research. Du Plessis and Boshoff [14] found that psychological capital both mediated and moderated the relationships between authentic leadership and work engagement in their cross-sectional quantitative study of 647 managers in a South African healthcare organization. In two separate studies of 606 and 384 high school students in the Philippines, psychological capital was shown to be associated with and to be a predictor of autonomous and controlled motivation, as well as academic engagement and achievement [11]. In their quantitative study, Xi et al. [41] found that psychological capital moderated the relationship between social support and work engagement, but not the other way around, providing support for the development of individual resources along with organizational support when looking to increase work engagement. Engaged employees were shown to use positive organizational behaviors, like psychological capital, as coping tools to help them manage a dynamic virtual work environment [13] and demanding situations [27].

114

R. L. Clarke

The results of this study corroborate the findings in one recent article. The premise for the second research question in this study is comparable to that of Van Steenbergen et al.’s [40] who suggested that employees with a larger number of personal resources, such as psychological capital, may be better equipped to cope while staying engaged, which is particularly pertinent as the 2020 global pandemic has brought on massive organizational changes. In their longitudinal quantitative study of 126 employees of a financial services provider in Holland, Van Steenbergen et al. [40] found that psychological capital did not moderate the relationship between work engagement and the transition to new ways of working. By conducting MANCOVAs, they reported that the univariate main effect for psychological capital was significant for task ambiguity but was not significant for job demands and mental demands. Although support was not found for the moderating role of psychological capital, Van Steenbergen et al.’s [40] findings did show that employees who had higher psychological capital in all three data waves also showed higher levels of engagement and autonomy compared with colleagues with lower psychological capital scores. In this study, psychological capital was also shown to have a conditional effect on work engagement. An ancillary observation in the results of the moderated multiple regression were significant relationships between type of work environment and psychological capital on work engagement, conditional on the other predictor variable being = 0. The conditional effect of psychological capital on work engagement was positive and significant (B = 1.02, se[HC3] = 0.04, p < 0.001), conditional on type of work environment = 0. This coincides with research stating that psychological capital has a positive relationship with engagement [14] and that it strengthens engagement levels [11]. When an individual has higher levels of psychological capital and their psychological needs — as defined by self-determination theory — are satisfied, they are likely to exhibit more positive organizational behaviors and may be more engaged in their work [31, 37]. Type of work environment was also shown to have a conditional effect on work engagement. In the results of the moderated regression analysis, type of work environment was also shown to significantly predict work engagement (B = 0.47, se[HC3] = 0.09, p < 0.001), conditional on psychological capital = 0. This result provides evidentiary support for the influence of an individual’s work environment on their work engagement. Shaik and Makhecha [32] noted that it is likely that employee engagement is impacted as more of the workforce confront new challenges and circumstances of a virtual work environment. Moreover, the findings in this study partially corroborate Duque et al.’s [15] conclusion that work engagement was directly and positively affected by physical working conditions. They also reported that this relationship was mediated by new ways of working, but that all facets of new ways of working did not have to be implemented to increase employee engagement. The inference drawn from their results was that investments into improvements of the physical work environment may enhance engagement more when the implementation of at least one facet of new ways of working is implemented as well. While the ancillary observations in the present study are interesting, the answer to the second research question for this study did not provide statistically significant evidence to address issues identified in the problem space.

Exploring Differences in Work Environment and Work Engagement …

115

The secondary purpose of this study was to determine if there was a significant moderation effect by psychological capital on the predictive relationship between types of work environment and work engagement for instructional designers. This focus was guided by a call for research to understand how one’s personal resources, specifically psychological capital, relate to work engagement in a sample of knowledge workers [39]. Furthermore, Carnevale and Hatak [10] indicated that the demands placed on instructional designers have increased with the onset of the coronavirus pandemic, but, prior to this study, there was no known research that delineated how instructional designer’s work engagement may be affected based on these changes. Van Steenbergen et al. [40] proposed that employees with a larger number of personal resources, such as psychological capital, are better equipped to cope with shifting demands and that their engagement levels remained stable. While the findings failed to support these concepts, the results of this study extend existing research.

4 Recommendations Based on the findings and conclusions of this study, there is opportunity for further exploration and practice. Due to the complexities of virtual work, future research could employ a qualitative method to understand the organizational context and environmental factors that influence the differences of work engagement between those who work predominantly at home and those who work predominantly in the office. Second, while the results of this study have shown there are significant differences in work engagement, there is more to be learned about why those working in the office were shown to be more engaged than those working predominantly at home. Replication of this study could determine whether the results remain true when environmental factors change. Given the timing of data collection for this study, when offices were re-opening after the coronavirus pandemic, the levels of engagement may change once work practices and job demands return to previous levels of normalcy. Third, additional research should focus on the amount of time spent working virtually. Future researchers could explore differences in work engagement by assessing time spent working virtually versus in the office by the number of days. Finally, a valuable understanding of work engagement could be gained by additional research using target populations of other types of knowledge workers, specific industries, organizations, or regions to enable better generalizability of the study results. Based on these findings, there are three recommendations for future practice for organizational leaders and human resources practitioners. First, knowing that there are differences in work engagement for employees working predominantly at home and predominantly in the office, organizational leaders and human resources practitioners should design flexible work practices in a way that accommodates the diverse needs of employees based on their type of work environment. Work contexts should be designed with the goal of creating the right fit between what employees expect for their roles and the type of work environment in which they want to work

116

R. L. Clarke

[3]. Bakker [3] also proposed that human resource managers should consider work environment designs that lessen job demands and enable job resources to proactively support work engagement. A second recommendation for future practice is for organizational leaders and human resources practitioners in the learning and development industry. Based on the results of this study, it would be prudent for learning and development department heads to allow their instructional design staff to choose the type of work environment where they feel they will be most effective. This recommendation is in line with Spivack and Woodside [35], who noted that intrinsic work motivation is likely to influence work environment choices. Individuals who have high intrinsic motivation, specifically knowledge workers, are likely to prefer and choose work environments that help their productivity [35]. Moreover, knowledge workers are also more likely to select a work environment that will have a positive influence on their productivity and well-being, particularly if they have perceived location autonomy [34]. Lastly, while this study did not show psychological capital to be a moderator of the relationship between type of work environment and work engagement, it is clear through the ancillary results of the moderated multiple regression that work environment and psychological capital each have a positive conditional effect on work engagement. Therefore, human resources practitioners should consider contextual factors influencing an individual’s work environment as well as establish programs to develop their psychological capital. Kotzé and Nel [22] posited that organizational leaders can increase an employee’s work engagement by investing in human resource practices and a work environment that enhances both job and personal resources, specifically psychological capital. When an individual has higher levels of psychological capital and that individual’s psychological needs are satisfied, that person is likely to exhibit more positive organizational behaviors and be engaged in their work [31, 37]. Acknowledgements Thank you, Dr. Lisa Rutner, Dr. Patricia D’Urso, and Dr. Brian Cicero for their guidance and support of this research.

References 1. Adil, A., Kamal, A., Shujja, S., Niazi, S.: Mediating role of psychological ownership on the relationship between psychological capital and burnout amongst university teachers. IBA Bus. Rev. 13(1), 69–82 (2018) 2. Alessandri, G., Consiglio, C., Luthans, F., Borgogni, L.: Testing a dynamic model of the impact of psychological capital on work engagement and job performance. Career Dev. Int. 23(1), 33–47 (2018) 3. Bakker, A.B.: Strategic and proactive approaches to work engagement. Organ. Dyn. 46(2), 67–75 (2017) 4. Bakker, A.B., Albrecht, S.: Work engagement: Current trends. Career Dev. Int. 23(1), 4–11 (2018) 5. Bakker, A.B., Demerouti, E.: Towards a model of work engagement. Career Dev. Int. 13(3), 209–223 (2008)

Exploring Differences in Work Environment and Work Engagement …

117

6. Bakker, A.B., Hakanen, J., Demerouti, E., Xanthopoulou, D.: Job resources boost work engagement, particularly when job demands are high. J. Educ. Psychol. 99(2), 274–284 (2007) 7. Bakker, A.B., Schaufeli, W.B., Leiter, M.P., Taris, T.W.: Work engagement: an emerging concept in occupational health psychology. Work Stress. 22(3), 187–200 (2008) 8. Belzunegui-Eraso, A., Erro-Garcés, A.: Teleworking in the context of the Covid-19 crisis. Sustainability 12(9), 3662 (2020) 9. Brynjolfsson, E., Horton, J., Ozimek, A., Rock, D., Sharma, G., & Ye, H.-Y. T.: COVID-19 and remote work: an early look at US data. National Bureau of Economic Research Working Paper Series (2020) 10. Carnevale, J.B., Hatak, I.: Employee adjustment and well-being in the era of COVID-19: implications for human resource management. J. Bus. Res. 116, 183–187 (2020) 11. Datu, J., King, R., Valdez, J.P.: Psychological capital bolsters motivation, engagement, and achievement: Cross-sectional and longitudinal studies. J. Posit. Psychol. 13(3), 260–270 (2018) 12. de Vries, H., Tummers, L., Bekkers, V.: The benefits of teleworking in the public sector: reality or rhetoric? Rev. Publ. Pers. Admin. 39(4), 570–593 (2019) 13. Diedericks, J.C., Cilliers, F., Bezuidenhout, A.: Resistance to change, work engagement and psychological capital of academics in an open distance learning work environment. SA J. Human Resourc. Manage. 17(a1142), 1–4 (2019) 14. Plessis, M.D., Boshoff, A.B.: The role of psychological capital in the relationship between authentic leadership and work engagement. SA J. Human Resourc. Manage. 16, 1–9 (2018). https://doi.org/10.4102/sajhrm.v16i0.1007 15. Duque, L., Costa, R., Dias, Á., Pereira, L., Santos, J., António, N.: New ways of working and the physical environment to improve employee engagement. Sustainability 12(17), 6759 (2020) 16. Gallup: State of the American Workforce (2017). https://www.gallup.com/workplace/257549/ state-american-workplace-report.aspx 17. Gallup: The Real Future of Work (2018). https://www.gallup.com/workplace/235709/futurework-performance-download.aspx 18. Golden, T.D., Gajendran, R.S.: Unpacking the role of a telecommuter’s job in their performance: Examining job complexity, problem solving, interdependence, and social support. J. Bus. Psychol. 34, 55–69 (2019) 19. Gupta, M., Shaheen, M., Das, M.: Engaging employees for quality of life: mediation by psychological capital. Serv. Ind. J. 39(5–6), 403–419 (2019) 20. Hayes, M., Chumney, F., Wright, C., Buckingham, M.: The global study of engagement technical report. ADP Research Institute (2018). https://www.adp.com/resources/articles-and-ins ights/articles/g/global-study-of-engagement-technical-report.aspx 21. Kang, H.J., Busser, J.A.: Impact of service climate and psychological capital on employee engagement: The role of organizational hierarchy. Int. J. Hosp. Manag. 75, 1–9 (2018) 22. Kotzé, M., Nel, P.: Job and personal resources as mediators in the relationship between iron-ore mineworkers’ job demands and work engagement. South African J. Human Resourc. Manage. 17(1), 1–9 (2019) 23. Kulikowski, K.: Do we all agree on how to measure work engagement? Factorial validity of Utrecht work engagement scale as a standard measurement tool – a literature review. Int. J. Occup. Med. Environ. Health 30(2), 161–175 (2017) 24. Lazauskaite-Zabielske, J., Urbanaviciute, I., Rekasiute Balsiene, R.: From psychosocial working environment to good performance: the role of work engagement. Baltic J. Manage. 13(2), 236–249 (2018) 25. Luthans, F., Avolio, B., Avey, J., Norman, S.: Positive psychological capital: measurement and relationship with performance and satisfaction. Pers. Psychol. 60(3), 541–572 (2007) 26. Youssef-Morgan, C.M., Luthans, F.: Psychological capital and well-being: psychological capital and well-being. Stress and Health 31(3), 180–188 (2015). https://doi.org/10.1002/smi. 2623 27. Martínez, I.M., Youssef-Morgan, C.M., Chambel, M.J., Marques-Pinto, A.: Antecedents of academic performance of university students: academic engagement and psychological capital resources. Educ. Psychol. 39(8), 1047–1067 (2019)

118

R. L. Clarke

28. Raffo, D.: World of technology changed forever by the pandemic. Storage Mag. 18(3), 1–4 (2020) 29. Reynolds, B. W.: 159% Increase in remote work since 2005: FlexJobs & Global Workplace Analytics report [Web log post]. https://www.flexjobs.com/blog/post/flexjobs-gwa-report-rem ote-growth/. 29 July 2019 30. Schaufeli, W.B., Salanova, M., Gonzalez-Roma, V., Bakker, A.B.: The measurement of engagement and burnout: a two sample confirmatory factor analytic approach. J. Happiness Stud. 3, 71–92 (2002) 31. Shah, T.A., Khattak, M.N., Zolin, R., Shah, S.Z.A.: Psychological empowerment and employee attitudinal outcomes: the pivotal role of psychological capital. Manag. Res. Rev. 42(7), 797–817 (2019) 32. Shaik, F.F., Makhecha, U.P.: Drivers of employee engagement in global virtual teams. Australas. J. Inf. Syst. 23, 1–9 (2019). https://doi.org/10.3127/ajis.v23i0.1770 33. Soares, M.E., Mosquera, P.: Fostering work engagement: the role of the psychological contract. J. Bus. Res. 101, 469–476 (2019) 34. Spivack, A.J., Milosevic, I.: Perceived location autonomy and work environment choice: the mediating influence of intrinsic motivation. J. Appl. Behav. Sci. 54(3), 325–348 (2018) 35. Spivack, A.J., Woodside, I.: Applying complexity theory for modeling human resource outcomes: antecedent configurations indicating perceived location autonomy and work environment choice. J. Bus. Res. 102, 109–119 (2019) 36. ten Brummelhuis, L.L., Bakker, A.B., Hetland, J., Keulemans, L.: Do new ways of working foster work engagement? Psicothema 24, 113–120 (2012) 37. ter Hoeven, C.L., van Zoonen, W.: Helping others and feeling engaged in the context of workplace flexibility: the importance of communication control. Int. J. Bus. Commun. 24(1), 113–120 (2020) 38. Thorbecke, C.: Employers announce COVID-19 vaccine requirements as workplaces reopen. ABC News, 17 May 2021. https://abcnews.go.com/Business/employers-announce-covid-19vaccine-requirements-workplaces-reopen/story?id=77688270 39. Toth, I., Heinänen, S., Nisula, A.-M.: Personal resources and knowledge workers’ job engagement. Int. J. Organ. Anal. 28(3), 595–610 (2020) 40. Van Steenbergen, E.F., van der Ven, C., Peeters, M.C.W., Taris, T.W.: Transitioning towards new ways of working: do job demands, job resources, burnout, and engagement change? Psychol. Rep. 121(4), 736–766 (2018) 41. Xi, Y., Xu, Y., Wang, Y.: Too-much-of-a-good-thing effect of external resource investment—a study on the moderating effect of psychological capital on the contribution of social support to work engagement. Int. J. Environ. Res. Public Health 17(2), 437 (2020)

What’s Behind the Learning Management System: Algorithmic Design in Online Learning Simone C. O. Conceição

and Lilian H. Hill

Abstract Online learning requires intentional design of learner experiences. As we use Learning Management Systems (LMS) to design instruction, there is a need for awareness of what operates behind the LMS platform. Algorithms, coded within the LMS, are used to create personalized experiences, answer common questions, and provide immediate feedback. Algorithms built into LMS can assess learners’ current knowledge level and deliver tailored instruction to facilitate students’ mastery of material. It can show students what they have accomplished and what they still need to complete. Advanced analytics allows instructors to better understand students’ learning needs and adapt their instruction to meet them. Chatbots can answer common questions quickly which reduces frustrating delay. They can also augment students’ feelings of accomplishment. Despite the benefits of algorithms, there are some dangers. Algorithms used for decision-making may seem impartial. Instead, depending on the information used to train artificial intelligence and machine learning, many algorithms take human bias and magnify discrimination. This paper considers the power of algorithms and ways to incorporate algorithmic design in online learning. We will share practical implications for designers and educators so that they maintain control over what’s behind the LMS wall to ensure that they make decisions related to students’ ability to progress in their learning more effectively. This control allows designers and educators to protect individuals’ privacy, promote inclusivity, and foster diverse learning opportunities. Keywords Algorithmic design · LMS · Online learning · AI · Machine learning

S. C. O. Conceição (B) University of Wisconsin-Milwaukee, Milwaukee, WI 53211, USA e-mail: [email protected] L. H. Hill University of Southern Mississippi, Hattiesburg, MS 39402, USA © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_10

119

120

S. C. O. Conceição and L. H. Hill

1 Introduction In this paper, we use three terms to explain what is behind a Learning Management System (LMS): algorithms, artificial intelligence, and machine learning. Algorithms are a set of problem-solving steps used by a computer program to accomplish a task; they are operationalized in Artificial Intelligence (AI), which are smart machines that perform tasks usually associated with human intelligence such as “learning, adapting, synthesizing, self-correction, and use of data for complex processing” [1], para. 3]. Machine learning is an application of AI in which large data sets are analyzed, without direct instruction, to detect patterns that would not be visible to humans. Machine learning, deep learning, and natural language processing are among the functions used by AI to provide educational services to end users [2]. These three interrelated fields focus on conceptualizing learning and teaching using technology. Designers and educators need to become knowledgeable of the boundaries among the three fields to solve complex educational challenges [3]. Some of the capabilities of AI in LMS include automation and streamlining of content management, personalized curricula and learning, enhanced student engagement, improved content accessibility, immediate user communication and knowledge assessment, and sophisticated and curated content. Machine learning enables the LMS to continue learning and improving based on experience and accumulated data patterns. Therefore, the purpose of this paper is to explore the power of algorithms and ways to incorporate algorithmic design in online learning. First, the power of algorithms is addressed. Next, ways to incorporate algorithmic design in online learning is discussed. The paper concludes with practical implications for designers and educators.

2 The Power of Algorithms Algorithms and AI have been a topic of research interest and growing use in educational settings [2]. They have multiple applications and usage. They function as intelligent tutoring systems, adaptive learning/teaching, assessment design, and learning analytics and have penetrated the educational systems of many countries in the last decade. These applications are often implemented with goals of improving university services, helping teachers offer quality education, and supporting learning. Holmes and Tuomi [4] developed a comprehensive taxonomy of AI applications in education and learning that demonstrates the current power of algorithms. The

What’s Behind the Learning Management System: Algorithmic Design …

121

taxonomy focuses on student, teacher, and institution applications. The studentfocused AI includes intelligent tutoring systems, AI-assisted apps, AI-assisted simulations, AI to support learners with disabilities, automatic essay writing, chatbots, automatic formative assessment, learning network orchestrators, dialoguebased tutoring systems, exploratory learning environments, and AI-assisted lifelong learning assistants. The teacher-focused AI includes six functions: plagiarism detection, smart curation of learning materials, classroom monitoring, automatic summative assessment, AI teaching assistants, and classroom orchestration. The institution-focused AI includes applications that support admissions and e-Proctoring. AI applications are being used to support educational functions by automating and reducing repetitive tasks and time teachers normally invest in grading and tracking student attendance by image recognition and processing; using AI facilitators to support teachers’ classroom work; and providing students with rapid, automated feedback using chatbots, and customized tutoring systems [5]. In a systematic review about AI use in Latin American higher education, SalasPilco and Yang [2] documented five categories: 1. Predictive modeling of student drop-out rates, course performance, and academic performance. 2. AI-computer assisted content analysis employed to conduct student online assessment. 3. Assistive technology to simulate conversation among students. 4. Intelligent analytics, classification systems to detect students’ readiness to progress. 5. Image analytics including facial recognition, sometimes used to take attendance. In a similar systematic review, Zawacki-Richter et al. [6] found that AI in education is actively used in academic support, institutional, and administrative services for four purposes: (a) profiling and prediction, (b) assessment and evaluation, (c) adaptive systems and personalization, and (d) intelligent tutoring systems. They noted that much of the research is characterized by a weak theoretical framework, with little attention paid to ethics of AI use in education, and an absence of critical perspectives about the risks and challenges of AI in education. Tan et al. [7] also conducted a systematic review on the use of AI for collaborative learning. Thematic analysis produced results around learning outcomes (collective performances and content of learning) and social interactions and processes (sentiments and emotions, discourse patterns and talk moves, and learner characteristic and behaviors). Researchers articulated two relevant implications to this paper: the ethical and responsible use of AI related to prescriptive actions, which needs further investigations and the need to explore in more depth the applications of diagnostic and predictive analytics. Algorithms are known for their speed, precision, and objectivity. But ultimately, they were designed by human beings and operate based on different values and objectives coded into them. Algorithms need to be scrutinized for taking important decisions out of human hands such as online learning support and outcomes. Often,

122

S. C. O. Conceição and L. H. Hill

learning designers and educators are unaware of what operates behind the LMS platform and lack the power to ensure that they make decisions related to students’ ability to progress in their learning effectively. The advantages and dangers of algorithms for LMS are discussed next.

2.1 Advantages of Algorithms for LMS Algorithms have a tremendous impact on the design of online learning, particularly as the design relates to what occurs in the background of the LMS. Algorithms are useful in automating many repetitive tasks in the LMS to facilitate effective delivery of online courses. Well-designed algorithms in an LMS provide for efficient documentation of attendance, participation, submission of assignments, grading tests, and documentation and tracking of student achievement. One advantage of the use of algorithms in an LMS platform is automated grading and interactive tutoring. When there are a finite number of responses to questions, for example in multiplechoice quizzes, algorithms are responsible for the immediate response a learner receives. Students only wait for a grade when their written essay must be assessed by a human being. Some essays are even being graded by using algorithms capable of machine learning. Intelligent tutoring is often used in language learning and for remedial support. Intelligent Tutoring Systems (ITS) provide step-by-step tutorials, customized for each learner, for mathematics or physics topics, for example. ITS provides a pathway through the learning materials and activities within the LMS [8].

2.2 The Dangers of Algorithms for LMS Although algorithms can be very useful to accomplish tasks [5], once set in motion, decision-making based on algorithmic functions often occurs without adequate human supervision. Sometimes, shortcuts are used in their programming, so the system ignores some relevant factors resulting in decisions that are inaccurate and potentially unfair and punitive. For example, essay grading may assess students’ abilities to memorize facts, or use specific vocabulary the system is programmed to recognize, but not how well they are able to question and apply information. Data collection used for large datasets needed to train AI can be invasive of people’s privacy [9]. The difficulty of understanding the LMS functions of algorithms relates to the fact that few designers and educators are equipped with knowledge of the specifics of computer coding, but coding is fundamental to the way the LMS platform operates. Algorithms are designed to achieve designated outcomes and they do that by analyzing volumes of data and detecting patterns. Potential dangers of algorithms are sourced in the following characteristics.

What’s Behind the Learning Management System: Algorithmic Design …

123

Opacity. Algorithms are composed of layers of small mathematical functions that are trained with thousands of examples. Therefore, it is difficult to examine machine learning to understand which factors contribute to decision-making processes [10]. One thing they are not designed to do is reveal their inner workings. Errors may be difficult to detect in the data analytics of an online course resulting in inaccurate grading. Errors will magnify through repetition unless close oversight is conducted. Faulty assumptions can be disguised by mathematics resulting in untested and unquestioned outcomes [11] instead of having an instructor verify learner performance. Model of Reality May be Flawed. Algorithms may be based on human prejudice, bias, and even misunderstandings. Human beings can change their thinking and transform their belief systems. Big datasets do not have a moral imagination. Therefore, algorithms, AI, and machine learning are stuck with the values coded into them, unless someone steps in to change them. An example is that if a machine grades a student essay, the meaning of the student’s subjective reflection may not be recognized. If an algorithm is in error, errors can go largely untested and unquestioned and will magnify through repetition unless a feedback loop is built in [11]. Students’ Privacy Concerns. Student data are being collected in LMS raising trust issues regarding how student privacy is protected [12]. The question of what people have access to information behind the LMS regarding student performance, achievement, and conduct can be concerning. If more people than the student and instructor are involved, the potential for privacy breaches can increase. For example, staff members providing instructional technology support, student retention services, student athletic tutoring, and emergency alert systems may all have access to student information and course performance. The more people who are involved, the higher the potential for breach of privacy.

3 Ways to Incorporate Algorithmic Design in Online Learning We suggest five ways to incorporate algorithmic design in AI-based LMS: customized content, curriculum automation, AI-powered assessment, real-time student support, and personalized learning. These approaches predict learners’ behavior and provide personalized learning tasks.

3.1 Customized Content in the LMS AI-based LMS can keep track of a student’s prior content learning and provide new material based on the learner’s pattern of knowledge. This function can detect

124

S. C. O. Conceição and L. H. Hill

learner weaknesses in understanding and present suitable content and tasks to meet students’ learning needs. Conversely, algorithms can recognize when a student is more advanced and automatically skip tasks to adapt the content to the learner’s knowledge level.

3.2 Curriculum Automation AI-based LMS can reduce human labor needed to design online content and activities. Most LMS require the creation of courses with modules built by designers and educators and placed manually in the LMS with constant updating. The AI-based LMS can alleviate this time-consuming task as the AI can develop the course content and activities independently.

3.3 AI-Powered Assessment AI-powered assessment can serve as a labor- and time-saving device for instructors, but it is important not to rely solely on automatic results. Human oversight is still required. Caution must be used when implementing algorithmic-based automatic grading due to individual student’s characteristics and needs. Instructors can use AI-based formative assessment to analyze data on students’ learning [8].

3.4 Real-Time Student Support AI can provide immediate feedback to learners, saving time and optimizing the learning process. In this case, chatbots play an important role in increasing the quality and speed of learning of students. AI can also keep track of students’ learning progress by sending reminders about what students have completed and have yet to complete. This automated knowledge check makes the learning process more accurate and efficient.

3.5 Personalized Learning A personalized course in the LMS can be more appealing to learners than a generalized course because it is more relevant to the learner. The learner may feel the course was tailor-made to their needs because it can avoid repeating content and learning tasks the student has already mastered.

What’s Behind the Learning Management System: Algorithmic Design …

125

4 Practical Implications for Designers and Educators Designers and educators have used LMS manually to create learning experiences for students. However, with the advancement of AI-based LMS, this practice may change. This can be a blessing and a curse for online learning design. The blessings include customized content, curriculum automation, real-time student support, and personalized learning. AI-based LMS can significantly increase the quality of a user’s learning experience and make it more effective and efficient thanks to the personalized approach for each student. AI-based LMS can customize learning content and skill development based on learner progress to achieve goals in an efficient way. AI-based LMS automation may reduce time to task for students and personalize their learning resulting in increased learner motivation. Unfortunately, AI-based LMS can affect human decision-making. Coding within AI and machine learning may reflect the values coded into the platform unless a human being takes action to correct them. Designers and educators may be unaware of what goes inside the code. It is very difficult to create personalized learning within an LMS because the subjective learning experience may not be accounted for in the design. AI-based LMS collects private information of a student’s journey through the course. This information may be at risk if the coding is not carefully designed to protect students’ privacy. Because of the dangers embedded in algorithms, there is a need for oversight and regulation. UNESCO [8] and the Council of Europe [13] have recognized some of the dangers of algorithms and produced statements. The United States White House Office of Science and Technology Policy prepared a Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People [14]. This blueprint begins with the following statement [14], para. 1]: Among the great challenges posed to democracy today is the use of technology, data, and automated systems in ways that threaten the rights of the American public. Too often, these tools are used to limit our opportunities and prevent our access to critical resources or services.

The blueprint advocates five principles that we relate to algorithmic design:

4.1 Safe and Effective Systems The LMS should be developed with input from diverse communities, stakeholders, and domain experts. They should be pre-tested prior to deployment for risk identification and mitigation, and ongoing monitoring.

126

S. C. O. Conceição and L. H. Hill

4.2 Algorithmic Discrimination Protections Designers and educators should take proactive and continuous measures to protect learners from algorithmic discrimination and to use and design LMS in an equitable way.

4.3 Data Privacy Learners should be protected from abusive data practices and should retain agency over how their information is used and disseminated.

4.4 Notice and Explanation Educators and learners should be informed that the LMS is an automated system and educated about how and why it contributes to outcomes that impact teaching and learning.

4.5 Human Alternatives, Consideration, and Fallback Learners should be able to opt out, where appropriate, and have access to a support person who can quickly consider and remedy problems they encounter.

4.6 Other Guidelines and Policies Other countries have also developed guidelines and policies concerning algorithms that can be applied to the oversight of LMS. Canada proposed the Digital Charter Implementation Act in 2022. The Canadian government promotes an Algorithms Impact Assessment Tool that determines the impact level of an automated decisionsystem. It is composed of 48 risk and 33 mitigation questions and is intended to “help departments and agencies better understand and manage the risks associated with automated decision systems.” The European Union proposed the Artificial Intelligence Act, the United Kingdom has the Algorithmic Transparency Standard, and China has proposed legislation [15].

What’s Behind the Learning Management System: Algorithmic Design …

127

5 Conclusion Algorithms have more influence than we know. They are embedded in the LMS platform used to design online courses. Algorithms are highly efficient at automating laborious functions and have permeated many aspects of education, but their uses can have unpredictable consequences. Algorithmic design requires awareness, knowledge, values, and ethics of care. It must be noted that the ethics of educational data and learning analytics are still in the early days of development and little agreement about educational ethical standards has been established [9]. Designers and educators need to become knowledgeable of algorithmic literacy; be cognizant of both the benefits and the potential problems with algorithms, artificial intelligence, and machine learning in LMS; and find ways to engage in algorithmic design practices that support effective learning, maintain students’ privacy, maximize the benefits, and minimize the dangers.

References 1. Popenici, S.A.D., Kerr, S.: Exploring the impact of artificial intelligence on teaching and learning in higher education. Rese. Pract. Technol. Enhanc. Learn. 12, 22 (2017) 2. Salas-Pilco, S.Z., Yang, Y.: Artificial intelligence applications in Latin American higher education: a systematic review. Int. J. Educ. Technol. Higher Educ. 19(1), 1–20 (2017) 3. Rienties, B., Simonson, H.K., Herodotou, C.: Defining the boundaries between artificial intelligence in education, computer-supported collaborative learning, educational data mining, and learning analytics: a need for coherence. Front. Educ. 5 (2020). https://doi.org/10.3389/feduc. 2020.00128, Accessed 13 Apr 2023 4. Holmes, W., Tuomi, I.: State of the art and practice in AI in education. Eur. J. Educ. 57(4), 542–70 (2022). https://doi.org/10.1111/ejed.12533 5. Owoc, M.L., Sawicka, A., Weichbroth, P.: Artificial intelligence technologies in education: benefits, challenges and strategies of implementation. Artif. Intell. Knowl. Manage. 599, 37–58 (2021) 6. Zawacki-Richter, O., Marín, V.I., Bond, M., Gouverneur, F.: Systematic review of research on artificial intelligence applications in higher education – where are the educators? Int. J. Educ. Technol. Higher Educ. 16(1), 39 (2019). https://doi.org/10.1186/s41239-019-0171-0 7. Tan, S.C., Lee, A.V.Y., Lee, M.: A systematic review of artificial intelligence techniques for collaborative learning over the past two decades. Comput. Educ. Artif. Intell. 3(1), 100097 (2022). https://doi.org/10.1016/j.caeai.2022.100097.y 8. UNESCO. Recommendation of the ethics of artificial intelligence [SHS/BIO/PI/2021/1] (2022). https://unesdoc.unesco.org/ark:/48223/pf0000376709. Accessed 13 Apr 2023 9. Holmes, W., Porayska-Pomsta, K., Holstein, K., et al.: Ethics of AI in education: towards a community-wide framework. Int. J. Artif. Intell. Educ. 32, 504–526 (2022). https://doi.org/10. 1007/s40593-021-00239-1 10. Dickson, B.: What makes AI algorithms dangerous? (2020). https://bdtechtalks.com/2020/06/ 10/ai-weapons-of-math-destruction/, Accessed 23 Mar 2023 11. O’Neill, C.: Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown Publishing Group, New York (2021) 12. Boudreau, E. Learning in the age of algorithms (2020). https://www.gse.harvard.edu/news/uk/ 20/01/learning-age-algorithms. Accessed 12 Mar 2023

128

S. C. O. Conceição and L. H. Hill

13. Council of Europe: The Council of Europe and Artificial intelligence [CoE4AI] (2023). https:// www.coe/int/AI. Accessed 13 Apr 2023 14. U.S. Office of Science and Technology. White House blueprint for an AI bill of rights (2022). https://www.whitehouse.gov/ostp/ai-bill-of-rights/#safe. Accessed 13 Apr 2023 15. Ferguson, C., Png, J., Whiteside, H. The regulation of artificial intelligence in Canada and abroad: Comparing the proposed AIDA and EU AI Act. Fasken (2022). https://www.fasken.com/en/knowledge/2022/10/18-the-regulation-of-artificial-intell igence-in-canada-and-abroad. Accessed 13 Apr 2023

A Participatory Museum for Intercultural Development: Innovative Fruition of South-Asian Art Collections in Italy Luca Contardi

Abstract Museums are becoming aware of their role as facilitators of social inclusion processes. Recently the paradigm of the participatory museum has been spreading; it envisages the involvement of source communities as co-protagonists in the production of value: they thus become agents of intercultural dialogue. New technologies offer tools both for user profiling, to improve visitor experiences and educational activities, and for practices of re-appropriation of the narrative. Nevertheless, it is necessary to investigate to what extent innovative ways of heritage fruition and education centered on practices of co-design within the museum context, characterised by strong use of new technologies, can implement social inclusion and well-being of marginalised communities and the development of digital skills (and, therefore, critical thinking) fruitful to developing active citizenship according to the lifelong learning paradigm. The purpose of this paper is to illustrate the theoretical framework and the methodological premises for the development of a new paradigm of fruition of the South-Asian art collections in Italy, centered on the use of new technologies and through the involvement of the local source communities. Keywords Participatory Museum · Heritage Education · Well-being and Social Inclusion

1 State of the Art and Research Question The purpose of this paper is to illustrate the theoretical framework and the methodological premises useful for the development of a new paradigm of fruition of the South-Asian art collections in Italy, centered on the use of new technologies and through the involvement of the source communities settled on the territory. The example that will be taken into consideration — the collections held at the MuCiv L. Contardi (B) Sapienza Università Di Roma, 00185 Roma, RM, Italy e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_11

129

130

L. Contardi

(Museum of Civilisations) in Rome — is particularly emblematic: it is currently experiencing a long phase of reorganization and display renovation, as a result of varying events and a recent relocation. This theoretical and methodological framing is part of a broader project that will attempt to construct a new valorisation proposal that takes into account the most updated discussions about heritage decolonisation — a topic that is emerging also in the Italian context. The aim is also to investigate the extent to which co-designing practices and participation in cultural activities, characterised by strong use of technology, can implement: well-being and social inclusion; digital and critical thinking skills, as essential in fostering active citizenship according to the life-long learning model in non-formal learning contexts such as museums. For some years now, the paradigm of the participatory museum has been spreading; its vision is mainly based on two bywords: ‘people as creative agents’; ‘artefacts as social objects.‘ These elements combine in three guidelines underlying the paradigm: the first is that cultural institutions should be ‘audience-centered,’ that is focused on the visitors and their experience; the second is that visitors create their personal meaning from their own cultural experiences (according to Dewey’s socio-constructivist learning theory); the third is that visitors’ opinions can inform and invigorate the museum project design [24]. Applying the participatory museum method in the context of non-European art collections, and specifically South-Asian art collections enriches the model with further significant elements. First of all, the possibility of the involvement of the source communities, which guarantees their promotion to co-protagonists, so as to generate forms of community empowerment [16, 24]; in addition, museums can become ‘counter-narratives laboratories’, allowing for the decolonisation of collections and exhibition practices [12]. In this sense, migrants become active agents in an intercultural dialogue, which is today — more than ever — urgent in the globalised context of diaspora cultures, especially for the construction of relational cultural identities and access to heritage [2, 16]. The participatory paradigm lends itself, on the one hand, to the development and valorisation of the social role of museums and, on the other hand, it can be seen as a precious tool for heritage decolonisation. Social exclusion can be defined as a dynamic process which prevents a person — or a certain community — from fully participating in the various economic-political and socio-cultural systems which guarantee integration into society [27]; consequently, social inclusion is the configuration of those conditions which allow the active participation of those previously excluded. In this respect, museums are increasingly becoming aware of their role as facilitators of social inclusion processes. The importance of cultural institutions — such as museums — as agents of social change has been often emphasised since the 1990s, identifying them as places in which integration, promotion of intercultural dialogue and involvement of socially-excluded groups within national and international cultural discourse are made possible [23]; on the other hand, museums have often excluded certain social groups from participating in their functions of communication, research and conservation of heritage [13]: exclusion from cultural interpretation processes has thus supported forms of marginalisation [23]. However, such a rethinking of the museum’s social role, as suggested by scholars, seems to

A Participatory Museum for Intercultural Development: Innovative …

131

be gaining ground. The definition of ‘museum’ given by ICOM, updated in 2022, defines it as “a permanent institution in the service of society […]. Open to the public, accessible and inclusive, museums foster diversity and sustainability. They operate and communicate ethically, professionally and with the participation of communities, offering varied experiences for education, enjoyment, reflection and knowledge sharing”, thus marking a precise guideline. The issues of accessibility, representation and (gender, marginalised communities and disabled people) participation are increasingly imposing themselves on cultural agendas, with the spread of awareness that exclusion causes social and cultural impoverishment on the territory, a phenomenon which can be countered through greater inclusiveness, that generates more widespread well-being quantifiable through, precisely, increased participation in social life [20]. The idea that museums can contribute to the well-being of a community is becoming more and more entrenched. ‘Health’ is defined by the World Health Organisation as “a state of complete physical, social and mental well-being, and not merely the absence of disease or infirmity.” On the other hand, ‘well-being’ is defined as “a positive state experienced by individuals and societies […] determined by social, economic and environmental conditions. Well-being encompasses quality of life, as well as the ability of people and societies to contribute to the world in accordance with a sense of meaning and purpose […]” [28]. Numerous studies are focusing on the relationship between participation in artistic and cultural activities and events and the promotion of people’s well-being and health, investigating their psycho-social impact; however, it is not easy to assess the impact of non-clinical interventions in the health arena and, despite a vast evidence base, acceptance of the relationship between arts and health is still somehow limited. Nevertheless, the body of research offering evidence of the beneficial effect of arts and creative activities on health and wellbeing is still growing [6]. Among others, the bio-psycho-social impact of cultural activities has been specifically studied by UCL (University College London), testing their particular positive effect — both from a social and general health point of view — especially in marginalised communities [5]. Another example is the research, based on the principles of participatory action research (PAR), conducted by UCL and Loughborough University at the Helen Bamber Foundation, which showed that participation in cultural and social activities has a positive impact on practical and social skills, on mood and emotion, and contributes to the enhancement of social health, mental wellbeing, self-confidence and resilience [6]. However, new studies that focus primarily on assessing this impact are still needed. The second fundamental aspect to be taken into consideration is, as anticipated, that of heritage decolonisation. Many museums have colonial origins, especially those which host the so-called anthropological or ethnographical collections. They were entwined with colonialism: methods of classification and display were key in developing colonial viewpoints and ideologies regarding ‘the other.‘ They could be — and still can be to a certain degree — integral apparatuses of colonialism, a legacy some of them are beginning to question, firstly by debunking the idea of museum neutrality. Scholars have been debating about this for the last 30 years, but only recently the idea has permeated through society. Museums should enter into what has been defined as the “spirit of epistemic and aesthetic disobedience” [17] undoing

132

L. Contardi

what they did in modern and imperial history. In this sense, the involvement of local and source communities and the practices of co-design become pivotal.

2 Target Data about communities from the Indian subcontinent — meaning the current nations of Afghanistan, Bangladesh, Bhutan, India, Maldives, Nepal, Pakistan and Sri Lanka — in Italy are particularly relevant. As a matter of fact, in spite of a process of stabilisation on the Italian territory and their large number, migrant communities from the Indian subcontinent still experience a lack of integration and participation in cultural and public life, as witnessed by recent reports produced by the Italian Ministry of Labour and Social Policies. Some significant data concerning the most numerous South-Asian communities in Italy (the Indian, Bangladeshi, Pakistani and Sri Lankan communities) will be taken into consideration. According to the data illustrated in the reports produced by the Italian Ministry of Labour and Social Policies [18], there are 153,946 Indian citizens legally residing in Italy, 138,509 Bangladeshi citizens, 122,406 Pakistani citizens and 100,033 Sri Lankan citizens (all data updated to January 1, 2021), and they constitute respectively the fifth, eighth, ninth and eleventh largest communities (of non-EU citizens) in Italy. The Indian, Bangladeshi, Pakistani and Sri Lankan communities in Italy are the largest in the European Union; moreover, Bangladeshis in Italy constitute the majority of those present in the entire European continent, as do Sri Lankans, almost half of whom have settled in Italy. Although the process of social integration of the migrant population represents a complex and multifactorial phenomenon, difficult to survey and measure, there are several indicators that — analysed over the long term — can help in understanding trends and patterns. For the most part, these indicators are about territorial localisation (i.e., the tendency to concentrate in certain areas of the country, a sign of communities that are historically more rooted), stabilisation (the number of longterm residents and applications for residence permits, which indicate the intention to settle for a long period, as well as the acquisition of citizenship), participation in social activities and other aspects of social life (work, training, associative activities), as well as other relevant data such as marriages. Among these factors, some of the most decisive in triggering the process are labour market participation, migration and integration policies, access to the welfare system, public and media narratives on migration. It is in fact through integration into the local labour market and into school, access to stable housing, and the possibility of participating in the social and political life of the country that inclusion can really be determined. Being complex and multidimensional, inclusion stems from a plurality of factors affecting both the individual and collective dimensions. From the analysis of these data, a rapidlychanging picture emerges: in the case of the Indian and Bangladeshi communities, indicators show how the stabilisation process in the Italian socio-economic context is not yet fully mature, although there are signs of acceleration; the stabilisation

A Participatory Museum for Intercultural Development: Innovative …

133

process of the Pakistani community is not yet mature either, although the available indicators show signs of progress; in the case of the Sri Lankan community, on the other hand, stabilisation is quite advanced — in the face of a migratory phenomenon that started as early as the 1970s. As shown, the South-Asian communities in Italy are numerous and rooted in the territory, but still experience low levels of integration. Nevertheless, younger generations seem to be aware of the importance of social and cultural inclusion: in the 2022 Manifesto of New Italian Generations (promoted by the Coordinamento Nazionale Nuove Generazioni Italiane, which represents associations of young people with a migrant background) a specific section is dedicated to the theme ‘Culture, sport and participation’. It states: “In order to foster processes of inclusion […], we believe it is important to move in two directions at the same time: enhancing and encouraging the preservation of the culture of the country of origin and strengthening the link with Italian culture. […] Promoting and reinforcing young people’s ties with their family’s culture of origin does not mean labelling young people who feel completely Italian as eternal ‘migrants’. Instead, it means valuing the richness of experiences and belonging” [7]. As the quote clearly shows, younger generations seem to be well aware of the important role the culture of origin can play in terms of social inclusion and intercultural development, to which is added the need for the exploitation of the available cultural heritage, also through the creation of opportunities for contact and exchange between people and communities, which is fundamental in a democratic society.

3 Research Context From the perspective of a participatory project, the South-Asian art collections in Italy are the perfect medium: they house a vast symbolic and typological heritage, with great internal diversity that allows the involvement of culturally heterogeneous communities; they raise interest within the general public, but their knowledge is often only partial; in addition, the largest collection in Italy (held at the MuCiv, Museum of Civilisations, in Rome) is currently undergoing a long process of reorganization. In this regard, it is worth briefly dwelling on the complex history of MuCiv collections, a history that further urges the need for well-structured mediation operations and direct involvement of the source communities. The MuCiv was established in 2016, with the aim of bringing together in a single institution the heterogeneous collections of the various different museums: the Prehistoric and Ethnographic Museum ‘Luigi Pigorini’; the Museum of Folk Arts and Traditions; the Museum of the Early Middle Ages; and the National Museum of Oriental Art ‘Giuseppe Tucci’ (MNAO). To these, the collections of the Former Colonial Museum, previously known as the Italian-African Museum, were added in 2017. The birth of the Museum of Civilisations happened in the wake of other international museums focusing on the theme of the human and its cultures, for the valorisation of heritages and testimonies of different identities and memories. The MuCiv is

134

L. Contardi

currently located in the EUR neighbourhood in Rome, at Palazzo delle Scienze and Palazzo delle Arti e Tradizioni. The entire complex was originally designed in the 1930s as an urban-architectural project to host the Universal Roman Exhibition that was to be held in 1942 — and later cancelled due to the outbreak of the Second World War. The project was thus born within the fascist regime — as is well demonstrated by the architectural and urbanistic layout that still characterises the district and the buildings of the complex — and the Exhibition it was to host was born from a utopian vision of an ‘Olympiad of Civilisations’ [21]. If, on the one hand, the decision to locate the MuCiv’s headquarters here seems a way to overturn the — hierarchical and Eurocentric — vision of fascism and its Exposition, declining its spaces in a new way, certainly the intended destination and its epistemological assumptions cannot be hushed up and must, instead, be considered and problematised. In addition to the functions of conservation, enhancement, and promotion of study and research, MuCiv declared mission is to foster cultural citizenship and promote dialogue and integration, through projects of cultural inclusion (from the Museum’s website). As for the National Museum of Oriental Art (MNAO), it was founded in 1957 and opened to the public in 1958. Its collections were previously housed in the neosixteenth century Palazzo Brancaccio on the Oppio hill in the center of Rome. In 2010, the Museum was officially named after Giuseppe Tucci (1894–1984), one of the leading academics specialising in Asian cultures in the twentieth century, who promoted its foundation. The initial nucleus of the collections consisted of artefacts deposited by the Italian Institute for the Middle and Far East (IsMEO, later merged into IsIAO, or Italian Institute for Africa and the East, which was dissolved in 2012 due to financial problems), and artefacts from excavations carried out by Italian archaeological missions in Iran, Afghanistan and Pakistan. The collections were later augmented by the Italian State, thanks to numerous purchases on the antiquities market and a flourishing exchange policy, and by various donations coming from private individuals. The Asia section of the Museum of Civilisations is therefore composed of the collections from the National Museum of Oriental Art ‘Giuseppe Tucci’, which were brought together there in 2016, plus objects from the National Prehistoric and Ethnographic Museum ‘Luigi Pigorini’ (more than 15,000 objects, mostly from purchases and gifts of diplomats, travelers, traders, scholars and artists). The confluence of these two institutions, with such a problematic and complex past, within the framework of the MuCiv, located in an area also characterised by a problematic past, as well as the consequent partial convergence and fusion of their collections, which stem from their two distinct historical-artistic and ethnographic matrices makes even more necessary and urgent an operation of total rethinking of the ways in which the museum can narrate and valorise its heritage, not forgetting its history, but instead opening it up to the interpretative and community involvement needs of a museum that wants to be in step with contemporary theoretical elaborations.

A Participatory Museum for Intercultural Development: Innovative …

135

4 Methodology As previously pointed out, special attention must be paid to the definition of new practices of cultural mediation and in the design of educational activities centered on the museum collections, not forgetting the fundamental importance of the involvement of source communities. In this sense, the technological tools now spreading also in the museum context can be very useful. New technologies offer tools both for user profiling, to improve visitor experiences and educational activities (AI; big data; eye-tracking), and for practices of re-appropriation of the narrative (digitisation and digital storytelling). Of these, some can be briefly considered to illustrate their usefulness and their purpose in relation to the aims of the project under consideration. Considering the technologies that allow the analysis of visitors’ habits and their experiences, a significant example is certainly eye-tracking. Mobile eye tracking (MET) technologies can be useful in the investigation of museum accessibility and fruition, allowing the observation of people’s spontaneous behaviour in selected settings and offering a wealth of potential information for analysis. Eye-tracking can be used in museums to map the visitors’ emotional responses and understand how they watch museum objects. This method also guarantees insights into momentary, ephemeral emotions experienced in naturalistic settings. Mobile eye-tracking technology can investigate guests’ experiences of awe in museums by analysing visitors’ visual attention. During the visit, visitors are asked to wear a lightweight mobile device (also in the form of glasses in the latest versions) and a number of cameras and sensors record eye movements and indicate the points on which the observer’s gaze lingers. The following measures can be extracted and analysed: total and average visit duration; total and average fixation duration; total and average fixation count; visit count. Besides this quantitative analysis, audio and video recordings from the MET can be individually analysed, to map each user’s experience and gaze pattern. It is important to notice that meaningful results can be obtained even with small numbers of participants. Many are the opportunities offered by this kind of technology: data richness, high external data validity (eye-trackers objectively record data through field and eye cameras, with a proven impossibility for the participants to control their behaviour but for fragments of seconds) and nonreactive measurement [9]. Experiments using this type of technology in the museum context have been conducted in various countries, including the USA [11], Germany [10] and Italy [9]. As stated by Jan Louis Kruger [14] when discussing applications of eye tracking research in media accessibility studies, eye movements provide a window into the internal systems of the mind. The use of technologies is also linked to the practice of digital storytelling (DST), which is now increasingly widespread and fundamental in the dynamics of the reappropriation of narratives by those subjects normally excluded from the voices considered. Digital storytelling refers to the creation of a narrative through the use of digital content, be it in the form of video, photos, text or audio [15]; it consists of the co-creation of individual and/or community narratives, based on one’s own personal stories and developed in the first person, by one or more subjects, and then

136

L. Contardi

realised through the use of various devices that allow for the construction, recording, editing and dissemination of the final product. Widespread since the beginning of the 1990s, its initial aim was the democratisation of art, in a dynamic of emancipation and consciousness-raising through self-legitimisation resulting from the possibility of being masters of telling one’s own story and experience [26]. Currently, it is widely applied in the pedagogical, cultural and, above all, museum contexts, due to its effectiveness related to the capacity for emotional and empathic involvement. As a matter of fact, digital storytelling can revolutionise the way visitors engage with cultural heritage. DST could be incorporated into any museum narrative through every imaginable format museum professionals deem feasible. It has been widely recognised as an important tool for attracting and satisfying the audiences (especially]digital natives’) of museums and other cultural heritage sites as well as for supporting teaching and learning at every level of education. It can stimulate emotional engagement, make difficult content culturally accessible and rememberable, thus promoting a sense of cultural belonging, and develop transversal and digital skills [22]. From these examples, it is possible to consider the relationship that can link digital skills and cultural heritage, showing the way in which their interrelation can be useful in enhancing communities’ well-being and social inclusion. Some predictions on student learning achievement in the COVID-19 era underline that social and digital marginalisation has increased cultural inequality [1]. Moreover, displaced people with low levels of digital literacy risk wider forms of political and social marginalisation, if they are unable to reintegrate into innovative professional and cultural contexts [4]. In this sense, organising cultural activities involving the use of technology can be a way to counteract the phenomena of social isolation and marginalisation by giving skills and familiarity with the use of certain digital and technological tools to those who, for various reasons, do not have them. Moreover, the development of transverse skills, through specific educational activities and cultural integration, can also contribute to the promotion of democracy and participation policies, as well as the welfare of the society itself. In general, the promotion of 4C skills (Creativity, Communication, Collaboration and Critical Thinking, all of them linked also to digital literacy) is highly supported by Cultural Heritage [19].

5 Evaluation Although the studies referred to in the previous paragraphs partially illustrate the potential — in terms of well-being and inclusion — associated with participation in cultural activities, and although the attention of the scientific community is beginning to focus on these aspects, further attention to these issues and further studies are needed, especially in the Italian context. It is necessary to investigate to what extent innovative ways of heritage fruition and education centered on practices of co-design within the museum context, characterised by strong use of new technologies, can implement social inclusion and well-being of marginalised communities and the

A Participatory Museum for Intercultural Development: Innovative …

137

development of digital skills (and, therefore, critical thinking) fruitful to developing active citizenship according to the lifelong learning paradigm. The bio-psycho-social impact of cultural activities has been studied, specifically, by UCL (University College London), which has developed a tool (the Museum Wellbeing Measures Toolkit) useful for testing it, and through which their particular positive effect — both from a social and general health point of view — has been noted, especially in marginalised communities [5]. This is a set of measurement scales, piloted throughout the UK, designed to assess the impact of participation in museum and gallery activities. The UCL Toolkit is designed to help people involved in designing, running, setting up and managing museum projects and those involved in visitor outreach and engagement to assess the impact of their work on public wellbeing. The UCL Toolkit measures psychological well-being as an indicator of an individual’s mental state. There are also other aspects of wellbeing, such as physical and social wellbeing, but the UCL Toolkit focuses on self-perceived levels of mood and emotional changes, because these aspects of wellbeing are more likely to vary following a short-term experience, such as participation in a cultural activity in a museum or gallery. The UCL Toolkit consists of two general well-being questionnaires (a short version and an extended version) and four umbrella-shaped survey instruments to measure well-being, differently formulated to be administered to different groups of users. There is an umbrella for positive moods, one for negative moods, one for the elderly and a last one for young adults, with instructions (included on the same page) and a separate sheet reserved for comments. All umbrellas are hexagonal in shape, with six differently-coloured sections. Different colour schemes were tried out, including warm colours for the positive umbrellas, and cool colours for the negative umbrellas. At the top of each section is a word expressing a mood or emotion and a series of numbers, from one to five. Following the instructions, participants have to assess to what extent, at that moment, they feel the emotion or mood described by the word by circling the appropriate number. The well-being measuring umbrella is a highly visual instrument. Its compilation is intuitive: the higher the number, the more intense the colour and the larger the area occupied. It is possible to take two measurements, the first before the activity, to establish a baseline, and the second afterwards, so that any changes in participants’ well-being can be compared. Measurements can also be taken over time, for example at the beginning of a programme, after a few weeks and at the end of the activity [25]. For evaluating levels of social inclusion, the Inclusive Processes Evaluation Scale defined by Cottini can be considered. It draws inspiration from the Index for Inclusion: Developing Learning and Participation in Schools [3], an important reference point for what concerns the development of inclusive processes in various educational contexts from a participative point of view. It refers to the social model of disability (and learning difficulties), which promotes, through a clear definition of indicators and descriptors, the removal of obstacles that limit social participation and learning, enhancing the potential of all those involved and the transformation of contexts. The Index explores the concepts of inclusion and exclusion through three key dimensions, each subdivided into two interconnected sections, which should guide the process of

138

L. Contardi

change in the educational context: creating inclusive cultures (building communities; affirming inclusive values); producing inclusive policies (developing the school for all; organising support for diversity); developing inclusive practices (coordinating learning; mobilising resources). From this, the tool proposed by Cottini provides: two self-assessment scales (one on the organisation itself, with 20 items; one on teaching methods, with 20 items), plus an objective assessment scale (15 items) [8]. Finally, in the European context digital competencies are defined by the Digital Competence Framework for Citizens (or DigiComp). There are several assessment tools that can be used and that have been designed over the years. What must be noted is the stratified nature of these competencies: they consist not only of technological and critical-cognitive components, but also of relational and social ones; in this sense, three dimensions must be considered and integrated: the technological dimension (ability to assess, present and exchange information; ability to choose appropriate technologies to tackle real problems); the cognitive dimension (ability to read, select, interpret and evaluate data, build abstract models and assess information considering its relevance and reliability; ability to critically evaluate); and the ethical dimension (social responsibility).

6 Conclusion As seen in the previous paragraphs, the definition of a new paradigm of fruition of the South-Asian art collections in Italy is not only necessary, but also useful on several levels. On the one hand, there is a pressing need for interventions aimed at fostering the inclusion of South Asian communities in the Italian context, given their long-term stability in the territory; such a need is also deeply felt by new generations, together with a willingness to reflect on their own culture of origin, even more so in the globalised and diaspora context. On the other hand, ensuring that museums regain their social role and operate in an inter- and transcultural key cannot be separated from a commitment to the decolonisation of their collections, a fundamental issue for institutions that wish to operate ethically in the contemporary world, contributing to the formation of new citizens, who are active, engaged and critical. The methodological tools to implement these processes exist and are manifold, as shown; however, it is crucial that these kinds of activities and interventions be evaluated with appropriate tools, so as to not only affirm their importance and urgency, but also to demonstrate their effectiveness even more strongly.

References 1. Azevedo, J.P., Hasan, A., Goldemberg, D., Geven, K, Iqbal, S.A.: Simulating the Potential Impacts of COVID-19 School Closures on Schooling and Learning Outcomes: A Set of Global Estimates. World Bank Res Obs. (2021)

A Participatory Museum for Intercultural Development: Innovative …

139

2. Bodo, S., Cifarelli, M.R.: Quando la cultura fa la differenza. Patrimonio, arti e media nella società multiculturale. Meltemi (2006) 3. Booth, T., Ainscow, M.: Index for INCLUSION: Developing Learning and Participation in Schools. Centre for Studies on Inclusive Education (2002) 4. Carretero Gomez, S., Vuorikari, R., Punie, Y.: DigComp 2.1: The Digital Competence Framework for Citizens with eight proficiency levels and examples of use. Publications Office of the European Union (2017) 5. Chatterjee, H., Noble, G.: Museums, Health and Well-Being. Ashgate (2013) 6. Clini, C., Thomson, L.J.M., Chatterjee, H.J.: Assessing the impact of artistic and cultural activities on the health and well-being of forcibly displaced people using participatory action research. BMJ Open 9(2), e025465 (2019). https://doi.org/10.1136/bmjopen-2018-025465 7. CoNNGI. https://www.integrazionemigranti.gov.it/Dettaglio-approfondimento/id/29/Manife sto-delle-nuove-generazioni-italiane. Accessed 13 Mar 2023 8. Cottini, L.: Didattica speciale e inclusione scolastica. Carocci Editore (2018) 9. Di Giovanni, E.: Museum through the visitor’s eye: eye tracking and museum fruition. In: Poce, A. (ed) Advanced Studies in Museum Education. Lectures. Edizioni Scientifiche Italiane (2019) 10. Eghbal-Azar, K., Widlok, T.: Potentials and limitations of mobile eye tracking in visitor studies: evidence from field research at two museum exhibitions in Germany. Soc. Sci. Comput. Rev. 31(11), 103–118 (2013) 11. Filippini Fantoni, S.: Capturing Visitors’ Gazes: Three Eye Tracking Studies in Museums, MWMuseum and the Web 2013 conference. https://mw2013.museumsandtheweb.com/pa-per/cap turing-visitors-gazes-three-eye-tracking-studies-in-museums/. Accessed Jan 2019 12. Grechi, G.: Decolonizzare il museo. Mostrazione, pratiche artistiche, sguardi incarnati. Mimesis (2021) 13. Kinsley, R.P.: Inclusion in museums: a matter of social justice. Museum Manage. Curatorsh. 31(5), 474–490 (2016) 14. Kruger, J.-L.: Eye tracking in audiovisual translation research. In: Pérez-González, L. (ed.) The Routledge Handbook of Audiovisual Translation, pp. 350–366. Routledge, Milton Park (2018). https://doi.org/10.4324/9781315717166-22 15. Lambert, J.: Digital Storytelling: Capturing Lives. Creating Community. Routledge, Milton Park (2013) 16. Lattanzi, V.: Musei e antropologia. Carocci Editori (2021). 17. Mignolo, W.: Museum in the colonial horizon of modernity. Fred Wilson Mining the Museum (1992). In: Harris, J. (ed). Globalization and Contemporary Art. Blackwell Publishing Ltd. (2011). 18. Ministero del Lavoro e delle Politiche Sociali, https://www.lavoro.gov.it/priorita/Pagine/Pub blicati-i-nuovi-rapporti-Le-comunita-migranti-in-Italia.aspx, last accessed 2023/03/13 19. Poce, A.: Il patrimonio culturale per lo sviluppo delle competenze nella scuola primaria. Franco Angeli (2018). 20. Poce, A.: La ricerca empirica al museo: metodologie, strumenti e funzioni. Edizioni Scientifiche Italiane (2020). 21. Ramasso, C.: Orienti. 7000 anni di arte asiatica dal Museo delle Civiltà di Roma. Silvana Editoriale (2018) 22. Robin, B.R.: The power of digital storytelling to support teaching and learning. Digital Educ. Rev. 30, 17–29 (2016) 23. Sandell, R.: Museums as agents of social inclusion. Museum Manage. Curatorsh. 17(4), 401– 418 (1998) 24. Simon, N.: The Participatory Museum. Museum 2.0 (2010) 25. Thomson, L.J., Chatterjee, H.J.: UCL Museum Wellbeing Measures Toolkit. https://www. ucl.ac.uk/culture/sites/culture/files/ucl_museum_wellbeing_measures_toolkit_sept2013.pdf. Accessed 13 Mar 2023 26. Waycott, J., Davis, H., Warr, D., Edmonds, F., Taylor, G.: Co-constructing meaning and negotiating participation: ethical tensions when ‘giving voice’ through digital storytelling. Interact. Comput. 29(2), 237–47 (2017)

140

L. Contardi

27. Walker, A.C.: Britain Divided: The Growth of Social Exclusion in the 1980s and 1990s. Child Poverty Action Group, London (1997) 28. World Health Organization, Health Promotion Glossary of Terms 2021. https://www.who.int/ publications/i/item/9789240038349. Accessed 13 Mar 2023

Promoting Positive Ageing Lifestyles and Wellbeing Through the Use of Social Media to Facilitate and Enhance Creative Decision-Making Ron Corso and Charlie-Helen Robinson

Abstract This paper represents work in progress with aged communities, on a project to explore the creative potential that can be harnessed from aged communities using social media platforms to assist in the articulation and dissemination of ideas developed through creative collaborative decision making. Our aim is to promote creative thinking and ideas through the introduction of concepts around deliberate creative practices within our aged communities. We wanted to make a case for how this could be instructive for how these communities are empowered to participate in identifying issues and challenges that are able to be actively disseminated through social media, demonstrating the application of creativity processes to decisions and policy impacting and influencing their lifestyles, health and wellbeing across a wide spectrum of social and community applications. Keywords Ageing Population · Creative Thinking · Social Media

1 The Perception of Aged Communities as a Burden on Society A significant observable aspect of twenty-first century communities worldwide, and especially in industrialised countries, is that they are increasingly being faced with a growing aged population. Our constant drive to improve the quality of human life at all levels has resulted in people living longer, living better through advances in preventative medicine and the promotion of healthy lifestyles [3, 22].

R. Corso (B) University of South Australia, Adelaide, Australia e-mail: [email protected] C.-H. Robinson Communications Specialist, Adelaide, Australia © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_12

141

142

R. Corso and C.-H. Robinson

A result of this positive outcome is the creation of a potential growing burden or threat, rather than an opportunity when it comes to the question of how we accommodate, in all manner of ways, people living longer and the extra demands placed on society [18]. Elderly people have been marginalised, institutionalised, and stripped of responsibility, power, and, ultimately, their dignity [27, 28]. That the elderly cannot provide any new social, economic or surplus value in forms such as products, services and influence on society is leading to an age discrimination or ageism prejudice [5, 38] and the likely growth of inter-generational conflict [23, 40]. Western society is moving away from the notion of the extended family, and this is becoming especially pronounced with an increasing emphasis on the nuclear family seen as narrow, harmful and diminishing much of family life [34]. The responsibility of state-provided care for the elderly in the form of pensions, accommodation, mobility and health has resulted politicians and others predicting the economic Armageddon and negative consequences that a growing ageing population will present to younger generations having to generate the income to provide this support [14, 18].

1.1 Changing Perceptions of Aged Societies In spite of these negative perceptions, the ageing phenomena according to ref. [17], is taking on new meanings because of its role as a change agent in society. The perception and definition of being senior is undergoing change away from preconceptions of previous generations, now described as healthier, living longer, better educated and more engaged in a range of interests beyond senility in a retirement village. Perspectives are shifting from a ‘burden’ view of diminished value in being old, to a more positive position where experience, wisdom and leadership are the positive attributes to offset the perceived negative challenges of ageing. Ray, Suar, and Mukhopadhyyay [42] contend that senior citizens have a preference to stay active for as long as they can, and in order to do this, support must be forthcoming from the general community to harness their abilities and reverse adverse perceptions about them. Elderly people have knowledge, work experience and culture and contribute to the workforce as a resource, rather than a burden [38]. Links between creativity and wisdom are being recognised — especially the capacity of the older mind for creative output and, rather than just assigning older people as ‘keepers of the culture,’ there is a need to recognise their creativity as a change agent in society [9, 32]. There is increasing evidence that creativity can deal with aged challenges by enhancing cognitive performance, a greater sense of personal influence and integrity, physical functioning and social skills, through a more confident and motivated form of creative expression [2, 37].

Promoting Positive Ageing Lifestyles and Wellbeing Through the Use …

143

1.2 Positive Ageing Active ageing, Creative ageing, Reframing Age, Embrace Age and positive images of old age [13] are among a range of initiatives that are being embraced by communities, to counter the perception that a growing ageing population will present serious challenges to society in accommodating a section of the population that is no longer seen as productive. The ability to contribute to society overall should not be the domain of only certain sectors of the community. As a solution, our ongoing work has been aligned to the adaption of an ageing policy around the concept of ‘Creative Capital’ introduced by ref. [12], tapping into the creative capital of older people. If we can presume that the general population has a potential for creative capacity [6], then this should also translate to the elderly and, with that, the potential to make adaptions and transformations to their lives and environment based on imaginative cognitive modelling [30]. There is merit in exploring how the wisdom of the aged can continue to contribute to the economic wellbeing of a community by being engaged in active work and a range of support, mentoring and teaching roles [1]. Reference [21] highlights factors that can stimulate creativity at all stages of life and especially in the promotion of positive ageing. Factors such as openness to new ideas, a questioning attitude and a tolerance for ambiguity and uncertainty. Our methodologies aimed to promote these qualities by building confidence and assertiveness, focusing attention to issues people identified as important to them and the community the elimination of stereotypes around ageism and the skills necessary in exercising these choices in a supportive creative environment. Ageing communities, like all age groups, are not immune to the concept of ‘Future Shock’ [39] and the challenges presented through the rate and scale of enormous change and disruptions that can be overwhelming. This can cause disconnection and uncertainty in people’s desire for stability and security, and perhaps even more so for elders when desiring to retreat from the pressures of work and life in general. Reference [19] argues that, in this environment, affecting all of society, it is unreasonable to consider the ageing population only in negative consequences and ignore the challenges facing all of society and the positive change that creativity can generate (inclusive of the aged) in dealing with multi-level cross-disciplinary collaborative challenges. Our contention is that new models of thinking to deal with future scenarios that cannot always be confidently dealt with using traditional thinking are challenges for all community sectors, and building a case that excludes the aged is to ignore the potential contribution they can make to the overall challenge. Our work supports giving aged communities the opportunity to eliminate stereotypes on ageism and provide an environment to be creative enhancing the identifying issues of concern to them and the broader community and take deliberate action in presenting and disseminating solutions that will make a difference.

144

R. Corso and C.-H. Robinson

1.3 Social Media Concepts and Approaches In attempting to build a creative dimension as a skill set and advocacy tool for aged communities, we explored the impact of social media and ways we could encourage older people to participate and engage in identifying and sharing new ideas. That Facebook can act as a catalyst for increasing social capital and the implication of “awareness” is promoted by ref. [11] as a new dimension. This was an aspect we wanted to explore in our workshops negating the perception that social media is something not commonly embraced by the older generation, often being perceived as a younger person’s domain. The use of social media, in particular blogging, and online professional forums such as LinkedIn, is growing across all age sectors. In parallel, more and more people research potential health professionals or organisations through social media or online via Google searches and in general many services are moving online necessitating mastery of the medium to keep up [41]. Online communities are also being promoted as suitable for providing and receiving social support when confronted with a difficult life situation, regardless of geographical location or time. Further positive consequences have been shown to be overcoming loneliness, relieving stress, and raising feelings of control and self-efficacy [35]. Increased participation in social networks can also empower older people providing a sense of connectedness and greater control and self-efficacy [7]. The Active Seniors Association Helsinki, Finland, is a prime example of aged communities actively co-designing practices through a social collective, willing to challenge established modes, traditions and attitudes in proposing creative alternate ways to grow old. New design tools were developed as well as the skill and confidence to embrace new technologies and social media platforms in not only presenting new ideas but the advocacy for change [4]. In general, social media provides opportunities for older adults to participate in discussions and stay contacted with the broader community through a medium that enables them to express themselves. Participating in social networks, it is argued, can empower older people and provide them with a sense of connectedness and greater control and self-efficacy [20].

1.4 A Creative Dimension Our work in building a creative dimension in aged communities began with establishing an understanding of what is meant by creativity and creative thinking and demystifying its relevance to human behaviour. Creativity is a term that can be interpreted many ways, and it carries many misconceptions and assumptions not only about what it is but how it works [33]. Traditionally considered to be personal, inborn, and individual; specialized; and about inventing something, it is now acknowledged to be collaborative, learnable, community based, cross or multi-disciplinary and

Promoting Positive Ageing Lifestyles and Wellbeing Through the Use …

145

about ideas and making connections [25]. Unsurprisingly, this reflects the way most of us work best [31]. Our own take is that creativity involves the ability to make a perception shift enabling the process of reconceptualising new interpretations from existing bodies of knowledge and scenarios [16]. Of interest to our work with aged communities is the notion that creativity has been found to be collaborative [8] and community based [12], about transference across disciplines and about communication and sharing ideas and making connections, and we contend that it can be taught [25]. These understandings that creativity is an inherent human and social attribute, we believed, would have a big influence on how we structured experiences to nurture and take advantage of an inclusive creative way of engaging the elderly with their communities, through their extensive experience and wisdom. We encouraged coproduction to facilitate problem solving through an organic growth of ideas and strategies for implementation [24, 26], where, according to ref. [29], better outcomes are realised rather than the traditional unilateral decision making of largely government bodies and aged care institutions. Reference [10] contends that creative output is perfectly possible in older people who are capable of exhibiting the attributes of creative behaviours such as sensitivity to issues, openness to new and novel approaches that may challenge conventions, willingness to take risks, and the ability to work collaboratively. Reference [21] argues that creativity at all stages of life can be stimulated and helps to foster successful ageing. Attributes such as openness to new ideas, a questioning attitude, a tolerance for ambiguity, unorthodoxy, intuitive, being playful and flexible, and the ability to see opportunities can contribute to supporting creative activities and the elimination of stereotypes on ageing if given the right environment. Reference [9] argues for the promotion of creativity in aged communities as something that is a part of all human activity and that can have specific value beyond a group that is just seen as the “keepers of the culture” to one that can play a role in changing a society and culture.

1.5 Introducing Creativity Processes and Approaches Through a series of workshops, a program to introduce an aspect of creativity and idea generation in aged communities was introduced. This incorporated a range of specific creative-thinking approaches, structured in a way that demonstrated understandings of the nature of creativity and where ideas could be developed and applied. People seldom work in isolation due do the social nature of humans [15, 36], so we established a collaborative, learning environment based on an unstructured group discussion encouraging egalitarian cross disciplinary teamwork modelled on the way enterprising organisations work. Fluent thinking encouraged the group to generate lots of ideas without critique and by withholding the judgment sense, no matter how crazy, seemingly silly or inappropriate, thus allowing all ideas an opportunity to be considered no matter how

146

R. Corso and C.-H. Robinson

irrelevant they may have initially appeared. We built confidence by removing the fear of making mistakes, the need to be right, peer acceptance, the need to have an expected answer, the need to follow a predetermined process, the expectation that a process needs to lead to a solution, and the need to follow a set routine using only logical, analytical, routine or judgmental thinking. Flexible thinking encouraged lots of different ideas, reinterpreting ideas and then restructuring these into new configurations by comparing or substituting things with similar qualities. Taking an existing idea from one situation, discipline or application and applying it to a different one, considering the opposite or negative of an idea, etc. Elaborative thinking encouraged originality, personal interpretations, playfulness, having fun, risk taking and using humour and absurdity as well as using analogies, metaphors and mistakes to change thinking or ways of looking at the problem by using ‘wrong’ ideas or ‘errors’ as a springboard for the generation of new ideas. We placed emphasis on unblocking the associational, cultural, professional, emotional, social, language and other impediments to creative thinking (often developed over a lifetime of habitual process) by challenging ingrained assumptions or preconceived ideas. Using our experience of teaching creativity in university programs, we began with small steps to develop confidence, provide positive experience of the process and model effective ways of working before slowly adding complexity as participants’ capabilities and self-assurance developed for the appropriate level. Using design thinking methods enhanced the group’s ability to create new knowledge and ideas based on a process of prototyping, experimenting and exploration. Participants were challenged to think reflectively and externalize their skills in the process of understanding the ‘thinking behind the thinking.’ They learned the art of structuring an argument to elaborate their thinking process, thinking critically and objectively about ideas and approaches, and avoiding impulsive narrow responses based on emotion and possible biases.

1.6 The Project A group of aged volunteers came together to explore the topic of positive ageing through the use of social media and creative thinking. We concentrated on ways to unpack a number of factors relating to the aged theme so as to gain new insights. Working collaboratively aimed to give participants the opportunity to present and share ideas building the confidence to explore new initiatives to a number of aged topics. The group had discussions around an exploration of what our life will look like as valued citizens contributing to society in our retirement years. Initially, people gravitated the conversation towards things that were of a personal concern to them, rather than look at the broader context. It was agreed that this casual conversation did give them the ability to have a voice. Social media facilitated and extended

Promoting Positive Ageing Lifestyles and Wellbeing Through the Use …

147

these conversations, with some people breaking away and continuing a more private debate. Many good discussion themes were established during these sessions.

1.7 The Ideas There were a number of topics discussed and from these the following were highlighted: Co-housing Communities We discussed where people of all ages live together in a shared housing complex. Each individual or family has their own private living space, but there are also shared common areas, meals, and activities. This model allows the elderly to have their own independence while still being able to receive support and socialization from their younger neighbours. Combining Facilities: We discussed combining aged care facilities with schools, kindergartens, medical centres, social facilities, sporting complexes, retail and small business etc., so as to desegregate the trend towards separate isolated specialist environments and people interaction based on convenience and ease of management, destroying the opportunity for integration and interaction. Community-Supported Ageing: We discussed variations on how networks of volunteers and/or paid caregivers come together to support elderly individuals who wish to continue living in their own homes. Intergenerational Co-home Ownership: We discussed a group of elderly individuals living together and cared for by a younger adult or group of adults who reside with them. This arrangement allows the elderly to receive personalized care and support while also providing the younger caregivers with an opportunity to learn from and connect with their elderly charges. The elderly person establishes joint arrangements under co-home ownership, which includes a financial incentive. Elder Foster Care or Adoption: We discussed a model similar to traditional foster care; however, instead of caring for children, individuals or families open their homes to elderly individuals who need support (coined ‘adoption flip’). The elderly person becomes a member of the family and is integrated into their daily lives, rather than being isolated in a separate facility. Expanding Educational Experiences: We discussed enhancing the curriculum, by formally integrating older people as co-teachers and mentors in the classroom using a range of lifelong skills, knowledge and wisdom. This could expand to social skills and reinforcing cultural values. An example was given where elderly people were teaching students how to dance rock ‘n’ roll — something they did well as young people and that was an important part of their social lives and something that young people today thought was cool and wanted to learn. The elderly become another reference for young students with “Work with the Wizard,” “Phone an old friend,” “Gray

148

R. Corso and C.-H. Robinson

Google,” and “Seniors in Schools” being some of the terms offered. All university students, no matter what discipline, must undertake a series of mandatory elective offerings involving working with, associating with, and consulting with elders in that discipline. Aged Consultancy: We discussed people in various disciplines, trades, occupations etc., as a business model, who can reference elderly experts, for a fee. The term “Coming out of Retirement” was used when it was identified certain skills in the workforce had been lost or replaced and where a reference to these might be seen as valuable to be reactivated or reconsidered, to then re-engage people with all the accumulated wisdom. Some of these ideas are now being taken onboard by teams within the group, with consideration of being further expanded and articulated for possible support and implementation through a range of government and private bodies.

1.8 Conclusion The project emanated from the application of combining social media and creative thinking to the dimension of aged communities, where the perception of ageing being a burden on society was changed. The project is very much a work in progress, but initial outcomes and ideas have given encouragement that both communication and connection through social media and the ability to articulate ideas using creativethinking methodologies can have a positive change factor in the wellbeing of elderly people determining their future. Witnessing the initial enthusiasm, originality of ideas and the empowerment and belief in initiating change provides a great incentive to formalize our findings in ways that can have a greater impact and provide support at many levels for ideas to be disseminated and implemented.

References 1. Ardelt, M.: Antecedents and effects of wisdom in old age: a longitudinal perspective on aging well. Res. Aging 22(4), 360–394 (2000). https://doi.org/10.1177/0164027500224003 2. Blazer, D.: Self-efficacy and depression in late life: a primary prevention proposal. Ageing Mental Health 6(4), 315–324 (2002) 3. Bloom, David E., Canning, D., Lubet, A.: Global population aging: facts, challenges, solutions & perspectives. Daedalus Spring. 144(2), 80–92 (2015). (The MIT Press on behalf of American Academy of Arts & Sciences) 4. Botero, A. Aging together: social media and the practices of coordinating everyday life in the project of an association of active seniors. University of Art and Design Helsinki. Media Lab. Hämeentie 135C 00560 Helsinki – FIN (2011). 5. Butler, R.: Successful aging and the role of the life review. J. Am. Geriat. Soc. 22, 529–535 (1974)

Promoting Positive Ageing Lifestyles and Wellbeing Through the Use …

149

6. Bramlett, M.: The Relationship Between, Creativity and Reminiscence in the Elderly. Medical College of Georgia (1990) 7. Chen, Y., Gao, Q.: Effects of social media self-efficacy on informational use, loneliness, and self-esteem of older adults. Int. J. Human Comput. Interact. 39(5), 1121–1133 (2023). https:// doi.org/10.1080/10447318.2022.2062855 8. Csikszentmihalyi, M.: Creativity Flow and the Psychology of Discovery and Invention. HarperCollins, NY (1996) 9. Cohen, G.D.: New Theories and research findings on the positive influence of music and art on health with ageing. Arts & Health. 1(1), 48–63 (2009) 10. Cropley, A.: Creative Performance in Older Adults. Reflections on Educational Achievement. 75–871. Waxmann academia.edu, Germany (1995) 11. Erickson, L.B.: Social media, social capital, and seniors: The impact of Facebook on bonding and bridging social capital of individuals over 65.2011. In: Conference: A Renaissance of Information Technology for Sustainability and Global Competitiveness. 17th Americas Conference on Information Systems, AMCIS 2011, Michigan, USA (2011) 12. Florida, R.: The creative class and economic development. Econ. Dev. Q. 28(3), 196–205 (2014). https://doi.org/10.1177/0891242414541693 13. Shankardass, M.K., Formosa, M.: The United Nations Madrid International Plan of Action on Ageing: Global Perspectives. Routledge, London (2022). https://doi.org/10.4324/978100330 0724 14. Gee, E.M.: Misconceptions and misapprehensions about population ageing. Int. J. Epidemiol. 31(4), 750–753 (2002). https://doi.org/10.1093/ije/31.4.750 15. Gardiner, P.: Thinking Skills and Creativity. Learning to Think Together: Creativity, Interdisciplinary Collaboration and Epistemic Control. Elsevier Ltd. (2020) 16. Gluth, S., Corso, C.: The Creative Application of knowledge in University Education: A Case Study. researchgate.net (2008). 17. Jaakola, H., Ekström, M., Guilland, A.: Changing attitudes towards seniors as learners. creating an understanding of seniors as digital storytellers. In: Proceedings of EDULEARN15 Conference. Barcelona Spain, pp. 1635–1644 (2015) 18. Klimczuk, A.: The politics of ageing and the challenges of ageing populations. In: Klimczuk, A. (ed.) Economic Foundations for Creative Ageing Policy, Volume II, pp. 1–15. Palgrave Macmillan US, New York (2017). https://doi.org/10.1057/978-1-137-53523-8_1 19. Klimczuk, A.: Economic Foundations for Creative Ageing Policy, Volume II. Palgrave Macmillan US, New York (2017). https://doi.org/10.1057/978-1-137-53523-8 20. Leist, A.K.: Social media use of older adults: a mini-review. Gerontology 59(4), 378–384 (2013). https://doi.org/10.1159/000346818 21. Levy, B., Langer, E.: The Life Span Developmental Model of Creativity. mitchmedical.us (2016) 22. Living longer. Living better. Department of Health (Australia) (2012). http://apo.org.au/node/ 29086 23. Meisner, B.A.: A meta-analysis of positive and negative age stereotype priming effects on behavior among older adults. J. Gerontol. Ser. B Psychol. Sci. Soc. Sci. 67B(1), 13–17 (2012). https://doi.org/10.1093/geronb/gbr062 24. McIntyre-Mills, J.: Participatory design for democracy and wellbeing: narrowing the gap between service outcomes and perceived needs. Systemic Practice and Action Research. Springer (2010) 25. McWilliam, E.: Teaching for creativity: from sage to guide to meddler. Asia Pac. J. Educ. 29(3), 281–293 (2009). https://doi.org/10.1080/02188790903092787 26. Needham, C., Carr, S.: Co-Production: An Emerging Evidence Base for Adult Social Care Transformation. SCIE Research Briefing 31. Queen Mary University, Social Care Institute for Excellence, London (2009) 27. Nelson, E.A., Dannefer, D.: Aged heterogeneity: Fact or fiction? The fate of diversity in gerontological research. The Gerontologist 32(1), 17–23 (1992). https://doi.org/10.1093/geront/32. 1.17

150

R. Corso and C.-H. Robinson

28. Nelson, T.D.: Ageism: Prejudice against our feared future self. J. Soc. Issues 61(2), 207–221 (2005). https://doi.org/10.1111/j.1540-4560.2005.00402.x 29. Ottmann, G., Laragy, C. Allen, J., Feldman, P.: Coproduction in Practice: Participatory Action Research to Develop a Model of Community Aged Care. Systemic Practice and Action Research. Springer (2011) 30. Puccio, G.J., Chimento, M.D.: Implicit theories of creativity: laypersons’ perceptions of the creativity of adaptors and innovators. Percept. Motor Skills 92(3), 675–681 (2001). https://doi. org/10.2466/pms.2001.92.3.675 31. Puccio, G.J.: Creativity as a life skill (TEDx Gramercy) Retrieved February 2023 32. Reed, I.C.: Creativity: self-perceptions over time. Int. J. Aging Human Dev. 60(1), 1–18 (2005). https://doi.org/10.2190/WM5Y-FHEM-CXQT-UEXW 33. Runco, M.A., Jaeger, G.J.: The standard definition of creativity. Create. Res. J. 24(1), 92–96 (2012). https://doi.org/10.1080/10400419.2012.650092 34. Sarkisian, N., Gerstel, N.: Nuclear Family Values, Extended Family Lives. Routledge, New York (2012). https://doi.org/10.4324/9780203141977 35. Newman, L., Stoner, C.: Social networking sites and the experience of older adult users: a systematic review. Ageing Soc. 41(2), 1–26 (2019). https://doi.org/10.1017/S0144686X190 01144 36. Segerstrale, U.: Education for Creativity, Skills, and Cross-disciplinary Collaboration. In: IEEE 18th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC) 147–334 (2019). https://doi.org/10.1109/iccicc46617.2019.9146076. ISBN: 1-7281-0496-3, 1-7281-1419-5 37. Stephenson, R.C.: Promoting wellbeing and gerotranscendence in an art therapy program for older adults. Art. Ther. J. Am. Art. Ther. Assoc. 30(4), 151–158 (2013) 38. Thane, P.: The growing burden of an ageing population? J. Publ. Policy 7(4), 373–387 (1987). https://doi.org/10.1017/S0143814X00004566 39. Toffler, A.: Future Shock. Random House, Bantam (1984) 40. Walker, A.: The economic ‘burden’ of ageing and the prospect of intergenerational conflict. Ageing Soc. 10(4), 377–396 (1990). https://doi.org/10.1017/S0144686X00007376 41. Wilding, W., Baldassar, L.: Ageing, migration and new media; the significance of transnational care. J. Sociol. 54(2), 226–235 (2018) 42. Ray, S., Suar, D., Mukhopadhyay, S.: Are senior citizens resources? A qualitative inquiry. J. Geriat. Care Res. 5(2), 43–51 (2018)

Learning and Performance Technology for the Performing Arts Gary J. Dickelman

Abstract Throughout the SARS-CoV-2 pandemic, performing arts organizations worldwide became creative with technology-based learning and performance support tools to maintain continuity for their respective crafts. While shared cloud repositories and collaboration tools were in use prior to the pandemic, circumstances catapulted many advances. Virtual rehearsals were initially problematic because of bandwidth variations from performer to performer. Early adopter tools, like Jamulus, emerged to synchronize real-time rehearsal for choirs, ensembles, and orchestras (Jamulus: Play music online with friends. https://jamulus.io/. Accessed 10 March 2023.). Applications also emerged to support artist learning, performance, composition, production design, and adjudication. At the same time, technologies have matured that significantly enhance the audience experience. What we now refer to as workplace learning and performance (L&P) was once simply “training.” The transition began in the late 1980s when focus shifted from the teacher (trainer, instructor, lecturer) to the learner, with primary focus on doing — the performance outcome. Arguably, the workplace performance metaphor was borrowed from the performing arts. This paper reexamines this metaphor, surveying tools, techniques, and protocol that have emerged, through interviews with performing arts professionals, and discusses how workplace learning might further benefit from such tools and protocol. Included are innovations that foster shorter time to performance competency, lower costs, and improved quality, while also uncovering unintended consequences of technology-mediated performance. Keywords Learning · Performance · Training · Performing Arts · Technology · Performance support · Drum corps · Theater · Adjudication

G. J. Dickelman (B) EPSScentral LLC, Boynton Beach, FL 33437, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_13

151

152

G. J. Dickelman

1 Background Imagine you are watching an NFL Super Bowl half-time show. It is an impressive spectacle where the football field is transformed in minutes to a huge stage, featuring scores of musicians, dancers, props, fireworks, costumes, and technology-enhanced illusions. “The general consensus is that Super Bowl halftimes last 20 to 30 min… roughly 12 to 14 min are reserved for the artist’s performance. Time taken to prepare and dismantle the on-field stage, as well as on-field warmups by the teams, must also be taken into consideration [1].” Now imagine all that is involved in rehearsals, setup, performance, and teardown. The most basic electronic technologies ensure that the performance is projected and enjoyed throughout the stadium while being broadcasted to millions around the world, including high-quality sound systems, mic’ed singers and musicians, leading-edge cameras, and technicians to ensure balance and synchronization. Recorded tracks are simultaneously broadcast, along with real-time digital sampling and mixing. In the YouTube video [2], Hayley Collett, Assistant Director Script and Continuity Department, calls shot numbers and camera numbers while counting the musical beats for the camera team to follow. Now pivot for a moment to a world-class performance organization, Drum Corps International (DCI) — often referred to as “Marching Music’s Major League” [6]. There are many similarities between a Super Bowl show and a DCI production. They are both performed on a football field and are, on average, about 13 min in length, with similar, stringent setup and teardown parameters. Productions are highly visual and frenetic, where musicians of the World Class competitive category are largely university music majors, plus “color guard” consisting of dancers who expertly handle several kinds of props, spinning, tossing, and catching them for visual effect and synchronization with music. A unique aspect of DCI is that visuals require athleticism on par with professional sports teams while performing on instruments and handling props. Production broadcasts are similar to Super Bowl shows behind the scenes [3]. The reader is encouraged to view many samples of DCI productions provided [7, 58–76] The drum and bugle corps activity (aka “drum corps”) evolved from the 1920s as paramilitary performance organizations — sometimes referred to as “field music” — in the wake of WWI. Today, they are regarded as elite “Marching Arts” performance ensembles. While this inquiry addresses technology in the performing arts generally, DCI ensembles exemplify that which drives technology enhancements to benefit student performers, audience, adjudicators, and production engineers. Show design is constrained by time and space for setup, testing, performance, and teardown, all within an extremely short period of time. “Whenever I try to explain the time constraints for deploying a sound system that is supposed to cover half of a football stadium [to audio pros outside of DCI], it’s always met with laughter” [80]. DCI was formed in 1972 as performing and competing units of the drum corps activity sought to self-govern, breaking away from protocol that celebrated military pageantry but restricted the activity’s musical and visual creativity. Because the

Learning and Performance Technology for the Performing Arts

153

paper serves to address learning and performance, DCI provides an excellent microcosm of a program that fosters performing arts excellence by students and budding professionals, while attracting tens of thousands of fans to the activity. Many of the interviewees for this paper got started in the performing arts in drum corps, were instrumental in its evolution to the high standard it enjoys today, and are currently engaged in theater productions, composition, visual design and choreography, broadcast journalism, audio engineering, film production, and sports medicine. Several are Grammy Award nominees and recipients. In addition to a life’s work in learning and performance, the author has been a journalist in the activity for 25 years, covering drum corps performances and competitions worldwide [4, 5] and an active performer, arranger, music designer, and administrator over six decades. DCI productions also provide an excellent laboratory for this inquiry because they have morphed from simple 13-min military drills on a football field, accompanied by brass and percussion (bugles and drums) and a military color guard, to pageantry that rivals a Super Bowl halftime show, a Cirque du Soleil performance, or a modern stadium rock concert. Through its 50-year evolution, the activity has assumed many of the characteristics of Broadway productions, with elaborate costumes (versus military uniforms), ancillary percussion instruments (vibraphone, marimba, tympani), and props that transform a football field into a performance stage that helps tell a story. Central to that transformation is how technology has found its way into the activity. Audio engineering now plays a key role in productions, the aim of which is to enhance the fan experience, present competitive elements more clearly to adjudicators up in the stadium press box, and to accommodate livestreams and live movie theater broadcasts through FloSports/FloMarching [11] and Fathom Events [12]. In addition, high-quality digital recordings are prepared to commemorate championship events, which require additional audio and video engineering. Like workplace learning and performance over the years, DCI ensembles have struggled with how best to integrate technology. Traditionalists decry anything synthetic, especially around audio engineering, but also costumes, physical props, staging, and choreography. All such enhancements are comprised of technology, electronic or otherwise. The dichotomy is around the organic versus synthetic, the conventional versus progressive, and what are sometimes considered unfair technological enhancements to performance. It is the struggle between focus on organic musical performance and individual acumen versus elaborate production design that marginalizes individual performance in some respects. DCI productions provide a mirror of workplace learning’s evolution over a similar time period with respect to the struggles, advances, and so much trial and error to get the right learning and performance (L&P) processes right. By analogy, think of adjudication results and fan response as measures of overall workplace business performance, while the means to such ends falls squarely on individual competency. Like workplace learning and performance, there are individual performance evaluations and then there are annual reports. It is often the disparity between the two that we need to question and normalize.

154

G. J. Dickelman

Consider the world of work as a total production, not just the individual or department charge. In the world of work, people touch multiple, disconnected applications in the course of a day. A realistic journey map includes discontinuous processes and isolated task nodes. Learning content productions are often incomplete, as they focus on an isolated system or process. If cognition is distributed, then L&P productions need to include and integrate the artifacts of work if it is to be effective. If the goal of learning content is to enable competency on a new application and an appreciation for its utility on the job, all toward adoption, then the content needs to reflect how the application is used in the work context. There are many more learning design considerations when the application is used for only 10% of the time within a workday. How might we change the L&P strategy when the total show is considered? Like performing arts productions, work is a dynamical system. L&P design cannot simply address a snapshot in time. Disruptive events are followed by unintuitive transients before a system reaches its new steady state. The system state may take a nose dive before increasing to a new, stable level, depending on the range of control variables. What happens if the design model is a snapshot within such a transient? You might win a competency battle but lose the performance war. Tasks are not goals [35]. You cannot model that which is dynamic and nonlinear with myopic linear techniques. Reexamining performance from the perspective of performing arts design and production is therefore an L&P imperative. Readers not familiar with DCI productions are encouraged to view samples through the DCI Web site [6] and a plethora of YouTube videos, many referenced below. Also consider Winter Guard International (WGI) [29], marching band competitions throughout the world, or show bands of historical black colleges and universities (HBCUs), as depicted in the popular 2002 movie Drum Line [10]. While field productions are similar, DCI presents additional challenges — and opportunities for technology support — because of a demanding three-month touring schedule throughout the United States, athleticism central to visual performance, and evolving trends around applications of technology to enhance performance. Furthermore, membership is not from a single school or location. Like professional sports teams, DCI performers hail from around the globe. During the off-season (September through May), ensembles rehearse music and basic skills one weekend per month and then spend about a full month at a suitable facility for spring training, where they learn the field production before the competitive tour. A variety of technologies are applied for remote auditions and performance exercises between offseason rehearsals. In some cases, live, monthly rehearsals have been eliminated in favor of virtual activities, particularly through necessity and lessons learned from the SARS-CoV-2 pandemic. Noteworthy for this paper are the DCI performance adjudication categories, including General Effect, Visual Proficiency, Visual Analysis, Color Guard, MusicBrass, Music-Percussion, and Music Analysis [76]. DCI World Championship Finals 2022 were held on August 13th at Lucas Oil Stadium in Indianapolis. The event draws around 20,000 fans to stadium concert side, is broadcast to theaters via Fathom Events [12], and into living rooms via FlowSports/Flow Marching [11].

Learning and Performance Technology for the Performing Arts

155

Both performance and production design are integral to adjudication. The show has to be well designed musically and visually and performed with excellence to earn high scores. For each category, there is an evolving set of technology enhancements for individual and ensemble rehearsal and performance, plus tools for production analysis, design, and development. Equally important are the tools and techniques that help communicate the show to the audience for the best visual and auditory experiences. Not only are there brass and percussion musicians and color guard, but audio engineers and prop crews that make the magic happen. Ensembles consist of 160 performers, plus about 100 design, instruction, and administrative professionals, plus scores of support volunteers. An average annual budget for a DCI World Class ensemble exceeds US$2.5 M. Why focus on DCI, a mostly unfamiliar activity to L&P professionals? Because it pulls us out of our comfort zone, disrupting tacit assumptions and resulting human error born of immersion in the discipline. To err is human. Sometimes we need a device to help us avoid those very human capture, activation, description, and mode errors [45]. Hopefully questions raised by this inquiry will enable L&P professionals to stick their heads up above their familiar world and see possibilities that only other disciplines and activities can inform. As DCI transitioned from local youth activities of military pageantry to a performing arts (non-profit) business, there were growing pains. Boards of directors were filled with seasoned drum corps people with little or no business experience, as their organizations faced significant business survival challenges. Recognized as a huge competency gap, many ensembles eventually filled their board seats with business people from outside the activity to stop economic bleeding and loss of ensembles that could not step up to the new touring requirements.

2 Status of Learning and Performance There seems to be a growing trend in business that is marginalizing the value and benefits of the L&P practice. Peruse L&P job openings across LinkedIn and it’s easy to find thousands of applicants for each of a very few job openings in the discipline. During the pandemic, many organizations reduced or eliminated either the number of L&P employees or the entire practice. This was especially true in the hospitality and travel industries, which were devastated by social distancing concerns and restrictions, plus fear of death and illness. Focus seems to have shifted toward data analytics and AI for basic business survival. As businesses emerged from the pandemic, such focus has only increased, leaving L&P as persona non grata, with dire consequences [46, 47]. Consider Gartner’s top 10 strategic technology trends for 2023: Digital Immune System, Applied Observability, AI Trust, Risk and Security Management (AI TRISM), Industry Cloud Platforms, Platform Engineering, Wireless-Value Realization, Superapps, Adaptive AI, Metaverse, and Sustainable Technology. To be sure, L&P may benefit from these trends and initiatives to some extent, but is largely left

156

G. J. Dickelman

out of strategies. L&P is grounded in IT, as its constituency is primarily learning and performance technologists. Does L&P have a seat at the IT trending table anymore? Not so much as it truly requires. This inquiry therefore also aims to start conversations around reinvigorating and refocusing the value and benefits of L&P in the face of technology. L&P science and practice are business imperatives. We need to identify where modern technology intrudes on that which is fundamentally humanity centered. When we design new things to address challenges in the world, the world, in turn, designs us [33]. It changes our point of view, often for the worse. Let’s reel L&P back to where it belongs, or at least, let’s not throw the proverbial baby out with the bath water.

3 An Ad Hoc Inquiry Interviews were conducted with nine performing arts professionals with credentials in music composition, performance, production design, audio engineering, broadcast journalism, publishing, and sports medicine. All but one has direct experience with DCI productions. We discussed learning and performance technologies, identity and tradition versus art and innovation, organic versus synthetic performance, and what comprise the good, the bad, and the ugly applications of technology, including the range of artifacts from the non-electronic to the artificially intelligent. The purpose is to begin reexamining performing arts as a means to stimulate thinking that advances L&P protocol in the workplace. Brenda Laurel regarded Computers as Theater in her groundbreaking 1993 work by that name, revised in 2013 [34]. It was a clear performance support enabler, on so many levels. As a result of her work, dozens of online learning development tools emerged that appropriated theater terminology: actors, cast members, stage, track, storyboard, etc. The L&P lexicon also shifted accordingly. But the performing arts persona back then was quite different in many respects from its contemporary counterpart, largely because innovative artifacts — audio and video technology, space-age materials, props, illumination, special effects, broadband, WIFI, GPS, smart phones/tablets, BLE/UWB technologies (for precise positional guidance), and AI — have found their way into performing arts learning, performance, and design. The following questions provided a framework for discussion during each interview and a means to uncover professional opinions, practices, and trajectory of L&P technology in the performing arts: • What do you see as the most effective uses of technology to support and enhance performance? • What is the value of such technology for students/performers, teachers (instructors, composers, arrangers, caption heads)? • When do you feel technology-facilitated performance is authentic versus artificial?

Learning and Performance Technology for the Performing Arts

157

• Do you foresee or have you experienced applications of AI, machine learning, or data science in the performing arts? • Do you think that virtual rehearsals can produce performance results comparable to live rehearsal? • What is the direction of technology-facilitated performance in the performing arts? • Are there unintended consequences of technology in the performing arts? Our conversations did not move through these questions lockstep, as one would on a survey. Rather, the conversations were free form and open, touching on the many topics inferred from the questions. In this regard, learnings from the interviews are largely informal. Besides, the questions as written assume a great deal about one’s familiarity with L&P in the workplace. For example, it was about 10 min into one conversation before I realized that the interviewee interpreted the word learning in learning technology as a verb rather than an adjective (“I am learning [to use] technology” rather than “I apply learning technology in my work”). Better that this journey affords the advantages of informal learning [41].

4 Technology that Supports Learning and Performance While DCI member ensembles are considered performing arts education programs, it is performance quality, fan response, and adjudication that foster performance excellence. How we measure success comes into question when performance is not organic but mediated and enhanced by technology. People become suspicious when unfamiliar artifacts appear in performances. As in any competitive discipline, innovation rears its head as disruption to the status quo — like “a disturbance in the Force” — which leads us to reexamine rules and practices. Performance takes on new meaning when technological artifacts are included in the mix. In the workplace, technology is introduced and democratized to expand capabilities, get more done faster, and sharpen focus on real data and business trends. That sort of innovation draws attention from competitors, occupational safety folks, IT security, human resources, financial regulators, and more. Reactions to technology in the performing arts is similar. Bear in mind that tech innovation not only changes processes but changes us as well. “The things we design — the artifacts — change the way we behave and act so that, just as we have changed things through our designs, those things then change us, affecting how we behave and live: we design the world, and it, in turn, designs us” [33]. It answers the question, “Why does our artificial world seem so natural?”. Brass musicians on the back sideline of a football field who are now mic’ed no longer need to increase volume and expression above what is musical to connect with the audience. But there was a time when bending the bell of the horn — overblowing in a very non-musical way — moved fans and even earned high scores. In the early days of drum corps, performances were metaphors for military battle and pageantry;

158

G. J. Dickelman

drums, bugles, rifles, sabers, and flags were ordnance. Today it is more akin to a Broadway production, where each ensemble strives for artistic acumen. In this regard, technology clearly enhances both learning musicianship and performing with excellence. And when students return to their college and university music programs after the DCI summer tour, their orchestra educators no longer have to re-teach how to be musical. But there are always unintended consequences around innovation, often born of competing influences. When an ensemble reveals a novel, technology-enabled feature, how is it received by fans, adjudicators, and production engineers? It may bring the fans immediately to their feet, but the general effect bump may be offset by a lower music analysis score. And maybe the effect simply won’t come across on a recording without post-production enhancements. Why the disparity? Have we lost focus on who is supposed to benefit, namely the student performer? Technology as automation has always raised fears around dumbing down the workplace, lowering the bar on human performance excellence, marginalizing the knowledgeable, and devaluing conceptual thinking. In reality, it mostly redefines what workers do toward more pertinent and productive endeavors, in the same way that agriculture and domesticating animals freed people from the burdens of hunting and gathering, ultimately enabling intellectual pursuits that led to industrial and technological revolutions. But there are certainly cases, as in early education, where technology causes neural development atrophy by limiting fine motor skill development [36]. Self-directed activities with technology, like game apps driven by algorithms that inform predictable outcomes, do not necessarily provide open-ended, analog exploration that a dramatic play area, say with blocks and other physical artifacts, provide [37]. Many such unintended consequences follow us into adulthood. “Mic’ing and amplifying are good examples. It was pretty experimental in the beginning but has gotten a lot better. Today performance quality in DCI is much higher because designers have figured out how to better project and appeal to the audience and to the judges, although I think there is often more of an effort to appeal to judges than there ought to be” [84]. We too often consider the advantages that new artifacts afford as unfair, even cheating. “You’re experiencing a manufactured representation of what is actually being performed by musicians in the stadium. It’s not just louder. You’re hearing something the technology has manipulated” [77]. What is actually being judged, either formally by adjudicators or by fan applause or disapproval? Is it the art or the performers, or both? “I say if a show is entertaining and moves people, then it’s good, no matter how you get there. Making someone clap involuntarily takes both a designer to create and performers to pull it off; you can’t have one without the other” [80]. On the other hand, “Cheating is cheating. Mic’ing good players in an ensemble and distracting from real playing by the entire ensemble is unfair manipulation and deception. Just like the Houston Astros sign-stealing scandal” [77], which crossed the line when the Astros used a video camera in the center field seats to film the opposing catcher’s signals to the pitcher [44]. Interviewees frequently invoked sports metaphor and simile as we addressed art, performance, adjudication, rule bending, and more [77]. It is most appropriate for

Learning and Performance Technology for the Performing Arts

159

this inquiry, especially since DCI adopted the moniker “Marching Music’s Major League,” to emphasize athleticism in its productions [63], even engaging sports personality Curt Gowdy to join DCI’s broadcast journalist Steve Rondinaro [83] for commentary in its formative years. Sports medicine is also integral to DCI ensembles and many other physically demanding performing arts. Technology associated with this domain has become essential. “Athletic Trainers (ATs) are professional healthcare providers who specialize in sports medicine. We focus on preventing injury and illness, apply first aid and emergency care, evaluate and diagnose, provide therapeutic rehabilitation, and generally promote good health and wellbeing. Our job is to ensure that patients (DCI corps members, student-athletes, professional athletes, etc.) are in their best physical condition to deliver optimal performance. “When someone gets hurt, we look for signs and symptoms of what the chief complaint is. First we need to look for any signs of obvious deformities or fractures and then work from there. If a tuba player complains of shoulder pain, we figure out a treatment plan for them that may include stretching and shoulder strengthening exercises outside of rehearsals. If a percussion player complains of forearm and/ or wrist pain, we diagnose first, then apply treatment to alleviate pain and reduce symptoms. “Sometimes we use technological devices, depending on the injury and pain level. They include kinesiology tape (KT), other varieties of tape and pre-wraps, Transcutaneous Electrical Nerve Stimulation (TENS) units, ultrasound, moist heat packs, ice packs, muscle and joint pain relief creams, and orthopedic braces” [32]. Rebecca Cole [32] continued to explain how critical the need is for athletic trainers in school sports and other physically demanding educational programs, like DCI ensembles. Rehearsals typically begin with stretching to warm up muscles and ligaments and conclude likewise. As the complexity of DCI productions has increased with so many props and things being tossed in the air, and upwards of 160 performers per ensemble moving around frenetically, the risk of injury has risen dramatically over the years. More spectacular, complex visuals are critical to competition. Ensembles cannot get production design across when performers are missing because of injuries. DCI is therefore heavily invested in performer health and safety. Ensembles employ seasoned athletic trainers for rehearsals and tours. But generally for athletic trainers, unless you work for the NFL or other wellfunded organizations, there will be one trainer for huge numbers of athletes in multiple sports, and there won’t be very much high tech. While many effective technologies are available for sports medicine, not many organizations can afford them. Leading-edge technologies include those for preventative genomics, sensors and wearables, exoskeletal devices, antigravity treadmills, dynamometers, plus applications of virtual and augmented reality (VR/AR) [56]. From tape to thermal therapy to hyperbaric oxygen chambers and VR/AR, technology has changed and continues to improve how performers recovery from physical injury. Ergonomics and assistive technologies have become integral to keeping knowledge workers injury free. The more we sit, the more things we carry and move because of supply chain demands (think Amazon warehouse workers), and the more

160

G. J. Dickelman

the workplace demands the physical, the more imperative athletic trainers, physical therapists, yoga sessions, and massage therapists are to health and safety. Attention to physical aspects of work is critical to L&P and should be included in program design. Do we look at workplace L&P as a complete production? Can we draw a clean line from the activities, investments, and outcomes of L&P to the top line or the bottom line? Are we judging success by how the people who do the work perform, by the design of a top performing enterprise, or both? How do we ensure parity of objectives and success factors? Does the world judge things on the basis of individual achievement or the art and elegance in design that results in a healthy bottom line, brand recognition, and a working organizational mission? Or perhaps it’s impossible to separate the two: “…you can’t have one without the other” [80]. “There’s a level of performance maturity and professionalism that result from a student [performer] maneuvering around a field or stage while mic’ed. It’s not just hitting your dot and performing at the right time, but being aware of everything going on around you. When you become professional, say in a jazz ensemble, a recording studio, Broadway, and more, you need to learn how to perform with microphones. It’s not always about playing as loud as you want” [80]. “In 1969–70, I was able to use audio recordings [of top drum lines]…to listen and apply new skills. I’d pick up what was being played audibly and take it to my hands. Visual learning happened at drum corps shows. We did not have video or computers back in the day. Dad brought me along when he judged, where I got to watch and listen at the same time. Those drummers I heard on recordings definitely set a bar of excellence for me that I strove to match, as primitive as that technology was. Many new tools and techniques to aid learning have emerged since then, so learning technology is a work in progress. We adapt and take advantage of them. I think the line falls between where the technology benefits the learner/performer and where it is actually a hindrance. Again, it’s a work in progress” [81, 82]. “I’m thinking of WGI competition which is performed in gymnasiums. You’re inside so you can evaluate things in that environment more directly than you can on a football field. One way you can look at the DCI activity is field music and visual/guard where performers are so spread out. The overall sound is different inside a building as opposed to outside on a football field or very large indoor stadium…electronics [used inside a gymnasium] could allow for use of screens as part of a performance. Think of the eight screens to view like they do for team evaluation on the NFL Red Zone Octobox [30]. “But what happens if those electronics go out? Performances are constrained by time, and the time to fix a problem may result in timing penalties. What happens if you can’t project that special visual feature that depends on technology? What would be the explicit and implicit deductions? What happens if the audio engineer doesn’t get the sound right? What if a key solo doesn’t get up to the box for evaluation by judges, is distorted, or the mic cuts out? Where do you draw the line for adjudication? There should be criteria in place and possibly a deduction range to provide for evaluation under those circumstances” [82].

Learning and Performance Technology for the Performing Arts

161

Question: What about virtual rehearsals and the tools that have emerged because of the pandemic? “If it’s only you playing locally but in some amazing listening environment that creates a feel for the other performers, there will still be gaps as compared with live rehearsal. For example, an RCA catalog includes instructions for placing a single 77DX microphone for recording a symphony orchestra. How is that possible? It’s because of the placement and expertise of the musicians and skills of the conductor. The softest instruments, like oboes and flutes and violins, are near the front, and the loudest, like brass and percussion, in the back. A good conductor manages the dynamics, reducing forte to mezzo forte and the like so there’s a better organic blend. That happens when a group of good musicians play together, and everybody finds their place within the ensemble. They make the room sing. You can’t possibly get that in a virtual rehearsal. You can get value from it, but it doesn’t mean you create the intuition and reflexes to play a gig together. It’s like the difference between crossing your arms to comfort yourself and hugging the love of your life. Not the same” [77]. “I’m in a little niche where I’m not a performer but from my perspective—like in the studio where we transitioned from analog tape to digital—it’s a big deal. There are many who still prefer analog sound. It’s not so much about technology as it is aesthetics and preference. A lot of people still haven’t completely wrapped their ears around digital…With apps like ProTools [39] and Melodyne [40] you can manipulate audio in ways you couldn’t even dream about before. I’m thinking about how we used razor blades to cut out pieces of tape and splice them into a project. Tech tools are certainly time savers as you no longer have to do retakes to correct pitch, for example. A good engineer who knows the tools can do digitally in minutes or seconds what used to take hours and days to do with analog” [79]. “I remember writing drill by hand on paper and using a light board to see the page you were on before. I started back then and then experienced Pyware and then the amazing Pyware 3D where you can see your visual design in a real 3D view [24]. I also use Finale for music notation [38] and the whole online Zoom thing. So I’ve progressed through using technology tools for the marching arts, sometimes kicking, and screaming, with a huge learning curve” [78]. “We’re able to evoke more emotion since technology was introduced. Years ago in drum corps there were no mallet players [vibraphone and marimba]. Then they were allowed and potentially provided additional texture to a production, but you couldn’t hear them past 10 yards. It was uninspiring to just see hands move but not hear what they were playing. Amplification was proposed and ultimately permitted and your ability to really mix and balance emerged. When I started directing theater, amplification wasn’t quite where it is today. Actors had to project strongly, or the audience would lose the storyline. Dialogue between actors in a performance are hand-offs that need to be clear and concise, so you would also lose track of the plot if you missed such an exchange. Imagine watching The Godfather but not hearing Marlon Brando. Would you understand the movie at all? You have to ensure that actors project to the audience. It’s about angles and position as much as volume. With amplification, good microphones, and headsets, some things go more smoothly,

162

G. J. Dickelman

while other [unintended] things happen. You might see a microphone mouthpiece in a fifteenth century scene, or you might amplify things that you don’t want the audience to hear” [81]. “In the theater the back row needs to be heard as well as the front. Early body mic’s were largely unreliable and intrusive. Today they thread little mic’s through the actors’ hair and high onto the forehead, or use lavaliers and headsets. But you can still see them from some distance. We are expected to suspend reality when the actor is supposed to be an historical figure, but it bugs me. There will hopefully be a next generation improvement but that’s where we are now” [79].

5 Identity and Tradition Versus Art and Innovation “During the height of the space race in the 1960s, legend has it that NASA scientists realized that pens could not function in space. They needed to figure out another way for the astronauts to write things down. So they spent years and millions of taxpayer dollars to develop a pen that could put ink to paper without gravity. But their crafty Soviet counterparts, so the story goes, simply handed their cosmonauts pencils…But, alas, it is just a myth” [50]. It is nonetheless a great story to contrast the simple and elegant with design fraught with complexity and cost. In reality, according to an Associated Press report from February 1968, NASA ordered 400 of Fisher’s antigravity ballpoint pens for the Apollo program. A year later, the Soviet Union ordered 100 pens and 1,000 ink cartridges to use on their Soyuz space missions…[and] both NASA and the Soviet space agency … paid $2.39 per pen instead of [list price of] $3.98. … Paul C. Fisher and his company … invested $1 million to create…the space pen. None of this investment money came from NASA’s coffers — the agency only became involved after the pen was dreamed into existence [50].” The false space pen story is great for those who wish to underscore technology bloat and unnecessary complexity, but there were clear benefactors of Fisher’s efforts and investment. Not only did the U.S. and Soviet space programs benefit, but there were wider applications for undersea exploration and expeditions to frigid areas. Hearing about something versus actually seeing it results in a very different experience. “I never thought I’d be a fan of an electric cello in drum corps, but I was. They [the Troopers of Casper, Wyoming] used it masterfully and it contributed so much [69]. Without amplification, electronics, and technology, that show never would have happened. I have to eat my own words on that count and that’s what makes it such a challenge to figure out how to manage integrating technology…I have mixed emotions with all of this as an old timer, a purist if you will, but one who was gassed out when the Bluecoats pulled off that ‘pitch bend’ [67, 68]. The first time I heard it I said, ‘Wow, cool idea!’ Then I wanted to find out how they did it and was mesmerized. That opened a door to many other such innovations. Bluecoats do it

Learning and Performance Technology for the Performing Arts

163

extremely well, but they’ve got top rate audio professionals. Not everyone has those resources, so it potentially creates an uneven playing field, and that bothers me” [83]. “There is nothing, nothing, nothing I love more than the full throated punch from 60-plus finely trained brass players with percussion driving them forward [58]. When you start amplifying you’ve got to be really careful, or you take away from the purity of that experience. To me it is not necessarily better and sometimes just louder for the sake of being louder because we can” [83]. “I think doing it the same way doesn’t always work. Without the changes we’ve seen over the years the activity would have died a long time ago. But sometimes I feel like they go too far with trying to jump ahead, especially when it comes to financial impact. All of the ensembles want to keep up with the top corps’ props, for example. You know it’s very expensive, often requiring another semi just to carry the props all summer. I’d almost like to see some way to level requirements so that everybody is at the same place. In competition you want a level playing field. I know of some school music festivals where everybody plays the same arrangements for adjudication. Thankfully that’s not the case with drum corps, where the shows are completely different from one ensemble to the next and different every year” [84]. “[Technology] shifts adjudication from performers to the electronic engineering, which calls into point that scores do not reflect the performers. Also, what happens to scores when the electronics fail in the middle of a show?” [83]. Question: Do you think it’s appropriate to apply technology, like in-ear metronomes, to help performers stay synchronized when spread across a 100 yard football field? “Try to synchronize many musicians acoustically and electrically across a 100 × 40 yard field. It’s a pretty human requirement at this point because of sound delay and trying to be surgically, rhythmically accurate across that span. The problem is peculiar to marching band and drum corps. A Broadway show doesn’t have the same challenges. Performers are all together and they’re able to listen to each other and play together. Jazz musicians at Lincoln Center or the Blue Note are able to listen and communicate, homing in on each other and just being great musicians. You have to be an intuitive performer as you move across the field…Technology has not fixed that yet. You would need a smart metronome in your ear that was able to react to where you are on the field…and it would have to graphically map the field and be aware of your position to adjust metronome speed. We’re just not there yet. [80]. Author’s note: Technology to address such problems clearly exists, like what’s behind Apple Air Tags [26] and posture-correcting apps [27]. Incentives to create a marketable solution and obtain cross-discipline cooperation to get there are quite another matter. “I honestly hope not. I want to see a group do exactly that with spatial relations in the raw, do it beautifully, and be rewarded for it as they should. I’m thinking back to some of the most memorable moments of over the years, as when Santa Clara Vanguard, Phantom Regiment, or Blue Devils would turn backfield and play soft and use that stadium as an amphitheater [71]. I get goosebumps just thinking of those moments. It is unadulterated acoustical sound born of fine playing and incredible musicianship” [83].

164

G. J. Dickelman

“We still need ensemble and individual skill evaluation as part of field performance. It’s those very elements of what and how that make the marching activity unique. Let’s give performers’ efforts and what they are playing a fair, deserved shot. What and how are very much valid in EVERY section of a performing ensemble…that means percussion, brass/winds, and guard/visual ensembles. Visual or movement today require everyone to contribute in ways that were not possible back in day. As an example, take marching drums. They were secured only by slings and leg rests. Current marching drummers now use a more stabilizing technology like a harness that allows for more control over what and how. This also allows the marching battery to move in ways that were not possible before. This is a fantastic opportunity for visual enhancement, and I applaud it! However it should not reduce or disregard the elements of music evaluation. Taking judges off the field does the music caption a disservice, in spite of the remarkable things that it affords the visual caption” [82]. Noteworthy is a Marching Roundtable podcast in which drum corps percussion legends lamented the decline in intricate rudimental skills during almost two decades, when body movement and visuals were demanded of and evolved in the battery [51]. In simple terms, marching quickly or running for visual effect while carrying a drum made it difficult to play the more difficult, intricate things that were always trademarks of rudimental drumming. Many of those skills disappeared through this period, but eventually reemerged as the battery figured out how to perform difficult choreography and play sophisticated passages at the same time.

6 Organic Versus Synthetic Performance “The topic of amplification on the field is nuanced and complicated. The performer may be mic’ed but they still have to be able to play in tune. Thank goodness we don’t have real-time autotuning technology on the field. I do have issues mic’ing individuals and amplifying as if they are the whole section. In the early days without any of this stuff, your woodwinds sounded like your woodwinds, you put them in the right place and staged them properly as part of the fun and artistry of the whole design process. Technologies now allow us to perhaps have more freedom with staging, where the audience hears the production from wherever they are… I was once very disappointed when I sat down close to a corps I love and all I heard was coming through speakers. The technicians corrected that in subsequent shows, but getting it right is a real issue. I’m not saying we shouldn’t use technology, but we should apply it a little more artfully, carefully, and thoughtfully. Just because you can do something doesn’t mean you should” [78]. “Performing with the JAMMitors at Disney [31], I’m thinking of what makes an ensemble unique. You can do organic things that are much more effective than applying electronic technology. Depending on the time of year, we were sometimes custodians, sometimes gardeners, sometimes chefs. We used different size metal or aluminum buckets that sounded like pitched instruments. We even cut the bars on a ladder different sizes to produce a range of pitches. We’d use trash can lids and

Learning and Performance Technology for the Performing Arts

165

turn our props into a drum set. We used a plunger as a microphone. I used toilet brushes on an oil pan to produce the effect of brushes on a snare drum. Please keep in mind, our environment also dictates that we perform on certain items without the use of electronics or microphones. There were instances where we were placed on a stage, far from an audience. In that case, it was beneficial for the group to have microphones and a mix that was more pleasing to our audience. Also, there was trial and error for use of technology through rehearsals that ultimately provided a best performance scenario” [82]. The dichotomy between technology-enhanced performance and organic performance continues to rear its head. What is fair innovation and what is not? In an effort to close the learning-doing gap, we often apply technology that promises to accelerate the ability to perform. The sleight of hand is that such tools do not make it possible to grab an arbitrary person off the street and have them performing and productive immediately. Such hype was heard around enterprises in the early days of performance support. The best of such technology simply reduces the time to competency and eliminates error. Atul Guwande’s seminal work, The Checklist Manifesto, addresses unintended consequences born of complexities in the medical operating room, like inadvertently exacerbating a condition or tragically causing death. The checklists he describes that substantially reduced such unfortunate outcomes are successful because they are developed by the people who do the operating room work and designed to keep their arms around massive amounts of technology in the room, give authority to more than just the surgeon, and allow a team to pause and scrum when there is concern about the next step [23]. Such checklists and their automation are precisely the kind of things that benefit performance, as the stage becomes more and more crowded with technological gizmos. Technology at its best subsumes process and timing, adding efficiency and freeing humans to do more pertinent things. Performance support technology does not instantly transform cashiers into nurses, doctors, or anesthesiologists, but it reduces a medical professional’s cognitive burden and risk by taming the complexities of equipment, authority, and responsibility, all for improving medical care and practice. The same goes for technologies that enable performing arts. The (non-electronic) technology, that is the checklist, does not instantly conjure medical staff expertise, but increases the ability of highly skilled people — and those developing those skills — to better manage the operating room experience [23]. If cognition is truly distributed, then all process artifacts need be included in the things that make us smart and help us to perform with excellence. In so many ways, Guwande addresses the human side of performance and design similarly to how Don Norman sees the world. The key to changing the mess the world is in is through “meaning, sustainability, and humanity-centeredness” in the things we design [23, 33]. It’s about making the complex easier to manage through careful arrangements of artifacts. Look around any hospital these days and you’ll see checklists on walls. A familiar outgrowth of this practice is how nurses, doctors, and anesthesiologists each initial the exact body part on which they are about to operate before you are wheeled into the operating theater. An artifact so simple has prevented scores of incorrect

166

G. J. Dickelman

amputations. In a related example, ophthalmologists place a colored sticker above the patient’s eye that is about to get laser cataract surgery. “I have been a Barrymore Awards reviewer for many years, the Philadelphia counterpart to New York’s Tony Awards. There are about eight categories to score on the sheet, and many are influenced by how microphones and speaker systems are configured. Balance is so important between vocals and music, for example. Maybe they are picking up background talk and I can’t understand the score. Proper use of technology can really improve evaluation… I’m also a judge for high school jazz bands. I love helping the students get better and better. But as amplification has been introduced, we’ve had to allow each band—sometimes with as many as 15 in a competition—to do individual sound checks. We used to do a single check of the room for all, but that did not permit groups to fine tune setup for the best audience and adjudication experience” [81]. How inconvenient or cumbersome managing innovation might become, it informs our willingness to employ it. Early adopters will tolerate it, in spite of product clunkiness. It is not until it crosses that chasm from early adopters to majority use, when improvements make it convenient and unobtrusive, that its domain will feel its full utility [2252]. If only select techies who pride themselves on juggling multiple variables and solving difficult problems in their heads are able to master the technology, it will have limited application and adoption. People will revert to what they know to be successful, rather than risk a failed outcome because the technology that was supposed to help got in the way. In this respect, most humans tend to fall in the identity and tradition camp. Remember Grudin’s Law: “When those who benefit are not those who do the work, then the technology is likely to fail or, at least, be subverted” [57].

7 The Good, the Bad, and the Ugly Tech in Performing Arts Nine one-hour discussions by performing arts practitioners have parsed technology’s role in L&P into these three categories, summarized here: The good • • • • • • • • • •

Expands opportunity for innovation Enhances audience (customer) experience and appreciation Improves communicating performance elements to adjudicators Increases musicianship and performance acumen Affords merchandising commemorative recordings and swag Expands the audience (customer) base through livestream broadcasts Obviates tedious tasks in design, composition, and content distribution Closes the learning-doing gap Benefits the student performer Provides formative improvements across the entire activity The bad

Learning and Performance Technology for the Performing Arts

167

• • • • •

When technology is applied for technology’s sake Trial-and-error glitches and risk that diminish fan (customer) experience. Dilutes individual and small ensemble adjudication by taking judges off the field Shifts organic student learning and performance focus to production designers Substantially increases costs (equipment, transportation, logistics, risk mitigation), especially around touring requirements • Creates disparity of intent. Is it for the student performers, the fans, the adjudicators, the designers, or the production engineers? • Adds an enormous amount of complexity to all aspects of productions and organizational management The ugly • • • • • • •

Crossing the line from innovation to deception, even cheating Further opening the divide between the haves and the have-nots Substantial reduction in the number of competing ensembles Learning and performance opportunities in the activity for fewer students Loss of tradition and identity High cost of event tickets driving fans away It not only changes what we do, but also changes the way we see and evaluate things

“When an ensemble has requisite expertise, money, and power to harness all of those things in a show it can be a fantastic experience. It’s so exciting to have costume changes and props that move and change, and there’s amplification, but as a band director you’re sweating it out on the sidelines. I’ve seen so many show disasters for everybody, including the audience, because something failed. You have to balance risk and reward. Besides, most bands are small and don’t have much money, so band directors just thrown up their hands and say, ‘Forget it’ to all those other layers. They just want to get their band out there and do a good show on a Friday night for the crowd. The wealth gap in our country reflects the bands that are applying these layers with tech tools and reaping the rewards. But most bands don’t have all that. Perhaps they have a meager sound system to mic soloists, but often there isn’t even a drum instructor. Many bands are being left behind or they’re just saying forget it if I have to have props and show shirts and amplification and a huge staff to compete. We’re pricing ourselves out, so to speak” [78]. The trial and error that DCI ensembles experience is not unusual. Consider the analogy of a blind person on a park bench manipulating a Rubik’s Cube and, each time, turning to the sighted friend beside him, each time asking, “Is this it? Is this it? Is this it?” Blindness manifests in not knowing what the consequences of technological support will be to an audience or adjudicators. Being willing to take risks and put an innovative idea forward to test the waters is as noble as it is human. Without such flexibility and risk taking, drum corps — like so many evolving activities — would have withered and died long ago. Put another way, the difference between success and failure in the arts is having the fortitude to weather just one more failure and press on with that spark of an idea that may just cross the chasm.

168

G. J. Dickelman

8 Ongoing Studies in Technology-Assisted and Technology-Enabled Performance Performing arts organizations are no strangers to digital transformation [15]. Their survival through the pandemic benefitted from many of the same strategies and tools as businesses in other economic sectors. Even before the pandemic, apps for art event management included data analytics modules for inventory, supply chain, venue utilization, ticket sales, booking, recruitment, and compensation [21]. These applications provide statistical measurements for financial and process management, prediction, and decision making. Although live attendance has not yet returned to 2019 levels, data analytics have been central to performing arts’ recovery. The data-driven-everything trend has resulted in tools for data visualization and intuitive graphical analysis. Basic and intermediate level analysis can be conducted by subject matter experts rather than requiring the most skilled data scientists. Consumers of data analytics get most of their reports automatically these days, while deeper analysis is address by so-called creators and advanced creator analysts and data scientists. Predictive analytics, for example, have begun to dominate the digital advertising space in the performing arts [48]. Survival has also been possible through hybrid performances that merge technology with live performance. That has helped maintain baseline employment and consumer access to some performance genres. For example, the Metropolitan Opera maintained operations, customer loyalty, and market mindshare by offering livestreams and on-demand performances. Rehearsal and performance have been met with obvious challenges through the pandemic around infectious disease protocol. While technology can be of some value to virtual rehearsal and performance, there are obvious shortcomings. “The physical gestures that chamber ensembles and other musicians enact in order to seamlessly perform ‘as one’ include cues such as a fleeting glance, a subtle nod of the head, or the rise of the torso. These physical gestures are the root of how musicians share vital information to one another empowering them to share ideas and stay open to improvisational interpretations” [14]. In the winter of 2022 the Visualizing Telematic Performance project conducted workshops on its findings in an effort to shift telematic performance from the limitations of video to a much larger scope of what is possible to close the gap between virtual and live performance [14]. Interviewees discussed how telematic performance protocol might aid students in accelerating learning brass synchronization and counterpoint when spread across 100 yards of a football field [74]. In the example, trumpets are staged between the left 15 and 40 yard lines, the mellophones (middle voices) between the left and right 40 yard lines, and the lower brass between the right 40 and 15 yard lines. That’s a spread of 270 feet, which translates to as much as a 0.3 s delay between performers, an eternity with respect to playing together. Consider the percussion, where there are about 40 yards between the battery (marching percussion) backfield and the front ensemble. The Little Fugue was perfected through a mostly organic process and

Learning and Performance Technology for the Performing Arts

169

hours of rehearsal, which could potentially have been shortened with a technology solution. “One of my mentors once took two Long Rangers [high selectivity sound devices] and stood on the front 50 yard line to measured how far apart tempos were between performers…He started one, then the other, and the gap was audible. He slowly walked on an angle and back until they synced, to show evidence that in order for the sum of all parts to line up, one group had to be playing later than the other, depending on the distance between them. Technology has not yet fixed that. You would need a smart metronome in your ear [28] that would also respond to your location on the field in relation to the grid” [80]. Wearables make certain technology invisible, unobtrusive, and therefore attractive to performers. They are applied to health, fitness, and entertainment, with abilities to monitor and transmit data for trends, communication, and sharing with others. The trajectory seems to be in miniaturization, specialization, and feature creep. Fitness monitoring, posture correction, and location services are just a few of the more popular trends with things like FitBit [54] and Apple Watch [55]. Technology has transformed the theater in four important ways. Sampling and mixing “help enhance and supplement the sound of the ever shrinking size of theatre pit orchestras. Moreover, musical instruments now rely heavily on digital tech to create the required contemporary instrumentations” [20]. Instrumental and vocal processing involves capturing audio and presenting it to an audience in a particular way. Rigging and moving large props are automated and timed, reducing technician requirements. Similarly for lighting. Social media also plays a role around connecting the audience to the performance. For example, in 2022 Carolina Crown introduced the Crown Live app, which engaged DCI audiences’ mobile phones to flash in sync with music, and to cue audience countdowns, claps, and other interactions [75]. Industry updates on the technology in the performing arts are detailed in an insightful series of articles [17, 18, 19]. “Commercial theaters and larger institutions have the budgets to invest in technology and automation, and their audiences expect it. Smaller theaters don’t often have the financial resources to invest in technology, but they usually have boundless energy and great intimacy. They also typically don’t have audiences that expect slick production. It’s the mid-sized professional theaters and larger community or civic theaters that are getting squeezed. They have overhead, which drives a higher ticket price and results in increased audience expectations. But those theaters have limited production budgets to compete” [17]. “Technology continues to evolve in all facets of American life, and these newly embraced technologies are finding their way into the theatrical space [16]. I think the three biggest areas of ‘revolution’ are LED lighting, motorized rigging, and active acoustics, which some call ‘electronic architecture.’ I’d also add digital visualization and projection technology. Not surprisingly, each of these technologies is built on the backbone of the digital/computer revolution” [18]. “I see more automated rigging or a combination with manual rigging, and active acoustic systems. I actually think active acoustics is still in its infancy in the performing arts space. Also, whether it’s a new-build or renovation, there will be

170

G. J. Dickelman

more early design collaboration to maximizing venue versatility” [19]. The article also discusses technology that connects performance with the audience.

9 AI, Machine Learning, and Data Science “The only data we really deal with include where the students/performers are from, where they have performed before, what is the average age. The older, more experienced groups are consistently meddling in innovative designs that change show design from one year to the next. I would think data would be helpful if we were doing the same thing over and over, but we’re not. It’s a work in progress” [80]. “I have experienced some AI programs where you can say something like, I want a composition that reflects both 1960s James Brown and Mozart. Such programs are replacing composers. You just tell it the influences that you want, it cobbles together the right characteristics, and spits out multiple compositions to choose from. But if you asked for a Miles Davis-like piece, for example, exactly which era are you talking about? You get some choices, and there you have replaced a composer and obscured the legacy of Miles Davis. You hear jingles on TV made with democratized tools like GarageBand or videos with iMovie—things that come free with your Apple computer. I’m a professional drummer, but was replaced by drum machines. Then there were synthesizers, and before that the Mellotron [42], which was a keyboard instrument that grabbed four-second sustained notes from a tape of real instruments. I remember the International Federation of Musicians Local 802 freaking out in fear that it was going to replace string players. That didn’t happen because it didn’t sound authentic enough, but grew a life of its own in Pink Floyd and Beatles recordings, like the beginning of ‘Strawberry Fields Forever’ [43]. Later synthesizers came and replaced almost everything, and now AI programs are replacing composers and artists. The unintended consequences include many ongoing lawsuits against companies that are accused of stealing intellectual property. They falsely associate famous names to their AI-generated wares. For the years such things are in litigation, many are making millions of dollars at the expense of others” [77]. “I think [applying tech tools to performance] can be a work in progress as there’s much trial and error involved. Too often in the marching arts once something is decided regarding criteria or even a rule, it’s set in stone and it’s done for the sake of change, but they are lean on evaluation. There needs to be an investment in evaluation. Unfortunately due to the competitive aspect of the marching activity, too often it seems that ego or pride displaces evaluation and distracts from improvement opportunities” [82]. “Each show is different. One year we have 40 people playing trumpet and the next year it shifts to many more tubas, or we’re dancing one year and marching more the next. The goalpost keeps moving. You may collect data on a production one year but the next year the show demands an entirely different constituency and skill set. Designers almost shoot from the hip every year” [80].

Learning and Performance Technology for the Performing Arts

171

Most of the interviewees tried to imagine how deep intuition and musicianship can be supported by technology, including data analytics and AI. In almost every case, they felt that tech solutions are appropriate during the learning process, to make it simpler and to save time, but largely not to directly enhance human performance real-time. Reducing time to competency is regarded as a benefit, given the constraints of rehearsals and touring. But the kind of performance acumen that results from great teachers, great content, and dedication to rigorous rehearsal is what they relate to mostly. “I am familiar with software and content from Alfred Music for music students called Sound Innovations [25]. I discovered it right before the pandemic. You have the sheet music, and you record into the computer. The technology evaluates you on things like melodic pitch, timing, rhythmic accuracy, note accuracy, right there. There are exercises on timing and pitch dynamics built into that technology. The coolest thing is being able to send things to your instructor/teacher for evaluation. So you get a hybrid of technology-generated feedback and human evaluation, and a grade is given. It certainly accelerates learning and is a great way to become creative and play great music” [82].

10 Implications for Workplace Performance The following list of questions is not meant to be comprehensive, but to suggest where we might rethink workplace performance today. Readers are encouraged to draw their own conclusions. Which are the things most relevant to your L&P practice and experience? Consider: • How do businesses establish and maintain parity between individual performance and organizational performance? • What is unfair technological advantage and fraud versus creative, strategic use of technology? • How do we codify performance excellence? Is it what a person does or can do on the job, or the combination of the individual, plus artifacts of the work environment, plus ethics and all the things that make us human? • To what extent should we consider technology enablement in L&P taxonomies? • What are appropriate applications of AI in workplace L&P? • Should workplace L&P have an explicit ethical code that addresses applications of performance technology? • What is an appropriate budget for L&P in this technology-driven, world as a fraction of revenue? • How is cognition distributed in a technological world? • How are we to contain costs associated with technology? • What are the tradeoffs between employing leading/bleeding edge tech tools to the business versus refining more organic and elegant humanity-centered processes?

172

G. J. Dickelman

The vision seems to be that L&P requires a more holistic treatment, a complete knowledge ecosystem. We should perhaps revise our view of the world of work through a more human-centered lens around design and dispose of the tech influences that have poisoned the design and evaluation wells. Organic performance acumen is, after all, always at the core of what we consider excellent, regardless of how much technology is involved. In this manner, we may be able to level the playing field around costs and resources to achieve outstanding performance. It is often felt that the best productions are ones that move us because of their simplicity and elegance.

11 Conclusions and Next Steps What we really aim to do is shift the competency curve toward fewer novices, more intermediates (who perform most of the work), and a keenly focused group of experts, in any field. From a data science perspective, it means applying the most efficient and effective strategies to wiggle the mean and standard deviation of organizational competency toward a new, more productive and satisfying normal. There is clearly fear, doubt, and practical concerns, like budget, around integrating technology into the arts and into our business establishments. We respect organic performance and individual acumen, but also admire technological innovation in L&P design. It is a dichotomy that brings to mind a favorite quote from over 70 years ago: “Thousands of years ago, the first man discovered how to make fire. He was probably burned at the stake he had taught his brothers to light. He was considered an evildoer who had dealt with a demon mankind dreaded. But thereafter men had fire to keep them warm, to cook their food, to light their caves. He had left them a gift they had not conceived and he had lifted darkness off the earth. Centuries later, the first man invented the wheel. He was probably torn on the rack he had taught his brothers to build. He was considered a transgressor who ventured into forbidden territory. But thereafter, men could travel past any horizon. He had left them a gift they had not conceived and he had opened the roads of the world.” [52]. Acknowledgements Many thanks to Rebecca Cole, Ted Greenberg, Tim Hinton, Larry Kerchner, Kevin LeBoeuf, Joe Marrella, Steve Rondinaro, and Steve Vickers for their generous contributions of time, insight, wisdom, and profound willingness to share thoughts. L&P science and DCI performance will no doubt reap great rewards from their participation in this inquiry. Thanks also to my life partner, Jan Greenberg, for her insights around early learning and supporting me through several physical challenges as I developed this inquiry. Finally, to my felines, Namare, Lila, and Evie, who sat on my lap and purred me through some rather intense cognitive acrobatics.

Learning and Performance Technology for the Performing Arts

173

References 1. How long is halftime at the Super Bowl? https://www.sportingnews.com/us/nfl/news/superbowl-halftime-arizona-length-2023/espv7mxa5jdy3edzqbsaforv. Accessed 15 Mar 2023 2. Inside the Super Bowl 50 Halftime Show Control Room. https://www.youtube.com/watch?v= gfjWjkTP4p8. Accessed 15 Mar 2023 3. Behind the Scenes of a DCI Broadcast Production. https://www.youtube.com/watch?v=TLG 1z2sMoho. Accessed 15 Mar 2023 4. A weekend of Japan’s 2019 marching music. Drum Corps World. 48(14), 10–28 (2019). https:// www.drumcorpsworld.com/publications/2019/december/#page=10. Accessed 18 Mar 2023 5. 20th title goes to Concord Blue Devils. Drum Corps World. 51(10), 8–21 (2022). https://www. drumcorpsworld.com/51-10/. Accessed 18 Mar 2023 6. Drum Corps International. https://www.dci.org/. Accessed 15 Mar 2023 7. About Drum Corps International. https://youtu.be/RwIx22Udjic. Accessed 5 Mar 2023 8. Jamulus: Play music online with friends. https://jamulus.io/. Accessed 10 Mar 2023 9. What a sound engineer really does. https://youtu.be/G2Rhh_4GZmU. Accessed 5 Mar 2023 10. Drum Line (the movie). https://www.imdb.com/title/tt0303933/. Accessed 15 Mar 2023 11. Flo Marching (a division of FloSports). https://www.flomarching.com/. Accessed 5 Mar 2023 12. Fathom Events. https://www.fathomevents.com/. Accessed 16 Mar 2023 13. Visualizing Telematic Performance. https://artsinitiative.umich.edu/projects/project1/. Accessed 25 Feb 2023 14. 13a. Visualizing Telematic Performance (YouTube video). https://youtu.be/O6CxfnURJKU. Accessed 16 Mar 2023 15. Technology and the Performing Arts Field: Usage and Issues. https://www.giarts.org/sites/def ault/files/Technology-and-the-Performing-Arts-Field-2010.pdf. Accessed 15 Feb 2023 16. How Technology Has Changed the Theater. https://illuminated-integration.com/blog/how-tec hnology-has-changed-theater/. Accessed 18 Mar 2023 17. Industry Update, Part One: Technology in Performing Arts. https://performance.wengercorp. com/industry-update-part-one-technology-in-performing-arts/. Accessed 20 Feb 2023 18. Industry Update, Part Two: Technology in Performing Arts. https://performance.wengercorp. com/industry-update-part-two-technology-in-the-performing-arts/. Accessed 13 Mar 2023 19. Industry Update, Part Three: Technology in Performing Arts. https://performance.wengercorp. com/industry-update-part-three-technology-in-the-performing-arts/. Accessed 18 Mar 2023 20. How Technology has Changed the Theater? 4 Ways. https://www.theatreartlife.com/stayingstill/4-ways-technology-changed-theatre/. Accessed 13 Mar 2023 21. Perks of Advanced Reporting and Analytics for Theaters & Performing Arts Centers. https:// venuearc.com/blog/perks-of-advanced-reporting-and-analytics-for-theatres-performing-artscenters/. Accessed 23 Feb 2023 22. Moore, G.: Crossing the Chasm, 3rd edn. HarperBusiness, New York (1991, 1999, 2002, 2014) 23. Guwande, A.: The Checklist Manifesto: How to Get Things Right. Metropolitan Books Henry Holt and Commpany, LLC, New York (2010) 24. Pyware 3D. https://www.pyware.com/3d/. Accessed 5 Mar 2023 25. Sound Innovations by Alfred Music. https://www.alfred.com/sound-innovations/. Accessed 25 Mar 2023 26. Apple Air Tags. https://www.pcmag.com/how-to/apple-airtag-tips. Accessed 25 Mar 2023 27. Best Posture Correctors 2022. http://bit.ly/3lPkvDo. Accessed 25 Mar 2023 28. In-ear metronome. http://bit.ly/40dZBNi. Accessed 25 Mar 2023 29. WGI (Winter Guard International) Sport of the Arts. https://wgi.org/. Accessed 25 Mar 2023 30. NFL Red Zone. https://www.nfl.com/redzone/. Accessed 25 Mar 2023 31. Disney’s JAMMitors. https://disneyworld.disney.go.com/entertainment/epcot/jammitors/. Accessed 25 Mar 2023

174

G. J. Dickelman

32. Rebecca (“Becca”) Cole ([email protected]) is a graduate of University of North Carolina at Greensboro (UNCG) where she received her bachelor’s degree in Kinesiology (2019). She returned to her alma mater UNCG in May 2020 to obtain her master’s degree in the Science of Athletic Training. During her master’s program, Becca worked in multiple athletic training settings including Grimsley High School, Carolina Crown, Meredith College, and her last setting was in a physical therapy setting with EmergeOrtho. Upon completing her master’s degree, Ms Cole joined the Select Physical Therapy team and became the Head Athletic Trainer for Wake Christian Academy. Interviewed 17 March 2023 33. Norman, D.: Design for a Better World: Meaningful, Sustainable, Humanity Centered, 1st edn. MIT Press, Cambridge (2023) 34. Laurel, B.: Computers as Theater. 2nd edn. Addison-Wesley Professional, Boston (1993, 2013) 35. Cooper, A.: The inmates are running the asylum: why high tech products drive us crazy and how to restore the sanity, 1st edn. Sams-Pearson Education, Carmel, IN (2004) 36. How Does Technology Affect Fine Motor Skills? http://bit.ly/3z9AG1x. Accessed 5 Mar 2023 37. Parenting Around Screen Time | Harvard Graduate School of Education. http://bit.ly/3Mc mSLF. Accessed 28 Feb 2023 38. Makemusic Finale. https://www.finalemusic.com/. Accessed 27 Mar 2023 39. Avid ProTools. http://bit.ly/3Zf751r. Accessed 6 Mar 2023 40. Celemony Melodyne. https://www.celemony.com/en/melodyne/what-is-melodyne. Accessed 27 Mar 2023 41. Cross, J.: Informal learning: rediscovering the natural pathways that inspire innovation and performance, Pfeiffer (and imprint of Wiley & Sons Inc., San Francisco, CA (2007) 42. Mellotron. http://bit.ly/3FOw7xv. Accessed 3 Mar 2023 43. Strawberry Fields Forever. https://youtu.be/10LSq_J5ol4. Accessed 7 Mar 2023 44. Houston Astros sign stealing scandal. http://bit.ly/40y2p7G. Accessed 12 Mar 2023 45. Norman, D.: The design of everyday things, Doubleday, NY, and Enfinito (https://enfinito. com/) (1990, 2012) 46. The consequences of a lack of training in the workplace. https://www.linkedin.com/pulse/con sequences-lack-training-workplace-dieter-moll/. Accessed 5 Jan 2023 47. Millions of jobs probably aren’t coming back, even after the pandemic. https://www.was hingtonpost.com/road-to-recovery/2021/02/17/unemployed-workers-retraining/. Accessed 25 Mar 2023 48. Your Detailed Guide to the 2023 Gartner Top 10 Strategic Technology Trends ebook. https:// www.gartner.com/en/information-technology/insights/top-technology-trends. Accessed 12 Mar 2023 49. Rogers, E.M.: Diffusion of Innovation, The Free Press, a Division of Simon and Schuster, Inc., New York 50. Fact or Fiction?: NASA Spent Millions to Develop a Pen that Would Write in Space, whereas the Soviet Cosmonauts Used a Pencil. https://www.scientificamerican.com/article/fact-or-fic tion-nasa-spen/. Accessed 12 Mar 2023 51. Old School Percussion: a livestream with Tom Aungst. https://bit.ly/3CAyBMy. Accessed 12 Aug 2021 52. Rogers, E.M: Diffusion of Innovation, Simon and Schuster Free Press, New York (1962, 1971, 1993) 53. Rand, A.: The Fountainhead. New American Library, New York (1952) 54. FitBit. https://www.fitbit.com/global/us/home. Accessed 1 Mar 2023 55. Apple Watch. https://www.apple.com/watch/. Accessed 1 Mar 2023 56. How technology is changing Sports Medicine. https://orthostreams.com/2018/07/technologyis-changing-sports-medicine/. Accessed 13 Mar 2023 57. Norman, D.: Things that Make Us Smart. Addison-Wesley Publishing Company, Menlo Park (1993) 58. DCI 2022 World Championships Finalists Best Moments. https://youtu.be/qIExsbt8ZGc. Accessed 2 Mar 2023

Learning and Performance Technology for the Performing Arts

175

59. 2013 Carolina Crown | Finals Show Chunk (Judges Tape). https://youtu.be/onoXY_KeYNo. Accessed 4 Jan 2023 60. Carolina Crown visual designer breaks down “Right Here, Right Now.” https://youtu.be/TwK mzmJshuY. Accessed 18 Mar 2023 61. Designing a Drum Corps Show. https://youtu.be/9VHTMRq3J1E. Accessed 18 Mar 2023 62. 2018 Santa Clara Vanguard - “Babylon” (My Body is a Cage by Arcade Fire/Peter Gabriel. https://youtu.be/F4_j5W8-L3g. Accessed 12 Nov 2023 63. The Athleticism of a Drum Corps marching member. https://youtu.be/IuP8O-KYok0. Accessed 12 Mar 2023 64. Blue Man Group on DCI. https://www.youtube.com/watch?v=0ViHdBfhKNk. Accessed 25 Feb 2023 65. Carolina Crown 2012. https://youtu.be/6GJhvZRdf6U. Accessed 12 Mar 2023 66. Boston Crusaders 2021. https://www.youtube.com/watch?v=9DYsJg8ynr4. Accessed 25 Jan 2023 67. Bluecoats 2014 “pitch bend”. https://youtu.be/hbeBsx3jq-0. Accessed 12 Feb 2023 68. Bluecats 2014 “pitch bend” crowd reaction. https://www.youtube.com/watch?v=CTEud2 0sDvM. Accessed 12 Feb 2023 69. Troopers 2022 with cello. https://youtu.be/cpUkedQyMzY?t=54. Accessed 12 Feb 2023 70. On the design of Vanguard’s 2018 “Babylon” | Michael Gaines. https://youtu.be/YKuK6n cbgBo. Accessed 25 Feb 2023 71. Blue Devils 1990 - Selections from Tommy. https://youtu.be/C_x6JV0wI94?t=149. Accessed 21 Mar 2023 72. Highlights: 2022 SoundSport International Music & Food Festival | #DCI2002. https://youtu. be/qvt1oUVUKAk. Accessed 21 Mar 2023 73. Hawthorne Caballeros Alumni 2022. https://youtu.be/r7alUAX1B9M?t=532. Accessed 21 Mar 2023 74. Carolina Crown 2017, Bach’s “Little Fugue” in G minor. https://youtu.be/k-CxYRCWjIM. Accessed 12 Mar 2023 75. Crown Live App. https://apps.apple.com/gb/app/crown-live/id1628954467 and https://youtu. be/m7wI3gt5dZk?t=48. Accessed 13 Mar 2023 76. DCI Adjudication Categories. https://dci.org/scores/recap/2022-dci-world-championshipfinals. Accessed 2 Jan 2023 77. Ted Greenberg - Ted Greenberg ([email protected]) is a musician, music producer, audio engineer, film/TV composer, acoustician, and teacher. He is the recipient of two Grammys and two Tec awards for his audio art on the soundtrack for the movie “Standing In The Shadows Of Motown.” He has earned numerous instrumental performance, recording, and film accolades, and consults on major studio/audio engineering projects. He is affiliated with the National Academy of Recorded Arts and Sciences, Audio Engineering Society, and the International Federation of Musicians unions. He resides in the LA area. Interviewed 19 March 2023. 78. Tim Hinton ([email protected], https://marchingartseducation.com) is a Tampa, FL based composer and arranger who has been creating marching arts productions for almost 40 years. He serves as a consultant and designer who cherishes helping performance ensembles to be more interesting and entertaining. He received his bachelor’s degree in music education from the University of Tennessee, Knoxville, and his masters from Georgia State University. Tim is earned top spot in the DCI Individual and Ensemble Competition on French horn in 1982, being the first in DCI history to receive a perfect score in individual competition. He is the host of the Marching Roundtable podcast and the creator of Marching Arts Education. Tim shares his passion for the marching arts by providing content to help everyone be more successful and enjoy their work. Interviewed 7 March 2023. 79. Larry Kerchner ([email protected], http://www.larrykerchner.com/) is a composer, lyricist, arranger, and producer with years of experience in the music industry, having received his extensive musical education at Berklee College of Music and the Boston Conservatory of Music. He is a voting member of The National Academy of Recording Arts and Sciences, a two-time GRAMMY Award nominee, and an Individual Artist Fellowship recipient. He is

176

80.

81.

82.

83.

G. J. Dickelman also a member of the American Society of Composers, Authors, and Publishers, and has had his work performed at several ASCAP Songwriters’ Showcases. He is on the Board of Directors of The Duke Ellington Center For the Arts, and is a member of The New York Sheet Music Society, The American Federation of Musicians, as well as the Manhattan Association of Clubs and Cabarets. Larry has conducted his music on the Tonight Show, the Jerry Lewis Muscular Dystrophy Telethon, the Paul Anka Cerebral Palsy Telethon, the Mike Douglas Show, the Merv Griffin Show, and in venues such as the Copacabana, Westbury Music Fair, the Waldorf-Astoria, and Harrah’s in Las Vegas, Reno, and Lake Tahoe. He was inducted into the Drum Corps International Hall of Fame and the World Drum Corps Hall of Fame. Interviewed 10 March 2023. Kevin LeBoeuf currently serves as Brass Caption Head for the world champion Bluecoats Drum and Bugle Corps and previously served as Brass Caption Manager for the DCI world champion Cavaliers Drum and Bugle Corps. He previously served on the Carolina Crown brass staff, and was a marching member of the corps from 2011–2014. He holds an undergraduate degree in music education from Rutgers University, and a graduate degree in wind-band conducting from the University of California, Los Angeles. LeBoeuf serves as the Brass Caption Head for the Aimachi marching band in Nagoya, Japan and presents clinics throughout the US, Japan, and Thailand. Outside of drum corps, LeBoeuf balances a career split between New York, Los Angeles, and Nagoya, Japan as a clinician, composer, arranger, bespoke sound system designer, and general contractor. Interviewed 3 March 2023. Joe Marrella ([email protected]) has over twenty-five years as a performing arts adjudicator in the U.S.A. and Canada. He is an individual snare drum champion and has instructed five national champion percussion ensembles. He recently joined the likes of Buddy Rich as a Drumming Hall of Fame inductee. Joe consults and has conducted seminars for music directors throughout the country, including a seminar on marching bands at Berklee University, and has conducted training and evaluation for many judging associations. He has also written articles for musical publications utilized by 35,000 high schools. He is an instructor, arranger, and judge for a number of prestigious halls of fame in the performing arts, including DCI. Joe has directed 22 theatrical musicals and was nominated for Best Direction twice by the Perry Awards of New Jersey. He has been selected as a Hall of Fame instructor, arranger, and judge to seven different Halls of Fame. Among them is the Drum Corps International Hall of Fame. Joe has served for three years as a television color commentator for the of various musical competition broadcasts and served as the Cavalcade of Bands Education Director for four years. Interviewed 1 March 2023. Danny Raymond (www.dannyraymonddrums.com, [email protected]) is a top performer in the marching arts who has enjoyed a varied career in the world of percussion. He was a member of the New York Skyliners Drum Corps as well as instructing or consulting with the Syracuse Brigadiers, Carolina Gold, Boston Crusaders and the Santa Clara Vanguard. Danny is a two time Drum Corps Associates Individual Snare Drum Champion (1989, 1990). He enjoyed over thirty years with the Walt Disney World Company which included performing, writing/show design, and Casting Associate. Danny cofounded and performed for 25 years with Disney’s popular “JAMMITORS.” He has authored a number of published articles and music scores. He performs as a freelance drummer in Michigan. Danny has given clinics and performed throughout the United States, Europe, Japan, and Canada performing with percussion greats such as Danny Gottlieb, Evelyn Glennie, Zoro, and the late great Joe Morello. Interviewed 17 March 2023. Steve Rondinaro ([email protected]) has enjoyed a storied career in TV news and broadcasting, including station ownership, with a second chapter in marketing and production, and still a third as a journalism professor at Coastal Carolina University and consultant. He has been the annual host of live telecasts, cinecasts and webcasts for Drum Corps International since 1979; 29 years were on PBS and ESPN. He holds Emmy awards from the TV days and Telly Awards now, including a 2021 gold Telly. He is a DCI Hall of Fame inductee and currently chairman of the DCI Hall of Fame. Interviewed 16 March 2023.

Learning and Performance Technology for the Performing Arts

177

84. Steve Vickers ([email protected]) has been editor and publisher of Drum Corps World magazine since 1973–4. He grew the magazine to be the premier global publication for drum corps activity. Through Vickers’ leadership, every drum corps show - no matter how big or small - receives coverage. For many years, his passion for drum corps ensured that most Drum Corps International (and smaller regional association) events received publicity in Drum Corps World as he believed every corps is important and worthy of recognition. Coverage over the years has expanded to drum corps activity in the UK, EU, and Asia. He is a Drum Corps Hall of Fame inductee and a former marching member of the Hutchinson, Kansas Sky Riders Drum and Bugle Corps. Interviewed 16 March 2023.

Fuel Your Top Five: Using What You Know to Make Real Change Erin Donovan

Abstract The purpose of this article is to work against the theory that professional development is in the hands of the institution. With strategies that allow a change in the narrative of institutional control, as it exists both in the education and corporate spheres, it is possible to build a learning path through aligning a sense of self with future career goals. By combining personal choice and empowerment, this article investigates how it is possible to construct growth proactively. This sense of growth can be achieved through applying and aligning strategies that allow the educator or corporate partner to define their own success both in and out of the traditional career structure. The argument begins by providing a contrasting view of what employee development was and what personal development is and perhaps can be. Next, it outlines five steps and corresponding strategies that actively build personal development paths. The article conclusion explores the idea of building a unique learning path to focus next steps. Keywords Development · Employee · Teacher · Goals · Empowerment · Choice

1 Learning from My Certainties When I graduated from college (the first time), I thought I had it made. I had a job, a partner and even a car. After my last exam, I vowed to never take another test and was convinced that I would work against the trend of job hopping. Sure enough, I would retire from my retail mid-management job that was, at the time, a very secure career choice. Fast forward: two other partners, a virtual bankrupting of mall culture, countless jobs, endless contracts, multiple career shifts, three additional degrees (all which entailed exams) and that old Dodge Neon, long gone. And even with all the changes I experienced, I honestly believed that if I continued to devote myself to what my E. Donovan (B) Fuel Training Consultants, Townville, SC, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_14

179

180

E. Donovan

organization wanted, I would be successful. They would take care of me. In fact, until the most recent market shifts, I invested in the concept of institutional-based security. Until my investments ran dry. Minutes after being hired in a new Learning and Development Director role, we were tasked with firing most of our development team. It became immediately clear that everyone, including me, was on the proverbial chopping block. My manager gave me what would become some of the most inadvertently important advice that dramatically shifted my future mindset. No one is safe. In the U.S., we have no security. He was right, and yet he was wrong. The purpose of this article is to work against the theory that professional development is in the hands of the institution. With strategies that allow a change in the narrative of institutional control, as it exists both in the education and corporate spheres, it is possible to build a learning path through aligning a sense of self with future career goals. By combining personal choice and empowerment, this article investigates how it is possible to construct growth proactively. This sense of growth can be achieved through applying and aligning strategies that allow the educator or corporate partner to define their own success both in and out of the traditional career structure. The argument begins by providing a contrasting view of what employee development was and what personal development is and perhaps can be. Next, it outlines five steps and corresponding strategies that actively build personal development paths. The article conclusion explores the idea of building a unique learning path to focus next steps.

1.1 Development: Past, Present and Future Traditionally, professional development has been path, program and organizationally focused. McCauley and Hezlett [1] suggest that employee development is “the expansion of an individual’s capacity to function effectively in his or her present or future job and work organization.” This task-based definition aligns what the employee will learn and do to the jobs, both future and present, that are needed most by the organization. It sets the onus and gives the power to the organization as the individual functions only as a component of their community. Hedge and Rineer [2] add that, by defining the historical career-pathway as situated vertically, with ascending steps, it only allows the employee to advance through a structured hierarchy, that is built by adding more complexity and responsibility, rather than cross-trained skills. This supports the mentality that employees are confined to a single occupational area and are tasked to follow a progression schedule that has a timeframe tied to title. Professional development for educators follows much the same trend. Costing billions of dollars with results that still struggle to be quantified, teachers are mandated to complete a certain amount of professional development hours each year as documented by Kennedy [3] and Hill [4]. While the results are unclear, Desimone

Fuel Your Top Five: Using What You Know to Make Real Change

181

[5] claims the need for professional development is certain. Unlike their historic corporate peers, the role of the teacher doesn’t follow a strict hierarchical definition. They take on roles ranging from data analyst, creative director and behavior management. The idea that professional development that is often one size fits all, can impact what they do is hard to fathom. What is clear is that the main result should be an improvement in their student’s achievement, but little is said about what professional development should do for them personally, let alone how it could be utilized to build their future selves. Even more, Mendes et al. [6] suggest that the concept of the teaching profession is losing its value in the larger society. Factors such as economic trends, democratization of knowledge and failing schools have impacted the idea of the teacher. Indeed the #transitioningteacher movement is one catalyst that is resulting in a crippling loss of talent and innovation. And Hill [4] argues the idea that professional development that only impacts student scores is not a sustainable solution. Rather, Mosquera, Stobaus, Jesus, and Herminio [6] argue that a healthy teacher with strong interpersonal relationship will not only lead their students more authentically, but they can reach a point of self-realization, which can be used as a factor of motivation. But how do we change these mandates and top-down expectations in both the corporate sphere? Let’s look at the new reality that can exist for both the educator and the employee. In the new development model, Molloy and Noe [7] cite that employees are expected to take more responsibility for enhancing current skills and adding new ones to meet current job demands, as they prepare for leadership opportunities, and ensure their own employability to move and adapt within and between organizations. Individuals need to be motivated to learn and should advocate that the organization will support their motivation by providing the necessary culture and resources that allow flexible learning answering individual needs. Bewell, Weaver, Salas, and Tindall [8] emphasize that technological and demographic, as well as changes in the employment relationship, signify that lifelong learning will increasingly involve opportunities that employees seek on their own rather than attending formal mandatory programs sponsored by the organization. Mosquera and Stobaus [6] add that self-image and self-esteem is influenced through (a) potential, (b) perception, and (c) personal dynamics. All factors should not only be self-defined by should be managed by the self rather than the organization.

2 Fueling the Five So let’s review: We own our success. We define our success. Rather than wait for the pink slip, it is up to each individual to not only outline what they want but to actively seek out opportunities to fund that want with knowledge. That can’t happen by choosing a new certification program and jumping through the hoops of e-learning. Rather, we use the following steps to align strategic approach to future goals:

182

• • • • •

E. Donovan

Build the self-image. Believe in agency. Create a pro-active plan. Resist a linear path. Define and fill skill gaps.

2.1 Self-Image: Learn to Live As-If In the traditional classroom and hierarchy, we are defined through our roles. As we move into society, and people ask what we do, we answer through our job title. This inexplicably ties who we are with what we do and the self-image that follows fluctuates with our success, as it is defined by others. Mosquera et al. [6] suggest that self-knowledge builds the idea that it is possible to “self-update” independently of the institution. Parker [9] adds that growth is dynamic and we should work against a static state of mind. Dweck [10] confirms this as she suggests that talents can be developed. In other words, we are not born with a fixed set of skills that can only be stretched so far. Based on these concepts, the first step in making personal change is to believe and build the self. Begin by asking yourself what if questions: • • • •

What if I had that opportunity? What if I made that salary? What if I could do that skill? What if I felt that way about that?

What would change? How would you describe yourself, own your space, and choose the way you spend your time and use your resources? How could you move from living in a state of “have to,” to living in a state of as if. Think through these statements and start to envision a new sense of identity that no longer is tied to what you do, but rather espouses who you are. • • • •

I live as if I am a leader. I live as if I am a writer. I live as if I am a great parent. I live as if I have the life I want.

2.2 Believe in Agency The idea of as if is lofty. However, in order to build both professional and personal change, you need to believe not only that change is possible, but it is within your control. Often, when working with transitioning educators, their despair originates in the feeling that they have no individual control, their words aren’t heard and ultimately they receive no support. Corporate employees have similar complaints:

Fuel Your Top Five: Using What You Know to Make Real Change

183

they aren’t considered for promotions, no one listens to them and they aren’t treated with the equity they deserve. Dachner, Ellingson, Noe, and Saxton, [11] remind employees that a component of their role is to take on the viewpoint of active agency. Within this point of view, the influence of internal forces far outweighs those of external forces. To build your sense of agency start by creating your own set of rules and personal expectations. Your daily life, opportunities and wins should be charted on a performance development plan that you own. When you take back control of your selfview, define who you want to be and then give yourself the power to achieve it, you can begin to believe in the possibility of achieving real results. Try these strategic exercises that can help build the agency mindset. 1. Build, edit, and rate your daily to-do list. Decide when you need to get things done and how you will measure the success of each task. 2. Create goals that exist outside of your workspace. Make sure success isn’t always tied to what you do. 3. Start to define what good looks like to you: good work, good days, good relationships, etc. Make sure that definition is within your power to evaluate.

2.3 Live Through a Pro-Active Approach Being proactive is about making things happen, anticipating and preventing problems, and seizing opportunities. Proactivity is judged primarily through the act of initiation. Crant [12] suggests that proactivity begins with removing uncertainty and ambiguity and fighting against the urge to passively react to life events. Parker, Bindl and Strauss [13] remind us that proactive behavior is routed in self-regulation, the desire to choose and the motivation required to achieve and work autonomously. What proactive behavior allows is a greater sense of control that leads to making sense of life events. Also, when you think proactively, you are not waiting to be mired in reactionary practices. Rather, you can create multiple plans that allow you to fail or succeed logically. You can check and adjust your actions, learning as you live with the understanding that “destiny” is not in control. You are.

2.4 Resist a Linear Life One of the first lessons I learned as a reading teacher is that academics no longer believed in linear literacy learning. Rather, learning to read and write entailed a circular path with influences all merging to create milestones of achievement for the reader. Hedge and Rineer [2] extend this understanding to corporate talent pipelines. The idea of linear development for both the organization and the employee is no longer relevant. Instead, talent development should be charted along pathways and

184

E. Donovan

trajectories that respond to and reflect the changing needs of all those involved. Now apply that thinking to building your best self. How many times have you taken apart a process from beginning to end? When you are doing this work, how many times do you stop, reframe, get a new YouTube video, a new book, a new source of knowledge to re-design your learning structure? Because learning has become so populous, and we find expertise in many different directions. We are no longer corporately locked down with three-day leadership retreats or prescribed mandatory time served by the school district. It is up to us to learn in our own style. Same with charting our development. It is counter-intuitive to move from position A to B to C. Because of the disruptive force and speed of change, those positions might not even exist by the time you are ready for them. Instead, chart a path that is defined by what you want to know and what you want to do, rather than what title you want to have. This will create a naturally winding path driven by your self-image, agency and proactive planning.

2.5 Define and Fill Your Skill Gaps Let’s start this fifth section with two questions: What do you want to know? How would that knowledge change your situation? Learning as an adult must have a why behind it. As such, the why needs to belong to us, not the institution. When we turn our why over to the institution, it is easy to rationalized learning that has nothing to do with who we are or what we want to do. In the corporate space, learning might make you better at your job or you might receive a promotion because you completed said learning. For teachers, your employment may be contingent upon becoming certified in a district-adopted program. But again, your learning is based on external forces, rather than what you want to know to pave the steps on your self-determined path. To change this externally imposed narrative, start by looking at job descriptions and speaking with people who are doing functions that you can see in your future. What are the commonalities? What are you missing? Making a plan to fill those gaps will serve as a vital component of your overall proactive plan and because you are actively working toward your goal, your inherent motivation will build agency, allowing you to take part in the development activities that you need to succeed. Once again, not only will you own the learning space, but you will also own the rewards of your hard work.

3 Conclusions and Next Steps The purpose of this article is to provide solid steps that allow you to fuel and form your individual path to success. Based on your identity and agency, it discussed how you can take a proactive approach to build success from what you want to do and what you do best. To put these strategies into practice, remember to push against

Fuel Your Top Five: Using What You Know to Make Real Change

185

the expected. Learn where you live. Develop habits that serve your success and find ways each day to reach beyond the paved paths to discover your own.

References 1. McCauley, C.,Hezlett S.: Individual development in the workplace. In N. Anderson Handbook of Industrial Work and Organizational Psychology. Sage Publications, London (2001) 2. Hedge, J.W., Rineer, J.R.: Improving Career Development Opportunities Through Rigorous Career Pathways Research. RTI Press Publication No. OP-0037–1703. Research Triangle Park, NC: RTI Press (2017) 3. Kennedy, M.M.: How does professional development improve teaching? Rev. Educ. Res. 86(4), 945–980 (2016) 4. Hill, H., Beisiegel, M., Jacob, R.: Professional development research: consensus, crossroads and challenges. Educ. Res. 42, 476 (2013) 5. Desimone, L.M.: Improving impact studies of teachers’ professional development: toward better conceptualizations and measures. Educ. Res. 38(3), 181–199 (2009) 6. Mendes, A.R., Dohms, K.P., da Conceição, C., Lettnin, J.J., Mosquera, M., Stobäus, C.D.: Elements for the personal and professional development of teachers. Create. Educ. 07(10), 1444–1455 (2016). https://doi.org/10.4236/ce.2016.710150 7. Molloy, J., Noe, R.: “Learning” a Living: Continuous Learning for Survival in Today’s Talent Market. Learning, Training and Development in Organizations. Routledge (2009) 8. Bewell, W., Weaver, S., Salas, E., Tindall, M.: Emerging conceptualizations of adult training and learning. In: London, M. (Ed.) The Oxford Handbook of Lifelong Learning. Oxford University Press (2011) 9. Parker, S.: Work Design Growth Model: How Work Characteristics Promote Learning and Development. Routledge (2017) 10. Dweck, C.: What Having a “Growth Mindset” Actually Means. Harvard Business Review (2016) 11. Dachner, A.M., Ellingson, J.E., Noe, R.A., Saxton, B.M.: The future of employee development. Human Resourc. Manage. Rev. 31(2), 100732 (2021). https://doi.org/10.1016/j.hrmr. 2019.100732 12. Crant, M.: Proactive behavior in organizations. J. Manage. 26, 3 (2000) 13. Parker, S.K., Bindl, U.K., Strauss, K.: Making things happen: a model of proactive motivation. J. Manage. 36(4), 827–856 (2010)

Online Education Innovation Strategies to Gain Support and Accomplish Team Goals Joseph Evanick

Abstract Online education has become increasingly popular in recent years, resulting in new challenges for educational institutions to develop innovative strategies to support and enhance their programs. This paper discusses the different forms that innovation strategies can take, such as new approaches to online instruction and the development of new technologies and tools to enhance the online learning experience. The effective use of innovation strategies can lead to gaining support from stakeholders and achieving team goals more efficiently. Furthermore, innovation strategies can help institutions to overcome challenges that arise during the implementation of new approaches and technologies, such as resistance to change, lack of resources, and difficulties in managing the complexity of technology integration. By employing successful strategies to overcome these challenges, institutions can enhance the educational experience for students and instructors alike. The purpose of this paper is to explore how online education innovation strategies can be used to gain support and accomplish team goals within higher education institutions. The significance of innovation strategies in ensuring success and sustainability in online education is discussed, providing insights for institutions and online education leaders to position themselves for success in a rapidly evolving education landscape. Keywords Online education · Change · Faculty

1 Introduction Online education continues to grow, creating new challenges for educational institutions in terms of finding innovative strategies to support and enhance their programs. Innovation strategies can take different forms, from new approaches to online instruction, to developing new technologies and tools to enhance the online learning experience. Institutions that effectively leverage these strategies are more likely to gain J. Evanick (B) Geisinger College of Health Sciences, Scranton, PA 18509, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_15

187

188

J. Evanick

support from stakeholders such as students, faculty, and staff and help them achieve team goals more efficiently. Online education innovation can also help institutions to overcome challenges that arise during the implementation of new approaches and technologies. These challenges can include resistance to change, lack of resources, and difficulties in managing the complexity of technology integration. By employing successful strategies to overcome these challenges, institutions can enhance the educational experience for students and instructors. The purpose of this paper is to explore how online education innovation strategies can be used to gain support and accomplish team goals within higher education institutions. By understanding the importance of innovation, different types of strategies, and how to overcome challenges, institutions and online education leaders can position themselves for success in a rapidly evolving education landscape.

2 Online Education Innovation Online education is expanding. The modality has grown exponentially over the past decade, with an ever-increasing number of universities adopting online education to expand their reach and students’ learning options [3]. Online education plays a significant role in broadening educational access and expanding higher educational opportunities. The success, however, of any online education initiative depends on a critical and core resource, namely faculty, who are fully invested in providing quality instruction [31]. Because of diminishing state and federal aid to colleges and universities, higher education leadership often seeks alternative revenue streams to help support operational costs [6]. Without faculty inclusiveness, an institution may set itself up for failure or, at the very least, a sluggish start to developing its online offerings. However, faculty inclusiveness alone is not enough. Faculty inclusiveness must result in faculty buy-in, formally partnering with faculty for program development. Online learning offers the chance to reexamine education’s teaching and learning endeavors broadly. Online learning can be conceived as a new pedagogy, where strategies such as interaction and dialogue are introduced back into the higher education model [9].

2.1 Organizational Change While steps are commonly taken to avoid negative experiences with organizational change, it is difficult to ensure every stakeholder experiences an entirely positive experience. The complexities of institutions lend themselves to disagreements and confusion. Some institutions have attempted to prevent potential faculty opposition by establishing a new and independent administrative entity to handle online programs [5]. This eliminates the need to integrate the curriculum into the existing institutional culture. However, it also eliminates one of the most valuable assets

Online Education Innovation Strategies to Gain Support …

189

available to an online program: full-time faculty support. Creating a new administrative unit that does not align with its mission and purposes could lead to widespread animosity against the online program. There may be various occasions where courses must be structured to quality standards and expectations when institutions are facultydriven. When there is little or no standardization of quality assurance, each faculty member is entirely responsible for the course’s content and execution [32]. Faculty members often fail to identify the differences between online and traditional instruction and wrongly approach both modalities similarly. According to Carroll and Burke [5], it is unrealistic to expect that the syllabi of face-to-face courses can be successfully adopted in an online degree program. Although the material would be identical, if faculty are to provide successful online learning, they must make significant changes and improvements to their teaching methodologies.

2.2 Faculty Concerns Studies show faculty worries about online learning outcomes may be understandable as this method of instruction is still relatively new, at least when contrasted with classroom instruction [29]. Accessible and reliable high-speed internet allowed institutions to create high-quality online programs. If full-time faculty are involved in teaching in the program, an institution is more likely to launch an online degree with full-time faculty support and acceptance. Faculty participation affects every aspect of the program, from designing a consistent curriculum and relevant learning outcomes to constantly evaluating student learning and ensuring ongoing quality improvements; the value of such acceptance cannot be overstated. Vital institutional leaders must support the plan to deliver an online degree program [5]. The institution’s structure and culture determine the identity of these key leaders.

2.3 Online Education Acceptance Furthermore, acceptance of online learning will vary by institution. Creating an online curriculum is usually much simpler if the organization has an entrepreneurial culture [5]. If the institution is not entrepreneurial or innovative, attaining faculty support is essential. Approaching a small group of faculty members who have been early adopters of technology to gain their support and ability to create an online course is a successful way to generate interest. Such faculty are commonly eager to try out new technological applications and are open to using various teaching methods [5].

190

J. Evanick

3 Institutional Climate Higher education has its own culture, and each institution is unique. Culture is a learned phenomenon for groups, just as personality and character are for individuals [28]. Higher education institutions are complex, containing multiple groups of stakeholders. All stakeholders contribute to the institutional community, often influenced by tradition [1, 11]. Thus, organizational changes are experienced differently by each group and individual. Understanding community, attitudes, and customs within higher education institutions is integral to the decision-making process [1, 33]. Stories and institutional norms are linked to these behaviors [1, 4, 12, 30]. When leadership announces policies, employees do not necessarily obey, unlike in the corporate world.

3.1 Main Stakeholders Leadership, faculty, and staff are the three types of higher education employees. Every institution is different, but faculty have much more control and power than staff; sometimes faculty are almost as influential as leadership. Many universities have faculty governing bodies, collective bargaining unions, and institutional bylaws that give faculty expansive responsibilities and powers, including recruiting new colleagues, granting tenure, creating curricula, and evaluating teaching and learning. Through shared governance, faculty can apply a “we are the institution” attitude. In some situations, this “we” progressed to the point where faculty and administration are at odds, which does not have to be the case. A joint agreement between faculty governing bodies and administrations is standard on several campuses. This standard can generally be beneficial [8].

3.2 Leadership Challenges At higher education institutions, some individuals have their interpretations about what is and should be happening [2], complicating leadership. A crucial aspect of leadership is setting an effective team work strategy that stimulates, inspires team members, and generates a solid collective incentive to perform well [2]. The most prevalent challenge for learning leaders is coming to terms with their lack of experience and wisdom [28]. Leaders’ effectiveness is damaged if they over-control, disregard emotions, or are oblivious to their effect on others [2]. At these institutions, leading is tricky because each move and every decision supports or undermines the perceived levels of status, certainty, autonomy, relatedness, and fairness. Words and gestures are interpreted, magnified, and combed for unexpected reasons [25]. Leaders

Online Education Innovation Strategies to Gain Support …

191

may lack the tools to do their job unless they can recognize and develop successful relationships with key stakeholders on their team [13]. Organizational culture and transformation must be handled from the top, with senior leaders acting as efficient change agents [10]. Senior leadership that is both strong and progressive has a significant impact on how the organization is influenced and how individuals function. These vital qualities of senior leaders are also important for higher education institutions. Higher education institutions benefit from a positive atmosphere that enables instructors to better assist students in achieving their academic and personal goals [14]. Positive results from observing institutional and organizational structures, attitudes, principles, and underlying assumptions can reward higher education institutions [26, 27].

4 Faculty Inclusiveness Faculty inclusiveness in the decision-making process regarding online programs is a necessity. When faculty members are purposely denied a role in decision-making, they will predictably approach the situation from a standpoint fueled by negativity. Policy regarding the conditions that encourage or discourage faculty participation may help sustain academic quality and respectability [31]. It is crucial to have leadership’s support when undertaking an initiative such as entering the online education marketplace. Leadership can set the tone and strongly encourage faculty to join them in their support, but they cannot be forced. One way to achieve faculty buy-in is through incentives. The absence of rewards and incentives and the exclusion of technology and innovative instruction as a significant aspect of promotion and tenure reviews may influence faculty decisions on adopting new technologies or engaging in distance education [30]. If faculty are formally evaluated on innovation, technology, and online education, they will likely have increased motivation towards online education. Faculty, both the recipients and agents of change, must be open to online education to increase the probability of effective implementation [21]. Faculty hesitation can derail initiatives, so obtaining their support is very important. Multiple studies from the early 2000s to the present day point to common issues that explain faculty members’ aversion to online teaching [19]. First, there are technical issues [15] about insufficient technological support and preparation [18, 23, 31]. Second, several student-related issues must be considered [15]. Some professors question whether online courses can achieve the same learning results as face-to-face courses [16, 17]. Third, there are questions about pedagogy [15]. Faculty members are dismissive of the time and effort required to develop a well-designed course [7, 16, 17]. Fourth, there are questions about institutions [15]. Faculty members are unhappy with the lack of credit for online teaching in the tenure and promotion process and insufficient pay and release time [16, 18, 24]. Institutions must create an environment that is conducive to faculty inclusion. They must have a strategy to build an atmosphere where campus leaders do not neglect or abandon those who are resistant to change but rather work with various

192

J. Evanick

groups to meet student needs through technology when necessary [20]. The fear of a lack of quality and sufficient contact with students, a heavy workload, and a lack of administrative support are common reasons for faculty reluctance and resistance to teaching online [22]. Nevertheless, regardless of education delivery mode; face-toface, online, distance, or some combination through blended learning, teaching and learning are evolving” [9].

5 Gaining Support from Stakeholders Many difficulties encountered can be traced to an institution’s culture. Higher education institutions should have an atmosphere that encourages creativity to effectively promote progress. Institutional leadership should begin by thoroughly understanding the organization’s culture to recognize numerous stable factors contributing to its previous successes [28]. Leaders should understand that cultural alignment necessitates cultural modesty and the ability to bring various subcultures together in a discourse that fosters mutual respect and concerted action [28]. This can be a powerful accelerant for cultural shifts for organizational effectiveness when combined with the bringing together of various subcultures. The process of achieving cohesion amongst all stakeholders will begin with disconfirmation. Disconfirmation refers to evidence that some company goals are not being met or that some procedures are not doing what they are supposed to do [28]. The following are specific suggestions for shifting perception of leadership’s commitment to online education. Given the current state of higher education, online education will be regarded as a secondary or less-than face-to-face education without adequate leadership engagement and support. Leadership support may include funding, encouragement, supervision, and assistance to avoid obstacles to a successful and well-run online program. The following are primary recommendations to increase leadership’s commitment to online education.

5.1 Gain Leadership Support First, scholarship suggests online education should have a place at the table with executive leadership as institutions continue to expand and improve their online education offerings. Senior leadership involvement in online education would help to legitimize online education at the institution. Second, as institutions continue to evolve, infrastructure support for online teaching faculty should be well-organized and ongoing. Introducing an online education program necessitates specific considerations that are not usually problematic in a traditional classroom environment. A formal department dedicated to online education provides clarity and continuity.

Online Education Innovation Strategies to Gain Support …

193

Third, research suggests online faculty have complete pedagogical, technological, financial, and emotional support. Faculty supported by a well-developed, welldesigned system will have a better chance of succeeding. Leadership must be systematic in faculty support and development, as successful online teaching does not happen by accident. A structured and mandatory faculty training program and a stipend for participation would be advantageous. In addition, a technology support system should be implemented and available at all times. Fourth, launching an online program for the first time takes much effort. A new online program impacts instructional design, faculty, admissions, registrar, financial aid, and information technology. As an institution’s online enrollment grows, leadership should hire and let teams grow as needed. Employees who contribute to an online program’s success should be rewarded with promotions and opportunities for professional growth. Personnel investment is an investment in online education for the institution. Regarding faculty’s acceptance of online education, many educators have primarily focused on developing and teaching in traditional classroom settings throughout their careers. Additionally, faculty members may have limited experience teaching outside their own institution, which can influence their perspectives and opinions on instructional methods. This can lead to a perception that online degree programs are less rigorous than their traditional counterparts. The success of any online education initiative depends on faculty members who are committed to providing quality instruction and building a strong program reputation. The following guidelines can help increase faculty acceptance of online education [31].

5.2 Gain Faculty Support First, research suggests faculty should be partners. Faculty want to be more involved in online education’s decision-making processes, according to the findings of this study. It is highly beneficial for leadership and faculty to collaborate and find common ground. Second, market forces suggest leadership leverage faculty champions. These faculty champions are committed, have adequately educated themselves by attending training sessions, and work with instructional designers regularly. They recognize that online education entails far more than lecturing over a web-conferencing platform. These professors may be used to persuade their peers to follow in their footsteps. Third, it is important to promote student interaction. There must be significant changes if professors treat their online classes like they approach their faceto-face classes, often lecture-style. There are several alternatives to this teaching mode, and online education provides numerous opportunities for synchronous and asynchronous student interaction. Faculty should aim to build online learning communities in their online courses rather than relying heavily on lectures.

194

J. Evanick

Regarding institutional culture change and collaboration, institutions should build an atmosphere that encourages change and creativity to support the organization as it matures and expands to facilitate change effectively. To do so, a thorough understanding of its culture is needed to recognize various stable elements that contributed to the organization’s success [28], all while keeping an eye on the future. Leaders should understand that cultural alignment necessitates cultural modesty on their part and the ability to bring various subcultures together in the kind of discourse that fosters mutual respect and concerted action [28]. All planned change begins with acknowledging that something is not going as planned [28]. A lack of leadership engagement and transparency, faculty resistance and hesitancy, and a climate that is not conducive to a disruptive institutional innovation like online education. The following are the most notable recommendations for promoting culture change and collaboration.

6 Institutional Change and Collaboration First, it is recommended that decision-making processes be formalized. Formalizing these processes for online education would provide clarification. It would also have several advantages, such as saving time and ensuring that everyone in the organization knows what is going on. The result would be more efficient functions. Second, research suggests communication be intentional and clear. Communication guidelines should be established and followed consistently. Doing so will require a shift in actions by all stakeholders, but it is necessary. Clear and deliberate communication enhances transparency and accountability and ensures all stakeholders are on the same page, which is essential to success. Third, there should be motivation to change. Following the development of motivation for change, the real change and learning process should be implemented. This is the process of implementing all of the modifications and solutions. Institutions must be able to adapt to the changing and evolving nature of online education. Change motivation and quality improvement must become the new standard. The final step in a change process is refreezing, which means the changes will not last until they are backed up by tangible results [28]. Institutions should achieve full support for online education in this step to ensure they view it as a critical component of its future. Achieving this goal will not always be easy. There is another choice if problems occur or the organization falls behind schedule. Some institutions have avoided faculty opposition by creating a separate and autonomous administrative body to manage online programs [5]. This removes the need for the program to be integrated into the current institutional culture. However, it also removes one of an online program’s most important assets: full-time faculty support. Creating a new administrative unit not aligned with the online program’s mission and goals could result in widespread hostility. Obtaining the full support of dedicated faculty members is almost always in an institution’s their full support would be in its best interest.

Online Education Innovation Strategies to Gain Support …

195

7 Conclusion The rise of online education has presented new challenges for educational institutions, but it has also opened up new opportunities for innovation. This paper has explored the importance of online education innovation strategies. The effective use of these strategies can lead to gaining support from stakeholders and achieving team goals more efficiently. By employing successful strategies to overcome these challenges, institutions can enhance the educational experience for students and instructors alike. As the education landscape continues to evolve, it is important for institutions and online education leaders to understand the significance of innovation strategies in ensuring success and sustainability in online education.

References 1. Bartell, M.: Internationalization of universities: a university culture-based framework. High. Educ. 45(1), 43–70 (2003). https://doi.org/10.1023/A:1021225514599 2. Bolman, L., Deal, T.: Reframing Organizations: Artistry. Choice, and Leadership. Wiley, Hoboken (2008) 3. Bunk, J., Li, R., Smidt, E., Bidetti, C., Malize, B.: Understanding faculty attitudes about distance education: the importance of excitement and fear. Online Learn. 19(4), 132 (2015). https://doi. org/10.24059/olj.v19i4.559 4. Cameron, K.S., Freeman, S.J.: Cultural congruence, strength and type: relationships to effectiveness. Res. Organ. Chang. Dev. 5, 23–58 (1991) 5. Carroll, N., Burke, M.: A framework for developing an online degree program. J. Acad. Bus. Educ. 12, 101–112 (2011) 6. Castillo, T.M.: Traditional faculty resistance to the corporatization model in continuing education: a case study (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database. (Order No. 10288299) (2017) 7. Chen, B.: Barriers to adoption of technology mediated distance education in higher education institutions. Q. Rev. Distance Educ. 10, 333–338 (2009) 8. Ciabocchi, E., Ginsberg, A.P., Picciano, A.G.: A study of faculty governance leaders’ perceptions of online and blended learning. Online Learn. 20(3) (2016). https://doi.org/10.24059/olj. v20i3.974 9. Cleveland-Innes, M., Gauvreau, S.: Faculty role change. Eur. J. Open Distance eLearning EDEN Special Issue 134 (2015). http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-206494 10. Cummings, T.G., Worley, C.G.: Organizational Development and Change, 5th edn. West Publishing, Minneapolis (2014) 11. Deal, T., Kennedy, A.: Corporate Culture: The Rites and Rituals of Corporate Life. AddisonWesley, Reading (1982) 12. Fralinger, B., Olson, V.: Organizational culture at the university level: a study using the OCAI instrument. J. Coll. Teach. Learn. 4(11), 85 (2007). https://doi.org/10.19030/tlc.v4i11.1528 13. Hill, L.A.: Becoming the BOSS. (cover story). Harv. Bus. Rev. 85(1), 48–56 (2007) 14. Hofman, R.H., Hofman, W.H.A., Guldemond, H.: School governance, culture, and student achievement. Int. J. Leadersh. Educ. 5(3), 249–272 (2002). https://doi.org/10.1080/136031202 760217009 15. Hunt, H.D., Davies, K., Richardson, D., Hammock, G., Akins, M., Russ, L.: It is (more) about the students: faculty motivations and concerns regarding teaching online. Online J. Distance Learn. Adm. 17(2), 62–71 (2014)

196

J. Evanick

16. Jaschik, S., Lederman, D.: The 2013 Inside Higher Ed survey of faculty attitudes on technology. Inside Higher Education (2013). https://www.insidehighered.com/news/survey/survey-facultyattitudestechnology 17. Jaschik, S., Lederman, D.: The 2016 Inside Higher Ed survey of faculty attitudes on technology. Inside Higher Education (2016). https://www.insidehighered.com/booklet/2016-sur vey-faculty-attitudestechnology 18. Lloyd, S.A., Byrne, M.M., McCoy, T.S.: Faculty-perceived barriers of online education. J. Online Learn. Teach. 8(1), 1–12 (2012) 19. Mansbach, J., Austin, A.E.: Nuanced perspectives about online teaching: mid-career and senior faculty voices reflecting on academic work in the digital age. Innov. High. Educ. 43(4), 257–272 (2018). https://doi.org/10.1007/s10755-018-9424-4 20. McBride, K.: Leadership in higher education: handling faculty resistance to technology through strategic planning. Acad. Leadersh. J. 4, 260 (2010) 21. Mitchell, L.D., Parlamis, J.D., Claiborne, S.A.: Overcoming faculty avoidance of online education: from resistance to support to active participation. J. Manag. Educ. 39(3), 350–371 (2015). https://doi.org/10.1177/1052562914547964 22. Nicoll, L.A.: Bringing education online: Institutional logics in the legitimation of and resistance to online higher education (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (Order No. 1785398290) (2016) 23. Oomen-Early, J., Murphy, L.: Self-actualization and E-learning: a qualitative investigation of university faculty’s perceived barriers to effective online instruction. Int. J. E-Learn. 8(2), 223–240 (2009) 24. Orr, R., Williams, M.R., Pennington, K.: Institutional efforts to support faculty in online teaching. Innov. High. Educ. 34(4), 257 (2009). https://doi.org/10.1007/s10755-009-9111-6 25. Rock, D.: Managing with the brain in mind. Strategy+Business (56) (2009). https://www.str ategy-business.com/article/09306?gko=9efb2 26. Schein, E.H.: The Corporate Culture Survival Guide. Jossey Bass, San Francisco (1999) 27. Schein, E.H.: Organizational Culture and Leadership, 4th edn. Jossey Bass, San Francisco (2010) 28. Schein, E.H.: Organizational Culture and Leadership, 5th edn. Wiley, Hoboken (2017) 29. Shea, P., Bidjerno, T., Vickers, J.: Faculty attitudes toward online learning: failures and successes. SUNY Research Network (2019) 30. Sporn, B.: Managing university culture: an analysis of the relationship between institutional culture and management approaches. High. Educ. 32(1), 41–61 (1996). https://doi.org/10.1007/ BF00139217 31. Tabata, L., Johnsrud, L.: The impact of faculty attitudes toward technology, distance education, and innovation. Res. High. Educ. 49(7), 625–646 (2008). https://doi.org/10.1007/s11162-0089094-7 32. Tannehill, D.B., Serapiglia, C.P., Guiler, J.K.: Administrative or faculty control of online course development and teaching: a comparison of three institutions. Inf. Syst. Educ. J. 16(3), 26–34 (2018) 33. Tierney, W.G.: Organizational culture in higher education. J. Higher Educ. 59(1), 2–21 (1988). https://www.tandfonline.com/doi/abs/10.1080/00221546.1988.11778301

Overcoming Learning Gaps and Building Transferability Skills in a Higher Education Math Course Subhadra Ganguli

Abstract The “unfinished learning” resulting from the COVID-19 pandemic has led high school graduates to remain unprepared for college courses in the post-pandemic era. The “unaddressed consequences” of this academic “deficit” can create financial losses both for the individual as well as the economy at large. The impact on the US economy could manifest in the form of slower economic growth and loss of GDP amounting to approximately USD 150 billion by 2040. Personal ramifications could result in a lifetime of accrued income loss for employees and workers. This research considers the application of Universal Design for learning (UDL) combined with technology in a Foundation Math course for helping students bridge the learning gap and successfully complete the course. Technology-driven UDL application can enhance the learning experience by encouraging the development of transferable skills (in preparation for future jobs), provide choices of representation of learning materials to students and provide students with a choice of expression in areas of assessment. Teamwork and presentation with technology based UDL framework can develop a student’s lifelong transferable skills for future jobs. Keywords UDL · Technology · Transferable skills · Higher education · Collaborative assessments · Math · Learning gap

1 Background of Student Learning After COVID-19 By the end of the 2020–21 school year, students were, on average, five months behind in Math and four months behind in reading. The impact of such “unfinished learning” is likely to lead to a potential annual GDP loss of $128 billion to $188 billion by 2040 [1]. It is estimated that with the “unfinished learning” and without any immediate intervention in bridging the gap, students could suffer potential lifetime income losses up to $60,000 approximately coupled with lowered output and growth for the US S. Ganguli (B) Penn State University Lehigh Valley Campus, Lehigh Valley, PA 17815, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_16

197

198

S. Ganguli

economy [1]. Universal Design for Learning (UDL) is one of the powerful tools available to level the playing field by granting wider accessibility to students from different backgrounds. This paper considers the application of UDL for teaching Foundations Math course at the Bloomsburg University Campus of Commonwealth University of PA during 2021–23. The paper demonstrates, through evidence-based research, how the use of UDL in teaching and learning of Foundation Math course for Freshmen students can help to bridge the COVID-19 learning gap and further build transferability skills for future jobs in their career.

2 Universal Design for Learning or UDL UDL originated in the 1950s as a principle to help students with disabilities navigate through buildings. UDL is now a framework with the potential for accessibility, inclusivity, and equality in education across secondary and post-secondary levels. The three pillars of this framework can be illustrated by performing multiple methods of representation; multiple methods of expression and multiple methods of engagement for students. Apart from dealing with “unfinished learning” students entered college during post—COVID-19 era with sub-par social skills and poor mental health. Given the diversity in student levels of preparedness for college education particularly in STEM and specifically in Math during post—COVID-19 years, it was necessary to provide as much flexibility to students to catch up on the learning backlog without affecting their progression in college courses. This paper shows how UDL can provide a robust platform to help students overcome lack of prior knowledge and bridge the academic gap, due to COVID-19 pandemic, necessary for post-secondary education in Math. This paper provides evidence-based research of UDL application to Math teaching and learning during 2021–23 at Bloomsburg University of PA. Finally, it discusses how transferable skills can be fostered in the process of teaching and learning of Math using technology driven UDL.

3 Three Principles of UDL Implemented in Math Course 3.1 Multiple Methods of Representation UDL has the advantage of being able to reach out to diverse learners with unique learning abilities [2]. Textbooks offer little or no flexibility for a diverse group of students that have different learning needs [3]. However, technology can enable various ways of delivering the same information to students. I adopted a Math courseware (MyLabMath) by Pearson Publishing for teaching and learning Foundation Math course. The courseware costs less than the market price through the Inclusive Access system provided to universities in collaboration with Pearson Publishing. The

Overcoming Learning Gaps and Building Transferability Skills …

199

e-text embedded in the course can be printed as hard copy at a minimum additional cost for those who are not comfortable with digital print due to disabilities or other reasons. The e-text has several customized features such as online note-taking ability, highlighting e-text ability, access to optional Power Point slides for a summary of the e-text, animations, and videos. It also provides links to external resources to cater to advanced learners. Figures 1a and 1b demonstrate the features of multiple methods of content representation in the course. Figure 1a shows how the courseware provides multiple options for learners to access information provided in the e-text. Use of technology in learning breaks down learning barriers and provides options for students to express themselves in multiple ways [4]. Options provided in the e-text help students access the same information through different means, thereby appealing to different learning abilities, including disabilities. Figure 1a displays the options available for reading and learning from the e-text in the courseware provided for the Math course. Figure 1b shows that students can use PowerPoint slides on each topic of the e-text as well as videos in the Animation section. UDL acknowledges that students learn in a variety of ways and technology-driven learning materials can offer such customized solutions to students.

3.2 Multiple Means of Expression The second principle of UDL caters to “expression” of learning outcomes. This is manifested by providing students the flexibility and options in their submission for grading by the instructor. Based on prior knowledge, skills, and abilities each student performs better than others in certain tasks. Students may feel anxiety and stress during traditional timed tests. [3] discusses the use of scaffolding and support for students to allow them the use of multiple expression formats e.g., reviews of work before submitting a project, several practice sessions for high stake assessments like midterm and final exams, and several attempts on homework assignments before submission, providing formative assessments with guided help to learn from mistakes and advanced readings for students with prior knowledge in the course.These are some of the UDL based approaches to help students overcome challenges they face with traditional time based assessments. In the Math course, my students use a variety of assessments which include technology-based online tests on MyLabMath, group presentations, online homework with flexible submission dates after several attempts, unlimited number of practice problems as part of a study plan providing immediate feedback mechanism, practice problems with numerous simulations and selfreflection exercises. More practice opportunities for students, multiple attempts on homework assignments and instant feedback on student attempts encourage students to bridge the learning gap and complete the course successfully. At the start of the course, a review chapter is covered carefully to help all students gain prior knowledge due to possible lack of preparation in high school. Options incorporating flexibility and choices of topics to practice for review (See Figs. 1a and 1b) provide students

200

S. Ganguli

Home

Contents

Search

Notebook

Flash Cards

Video Channels

Change Text Size for Better Readability

(a)

(b) Fig. 1 a Multiple methods of representation using technology. b Multiple methods of Learning using technology

Overcoming Learning Gaps and Building Transferability Skills …

201

with the much-needed help to overcome the lack of preparation for attending college Math courses. Figure 2 shows that students are given a maximum of five attempts for each weekly homework assignment before their final submission. Pearson’s MyLabMath adjusts the final grade by calculating only the best of all attempts in each assignment. This is to help students overcome barriers to master a topic. MyLabMath includes both formative and summative assessments. Figure 2 shows UDL application in Math homework assignments by providing multiple attempts in each assignment. Student Feedback on MyLabMath Based Practice of Math Problems. Students provided feedback as part of Midterm review of the course and mentioned how useful the Math practices have been for them to master topics for exams. These include

Fig. 2 Students attempt homework multiple times before final submission

202

S. Ganguli

MyLabMath based Homework problems with multiple attempts, practice quizzes in class with multiple attempts and instant feedback at the end of each practice. • “One thing that can continue is all the great homework problems, they’re awesome practice, and all the practice in class we do.” • “Something we need to continue is how you go over your problems and explain them until everyone understands them.” • “I believe we should continue doing the practice quizzes and correcting our mistakes.” • “One thing we should continue is the practice quizzes, they really strengthen my knowledge of the material.” • “I think you are doing good with the practice tests to continue.” • “The practice quiz helps give an idea of what to study more for the real quiz”. • “Continue-Small quizzes to work on types of problems.” • “One thing we need to continue is doing more practice quizzes to help us better prepare for the tests”. Students feel that the practice and flexibility built in the courseware helps them to overcome their fears around high-stake exams and perform decently in those. [3] mentions such support as part of UDL application for helping students to succeed through scaffolding and support. Self-Reflection for Metacognition: Students are required to reflect on their learning of Math using self-reflection exercises. This is a free format assessment graded on completion where students ruminate on their performance in low stake assessments and develop metacognition [5]. They then work on the areas of improvement in readiness for the high-stake assignments. Hence low stake or formative/practice assessments and metacognition help them prepare for the high stakes assessments [6] (Fig. 3). Collaborative Project or Team Presentation. Students are encouraged to form teams and develop project-based presentations, where they apply their in-class learning of Math tools to a real-world business of choice. [7] mentions that group or team presentations have both positive and negative outcomes, yet it is one of the most effective ways to make students work together. This is a summative assessment and has the maximum weightage (25%) towards the final grade. UDL allows students to self-select themselves into teams, select their team leaders, work in teams, present their problem-based projects, and assess performance of other teams (except their own) using peer evaluation. [8] demonstrated through action research that students benefitted in their research projects when provided with peer feedback. The detailed phases of the presentation project remain outside the scope of the current paper. The formation of teams, using self-selection methods, leads to collaboration among peers [9–11]. Another study [7] mentions that preparation of the presentations also develops interpersonal skills and learning skills of students. Teamwork leads students to prepare three phases of presentation namely, create their business; and provide

Overcoming Learning Gaps and Building Transferability Skills …

203

Fig. 3 Self-reflection leads to Metacognition

solutions to two important problems in their business applying Math tools learnt in the course [9–11]. At the beginning of the course, I provided step-by-step guidance for preparation of team presentations. However, students have the flexibility to prepare presentations using their imagination, creativity, Math knowledge and applications with technology. Figure 4 shows the theoretical model of collaborative learning which uses UDL and technology to develop transferable skills among students. Figure 4 explains the steps in the preparation and final presentation of the team presentation where students get to choose to work with fellow peers and present a project where they apply their in class learning of Math to real world problems of business to solve them. Figure 5 presents a sample extract from D2L and illustrates a student exchange prior to team formation. Students are allowed to interact with one another through messages posted on the D2L learning platform of Bloomsburg University of PA. Based on their special interest in a business which they may want to set up from scratch or a company they would like to work in the future, they self-select themselves in teams to work with fellow peers. After teams are formed, each team submits a contract specifying roles and responsibilities of team members throughout the semester. All teamwork is focused on the preparation and delivery of presentation in three phases.

204

Fig. 4 Collaborative model of learning with UDL and technology

Fig. 5 Sample of student interactions prior to team formation

S. Ganguli

Overcoming Learning Gaps and Building Transferability Skills …

205

Table 1 demonstrates the broad guidelines of structure that the presentations should follow. Teams prepare presentations using creativity, imagination, interest in business and willingness to solve a real-world problem by applying knowledge of Math learned in the class. UDL provides the framework whereby students are encouraged to prepare their preferred form of presentation using their choice of business. This idea of applicability of in-class knowledge has the potential to create lifelong learners. The transferable skills in Table 2 as provided by the National Association of Colleges and Employers, NACE (2021) [12] are the skills students can learn in the process of preparation and submission of the project. At the end of each phase of presentation (there are three phases to the presentation), students are asked to self-reflect, as a team, on various skills they have exercised or developed in the process of presentation preparation and delivery. Figure 6 presents the guidance shared with the teams to reflect on their skills-based actions. Figure 7 presents sample feedback from one of the teams on their reflection of skills they practiced. The team picked three important areas of transferable skills namely communication, time management and teamwork and expressed their challenges and successes through their team collaboration in exercising the skills. Moreover, learning gaps from COVID-19 period are partly bridged as students interact and learn from one another in teams [13]. UDL’s multiple engagement methods include various ways faculty members can create opportunities in the classroom for students to be involved in the learning process. It may include mini lectures, discussions, games, presentations, short quizzes and discussions, applications of in-class learning in Math and many others. This part of UDL is beyond the scope of the current research. Boston’s Suffolk University has applied UDL in its Math department after noticing a decline is success rates in First level Math courses. Taking student diversity into account, the department started Table 1 Sample guidelines for team presentations Slide

Expected content

Cover slide

Team: Business Solutions for in Marketing/Accounting/ Finance/Economics/Other

Slide 1

Introduction to business or company • Business type • Product name • Client profile • Details of product and use and benefits

Slide 2

Explain the problem you want to solve

Slide 3

Explain the solution method/technique used

Slide 4

Write the problem in mathematical language and solve it Also mention the problem in simple language and explain the steps in nonmathematical language

Slide 5

Conclusion

Slide 6

References

206

S. Ganguli

Table 2 Definition of transferable skills and measurable metrics Transferable skill or core competency

Definition based on NACE (National Association of Colleges and Employers)

Career & Self-Development

Proactively develop oneself and one’s career through continual personal and professional learning, awareness of one’s strengths and weaknesses, navigation of career opportunities, and networking to build relationships within and without one’s organization

Oral and Written Communication

Clearly and effectively exchange information, ideas, facts, and perspectives with persons inside and outside of an organization

Critical thinking

Identify and respond to needs based upon an understanding of situational context and logical analysis of relevant information

Equity and Inclusion

Demonstrate the awareness, attitude, knowledge, and skills required to equitably engage and include people from different local and global cultures. Engage in anti-racist practices that actively challenge the systems, structures, and policies of racism

Professionalism

Knowing work environments differ greatly, understand, and demonstrate effective work habits, and act in the interest of the larger community and workplace

Leadership

Recognize and capitalize on personal and team strengths to achieve organizational goals

Teamwork

Build and maintain collaborative relationships to work effectively toward common goals, while appreciating diverse viewpoints and shared responsibilities

Technology

Understand and leverage technologies ethically to enhance efficiencies, complete tasks, and accomplish goals

Source https://www.naceweb.org/uploadedfiles/files/2021/resources/nace-career-readiness-compet encies-revised-apr-2021.pdf, last accessed 3/13/2023

teaching math using practical and real-life applications leading to positive student feedback [14].

Overcoming Learning Gaps and Building Transferability Skills …

Fig. 6 Instructions on self-reflection of transferable skills

Fig. 7 Skills development as part of self-reflection by teams (a sample)

207

208

S. Ganguli

4 Conclusion The “unfinished learning” in Math from COVID-19 pandemic led to unprepared Freshmen students taking college level Math courses during 2021–23 at Bloomsburg University of PA. Unpreparedness of incoming Freshmen students is an “environmental” problem in education [3]. There are “individual” problems as well where students may not be engaged or display under-par performance due to several reasons (may or may not be related to disability) [3]. Creating a universal solution to the problem will be more cost effective for institutions. Technology driven UDL has been implemented to remove barriers for students with learning gap in Foundation Math course post COVID-19 pandemic period. Flexibility and choices incorporated into the UDL based system such as flexible submission dates of assignments, multiple attempts on assignments, instant feedback on attempted problems on MyLabMath help students to complete course successfully. Finally, in the process of engaging themselves in UDL based collaborative assessments in teams, students get the opportunity to practice and develop transferability skills. UDL based learning approaches can remove “environmental” problems of “unfinished learning” from the pandemic without having to consider “individual” students on a case-by-case basis. Moreover, it can prepare them for future jobs by inculcating transferability skills. Current research has been applied to economics courses which are general education courses in areas of Microeconomics and Macroeconomics. Similar methods have been followed in those courses using online courseware to complement in-person or synchronous virtual classes using UDL to build transferability skills. Further research can be initiated by universities to examine how examples used in this paper can be further extended to other areas of learning and teaching for removing barriers to learning as well as enhancing employability skills for future job seekers.

References 1. Mckinsey & Company: COVID-19 and education: The lingering effects of unfinished learning. mckinsey.com. Accessed 13 Mar 2023 2. Rose, D.H., Meyer, A.: Teaching Every Student in the Digital Age: Universal Design for Learning. Association for Supervision and Curriculum (2002). Development, 1703 N. Beauregard St., Alexandria, VA 22311-1714 3. Rose, D.H., Harbour, W.S., Johnston, C.S., Daley, S.G., Abarbanell, L.: Universal design for learning in postsecondary education: reflections on principles and their application. J. Postsecondary Educ. Disabil. 19(2), 135–151 (2006) 4. Rose, D.H., Meyer, A., Hitchcock, C.: The Universally Designed Classroom: Accessible Curriculum and Digital Technologies. Harvard Education Press, Cambridge (2005) 5. McGuire, S.Y.: Teach Students How to Learn: Strategies you Can Incorporate into Any Course to Improve Student Metacognition, Study Skills, and Motivation. Stylus Publishing LLC, Sterling (2015) 6. Lang, J.M.: Small Teaching: Everyday Lessons from the Science of Learning. Wiley, New York (2021) 7. Michaelsen, L.K., Knight, A.B., Fink, D.L.: Team-Based Learning: A Transformative Use of Small Groups in College Teaching. Stylus Publishing, Sterling (2004)

Overcoming Learning Gaps and Building Transferability Skills …

209

8. Weaver, R.L., Cotrell, H.W.: Peer evaluation: a case study. Innov. High. Educ. 11(1), 25–39 (1986) 9. Weimer, M.: Designing Small Group Activities: A Resource Guide (2019). www.teachingp rofessor.com. https://www.teachingprofessor.com/topics/preparing-to-teach/designing-smallgroup-activities-a-resource-guide/. Accessed 13 Mar 2023 10. Weimer, M.: Leadership in Student Groups. The Teaching Professor (2021). www.teachi ngprofessor.com. https://www.teachingprofessor.com/topics/for-those-who-teach/leadershipin-student-groups/. Accessed 13 Mar 2023 11. Weimer, M.: Exploring How Students Select Group Members. The Teaching Professor (2021). www.teachingprofessor.com. https://www.teachingprofessor.com/topics/resource-collections/ assignments-of-note/exploring-how-students-select-group-members/. Accessed 13 Mar 2023 12. www.nace.org. https://www.naceweb.org/uploadedfiles/files/2021/resources/nace-career-rea diness-competencies-revised-apr-2021.pdf. Accessed 13 Mar 2023 13. Darby, F., Lang, J.M.: Small Teaching Online: Applying Learning Science in Online Classes. Jossey-Bass, New Jersey (2019) 14. Behling, K.T., Tobin, T.J.: Reach Everyone, Teach Everyone: Universal Design for Learning in Higher Education. West Virginia University Press, Morgantown (2018)

Evaluating the Effectiveness of a New Programming Teaching Methodology Using CodeRunner Siba Haidar, Antoun Yaacoub, and Felicia Ionascu

Abstract Modern teaching technologies, such as CodeRunner, significantly enhance the quality of programming courses and contribute to the ongoing revolution in teaching. This paper presents our experience using a novel and innovative approach to teach and assess programming courses through the use of automated grading systems like CodeRunner. Our research exposes the transition from the traditional to the new method and focuses on enhancing student engagement and learning outcomes. The study demonstrates that the utilization of CodeRunner questions in practices, quizzes, and evaluations offers valuable insights into the academic progress and learning outcomes of students. These findings have important implications for informatics education and can guide educators in adopting innovative and effective teaching methods. Keywords CodeRunner · Automated grading · Programming courses · Student engagement · Learning outcomes · Innovative teaching methods · Pedagogical innovation

1 Introduction Over the past two decades, the teaching of programming has undergone significant changes because of advancements in technology and changes in educational practices. In the early 2000s, programming education was often characterized by traditional lecture-based instruction, with limited opportunities for hands-on practice or peer collaboration. As technology evolved, however, educators began to explore new approaches to teaching programming, including online courses and interactive S. Haidar (B) · A. Yaacoub LDR Lab - ESIEA, 9 rue Vésale, 75005 Paris, France e-mail: [email protected] F. Ionascu ESIEA, 9 rue Vésale, 75005 Paris, France © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_17

211

212

S. Haidar et al.

learning environments. These approaches allowed for greater flexibility and accessibility, as well as more opportunities for student-centered learning and collaboration. Additionally, educators began to experiment with new tools and technologies, such as visual programming environments and game-based learning, to engage students and enhance their learning outcomes. Today, programming education continues to evolve, with a focus on personalized and adaptive learning, as well as the integration of emerging technologies such as artificial intelligence and machine learning. The teaching of programming is a critical topic in the field of education and technology, given the increasing importance of computer science skills in today’s economy. As more industries adopt digital technologies, the demand for skilled programmers has skyrocketed, making programming education a vital component of workforce development. Additionally, programming education can help students develop critical thinking, problem-solving, and logical reasoning skills that are valuable in a wide range of fields. We will aim in this study to evaluate the effectiveness of using CodeRunner for teaching programming and to analyze its impact on student learning outcomes in programming courses. We will seek to provide recommendations for educators interested in incorporating these new methods into their programming courses.

2 Literature Review In today’s rapidly advancing technological landscape, the ability to write code and solve problems efficiently has become increasingly important. As a result, there has been a rise in the popularity of various online platforms and tools designed to assist programmers and problem solvers. We explore three such platforms: CodeRunner, online coding platforms, and problem-solving sites. Each of the following subsections will provide an overview of the platform or tool, highlighting its features and discussing its usefulness in improving coding skills and problem-solving abilities. An enhanced comprehension of the strengths and limitations of these platforms facilitates a better understanding of the advantages of utilizing online resources to enhance programming skills.

2.1 CodeRunner CodeRunner is a popular Moodle plugin that automates the evaluation of students’ programming assignments. It supports a wide range of programming languages and includes built-in tests that allow educators to create questions that require students to write code that performs certain tasks or meets specific requirements. One of the key benefits of CodeRunner is its ability to provide immediate feedback to students, which can be particularly helpful in large programming courses.

Evaluating the Effectiveness of a New Programming Teaching …

213

Several studies have been conducted to evaluate the effectiveness of CodeRunner in various educational contexts. For example, Hatano [6] proposed a framework for analyzing the logs generated by CodeRunner that can help teachers to understand how students approach programming problems and identify their mistakes. Aldriye et al. [1] reviewed various automated grading systems, including CodeRunner, and highlighted the need for more beginner-friendly and localized systems. Pringuet et al. [9] used CodeRunner for formative and summative assessments in a blended/online teaching and learning methodology for a first-year undergraduate programming curriculum. The study found that CodeRunner was useful for providing timely feedback and motivating learners to engage in their learning. Wünsche et al. [12] adapted CodeRunner to OpenGL assignments for computer graphics courses and proposed three extensions to address issues with formative feedback and experimentation with OpenGL code. Croft et al. [3] reported on using CodeRunner for summative assessments in a first-year undergraduate programming curriculum at Coventry University. The study reported changes in key metrics following its use. Chauhan [2] used CodeRunner in a model for teaching database courses and found that the model was advantageous for learning and teaching practices. Tangaraj et al. [10] discussed the challenges of teaching introductory programming modules in higher education and the use of automated assessment and feedback systems, including CodeRunner. CodeRunner is, thus, a valuable Moodle plugin that supports automated grading, formative and summative assessments, and feedback for programming assignments. Its customization options, support for multiple programming languages, and immediate feedback capabilities make it a powerful tool for enhancing student learning outcomes and engagement in programming courses.

2.2 Online Coding Platforms Online coding and programming platforms have become increasingly popular in recent years, providing a flexible and accessible way to learn and practice programming skills. These platforms offer a range of features such as interactive coding environments, automated testing, and gamification to engage and motivate learners. Zinovieva et al. [13] highlighted the potential of online programming simulators in organizing effective distance learning systems, recommending their use as an additional tool for teaching computer science disciplines. The study compares different online platforms for teaching programming and emphasizes the importance of functionality and level of student preparation in selecting an appropriate platform. The paper also notes the benefits of gamification in increasing cognitive activity and improving the quality of the educational process. Di Mascio et al. [5] focused on personalized competitive programming training and proposed a framework that allows for personalized recommendations to learners about the next programming problem to undertake. The paper emphasizes the importance of training for programming contests, which emphasizes the algorithms and

214

S. Haidar et al.

data structures used to compose solutions to problems, as well as the quality of the solution program itself. The framework proposed in this paper utilizes a badge-based, gamified approach to foster motivation and engagement among learners. Di Luigi et al. [4] focused on learning analytics in competitive programming training systems. The study analyzes data collected from the OII-web platform, which is designed to train students for the Italian Olympiads in Informatics. The paper compares two distinct groups of users in two separate platforms built on OII-web, one for students and one for teachers. The findings indicate that the two groups are more similar than expected when it comes to programming contest training. Pervolarakis [8] discussed the development of an automatic code grading platform called Eurytus. The platform is designed to assist in fast and efficient programming learning, with features such as input and output tests, structure checking, and design pattern checking. The platform also allows individual users to create and publish programming challenges for others to join, providing an opportunity for community members to improve their programming skills. Online coding platforms and problem-solving sites are web-based platforms that provide students with a virtual environment for coding and testing their programs. These platforms have become increasingly popular in recent years, as they offer a range of benefits for programming education. One of the key benefits of online coding platforms is that they provide students with immediate feedback on their code. As students write and test their programs on the platform, the platform can automatically check the code for errors and provide feedback on how to correct them. This immediate feedback can help students learn from their mistakes more quickly and improve their programming skills faster.

2.3 Problem Solving Sites Problem-solving sites are web-based platforms that provide programming challenges and problems to students, offering numerous advantages for programming education and recruitment. These sites offer real-world programming challenges, ranging from basic programming exercises to advanced algorithmic problems, that help students develop critical and creative problem-solving skills and design effective and efficient code. Moreover, they provide community features that allow students to collaborate with other programmers, share tips, advice, and code snippets, and learn from each other, accelerating their programming skills development. These platforms also offer feedback mechanisms that help students improve their coding efficiency and conform to programming best practices. Some popular problem-solving sites include HackerRank, CodeWars, and LeetCode. Kumar et al. [7] developed a performance-based coding test platform that provides real-time feedback to help companies assess candidates’ coding skills for performance benchmarking and effective recruitment. The platform offers comprehensive programming libraries to administer coding tests for hiring programmers, including Python, Java, C programming, SQL, and PHP developer tests. With this coding test

Evaluating the Effectiveness of a New Programming Teaching …

215

platform, organizations can design an efficient developer recruitment cycle, bridge skill gaps for employee career progression, build customized coding scenarios based on business needs, and tailor pre- and post-training requirements. The application developed with Python and Django allows for easy implementation, input, and output of code. Furthermore, Vamsi et al. [11] proposed a framework for auto-classifying programming problems on online coding practice platforms, like HackerRank, into easy, medium, and hard levels based on attempt statistics for each problem. This framework helps students choose the right problems to solve on a learning track, boosting their motivation levels and improving their learning outcomes. This automatic classification system helps students to determine which problems to solve based on their skill level, providing a more personalized and effective learning experience.

3 Methodology Now that we have explored the current state of the art, we discuss the implementation of our new method in programming courses and share observations about the course transformation and student reactions.

3.1 Implementation of the New Method in the Programming Courses Our team implemented the new method in a range of programming courses across multiple years of study, as shown in Table 1. In the second undergraduate year, we implemented it in the Advanced Programming & Graph Applications course. In the third undergraduate year, it is used in Object-Oriented Programming as well as Python as an Engineering Tool courses. In the first graduate year, the new method is implemented in three courses: Optimization in Python, Image Processing, and Machine Learning. Table 1 Courses by semesters and acronyms Semester

Course

Acronym

4

Advanced Programming & Graph Applications

APG

5

Object-Oriented Programming

OOP

6

Python as an Engineering Tool

PET

7

Optimization in Python

OP

7

Image Processing

IP

8

Machine Learning

ML

216

S. Haidar et al.

We developed an interactive course involving student participation during the lectures. In addition, we elaborated guided tutorials containing a series of independent but chapter-related exercises, as well as practical assignments, in which students solved mini-projects divided into many sub-tasks. To create these exercises, we team mixed different types of Moodle questions, using the appropriate type to deliver each idea. This included multiple-choice questions (MCQs) and other question types. However, we primarily used CodeRunner exercises, programming hundreds of exercises with numerous test cases in order to demonstrate to students the limitations of poor programming and code. Data analysis procedures involved collecting data on student performance on the programming exercises and using this data to evaluate the effectiveness of the new technological methods. We collected data on student grades and used this information to identify areas where students were struggling and to refine the course materials and exercises accordingly. In the remainder of this section, we will focus on the Object-Oriented Programming in Java course, at the undergraduate level. Before Adopting the New Approach. The course had been taught in the school in the past years by a classical mix of lectures, tutorials, and practical work. One teacher was in charge of a cohort of around 125 students, dispensing 9 h of lectures for the entire cohort and 15 h of practical work per class. An equivalent number of hours of supplementary autonomous work was expected from the students. The background of the students attending this course is very diverse. At one end of the knowledge spectrum, some of the students have good bases of objectoriented programming in Java. At the other end of the knowledge spectrum, some students have very little programming experience in procedural languages or even no experience at all. At the end of the course, the aim was that students without previous object-oriented programming experience were able to run a Java program at the console; were able to write basic Java programs with classes; had notions of abstractions, inheritance, and encapsulation; made a basic use of collections; were able to manipulate input and output streams; and understood basic UML class diagrams. The subject of the practical works was printed on paper and handed to the student, with step-by-step instructions on what was expected from the student and how to achieve it. Evaluation was composed of graded practical works, a small project, and an exam on paper. One of the three practical works, randomly chosen, used to be evaluated by running unitary tests, the small project was evaluated by assessing the quality of the code, and executing acceptance testing. After Adopting the New Approach. With the new approach, introduced in the 2021–2022 academic year, the same course is taught by two teachers dispensing 12 h of lectures for a cohort of 125 students and 24 h of practical work per class. The background of the students stays the same. At the end of the course, the aim is that students without previous Java programming experience are able to: write

Evaluating the Effectiveness of a New Programming Teaching …

217

intermediate-level Java programs with classes, abstractions, inheritance and polymorphism, use interfaces, create generic classes, use collections in a more advanced manner, manipulate input and output streams, understand basic UML class diagrams. Meanwhile, we sought to provide opportunities for more advanced students to deepen their knowledge. We used Moodle with CodeRunner for all the assignments and for the exam. Evaluation was composed of 2 practical works, 4 guided tutorials, one mini-project, and an exam. The students have no notion of unit testing. For some exercises, penalties were given for erroneous attempts. At the exam, no penalty was applied for coding exercises.

3.2 Implementation of the New Method in the Programming Courses In the following, we detail the course transformation highlighting the qualitative benefits. Next, we summarize the student reaction. Course Transformation. The introduction of the new teaching method with CodeRunner coincided with a management decision to increase the face-to-face time spent in practical works by a factor of 1.6 as shown in Table 2. Nevertheless, thanks to the immediate feedback of CodeRunner (compilation errors, automatic testing) the number of exercises that the students solved was increased by a factor of 5.66. Before CodeRunner, the teacher time spent in grading was very long; despite this effort, few qualitative feedback was given to the student about his or her achievement. Introduction of the assessment with CodeRunner led to an increase in teacher time spent in preparation but to a decrease in teacher time spent in grading. In the second year of CodeRunner, time spent in preparation dropped sharply and time spent in grading went down to zero because the mini-project was also assessed with CodeRunner. Given that, effort spent on preparation is reusable from one year to Table 2 Statistics related to the assessments Year

Total face to face per student [min]

Teacher Time/ Student/ exercise

Teacher time preparation [h]

Teacher No. of Mini-Project Exam with time exercises in Moodle and grading CodeRunner CodeRunner [h]

2020–2021 36

3 min

45

60

12

No

No

2021–2022 58

1 min

90

20

61

No

Yes

2022–2023 58

51 s

12

0

68

Yes

Yes

2020–2021: classical method; 2021–2022: first year with Moodle and CodeRunner; 2022–2023: second year with Moodle and CodeRunner

218

S. Haidar et al.

another, while effort spent on grading is not. With CodeRunner, the focus can now be shifted to activities with more added value for the quality of the course. Using CodeRunner during the practical works and tutorials led to more teacher time dedicated to explanations and less teacher time dedicated to solving individual student issues related to the development environment. It also allowed addressing a larger palette of exercises, from the most basic to complex or tricky ones, therefore better adapted to the diverse background of the students in the cohort. In the guided tutorials, the exercises were each focused on a Java concept (class, inheritance, generics, object equality, abstraction, etc.), without any logical link between the exercises. In the practical works, the exercises were forming a logical suite, including some more elaborated algorithms to implement a game. We observed the following: 1. It is more straightforward for students to solve exercises of the guided tutorials with CodeRunner. 2. Automatic testing with CodeRunner leads to excellent feedback for the exercises of the practical works because tests executed by the students in their own environment were proven to be incomplete. It should be noted that an important aspect of Java programming, which is part of the course objectives, cannot be simulated using CodeRunner—namely, the Java development environment. This objective has been included in another course. Student Reaction to the New Method. The main issue with continuous assessment with CodeRunner during the practical works is student acceptance. Students with some programming experience preferred to use an IDE to code and test before proceeding with CodeRunner. They were then likely to blame the unit tests implemented in CodeRunner, having less trust in an automatic correction “made by the machine” than in their own tests. The penalties induced by erroneous runs were reasons for stress and some students developed group strategies to avoid them, checking their work on different accounts before getting to the correct version that would lead to no penalty when submitted for verification to CodeRunner. Also, students had a hard time accepting at the beginning that seemingly correct code would lead to a grade of zero because of failed tests. The satisfaction of submitting successfully tested code grew toward the end of the course, together with the level of Java programming skills, and led to better acceptance of the method.

4 Results After discussing the qualitative advantages of our method, we present some statistical measures to validate our approach. We begin by introducing the sizes of the question banks in the different courses. Then we compare the effectiveness of CodeRunner

Evaluating the Effectiveness of a New Programming Teaching … Table 3 Question bank sizes by course

Course

CR

219

Other

Total

APG

71

123

194

OOP

86

130

216

PET

211

157

368

OP

42

75

117

IP

51

24

75

ML

41

105

146

Total

842

884

1726

question type versus other types of Moodle questions. Finally, we validate our quizzes by reference to classical validation measures for exams.

4.1 Question Banks When evaluating the effectiveness of a course or educational program, the size and quality of the question bank can be an important factor to consider. We describe, in Table 3, the question bank sizes for each course, including the total number of questions available for each course along with the distribution of question types between CodeRunner (CR) and other types of questions (Other).

4.2 Facility Index and Discrimination Efficiency In this subsection, we will examine two key measures of the effectiveness of quiz questions—the Facility Index and Discrimination Efficiency—and compare their performance across different courses and question types. Specifically, we will present the averages of these measures for three different courses and analyze how they relate to student performance. The three courses that have been selected here have diverse learning objectives and levels. After presenting our results, we will explore the relationship between question type and these measures, focusing on the CodeRunner question type in particular. The Facility Index (FI) of a quiz question is a statistic that measures the difficulty level of the question. It is calculated using the following formula: FI =

nb_corr ect × 100% nb

(1)

Here, nb_corr ect is the number of students who answered the question correctly, and nb is the total number of students who attempted the question.

220

S. Haidar et al.

The FI value can range from 0 to 100%, with a higher value indicating that the question is easier. A FI of 50% means that half of the students answered the question correctly and half answered it incorrectly, indicating that the question is of moderate difficulty. The FI is a useful tool for evaluating the effectiveness of quiz questions and identifying areas where students may be struggling. However, it should be used in conjunction with other assessment measures and should not be relied upon as the sole method of evaluating student performance. The Discrimination Efficiency (DE) of a quiz question is a statistic that measures how well the question can differentiate between high-performing and low-performing students. It is calculated using the following formula: DE =

Upper 27% score − Lower 27% score Total score range

(2)

Here, the Upper 27% score is the score obtained by the top 27% of the students who took the quiz, while the Lower 27% score is the score obtained by the bottom 27% of the students who took the quiz. The Total score range is the difference between the highest possible score and the lowest possible score for the quiz. The DE will very rarely approach 100%, but values in excess of 50% should be achievable. Lower values indicate that the question is not nearly as effective at discriminating between students of different ability as it might be and therefore is not a particularly good question. Table 4 shows the quizzes structure analysis including FI and DE for CodeRunner type of questions versus other types of questions, for the three chosen courses. The same information is graphically represented in Fig. 1. We can check that the FI of CodeRunner questions varies across different topics. In the topic of OP, CodeRunner questions have a lower FI compared to other types of questions. However, in the topics of OOP and IP, CodeRunner questions have a higher FI compared to other types of questions. The varying FI of CodeRunner questions across different courses is attributed to differences in pedagogical and conceptual goals across those courses. For instance, in the Image Processing course, the CodeRunner questions were designed to be semi-filled, with a small amount of code left for students to complete. The purpose was to introduce different Python libraries available for use in Image Processing, rather than to teach algorithms and programming. In the Object-Oriented course, the relatively large number of CodeRunner Table 4 Quiz structure analysis. Facility index (FI) and Discriminative efficiency (DE) for CodeRunner (CR) type of questions versus other types of questions FI CR [%]

FI Other [%]

DE CR [%]

DE Other [%]

OP

66,38

73,07

57,87

38,04

OOP

79,49

65,57

71,35

41,86

IP

90,92

70,48

66,75

52,22

Evaluating the Effectiveness of a New Programming Teaching …

221

Fig. 1 Quiz structure analysis comparison between CodeRunner and other types of questions

questions (86 versus 42 and 51 in the two other courses) the students had to complete during the course, led them to be more comfortable with programming, which was part of the course objectives. On the other hand, the table shows that CodeRunner questions consistently have a higher Discriminative Efficiency compared to other types of questions in all three topics. This indicates that CodeRunner questions are better at distinguishing between high-performing and low-performing students. In conclusion, the use of CodeRunner questions in quizzes and assessments can provide valuable insights into student performance and learning outcomes. Therefore, incorporating CodeRunner questions can be an effective way to enhance assessment practices and promote deeper learning.

4.3 Evaluation of Final Exam Measures In this subsection, we aim to validate the measures used in our statistical analysis. We considered the statistics of the three final exams (OP, OOP, and IP) done on Moodle including exercises of different types including CodeRunner. In Table 5, we show skewness, kurtosis, standard deviation, and other relevant measures. Despite variations in the measures across the three courses, the measures reflect a consistent examination method for each course. Based on the measures presented in Table 5 Quizzes information

Course

OP

OOP

IP

# grades

255

101

61

Average

54.15%

33.24%

60.53%

Median

52.63%

26.67%

63.92%

Std

17.52%

21.72%

17.17%

Skewness

0.5049

1.0532

−0.8735

kurtosis

0.2917

0.3289

0.6958

222

S. Haidar et al.

the table, we can draw several conclusions about the final exams in each course. In terms of standard deviation, the OP course has a value of 17.52%, the OOP course has a value of 21.72%, and the IP course has a value of 17.17%. All three values fall within the desired range of 12% to 18%, indicating that the scores are not too bunched up and are reasonably spread out. Skewness is a measure of the asymmetry of the distribution of scores. A value of zero indicates a perfectly symmetrical distribution, with scores distributed equally on either side of the mean. Kurtosis is a measure of the “peakedness” of the distribution. A positive kurtosis value indicates a relatively peaked distribution, while a negative kurtosis value indicates a relatively flat distribution. In general, skewness and kurtosis values between −2 and +2 are considered acceptable, as they indicate a relatively normal distribution. The skewness and kurtosis values in the table fall within this range, suggesting that the distributions of grades in the three exams are relatively normal. Therefore, based on this measure, we can confirm that the three exams were welldesigned and validated, as they all have standard deviation values that fall within the desired range. In the context of the literature review, these findings are consistent with previous research that has shown the importance of careful assessment design and grading processes in ensuring fair and accurate evaluations of student learning.

5 Conclusion In conclusion, the adoption of CodeRunner as a teaching method has led to numerous benefits, including increased course quality, the ability to cover more complex topics, and reduced effort in preparation and grading. More importantly, the immediate feedback provided by CodeRunner has encouraged students to adopt good programming practices and be more rigorous in their work. Furthermore, our analysis showed that CodeRunner questions tend to have higher DE scores, indicating their effectiveness in distinguishing between high- and low-performing students. Incorporating CodeRunner questions into assessments can provide valuable insights into student performance and promote deeper learning. The results of our study confirm the importance of careful assessment design and grading processes in ensuring fair and accurate evaluations of student learning. Overall, CodeRunner has proven to be a valuable tool for enhancing assessment practices and promoting student success.

References 1. Aldriye, H., Alkhalaf, A., Alkhalaf, M.: Automated grading systems for programming assignments: a literature review. Int. J. Adv. Comput. Sci. Appl. 10(3) (2019) 2. Chauhan, S.: Implementation of Code Runner for SQL tasks in Database Course (2022) 3. Croft, D., England, M.: Computing with CodeRunner at Coventry University: Automated summative assessment of Python and C++ code. In: Proceedings of the 4th Conference on

Evaluating the Effectiveness of a New Programming Teaching …

4.

5.

6.

7. 8. 9. 10.

11.

12.

13.

223

Computing Education Practice, CEP 2020, New York, NY, USA, pp. 1–4. Association for Computing Machinery, January 2020 Di Luigi, W., Fantozzi, P., et al.: Learning analytics in competitive programming training systems. In: 2018 22nd International Conference Information Visualisation (IV), pp. 321–325, July 2018 Di Mascio, T., Laura, L., Temperini, M.: A framework for personalized competitive programming training. In: 2018 17th International Conference on Information Technology Based Higher Education and Training (ITHET), pp. 1–8, April 2018 Hatano, K.: A framework to analyze logs of coderunnder for improving programming education. In: 8th International Conference on Educational Technologies 2021, ICEduTech 2021 and 17th International Conference on Mobile Learning 2021, ML 2021, pp. 269–270. IADIS Press (2021) Kumar, G.U., Varun, B., Sangameshwar, S.: Coding Website (like Hacker Rank) Using Python and Django, Number: 4827. EasyChair, December 2020 Pervolarakis, M.: Development of an automatic code grading platform, October 2022 Pringuet, P., Friel, A., Vande Wiele, P.: CodeRunner: a case study of the transition to online learning of a Java programming course, June 2021 Thangaraj, J., Ward, M., O’Riordan, F.: Use of assessment and feedback systems for introductory computer programming modules of higher education: a comparative study. In: 8th International Conference on Higher Education Advances (HEAd 2022), Valencia, Spain, 14–17 June 2022. Editorial Universitat Politecnica de Valencia, May 2022 Vamsi, S., Balamurali, V., et al.: Classifying difficulty levels of programming questions on HackerRank. In: Satapathy, S.C., Raju, K.S., et al. (eds.) Advances in Decision Sciences, Image Processing, Security and Computer Vision, Learning and Analytics in Intelligent Systems, pp. 301–308. Springer, Cham (2020) Wünsche, B.C., Huang, E., et al.: CodeRunnerGL - an interactive web-based tool for computer graphics teaching and assessment. In: 2019 International Conference on Electronics, Information, and Communication (ICEIC), pp. 1–7, January 2019 Zinovieva, I.S., Artemchuk, V.O., et al.: The use of online coding platforms as additional distance tools in programming education. J. Phys. Conf. Ser. 1840(1) (2021). Publisher: IOP Publishing

Institutional Effectiveness of Innovative Learning Experiences: How MOOCs Transform and Encourage Lifelong Learning Ryan Hamilton

Abstract Online learning is changing rapidly, and so is the way we absorb information and encourage new delivery methods. The traditional synchronous and in-person methods no longer work for everyone, but corporations want to encourage lifelong learning. So, what are we doing about it? Studies have shown that MOOCs highly encourage the learning experiences of audiences around the globe through innovative technologies and require workplaces to be ready to deploy resources. The article’s purpose is to examine and discuss the practical usage of MOOCs in the classroom and in a corporate environment to encourage life-long learning. This will be done through exploring examples of MOOCs currently on the market, and how effective they are for learning retention. Additionally, recommendations on how to successfully integrate and incorporate these learning modules in learning paths are made. Suggested recommendations include the best path to approach how and why MOOCs work for learners through their independence of a traditional university or professional course, ease of access, and being one of the most cost-effective learning modules in education. Keywords MOOC · Life-long learning · Adult education · eLearning

1 Introduction Research shows that one of the best ways to encourage life-long learning is to provide open opportunities to learners, in a variety of formats [1]. Not only do learners need to have accommodations to their learning paths to achieve their goals, but it provides an opportunity for institutions and companies alike to be able to attract practitioners to sign up for their content [2]. Along with this opportunity comes the integration of massive open online courses (MOOCs) for lifelong learning. MOOCs are defined as any form of online or eLearning content that is accessed by providers with an aim to R. Hamilton (B) Swiss School of Business and Management (SSBM), Geneva, Switzerland e-mail: [email protected]; [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_18

225

226

R. Hamilton

make content that is relevant, flexible, affordable and of good quality [2]. The idea behind MOOCs (massive open online courses), “spearheaded by edX, began as an opportunity for organizations to offer online courses to students all over the world, in the millions, for free” [5]. With a heightened need and desire to learn more at the tip of our fingers, MOOCs solve this problem with high-quality education that is free and can be accessed by anyone around the world. By bringing MOOCs to the corporate environment, companies will be able to ensure that a variety of learning styles are being considered, and that the content that is relevant to individuals is being taken. MOOCs have advanced ten-fold since their inception by including credentials beyond a certificate of completion to programs such as MicroMasters® and Professional Certificate programs [5]. Not only can learners get these certificates and programs, but they also have the ability, through university partnerships, to take MOOCs for the purpose of getting post-secondary credit towards a stated program [5]. MOOCs have advanced well beyond their initial audience of providing open education opportunities for all in the context of micro learning, to now catering to learners of every audience—young students, early career professionals, and senior executives and the list goes on. To provide more context around what lifelong learning is, and its characteristics, we will define it. Lifelong learning is simply the act of taking upon oneself the initiative to learn throughout various stages of life, through work, or personal engagements to upskill, or for interest’s sake [2]. The characteristics of lifelong learning include the following reasons why one may continue to learn: “professional development, career transition, college preparation, supplemental learning … and corporate e-learning and training” [2]. According to UNESCO, they have developed a “Lifelong Learning Initiative” that looks at lifelong learning beyond the typical bounds of education and “emphasizes the possible reconnection of learning to larger social-emotional domains” [2]. This is to say that lifelong learning is not bound to simply being a course to upskill, but to be rethought of as a tool to encourage greater well-being, and overall job, life, and career satisfaction.

2 Background and Problem Identification One of the biggest challenges of MOOCs is their implementation and tracking of its effectiveness on learners [4]. It is not simply enough to identify that a company requires their employees to take part in learning initiatives, it should be expected to identify the competency or outcomes that is a part of their journey [6]. It is easy for someone to sign up for a MOOC and take it, but for a company to promote a culture of life-long learning, it is necessary to clearly articulate the benefits of taking these courses. This can be done by offering solutions for credible resources to obtain this information and checks in place to report back on findings and how the learning relates to on-the-job duties. The question to answer is who will be benefiting from taking part in a MOOC course, and how will this be tracked and assessed [4].

Institutional Effectiveness of Innovative Learning Experiences …

227

One important thing to consider is that even if a learner enrolls in a MOOC, it does not mean that they are going to finish it; they can start, stop, pause or discard the initiative at any time [7]. This is because MOOCs are primarily free and can be looked at as a trial for learning a new topic with no strings attached. One thing to note and highlight is that for many organizations, MOOCs are not a replacement for a company initiated and internally created course that is specific for a role, or group of roles. MOOCs may be viewed more in the space of supplementary learning, not only for an individual’s interest, but also to further their careers with a broader scope to fit their unique learning path and goals. MOOCs are available in almost any subject, including but not limited to science, math, education, business, law, computer science, and design [4]. One of the philosophies behind the invention of the MOOC is that idea of connectivism [4]. Connectivism considers that learning in courses can connect people with support of cognition and retention to advance a current knowledge base [4]. In relation to lifelong learning, connectivism provides enrichment to the idea that learning is not a linear process, and the way in which individuals learn over their lifetime will change significantly, just as MOOCs have been for over a decade. Another consideration is the limitations of MOOCs as it relates to program offerings and how they are delivered. Often, MOOC’s have an end-of course evaluation that can utilize notes and answers saved throughout a course by a learner [2]. There is a limitation around authenticity and ensuring absolute academic integrity when working with MOOC courses, which require a critical eye. As MOOC’s are a primary tool for lifelong learning, it goes without saying that learners will often be taking these courses for interest’s sake, which provide more satisfaction to complete the work on their own [2]. A Work Institute report highlighted the key distinguishing features of corporate learning, and their findings included that lifelong learning is “the conscious choice to pursue knowledge” and the learning that is mandatory and assigned by a workplace is considered “continuous learning” as it is less conscious, and for a report of duty [11].

3 Theoretical Framework The Open University UK conducted a research study on their own programming to better understand their processes and the effectiveness of learning design [3]. One of their findings indicated that learning design activities can be broken down into types of activities and desired outcomes (or categories) to effectively manage assigned learning modules, and assessments [3]. Some of these categories included identifying outcomes such as “finding and handling information,” “experiential” and “interactive/adaptive” [3]. In the context of MOOCs, due to the nature of their relevance in today’s day-in-age, these characteristics all highlight the relevance of utilizing category coding of learning initiatives, even if the actual content is being delivered externally, as this is the case with most, if not all MOOCs.

228

R. Hamilton

As previously mentioned, we must be considering the competence that is the goal of a MOOC, which is “more than just knowledge or skills. It involves the ability to meet complex demands, by drawing on and mobilising psychosocial resources (including skills and attitudes) in a particular context” [6]. This idea is increasingly important given the nature of corporate learning, as things move quickly. An organization must be ready to not only deploy resources but be able to speak to why courses are chosen, and how to approach it [9]. Research conducted by Harvard Health concluded that “we stave off the effects of aging when we keep our brains active by trying new things” [9]. This provides insight into the fact that corporations are considering learning for the sake of improving employee’s well-being or, specifically, for on-the-job training. Lifelong learning certainly has an inadvertent effect on the lives of these individuals. MOOCs are offering independence of a traditional university or professional course, as well as being one of the easiest ways to access new information. Not only this, but by vetting the authenticity of MOOC providers, the quality of education is to a high degree and just as applicable as traditional approaches to programming (i.e., in-class simulations offered by consultants, or universities). The number of courses available in recent years is also at an exceptionally high amount; MOOC courses amounted to having over 60 million new learners in 2020, and over 16,000 new courses by almost 3,000 university and college professionals around the world were announced in the same year [4]. When considering MOOCs to encourage lifelong learning, there are somewhat limited statistics on its effectiveness due to the it being around for just over a decade. The key research that is available around key learner statistics includes the following: by 2020, there were over 180,000,000 active students in MOOCs around the world, from 950 universities, which also included over 30,000 courses for learners to take [2]. The learning landscape has changed significantly in just over a decade, with learners being able to take brand new courses daily and have thousands of new credentials being created by hundreds of MOOC providers. Research conducted by the University of Waterloo and cooperative education students found that having a lifelong learning mindset ensures both objective and subjective careers success through participant studies [13]. This meant that not only were participants successful in their careers post-coop work experience, but they also found job satisfaction and engagement opportunities [13]. This concludes a notion that careers success starts at a young age and that ensuring vast opportunities are available can set the younger generation up for success in not only getting promotions and senior level positions, but also in appreciating the ability to be engaged at work or experiencing overall job satisfaction. Among these findings, it is suggested that implementing MOOCs in these work environments can further encourage work satisfaction and starting these opportunities even within cooperative education in post-secondary studies can ensure that students and professionals alike start learning the topics that meet their diverse needs. These studies at the University of Waterloo also concluded three common themes among a lifelong learning mindset which are: epistemic curiosity, strategic thinking and resilience [13]. Epistemic curiosity

Institutional Effectiveness of Innovative Learning Experiences …

229

describes the innate need that people must relieve the “itch” that is had when something is unknown but wants to be uncovered. It encourages lifelong learning by enhancing knowledge that is of interest and curiosity which aligns with the notion of learning for both pleasure and professional contexts. In a case study by Marta-Lazo et al., a qualitative approach was used to derive the correlation between various participants (from students to seasoned working professionals) to determine the ability of sMOOCs (addition of social) to allow information transfer to a professional context [14]. The results concluded that the satisfaction of participants and their ability to transfer the knowledge (based on initial expectations) was just as high regardless of the level of learning being conducted [14]. This included both high level overviews of sMOOC content, as well as more technical content were just as well received through MOOC delivery methods. Much of the findings also concluded that there was an even higher level of knowledge transfer among those who voluntarily were completing MOOCs [14]. This further supports the idea that MOOCs are a true solution to encouraging lifelong learning, regardless of the stage of one’s life. MOOCs are not only for those who are earlystage professionals but also for those with 20 or more years of experience; it is about how and why the learning is being conducted that influences learning satisfaction and retention. Through research conducted by Morris, their findings on who benefits from self-directed learning, which is a major characteristic of MOOCs, found specific groups of individuals who get the most value [15]. Specifically, the conclusions of Morris’ research found that highly complex career paths of individuals in technical fields benefit greatly from self-directed learning [15]. In other words, MOOCs are an excellent way to benefit the self-directed learning of these individuals as they are completed completely on their own time and with little interaction from others. Other groups of individuals who were found to benefit greatly from self-directed learning included those impacted by work environments that are subject to rapidly changing economic conditions (especially cyclical industries) and overall, those wanting to be set up for long-term career success [15]. In the context of MOOCs for lifelong learning in different contexts, Morris’ research greatly supports the conclusion that these opportunities are vast and can benefit almost all individuals, depending on the motivations and reasons for learning. Lifelong learning with MOOCs can be both an empowerment tool, as well to get ahead in one’s career both short and long term.

4 Opportunities One distinguishment of utilizing MOOCs over a traditional in person, or structured course delivery is that these courses require a heightened amount of self-discipline and motivation [8]. This is due to the self-paced nature of the courses, and with no cost or fee, typically, there is less of an incentive for a learner to see the material through to the end to get the certificate that is or is not going to be provided. This provides an opportunity, and potential gap, for companies and Learning and Development departments as there are now considerations needed to ensure that learners are staying

230

R. Hamilton

motivated. As an employer, or People Leader, you may find yourself in a position of determining learning plans of individuals and trying to understand how to approach it. There are traditionally two types of learners in general, that need to be considered for a project such as implementing MOOCs to encourage lifelong learning. The first learner that you likely would come across in a corporate setting would be “the inquirer.” This learner profile has characteristics such as: a keen interest in excelling their knowledge, enjoys working on new projects, and has the intention of developing a new skill set. The other type of learner that you may have would be “the balanced learner.” This learner profile would be one who enjoys learning new things within the workplace but will not necessarily go out of their way to initiate taking a new course, or designation, for example. They are content with their knowledge and education, but consider opportunities as they arise, based on the business’ needs. By identifying or categorizing employees into one of these learner profiles, based on observations, or even self-declaration, you have an opportunity to tailor programs to their needs. You may also offer additional support for those who would like to actively pursue further courses to supplement their work. According to a study conducted by the Association for Talent Development (ATD), results found that companies offering training programs that are comprehensive and provide more than just on-the-job training have 218% higher income per employee compared with companies that do not offer structured programming [10]. This provides an exceptional opportunity for further advancements for employees to be working in an environment that fosters learning and engagement. These characteristics are at the forefront of what MOOCs are intended for and allow employees to not only reach their own goals, but further the goals of an organization. This all ties into the concept of life-long learning by enrolling in courses that provide context and interest to an employee. Given that MOOCs typically do not cost any money, except in situations where a learner may want a printed certificate, the ROI is exceptional as the primary investment you need to make is allowing employees the time to complete courses. In the context of what MOOCs can provide an organization by utilizing them, would be Corporate Social Responsibility (CSR). As institutions and businesses try to gain credibility in the field of education, providing opportunities and even developing MOOC materials, there is a philanthropic angle to the initiative [12]. Organizations can include these initiatives in their CSR annual reporting to highlight devotion to and mission to increase the availability of high-quality resources [12]. As this relates to MOOCs and lifelong learning, companies can take part in a movement towards fair, equal, and free access to courses to better their employees and communities alike.

Institutional Effectiveness of Innovative Learning Experiences …

231

5 Recommendations Based on the findings within this paper, the following conclusions and recommendations have been drawn to consider when implementing MOOCs within a corporate e-Learning context, or for general business training. • Consider evaluating the types of learners within an organization to position their learning plan correctly (are they “an inquirer” or a “balanced learner”). • Understand the target audience of the learning material, and what the purpose is of the training. Is it for upskilling, or to fill a specific gap in the knowledge base of an employee? • Consider creating tailored learning paths for employees to meet their unique needs, as opposed to a blanket approach to programming. • Select MOOC providers that have content that closely aligns to the objectives of the business. As the authenticity of the content is hard to track and monitor, it is recommended to establish providers that are known within the MOOC community. • In addition to providing MOOC opportunities for knowledge or skills gap, by also providing these learning opportunities for personal interest, the ROI can be exponentially high for overall employee performance. • Due to the lack of proctored evaluations for MOOCs, consider first the quality of the content, and secondly the motivation and shown ability to apply the learnings on the job by the employee. • Stay up to date with new MOOC offerings with thousands being created monthly; providers such as Harvard, or USC provide materials that offer certificates, and credentials that are highly regarded in the eLearning community [2]. • Overall, understand that online learning is not a one-size fits-all approach which will allow for tailoring content to specific learning styles, whether this will be auditory, kinesthetic, or a hybrid model.

Fig. 1 The above table was created using a summary of the recommendations posed to conduct and implement MOOCs for lifelong learning

232

R. Hamilton

6 Conclusion In summary, the availability of MOOCs in a corporate environment is at an all-time high, with many credible sources providing these programs. Lifelong learning with the inclusion of MOOCs has sparked the relationship between how we learn, and the greater retention we have over time when provided with ample opportunities to pursue information that is interesting and engaging. Enrolling in MOOC courses not only provides employees and individuals with new sources of information, but it can overall improve their quality of life and increase the ROI exponentially for employers. MOOCs provide opportunities to engage in content that stimulates an innate need to learn as we please and, overall, allow for curiosity to become a reality. Not only do MOOCs provide opportunities for learners to engage in further learning, but they also provide companies with a way to increase CSR, as well as providing industry knowledge to individuals all around the world.

References 1. van Woezik, T., Koksma, J., Reuzel, R., Jaarsma, D., van der Wilt, G.J.: How to encourage a lifelong learner? The complex relation between learning strategies and assessment in a medical curriculum. Assess. Eval. High. Educ. 45(4), 513–526 (2020). https://doi.org/10.1080/026 02938.2019.1667954 2. Ossiannilsson, E.: MOOCS for Lifelong Learning, Equity, and Liberation. MOOC (Massive Open Online Courses), IntechOpen, June 2022. https://doi.org/10.5772/intechopen.99659 3. Rienties, B., Toetenel, L., Bryan, A.: “Scaling up” learning design: impact of learning design activities on LMS behavior and performance Conference Item (2015). https://doi.org/10.1145/ 2723576.2723600 4. Buhl, M., Andreasen, L.B.: Learning potentials and educational challenges of massive open online courses (MOOCs) in lifelong learning. Int. Rev. Educ. 64, 151–160 (2018). https://doi. org/10.1007/s11159-018-9716-z 5. MOOC: What is a MOOC? https://www.mooc.org/about-moocs. Accessed 05 Mar 2023 6. Palacios-Hidalgo, F.J., et al. : EFL teachers’ perceptions on the potential of MOOCs for lifelong learning. IJWLTT 15(4), 1–17 (2020). https://doi.org/10.4018/IJWLTT.2020100101 7. Wang, Y., Baker, R.: Grit and intention: why do learners complete MOOCs? Int. Rev. Res. Open Dis. Learn. 19(3) (2018). https://doi.org/10.19173/irrodl.v19i3.3393 8. Milligan, C., Littlejohn, A.: Why study on a MOOC? The motives of students and professionals. Int. Rev. Res. Open Dis. Learn. 18(2) (2017). https://doi.org/10.19173/irrodl.v18i2.3033 9. Potvin, M.J.: Encouraging employees to become lifelong learners, 18 November 2022. https:// www.lga.cpa/resources/encouraging-employees-to-become-lifelong-learners/. Accessed 05 Mar 2023 10. Business Training Experts: Profiting From Learning: Do Firms’ Investments in Education and Training Pay Off? (n.d.). https://businesstrainingexperts.com/knowledge-center/training-roi/ profiting-from-learning/. Accessed 22 Apr 2023 11. Koumadoraki, A.: How to promote lifelong learning in the workplace (2023). https://www.lea rnworlds.com/lifelong-learning-in-the-workplace/. Accessed 11 Mar 2023 12. Wakefield, A., Cartney, P., Christie, J., Smyth, R., Cooke, A., Jones, T., King, E., White, H., Kennedy, J.: Do MOOCs encourage corporate social responsibility or are they simply a marketing opportunity? Nurse Educ. Pract. 33, 37–41 (2018). https://doi.org/10.1016/j.nepr. 2018.08.020. ISSN 1471–5953

Institutional Effectiveness of Innovative Learning Experiences …

233

13. Drewery, D.W., Sproule, R., Pretti, T.J.: Lifelong learning mindset and career success: evidence from the field of accounting and finance. High. Educ. Skills Work-Based Learn. 10, 567–580 (2020) 14. Marta-Lazo, C., Frau-Meigs, D., Osuna-Acedo, S.: Collaborative lifelong learning and professional transfer. Case Study: ECO European Project. Interact. Learn. Environ. 27(1), 33–45 (2019). https://doi.org/10.1080/10494820.2018.1451346 15. Morris, T.H.: Self-directed learning: a fundamental competence in a rapidly changing world. Int. Rev. Educ. 65(4), 633–653 (2019)

Project DOCE as a Pedagogical Experience for Innovative Teaching Susana Jorge, Paula Tavares, Manuel Albino, and Marta Madureira

Abstract School of Design (ESD) of the Polytechnic Institute of Cávado and Ave (IPCA) created, in 2017, a project in the field of drawing with the main objective of developing innovative pedagogical practices. The project is called DOCE, or “Drawing, Order, Chaos and Teaching,” and, so far, four editions have taken place. The project began with an invitation to several authors and designers who presented and discussed with the students’ various themes in the field of drawing. This allowed to dynamize the pedagogies developed in the curricular units and provide collaborative and holistic learning with group dynamics and with a successful involvement among all. It can be said that two stages took place: a first working session on methodologies and techniques applied in professional and artistic practice, promoting a wide debate with the students; and a second stage, with the definition of drawing exercises, carried out within the scope of the curricular units, based on the themes discussed in the working sessions, with the invited authors. The COVID-19 pandemic brought changes that influenced the methodologies of working and demanded a re-adaptation with consequences for the dynamic and functionality of the curricular units and some adaptations to this project. For example, last edition (2020) was entirely held online, a way to maintain the project but also as a way to get closer, even in a physical distance, to enhance the students’ connection with their interest in drawing. At the moment, the next editions are currently being analysed, and the intention is to enrich the event with two new approaches: strategies for its dissemination at various levels and for publicizing the pedagogical exercises carried out and their results. Keywords Teaching drawing · New pedagogical practices · Collaborative learning

S. Jorge (B) · P. Tavares · M. Albino · M. Madureira ID+ Research Institute for Design, Media and Culture, Polytechnic Institute of Cávado and Ave, Design School, Barcelos, Portugal e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_19

235

236

S. Jorge et al.

1 Introduction The DOCE—Drawing, Order, Chaos and Teaching—project, from the School of Design (ESD) of the Polytechnic Institute of Cávado and Ave (IPCA) had its first edition in 2017. The first goal was to create a work session bringing together students and several authors and designers linked to the teaching and practice of drawing. It was thus possible to promote debate, research and investigation in this area. It is important to mention that these moments of work proved to be pertinent as a complement to the students’ training, with them being called upon to respond to challenges based on the contributions made by the guests. As acknowledged professionals in their field of activity, the guest authors presented topics from their fields of work and specialisation, addressing the way in which drawing is present as a fundamental mechanism for its exercise. Within the scope of DOCE, two stages were initially set: a first stage of work on the day of the event, for the presentation and debate of various topics about drawing and methodologies and techniques applied in professional and artistic practice; and a second stage, now in the classroom, and within the scope of the curricular units, with the completion of brief exercises and a questionnaire, based on the topics dealt with by the guest authors. The impact achieved with the first edition of the event led us to maintain its continuity with an annual periodicity and to carry out four editions to date. The images of the posters are presented in Fig. 1. In 2020, the COVID-19 pandemic required changes to an online regime. Notwithstanding the interest in this type of event happening in a face-to-face format, the transition to an online format revealed some advantages, among which were the possibility for student workers to participate in the event; the possibility to open the event to a wider and geographically diverse community, outside the IPCA; the possibility to disseminate the ESD courses to the external community; and the possibility to invite authors who were outside national territory and who otherwise would not find it easy to participate.

Fig. 1 From left to right: the images of the DOCE poster are presented: of the first edition, in 2017; of the second edition, in 2018; of the third edition, in 2019; and of the fourth edition, in 2020

Project DOCE as a Pedagogical Experience for Innovative Teaching

237

Thus, although conviviality and face-to-face debate was not possible as in previous editions, a solution was found to maintain continuity of the event, much to the satisfaction of all those involved. The period after the last edition led us to a promising reflection on important changes to be implemented in teaching practices and pedagogical work, based on the DOCE project. Although the starting point was the organisation of a scientific, cultural and artistic event for students, it was important to move on to new stages of this project, framing it within the scope of a project for pedagogical innovation in the teaching of drawing in ESD’s various bachelor courses,1 with this being the reflection that is currently being carried out. The greatest challenge has been to structure this whole organisation, in parallel with ongoing pedagogical practices, and starting from the question: how to innovate in the teaching of drawing? Knowing that the event has presented benefits for the development of the drawing practice with student commitment, it was fundamental, in a first stage, to research some actions and initiatives already applied in higher education for an innovative practice in teaching. Notwithstanding other important initiatives, it is important to mention that this event is unique in ESD-IPCA, regarding the field of drawing.

1.1 About Pedagogical Innovation: How to Teach and How to Learn? In Portugal, there are several initiatives from Portuguese higher education institutions demonstrating the importance of Pedagogical Innovation in Higher Education and the debate promoted around it. As an example of experiences and practices already realised, there is a compilation present in the book of the Ministry of Education and Science (2015) on pedagogical innovation practices in Portuguese higher education [1] and, as a whole, presented in the National Congress on Pedagogical Practices in Higher Education (CNaPPES) [2]. In addition to this example, other situations can also be cited: pedagogical training and professional development for teachers, as is the case of the Interinstitutional Pedagogical Development Conference [3], or the dissemination and awarding of pedagogical experiments, as has been verified in several institutions [4, pp. 18–23]. At IPCA, the recent integration in the Future Advanced Skills Academy (FASA) programme within the scope of RUN-EU—Regional University Network (https:// run-eu.eu/fasa)—is an example of innovation, enabling the debate on “skills for the future, connection to the labour market, active involvement and student autonomy, diversified and practical learning, interdisciplinarity, stimulation of creativity” [5]. 1

“The educational offer of the School of Design (ESD) comprises a wide range of courses in the areas of Design, all of them complying with the prerequisites of the Bologna Process, accredited by the Agency for Assessment and Accreditation of Higher Education—A3ES and registered in the Directorate General for Higher Education. ESD offers courses which bestow the level of Bachelor (1st cycle) and courses which bestow the level of Masters (2nd cycle)”. Retrieved September, 2022 from: https://esd.ipca.pt/en/oferta-formativa/

238

S. Jorge et al.

All of the work developed always seeks to improve the teaching–learning process and several pedagogical actions are framed in this domain [5]. Notwithstanding the amplitude of possible interventions, we are mainly interested in the analysis of approaches and strategies that work the contents. As Carlos Nogueira Fino points out, “pedagogical innovation implies qualitative changes in pedagogical practices and those changes always involve a critical position, explicit or implicit, towards traditional pedagogical practices” [6, p. 1]. Therefore, it is important to analyse how these new pedagogical practices are developed so that they contribute to the improvement of the teaching–learning process, as mentioned by Cristiane Fiorese & Maria Trevisol [7]: “The innovative nature of the educative experiences is in the differential that they promote in the pedagogical thinking and doing. This is what constitutes an innovative pedagogical practice. To innovate is to think differently; it is to propose differentiated pedagogical alternatives that go beyond the commonplace and break the barrier of conventional teaching, thus promoting the transformation of the teaching practice” [7, p. 140].

Thus, and for an objective analysis, it is important to conduct, afterwards, a comparative study of the results obtained in the implementation of these new pedagogical practices, so that it is possible to evaluate their impact [8], since, according to Ana Ramos et al., “this process is more effective if it is translated into an academic process, these changes being based on data, and not assumptions, which is why it is necessary to have ways of collecting information and documentation for sharing new pedagogical practices” [8, p. 118]. The key point lies in how the student can develop certain skills, i.e., making the current, highly diverse universe of students effectively capable of developing them, promoting their participation [8, p. 119]. However, in addition to the intervention at the learning level, that is, to understand and comprehend the subjects in teaching models that are different from the usual [9, p. 101], the importance of those measures is also seen in the importance for the prevention of dropout and for the academic success of students, motivating them and allowing a greater involvement with the school and with learning [9, p. 109]. It is therefore important to underline that there are several improvements and benefits. There is also an increase in the “attractiveness of the higher education system for active professionals” [9, p. 109], who resume their academic activity with a different, more knowledgeable vision of a reality not yet experienced by those who are beginning their studies. Taking this into account, it is fundamental to adopt pedagogical strategies that prove to be “capable of responding to the needs of an increasingly massified, heterogeneous, globalised and technologically evolved student community” [9, p. 100]. Thus, it is fundamental to search for and recognise projects and initiatives already developed in the field of pedagogical intervention in order to analyse the goals set and achieved, as a way of internalising them in new teaching models. The study of practices and results are useful possibilities, relevant in this active interest in innovating in teaching and in its various fields, and our focus is centred on the benefits for the student and on the prospects of other learning models to be developed.

Project DOCE as a Pedagogical Experience for Innovative Teaching

239

1.2 About the Innovation in Drawing Teaching in ESD from the DOCE Project The reflection that is presented around the DOCE project is above all in the analysis and definition of strategies that allow the innovation of pedagogical practices, keeping in mind their importance, and starting from the challenges and the contributions brought by the event. One of the main concerns has been to structure the topics of the sessions with the goal of applying them in parallel with classroom practices. In order to set out all of this work, it is thus important to present the context of our field of action: the teaching of drawing in the bachelor courses at ESD, namely: the Graphic Design course, the Industrial Design course and the Audio-visual Design course, which started in the current academic year of 2022/2023. It is important to point out that the ongoing pedagogical practices allow for a training aimed at a professional practice, with a future action on a creative, innovative, technological and communication level [10]. ESD design courses are seen in a holistic perspective, taking into account the necessary action to be explored in each of these fields, as well as in the conception and development of new products for industrial production [10]. Thus, contact with companies and teaching–learning methodologies based on Design as a Project [10] is promoted. The development of the knowledge and competences to be acquired allow for “intervention in several productive fields, exploring new paradigms of innovation and development, allied with the most recent technological solutions” [10] in a field—Design—that is in constant change. As the author Victor Margolin points out to us, being aware of all the changes that are going on around Design is fundamental for a necessary action in the field of pedagogy capable of establishing an adequate training for current challenges [11, pp. 94–97]. In this regard, the authors Hugo Rocha and Ana Ferreira, when discussing the teaching of design in the education of future designers, point out the importance of considering the issues of “sustainability and social innovation” [12, p. 1]. Although complex topics should be part of the training: “Training future designers means training those responsible for designing a better world, which implies increasing the students’ abilities in critical thinking and developing design proposals that have social value” [12, p. 2]. Specifically in the disciplinary area of drawing, it is also of the utmost relevance to understand its goals at the educational level of the courses, a careful analysis of the changes and adaptations that may be necessary being fundamental, knowing that drawing is fundamental to the areas in question and to the main goals of that training [13]. In this reflection, it is important to consider the approaches that are intended to be explored in the next edition of DOCE. Note the following situations: 1. A pedagogical approach: knowing that it is up to the teacher to organise the contents and to plan out the curricular unit, it is important to constantly analyse these subjects and to prepare new approaches to be adopted in that practice. To this end, with the DOCE project, we intend to accentuate the collaborative dimension [14] and also to create mechanisms for students to share and disseminate their projects.

240

S. Jorge et al.

2. An approach around learning, in other words, about the way in which the student is going to work: that is, knowing that it is fundamental that the student develops a motivated learning, with attention and dedication, it is important to accentuate creative and challenging initiatives that promote critical capacity, and the DOCE project aims to involve the student in teamwork and in the relationship with others and let them be a participant in, and also an author of, the questions that they will have to resolve, given that: “In the short four years between the start of studies and graduation, technology and society change extremely dynamically. That’s why designers need to be trained as thinkers” [15, p. 73]. 3. Finally, it is also important to mention another component to be discussed at the very beginning of the next edition: the students’ previous experience with the drawing practice. This is a fundamental aspect, as each student will certainly have a different pathway. Ilídia Cabral and José Alves point out that another fundamental factor to consider refers to the heterogeneity and cultural diversity of the students, given that: “Pedagogical innovation processes require more flexible ways of working, which can adapt to the heterogeneity of the students, their characteristics and needs and that involve them in the production of knowledge” [16, p. 21]. Having the exposed approaches in mind, the next step consisted of reflecting on the various topics debated in the event and on the projects already explored in the classroom; to then define the structure and an entire planning scheme to put into practice in the next DOCE event, enabling the realisation of an innovative project in the pedagogical and learning component.

2 DOCE as a Pedagogical Promoter The DOCE event provided students with a basis for work that was subsequently developed in the drawing classes. It is important to mention that this curricular unit— Drawing—is included in the first year of the bachelor courses at ESD. It is, without a doubt, a fundamental subject where the exercise of observation and representation is explored, developing several contents, always bearing in mind the importance of this knowledge and these tools for a future performance as designers [17]. It is thus possible to mention the understanding of drawing strategies for representation, the comprehension of drawing functions and images and the drawing language in itself. A fundamental point for the development of these skills is to promote the knowledge of authors and of drawing images, also exploring the importance of drawing practice and manual skills, for the most diverse achievements in the field of Design [17]. To explore the contents, several exercises are developed, usually in sequential stages, with adaptations according to the particular reality of each student, which can be felt above all in the time set for the execution of each step.

Project DOCE as a Pedagogical Experience for Innovative Teaching

241

In an initial phase, the challenge was to consider to what extent the DOCE event, taking into account its characteristics and specificities, would be an important contribution to pedagogical innovation in the teaching of drawing. In the four editions already organised (Figs. 2, 3, 4, and 5), this concern has already taken place, although the intention is to apply new stages in the next edition. However, it is important to mention the possibilities that have already occurred: work sessions in a different space, other than the classroom, with guests, who also posed questions and challenges to the students; the application, already in the classroom, of a questionnaire and brief exercises taking into account the topics presented. This questionnaire was essential for obtaining feedback from the students and for a global perception of the event. An analysis of the answers showed a very positive satisfaction, with students wanting and intending to continue to benefit from the learning provided by DOCE. Above all, the importance of the interaction with professionals in the field and the exposure to work realities that were unknown to them was highlighted. Throughout the four editions held to date, the seventeen guests presented topics that could be encompassed in two areas: (a) a first area about the importance of the drawing practice in the professional field of illustration. In these instances it was discussed how drawing is essential within the scope of a creative project and in the execution of ideas, as was mentioned, for example, by the guest speaker André da

Fig. 2 Images from the first edition of DOCE (2017)

Fig. 3 Images from the second edition of DOCE (2018) and the third edition of DOCE (2019)

242

S. Jorge et al.

Fig. 4 Images from the third edition of DOCE (2019)

Fig. 5 Images from the fourth edition of DOCE (which took place online in 2020)

Loba (https://www.andredaloba.com/), in the first edition, or by the guest speaker Mariana Rio (https://marianario.com/), in the last edition, and (b) a second area about the importance of drawing in the artistic practice, with the emphasis being placed on a more authorial aspect and on a creative freedom that should be known and understood. For students who are still discovering and exploring so many possibilities for work, it is also fundamental that they know the importance of drawing in their artistic career. An example of this is the paper presented by the guest speaker Paulo Almeida (https://pauloluisalmeida.com/), in the first edition, or by the guest speaker Cláudia Amandi (https://claudiaamandi.weebly.com/), in the third edition.

3 The Next Edition of DOCE The structure that follows concerns the intended goals for the next edition of DOCE; starting with the invitation of several authors and designers, the aim is to act in innovative dynamics for teaching–learning. This situation will make it possible to focus on debating certain problems and to establish with the guests the structure for the exercises that will be suggested at the event, and that will be followed up on going forward. The goal is, without a doubt, to accentuate debate with the guests and to allow a greater involvement amongst all. The main goals can be presented as follows:

Project DOCE as a Pedagogical Experience for Innovative Teaching

243

• To bring students closer to real problems in the professional activity, based on the guest’s experience. • To get to know the career path of the guests and the answers given to the challenges faced, analysing to what extent drawing was present in these processes and how it was explored. • To promote cooperation between everyone—guest/student and student/student— through the organisation of work teams. To this end, it is important to group students from various years and from the three ESD bachelor courses. • Develop a drawing practice from this confrontation, with goals and results. • To involve students in the preparation of the exhibition with the obtained results, and also in the oral presentation of the completed work and drawings. • To involve the students in an active and creative participation that leads them to investigate the various dimensions of the project. Thus, in a first moment, it is intended to have a more specific thematic direction and also a more intense cooperation strategy between everyone, especially for the debate phase and the execution of the projects. This aspect is very relevant to create collaborative spaces, where autonomy is achieved by knowing how to share, listen and discuss. In a second moment, there is another benefit present in the exposure and dissemination of the results obtained. The organisation of the next edition is planned for the end of the present school year of 2022/2023, since it is understood to be the best period to conciliate with the students’ school time. In the future, we expect to develop and expand the proposal of the event so that it is integrated into the curricular plan of the courses, as an important training component.

4 Conclusion The DOCE event arose in 2017 from an interest in creating important work dynamics for the teaching of drawing that are capable of continuously updating and innovating in teaching practices, all the while maintaining the students’ motivation. From the outset, it was established that these work sessions would take place in another space outside of the classroom, in order to bring together all the undergraduate students of the IPCA’s ESD, thus forming a single collective, with several guest authors from the training areas of the courses. The works presented by each of the guest authors left remarkable and most pertinent contributions for the students. After a first stage of presenting these topics, the students had the opportunity to meet and debate with the guest authors on important issues related to professional topics. After the event, the feedback collected from the students led to the conclusion that there was a great interest in maintaining and continuing with DOCE, as a useful and fundamental moment in their education was being provided.

244

S. Jorge et al.

Bearing in mind all of these components, it was understood that the next edition of DOCE can go even further in the contribution that it has made. Several goals were established, taking into account the advantages and benefits of pedagogical innovation in higher education. Knowing that DOCE focuses all its attention on drawing and on the fact that it fosters and promotes important experiences for pedagogical innovation in this area, on the one hand, the pedagogical side should be accentuated, where it was understood to be pertinent to highlight the collaborative dimension. To do so, it is important to emphasise the organisation of the students in work groups, bearing in mind the various years and areas of the courses. On the other hand, it is important to highlight the component of how the students will learn, with learning mechanisms that help them to work motivated, to share and disseminate their conclusions and to be the authors of the challenging questions they will have to solve. It is also important to mention that the impact provided to students by these new strategies is fundamental: namely in terms of the learning and skills that the student will develop and that will contribute to achieving important results in their process. This kind of learning, outside the classroom and in direct contact with practice, promotes the acquisition of skills in a more spontaneous but also incisive way. Students are very receptive to external guests, which allows for fast and effective learning. Lastly, it is important to underline the importance of pedagogical initiatives and proposals focused on innovation and the teaching–learning process which allowed the analysis of possibilities to be explored. All of the reflection presented took into account, on the one hand, the results already achieved with the editions developed so far and, on the other hand, the desired goals for the future, with the continuation of the event being fundamental for the achievement of these new stages. Acknowledgements This work is financed by national funds through the FCT—Fundação para a Ciência e a Tecnologia, I.P., under the scope of the project UIDB 04057/2020.

References 1. Ministério da Educação e Ciência: Experiências de Inovação Didática no Ensino Superior, Gabinete do Secretário de Estado do Ensino Superior (2015). https://www.researchgate.net/ publication/283010065_Experiencias_de_inovacao_didatica_no_ensino_superior. ISBN 978972-729-087-1. Accessed May 2018 2. National Congress on Pedagogical Practices in Higher Education [Congresso Nacional de Práticas Pedagógicas no Ensino Superior – CNaPPES]. https://cnappes.org/. Accessed May 2022 3. Interinstitutional Pedagogical Development Conference [Jornadas Interinstitucionais de Desenvolvimento Pedagógico]. https://cms.ua.pt/jornadasidp/. Accessed Apr 2022 4. Reis, T., Santos, R.: Está em marcha uma revolução pedagógica. In: UPortoAlumni – Revista dos Antigos Estudantes da Universidade do Porto, n.º 22, II Série, Agosto de 2015, pp. 18–23 (2015). https://issuu.com/uporto/docs/alumni_22/18. Accessed July 2018 5. Polytechnic Institute of Cávado and Ave [Instituto Politécnico do Cávado e do Ave], IPCA website. https://ipca.pt/noticia/razoes-para-inovar-o-ensino-a-nossa-mostra-ped agogica-reuniu-varias/. Accessed July 2022

Project DOCE as a Pedagogical Experience for Innovative Teaching

245

6. Fino, C.N.: Inovação pedagógica: significado e campo (de investigação). In: Mendonça, A., Bento, A.V. (Org.) Educação em tempo de mudança, pp. 277–287. Grafimadeira, Funchal (2008). https://hdl.handle.net/10400.13/808. Accessed May 2020 7. Fiorese, C., Trevisol, M.: Práticas pedagógicas inovadoras no ensino superior: relatos e análise de experiências em cursos de formação de professores. Teoria e Prática da Educação 24(2), 122–141 (2021). https://doi.org/10.4025/tpe.v24i2.57827 8. Ramos, A., et al.: Implementação de novas práticas pedagógicas no ensino superior. Revista Portuguesa de Educação 26(1), 115–141 (2013). https://doi.org/10.21814/rpe.2986. ISSN 0871-9187 9. Estudo: Inovação Pedagógica: ventos de mudança no Ensino Superior. Federação Académica do Porto. Centro de Estudos (2021). Accessed May 2022. https://www.fap.pt/sites/default/files/ u29/fap_estudo_inovacao_pedagogica.pdf 10. School of Design (ESD) of the Polytechnic Institute of Cávado and Ave (IPCA) [Escola Superior de Design do Instituto Politécnico do Cávado e do Ave], ESD-IPCA Website. https://esd.ipca. pt/oferta-educativa/. Accessed Sept 2022 11. Margolin, V.: Design e Risco de Mudança. In: Verso da História, ESAD (2014) 12. Rocha, H., Ferreira, A.: Future designers as actors of change: exploring design education scenarios focused on social innovation and sustainability. In: Convergências - Revista de Investigação e Ensino das Artes XII(24) (2019). http://convergencias.esart.ipcb.pt/?p=article& id=364. Accessed June 2022 13. Zimmermann, A., Coutinho, S.: Teaching drawing in Graphic Design: an analysis of scenarios and a proposal for Brazil based on the design process and an interdisciplinary approach. [O ensino do desenho na formação em Design Gráfico: uma análise de cenários e uma proposta projetual e interdisciplinar para o Brasil]. DAT J. 5(2), 155–173 (2020). https://doi.org/10. 29147/dat.v5i2.200 14. Krucken, L., Mol, I.: Abordagens para cocriação no ensino do design: reflexões sobre iniciativas no contexto da graduação e pós-graduação. Blucher Des. Proc. 1(4), 992–1000 (2014). https:// doi.org/10.5151/designpro-ped-01062. São Paulo: Blucher. ISSN 2318-6968 15. Boninger, C., Schmidhuber, S., Frenkler, F., Gaines, J., Gaines, P.: Designing Design Education: Whitebook. IF Design Foundation, Hanover (2021) 16. Cabral, I., Alves, J.M.: Para um modelo Integrado de Inovação pedagógica e melhoria das aprendizagens. In: Cabral, I., Alves, J.M. (Orgs.) Inovação Pedagógica e Mudança Educativa Da teoria à(s) prática(s) Porto, Faculdade de Educação e Psicologia da Universidade Católica Portuguesa (2018). https://doi.org/10.34632/9789899948693 17. Polytechnic Institute of Cávado and Ave [Desenho I, Desenho II, Catálogo de Unidades Curriculares - Instituto Politécnico do Cávado e do Ave], IPCA website. https://ipca.pt/ensino/cat alogo-unidades-curriculares/. Accessed July 2022

A Comparison of Online and In-Person MBI Classes on Self-Compassion and Creativity Young Min Jung and Eunmi Kim

Abstract Research on mindfulness-based interventions (MBIs) has increasingly highlighted their influence on creativity and self-compassion. Diverse MBIs have been applied to different populations, including STEM-focused college students. During the COVID-19 pandemic, online MBIs were introduced as substitutes for in-person format MBIs, yet few studies have evaluated the comparative effects of both formats. This study examines the effects of both online (n = 19) and in-person (n = 27) MBI academic classes at a STEM college in South Korea. The Kaufmann Domains of Creativity Scale (K-DOCS) and the Self-Compassion Scale (SCS) were employed to measure participants’ creativity and self-compassion. With a one-year gap between the two formats, pre- and post-K-DOCS and SCS data were collected and statistically analyzed. Our main findings are as follows: a) Contrary to our expectations, the pre/post t-tests did not show a significant change in either format of MBIs. b) Silver’s test revealed that Pearson’s correlation between self/everyday, scholarly, and other subscales significantly increased after MBIs. This trend was consistent in both online and in-person MBI classes. c) The impact of gender on the self/everyday subscale varied between online and in-person MBIs. In both formats, the relationship between gender and the self/everyday subscale diminished after the MBIs. These findings suggest that students’ distinctive interpersonal and intrapersonal creativity significantly influences other facets of creativity and self-compassion through MBI, therefore instructional strategies for student groupings based on it appear essential. Additionally, MBI may serve as a tool to mitigate gender-based creativity disparities in STEM colleges. Future research warrant comparison with a control group. Keywords Mindfulness-based intervention · Interpersonal creativity · Education platform

Y. M. Jung · E. Kim (B) Korea Advanced Institute of Science and Technology, 291 Daehak-ro, Yuseong-gu, Daejeon 34141, Republic of Korea e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_20

247

248

Y. M. Jung and E. Kim

1 Introduction Mindfulness-Based Interventions (MBIs) have gained considerable attention in recent years due to their potential to improve various aspects of mental health and well-being. In particular, they have been shown to enhance emotional regulation, cognitive flexibility, and stress resilience among students in higher education [1, 2]. Among these aspects, self-compassion is important and highly related to others, which involves relating to oneself, with care and support during times of suffering [3, 4]. This ability could be a healthy way to maintain positive self-regard [2], and mindfulness and MBIs are well-known ways to cultivate self-compassion [5, 6]. As much as self-compassion is considered significant, creativity is also one of the focal points of educational interest alongside mental health and can be nurtured through MBIs. Emerging research supports the idea that practicing mindfulness aids creativity by allowing purposeful mind-wandering and reducing judgment or fear [8], therefore promoting divergent thinking [9]. In relation to this, various educational methods are being developed, such as body-mind training and emotion-related MBI, which yield significant improvements in verbal and visual creativity among university students [10]. Moreover, the relationship between self-compassion and creativity is also being studied, as self-compassion may promote a safe psychological space in which creativity can flourish [7]. Based on these studies, when developing a new form of MBI, it is necessary to check whether it is possible to cultivate both in the same way as the traditional one. Today, online MBIs are gaining traction owing to their ease of access, without being constrained by time and space [11]. In 2014, a survey reported that many people preferred mindfulness meditation interventions in the online format over group formats [12]. The COVID-19 pandemic, particularly, has accelerated the demand, since quarantine policies made in-person classes difficult [13]. Against this background, various studies related to online MBIs are being conducted, and similar benefits, including mental health, have been validated using in-person MBIs [11, 13–15]. However, there are relatively few studies on creativity. Also, mindfulness practice requires students to actively practice the given instructions and communicate their experiences to teachers. It is essential to establish trusting relationships between teachers and students [16]. Consequently, the advantage of not having to communicate directly and in real-time with an online expert can be a disadvantage in MBIs, rendering them less effective in areas such as knowledge acquisition compared to in-person classes [17]. Considering these points, further research is needed to determine whether replacing an in-person MBI curriculum with an online one would yield similar results. In particular, it is necessary to investigate the suitability and direction of improvement, especially among college students who attended most of their classes online after the COVID-19 outbreak. Moreover, it is considered a major challenge to address, as there is a lack of research on whether online MBIs can have the same effect on creativity as in-person MBIs, which is as important as mental health in the science, technology, engineering, and math (STEM) field.

A Comparison of Online and In-Person MBI Classes …

249

Given the increasing demand for creativity and adaptability in today’s rapidly changing world, it is crucial to explore how MBIs may impact these critical skills, especially among STEM students. Therefore, this study seeks to contribute to this emerging area of inquiry by comparing the impact of online and in-person MBIs on creativity and self-compassion among a sample of students in STEM focused college, Korea Advanced Institute of Science and Technology (KAIST).

2 Method 2.1 Participants The participants were recruited from STEM majors attending the MBI course the KAIST offered. Nineteen students were recruited from a course held online in the 2021 Fall, and 27 were recruited from one held in-person in the 2022 Fall. The groups consisted of Korean and international students. Responses from 6 students from the online class and 19 students from the in-person class were excluded during preprocessing because of incomplete or unreliable responses. See Table 1 for details of the participants.

2.2 Intervention The MBI course consisted of 10 ordinary classes of 2.5 h; a total of 25 h of practice, excluding orientation, and a one-day retreat class. Compared to the usual MBSR program of 28 h with a one-day retreat class, the intervention has shorter practice hours and a longer total duration. Each 2.5-h class had a unique theme, such as “coping with stress” and “mindful eating.” These themes were drawn from the original MBSR program. Each session started with a brief lecture and shared thoughts about the theme. Students were instructed to follow several mindfulness practices. Some practices were closely related to the theme, such as mindful eating, walking, listening, and speaking, with occasional practices such as breathing, meditation, and yoga. After 15–30 min of practice, the students shared their impressions of the experience. Each sharing session was randomly conducted in groups of three to five students per class. In the online course, Zoom’s Breakout Room feature was used for sharing, while in the in-person course, the students in each group and the teaching assistant participated in discussions. Each class had a 15- to 30-min break, during which students were encouraged to prepare for the next practice. Students were required to attend at least eight of the 10 classes to pass the course.

250

Y. M. Jung and E. Kim

Table 1 Demographic Characteristics. 3 Students in the in-person class were double majors: Computer Science & Electrical Engineering (2), Computer Science & Mathematics (1). Online MBI

In-person MBI

n

%

n

%

12

63.2

18

66.7

7

36.8

9

33.3

16

84.2

17

63.0 18.5

Sex Male Female Major School of Freshmen Computer Science

1

5.3

5

Physics

0

0.0

0

0.0

Mathematics

1

5.3

1

3.7 11.1

Electrical Engineering

0

0.0

3

Chemical and Biological Engineering

1

5.3

2

7.4

Biological Science

0

0.0

1

3.7

Aerospace

0

0.0

1

3.7

2022

0

0.0

18

66.7 14.8

Year of entry 2021

16

84.2

4

2020

1

5.3

0

0.0

2019

0

0.0

3

11.1

2018

2

10.5

2

7.4

Age 16

1

5.3

2

7.4

17

1

5.3

8

29.6

18

7

36.8

6

22.2 18.5

19

5

26.3

5

20

1

5.3

2

7.4

21

3

15.8

2

7.4

22

1

5.3

2

7.4

23

0

0.0

1

3.7

24

0

0.0

1

3.7

Total

19

27

2.3 Measures Participants completed an online survey before and after each semester, using Google Forms. The survey consisted of two 5-point Likert scale questionnaires: the Kaufmann Domains of Creativity Scale (K-DOCS) and the Self-Compassion Scale (SCS).

A Comparison of Online and In-Person MBI Classes …

251

Table 2 K-DOCS subscales and each description Subscale

Description

Self/Everyday

Self-perception of intrapersonal and interpersonal creative behaviors

Scholarly

Related to gaining knowledge

Performance

Self-perception of creativity in music, writing, and acting performances

Mechanical/Scientific Self-perception of creativity in science, engineering, and math Artistic

Self-perception of creativity in artistic activities

The questions were randomized using the internal function of Google Forms, and distributed to each student. Kaufmann Domains of Creativity Scale (K-DOCS). Creativity was measured using 50 items of the K-DOCS used in Kaufmann’s study [18]. K-DOCS comprises five domains: self/everyday, scholarly, performance, mechanical/scientific, and artistic (see Table 2) [18]. Responses were given on a 5-point Likert scale (1 = much less creative to 5 = much more creative) compared with respondents of a similar age and life experience [18]. Those subscales have been shown to be internally consistent with Cronbach’s alpha, ranging from 0.86 to 0.90 [19]. Self-Compassion Scale (SCS). Self-compassion was assessed using 26 items of the SCS. It comprises six factors: self-kindness, self-judgment, common humanity, reduced isolation, mindfulness, and reduced over-identification (see Table 3). Responses were self-reported on a 5-point Likert scale, with 1 being almost never and 5 being almost always. All items in the self-judgment, isolation, and overidentification subscales were negatively valenced. Items representing uncompassionate self-responses were reverse-coded before calculating the total score, to indicate the relative absence of a self-compassionate mindset. Means were calculated for each subscale, and a grand mean was calculated for the total self-compassion score [20]. The subscales have been shown to be internally consistent, with Cronbach’s alpha ranging from 0.75 to 0.81 [21].

2.4 Analysis The Shapiro–Wilk test was used to check normality, and the F-test was used to compare the variance of pre- and post-data of online and in-person interventions, respectively. Paired t-tests were used to test the hypothesis that there were no mean differences among the data. The correlation matrices of Pearson’s correlation coefficients, both before and after each intervention, were also calculated to identify the effectiveness of the intervention on the relationships between factors. Fisher’s z-transformation method with the core library of R was used to test the significance of the differences between correlations [22].

252

Y. M. Jung and E. Kim

Table 3 SCS subscales and descriptions Subscale

Description

Self-Kindness

Reflects an individual’s tendency to treat themselves with compassion and understanding when experiencing emotional pain or distress

Self-Judgment

The extent to which individuals are critical and judgmental of their own flaws and shortcomings

Common Humanity

The extent to which individuals recognize that suffering and challenges are a normal part of the human experience

Isolation

The tendency to feel more connected and less alone when facing challenges

Mindfulness

An individual’s ability to be present and aware of their thoughts and feelings, without getting caught up in them or reacting impulsively

Over-identification

The extent to which individuals get carried away by their emotions, particularly negative ones

SCS-Tot

The grand mean of SCS

All statistical analyses and visualizations were performed using R version 4.2.2 and Python 3.11.0.

3 Results 3.1 Descriptive Statistics Table 4 displays the descriptive statistics for the SCS and K-DOCS subscales, including the means and standard deviations before (pre) and after (post) the online classes in 2021. The Self-Judgement, isolation, and over-identification subscales were reverse-coded owing to their negative valence and are denoted as (reversed) in the table. It also presents the results of the Shapiro–Wilk test, indicating that the data for each subscale met the assumption of normality. An F-test was conducted to assess the equality of variances between the groups before and after the class, and no significant differences were found. Additionally, the results of the paired t-test showed no significant differences in any of the subscales before and after class. However, the values of the effect sizes (Cohen’s d) for the Artistic and Over-identified subscales were significant. The p-value of the t-test indicated that there were some differences between the groups, but this was not statistically significant; that is, the study was underpowered. Table 5 presents the descriptive statistics before and after the in-person classes in 2022. The result shows that no significant differences were observed in any of the subscales before or after the class.

A Comparison of Online and In-Person MBI Classes …

253

Table 4 Statistical values of before (pre) and after (post) online MBI class: Mean, standard deviation, p-values of Shapiro Wilk, F-test, and paired t-test. Values of Cohen’s d are also described (N = 19). Self-Judgement, Isolation, and Over-identified subscales of SCS are reverse-coded and denoted as (reversed). Scale K-DOCS

Subscale Self/Everyday

Scholarly

Performance

SCS

pre

post

mean

3.56

3.60

std.

0.61

0.66

Shapiro–Wilk

0.398

0.093

mean

3.52

3.54

std.

0.60

0.64

Shapiro–Wilk

0.279

0.123

mean

2.61

2.77

std.

0.73

0.85

Shapiro–Wilk

0.303

1.000

Mechanical/ Scientific

mean

3.11

3.02

std.

0.83

0.60

Shapiro–Wilk

0.305

0.531

Artistic

mean

2.87

3.30

std.

0.90

0.84

Self-Kindness

Self-Judgement (reversed)

Shapiro–Wilk

0.815

0.905

mean

3.49

3.57

std.

0.80

0.71

Shapiro–Wilk

0.363

0.416

mean

3.16

2.94

std.

0.79

0.90

Shapiro–Wilk Common Humanity mean

Over-identification (reversed)

SCS-Tot

0.576 3.72

std.

0.77

0.84

Shapiro–Wilk

0.486

0.618

3.16

3.04

Isolation (reversed) mean

Mindfulness

0.298 3.51

std.

0.85

0.94

Shapiro–Wilk

0.646

0.796

mean

3.55

3.58

std.

0.60

0.62

Shapiro–Wilk

0.761

0.330

mean

3.29

2.72

std.

0.67

0.68

Shapiro–Wilk

0.320

0.546

mean

3.36

3.26

p-value

Cohen’s d

F-test

t-test

0.781

0.838

0.060

0.755

0.917

0.031

0.541

0.561

0.199

0.173

0.714

0.130

0.785

0.163

0.505*

0.646

0.747

0.098

0.584

0.468

0.260

0.732

0.472

0.260

0.691

0.685

0.132

0.923

0.893

0.043

0.958

0.019

0.842**

0.722

0.615

0.162 (continued)

254

Y. M. Jung and E. Kim

Table 4 (continued) Scale

Subscale

pre

post

p-value F-test

std.

0.58

0.64

Shapiro–Wilk

0.835

0.904

Cohen’s d t-test

Effect sizes of Cohen’s d are reported; *d > 0.5, **d > 0.8

3.2 Pearson’s Correlation Among Subscales Significance levels and Pearson’s correlation coefficients were calculated and visualized by using the corrplot library in R (Figs. 1 and 2). In all the groups, statistically significant correlations were found between some subscales of the same scale, such as over-identified (reverse) vs. isolated (reverse). This suggests that the subscales of each scale share a common underlying construct.

3.3 Effects of Self/Everyday and Scholarly Subscales After the in-person classes, a noticeable increase was observed in the correlations between the self/everyday subscales and other subscales, whereas this increase was either small or nonexistent in online classes. The subsequent subscales of the group after the in-person class had sufficiently high correlation coefficients. Nonetheless, the p-values did not reach statistical significance, indicating the need for larger sample sizes. Silver’s test was conducted to see how the correlation changes were statistically significant [23], and showed that they were not (see Table 6). In addition, both in-person and online classes exhibited an increase in the correlations between the scholarly subscale and other subscales (see Table 6). This finding suggests that both types of classes had a similar impact on the scholarly subscales’ relationships with other subscales.

3.4 Gender, Creativity, and MBI Platform In both the online and in-person classes, the correlations between gender and the subscales approached zero after class (see Table 7). This result implies that the relationship between gender and the subscales became less prominent with the classes. Furthermore, both online and in-person classes demonstrated a reversal in the correlations between gender and self/everyday subscale scores. The p-values for these changes were statistically significant (see Table 7). This finding suggests that the relationship between gender and the self/everyday subscale was notably altered

A Comparison of Online and In-Person MBI Classes …

255

Table 5 Statistical values of before (pre) and after (post) in-person MBI class: Mean, standard deviation, p-values of Shapiro Wilk, F-test, and paired t-test. Values of Cohen’s d are also described (N = 27). Self-Judgement, Isolation, and Over-identified subscales of SCS are reverse-coded and denoted as (reversed). Scale K-DOCS

Subscale Self/Everyday

Scholarly

Performance

SCS

pre

post

mean

3.40

3.35

std.

0.69

0.58

Shapiro–Wilk

0.341

0.959

mean

3.34

3.36

std.

0.60

0.58

Shapiro–Wilk

0.200

0.126

mean

2.65

2.74

std.

0.89

0.80

Shapiro–Wilk

0.237

0.869

Mechanical/ Scientific

mean

3.11

3.13

std.

0.63

0.65

Shapiro–Wilk

0.336

0.529

Artistic

mean

2.99

3.16

std.

0.79

0.73

Self-Kindness

Self-Judgement (reversed)

Shapiro–Wilk

0.076

0.226

mean

2.79

2.96

std.

0.81

0.85

Shapiro–Wilk

0.508

0.215

mean

2.60

2.47

std.

0.77

0.92

Shapiro–Wilk Common Humanity mean

Over-identification (reversed)

SCS-Tot

0.265 3.35

std.

0.91

0.92

Shapiro–Wilk

0.063

0.202

3.05

2.90

Isolation (reversed) mean

Mindfulness

0.485 3.21

std.

1.01

0.90

Shapiro–Wilk

0.140

0.554

mean

3.25

3.40

std.

0.84

0.93

Shapiro–Wilk

0.149

0.719

mean

2.81

2.81

std.

0.82

1.10

Shapiro–Wilk

0.749

0.355

mean

2.95

2.98

p-value

Cohen’s d

F-test

t-test

0.781

0.838

0.079

0.755

0.917

0.045

0.541

0.561

0.113

0.173

0.714

0.038

0.785

0.163

0.227

0.646

0.747

0.213

0.584

0.468

0.149

0.732

0.472

0.152

0.691

0.685

0.155

0.923

0.893

0.166

0.958

0.019

0.010

0.722

0.615

0.048 (continued)

256

Y. M. Jung and E. Kim

Table 5 (continued) Scale

Subscale

pre

post

p-value F-test

Fig. 1 Pearson’s correlation matrix of before (left) and after (right) online class. Blue square: positive correlation; red square: negative correlation. Significance levels are reported; *p < 0.05, **p < 0.01, ***p < 0.001.

std.

0.68

0.72

Shapiro–Wilk

0.368

0.988

Cohen’s d t-test

A Comparison of Online and In-Person MBI Classes …

257

Fig. 2 Pearson’s correlation matrix for the in-person MBI class. Blue square: positive correlation; red square: negative correlation. Significance levels are reported; *p < 0.05, **p < 0.01, ***p < 0.001

by the class. Hittner’s test was conducted to determine the statistical significance of the correlation changes [24], and the results are presented in Table 7. These findings are also illustrated in Fig. 3, which reveals changes in the self/ everyday and artistic subscales, particularly in the online learning environment. These changes, when considered alongside the varying effects observed in other subscales depending on the self/everyday dimension and the distinct patterns of change observed in female participants, suggest that different educational platforms may be more effective for each gender. Alternatively, this may imply the need to design and implement learning experiences within one platform, that considers the unique characteristics and needs of both genders. This finding highlights the

258

Y. M. Jung and E. Kim

Table 6 Changes in Subscale Correlations Before and After In-person and Online Classes. Silver’s test (Silver’s p) is conducted to see how the correlation changes are statistically significant. Online MBI Correlation Self/ Everyday

Scholarly

In-person MBI Silver’s p

Correlation pre

Silver’s p

pre

post

Scholarly

0.47*

0.46*

0.968

0.40*

0.50**

post 0.657

Performance

0.50*

0.43

0.814

0.48

0.37

0.602

Mechanical/ Scientific

0.45

0.00

0.189

0.27

0.52**

0.218

Artistic

0.20

0.12

0.811

0.40*

0.48*

0.697

Self-Kindness

0.39

0.52*

0.639

0.32

0.51**

0.413

Self-Judgement (reversed)

0.44

0.41

0.897

0.18

0.49**

0.250

Common Humanity 0.23

0.24

0.959

0.24

0.54**

0.209

Isolation (reversed) 0.04

0.12

0.827

0.25

0.27

0.921

Mindfulness

0.55*

0.38*

0.548

0.32*

0.66***

0.093

Over-identification (reversed)

0.32

0.33

0.968

0.28

0.27

0.973

SCS-Tot

0.41

0.40

0.983

0.34

0.59

0.270

Self/Everyday

0.47*

0.46*

0.968

0.40*

0.50*

0.657

Performance

0.40

0.53*

0.632

0.43*

0.18

0.253

Mechanical/ Scientific

0.35

0.40

0.854

0.18

0.2

0.936

Artistic

0.23

0.47*

0.443

0.3

0.54**

0.230

Self-Kindness

0.37

0.36

0.974

−0.41*

−0.03

0.152

Self-Judgement (reversed)

0.22

0.19

0.944

−0.41*

0.04

0.129

Common Humanity 0.25

0.35

0.775

−0.43*

0.00

0.095

Isolation (reversed) 0.01

0.28

0.429

−0.13

0.09

0.431

Mindfulness

0.40

0.44

0.873

−0.43*

0.06

0.049*

Over-identification (reversed)

0.16

0.11

0.886

−0.25

−0.05

0.485

SCS-Tot

0.29

0.35

0.849

−0.43*

0.02

0.103

P Pearson’s correlation and Silver’s test are reported; *p < 0.05.

importance of tailoring educational approaches to accommodate the diverse needs of students and optimize their learning outcomes.

A Comparison of Online and In-Person MBI Classes …

259

Table 7 Changes in Gender-Subscale Correlations Before and After In-person and Online Classes. Hittner’s test (Hittner’s p) is conducted to see how the correlation changes are statistically significant Online MBI Correlation pre Gender

Self/Everyday

−0.49*

Scholarly

−0.02

Performance

−0.21

Mechanical/ Scientific

In-person MBI Hittner’s p Correlation

post

pre

Hittner’s p

post

0.004**

0.16

−0.17

0.163

−0.09

0.822

0.10

−0.16

0.207

−0.13

0.947

0.06

−0.30

0.109

−0.16

−0.19

0.391

−0.49**

−0.42*

0.586

0.34

0.03

0.503

0.58**

0.26

0.021*

Self-Kindness

−0.15

0.07

0.306

0.13

−0.03

0.549

Self-Judgement (reversed)

−0.33

0.06

0.979

−0.23

0.20

0.174

Common Humanity

−0.09

−0.07

0.692

0.25

0.18

0.785

Isolation (reversed) −0.34

−0.21

0.730

−0.21

0.06

0.343

Mindfulness

−0.35

−0.24

0.043*

−0.19

0.06

0.275

Over-identification (reversed)

−0.55*

0.11

0.342

−0.32

−0.13

0.511

SCS-Tot

−0.38

−0.06

0.828

−0.12

0.07

0.527

Artistic

0.37

Pearson’s correlation and Silver’s test are reported; *p < 0.05, **p < 0.01, ***p < 0.001

Fig. 3 Comparison of pre- and post-course K-DOCS subscales changes for each gender in online (top) and offline (bottom) learning environments. Males are represented in red (or light gray), while females are represented in blue (or dark gray).

260

Y. M. Jung and E. Kim

4 Discussion In this study, we aimed to investigate the effects of MBI courses on creativity and selfcompassion in students at a STEM-focused college, and to explore any differences in these effects between online and in-person platforms. The main result observed was the polarization of other subscale scores depending on self/everyday scores, indicating the importance of interpersonal creativity. However, no significant improvement was observed in the average subscale scores of the overall group. This might be because most participants were freshmen in their first semester, and considering the nature of STEM-focused colleges, it is possible that the overall population’s scores may have decreased due to exams or other factors. Therefore, to verify these findings, future research should collect data on the tendencies of a control group that did not take the course. Moreover, this finding is in line with previous research that demonstrated a relationship between self-compassion and creativity [7, 25]. By identifying the correlations between these aspects, our study adds to the growing body of evidence supporting the interplay between self-compassion and creativity. The importance of the self/everyday subscale in the MBIs, which is related to interpersonal and intrapersonal creativity [18], suggests that students with higher abilities in these areas may benefit from it in terms of enhancing creativity and self-compassion. However, the gap between males and females did not narrow for the mechanical/scientific subscale, emphasizing mechanical ability and interest in science and math [18], which is one of the most important subscales in STEM education. This suggests that MBI alone may not fundamentally increase interest in science and math, which is a reasonable conclusion considering the purpose and direction of the MBI curriculum. Given the importance of emotional intelligence for the self/everyday subscale [18], there is ample reason to develop ways to enhance emotional intelligence in MBI-related research and curriculum development, so that even students with lower emotional intelligence can benefit from MBIs. The results of the scholarly subscale, which is related to creative analysis, debate, and scholarly pursuits [18], indicate that in-person courses may help students who have neglected self-care due to academic focus return to their ‘original’ state. In contrast, online courses did not show this correlation before the course, which, along with previous discussions, adds weight to the argument that there is a need to examine the differences between STEM-focused college students and current students in greater depth, particularly during the COVID-19 pandemic. In relation to gender and MBI, there was a significant increase in the distribution of female scores on the self/everyday subscale in online courses. The association of this subscale with interpersonal creativity suggests that MBI courses can be helpful for students in STEM-focused colleges with a low proportion of females. However, the in-person courses showed the opposite effect, which was not statistically significant. Therefore, it was difficult to draw conclusions about the impact of the platform itself. The limitations of this study include small sample size and the fact that the two courses were not conducted simultaneously, which may have potential implications that are not statistically supported. Considering these limitations, follow-up studies

A Comparison of Online and In-Person MBI Classes …

261

on a larger scale may reveal additional potential findings from various perspectives. Moreover, for future gender-related research, it is necessary to balance the gender ratio among participants, and diverse statistical methods such as principal component analysis can be employed to understand how students with different characteristics are influenced by MBIs. Acknowledgements This study was funded by the KAIST G-CORE Project of 2021 (N11220008) and approved by the KAIST Institutional Review Board (KH2022-069). Data presented in this study are available upon request from the corresponding author. These data are not publicly available because of privacy restrictions. The authors thank the Post-AI and G-CORE project members at KAIST CCS. The authors declare no conflicts of interest. The funders had no role in the study design; collection, analyses, or interpretation of data; writing of the manuscript; or decision to publish the results. We would like to thank Editage (www.editage.co.kr) for English language editing.

References 1. Broderick, P.C.: Learning to Breathe: A Mindfulness Curriculum for Adolescents to Cultivate Emotion Regulation, Attention, and Performance, 2nd edn. New Harbinger Publications, Oakland (2013) 2. Neff, K.D., Dahm, K.A.: Self-compassion: What it is, what it does, and how it relates to mindfulness. In: Handbook of Mindfulness and Self-regulation, pp. 121–137 (2015) 3. Neff, K.D., Knox, M.C.: Self-Compassion. In: Zeigler-Hill, V., Shackelford, T.K. (eds.) Encyclopedia of Personality and Individual Differences. Springer, Cham (2020) 4. MacBeth, A., Gumley, A.: Exploring compassion: A meta-analysis of the association between self-compassion and psychopathology. Clin. Psychol. Rev. 32(6), 545–552 (2012) 5. Germer, C.: The Mindful Path to Self-Compassion: Freeing Yourself from Destructive Thoughts and Emotions. Guilford Press, New York (2009) 6. Boellinghaus, I., Jones, F.W., Hutton, J.: The role of mindfulness and loving-kindness meditation in cultivating self-compassion and other-focused concern in health care professionals. Mindfulness 5, 129–138 (2014) 7. Zabelina, D.L., Robinson, M.D.: Don’t be so hard on yourself: Self-compassion facilitates creative originality among self-judgmental individuals. Creat. Res. J. 22(3), 288–293 (2010) 8. Henriksen, D., Richardson, C., Shack, K.: Mindfulness and creativity: Implications for thinking and learning. Think Skills Creat. 37, 100689 (2020) 9. Hommel, B., Öztürk, A.B.: Meditate to create: The impact of focused-attention and openmonitoring training on convergent and divergent thinking. Front. Psychol. 3 (2012) 10. Ding, X., Tang, YY., Tang, R., Posner, M.: Improving creativity performance by short-term meditation. Behav. Brain Funct. 10, 9 (2014) 11. Andersson, G., Titov, N.: Advantages and limitations of Internet-based interventions for common mental disorders. World Psychiatry 13, 4–11 (2014) 12. Wahbeh, H., Svalina, M.N., Oken, B.S.: Group, one-on-one, or internet? Preferences for mindfulness meditation delivery format and their predictors. Open Med. J. 1, 66–74 (2014) 13. González-García, M., Álvarez, J.C., Pérez, E.Z., Fernandez-Carriba, S., López, J.G.: Feasibility of a brief online mindfulness and compassion-based intervention to promote mental health among university students during the COVID-19 pandemic. Mindfulness 12, 1685–1695 (2021) 14. Spijkerman, M.P.J., Pots, W.T.M., Bohlmeijer, E.T.: Effectiveness of online mindfulness-based interventions in improving mental health: A review and meta-analysis of randomised controlled trials. Clin. Psychol. Rev. 45, 102–114 (2016)

262

Y. M. Jung and E. Kim

15. Nadler, R., Carswell, J.J., Minda, J.P.: Online mindfulness training increases well-being, trait emotional intelligence, and workplace competency ratings: A randomized waitlist-controlled trial. Front. Psychol. 11 (2020) 16. Valerie, J.R., Schroeder, P.J.: In-person and virtual world mindfulness training: Trust, satisfaction, and learning. Cyberpsychol. Behav. Soc. Netw. 24(8), 526–535 (2021) 17. Gross, G., Ling, R., Richardson, B., Quan, N.: In-person or virtual training?: Comparing the effectiveness of community-based training. Am. J. Distance Educ. 37(1), 66–77 (2023) 18. Kaufman, J.C.: Counting the muses: Development of the Kaufman domains of creativity scale (K-DOCS). Psychol. Aesthet. Creativity Arts 6, 298–308 (2012) 19. Kapoor, H., Reiter-Palmon, R., Kaufman, J.C.: Norming the muses: Establishing the psychometric properties of the Kaufman Domains of Creativity Scale. J. Psychoeduc. Assess. 39(6), 680–693 (2021) 20. Neff, K.D., et al.: Examining the factor structure of the self-compassion scale in 20 diverse samples: Support for use of a total score and six subscale scores. Psychol. Assess. 31, 27–45 (2019) 21. Neff, K.D.: The development and validation of a scale to measure self-compassion. Self Identity 2(3), 223–250 (2003) 22. Diedenhofen, B., Much, J.: Cocor: A comprehensive solution for the statistical comparison of correlations. PLoS ONE 10(4), e0121945 (2015) 23. Silver, N.C., Hittner, J.B., May, K.: Testing dependent correlations with nonoverlapping variables: A Monte Carlo simulation. J. Exp. Educ. 73(1), 53–69 (2004) 24. Hittner, J.B., May, K., Silver, N.C.: A Monte Carlo evaluation of tests for comparing dependent correlations. J. Gen. Psychol. 130(2), 149–68 (2003) 25. Verger, N.B., Shankland, R., Sudres, JL.: High artistic achievements and low emotion dysregulation: The moderating and mediating role of self-compassion. Creat. Res. J. 34(1), 68–84 (2022)

Learning in Encounter: Collaborative and Project-Based Strategies for Learning in Culturally and Religiously Diverse Contexts in the Higher Education Sector Christoph Knoblauch

and Gökcen Sara Tamer-Uzun

Abstract Focusing on the intimate relationship between encounter and education, this paper discusses evaluation findings from a collaborative and project-based course in the higher education sector. The course engages students in the development and implementation of collaborative projects in the field of Religious Education with a special focus on cultural and religious diversity. The empirical findings, therefore, focus on students’ experiences in collaborative and project-based settings—face-toface (Face-to-face interaction in this article means analog, physical collaboration.) and digitally—with a special emphasis on encounter in religiously diverse groups. This paper analyzes the planning, execution, and critical reflection of learning experiences through collaboration and encounter. It thereby discusses the structure, the methodology, and the outcomes of the course with a focus on the experience of encounter. The study uses digital feedback and evaluation methods for the evaluation research. By doing so, the study investigates and reflects the quality of students’ experiences through encounter in religiously diverse groups and possible influences on learning. It also discusses the question of how these experiences can be constructively implemented to improve future collaborative scenarios in the higher education sector. Keywords Learning and encounter · Religious diversity · Experiential learning · Experience · Higher education sector · Empirical qualitative evaluation

C. Knoblauch (B) · G. S. Tamer-Uzun (B) Ludwigsburg University of Education, Reuteallee 46, 71634 Ludwigsburg, Germany e-mail: [email protected] G. S. Tamer-Uzun e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_21

263

264

C. Knoblauch and G. S. Tamer-Uzun

1 Culturally and Religiously Responsive Education Cultural and religious diversity influence social coexistence in societies all over the world [1]. People with different cultural and religious backgrounds and diverse histories of socialization live together in various contexts and share their individual religious and cultural concepts with their communities. Interrelations of society, culture, and religion can be observed in these contexts in various ways, as individuals develop beliefs, worldviews, and traditions based on socialization, education, and experiences [2]. Paul Tillich describes culture as one form of expression of religion. At the same time, he regards religion as content of culture [3]. “Religion is part and parcel of the culture in which it is lived” [4]. These interrelations not only show in families and societies but in various aspects of everyday life and human relationships. In times of globalization and migration, the importance of cultural and religious identities, and their visibility within society, have to be appreciated to embrace diverse cultural and religious affiliations constructively [5]. Therefore, the presence of diverse cultural and religious identities in societies produces a need for sensitive learning environments, which must be built on principles of appreciation and equality [6]. Learning involves various processes that are almost always embedded in lived relationships and, therefore, marked by diverse individual views and concepts. Ambivalent concepts of culture and religion and the resulting worldviews and practices demand a high level of resilience, often described as tolerance of ambiguity [7]. Against this background, the sensitive inclusion of diverse cultural and religious backgrounds becomes an important responsibility of educational institutions [8]. Culturally and responsive education is crucial for teachers to professionally meet the challenges of diverse classrooms in face-to-face and digital encounter. Therefore, the higher education sector should be prepared to implement culturally and religiously responsive education in teacher training. Higher education institutions, as social and democratic spaces of learning, have the responsibility to offer spaces for learning and growing up which are characterized by an appreciation for diversity and equality. Cultural and religious responsiveness and the willingness of the educator to establish an open and sensitive climate seem to be important traits for these learning spaces which have to include the various geographies of the students’ socialization. Against this backdrop, students’ diverse life worlds should become part of the educational processes, to offer learning potentials through encounter and collaboration [9]. Through encounter in culturally and religiously diverse project teams, students can develop relationships with others; they can enhance their relationships with themselves, and can foster relationships with organizations, groups, and society as a whole [10]. In this regard, project-based and collaborative scenarios are discussed as some of the most constructive ways to cope with the new challenges of a highly diverse world and to develop and foster competencies in collaborative and autonomous learning [11]. In this context, the course “Encounter- and Project-based Learning in religiously diverse contexts” was developed at Ludwigsburg University of Education (LUE).1 The course aims at students in teacher training, who want to 1

https://www.ph-ludwigsburg.de/en/ (access 2/1/2022).

Learning in Encounter: Collaborative and Project-Based Strategies …

265

teach Religious Education in schools. This paper discusses results from a qualitative evaluation study with a special focus on possible potentials for culturally and religiously sensitive education through encounter in teacher training. In this study, students share their ideas on and experiences with cultural and religious diversity in learning spaces and discuss the role of collaborative and project-based learning in culturally and religiously diverse contexts in the higher education sector. The processes of encounter and experiential learning in blended-learning environments have become increasingly important and have changed considerably in the face of new challenges. Experiential learning in the context of new information technologies has become an integral feature of societies and their educational systems all over the world. Against this backdrop, experiential learning has been discussed as an important approach in various educational contexts with a special emphasis on the concepts of experience and reflection [12]. Within the last few years, the ideas of experiential learning have been exposed to new opportunities and challenges through the development of digital learning environments [13]. The world of learners and teachers has become more flexible and interconnected and at the same time more isolated as a result of digital learning environments. “…people who collaborate with us in our projects, inhabit and co-constitute environments in which digital technologies and media are inextricably entangled.… the people we meet in the course of our projects move through worlds that are at once online and offline, and (…) are never separated from the digital or material elements of life” [14]. Accordingly, digital learning environments are understood as the various digital contexts in which learners are active, such as eLearning, interactive digital collaboration, digital research, and many more [15]. While, for example, collaboration options in digital settings may be readily accessible, the qualities and characteristics of digital and in-person collaboration are different [16]. Digital learning environments influence experiences in learning, offering new opportunities but also challenging existing potentials. This is discussed by many academics whose interests lie within the scope of experiential and digital learning scenarios [17]. In this regard, it shows that experience has to be respected as a crucial factor in digital learning environments as it is fundamental to learning processes in general. Experiential learning can be regarded as one of the most original learning processes, deeply rooted in human nature [18]. Of course, not all experiences are equally educative; however, personal experience and education are often closely related [19]. Thus, the quality of experience, especially in digital environments, has to be discussed in depth. Identifying experiences in digital environments, which are agreeable and have a positive influence on further learning processes, seems to be a challenge for future research. Looking at the intimate relation of experience and education in non-digital environments, the question arises of how exactly digital contexts influence experience in learning. However, wellresearched resources offering guidance on how to promote experiential learning in digital contexts and how to design experience-oriented digital courses for higher education remain rare.

266

C. Knoblauch and G. S. Tamer-Uzun

2 Collaboration and Encounter—Description of the Course Design This paper analyzes and discusses students’ learning experiences in collaborative settings that are based on encounter in culturally and religiously diverse teams. Therefore, it evaluates students’ experiences in collaborative and project-based settings, focusing on the role of encounter in culturally and religiously diverse project teams. The course “Learning in encounter in religiously responsive contexts” serves as the basis of this study and was conducted at LUE in the summer semester of 2021. The main foci of the course are (a) the discussion of cultural and religious diversity in educational contexts, (b) project-based and experiential learning, (c) the collaborative development of learning tasks in the field of Religious Education, and (d) the implementation of these tasks at learning festival.2 The character of the course aims for the interconnection of students with diverse cultural and religious backgrounds based on encounter and collaboration through a project in the field of Religious Education. The course was conducted face-to-face and digitally; approximately 50% of the course sessions were conducted face-to-face at LUE. The other 50% of the course was carried out digitally, using learning management systems such as Moodle. The face-to-face session mostly offered input and guided discussions on fundamental theoretical concepts. The digital sessions were based on students’ questions and reports, offering group discussions and individual tutoring in break-out rooms. These sessions offered a blend of learning arrangements; readings, podcasts, audio presentations, interactive forums, videos, and chats were used. The collaborative and project-based focus of the course aims at helping students to learn about cultural and religious diversity through encounter and to gain experience by developing learning tasks in the field of education. A speciality of this course is that the participating students reserve several weeks within the semester for the project-based, autonomous development of learning tasks in the field of Religious Education. These learning tasks are then part of a learning festival at the University of Education, where children can interact and learn with the students. Cultural and religious diversity can, therefore, be experienced by the students of this course within their project-based work and afterward by children who participate in the learning tasks developed by the students. Students can gain up to three credit points (as defined by the European Credit Transfer and Accumulation System—ECTS) for the course and can use their projects as a basis for their Master’s Thesis. Against this backdrop, the study reports on the development, implementation, and evaluation of a course in the program “Teacher Education”3 at Ludwigsburg University of Education (LUE) in Germany. Participants were thirty-six students enrolled in teacher education programs at LUE in the summer term of 2021—one of their majors being Religious Education. Students’ attitudes, practices, and preferences concerning 2

The Learning Festival of LUE invites pupils from schools in the area to visit the University Campus and participate in learning tasks designed by the students of LUE. 3 https://www.ph-ludwigsburg.de/en/studies/study-programs (access on 2/1/2022).

Learning in Encounter: Collaborative and Project-Based Strategies …

267

learning experiences in collaborative and religiously diverse settings play a major role in the design of the courses and their evaluation. The course discussed shows a mix of synchronous and asynchronous course sessions, thus using face-to-face and digital ways of learning and teaching. The focus of this paper is to discuss the potential for collaborative and experiential learning in diverse environments. To facilitate an insight into the learning experiences of students in the described settings, the paper discusses a qualitative study focusing on the reflections of students toward personal learning experiences through encounter. By doing so, the study assesses current practice and analyzes the course mentioned in detail. The discussion of fundamental pedagogical characteristics that influence learners’ experiences and focus on encounter leads to a theoretical framework for the development and analysis of the qualitative data.

3 Qualitative Design of the Study The study focuses on the reflections and discussions of the participating students of the course, concerning their individual experiences through collaboration and encounter in the described learning contexts. Therefore, it uses a creative qualitative design, which is conducted digitally. The complex research focus—students’ experiences in learning in digital environments—asks for an innovative approach observing multiple perspectives. Therefore, students were asked to discuss questions in their collaborative groups and share their answers via an online platform. The interview guide was sent to the students in advance via email [20]. The participants could take as much time for reflection as they considered appropriate. After the discussion of the questions in the groups, students sent their answers to the research team via a digital transfer system [21]. The digital asynchronous way of data collection is a rather new technique in qualitative research, which encourages respondents to reflect on their answers by allowing them to structure their ideas and responses beforehand [22]. Using this asynchronous digital method, the study looks for data that offer a comprehensive view through reflected, subjective feedback. The semi-structured questionnaire shows different categories which were developed in a deductive procedure: a) Experiencing project-based collaboration in diverse groups, b) Learning about others and learning about myself: The role of encounter, c) Experiential Learning in encounter in collaborative and project-based settings, d) Possible relevance for future teaching, e) Experiencing blended-learning environments in collaboration. These deductive categories were built based on the reflection of theoretical frameworks focusing on experiential learning. Each category includes several key questions, which are supported by specifying impulses or further questions. The questionnaire mainly focuses on the project-based and experience-oriented character of the course. Reflections on learning strategies and perceptions of change within learning are also part of the questions.

268

C. Knoblauch and G. S. Tamer-Uzun

4 Findings of the Study The qualitative study focuses mainly on the evaluation of learners’ experiences and attitudes towards learning in encounter in culturally and religiously diverse, collaborative, and project-based contexts. The analysis and discussion of the data show different combined categories, which are based on the deductive categories and new inductive impulses found within the data. The findings are structured and discussed according to these categories. Several answers and reflections show links to more than one category and are therefore discussed in various contexts. The students’ projects focused on culturally and religiously diverse topics in the field of Religious Education, such as “Experiencing religious diversity with all senses”, “Quran and Bible—comparative perspectives”, “The element of water in the Abrahamic context”, or “Symbols in Islam and Christianity.”4

4.1 Experiencing Project-Based Collaboration in Diverse Groups Project-based collaboration plays a major role in the discussed course as students (a) are part of a setting with participants from diverse cultural and religious backgrounds, (b) work together in culturally and religiously diverse groups in project-based settings and (c) develop learning tasks for children with diverse biographies. a) The interviewed students report various positive learning experiences in the course, as they could interact with students and teachers from different cultural and religious backgrounds. Furthermore, some students emphasized that the guided discussion, in religiously diverse groups and with teachers from diverse religious backgrounds, led to authentic and intense learning experiences: “The wide diversification of theological topics, related to authentic insights was very helpful.” Many students describe the collaboration in the course as exciting and motivating as one could discuss with others based on firsthand and authentic experiences. At the same time, some students emphasized the importance of a differentiated and comprehensive theoretical foundation which was developed in the course session through the teachers. b) Many of the interviewed groups report about experiences of membership, due to the common goal of their team. The common goal of developing learning tasks for children led to a stronger commitment to collaborations. Students experienced that they were responsible for the implementation of their collaborative project. Therefore, many students report strong motivation and active exchange in their groups. Against this backdrop, the group members could discuss similarities and differences between their individual religious affiliations and imprints. 4

All results, quotations, charts, transcriptions and analysis phases of the study are available through the authors.

Learning in Encounter: Collaborative and Project-Based Strategies …

269

These discussions sometimes outlasted the course-bound work and led to further dialogue. Some students report that they reflected on religious and cultural imprinting through the collaborative processes and beyond. “There were many intensive and profound discussions … and the interaction was interesting and authentic.” Against this background, students discuss the fact that they learned about diverse cultural and religious affiliations and therefore reflected their individual cultural and religious contexts. In this horizon, the development of a certain group dynamic and various collaborative tasks played a key role for many students. c) Some groups report about the importance of direct references to their future work fields: “The projects are relevant for our future work, as we are going to teach children with diverse religious backgrounds.” Collaboration in culturally and religiously diverse groups can prepare students for responsibilities in diverse classrooms and beyond. This shows in the students’ projects, which focus on cultural and religious diversity in various ways. “We wanted to illustrate similarities and differences (between cultural and religious contexts) … and we chose a multisensory approach.” Students report that collaborative, project-based, and diverse settings in teacher training can play an important role in the development of crucial competencies for diverse schools and societies.

4.2 Learning About Others and Learning About Myself: The Role of Encounter Encounter plays a crucial role in the collaborative processes of the course and the project groups. In general, the interviewed students rate the encounter-focused character of the seminar as very helpful. “Especially the various contents and the interaction. Getting rid of biases and discussing questions which one would not ask outside this setting.” It seems that encounter, in the described settings, can lead to experiences of mutual trust and therefore can help to establish a constructive and culturally and religiously responsive atmosphere. Against this backdrop, some students underline the importance of an open and authentic atmosphere in the course and the project groups: “Through the ongoing encounter one could break down prejudices. We learned that prejudices develop if one does not engage with other (religious) contexts.” The interviewed students report that encounter in the course and the project groups led to a respectful and prejudice-conscious learning atmosphere. “To realize that fellow students with other religious backgrounds were interested in my religion was good to see. (…) Some questions (about my religious backgrounds) could only be answered with difficulty, and I had to reflect on my religious affiliations and imprints again and again.” Some students also discuss experiences of emotional involvement, due to encounter. The development of empathy, especially through project-oriented tasks, seems to help students to discuss openly [23]. The development of empathy seems to be closely connected to encounter in project-based settings and shows a strong impact on the individual perspectives of the participating

270

C. Knoblauch and G. S. Tamer-Uzun

students. In this context, empathy is discussed as the development of a reflected awareness of the project’s topic and the involved participants.

4.3 Experiential Learning in Encounter in Collaborative and Project-Based Settings Experiential learning is closely connected to encounter and can profit from collaborative and project-based learning arrangements [24]. The interviewed students discuss their experiences in (a) the course, (b) the project teams, and (c) the learning festival: a) Interaction with other students and guided discussions are discussed as some of the most important elements of the course. Amidst various individual cultural and religious biographies students can become aware of their sociocultural imprints in cultural and religious contexts. The selected contents of the course and the discussion with fellow students and teachers opened up many constructive learning opportunities. Again, encounter was discussed as a key element of the learning experiences: “The encounters in the course … made the other religion more comprehensible and real.” In addition to this, the course is seen as a basis for the project-based approach and is discussed as a helpful starting point and guidance for the development of learning tasks. b) The importance of a constructive group dynamic and guidance through lecturers are mentioned many times as important traits of the project-based part of the course. Encounters in the project teams are discussed as highly authentic and real, as common goals helped to establish experiences of trust and belonging. Some students emphasize the importance of project-based work for their future teaching competencies: “The project-based work is relevant to our future work fields … as it prepares us for religious diversity in schools.” c) The Learning Festival and its preparation led to various learning experiences: Some students report that they were struggling with the heterogenous learning levels of the students: “Generalization just does not work.” Other students reported positive experiences with active learning approaches and collaborative tasks for the participating pupils. In general, the students are highly satisfied with the motivation and the level of participation of the pupils. A high level of differentiation, made possible through learning circles and various learning options, is discussed as an important trait for the pupils’ motivation.

4.4 Possible Relevance for Future Teaching The interviewed teams discussed strong practical relevance and encounter in culturally and religiously heterogeneous groups as highly relevant for their future work fields. Additionally, the reflection of biases and diverse ways of socialization seems to be very important for students in teacher training: “Reflecting on one’s awareness

Learning in Encounter: Collaborative and Project-Based Strategies …

271

of prejudice and one’s individual imprint. (…) The dismantling of stereotypes.” Also, many students discussed the development of a diversity-responsive attitude as fundamental for teachers and something that should be established during teacher training. Against this backdrop, culturally and religiously responsive education is discussed as an interdisciplinary approach that should be discussed in schools generally. Again, encounter and experience play a key role for many students:“… educating children in religiously diverse settings is easier if one has had experiences oneself.” Some students report that they experienced constructive impulses in their diverse teams which led to further reflection about theological issues in general: “It seems as if one never stops learning about religion—be it a different religious view or your own.” Some students, however, missed a focus on exegetical perspectives and current issues, to gain a more profound view of certain topics. Also, the visit to different places of worship was something that could have helped to relate more to different religious denominations.

4.5 Experiencing Blended-Learning Environments in Collaboration The interviewed students discuss experiences in digital environments in various categories and connection to different topics and processes. They used different strategies for the implementation of their projects in face-to-face and digital environments: Some students conducted their projects mainly face-to-face, whereas others balanced online and face-to-face modes. Students who met face-to-face mostly report that the implementation and reflection of projects benefit greatly from in-person collaboration, as encounter and dialogue are more authentic. Students discuss that the faceto-face encounters helped them to connect project-based topics and private attitudes. Therefore, personal ideas could be shared easier and openly. Some students report that they experienced the face-to-face meetings as more intense and more in-depth discussions developed. Again, empathy plays a role in the students’ answers; faceto-face interaction seems to encourage feelings of belonging and empathy in groups. Students who balanced online and face-to-face modes confirm these results. However, they discuss positive traits of online collaboration, such as the opportunity to gather and share information online and simultaneously, the chance to meet without using a car or public transport and other time-saving aspects of digital communication.

5 Discussion of the Findings The qualitative approach of this study is looking for the reflection of students’ experiences in collaborative and project-based settings—face-to-face and digitally— with a special emphasis on encounter in religiously diverse groups. The qualitative

272

C. Knoblauch and G. S. Tamer-Uzun

perspectives were implemented to gain a deeper understanding of the importance of encounter in collaborative and project-based settings in culturally and religiously diverse groups. At the same time, the results can help to discuss the question of how these experiences can be constructively implemented to improve future courses in the higher education sector. Therefore, the study analyzes the planning, execution, and critical reflection of learning experiences through collaboration and encounter. The methods employed largely focus on the same phenomena and the findings are discussed in this analysis, by triangulating the different qualitative perspectives [25]. Three main foci could be established during the analysis process and are discussed in the context of learning in encounter: (a) Experiences in learning in encounter in culturally and religiously diverse contexts; (b) the role of encounter in teacher training; (c) the use of blended-learning environments in project-based settings. a) Learning in encounter in culturally and religiously diverse contexts offers a variety of learning experiences. The study indicates that these experiences can be profound and sustainable if they are developed in authentic and open settings. Encounter between learners with different cultural and religious backgrounds can lead to the dismantling of prejudices and encourage understanding. In order to achieve this, however, it is helpful to offer encounter which is embedded in project-based settings. When students with diverse backgrounds share a common goal and their projects are supported by a solid course structure and mentoring programs, encounter can lead to constructive and sustainable learning [26]. Project-based work can lead to experiences of involvement and belonging and create strong connections toward common goals and other participants. Emotional involvement seems to encourage motivation and learning processes [27]. Within the projects, learners can experience a high level of autonomy as they can pursue individual interests and control fundamental processes within their projects [28]. At the same time, learners are dependent on collaboration with others and experience diverse ideas and imprints. Against this backdrop, learning in encounter leads to the development of new knowledge about other cultural and religious views, but at the same time about one’s own beliefs: Learning about others and learning about the self [29]. b) The role of encounter in teacher training is essential for the development of a culturally and religiously responsive attitude. Teachers in culturally and religiously heterogeneous classrooms must be self-conscious, critical, and analytical of one’s own beliefs, imprints, and behaviors [30]. The study indicates that students in teacher training profit from project-based and collaborative settings which encourage encounter with diverse teams. The exposure to new and unfamiliar cultural and religious backgrounds seems to encourage students to reflect on their contexts and discuss possible stereotypes. The results show that students in teacher training value encounter in culturally and religiously diverse teams and become motivated to reflect on individual ideas and imprints through projectbased collaboration. In addition to this, the project’s goal—the development of learning tasks for pupils—seems to offer many potentials for the development

Learning in Encounter: Collaborative and Project-Based Strategies …

273

of didactic competencies, as it connects the reflection on cultural and religious diversity with a practical orientation. Furthermore, it shows that the implementation of the projects and the communication with children and teachers at the Learning Festival created intensive learning experiences. c) Face-to-face contacts seem to play an important role in encounter in culturally and religiously diverse contexts. Digital tools, however, seem to be very efficient, when it comes to organizing and structuring collaboration. It shows, that digital environments are highly efficient for students who collaborate in projectbased settings. Experiences with encounters in digital environments, however, seem to be less sustainable [31]. Students seem to collaborate in face-to-face and digital environments in a flexible and largely independent manner. Furthermore, students discuss various positive experiences with the blending of face-toface and digital collaboration [32]. Thus, a flexible course structure should offer students opportunities to blend face-to-face and digital encounter.

6 Outlook Encounter is essential to learning as it leads to authentic and sustainable experiences among students. Collaborative and project-based strategies show vast potential for learning in culturally and religiously diverse contexts, as they encourage students to interact and collaborate. The study indicates that: • Learning in encounter in project-based settings is emphatic and authentic and leads to the dismantling of prejudices, as it brings students together with joint tasks and a common goal. • Learning in encounter encourages very strong participation when realized in collaborative and project-based contexts. The joint responsibility for a project leads to involvement and exchange. • Learning in encounter leads to authentic interaction when students collaborate in project teams. • Learning in encounter takes into account the level of the student’s individual development and the level of the group’s development. Learning happens in relationship to others. • Learning in encounter enables students to become teachers. In collaborative processes, students learn and teach at the same time. “The community moves toward a mutuality that encourages total commitment and a willingness to share” [33]. Against this backdrop, learning in encounter can be discussed as a crucial factor for learning in culturally and religiously diverse groups. Collaborative projects in the field of Religious Education and beyond should implement the idea of encounter to offer students authentic and constructive learning experiences with others. The structure of such courses should offer (a) a theoretical basis for all students, (b) a discussion of strategies for project-based learning, (c) a realistic and clear

274

C. Knoblauch and G. S. Tamer-Uzun

timeframe, (d) a mentoring program for the projects, (e) a platform where the projects can be presented and reflected, and (c) room for reflection on the learning experiences. In this context, blended learning strategies can play an important role in the successful implementation of the course: The development of encounter and relationships and the experiences of membership and belonging are more intense and authentic in face-to-face interactions. However, digital environments can make project-based learning more efficient, contextual, and personalized. Encounter can also take place in digital environments, even though of different quality. Therefore, the entire learning environment should offer various digital and face-to-face learning methods. Learning in encounter, initiated by project-based, collaborative, and blended learning designs, might be regarded as transformative as it meets the challenges of culturally and religiously diverse groups constructively and supports students in becoming self-reflective and culturally and religiously responsive teachers [34].

References 1. Mellor, P.A.: Religion, culture and society in the “information age.” Sociol. Relig. 65(4), 357–371 (2004) 2. Geertz, C.: The Interpretation of Cultures: Selected Essays. Basic Books, New York (1973) 3. Tillich, P.: Religionsphilosophie. Kohlhammer, Stuttgart (1962) 4. Witte, M.: Zu diesem Buch. In: Witte, M. (ed.) Religionskultur – zur Beziehung von Religion und Kultur in der Gesellschaft. Religion und Kultur, Würzburg, 11 (2001) 5. Beyers, J.: Religion and culture: revisiting a close relative’. HTS Teologiese Stud./Theol. Stud. 73(1), 1–3 (2017) 6. Auernheimer, G.: Einführung in die Interkulturelle Pädagogik (7), 15 (2012) 7. Frenkel-Brunswick, E.: Intolerance of ambiguity as an emotional and perceptual personality variable. J. Pers. 18, 108–123 (1948) 8. Gaus, R.: Global (Citizenship) Education as inclusive and diversity learning in Religious Education. J. Relig. Educ. (69), 179–192 (2021) 9. Selçuk, M.: Learning in encounter: crossroads, connections, collaborations. Relig. Educ. 113(3), 233–243 (2018) 10. Boschki, R.: “Beziehung” als Leitbegriff der Religionspädagogik. Stuttgart (2003) 11. Stefanou, C., Stolk, J., Prince, M., Chen, J., Lord, S.: Self-regulation and autonomy in problemand project-based learning environments. Act. Learn. High. Educ. 14(2), 109–122 (2013) 12. Miettinen, R.: The concept of experiential learning and John Dewey’s theory of reflective thought and action. Int. J. Lifelong Educ. (19:1), 54–72 (2000) 13. Yates, A., Starkey, L., Egerton, B., Flueggen, F.: High school students’ experience of online learning during Covid-19: the influence of technology and pedagogy. Technol. Pedag. Educ. 30(1), 59–73 (2021) 14. Pink, S.: Foreword. In: Frömming, U., Köhn, S., Fox, S., Terry, M. (eds.) Digital Environments, p. 9 (2017) 15. Iskakov, I., Kovalenko, B., Turovskaia, M., Getmanova, G.: Online blended learning in the digital environment. J. Phys. Conf. Ser. 1691 (2020) 16. Brooks, E., Selander, S.: Designing for collaboration. Frameworks for learning. In: Brooks, E., Dau, S., Selander, S. (eds.) Digital Learning and Collaborative Practices. Lessons from Inclusive and Empowering Participation with Emerging Technologies, pp. 1–4 (2022) 17. Bouilheres, F., Le, L.T.V.H., McDonald, S., et al.: Defining student learning experience through blended learning. Educ. Inf. Technol. (25), 3049–3069 (2020)

Learning in Encounter: Collaborative and Project-Based Strategies …

275

18. Boud, D.: Forward. In: Warner Weil, S., McGill, I. (eds.) Making Sense of Experiential Learning: Diversity in Theory and Practice. Open University Press, London (1989) 19. Dewey, J.: Experience and Education. Touchstone, New York (1997) 20. Ratislavová, K., Ratislav, J.: Asynchronous email interview as a qualitative research method in the humanities. Hum. Aff. 24(4), 452–460 (2014) 21. Salmons, J.: Qualitative Online Interviews. Strategies, Design, and Skills, 2nd edn. SAGE, Los Angeles (2015) 22. Thunberg, S., Arnell, L.: Pioneering the use of technologies in qualitative research – a research review of the use of digital interviews. Int. J. Soc. Res. Methodol. (25:6), 757–768 (2022) 23. Md Hashim, A., Syed Aris, S.R., Chan, Y.F.: Promoting empathy using design thinking in project-based learning and as a classroom culture. AJUE 15(3), 14–23 (2020) 24. Kong, Y.: The role of experiential learning on students’ motivation and classroom engagement. Front. Psychol. 2, 771272 (2021) 25. Denzin, N.K.: The Research Act. A Theoretical Introduction to Sociological Methods (2). McGraw-Hill, New York (1978) 26. Ratzke, C.: Hochschuldidaktisches interreligiöses Begegnungslernen. Dissertation. Münster, Waxmann, pp. 222–245 (2021) 27. Knoblauch, C.: Experiential learning in digital contexts - a case study. In: Guralnick, D., Auer, M.E., Poce, A. (eds.) Innovative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, vol. 581, pp. 181–191. Springer (2022) 28. Knoblauch, C.: Digital project-based learning in the higher education sector. In: Guralnick, D., Auer, M.E., Poce, A. (eds.) Innovations in Learning and Technology for the Workplace and Higher Education. Lecture Notes in Networks and Systems, vol. 349, pp. 170–179. Springer (2021) 29. Asaba, M., Hyowon, G.: Learning about others to learn about the self. Early reasoning about the informativeness of others’ praise. In: Brummelman, E. (ed.) Psychological Perspectives on Praise, 1st edn., pp. 67–75. Routledge (2020) 30. Gay, G., Kirkland, K.: Developing cultural critical consciousness and self-reflection in preservice teacher education. Theory Pract. 42(3), 181–187 (2003) 31. Liu, S.: Student interaction experiences in distance learning courses a phenomenological study. Online J. Distance Learn. Adm. 11(1) (2008). https://www.learntechlib.org/p/158563/. Accessed 10 Jan 2023 32. Knoblauch, C.: Combining and balancing project-based and blended learning in education. Int. J. Adv. Corp. Learn. (iJAC) 15(1), 35–44 (2022) 33. Dow, R.A.: Learning Through Encounter. Judson Press, Valley Forge (1973) 34. Gay, G.: Culturally Responsive Teaching. Theory, Research, and Practice, 2nd edn., pp. 22–47, New York (2010)

New Perspectives for Internationalization in Higher Education: Collaborative Formats in Project-Based and Blended Learning Contexts Christoph Knoblauch

Abstract This paper discusses a multi-method comparative intervention study on students’ attitudes and preferences toward project-based collaboration in blendedlearning courses in an international setting. It focuses on the comparison of international collaboration in digital and in-person contexts. The discussed courses were developed and taught in the context of an international blended-learning collaboration in teacher education between Dr. Ambedkar University (AUD) in Delhi, India, and Ludwigsburg University of Education (LUE), Germany. The paper focuses on the analysis and discussion of students’ experiences in project-based, collaborative, and blended learning settings. A special interest lies in data showing similarities and differences between digital and in-person collaboration between students. The study uses a multi-method design with qualitative interviews and different written feedback forms, concerning different aspects of the collaborative processes. By doing so, the study assesses current practice and analyzes the above-mentioned collaborative processes closely and in detail, thus seeking to investigate how students experience project-based, collaborative, and blended-learning environments, especially in an intercultural, international setting. The paper also discusses how the results can be constructively implemented to improve future blended-learning scenarios in the higher education sector. Keywords Project-Based Learning · Blended Learning in Higher Education · Collaborative Learning · International Collaboration · Digital and in-person Learning

C. Knoblauch (B) Ludwigsburg University of Education, Ludwigsburg, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_22

277

278

C. Knoblauch

1 Project-Based, Collaborative, Digital and In-person Learning: Background and Context of the Course Elements The process of learning and teaching in collaborative blended-leaning settings1 has become increasingly important and has changed considerably in the face of new challenges [1]. Project-based and collaborative learning and working with the assistance of new information technologies have become integral features of higher education systems and societies all over the world [2]. The world of learners and teachers has become more autonomous and, at the same time, more interconnected as a result of new challenges such as internationalization and digitalization. Internationalization itself is a broad and varied phenomenon in higher education, driven by various dynamic motivations, such as political agendas, economic interests, academic rationales, lecturers’ and students’ interests, and many others [3]. Higher education systems are in the need to react to the challenges and potentials of international collaboration, by offering students the possibility to experience and discuss international perspectives in digital and in-person environments. In this regard, project-based and collaborative learning approaches are discussed as some of the most important ways to cope with the new challenges of a globalized world and to develop and foster competencies in autonomous learning in the higher education sector [4]. Projectbased and collaborative learning approaches seem to have proven particularly relevant in this context, for they can offer tools for communication, autonomous learning, collaboration, and shared learning between students and teachers [5]. Consequently, there is a strong demand for the development of international project-based courses in increasingly autonomous and blended learning environments. The higher education sector plays a major role in these developments and must deal with unknown and known challenges to prepare students for learning and teaching in international, autonomous, project-based, and blended contexts. Digital and in-person collaboration show different qualities and characteristics [6]. Therefore, the question arises of how students experience international projects-based collaboration in digital and in-person contexts and how these can be blended constructively to combine the best features of both. Against this background, however, well-researched resources offering guidance on how to combine digital and in-person contexts in international project-based collaboration and how to design adequate courses for the higher education sector remain rare. This study reports on the development, implementation, and evaluation of two related courses in the Master’s program “Teacher Education” at LUE in Germany. The first course, in the winter term of 2021 was based on digital collaboration with students and lecturers from AUD in India. The second course in the following summer term of 2022 was based on in-person collaboration with Indian and German students and lecturers at AUD in Delhi. The interviewed students participated in both courses, 1

In the context of this paper, blended learning is understood as the integration of traditional inperson (face-to-face / physical) learning with digital contexts, such as digital platforms, videocalls, distance learning and others.

New Perspectives for Internationalization in Higher Education …

279

thus having the opportunity to experience both digital and in-person international collaboration in project-based contexts.2 The key question of the research study is how students experience international project-based collaboration in digital and in-person contexts. Therefore, students’ attitudes, practices, and preferences concerning digital and in-person collaboration play a major role in the design of the courses and their evaluation. The courses discussed, therefore, follow different approaches to project-based collaboration. The participating students collaborated (a) digitally in the first course and (b) in-person in the second course. During the digital collaboration, they worked autonomously for several weeks on the development, implementation, and reflection of a joint international project in the field of education. In order to facilitate an insight into the experiences of students in collaborative international project-based learning environments, the paper discusses a multi-method comparative intervention study focusing on the attitudes and preferences of students in digital and in-person contexts. The study looks at how German and Indian students experience online and in-person project-based collaboration.

2 Outline of the Courses The courses “Diversity and Resilience in Education (I) + (II)” serve as the basis of this study and were conducted at LUE and AUD simultaneously. The main foci of the courses are (a) the discussion of diversity and resilience in educational contexts, (b) project-based and collaborative learning, (c) the autonomous implementation of a project in the field of education, (d) the discussion of international perspectives, and (e) the presentation and reflection of the projects. The character of the courses aims for the interconnection of students with international partners, their future work area, and their future colleagues through a project in the field of education. The collaborative project-based focus of the course aims at helping students to gain international experience by actually implementing a project in the field of education. Students can gain up to three credit points (as defined by the European Credit Transfer and Accumulation System – ECTS) for each course and can use their projects as a basis for their Master’s Thesis.

3 Design of the Study and the Evaluation The study focuses on the reflections and discussions of the participating students of both courses, concerning their individual experiences in the described learning contexts. It uses a multi-method approach that focuses on students’ attitudes and 2

Participants are students enrolled in teacher education programs at LUE and AUD in the terms of 2021/2022. The students are between 20 and 24 years old.

280

C. Knoblauch

preferences toward project-based collaboration in digital and in-person contexts in an international setting. Data were collected through two quasi-experimental studies. One study was conducted before the in-person collaboration and the second one after the in-person collaboration. Each study was implemented with the same sample, namely students participating in both courses [7]. All students, therefore, received the intervention and were asked to reflect on their experiences in both, digital and inperson contexts [8], through a multi-method approach using qualitative interviews and various forms of asynchronous and written feedback [9]. The interviews and feedback options focused on the experiences in both contexts and the treatment of inperson collaboration. The complex research focus — students’ experiences in digital and in-person collaboration — asks for an innovative approach observing multiple perspectives through (a) synchronous dialogue-based interviews and (b) individual asynchronous feedback. Against this backdrop, the study looks for individual and subjective feedback and group discussion to gain a more comprehensive view. Semistructured qualitative interviews were carried out in (a) a synchronous digital way, using video calls, and (b) in an asynchronous digital way. The digital asynchronous way of interviewing students is a rather new technique in qualitative research, which encourages respondents to reflect on their answers by allowing them to structure their ideas and responses beforehand [10]. The (a) synchronous interview situations offered the possibility to discuss questions in deep, to clarify ambiguities, and to develop a constructive dialogue between the interviewer and the participants. The (b) asynchronous way offered the possibility to record answers as audio files independently. The participants could take as much time for reflection as they individually considered appropriate. Using this combined method, the study looks for data that offer a comprehensive view through individual, reflected, subjective feedback, and dialogue [11]. The discussion of fundamental pedagogical characteristics in the context of international, collaborative, and project-based learning [5] leads to a theoretical framework for the development of the questionnaires and the analysis of the qualitative data: (a) General experiences in the courses, (b) project-based learning in in-person and digital contexts, (c) collaboration in in-person and digital contexts, and (d) internationalization in in-person and digital contexts. These deductive categories were built based on the reflection of theoretical frameworks focusing on collaborative and project-based learning in international contexts. Each category includes several key questions, which are supported by specifying impulses or further questions. Reflections on learning strategies and perceptions of change within learning are also part of the questions. The data are structured according to these categories and analyzed with the perspectives (a) general experiences, (b) experiences with digital learning, and (c) experiences with in-person learning. The discussion of the data has a special focus on the students’ comparisons of in-person and digital contexts through their experiences with the intervention.

New Perspectives for Internationalization in Higher Education …

281

4 “Encounter Can Also Take Place Digitally, However…”: Findings of the Intervention Study The analysis and discussion of the data are realized in the discussed four combined categories, which are based on the deductive categories and combined with inductive impulses found within the data. All categories discuss experiences with and attitudes toward physical and digital learning. The findings within the data are structured and discussed according to these categories. However, some answers and reflections within the data show links to more than one category and are therefore discussed under the perspectives of different categories.3

4.1 Experiences with Learning in In-Person and Digital Contexts General Experiences in the Courses. For many students, the prospect of the implementation of a small-scale empirical project seems to be a strong motivation for enrolment in the course. In general, education is often discussed as strongly connected to experience in the conducted interviews and many students seem to see a benefit in the implementation of an individual project for their studies and their future work. “I consider the project and the course as an enrichment. In each project, one gains experience which is helpful for my future work.” Project-based settings create experiences that are helpful for further tasks and motivate the interviewed students to work autonomously and self-responsibly. The idea of autonomous and project-based work also seems to be especially motivating, as some students report that they missed working independently in their studies so far. Another factor seems to be the international and interdisciplinary approach of the course. Many students report that they appreciated discovering and pursuing individual interests and ideas and that they were able to connect different disciplines and perspectives in their work. Furthermore, students reported enjoying the experience of presenting their projects and at the same time listening to the presentation of other projects. It shows that the interviewed students want to learn about various projects in different fields of education to gain a more profound insight into specific themes. Experiences with Digital Learning. The interviewed students report on various experiences with the digital contexts of the course, the collaborations, and their projects. Many students agree that without digital tools neither the projects nor the collaborations would have been possible: “Only through digital ways it was possible to realize the projects and the collaborations.” International collaboration, especially, would not have been possible without digital options like shared documents, collaborative platforms, and video calls. Some students are very satisfied with the 3

All results, quotations, charts, transcriptions and analysis phases of the study are available through the author.

282

C. Knoblauch

use of digital methods and the produced outcomes: “All in all I was surprised how smooth everything worked and I am very happy with all the results.” The fact that all information was accessible to all members of the course and the project teams was also highly appreciated. The interview students highly appreciate the asynchronous mode of digital collaboration and report about their experiences with very flexible and independent ways of learning. However, some students also discussed problems with this flexibility: “I find it problematic that one is reachable all the time and everywhere through digitalization. (…) recreation suffers.” Experiences with In-person Learning. Many of the interviewed students report more intense learning experiences in in-person learning contexts. Students seem to experience presentations and discussions as more intense and report a higher level of involvement through stronger relationships in in-person contexts: “I think that the intensity of in-person learning contexts is very different (to digital learning contexts), especially when you discuss and share ideas.” The importance of encounter and relationship is highlighted in many interviews and is experienced more intensely in in-person learning contexts. Additionally, some students report about the significance of in-person meetings, as situations and modes can be experienced more profoundly and discussed directly: “…in the real encounter we can use all our senses to access situations as a whole…” This can lead to efficient collaboration as there seem to be fewer misunderstandings in in-person meetings: “It was clearer when we met in-person. And we usually found solutions faster.” The interviewed students also state that in-person learning can lead to more profound results: “One was able to learn more during the in-person meetings because one discussed issues which were beyond the topic. (…) more background information, to understand things better.” Result. It shows that the interviewed students make valuable experiences in digital and in-person learning contexts. Digital tools are the basis for international collaboration and are valued for their efficient, independent, and flexible features. At the same time, they show limitations and sometimes create misunderstandings. In-person encounter plays a crucial role for many students as it can offer a constructive and creative atmosphere for collaboration and learning. Finally, some students emphasize the importance of blended-learning arrangements: “One always has to reflect on what makes sense in which situation (of the learning and collaboration processes).” The dynamic and situation-oriented balancing of digital and in-person learning contexts seems to be essential.

4.2 Project-Based Learning in In-Person and Digital Contexts Experiences with Project-Based Learning in General. The students in the discussed courses developed and implemented projects in the field of diversity education and followed their interests and subjects in the finding of their themes. Projects

New Perspectives for Internationalization in Higher Education …

283

such as “Religious diversity in education in India and Germany – a comparative study,” “The impact of Covid on diversity and minorities in school.” “Domestic violence and impacts on schools,” and “The importance of diversity responsive teaching in math” were developed in the courses. The reactions to the project-based character of the course and collaborations are mainly positive. Many interviewed students value the possibility to pursue individual interests and thereby develop competencies within their chosen fields of education. The freedom and flexibility, project-based learning offers, and the development of competencies in fields such as organization, workload management, and collaboration were mentioned as positive features of project-based settings many times: “(…) experiences … especially in the areas organization and coordination and the self-discipline that goes along with these.” The interviewed students also valued the potential for creative work and a high level of involvement due to intrinsic motivation and the responsibility for their projects and team members: “It (project-based learning) gets you out of your comfort zone.” At the same time, many students emphasize the importance of a reliable course structure and close mentoring through their lecturers. The flexible and autonomous features of project-based learning should be embedded in a reliable and clear course structure and mentoring programs — the projects seem to need a reliable framework. All in all, the project-based character was discussed as highly relevant for the student’s future studies and work fields: “I thereby (through project-based learning) gained experiences in collaboration, organization, and coordination which I can use for my future studies.” Project-Based Learning in Digital Contexts. The interviewed students discuss digital spaces such as online platforms, clouds, and message boards as crucial for the successful development of their projects. The first steps of developing and organizing projects involved different digital methods as they supported easy and efficient access to information and the sharing of ideas. Digital spaces, which can be used asynchronously, were especially highly valued by the interviewed students: “In the first processes of establishing projects we posted our ideas in the shared Padlet. Then two Indian peers contacted us and we elaborated on these ideas. Then we arranged an (online) meeting.” The second step involved online meetings, mostly carried out via video conference systems. These video calls are described as time efficient but not necessarily constructive. Students report problems with time zones, connectivity, and language. Furthermore, some students regret a rather low level of involvement and various distractions during these meetings. Project-Based Learning in In-person Contexts. All interviewed students report that the implementation and reflection of projects benefit greatly from in-person collaboration: “Once we met (in-person) many of us realized how much better and faster it worked out.” Students report that the various encounters helped them to connect project-based topics and private attitudes. In these processes, “…it got easier to create memories which are connected to content-related things.” Some student report that the in-person meetings helped to include different levels of knowledge and led to an in-depth discussion of the topics: “Different contexts and fields of knowledge came together. (…) This helped to learn about oneself within the project and to

284

C. Knoblauch

broaden one’s horizon.” Some features of project-based learning were highlighted in the discussions: situation-oriented learning, practical orientation, focus on individual interests, communication, and collaboration. Motivation develops through individual interest and collaboration: “One is much more involved if you are really interested in the project and work together with others on it.” Result. It shows that digital spaces seem to be essential for the development of collaborative projects, as asynchronous options, such as shared documents and message boards, help students to get and stay connected. However, the implementation and in-depth discussion of projects seem to be problematic, if limited to digital meetings. In-person learning can fill this gap, as it offers students room for discussion, various perspectives, and different fields of knowledge. Most importantly, in-person learning in project-based settings seems to develop a high level of involvement among the students, as it is based on encounters and the development of relationships.

4.3 Collaboration in In-person and Digital Contexts Experience with Collaborative Processes in General. The interviewed students described the collaborative processes as very intense and constructive. Most students discussed the project-based and international character of the courses as reasons for strong involvement and high motivation. “(…) I never experienced such a strong collaboration within a team.” Against this backdrop, the coordination of the project and the inclusion of different interests were discussed as challenging, yet highly constructive tasks. Many of the interviewed students reported about collaborations with different partners outsides the Universities, such as teachers, headmasters, lecturers, and NGOs. These partners were part of the student’s projects and offered authentic perspectives for the various topics of the course. Collaboration in Digital Contexts. Digital contexts are crucial to international collaboration. The interviewed students discuss the benefits and problems of digital collaboration: “Digital spaces made this (collaboration) possible.” The students report in various contexts that digital ways of communication were crucial to their projects and all collaboration, may it be on an international or local level. Many students emphasize, that shared documents and digital platforms were used as a basis to establish collaborations and to save the results of the various projects. Also, the use of video conferences, where shared documents were processed collaboratively, was mentioned often. At the same time, many students report digital collaborations being very demanding as meetings had to be planned and structured beforehand to make the work processes smooth. Some students report that they found it more difficult to open themselves and share their ideas and thoughts in digital environments. This also could lead to a lack of creativity in digital contexts: “I could imagine that one loses some creativity due to the structure and character of video calls.”

New Perspectives for Internationalization in Higher Education …

285

Also, communication in digital contexts seems to suffer, as many students experienced misunderstandings in their collaborations because they could not interpret the facial expressions and gestures of their team members correctly. This is especially problematic in international collaboration. Email and messenger services are also discussed critically: “Messages leave quite some room for interpretation. And the interpretation of messages can be influenced by the readers’ context, the time and the mood. This also can create misunderstandings.” In classroom settings, digital tools such as laptops can create barriers as they divide the physical space between students and can create a distraction. Finally, many of the interviewed students criticize the lack of interpersonal relations in digital settings: “It is sad that the interpersonal aspect gets lost in digital settings. Therefore, I find it important to establish a mix of digital and in-person learning.” Collaboration in In-person Contexts. Most of the interviewed students find inperson collaboration more constructive and discuss the learning effects as being more sustainable: “ (…) because we experienced so much (in in-person contexts), as we discussed the topics intensively and connected these discussions with situations and memories…” Additionally, in-person collaboration was discussed as being more precise and clearer. Students report that they experienced stronger motivation and involvement in in-depth collaboration: “…the level of engagement is different in in-person collaboration. One participates with a different emotionality. (…) social contact and interaction lead to a more intense analysis of the topics.” In addition to this, some students emphasize the importance of establishing relationships and trust within their in-person collaboration: “One can establish mutual trust (in in-person collaboration) easier…”. Result. It shows that the interviewed students make valuable experiences with collaboration in digital and in-person contexts. Digital tools are the basis for international collaboration and offer flexible and time-independent ways of collaborating. In-person collaboration, however, can offer a more constructive and creative atmosphere for collaboration. The establishment of mutual trust and the in-depth discussion of topics seems to need in-person encounters and interaction.

4.4 Internationalization in In-person and Digital Contexts Experience with Internationalization in General. The interviewed students show a high level of appreciation for international perspectives in the courses and their projects. On top of that, many students report about the benefit of international views in their projects and beyond: “The contact with international partners … supports understanding, tolerance, and open-mindedness, and also intercultural learning…” The different backgrounds and different systems of socialization help students to develop a profound insight into their collaborative projects and the connected topics. Many students report that the discussion of similarities and differences, as well as

286

C. Knoblauch

the reflection of one’s ideas in the context of different perspectives, leads to highly constructive learning processes. Against this backdrop, some students emphasize the importance of the inclusion of the different lifeworlds in the projects. Internationalization in Digital Contexts. Again, digital contexts are discussed as crucial for internationalization in the interviews of this study. However, most students strongly relate their answers about experiences with internationalization to in-person contexts. Internationalization in In-person Contexts. “In-person encounter mattered a lot and had a sustainable impact on my life.” In-person encounter is discussed as a crucial basis for internationalization in all interviews: “It is completely different when you meet in-person … because you get to know the lifeworld of your peers differently and you can discuss similarities and differences completely differently in real life. And this is more fundamental and broader than digitally.” Many students state that internationalization and personal engagement go together with collaboration in inperson contexts. On top of that, students discuss that in-person collaboration leads to a sustainable engagement with the projects and connected matters beyond: “…the collaboration went further than the initial project … I received authentic insights into the lifeworld of my partners … experiences which would not have been possible online.” Result. It shows that internationalization needs in-person collaboration if it wants to create strong engagement and produce profound results. While digital contexts are crucial for the development and continuity of international collaboration, it needs in-person encounter to establish mutual trust and sustainable relationships: “I had the feeling that we had a very constructive discussion when we met in-person … this was extremely precious and broadened our horizon very much…”.

5 New Perspectives for Internationalization International collaboration needs a high level of commitment and exchange between the participating partners [12]. The evaluated courses show that project-based learning can lead to strong experiences of involvement and motivation in the collaboration between international partners [13]. Project-based courses in the field of education can generate a high level of motivation in students and encourage them to engage in autonomous and self-responsible work [14]. The implementation of collaborative projects with international partners can lead to the development of interdisciplinary competencies, a deeper insight into chosen topics, and many-sided connections to future work fields. It allows students to test and reflect on their ideas in autonomous, yet collaborative environments and thus promotes innovation competence and active knowledge construction [2]. Therefore, the collaborative development and implementation of projects by students can be identified as crucial for internationalization in higher education.

New Perspectives for Internationalization in Higher Education …

287

This collaboration needs exchange — digitally and in-person. The comparative study shows that students appreciate digital and in-person contexts for international collaboration. Therefore, international collaboration should be developed in blendedlearning contexts. While digital contexts can be very important for the development, planning, and continuity of joint projects, in-person learning should be involved when partners implement and discuss their projects, and possible results [15]. The comparison of experiences in digital and in-person contexts in this study shows the importance of real exchange through in-person encounter. In-depth and sustainable learning happens mostly in in-person contexts, especially when cultural diversity plays a role [16]. Collaborative project-based courses with international partners require a structure, which enables students (a) to find themes and develop projects individually and with others on suitable online platforms; (b) to share and discuss their ideas online; (c) to develop, and implement and discuss projects in-person; (d) to develop new content and tasks for themselves and others. Course designers need to present a solid course structure and a close mentoring program to support students during all processes of international collaboration. The study shows, the combined process of autonomous and collaborative work is a crucial factor in students’ learning experiences. By following individual interests and ideas and developing these in international teams, especially in project-based settings, students can develop and foster various competencies. Especially students in the higher education sector can gain from these processes as they can (a) set appropriate goals autonomously, (b) reflect and discuss their ideas in international contexts, (c) have frequent opportunities for individual and collaborative revision, and (d) connect with future work fields. These processes have to be supported through learning environments that blend digital and in-person collaboration. Exchange happens through encounter [17]. International encounter in higher education needs digital and in-person contexts to be feasible, productive, constructive, and sustainable. It shows that internationalization in the higher education sector can bring many benefits to both students and educators if developed in blended learning contexts. International collaborative courses can produce increased cultural understanding and awareness, expanded professional networks, and exposure to diverse perspectives and ways of thinking. The study shows that the analyzed courses can foster key aspects that can be especially important for internationalization: (a) Cross-cultural competency: Collaboration and encounter can help both students and educators to be equipped with the knowledge, skills, and attitudes needed to effectively navigate and engage with diverse cultures and perspectives. (b) Intercultural communication: Project-based settings offer opportunities for students and educators to develop their intercultural communication skills, such as active listening, empathy, and the ability to adapt their communication style to different cultural contexts. (c) Collaborative learning: The designed courses and activities foster collaboration and cooperation among students from different cultural backgrounds, and help to build mutual understanding and respect. (d) Virtual collaboration tools: Courses have to incorporate technology to facilitate collaboration and communication among students and educators in different locations, such as video conferencing, instant messaging, and

288

C. Knoblauch

file-sharing platforms. (e) Flexibility: Project-based and collaborative courses foster flexibility in accommodating and adapting to the different cultural and academic backgrounds of the students and educators. Overall, internationalization in higher education shows to be a crucial tool for fostering culturally responsive understanding, but it requires thoughtful planning and a commitment to providing the necessary support and resources to ensure constructive and successful collaboration and exchange. Against this backdrop, multiple benefits of blended learning arrangements are repeatedly evident in this study. The balance and arrangement of digital and face-to-face learning opportunities in the courses studied shows the importance of an adapted and flexible combination of different learning pathways. This combination needs to be individually adapted to different international collaborations.

References 1. Austin, R., Smyth, J., Rickard, A., Quirk-Bolt, N., Metcalfe, N.: Collaborative digital learning in schools: teacher perceptions of purpose and effectiveness. Technol. Pedagogy Educ. 19(3), 327–343 (2010) 2. Guo, P., Saab, N., Post, L., Admiraal, W.: A review of project-based learning in higher education: student outcomes and measures. Int. J. Educ. Res. 102, 101586 (2020) 3. de Wit, H., Altbach, P.G.: Internationalization in higher education: global trends and recommendations for its future. Policy Rev. High. Educ. 5(1), 28–46 (2021). https://doi.org/10.1080/ 23322969.2020.1820898 4. Stefanou, C., Stolk, J.D., Prince, M., Chen, J.C., Lord, S.M.: Self-regulation and autonomy in problem- and project-based learning environments. Active Learn. High. Educ. 14(2), 109–122 (2013). https://doi.org/10.1177/1469787413481132 5. Knoblauch, C.: Combining and balancing project-based and blended learning in education. Int. J. Adv. Corp. Learn. (iJAC) 15(1), 35–44 (2022) 6. Brooks, E., Selander, S.: Designing for collaboration. Frameworks for learning. In: Brooks, E., Dau, S., Selander, S. (eds.) Digital Learning and Collaborative Practices. Lessons from Inclusive and Empowering Participation with Emerging Technologies, pp. 1–4 (2022) 7. Sorrell, D., Brown, G.T.L.: A comparative study of two interventions to support reading comprehension in primary-aged students. Int. J. Comp. Educ. Dev. 20(1), 67–87 (2018). https://doi. org/10.1108/IJCED-08-2017-0018 8. Harris, K.: The state of educational intervention research as viewed through the lens of literacy intervention. Br. J. Educ. Psychol. 76, 1–19 (2006) 9. Ehlers, U.: Qualitative online befragungen. Lothar Mikos und Claudia Wegener (Hg.): Qualitative Medienforschung Ein Handbuch 2, 327–339 (2017) 10. Salmons, J.: Qualitative online Interviews. Strategies, Design, and Skills, 2nd edn. Sage, Los Angeles (2015) 11. Berg, B., Lune, H.: Qualitative research methods for the social sciences 9, 21–30/172–174 (2017) 12. Atalar, A.: Student exchange: the first step toward international collaboration. In: AI-Youbi, A., Zahed, A., Tierney, W. (eds.). Successful Global Collaborations in Higher Education Institutions. Springer, Cham, pp. 63–75 (2020). https://doi.org/10.1007/978-3-030-25525-1_7 13. Gay, G., Kirkland, K.: Developing cultural critical consciousness and self-reflection in preservice teacher education. Theor. Pract. 42(3), 181–187 (2003)

New Perspectives for Internationalization in Higher Education …

289

14. Chiang, C.L., Lee, H.: The effect of project-based learning on learning motivation and problemsolving ability of vocational high school students. Int. J. Inf. Educ. Technol. 6(9), 709–712 (2016). https://doi.org/10.7763/IJIET.2016.V6.779 15. Md Hashim, A., Syed Aris, S.R., Chan, Y.F.: Promoting empathy using design thinking in project-based learning and as a classroom culture. AJUE 15(3), 14–23 (2020) 16. Selçuk, M.: Learning in encounter: Crossroads, connections, collaborations. Religious Educ. 113(3), 233–243 (2018) 17. Liu, S.: Student interaction experiences in distance learning courses a phenomenological study. J. Distance Learn. Admin. 11(1) (2008). Accessed 10 Jan 2023. https://www.learntechlib.org/ p/158563/

Student Satisfaction and Graduation Rates in Finnish Master of Engineering Programs Matti Koivisto

Abstract During the recent years, many stakeholders like researchers, policymakers and even students have paid more attention to the efficiency and performance of higher education. Although no single indicator is sufficient to describe the organizational quality of the university, graduation rate has become one of the key indicators of university success. There are clearly many factors that affect students’ ability and willingness to complete their studies successfully, but previous studies have clearly shown that high student satisfaction has a positive impact on the graduation rate. The focus of this paper is on the part-time Master of Engineering programs of a Finnish university of applied sciences (UAS), and the empirical part of the study examines the relationship between student satisfaction and graduation rates based on the internal graduation records of the UAS and the data from the Finnish national student satisfaction survey (AVOP). The findings of the study suggest that student satisfaction is a multidimensional structure, and satisfaction with the content of the studies and the thesis has a statistically significant impact on graduation from the Finnish Master of Engineering programs. Keywords Student Satisfaction · Graduation · Dropout Rate

1 Introduction The educational mission of the university is to provide the highest education based on research and the requirements of the society [1]. In order to meet these expectations and provide a high-quality education, higher educational institutions must focus their limited resources on key activities. The institutional performance of the university can be assessed in several ways, but two most frequently used methods are student satisfaction and graduation. By studying how satisfied students are with their educational experiences, colleges can identify both their well performing functions M. Koivisto (B) South-Eastern Finland University of Applied Sciences, Patteristonkatu 2, 50100 Mikkeli, Finland e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_23

291

292

M. Koivisto

and areas requiring improvement [2]. On the other hand, a high graduation rate can give a positive image about the academic and administrative states of the university [3], and it can be seen as positive indicator of institutional performance [2]. Although scholars have paid a lot of attention both on student satisfaction and dropout rates at the undergraduate level, this topic is far less studied at the master level [4]. The empirical part of the study examines the student satisfaction and its impact on the graduation rates in Finnish Master of Engineering programs. More specifically, the aim is to find answers to the following research questions: • Which areas of education have the greatest impact on general student satisfaction? • Should we focus on general student satisfaction, or should student satisfaction be examined as a multi-dimensional structure? • Can we identify some areas of education which have a significant impact on the graduation rates? The first research question follows the tradition of earlier student satisfaction studies, and it aims at finding out the most important institutional factors affecting student satisfaction in a Finnish UAS. With the other two questions, we aim to find out first whether we should utilize one global or many area-specific items when linking student satisfaction and graduation rates. Second, we try to identify the areas that have the most significant links between student satisfaction and high graduation rates. The structure of the paper is as follows. Section 2 provides a short literature review on student satisfaction, graduation, and dropout rates. The study design and the aim of the empirical part of the study are presented in Sect. 3. The experiment focuses on the part-time Master of Engineering programs of a Finnish university of applied sciences (UAS). All students in these programs are adults with several years of work experience, and they have returned to college for further education. In our study, we combine the data from two different sources: from the internal graduation records of the UAS and from the Finnish national student satisfaction survey (AVOP). The AVOP survey measures graduating students’ satisfaction with different aspects of education, such as study content, teaching, learning environment, and support services. The results of the study are discussed in Sect. 4 and, finally, in Sect. 5, a few concluding thoughts and suggested directions for future research are provided.

2 Literature Review Universities all over the world are increasingly recognizing that higher education can be seen as a service industry [5]. Thus, they are placing greater emphasis on meeting the expectations and needs of different stakeholders like students, administrators, and policymakers. In this chapter, this topic is shortly reviewed from two perspectives: student satisfaction and graduation or dropout rates.

Student Satisfaction and Graduation Rates in Finnish Master …

293

2.1 Student Satisfaction From the students’ point of view, the key objective for higher education institutions is to provide good educational services [1]. Although there are several methods of evaluating the quality of education, student satisfaction is probably the most popular metric. Scholars have defined student satisfaction in many ways. For example, it has been seen as “an assessment of student outcomes and experiences in education and life on campus” [6] or as “the students’ perception of graduate educational experiences and values received when enrolling in a training institution” [1]. Scholars have tried to analyze the key elements of student satisfaction at higher education in different ways. For example, Shonfeld [7] recognized the following three components: instructors, quality, and teaching methods. On the other hand, Alnawas [8] identified the following factors: organization and management, quality of teaching, personal development, assessment and feedback, learning resources and academic support. Trang et al. [1] instead analyzed post-graduate students’ satisfaction from five different perspectives, and they found out that tuition fees, serviceability, and lecturers had statistically significant impact on satisfaction.

2.2 Student Dropout and Graduation Rates During the recent years, policymakers and government officials have paid more attention to the quality and accountability of the academic institutions, and they are looking at student outcomes as a measure of the quality of a university. Although student outcomes can be measured in many ways, graduation or dropout rates have received most of the attention [9]. High non-completion rates in higher education are a global phenomenon. On average across OECD countries only 39% of fulltime students in bachelor’s programs graduate within the theoretical duration of the program, and after an additional three years, the average completion rate increases to 68% [10]. These relatively low graduation rates sharply contrast with the social and economic objectives that have been formulated to achieve sustainable economic growth [11]. Therefore, student dropout is a serious problem, and it has several negative impacts on the well-being of the students and the community in general. For academic institutions, it leads to monetary losses either in the terms of missed tuitions fees or government funding [12]. For decades, scholars have tried to identify the variables that influence students’ graduation and have created different kinds of predictive models. Perhaps the most cited publication is Tinto’s [13] groundbreaking study, which postulates quitting as a long-term interaction process between a student and a university. Students’ experiences both in academic and social interaction affect their goals and commitments and decisions on continuation or termination of studies. The major finding of the study was that institutions, where students had better interaction with the academic and

294

M. Koivisto

social environment, had higher graduation rates. Other researchers have developed these ideas further. For example, the Input-Environment-Output model was based on the following three components: inputs (student’s background and experiences before entering the university), environment (student’s experiences during college), and outputs (student’s outcomes and performance at college) [14]. Fung [15], on the other hand, analyzed retention and graduation rates using two student related characteristics (student background and finance) and two environmental variables (academic and social environment).

2.3 Linking Student Satisfaction and Graduation Rates Earlier studies have indicated that student satisfaction has a positive effect on the following key institutional metrics: individual student retention [16], institutional graduation rates [2], and institutional alumni giving rates [17]. When analyzing the relationship between student satisfaction and graduation or dropout rates, scholars have used different approaches. Starr et al. [18], for example, applied the theory of work adjustment, which uses a general satisfaction component together with satisfactoriness metric. The theory suggests that not to drop out from the college, a student must fulfill the requirements of that environment (perform satisfactorily) and the college environment must meet the needs of the student (lead to satisfaction). Suhre et al. [19] focused on the impact of degree program satisfaction on students’ views on dropping out and Mostert and Pienaar [20] investigated the links between student burnout, intention to drop out, and satisfaction. All these studies and many others have supported a widespread belief that student satisfaction and graduation have a positive relationship as the satisfaction indicators significantly add our ability to predict student retention [21] and thus it can increase institutional graduation rates [2]. Models that combine the degree of student satisfaction to graduation rates can be applied in many ways. For example, it can be used in institutional performance evaluations or benchmarking analysis [15]. In addition, they can offer valuable information for resource allocation. In current economic climate academic institutions must focus their limited resources on those areas which have the greatest impact on their key objectives.

Student Satisfaction and Graduation Rates in Finnish Master …

295

Table 1 Number of graduates, graduation and admission rates of the degree programs Degree Program

N

Graduation rate (%)

Admission rate (%)

DG1

27

63.6

73.3

DG2

78

79.1

53,1

DG3

42

70.8

64.0

DG4

21

44.4

75.0

DG5

51

66.7

72.9

DG6

60

64.2

60.9

DG7

60

75.5

77.8

3 Study Design 3.1 Target Programs and Graduation Rates As mentioned earlier, the empirical part of the research examines the student satisfaction and its impact on the graduation rates in Finnish Master of Engineering programs. The sample of the study consists of six part-time Master of Engineering programs and one Master of Natural Resources program of a Finnish university of applied sciences (UAS). The indicative duration of the analyzed degree programs is two years, but each student is automatically granted a three-year right of study. If the student has not completed his or her studies within three years, he or she may apply for a discretionary extension of up to one year. It is also worth mentioning that there are no tuition fees in these programs for EU/EAA students. All students in these programs are adults with several years of work experience and they have returned to college for further education. Our sample contained all students who graduated from those programs in 2015–2018. The total number of graduated students was 339, and they earned their degree in seven different degree programs (DG1-DG7). Table 1 shows the number of graduates in each program together with graduation and admission rates. Although those degree programs that are easier to access (higher acceptance rates) have a slightly lower degree of graduation, the difference is not statistically significant at 0.05 level.

3.2 Satisfaction Data The student satisfaction data were retrieved from the national AVOP survey database. The AVOP survey is co-organized by Statistics Finland, the Ministry of Culture and Education and the Finnish National Agency for Education, and all students graduating from the Finnish universities of applied sciences participate in it. To ensure the privacy of respondents, it is not possible to retrieve an individual student’s answers from the AVOP database, but the results are summarized. The annual summary of the degree

296 Table 2 Question groups and the number of questions in each group

M. Koivisto

Question group

Number of questions

Study content

3

Planning studies, counselling

2

Teaching

3

Studying

2

Learning environments

3

Support services

3

Feedback and assessment

3

Internationality and multiculturalism

2

Connections with the working life

2

Career services

2

Thesis

2

General satisfaction

1

program includes the number of respondents and the mean values for each question. If there are fewer than five respondents per year, results for that year is not presented. The master’s degree version of the AVOP survey measures student satisfaction from eleven specific and one general perspectives, as can be seen in Table 2.

4 Results and Discussion The results of the study are reported below in two phases. First, we concentrate on satisfaction and the links between areas specific and the general student satisfaction. Second, we focus on the relationship between satisfaction and graduation rates.

4.1 Links between Specific and General Satisfaction As mentioned earlier, scholars have created different kinds of models to describe which factors lead to general or overall student satisfaction. Using the same logic, the model shown in Fig. 1 was created. In this model, the general student satisfaction is a result of 11 factors or question groups used in the AVOP survey. The correlation coefficient and the associated p-values (the probability that the correlation coefficient was in fact zero) between the factors of the model and general satisfaction are listed in Table 3. The results indicate that seven areas (study content, planning studies and counselling, teaching, studying, feedback and assessment, career services, and thesis) have a statistically significant effect on general student satisfaction at 0.01 level and one area (learning environments) at 0.05 level. Three areas (support services, internationality and multiculturalism, and connections

Student Satisfaction and Graduation Rates in Finnish Master …

297

Fig. 1 Model based on AVOP survey question groups

Table 3 Correlations between general and area specific student satisfactions Question group

Correlation

p-value (probability)

Study content

0.91 **

0.004

Planning studies, counselling

0.93 **

0.002

Teaching

0.95 **

0.001

Studying

0.93 **

0.002

Learning environments

0.87 *

0.011

Support services

0.62

0.136

Feedback and assessment

0.90 **

0.005

Internationality and multiculturalism

0.14

0.767

Connections with the working life

0.65

0.116

Career services

0.89 **

0.008

Thesis

0.93 **

0.002

**

p < 0.01 and * p < 0.05

with the working life) instead do not have statistically significant effect on student satisfaction.

4.2 Links between Satisfaction and Graduation Rate The correlation between satisfaction on graduation rates was analyzed from two different perspectives. First, we wanted to study the relationship between the general satisfaction and graduation, and the results (r = 0.71 and p-value = 0.07) suggested that there is no statistically significant correlation between them. Second, we shifted our focus from general to area specific satisfaction ratings. Table 4 contains the correlations between the question groups and the graduation rates.

298

M. Koivisto

Table 4 Correlations between graduation and area specific student satisfactions Question group

Correlation

p-value (probability)

Study content

0.80 *

0.03

Planning studies, counselling

0.65

0.11

Teaching

0.72

0.07

Studying

0.72

0.07

Learning environments

0.65

0.11

Support services

0.36

0.42

Feedback and assessment

0.47

0.28

Internationality and multiculturalism

-0.16

0.74

Connections with the working life

0.32

0.49

Career services

0.69

0.09

Thesis

0.79 *

0.03

General satisfaction

0.71

0.07

* p < 0.05

As can be seen from Table 4, there were two question groups (study content and thesis) that were highly correlated with the graduation rate, and the correlations are statistically significant at 0.05 level. Although teaching, studying and general satisfaction indicated strong correlations with a graduation rate (r > 0.7), we are not able to reject the null hypothesis because the p-value was over 0.05 in all three of them.

4.3 Discussion The results of the study suggest that the general student satisfaction or satisfaction with education as a whole is influenced by several institutional factors. This result is consistent with earlier findings [1, 7, 8, 22, 23]. On the other hand, according to our findings, the higher overall satisfaction does not directly lead to lower dropout rates, as our data did not indicate a significant correlation between them. Our results are also in line with previous studies that have highlighted the importance of using a number of indicators for different aspects of study satisfaction rather than a single measure of overall satisfaction [21]. When we followed this approach, we identified two factors (study content and thesis) that had a statistically significant impact on the degree of graduation.

Student Satisfaction and Graduation Rates in Finnish Master …

299

5 Conclusions This paper continues the substantial literature that examines student satisfaction and the determinants of academic performance. Previous research has analyzed the topic from many different views like student, social, financial, and institutional perspectives. In this study, the focus was limited only on how satisfied students were on different institutional factors of the UAS. The results of study indicated that student satisfaction is a multi-dimensional structure and for Finnish Master of Engineering graduates the most important factors affecting their satisfaction were: study content, planning studies and counselling, teaching, studying, feedback and assessment, career services, and thesis. This exploratory study has its limitations. First, the sample used was limited only to Master of Engineering degree programs at one UAS in Finland. In the future, the study should be extended both to other disciplines and educational institutions. Second, when comparing our results against other similar studies, it is important to bear in mind that there are no tuition fees in Finnish universities for EU/EAA citizens. Therefore, the results should not be generalized to the countries where high tuition fees have been identified as an important factor affecting both student satisfaction and graduation. Finally, the study does not necessarily imply causations but only correlations. However, the results suggest that there are measurable differences in student satisfaction that are related to general satisfaction and even further to the graduation rates. At this point, it is too early to make conclusions, and more research is still needed for better understand the factors promoting the academic performance and graduation of the students. Acknowledgements I would like to acknowledge Statistics Finland, the Ministry of Culture and Education and the Finnish National Agency for Education for collecting and sharing the AVOP data for research purposes.

References 1. Tran, T., Kha, G., Duyen, T., Linh, T.: Research on factors affecting the postgraduate students’ satisfaction in the quality of training services in accounting at the training institutions in Hanoi. Am. J. Educ. Res. 6(5), 512–518 (2018) 2. Bryant, J., Bodfish, S.: The Relationship of Student Satisfaction to Key Indicators for Colleges and Universities. National Research Report (2014) 3. Aljohani, O.: A review of the contemporary international literature on student retention in higher education. Int. J. Educ. Literacy Stud. 4(1), 40–52 (2016) 4. Rotem, N., Yair, G., Shustak, E.: Dropping out of master’s degrees: objective predictors and subjective reasons. High. Educ. Res. Dev. 40(5), 1070–1084 (2021) 5. DeShields, O., Kara, A., Kaynak, E.: Determinants of business student satisfaction and retention in higher education: applying Herzberg’s two-factor theory. Int. J. Educ. Manage. 19(2), 128– 139 (2005)

300

M. Koivisto

6. Avero, T., Tricahyono, D.: The relationship between collaboration factors, teamwork satisfaction, and student satisfaction: a conceptual paper. In: Sustainable Future: Trends, Strategies and Development – Noviaristanti & Hway Boon (eds). (2023) 7. Shonfeld, M.: Factors affecting student-teacher satisfaction with a multi-college online collaborative course. Australas. J. Educ. Technol. 37(6), 193–205 (2021) 8. Alnawas, I.: Student orientation in higher education: development of the construct. High. Educ. 69(4), 625–652 (2015) 9. Cook, B., Pullaro, N.: College Graduation Rates: Behind the Numbers. American Council on Education (2010). https://www.acenet.edu/Documents/College-Graduation-Rates-Behindthe-Numbers.pdf 10. OECD: Education at Glance 2022: OECD Indicators. OECD Publications (2022). 11. De Witte, K., Cabus, S., Thyssen, G., Groot, W., Maassen van den Brink, H.: A critical review of the literature on school dropout. Educ. Res. Rev. 10, 13–28 (2013) 12. Olaya, D., Vásquez, J., Maldonado, S., Miranda, J., Verbeke, W.: Uplift Modeling for preventing student dropout in higher education. Decis. Support Syst. 134, 113320 (2020) 13. Tinto, V.: Dropout from higher education: a theoretical synthesis of recent research. Rev. Educ. Res. 45(1), 89–125 (1975) 14. Astin, A.: Assessment for Excellence. Macmillan, New York (1991) 15. Fung, T.: Analysis of Graduation Rates for Four-Year Colleges: A Model of Institutional Performance using IPEDS, Doctoral dissertation, University of North Texas (2010) 16. Miller, K.: Predicting Student Retention at Community Colleges. Ruffalo Noel Levitz, Cedar Rapids (2015) 17. Bryant, J., Bodfish, S., Stever, D.: The Correlation between College Student Satisfaction and Alumni giving. Ruffalo Noel Levitz, Cedar Rapids (2015) 18. Starr, A., Betz, E., Menke, J.: Differences in college student satisfaction: academic dropouts, nonacademic dropouts, and nondropouts. J. Couns. Psychol. 19(4), 318–322 (1972) 19. Suhre, C., Jansen, E., Harskamp, E.: Impact of degree program satisfaction on the persistence of college students. High. Educ. 54, 207–226 (2007) 20. Mosert, K., Pienaar, J.: The moderating effect of social support on the relationship between burnout, intention to drop out, and satisfaction with studies of first-year university students. J. Psychol. Africa. 30(3), 197–202 (2020) 21. Schreiner, L.: Linking Student Satisfaction and Retention. Ruffalo Noel Levitz, Cedar Rapids (2009) 22. Hwang, Y., Choi, Y.: Higher education service quality and student satisfaction, institutional image, and behavioral intention. Soc. Behav. Pers. 47(2), 1–12 (2019) 23. Hew, K.F., Hu, X., Chen, Q., Tang, Y.: What predicts student satisfaction with MOOCs: a gradient boosting trees supervised machine learning and sentiment analysis approach. Comput. Educ. 145, 103724 (2020)

Ensuring Optimal Performance in Online Learning of STEM Subjects: An Autoethnographic Study Gopala Krishna Koneru

Abstract Learning is a process of assimilating facts or knowledge; making inferences from observations, experiments or experiences of the real world; and reflecting on them to build a coherent mental model of understanding of the subject. An ability to abstract from specific knowledge to form insights and also gain the expertise to realize practical implementations from abstracted models are the essential traits of a learned engineer or scientist. I have had nearly 30 years of experience of working in technology organizations, as well as a Learner-Teacher in a university teaching working professionals for the past 12 years. As part of this autoethnographic experiment, starting afresh as a student to learn new subjects which are different from my earlier academic credentials or past profession, I have accumulated several insights on the way I experienced learning along with the roadblocks to my understanding of the subjects. While a Teacher can afford to be a specialist or expert in one discipline of knowledge or the other, a Student on the other hand, would have to assimilate different courses being delivered in a disjointed manner by individual instructors in their own scheduled periods. Being empathetic to students’ cognitive barriers and delivering instruction as a symphony or synergistic portfolio of courses aligned to their mental bandwidth in a progressive bite-sized chunks of instruction is the key to effective assimilation of knowledge. By listening to the voices and noises in my head while studying new courses and analyzing my subject notes and diaries over the last 12 years of Learning-Teaching of electrical engineering and computer science courses, this paper highlights key observations and recommendations for effective instruction in STEM education based on my proposed SLATE (Structured Learning and Thinking Enablement) framework and Cognitive Load Theory (CLT). Whatever measures, observations and recommendations proposed in this lone experiment-onmyself are neither well-defined, conclusive or comprehensive. They’re mere thought triggers. A detailed study with formal hypothesis, survey and analysis of real students learning in the classroom will be continued by the author.

G. K. Koneru (B) WILP Division, BITS-Pilani University, Hyderabad, TS 500078, India e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_24

301

302

G. K. Koneru

Keywords STEM · Higher-Order Learning · Cognitive Load · Schema-based Learning · Cognitive Constructs · Instruction · Higher Education · Professional Education · Life-long Learning · Structured Learning

1 Introduction In today’s university education system, a typical student has to opt for a minimum of three to four courses in a semester. Often times, these courses are diverse in their demands placed on cognitive load on the part of the students. A typical coursemix in a semester consisting of courses in Science, Technology, Engineering and Mathematics (STEM) which would often require students to have prior-knowledge in the form of other related pre-requisite courses along with strong mathematical abstraction skills. While individual instructors deliver their respective courses in a discipline as per structured syllabus and schedule, it is the student who has to assimilate the content from these multiple instructors so as to form a coherent and integrated knowledge base ready to be applied to solve real-world problems which are often multidisciplinary in nature. Students come with varied cognitive skills such as learnability, mathematical thinking or abstraction skills, prior knowledge or experience in the subject domain, and also diverse socio-economic backgrounds that may have limited their opportunities for access to quality learning materials or instruction in their earlier academic years. Moreover, lack of access to live class instruction in these COVID-19 pandemic years denied student the opportunities for live tutoring and peer-to-peer learning typical in a campus environment.

1.1 Profile of Student in This Study Typical profile chosen for this study includes working professional in Indian IT industry with an average experience of 20 years who would like to enhance his knowledge and skills in the emerging areas of technology such as Digital Communications, Data Science, Internet of Things, and Artificial Intelligence. Many students of this profile would have had their under-graduate degrees in totally different discipline or their working experience or expertise not connected with the areas they have currently chosen for their professional education. For the current experiment, fitting into this profile, myself (author) has been chosen as the subject of this autoethnographic study. The author — who was a Mechanical Engineering graduate in the early’80 s, spent nearly 25 years of his early professional career in Software Project Management — pursues upgradation of his knowledge in the contemporary areas of technology described above. For this, a MOOC (Massive Open Online Course) platform was chosen which offered certification courses taught by faculty from India’s premier technology institutes. However, the observations and conclusions

Ensuring Optimal Performance in Online Learning of STEM Subjects …

303

derived from this experiment may be subjective and need further refinement by formal research before generalizing for wide applicability across various learning environments — off-campus and on-campus as well as non-STEM subjects.

1.2 Why MOOCs? With the advent of MOOCs and free access to wide choice of audio-visual content on the web along with assessment via online and proctored examinations, online course delivery emerged as the primary choice of pedagogy for effective scaling of quality instruction and assessment at affordable prices to students across geographies. This study is undertaken as that of lone student studying STEM courses in an online mode on NPTEL [1] — India’s largest MOOC platform with courses delivered by Indian Institutes of Technology (IIT) considered as premier technology institutes in India.

2 Autoethnography as a Qualitative Research Method Learning is intrinsically a subjective human experience that enhances human intellect and problem-solving skills. Apart from the role of passive instruction resources, learning is a social construct and it emerges as desired behaviour in an ecosystem of teachers, students and one’s living environment. Use of any explicit objective methodology to capture data in this innate subjective process may not be complete as it leaves out several important unvoiced data elements. Autoethnography [6] is a qualitative research method in which the author immerses himself in the subject of study and uses self-reflection and writing to explore his personal experiences and stories for broader understanding and sense-making. For the purposes of this immersive study, the author has chosen over 25 courses [8] for self-study in online mode during the COVID-19 pandemic years of 2019 to 2022. These courses are grouped under the broad categories of Technology, Mathematics, and Management.

2.1 Modelling as ‘Middle Bencher’ The phrase ‘Middle Bencher’ is used an epithet to indicate the customary student who neither gets recognized by his teachers for his cleverness unlike the studious ‘front benchers’ nor playful ‘last benchers.’ Middle bencher is stereotypical of a student who is ambitious and yet confused, lacks self-confidence in his knowledge to proactively standup and ask questions, and is willing to work hard but needs guidance or little tutoring to catchup with his front bench counterparts.

304

G. K. Koneru

Throughout his academic and professional career, the author considers himself as that Middle bencher who barely tries to swim above the water following the crowd. The courses chosen were all unrelated to his past academic credentials and current work or teaching assignments, and therefore demand requisite attention and effort as that of a young university student enrolling in those courses for the first time.

2.2 Effort and Resources The lessons for the courses are delivered every week with varying styles ranging from pre-recorded videos to lecture materials in the form of PDFs and weekly assignments with deadlines. All the videos (with varying durations for each course, i.e. weekly video-hours to be watched) are watched in normal mode making extensive notes all along. Assignments are worked out and submitted before deadlines. The following data on courses is culled from NPTEL course site along with the effort spent by the author as noted in his time-stamped notes pages along with performance in each of the courses completed during the Covid period (2020–’22).

2.3 Cognitive Load Factor (CLF) and Instruction Load Factor (ILF) These two factors are chosen to be the course parameters affecting students’ learning. Cognitive Load factor [3] for any course is measured as the ratio of cognitive task effort demanded in terms of number of individual key concepts to be mastered in a subject to the available resources in terms of expected hours for study, assignment practice or review and also required external resources such as reference materials, tutorial hours, prerequisite courses etc. to comprehend those concepts covered during the course. In addition to the course’s intrinsic load, ‘Instruction Load’ factor indicates additional effort imposed by the quality of instruction or ‘student-friendliness’ of the instructor. On average, four courses have been chosen for study in a semester of 3–4 months with relative assessment of the course’s cognitive load and instruction load. In the absence of precise definition of these two course parameters for formal measurement, a subjective self-assessment (Table 1) by the author (L-Low, MMedium, H-High) is adopted in this empirical study. Hence, the analysis of this experiment is purely drawn out of subjective self-evaluation which needs to be corroborated by formal research and quantitative analysis which will be pursued by the author in the coming year.

Ensuring Optimal Performance in Online Learning of STEM Subjects …

305

Table 1 Mix of courses along with their course parameters and performance by the student (arranged by the discipline group—Management/Professional, Technical) Month / Course Name Year

Credit Duration Video-Hours Instruction Intrinsic Marks Points (weeks) Factor Load %

Nov’22

New Product Development (NPD)

2

6

5

L

L

60*

Nov’22

Managing Innovation (MI)

2

6

5

L

L

75*

Nov’22

Introduction to Marketing Essentials (IME)

2

6

5

L

L

65*

Nov’22

Innovation and IT Management (ITM)

2

6

6

L

L

70*

Nov’22

Business Environment (BE)

2

8

8

L

L

60*

Sep’21

Innovation by Design (IBD)

1

4

12

M

L

58*

Oct’22

Introduction to Wireless and Cellular Communication (IWC)

3

12

60

H

H

57

Oct’22

Digital Image Processing (DIP)

3

12

30

M

H

66

Oct’22

Probability Foundation for Electrical Engineers (PEE)

3

12

48

H

H

22

Oct’21

Probability for Computer Science (PCS)

2

8

16

H

H

42

Sep’22

Principles of Communication Systems-II (PCS)

2

8

42

M

H

67

Mar’19

Data Science for Engineers (DSE)

2

8

24

M

H

70

Apr’22

Introduction to Machine Learning (IML)

3

12

48

M

H

46

Apr’22

Introduction to Artificial Intelligence (IAI)

3

12

42

M

H

48

Apr’22

Reinforcement Learning (RL)

3

12

36

H

H

48 (continued)

306

G. K. Koneru

Table 1 (continued) Month / Course Name Year

Credit Duration Video-Hours Instruction Intrinsic Marks Points (weeks) Factor Load %

Feb’22

Python for Data Science (PDS)

1

4

16

M

M

70

Oct’21

Deep Learning (DL)

3

12

30

H

H

76

Sep’22

Design for Internet 2 of Things (DIT)

8

32

M

M

76

Apr’21

Patent Law for Engineers and Scientists (PES)

3

12

24

M

L

64

Apr’21

IP Practical aspects—Patent drafting (IPD)

1

1

6

M

L

82

Aug’22

Managing IP in Universities (MIP)

1

4

8

M

L

81

Feb’22

Patent Drafting for 1 Beginners (PDB)

4

6

M

L

76

Oct’22

Patent Search and Analysis (PSA)

2

8

20

M

L

69

Feb’21

Towards an Ethical 1 Digital Society: From Theory to Practice (TED)

4

12

L

L

79

Feb’21

Introduction to Professional Scientific Communication (IPC)

1

4

8

L

L

78

Nov’20

Ethics in Engineering Practice (EEP)

2

8

20

M

L

71

Dec’20

Introduction to 3 Japanese Language and Culture (IJL)

12

36

H

M

79

Apr’21

Qualitative Research Methods and Research Writing (QRM)

3

12

30

M

M

64

Oct’20

Teaching and Learning in General Programs (TAL)

1

4

10

M

M

73

(continued)

Ensuring Optimal Performance in Online Learning of STEM Subjects …

307

Table 1 (continued) Month / Course Name Year

Credit Duration Video-Hours Instruction Intrinsic Marks Points (weeks) Factor Load %

Oct’20

Designing Learner-centric e-learning in STEM disciplines (DLS)

1

4

6

M

M

87

Sep’21

Accreditation and Outcome Based Learning (AOL)

2

8

20

M

M

67

Feb’20

Effective Engineering Teaching in Practice (ETP)

1

4

8

M

L

79

*:

Marks not received yet, estimated percentage. Instruction Load Factor: Low (L)/Medium (M)/ High (H) — indicates relative impact of Instruction/Instructor-style on the cognitive load for the student; Intrinsic Load: Required technical/analytical depth (relative) of the course which translates to number of study hours to be spent by the student post watching of instruction videos; Video Hours: Total number of hours of video to be watched for the entire course; Duration: Calendar period of study when the course material is made available online; Credit Points: Weightage assigned for the course which translates to academic credits earned by the student

3 Learning Dissected — Meta-Cognitive Constructs for Self-Learning and Problem-Solving Skills Online learning via MOOCs or otherwise is self-directed and self-motivated in the sense that without the support of personal live tutor the learner has to have the right mental scaffolding to make sense of knowledge that is being presented. Constructivist theory [5] proposes learning as a process of co-creation of knowledge structures in mind as the student interacts with the real world to solve problems. Learning is progressive and in class room its pace can be accelerated with simulated experiential learning via experiments, case-studies, challenge-problems etc. The motivation for this experiment is author’s earlier work on arriving at a single schema for learning of STEM subjects. For qualitative assessment of course parameter CLF, the author has chosen SLATE framework [4] which identifies the role of metacognitive constructs for effective learning. This cognitive framework, also known as ‘schema’ in the literature [3] has been arrived over more than two decades of his experience as working professional executing customers’ projects with on-the-job learning and retrospective analysis of projects’ performance. SLATE framework identifies the below ‘Learning Primitives’ for STEM subjects and these must serve as mental scaffolding for students to acquire knowledge as well as guide curriculum design and instruction delivery by the teachers. These primitives can also be mapped to the corresponding levels in the Bloom’s Taxonomy.

308

G. K. Koneru

Abstraction — Realization. Ability to generalize what one learns in the class-room and from his own experiences in the real world is the key construct to put him on the path of self-learning; Abstraction along with ‘Realization’ which takes an abstracted entity to realize it in the real world are the key cognitive constructs for learning as well as higher-order thinking and problem-solving skills. For example, if one is currently learning about Internal Combustion (IC) Engine of an automobile, a learner should be able to abstract the IC engine as a prime mover— one of several concepts or mechanisms that make objects move by themselves with examples such as steam engine, diesel engine, nuclear-powered engine, electric traction motor, etc. Realization is the inverse of Abstraction where upon working on the conceptual/abstract treatment of the subject, the learner derives a Realization which can be practically implemented in the real world. A learner can take comfort in the fact that there’re only few such concepts he/she has to master in any discipline and each such concept spawns most Realizations in the real world. Systems Thinking. Ability to think everything in terms of systems, sub-systems, elements and their interfaces and interactions helps student to abstract (hide) complexity and learn its behaviour by its interaction with other sub-systems; the simplest way to teach this is to hide complexity of any topic by enclosing it in a black-box; this approach is widely used in technology and engineering disciplines to introduce new ideas with emphasis on their application and relevance in the real world. This metacognitive skill enables a learner to view any topic or subject of study to be part of greater system in which it currently coexists. For example, an IC engine is an element of subsystem of an automobile system which is again a part of greater transportation system which includes automobiles, road infrastructure, and traffic signaling subsystems etc. Analysis – Synthesis. In most of the courses in this experiment, the author found undue emphasis given to ‘analysis’ wherein the topic or subject under study is broken up into a hierarchical structure and then each covered in a level of detail that makes student miss the forest for the tree. ‘Synthesis’ involves building up or engineering systems from available sub-systems or components to deliver useful functions, and analysis-synthesis is to be treated as a continuum as it allows student not only to make sense of knowing the sub-systems of systems but also fosters creative synthesis of existing sub-systems to design new products/services. Considering the example of an automobile system, analysis involves a hierarchical breakup of a car into IC engine, chassis, suspension, electrical system, airconditioning system, and entertainment system. Synthesis is an engineering activity where one builds up or synthesizes a desired system of car which involves interaction of the above several subsystems. Evolution of Systems. Once students are tuned to systems thinking, a kind of mental abstraction to make sense of things, they can imagine new systems or how current systems can evolve over a period of a time in terms of their functions or behaviour; a system can be an interaction of technology, people, and social patterns. Thinking in

Ensuring Optimal Performance in Online Learning of STEM Subjects …

309

terms of systems is an essential skill for inventive problem-solving in today’s digital world. Driven by one’s imagination, this metacognitive skill enables learners to foresee future systems based on their past history and evolution as well changing social and behavioral patterns of customers. Considering the example of automobile system, this would enable one to visualize how customers in future would commute if at all they do or what kind of new transportation system which is radically different from the current one would emerge. Trained as a skill, this enables a learner to foster predictive insights which help pioneer game-changing innovations. Technology Trending. Learning about the current state of the art is one thing and following how the technology is evolving over a period is what differentiates just knowledge-seekers from trend-setters or leaders; students are to be exposed not only to the current technology but also how it would be transforming in the future by identifying trends and stretching their imagination from the current to the most-likely future. For example, in the case of automobiles, this Trending skill would help identify directions in which the current automobile technology is evolving — different engines, alternative fuels, interior design of spaces, connected cars, and digital transmissions.

4 Analysis of the Experiment (Autoethnography) For qualitative assessment of course parameter CLF, the author has chosen the above SLATE framework which identifies the role of metacognitive constructs for effective learning. Below are some of the observations and inferences drawn from my own experiment as working-professional-student registered in a program to study new courses in STEM as well as management subjects for his professional development. • The amount of time a student need to spend (often related to ‘course credits’) for effective learning is not related to the amount of content or hours of live instruction or video-hours one has to read or watch, but on the intrinsic cognitive load factor and instruction factor of the course; for example, for achieving the same performance, highly analytical courses such as Probability Theory, Wireless Communications, etc., involving several mathematical abstractions took nearly 70% of author’s time compared to descriptive, fact-based or case-study type of courses such as those in management domain. • The ‘myth’ of prerequisite courses — when designated as ‘must-have’ prerequisites for a course and are not part of student’s daily work-life or studied by the student years ago, they increase the cognitive load for the student as additional effort to catchup with the instructor.

310

G. K. Koneru

• Academic style of instruction — many a time, a syllabus prescribed in a course handout resembles linear chronological structure of the developments in that technology/area or simply reflects the table-of-contents of the prescribed text-book; syllabi centered on problem-based learning or challenge-questions situated in students’ context with progressive disclosure of thinking and developments in that subject area would enhance stickiness. • Disjointed course-mix — courses in a typical semester are chosen purely based on academic requirements imposed by the university or regulator and rarely on the homogeneity of courses in terms of relatedness of topics or mutual dependency; in the present experiment, author has deliberately chosen a personalized coursemix from the MOOCs with varying cognitive load to match his cognitive comfort and available resources, which may not be possible in a typical semester-driven university system.

5 Recommended Practices for Effective Instruction Based on my interactions with students and my experience as an industry professional and as Teacher-Student in this experiment, here are my recommendations for effective instruction and learning.

5.1 Drive Curiosity, not Curriculum Excessive emphasis on ‘covering the syllabus’ as prescribed should give way to creating excitement and curiosity about the subject by focusing on the challenging problems it attempts to solve; live instruction plays an important role in engaging students to stimulate them for continuous learning beyond classroom; content structured around active problem-based learning with problems centered around student’s social context engages students more effectively than driving passive content modeled around academic text-books.

5.2 Empathetic Course-Mix Every classroom is unique in terms of student profiles and each course is to be structured around topics that connect with them; courses are not be treated as isolated entities being delivered independently by respective instructors. Taking cognizance of student profiles in each program of study, using the right combination of courses based on CLF, and connecting with each other delivers a unified learning experience; it demands fluidity in planning and delivery of instruction for each batch of students

Ensuring Optimal Performance in Online Learning of STEM Subjects …

311

in a program. In an online MOOC environment, this role is relegated to students themselves who can chose their own courses, time, place and pace of learning.

5.3 Instructor Teaming Learning is an experience with several cognitive roadblocks and ‘aha’ moments in its path when courses are being delivered by instructors. In medieval times, there were no specific disciplines, and most sciences were grouped under ‘natural philosophy.’ Today, with plethora of disciplines and sub-disciplines, learning has become fragmented with individual instructors focusing on their select courses delivering them as per their specific course content, text-books, delivery and preferences in their allotted time and space. This compartmentalization of course delivery exacts a toll on the part of the student who is expected to integrate knowledge across courses to form a cohesive wholesome learning experience. Often times, even related courses are delivered in parallel without common pool of examples, terminology and stories. If all the courses in a semester are chosen in such a way that they are delivered as a team with a common story (content) with individual instructors playing their respective role as directed by program director, it would be an integrative learning experience that sound music to students’ ears.

5.4 Telescopic Treatment of Course Content With distracting digital devices and always connected social media, student engagement has become a major challenge for teachers to gain their attention. A typical course delivery takes around 40–50 h of instruction spanning several weeks in a semester. Often times, it is observed that student attendance and engagement is high during the initial weeks and slowly wanes as it progresses with only few selfmotivated students (‘front benchers’) stay put till the end. Taking cue from the above ‘movie-like’ experience of learning via instruction teaming, if course content can be structured as story-centric with initial challenge problem followed by drama/ suspense (various paths, solutions, constraints) and unravelling the mystery (climax) at the end — with either a solution or make students learn the right model to solve problems.

5.5 Focus on Cognitive Constructs The learning primitives identified as part of SLATE framework can form the basis of designing course content. For example, if the course demands more of ‘abstraction’ and ‘analysis’ skills, relevant cases, examples and assignments should dominate

312

G. K. Koneru

much of the content with passing references to advances in that technology area (part of ‘trending’). On the other hand, if the course demands engineering or system development skills (part of ‘synthesis’), course content should be enriched with Problem or Project-Based Learning (PBL) exercises, group assignments, and development methodologies. with access to on-demand learning resources. More or less, today as part of Outcome Based Education (OBE), this model is being adopted in many Indian institutions of higher education by designing course content and assessment by mapping to the levels of the Learning Pyramid of Bloom’s taxonomy.

5.6 Assessment and Evaluation While a teacher can motivate a student for deeper engagement, it is the assessment and evaluation — an inevitable part of education system — that drives students to put in their best efforts towards attaining grades and degrees. In the connected world where information and knowledge is available at one’s fingertips and emergence of intelligent answering systems like ChatGPT [7] will be transforming the assessment models with emphasis on assessing the higher-order learning and thinking skills replacing rote learning and memorization of content.

6 Conclusion Education is an unsolved problem. It is debatable how much of what a teacher and student invests in terms of his time and effort would contribute to one’s real learning beyond classroom. Most of the studies in Teaching–Learning are empirical in nature and learning models are based on cognitive psychology. However, today’s digital communication technologies democratize learning for any aspiring individual at any time and place with digitized content delivered almost for free. Our current models of university and online education still carry remnants of bygone era when education was considered as prerogative of few individuals and universities deemed as ivory temples of learning compartmentalized with individual departments and faculty disciplines. The role of a live teacher in this digital learning ecosystem is more pronounced than ever — with teacher playing the role of mentor, motivator, and guide for the student. Instruction models have to be relooked from a learner’s perspective with empathy and ethnographic studies. Let’s hope that we shall see some transformation in the near future in this evolutionary path of education. In this autoethnographic study, the author — as Teacher-Student immersed in learning new courses in virtual mode — shared his observations and insights. As learning is intrinsically a subjective human experience, the ideas and conclusions drawn may be biased or opinionated or subject to debate and therefore demand indepth study and formal research methods, which will be undertaken and presented by the author in the follow-up paper.

Ensuring Optimal Performance in Online Learning of STEM Subjects …

313

References 1. NPTEL homepage. https://nptel.ac.in. Accessed 05 Mar 2023 2. Sweller, J.: Cognitive load theory. In: Psychology of Learning and Motivation, Academic Press, vol. 55, pp. 37–76 (2011) 3. Kalyuga, S.: Schema acquisition and sources of cognitive load. In: Plass, J., Moreno, R., Brünken, R. (eds.), Cognitive Load Theory, pp. 48–64. Cambridge University Press, Cambridge (2012). https://doi.org/10.1017/CBO9780511844744.005 4. Gopala Krishna, K.: SLATE--a framework for systems learning and thinking enablement. In: The 9th World Engineering Education Forum (WEEF - 2019), Elsevier (2019). https://doi.org/ 10.1016/j.procs.2020.05.066 5. Cecilia Obi, N.: Constructivists’ theory and science education classroom. Eur. J. Sci. Res. 154(4), 549–553. ISSN 1450–216X/1450–202X. http://www.europeanjournalofscientificresearch.com 6. https://en.wikipedia.org/wiki/Autoethnography. Accessed 09 Jul 2023 7. https://en.wikipedia.org/wiki/ChatGPT. Accessed 12 Mar 2023 8. https://archive.nptel.ac.in/noc/transcript_verify/?val=F16D9888010349BB2BF8C16B7EC 9A6C08B257EBCFD16B4BCEA4D8B8FCF93E6E5. Accessed 12 Mar 2023

The Development of a Learning Arrangement in a Characteristic Curve Remote Laboratory Ingrid Krumphals , Thomas Benedikt Steinmetz , Christian Kreiter , and Thomas Klinger

Abstract Remote laboratories can be an essential support for teaching and learning. They offer possibilities that otherwise cannot be implemented in the classroom. The project OnLabEdu (Online Laboratories for School Education) focuses on developing remote labs for school, including the development of appropriate accompanying teaching and learning materials. This paper presents the first design of a learning arrangement for a characteristic curve remote lab. The learning arrangement was designed based on the model of educational reconstruction and therefore, takes the clarification of the scientific content and students’ perspectives into account. The topic is RGB LEDs and the connection of the terms energy, forward voltage, and wavelength of light. Following a design-based research approach, the learning arrangement was evaluated through two probing acceptance interviews with two high school students. The main goal was to identify first hints on learning obstacles, along with elements that support learning within the learning arrangement together with the handling of the remote lab itself. The results showed barriers to conceptual understanding of energy and forward voltage and that students had problems writing down their ideas in appropriate technical language. Furthermore, the students noted room for improvement concerning the interface of the remote lab. Based on these findings, ideas for re-designing the learning arrangement are discussed. Keywords Remote Labs · Development of Learning Arrangements · Probing Acceptance

I. Krumphals (B) · T. B. Steinmetz University College of Teacher Education Styria, Hasnerplatz 12, 8010 Graz, Austria e-mail: [email protected] C. Kreiter · T. Klinger Carinthia University of Applied Sciences, Europastraße 4, 9524 Villach, Austria © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_25

315

316

I. Krumphals et al.

1 Introduction Remote laboratories are an essential extension for teaching and schools – especially regarding school feasibility. Thus, the technical opportunities created by remote labs for education are very welcome. Often, experiments are not being conducted in an authentic setting due to the teacher’s workload in relation to the design of the experiment as well as safety aspects, which sometimes cannot be maintained in school [1]. In this case, remote labs help to take complex experiments worth teaching and learning into the classroom [2]. The project OnLabEdu (Online Laboratories for School Education), funded by the Austrian Innovation Foundation for Education, addresses the challenge of developing remote labs like this for schools. An essential factor for the general use of remote labs is, among other things, the development of appropriate accompanying teaching and learning materials. Currently, three prototypes of remote labs are developed within the project and therefore are available for the first piloting phase. In addition to developing the software and hardware of the remote labs, teaching and learning materials are being designed to enhance and support teachers in using the remote labs in the classroom. The first version of the learning arrangements are developed based on the current state of educational research. Furthermore, the learning arrangement and the remote labs are refined based on empirical evidence generated by accompanying educational research within the project. Thus, it is possible to incorporate current teaching and learning research – e.g., regarding learners’ conceptions, basic subject-related concepts, and competence orientation – into developing teaching materials. Furthermore, new research results relating to learning processes are generated based on our empirical findings. The learning environment and learning materials developed within the project can be optimized for students on the secondary level, the current target group of the project. The paper will present one of the developed remote labs (characteristic curve lab) shortly, including the hardware structure and the interface. The focus is on the accompanying development of the learning arrangement and the first empirical findings students encounter while working on the tasks. Specifically, the first empirical data collection aims to identify elements that hinder learning processes and components that promote learning processes to refine the developed learning arrangement on a design level (concerning the interface) and on a content level.

2 A Learning Arrangement in a Characteristic Curve Remote Lab The characteristic curve remote lab is a comprehensive electrotechnical lab and can measure the current-voltage characteristics of several electronic components (e.g., resistors and diodes). The hardware consists of a single circuit board with 16 different measuring sockets (see live image in Fig. 1). The laboratory can be controlled via a web client, and the current-voltage characteristic can be measured for different

The Development of a Learning Arrangement in a Characteristic Curve …

317

Fig. 1 Characteristic Curve Web Client

components. It is also possible to set the voltage range (see Fig. 1). Additionally, via webcam, it is possible to follow the measurement (e.g., when measuring an LED, it can be seen when an LED light turns on).

2.1 Development and Key Ideas The development of the first version of the learning arrangement follows a designbased research approach [3, 4]. Therefore, iterative cycles of design, evaluation, theory impact, and re-design are done (see Fig. 2). The first version of the learning arrangement is designed based on design principles subjected to current science education research literature. Furthermore, the model of educational reconstruction provides a foundation for development [5]. So, to design learning environments, the main aspects that must be considered are clarifying science content and learners’ perspectives. The main principle we follow in the design of the learning environment is that the learning processes of students take place in a constructivist way [6]. The first learning arrangement to the characteristic curve remote lab is developed for high school students (11th –12th grade) in Austria. The topic focused on is light emitting diodes (LEDs) and RGB diodes are used in particular. The scientific key ideas we follow within the learning arrangement are extracted and derived from tertiary physics textbooks and literature (e.g., [7, 8]). These are the key ideas we focus on: 1. There is a relation between energy and the wavelength of light.

318

I. Krumphals et al.

Fig. 2 Design-Based Research Cycle (Design-Based Research Approach, Sarah Zloklikovits, CC BY 4.0, https://commons.wikimedia. org/wiki/File:DBR_english_ greyscaler.svg)

2. Considering the wavelengths of the light, shorter ones are related to higher energy and the other way around. 3. The wavelength of light provides information about the assigned position of the light in the electromagnetic spectrum and therefore it can be assigned to a type or color of light. 4. The current-voltage characteristic can provide information about the properties of electronic components. 5. The forward voltage indicates the energy level which must be reached to turn on an LED. While these overall key ideas form the basis of the learning arrangement, students’ perspectives, in particular students’ conceptions concerning the topics energy (e.g., [9, 10]) and electric current (e.g., [11, 12]), as well as the electromagnetic spectrum (e.g., [13, 14]), are considered. Based on the above considerations, we defined the following primary learning objectives for the students: 1. The learners can argue that red, green, and blue colors have different wavelengths. Red has a bigger wavelength than green than blue. 2. The learners can connect the wavelength of different light colors to its energy. 3. The learner can explain how the different forward voltages are related to the colors of the LED. 4. The learners can explain the term Fermi energy and use it for their explanation concerning the different forward voltages of the red, green, and blue LEDs.

2.2 The Learning Arrangement At the beginning of the developed learning arrangement, the students start with an introduction and a sheet of information (3 pages) where the main terms and underlying concepts (electromagnetic spectrum, conduction band and valence band, insulator, semiconductor, conductor, forward voltage) are shortly explained. Additionally, the sheet contains a short introduction concerning LEDs, and Fermi energy is defined. The remote lab is presented briefly at the end of the information sheet. After this introduction, the students must solve three tasks that focus on investigating the current-voltage curve of three LEDs (RGB). In particular, the following tasks must be worked on:

The Development of a Learning Arrangement in a Characteristic Curve …

319

1. Phrase a hypothesis on the forward voltages of the different LED colors. If they differ and how or if they do not differ. Give elaborated reasons for your hypothesis. 2. Check your hypothesis with the characteristic curve lab. Document your approach. In the datasheet of the LED used in the characteristic curve lab, you can find the wavelengths of the emitted colors. Using the formula below, explain why the forward voltage of the respective colors is different and has the given order: red, green, and blue (from small to large). Justify your answer with pictures of the remote lab’s measurement series.   1240 eV · nm (E . . . Energy, λ . . . wavelength) (1) E= λ nm The students should achieve the learning goals above by working on these tasks and using the information sheet.

3 Educational Research In the sections above, we described the characteristic curve remote lab and the learning arrangement’s development and objectives. One objective of the study is to optimize the accompanying developed learning arrangement by following a constructivist perspective on learning (e.g., by addressing students’ conceptions) and mainly to foster conceptual change [6]. Therefore, our educational research follows a designbased research approach [3, 4], which involves iterative cycles of developing learning arrangements linked with evolving learning theories.

3.1 Research Question The main goal of the first evaluation of the learning arrangement is to investigate students’ learning obstacles and beneficial elements. Therefore, we follow the main research question: Which aspects of the learning arrangement can be identified that foster or hinder the students’ learning process?

3.2 Design of the Study To answer the research question, we are using the method of probing acceptance [13, 14]. This method allows us to identify first hints at learning obstacles along with elements that support learning within the learning arrangement together with the

320

I. Krumphals et al.

handling of the remote lab itself. The procedure of the probing acceptance interviews was the following: 1. First, the students get the information sheet with the essential scientific terms and explanations they need for the task (see Sect. 2.2.). 2. The students must complete the tasks (see Sect. 2.2.). 3. For each task, the students were asked the following questions: a. b. c. d. e.

How comprehensible was the task for you? Please describe how you worked on the task. What information provided in the material helped you to solve the task? Describe what was easy and what was challenging. What else would you have needed to solve the task?

4. Finally, the students got questions concerning the website itself: a. How intuitive and easy was it to navigate and use the lab? b. Were there any ambiguities and irritations using the remote lab? If so, where? c. Do you see room for improvement? If so, where? The data collection was done in February 2023. Two probing acceptance interviews with students were conducted (two high school students, both female). The interviews lasted, on average, about 53 min and were transcribed, anonymized (the two cases were labeled with other names), and analyzed using qualitative content analysis [15]. In addition to the interviews, the students’ worksheets form the data basis of our first evaluation.

4 Results The results section focuses on the significant hints we found for a first re-design cycle. This initial evaluation must be considered an exploratory study to discover the most obvious stumbling stones and essential beneficial elements within the learning environment. The results will be presented in order of the tasks the students worked on (see Sect. 2.2.).

4.1 Task 1 Both students partly solved the first task of phrase a hypothesis. Both were able to connect the information on energy and wavelength. Monica mixed red and blue concerning higher and lower energy: “The forward voltage can distinguish between the individual colors red, green and blue, because the energy or the wavelength of the colors is also different and the forward voltage can, for example, transmit blue (lower

The Development of a Learning Arrangement in a Characteristic Curve …

321

energy) but not red (higher energy)” [Monica, written answer to task 1 – translated from German]. Furthermore, Monica shows gaps in technical language and puts the forward voltage as the active part, distinguishing between the individual colors and transmitting blue. This also hints at gaps in understanding that the term forward voltage is just an indicator for the energy level that must be reached for specific colors of LEDs. When asked what was easy and difficult, both students – Monica and Julie – noted that phrasing the hypotheses was difficult [Julie, 124; Monica, 139; numbers refer to the paragraph in the transcription].

4.2 Task 2 Monica and Julie solved the second task of testing the hypothesis with the help of the characteristic curve remote lab. While Monica answered with two sentences without any website documentation, Julie managed to document her measurements with screenshots and an explanation. Nevertheless, both mentioned problems with using the interface. The main obstacle was setting up the starting and stopping point for the voltage [Julie, 132; Monica, 146]. Especially for Monica, the interface was not intuitive: “I opened the website, clicked around a bit and tried it until I was helped because I was lost at the highest degree and [then I] looked at the graphs” [Monica, 146 – translated from German].

4.3 Task 3 Both students accomplished the third task to argue with the formula above (1) why red, green, and blue LEDs need different forward voltage. Both interpreted the formula and explained that wavelength and forward voltage are inversely related. Nevertheless, Monica needed help to solve this task: “I remembered the formula, then took my first best thought, which was wrong, and then analyzed other connections with the help of the teacher so that I then came up with the correct connection” [Monica, 157].

4.4 Others and Summary The concept of forward voltage in combination with LED wavelength seems difficult to understand for the students. Both showed problems with technical language: Julie: Well, I think there is a difference because red, yellow, and blue, and red and blue have different nanometers.

322

I. Krumphals et al. Interviewer: what do you mean by nanometers?

Julie: Well, they have a different range in visible light. And that’s why I think there is a difference in the forward voltage because they have different nanometers, and if you have a formula, for example, it depends on that, so if you take the formula from above, the result is also different for the period length. Isn’t it? [Julie, 18–20]

Concerning the interface, both students see room for improvement [Julie, 161; Monica, 166], while the overall feedback of both students concerning the learning arrangement was positive [Julie, 163; Monica, 171].

5 Discussion and Outlook Our results show that the students need help connecting the concepts of energy, wavelength, and forward voltage concerning an LED. In detail, students face problems when using technical terms to phrase a hypothesis and answer the tasks. This points out the need for scaffolding materials concerning technical language for the next re-design cycle of the learning arrangement. Additionally, the previous information sheet must consider the technical language problems while connecting wavelength, energy, and forward voltage. Here one idea is using frequency instead of wavelength due to the direct proportionality, which is more difficult to understand and connect than the indirect correlation. Furthermore, the concept of forward voltage in connection to current-voltage characteristics is not entirely clear to the two students. Additional learning aids – e.g., hints on the start and stopping point of the voltage – may help optimize the learning arrangement. These hints could be directly implemented on the interface. These probing acceptance interviews are the first explorative interviews to find hints on learning obstacles and elements that foster learning within the learning arrangement. Our results are limited to this characteristic curve remote lab and the developed learning arrangement. Nevertheless, the methodological approach to the empirically based development of remote labs is a novelty, as this currently needs to be shown in the literature on remote labs. Applying this approach concerning the design of remote labs, including accompanying learning arrangements, is as fruitful as it is in already common research-based and empirically supported development of learning arrangements and teaching and learning materials. Our results will be used to adjust the remote lab and the learning arrangement. In the next step, the second version of the learning arrangement will be developed and evaluated in the following circle according to the design-based research approach. Acknowledgements This project is funded by the Innovation Foundation for Education and is carried out as part of the Innovation Labs for Education program.

The Development of a Learning Arrangement in a Characteristic Curve …

323

References 1. Schlichting, L.C.M., et al.: Remote laboratory: application and usability. In: 2016 XII Congreso de Tecnologia, Aprendizaje y Ensenanza de la Electronica (XII Technologies Applied to Electronics Teaching Conference) (TAEE) 22.06.2016–24.06.2016, pp. 1–7. IEEE, Seville, Spain (2016). https://doi.org/10.1109/TAEE.2016.7528355 2. Pester, A., Klinger, T.: Distributed experiments and distributed learning. Int. J. Onl. Eng. 16(6), 19–33 (2020). https://doi.org/10.3991/ijoe.v16i06.13661 3. Barab, S., Squire, K.: Design-based research: putting a stake in the ground. J. Learn. Sci. 13(1), 1–14 (2004). https://doi.org/10.1207/s15327809jls1301_1 4. Haagen-Schützenhöfer, C., Hopf, M.: Design-based research as a model for systematic curriculum development: the example of a curriculum for introductory optics. Phys. Rev. Phys. Educ. Res. 16(2), 020152 (2020). https://doi.org/10.1103/PhysRevPhysEducRes.16.020152 5. Duit, R., Gropengießer, H., Kattmann, U., Komorek, M., Parchmann, I.: The model of educational reconstruction – a framework for improving teaching and learning science. In: Jorde, D., Dillon, J. (eds.) Science Education Research and Practice in Europe: Retrospective and Prospective, pp. 13–27. Sense Publ., Rotterdam (2012). https://doi.org/10.13140/2.1.2848. 6720 6. Duit, R.: The constructivist view in science education–what it has to offer and what should not be expected from it. Investigaçôes em ensino de ciências 1(1), 40–75 (1996) 7. Feynman R.P., Leighton R.B., Sands M.L.: The Feynman Lectures on Physics, Volume II: Mainly Electromagnetism and Matter. Addison-Wesley, San Francisco, CA (2011) 8. Jaeger, R.C., Blalock, T.N., Blalock, B.J.: Microelectronic Circuit Design, 6th edn. McGraw Hill, New York (2023) 9. Liu, X., Ebenezer, J., Fraser, D.M.: Structural characteristics of university engineering students’ conceptions of energy. J. Res. Sci. Teach. 39(5), 423–441 (2002). https://doi.org/10.1002/tea. 10030 10. Schecker, H., Duit, R.: Schülervorstellungen zu Energie und Wärmekraftmaschinen. In: Schecker, H., Wilhelm, T., Hopf, M., Duit, R. (eds.) Schülervorstellungen und Physikunterricht, pp. 163–183. Springer Spektrum, Berlin (2018). https://doi.org/10.1007/978-3-662-572 70-2_8 11. Saputro, D.E., Sarwanto, S., Sukarmin, S., Ratnasari, D.: Students’ conceptions analysis on several electricity concepts. J. Phys.: Conf. Ser. 1013, 12043 (2018). https://doi.org/10.1088/ 1742-6596/1013/1/012043 12. Wilhelm, T., Hopf, M.: Schülervorstellungen zum elektrischen stromkreis. In: Schecker, H., Wilhelm, T., Hopf, M., Duit, R. (eds.) Schülervorstellungen und Physikunterricht, pp. 115–138. Springer Spektrum, Berlin (2018). https://doi.org/10.1007/978-3-662-57270-2_6 13. Wiener, G.J., Schmeling, S.M., Hopf, M.: The technique of probing acceptance as a teacher’s professional development tool: a PCK study. J. Res. Sci. Teach. 55(6), 849–875 (2018). https:// doi.org/10.1002/tea.21442 14. Jung, W.: Probing acceptance, a technique for investigation learning difficulties. In: Duit, R., Goldberg, F., Niedderer, H. (eds.) Research in Physics Learning - Theoretical Issues and Empirical Studies, Proceedings of an International Workshop held at the University of Bremen, pp. 278–295. IPN, Kiel (1992) 15. Kuckartz, U.: Qualitative Text Analysis: A Guide to Methods, Practice and using Software. Sage, Newcastle upon Tyne (2013)

How’s Identity Being Learned in City Museums? An Identity Education Approach to Museum Education Zhichao Lei

Abstract This project addresses the need to develop cultural identity of citizens in the urban context, and well as sharing of such identities with others as global citizens. This is seen as a necessity to develop the sense of one’s own identity and culture, as well as the understanding for each other, especially in the major cities of “immigrant nations” where people from different backgrounds live, work and/or study together. In order to address this need, identity education (IdEd) is used as a framework to present a solution, a way to support the development of cultural identities through city museums. The needs for the inquiry in city museums and identity education through the institutions, and the theoretical framework to support the inquiry are analyzed and articulated in the paper. More importantly, a learner-centered IdEd assessment framework is introduced to facilitate the overall analysis on the identity narrative and the learning experience design of city museum, which is supposed to offer some guidelines for direct observations of the visitors’ experiences and evaluations of curators’ decisions as well as technological supports for instructional purposes from the IdEd perspective. Keywords Identity Education · Museum Education · City Museums

Z. Lei (B) Department of Communication, Seoul National University, Seoul 08826, South Korea e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_26

325

326

Z. Lei

1 What Shall Be Known about Identity in this Urban Era? 1.1 Migration, Acculturation and Identity Issues in Burgeoning Cities Orloff [20] argues that the 21st century is “an urban era” as “the century’s problems will be found in cities,” and that city museums could play significant and meaningful roles in this era; bold as it may sound, today ought to be “the moment of city museums.” As early reported by United Nations [32], more than half the earth’s population now lives in cities, and if projections hold, by the year 2050, the proportion of urban residents globally will increase to two-thirds. Human civilization is marching towards an era of city, during which new opportunities brought by urbanization always come with emerging dark-sides in supposedly every aspect of human society. It is believed by Tisdale [30] that “cities not only provide economic opportunities but constitute our best shot at solving systemic societal problems to achieve social justice.” Myriads of class-based critiques have been made on social justice issues taking place in cities, such as income gap, unemployment rate, educational resource allocation, homelessness, drug addiction, prostitution [30]; gender and racial equality is also believed to be at stake by some scholars [18]. As a matter of fact, the discussion of city and social justice could go far beyond these dimensions, especially when the conceptual lenses of cultural globalization and cosmopolitanism are taken into account, accompanied by an unfolding age of international migration. In line with World Population Prospects 2022 [31], international migration is having important impacts on population trends for some countries; in some parts of the world, international migration has become a major component of population change; over the next few decades, migration will be the sole driver of population growth in high-income countries. Meanwhile, the influxes of immigrants to a city could, if examined through the cultural lenses, challenge the intellectual, cultural and ideological monopoly of the local community; the prevailing attitudes towards immigrant families shared by local residents could be indifferent, refusing and even xenophobic. Rising extremism and hate crimes against certain groups of people could be found closely connected to these cultural conflicts.

1.2 Identity Education as a Key to Unlocking a Peaceful, Inclusive World Some scholars working on international studies have noticed that the cultural conflicts constitute a common issue in an intercultural context or during an acculturation process, namely a process of assimilation to a different culture, mainly happening to immigrants; and immigrant youth, in particular, could suffer a lot mentally and

How’s Identity Being Learned in City Museums? An Identity Education …

327

physically, from the process, as reported by some acculturation researchers [4]. It has also been noted that multicultural youths may have a complicated understanding of their identities and lives in multilingual contexts [34]. As maintained by Bai and Nam [3], the term Identity can be defined and comprehended in a broad spectrum of ways, as each individual or group may socially categorize themselves based on their own “looks, beliefs, norms, attitudes, and other factors”; thus, people may represent their identities according to sociocultural factors such as gender, sexual orientation, race, ethnicity, and national origin. In other words, one’s identity could be constructed based on one’s own selection of the available cultural factors formed in history, which means that an individual’s initiative takes a pivotal role in the development of identity. Besides, in accordance with Falk [10], all individuals enact multiple identities, many of which are situational and constructed in response to a social and physical context. These fundamental understandings could be used as the rationale of why identity can be learned and could help explain the identity issues usually seen in a multi-cultural context. More importantly, the identity issues inflicted on immigrants are arousing concerns for public security, as the linkage between the “homegrown threats” and “foreign terrorist organizations” has been found to play out in dangerous ways [21], marked by naturalized citizens waging terrorist attacks on the very city where they live. It could successively aggravate the misunderstandings, fears and even hates of local communities against immigrants—in fact, even some academic papers appear to pick up a tone demonizing certain ethnic or racial groups, using potentially misguiding terms like “Islamist Terrorism.” As an all-out war on terrorism has been waged in this post-911 era, the author believes that educational interventions in identity development are also worthy of some attempt for this endeavor. In this concern, the author turns to Schachter & Rich [25], who present the concept of Identity Education, IdEd—the purposeful involvement of educators with students’ identity-related processes or contents—as a conceptual framework for educational researchers and practitioners. According to their work, there are three facets of identity development for educators to work on, as follows. • Content: “the specific ideation that is the object of a person’s identifications or commitments. These can be ideals, knowledge, institutions, people, etc.” [25; p. 27] • Structure: “the various ways identity elements are embedded, organized and related to one another, to identity’s degree of complexity, and to its stability and flexibility” [25; p. 27] • Process: “those processes that are involved in the acquisition, maintenance and transformation of identity contents or structures” [25; p. 27] With a long-term interest in narrative analysis, the author chooses to interpret the identity content in an IdEd program as identity narratives, or educational materials serving to construct certain identity narratives, while Structure is understood as the narrative structure behind the materials, which is constructed by instructional designs to assist the learners to engage with the Content. Process constitutes the essential

328

Z. Lei

outline of an IdEd program, and explains the reason why such programs can be assessed based on Theory of Change, which will be discussed later in this paper.

2 What Shall be Known about Identity Education in Museums 2.1 Re-Approaching Museum Education with Identity Education The academic understandings of Museum and Museum Education are wellestablished and multifaceted. In line with the latest definition approved by the International Council of Museums (ICOM), a museum is “a not-for-profit, permanent institution in the service of society that researches, collects, conserves, interprets and exhibits tangible and intangible heritage” [16]. Definitions introduced by different professional museums associations around the world could be slightly different, but their common point is to insist upon the activities of a museum which make it differ from other institutions: conservation, research and communication [12]. A group of education scholars have interrogated how museums can be utilized as an educational space for both formal and informal learning/teaching to happen [2, 15, 29]. History Education, Arts Education, Science Education have been found to be able to take advantage of relevant museums [15], such as history museums, art museum, and science museum. However, the author looks more into previous research that established museums as narrative tools in education. As museums are believed to depict narratives that can be crucial to understandings of the past for their visitors [22], the relations between the learning of identity narratives and Museum Education could be further clarified. Museums, as active social participants, serve many social purposes, which includes defining and expressing major social narratives; the narratives conveyed by museums are observed as definitive and authoritative, and the objects displayed are understood as emblematic or normative culture [7]. To be more specific, as maintained by Roberts [35], it is through collecting, cataloguing and conserving art objects that society’s ontology of objects is displayed; the social construction of value is revealed in curatorial decisions. Therefore, what kind of narratives and how the narratives are learned by a museum visitor could be considered essential learning problems in Museum Education. Regarding this, Golding [13] describes museums as frontiers where learning is created, new identities are forged and new connections are made between disparate groups and their own histories. Moreover, museums have been considered important collections of ideological symbols and perform a special communication as well as legitimizing role [7]; and the educative, social and political roles of museums can be put in use, for the purpose of promoting social, cultural, and environment justice movements by motivating audiences to engage in special events and activities [3]. In this sense, the significance

How’s Identity Being Learned in City Museums? An Identity Education …

329

of this study also lies in the fact that identity is an inevitable issue that is rooted deep down in the core of every culture and can be discussed on both individual and social levels. On the individual level, the feeling of being part of a city could be a source of the belief that “I belong here”; moving on to the social level, the communal belief that “we all belong here” could serve as a precondition for local people bonding with each other. Therefore, to achieve and maintain the mental health of an urban resident, as well as the general harmony of the urban communities, strategies could be built upon identity that is being formed in a particular city, and also its city museum.

2.2 Growing Responsibilities on Museums in the Age of Decolonization Stakeholders of this paper might transcend museum visitors, educators, local communities and municipal offices, as more social expectations have been put on the museum industry. Museums have been criticized for being a symbol of colonialism; and the challenge to the industry is how to decolonize a product of colonization without removing all museum structures and their services from the scene [19]. In this endeavor, museums have become comfortable embracing a learning mission, turning into more common locations for learning research, as they are filled with complex, rich, and fascinating learning problems [9]. With these learning problems being solved, the beneficiaries might turn out to be more than one generation. Bai and Nam [3] argue that the prominent educative roles of museums should be played to encourage visitors to raise their own critical voices, thereby they can take social roles and responsibilities to promote a more democratic cultural environment for their future generations. The focus on learning and the fulfilling of museums’ social responsibilities, as believed by some scholars, are motivated by increasing pressure on museums to demonstrate that they serve a broader public, and not only an educated and cultured elite [9]. More importantly, by transmitting a new reality not only to itself, but also to its immediate community, to the world beyond, and even to the seats of national governance, as believed by Maranda [19], museum can become a standard-bearer for the decolonization of itself and for a societal realignment towards those who have been disenfranchised for so long. In this sense, this paper could also provide some insights for people working in the museum industry, in terms of how to better perform a museum’s duty by playing its educative role, which is supposed to facilitate, even putting aside IdEd, their branding strategies about building a social image of cultural diversity and inclusiveness in the age of decolonization.

330

Z. Lei

2.3 City Museum as the Starter of a Long Discourse Approaching this era from its urban facet—a main feature but not the full picture— the author realized the potentials of city museum in IdEd and thus proposed the IdEd in city museums as an educational solution to the matter of “winning hearts and minds on all sides” in this multicultural urban era. It is underlined that for most immigrant children and adolescents, schools and other education settings are the major arenas for inter-group contact and acculturation [23]. Hence, education practitioners and researchers are expected to help immigrants, especially the multicultural youths, appreciate their legacy, fit into the local culture and embrace their new identities as new citizens of the country, and in many cases, as new residents of the city where they choose to live. As maintained by Orloff [20], the urban region, rising as an increasingly fundamental, political, social, and economic unit, will increasingly displace the nation-state as the basic unit of self-identification and culture; it will be increasingly in metropolitan regions that people will seek rootedness, and consequently the diversity of the urban population will be enhanced and call for more diversified representation of their cultures and identities. Hence, for the further, harmonious development of cities and the world, city museums, namely the institutions located in major metropolitan areas that collect and interpret the history of their city [30], could assume an important role to address identity-based cultural issues. That being said, the current situation is, “cities have their dark sides, and city museums don’t have a great track record of addressing them” [30], which leaves us a huge gap worth theoretical and empirical attention from the educative perspective. On the part of urban curators working with city museums, their job is “not merely to collect and share historical knowledge, but to help change and shape the lives of our cities and their citizens” [20], which means a positive, transformative function has been expected. Therefore, the framework introduced later for the assessment of city museum’s performance in IdEd could also serve as guidelines for city museum managers to organize the exhibits and events in their museums.

3 What Could Be Used for Theoretical Framework The framework serves the purpose of assessing the effectiveness of the IdEd offered by a city museum. In other words, the author is interested in the design of the narrative structure about a cultural identity, which is presented by a city museum and open for the visitors to do personalized learning, as well as what can be learned from it. The author starts off exploring the relatively recent findings in museum education, notes what has been suggested in terms of how to design museum experiences to support more powerful learning, and combines some of the theoretical achievements with classic theories and principles from the perspective of instructional design.

How’s Identity Being Learned in City Museums? An Identity Education …

331

3.1 Starting from the Visitors to City Museums To begin with, the questions below, put forward from a learner-center perspective, are worth thinking: • What could be the visitor’s cultural identity? And is it the one represented through the city museum? • What do the visitors expect to engage with upon entering a city museum? And how do they engage with the contents provided by the city museum? A simple dichotomy can be used to divide the city museum visitors into local residents, perhaps frequent visitors, and travelers from outside the city, namely, the transient visitors. For the further classification on the possible cultural identities of them, the author turns to the lens of nationalism: anyone who had or has another citizenship than the dominant one shared among the other local residents, could be identified as an immigrant in some cases, or an international traveler/worker/student in other cases, and therefore possibly have a cultural identity different from the one shared among the local community. Such visitors could find many differences between their cultural identity and the local, dominant identity shared among the local community and supposedly presented in the city museum. More importantly, as a matter of fact, people having or claiming to have multiple cultural identities are no more rare cases, because of not only international migration, but the increasingly developed intercultural communication, largely propelled by social media [6]. Then, how do the multiple identities interact in a city museum? And how do such interactions affect the IdEd in a city museum? Answers to these questions are supposed to help address how the visitors engage with the contents in a city museum. A museum-specific method of classification on museum visitors is adopted to provide a specialized lens to define and understand a museum visitor’s role based on a certain interaction style of multiple identities. In line with Falk [10], “visitors to museums tend to enact one or various combinations of five museum-specific identities,” which is summarized and combined with the learning concern below. • Explorer: they visit a museum “because of curiosity and/or a general interest in discovering more about the subject or the content of the institution” [10; p. 156]; effective learning can be expected on them, while the efficiency of their learning could vary. • Facilitator: their visit to a museum is mainly “in order to satisfy the needs and desires of someone they cared about (other than themselves)—in particular their children” [10; p. 157], thus, in contrast to the Explorer group, always assuming the role of teacher; that being said, it has been proved by research that great learning potentials can be found in the process of teaching [8], which implies that Facilitators could still learn by teaching during the visit. • Professional/Hobbyist: related to but distinct from the explorer group, they possess or claim to possess “a strong knowledge, and interest in the content of the institution, and their primary motivation was not general but specific” [10; p. 157; they can delve deep into the content without assistance, and could even provide

332

Z. Lei

guidance for other visitors; it could be estimated that they anticipate in-depth contents as possible for their own learning, and that a strong connection exists between Facilitator and Professional/Hobbyist. • Experience Seeker: such visitors choose a museum “primarily in order to collect an experience, so that they could say they have been there, done that” [10; p. 157]; according to Falk’s observation [10], they are often tourists and tend to engage with the content on site merely at a basic, shallow level, requiring interventions to boost their motivations for deeper engagement and more effective learning. • Spiritual Pilgrim: though in a small number, they visit a museum “in order to reflect, rejuvenate, or generally just bask in the wonder of the place” [10; p. 158]; showing a similar inclination with Experience Seeker, they place their main focus on the place per se, from which they can gain knowledge about the museum, probably even in a Professional/Hobbyist style, but meditations are needed to encourage them to become Explorer. It is assumed by the author that each motivational category leads to a distinct style of interaction between multiple cultural identities, which happens to a museum visitor during a museum visit, in that a museum visitor’s motivation could be a reflection of prior knowledge, beliefs and concepts, and could dramatically influence what the visitors notice about a context and how they organize and interpret it [5].

3.2 Considering Museum as a Designable Multi-cultural Context In practice, a museum institution mainly serves to enable the visitors to guide themselves through carefully selected contents from a vast collection in a way that is often linear, thereby providing historical contextualization for the displays [24]. Nevertheless, how to arrange all the learning opportunities mentioned before is more than a matter of being “linear” or, to name another, “classification.” If the learning resources in a city museum are arranged in a manner lacking a systematic mindset, it could be hard for the audience with average or below-average prior knowledge about the city, such as new arrivals, to even find a starting point. In light of this, one alternative is to think of providing a “context” for the audience, as “context” can be seen as a multilevel body of factors in which learning and performance are embedded [36]. From this perspective, assessing the IdEd of a city museum is essentially an examination of IdEd-relevant contextual factors available on site. While various contextual factors are believed to be significant in determining visitor responses, Skydsgaard, Møller Andersen, and King [26] argue that four design principles serve to put the contextual factors to work in different manners that facilitate reflection and discussion among museum visitors, especially young visitors. In combination with the five motivational categories discussed before, the four design principles can be used as a design-specific approach to examining whether an ideal visitor experience could be engendered, and whether the experience design

How’s Identity Being Learned in City Museums? An Identity Education …

333

corresponds to the exhibition aims and satisfies the visitors’ personal needs determined by their motivations. The findings below [26] about the four principles and their secondary concepts can be useful in the later construction of the assessment framework. • N: Exhibits with Narratives (N), defined as both personal (N-PN) and expert narratives (N-EN), were found effective in terms of facilitating personal reflection and prompting discussion; N-PN is implemented to reflect other people’s ideas and experiences, thus acting as an important source of inspiration and information— “narratives from members of the same target group as the audience help to increase the personal relevance of the exhibition, whereas narratives from other groups or generations can serve as documentation of cultural differences and changes over time” [26; p. 4]—while N-EN, on the other hand, serves to present relatively in-depth views with expertise and make them available to non-expert visitors. • P: Participation, defined as including both physical interaction (P-PI) with exhibits, and dialogic interactivity (P-DI) between visitors, is usually designed to encourage the sharing of ideas and feelings between visitors; recent findings in the fields of human perception, cognition and physiology have led researchers to propose a new conception of learning with physical interaction at its core, thereby laying theoretical foundations for the implementation of P-PI in museum contexts (interactive devices and activities on site), while P-DI is introduced to enrich the mainly uni-directional communication realized by N designs. • C1 & C2: Exhibits with factors of Curiosity (C1) and Challenge (C2) were found to attract visitors’ attention, but also “work well with other design principles to engage the visitors in sustained reflection and discussion” [26; p. 1]; with the help of C1, a visitor’s curiosity can be aroused by “objects never seen before, new information relating to existing knowledge, and fascinating pictures or surprising effects” [26; p. 3], and then help to determine which exhibit to see and where to engage, while C2 is introduced to challenge visitors to struggle, either physically or intellectually, evoking strong emotions—a challenge can even come from “exhibits that evoke emotions, confront visitors with dilemmas or address prejudices and taboos” [26; p. 4]. Taking this design-specific approach, the author classifies the contextual factors provided in a city museum for qualitative analysis concentrating on how these factors impact learners of different motivational types. It is noteworthy that spatial analysis on the museum space per se, or the systematic examination on the architectural elements of a museum will not be taken into account in this projection. Withstanding the fact that spatial factors are by no means irrelevant to the IdEd learning outcomes, due to the focus of this paper, the devices, interior styles, layout details and so on will be merely interpreted as certain contextual factors embodying some of the four design principles. Such interpretations leave behind a huge gap for the follow up studies with the introduction of applied spatial analysis.

334

Z. Lei

3.3 Introducing Cultural Hybridization and Theory of Change In fact, work has been done in academia to evaluate a museum’s role and performance. According to Vergo [33], roles of museums in Western society can be viewed by “how museums promote social and cultural capital that can be exchanged to produce economic capital by providing sites where people can obtain social network circles, cultural tastes, and edutainment, and thereby have positive memories of their participation.” Considering the growth of international migration and the rise of cultural globalization and cosmopolitanism, a city museum should take more steps forward and place more weight on its potentials in terms of contributing to social integration and cultural democracy in a multicultural society. Being advocated currently by some cultural scholars as an idea of cultural democracy, or “a third way between convergence and divergence” [17], Cultural Hybridization is understood as a one-dimensional cultural flow that interacts with other cultural flows, both internally and externally, to create a unique and innovative culture. Compared with the concept of acculturation discussed above, Cultural Hybridization has largely weakened the culturally dominant body and presented the possibility of breeding a composite new culture, through which a new identity could be constructed. In this sense, a city museum embracing the value of Cultural Hybridization could impart a cultural identity into its visitors, which values each component cultural identity and provides a compound identity for the visitors bonding with each other. It is the kind of message that would be appreciated in the name of cultural democracy and diversity. In order to facilitate the examination on a city museum’s IdEd performance to achieve a targeted cultural identity hopefully meant for Cultural Hybridization, Theory of Change should be then introduced. In line with [28], Theory of Change, or ToC, could help identify “measurable indicators of success as a road map to monitoring and evaluation,” as it “spells out initiative or program logic.” Therefore, the framework for assessment will be built upon an illustration of an IdEd targeted ToC.

3.4 Constructing the Framework for Overall Assessment For a target cultural identity to be learned, a series of subordinate learning objectives must be accomplished in an intertwined manner. These objectives could be identified on the basis of the three levels of Identity, as maintained by Schachter and Rich [25]. A city-related cultural identity could be split into the following three levels of Identity. • Social level: a person’s subjective sense of belonging to a city; this level can derive from the knowledge of the membership of a city (who can be considered as a city member) and the value and emotional significance attached to the membership, which is mainly conveyed through storytelling on the basis of local history

How’s Identity Being Learned in City Museums? An Identity Education …

335

• Personal level: a person’s unique goals, values, sentiments and preferences that can be learned from a city, and that are claimed as one’s own and serve to distinguish oneself from others in daily interaction; it provides a valued sense of uniqueness and singularity, which implicates that even for two persons claiming to be the members of the same city they could still differentiate from one another by comparing distinct aspects of the city culture that they choose to embrace, and that an essential precondition for this level of cultural identity is the existence of cultural diversity—multiple topics encompassed in the narratives—which is perceivable to the visitors and open for them to interpret in their own ways and have related discourses with each other, instead of cultural homogeneity imposed upon them in a propaganda-like style. • Ego level: a person’s sense of invigorating sameness and continuity that could be enhanced by living in a city, or, during a city museum visit, one’s subjective feeling that “one is the same individual leading a life that is coherent, imbued with purpose, moving from a reasonably understood past to a manageable future despite the diverse and often unpredictable social situations, circumstances and life events one meets” [25; p. 22], which is understood by the author as a package of an IdEd learner’s prior knowledge and motivates the learner to approach the contents on site in a personalized manner; this level is considered the underlying basis for the individual’s ability to be an active learner in IdEd, as it drives a person to “positively adopt, flexibly manage and freely commit towards personal and social identities” [25; p. 23], and could be somewhat reflected by one’s motivation behind the engagement in an IdEd experience. Moreover, while the museum provides the intermediate-level, designable context for IdEd, the larger context, or the social environment should also be taken into consideration, as identity needs and goals could be initially framed in different social systems [25] and afterwards have a huge influence on the intended identity of a city museum. Hence, for the starter of conducting an IdEd assessment, the macrosystem context should be first identified, which clarifies the social expectation on the targeted cultural identity that can be brought by a city museum; besides, whether or not Cultural Hybridization is championed in a museum context directly depends on the curatorial decisions made by curators, which can be discerned by qualitatively examining the Identity Contents and Structures in the IdEd Process. Based upon what has been introduced so far, the framework can be illustrated in the following conceptual model, outlining the basic logic behind the IdEd process in a city museum and signifying the directions for analysis on the design-specific contextual factors (Fig. 1). Taking the form of bold text, the four types of design-specific museum contextual factors (their secondary concepts mentioned before, such as N-PN, P-PI, are not included in the illustration, merely for simplicity, but could be used in the analysis of specific cases) not only contribute to the construction of the identity narratives, but also serve as meditations for learners, which indicates two main directions for the analysis. Thus a table is drawn below, and could serve as a scaffolding for a

336

Z. Lei

Fig. 1 The IdEd process in a museum context. Dotted arrow-lines stand for interventions/ meditations, either from the museum contextual factors or between the learners

researcher’s field notes taken down during the museum visits for the case studies (Table 1). When a museum item is inspected, it could be found to fall into more than one block in this table, as it could assume two or even more factors at the same time. The application of the framework and the table above is supposed to help address the following four questions, preferably in order: • What is the ultimately intended cultural identity, as expected and implicated by the macro-system context, in the city museum? Table 1. Table of IdEd analysis on museum contextual factors

How’s Identity Being Learned in City Museums? An Identity Education …

337

• Does the cultural identity, as presented by the narratives on site, embrace Cultural Hybridization? • Is the city museum doing its work well in teaching visitors about the intended cultural identity by utilizing different types of contextual factors? • What can be done, especially with the help of latest learning technologies, to better the IdEd performance of the city museum, on certain Factors/Directions?

4 What Could Be Expected in Future Endeavors With the framework above, conclusions of case studies could be reached on how the city museum takes advantage of different contextual factors to engage the visitors of five motivational types, whether there is any gap between the target identity expected by the city and the identity narrative presented by the city museum, and if any, how huge the gap is. Then, this closing part will land on the discussion of current issues and emerging trends related to Identity Education and Museum Education, as well as the possible directions for future research in this field.

4.1 Current Issues, Emerging Trends and the Imperatives The recent Migrant Crisis in New York City has drawn wide attention to the topics of growing multicultural societies, chiefly characterized by the co-existence of different ethnic groups. And Asian Hate, irrationally ignited by the COVID pandemic situation, is causing discrimination in either implicit or explicit manners and even violence against Asian American families. Even beyond the borders of the states, in recent past, attacks in Norway on July 22, 2011, can be found as another case in point and a constant reminder of the fact that terrorist attacks could derive from the antiimmigration and anti-multiculture emotions going unnoticed and unchecked. Such tragedies cannot be just taken as unfortunate and phrasal byproducts of a growing multicultural society in the age of decolonization. The moral imperative for educators, museum practitioners, municipal officers and others of interest to work in concert has manifested itself in these migration-related, culture-rooted social justice issues. While traditional immigration countries like the United States have a long history of making efforts to better position the minority groups in their societies for the good of all, lately well-developed nation states like South Korea have begun to invite people of diverse origins into their aging population, and even China and other countries with a relatively conservative attitude towards “foreigners” in their traditions, either originally multinational or homonational, have become increasingly open in terms of naturalization. In this sense, responsive cultural strategies should be underway, which could, as recommended before, start with city museums for a possible breakthrough. Even stepping away from the immigration topics, museum, by its very nature, has an inclination of trans-nationalism, and thus can be used for both nostalgic

338

Z. Lei

and bonding purposes in a multicultural society, like a metropolis, where the social members have various origins within or beyond the borders. Looking out on the changes in this pluralistic era, Tisdale [30] argues that now the main challenge for most city museums is to collaboratively start a new phase in social life and build up scientific awareness in a population which has historically been on the fringes of society, thereby offering the community a place for great cultural and human enrichment. In other words, city museums and urban curators must keep an eye on the growing complexity of city living and the increasing diversity of the urban population; thus, their jobs require the establishment of closer cooperation with other agencies and organizations. Educators, learning scientists and instructional designers, who are equipped with the learner-centered mindsets and relevant theoretical tools, and acquainted with myriads of instructional technologies, could provide insights for the planning and operation of museum exhibitions and activities, thereby assisting urban curators to build more learner-centered city museums, and turning the museum visitors into more active participants/leaners. Progress is being seen, however, with regard to interactive designs and joint programs introduced to museums for more efficient fulfillment of educational purposes. Reports have been made that the success of newer interactive form of museum has encouraged many more traditional forms of museum to reposition themselves as educational institutions, especially for school-aged children who visit with families or in school groups [9]. The rising interrelationships between museum visitors of different roles, museums of various functions/themes and the museumutilizing learning facilitators have proved how huge the benefits can be brought to formal and informal education by unified endeavors. For a city museum, such interrelationships must be underlined in particular, as a city museum is expected to take more social responsibilities for its community in the urban era. Another noteworthy trend is the rise of richer possibilities for personalized history, allowing visitors to explore and reinforce their individual identities [30]. It comes with the enrichment of learner experiences, boosted by new media technologies to a large extent, as mobile apps, social media, website homepages, etc., have enabled highly customized museum visits; even the concept of “museum visit” per se is being redefined and expanded by emerging concepts such as “e-visit.” Digital museums, and the museums taking virtual forms while probably possessing no physical entity, are now working with their real-world counterparts, providing an alternative or a supplement for visitors. For instance, in order to expand its scope and to maximize its education and communication functions, some museums utilize three-dimensional scanning and modelling to build a virtual museum that enables users to conduct a virtual tour [3]. There are also museum professionals dedicated to the building and maintenance of the museums’ websites and other digital products. Similar cases can be easily found lately and count as part of the efforts made by traditional museums to become a spectacular multimedia extravaganza.

How’s Identity Being Learned in City Museums? An Identity Education …

339

4.2 Two Directions for Future Education Research Taking This Approach This final section is written in hopes of providing researchers from different scholarly traditions with a common framework for constructive dialogue in terms of how to better exploit museums for IdEd, one possible identity-specific solution to problems arising in multicultural societies, and serving as a basis for generating focused and productive research directions related to Museum, a rich gold mine for interdisciplinary studies. While bearing in mind a common commitment to building a society that embraces understandings and differences, in which all individuals may live and thrive safely, educators and education researchers could turn to museums for their future work following at least two general directions, either separately or in an intertwined manner: the social dimension and the individual dimension. First, museum learning could be public and social, whether in peer groups or with families. Social interactions in museum settings fraught with political complexities render the multicultural participants, such as the visitor with an immigrant background, unable to rely on a single interpersonal identification, but must instead navigate among various identity options coming loaded with social expectations and biases. Therefore, the motivation-based treatment for the interactions between multiple identities in this paper might be accused of being oversimplified and neglecting certain principal social mechanisms by critical thinkers from the sociopolitical arena. Moreover, in the light of the complexities of multilingual contexts, some sociolinguists also advocate for the examination of identity as a narrative emerging through language - a fragmented, decentered, and shifting narrative, instead of fixed identities underlying certain discourse strategies [11]. This argument implies the necessity of combining linguistic thoughts into IdEd programs. Looking at the homogenizing, monolingual education systems of the three countries discussed in this paper—none of them being an “officially” multilingual country, like Canada— some critical scholars are concerned with the mainstream education institutions that are believed to devalue the first language of immigrant families [1] but also serve as the life-changing locus for these households, especially for their children, thus having considerable implications on the identity development within an immigrant family, usually characterized by bilingualism [14, 27]. The multilingual learners, especially the non-native speakers of the dominant language, could approach the museum resources very differently in comparison with the native speakers. In this sense, the lens of linguistics, along with a focus on language issues in education practices, could encourage the discourses on the establishment of a multilingual system in museum contexts. Second, the author also hopes that this paper might inspire a new generation of learning scientists to use museums as laboratories for their work on the individual level. Museums expand our definition of learning and require learning scientists to account for forms of knowledge, behaviors, and interactions that are often different from those in school, which requires interdisciplinary wisdom, including physiology,

340

Z. Lei

psychology, cognition and other disciplines that aim to explore individual entities. Turning to cognitive science, for example, we have so many questions to ask regarding what exactly happens “inside” us when we think, see, hear, understand, feel, act and learn in a museum setting: How does human cognition influence the design and use of certain museum items? How does the introduction of an instructional technology, like Augmented Reality or some other interactive devices, support our cognitive mechanism for learning in a museum? Besides, visitors with special needs are also noteworthy and accentuate the necessity of the research and development (R&D) in assistive technology, currently used by people with and without disabilities. Some of these technologies that can be found in daily life include voice typing, mobile devices capable of recognizing sounds and reading texts to the users, the audio versions of text articles, etc., most of which already hold robust records of assisting learning in museum settings. Such individual-level investigations are supposed to guide us to better understand and cope with emerging trends in museum education, namely the use of new technology and the pursuit of personalized learning. Combining the social and the individual dimensions, the academia can be expected to assume a driving force at both the strategic and the tactical level in museum education—not only for IdEd but also for possible solutions to other issues that are and will be confronted by human society. Acknowledgements The birth of this project can be traced back to the author’s graduate program at Teachers College, Columbia University; thus, special thanks should be given to Dr. Yoo Kyung Chang for offering guidance and advice. Also, warm thanks should be extended to BK21 FOUR Free and Responsible AI Media Group, Seoul National University for providing supports and resources needed by the author to complete this paper.

References 1. Adair, J.K.: The impact of discrimination on the early schooling experiences of children from immigrant families. Migration Policy Institute, Washington, DC (2015) 2. Altintas, I.N., Yenigül, Ç.K.: Active learning education in Museum. Int. J. Eval. Res. Educ. 9(1),120–128 (2020) 3. Bai, Q., Nam, B.H.: Where ‘West Meets East’: the cross-cultural discourses regarding the Chinese arts collections at the metropolitan museum of art. Identities 29(6), 883–902 (2022) 4. Berry, J.W., Phinney, J.S., Sam, D.L., Vedder, P.: Immigrant youth: acculturation, identity, and adaptation. Appl. Psychol. 55(3), 303–332 (2006) 5. Bransford, J.D., Brown, A.L., Cocking, R.R.: How People Learn. Expanded edn. National academy press, Washington, DC (2000) 6. Chen, G.: The impact of new media on intercultural communication in global context. China Media Res. 8(2), 1–10 (2012) 7. Coffee, K.: Museums and the agency of ideology: three recent examples. Curator Museum J. 49(4), 435–448 (2006) 8. Cortese, C.G.: Learning through teaching. Manage. Learn. 36(1), 87–115 (2005) 9. Crowley, K., Pierroux, P., Knutson, K.: Informal learning in museums. In: Sawyer, R. K. (ed.), The Cambridge Handbook of the Learning Sciences pp. 461–478. Cambridge University Press, Cambridge (2014)

How’s Identity Being Learned in City Museums? An Identity Education …

341

10. Falk, J.H.: An identity-centered approach to understanding museum learning. Curator: Museum J. 49(2), 151–166 (2006) 11. Ganat, G.L.: Book Review: Negotiation of Identities in Multilingual Contexts. Teachers College, Columbia University, New York City (2004) 12. Ginsburch, V., Mairesse, F.: Defining a museum: suggestions for an alternative approach. Museum Manage. Curatorship 16(1), 15–33 (1997) 13. Golding, V.: Learning at the Museum Frontiers: Identity, Race and Power. 2nd edn. Routledge, London (2016) 14. Honig, A.S., Xu, Y.: Chinese immigrant families and bilingualism among young children. NHSA Dialog 15(4), 303–318 (2012) 15. Hooper-Greenhill, E., Moussouri, T.: Researching Learning in Museums and Galleries 1990– 1999: a Bibliographic Review. University of Leicester, England (2000) 16. ICOM Homepage. https://icom.museum/en/resources/standards-guidelines/museum-defini tion/. Accessed 21 Nov 2022 17. Kwok-Bun, C., Peverelli, P.J.: Cultural hybridization: a third way between divergence and convergence. World Futures 66(3–4), 219–242 (2013) 18. Leslie, D., Catungal, J.P.: Social justice and the creative city: class, gender and racial inequalities. Geogr. Compass 6(3), 111–122 (2012) 19. Maranda, L.: Decolonization within the museum. ICOFOM Study Ser. 49–2, 180–195 (2021) 20. Orloff, C.: Museums of Cities and the Future of Cities, City Museums and City Development, pp. 27–39 (2008) 21. Rascoff, S.J.: The law of homegrown (counter) terrorism. Tex. L. Rev. 88, 1715 (2009) 22. Saeji, C.T.: Creating regimes of value through curation at the national museum of Korea. Acta Koreana 17(2), 609–637 (2014) 23. Sam, D.L., Vedder, P., Ward, C., Horenczyk, G.: Psychological and Sociocultural Adaptation of Immigrant Youth. In: Immigrant youth in cultural transition, pp. 119–143. Routledge, Milton Park (2006) 24. Sauvage, A.: To be or not to be colonial: museums facing their exhibitions. Culturales 6(12), 97–116 (2010) 25. Schachter, E.P., Rich, Y.: Identity education: a conceptual framework for educational researchers and practitioners. Educ. Psychol. 46(4), 222–238 (2011) 26. Skydsgaard, M.A., Møller Andersen, H., King, H.: Designing museum exhibits that facilitate visitor reflection and discussion. Museum Manage. Curatorship 31(1), 48–68 (2016) 27. Song, K.: Immigrant parents’ ideological positioning on bilingualism. Theor. Into. Pract. 58(3), 254–262 (2019) 28. Taplin, D.H., Clark, H., Collins, E., Colby, D.C.: Theory of change. Technical papers: a series of papers to support development of theories of change based on practice in the field. Act Knowledge, New York City (2013) 29. Taylor, E.W., Neill, A.C.: Museum education. J. Museum Educ. 33(1), 23–32 (2008) 30. Tisdale, R.: From the guest editor: city museums and urban learning. J. Museum Educ. 38(1), 3–8 (2013) 31. United nations department of economic and social affairs, Population Division.: World Population Prospects 2022: Summary of Results. UN DESA/POP/2022/TR/NO. 3 (2022) 32. United nations.: More than half of world’s population now living in urban areas, UN survey finds, https://news.un.org/en/story/2014/07/472752. Accessed 11 Feb 2023 33. Vergo, P.: The reticent object. The new museology, pp. 41–59 (1989) 34. Pavlenko, A., Blackledge, A.: Negotiation of identities in multilingual contexts (Clevdon). Multilingual Matters (2003) 35. Roberts, L.C.: Educators on exhibit teams: a new role, a new era. J. Museum Educ. 19(3), 6–9 (1994) 36. Tessmer, M., Richey, R.C.: The role of context in learning and instructional design. Educ. Tech. Res. Dev. 45(2), 85–115 (1997)

Motivational Factors for College Success: A Focus on First-Generation and Immigrant Students Meitong Lu

Abstract This review paper aims to synthesize and evaluate the literature on the role of motivation in student success in college. Based off the self-determination theory (SDT) and cultural mismatch theory, it explores the motivational factors that seem to be especially important predictors of success for immigrant and first-generation college students (FGS) at four major dimensions: student, teacher, content, and environment. Specifically, the following sub-factors on each level are found to be associated with these non-traditional attendees’ success in college: Student’s demographic factors such as socioeconomic-status (SES) and race/ethnicity, teacher’s socioemotional competence, expectation for student achievement and teaching expertise, learning content’s level of challenge and its social-cultural connection with learner background, and family and/or home environment’s level of support they can provide with college students. Importantly, the SES and racial/ethnic factors may explain why students among the targeted population have lower levels of learning motivation due to social and cultural barriers induced by the cultural mismatch from their low-SES and ethnic-minority identity. Preceding the review synthesis, suggestions on how schools and educators can improve motivation for low-motivated FGS and immigrant students is provided; specific directions on how researchers can conduct future studies on the learning experience of the ethnically diverse students (e.g., FGS, immigrants) is also discussed. Keywords College success · Cultural mismatch · First-generation Student · Immigrant · Motivation · Self-determination theory

M. Lu (B) New York University, New York City, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_27

343

344

M. Lu

1 Introduction Motivation is a stimulus, need, or desire that prompts a person to behave in certain ways to achieve their goals [1]. It is also defined as a bi-dimensional attribute that includes extrinsic motivation, which is driven by external rewards, and intrinsic motivation, which is driven by intrapersonal incentives and interests [2]. Several studies also examined the linkages between motivation and college students’ academic success, which is deemed by many researchers as a primary indicator for college success [1, 3–6]. While some studies have explored the motivational factors of firstgeneration students (FGS) in relation to their non-FGS peers [3, 5], others investigated the motivators that are significantly relevant to collegians from immigrant and/ or minority families [4, 6]. Those literature essentially focused on how considerable barriers can become a source of motivation for FGS and immigrant students to pursue their personal and academic goals in college, but they did not focus on the motivators that are unique to FGS and immigrant students’ college success.

1.1 Immigrant and First-Generation Student Background Generally, immigrant and FGS are a significant proportion of the total student population in the U.S. higher education institutions. As of 2016, FGS consist of 56% of undergraduate population in the United States [7]. Compared to their NFGS peers, FGS are likely to score lower in standardized tests (e.g., GRE, ACT), have lower GPA, lower household income, and are less likely to be employed full-time upon graduation from college [5]. The study also found that, even after controlling for demographic factors such as gender, race, GPA, and family income, the FGS population still has a much higher probability than NFGS (i.e., 71%) to drop out from college [5]. As of 2018, immigrant students consist of 29% of all attendees in U.S. public higher education institutions [8]. For all college attendees enrolled, 43% of immigrant students are Latino, 13% are Black, and 22% are identified as AAPI (Asian American and Pacific Islander) [8]. Several studies found that FGS and immigrant students are less likely to be successful compared to their non-immigrant peers, in terms of academic and socialization experiences, because the former face considerable challenges due to the cultural mismatch between their home culture and the school environment [9–11]. The current review aims to answer the following questions: First, what are the motivational factors that seem to be especially important to FGS and immigrant college students’ success? Second, based off the current review, what are some suggestions for post-secondary institutions to improve learning motivation for these non-traditional attendees?

Motivational Factors for College Success: A Focus on First-Generation …

345

2 Theoretical Framework Numerous contemporary theories explain the connection between motivation and achievement, including attribution theory, goal-orientation theory, social-cognitive theory, expectancy-value theory, and self-determination theory [12]. Attribution theory essentially examines an individual’s attribution to, or “subconscious causal explanations” of event outcomes [12]. Attribution is influenced primarily by factors of locus, stability and controllability. Locus means the source of the cause — whether it is internal or external to the learner; stability refers to whether the locus is fixed or flexible; controllability refers to whether the cause is beyond the individual’s control or not [12]. Goal-orientation theory focuses on two types of motivation goals: Mastery goals that are related to a growth mindset, as well as performance goals, which are related to a fixed mindset. According to this theory, desirable academic outcomes is linked with a learner’s orientation towards the mastery of content, instead of towards the demonstration of their competence on the content. A focus solely on performance undermines both learner’s personal interest (or intrinsic motivation) towards and deep learning of the content [12]. Social-cognitive theory emphasizes the role of interpersonal interactions and environmental response on people’s learning and achievement motivation. According to this theory, a person needs both self-regulation and self-efficacy to be successful. Self-efficacy refers to one’s subject evaluation about their competence, and selfregulation is a cycle whereby a learner applies that self-evaluation into their process of goal realization [12]. Expectancy value-theory (EVT) places an emphasis on an individual’s expectation of achievement and perception of task value. Specifically, the theory explores the degree to which one expects to succeed if they work hard towards a goal. It also explores the extent to which one finds a task interesting (i.e., intrinsic interest), as well as how one perceives a task important to goal achievement [12]. This review essentially focuses on the self-determination theory and the cultural mismatch theory to describe motivational factors related to the success of FGS and immigrant college students. Self-determination theory suggests that a person’s desire for success stems from their intrinsic motivation, which is generated by factors including competence, autonomy, and relatedness: Competence refers to one’s perception of their ability to succeed, autonomy concerns one’s ability to control their own behaviors related to their personal goals, while relatedness refers to one’s need to connect with other individuals in order to find a sense of belonging in the process of working toward that particular goal [12, 13]. Although SDT places a heavy emphasis on competencerelated motivation, it sees all three components as important for human success [14].

346

M. Lu

2.1 Self-determination Theory Specifically, the self-determination theory sees human motivation as a continuum from amotivation, extrinsic motivation, to intrinsic motivation. Each of the three motivation types also corresponds with certain regulatory styles and regulation processes [13]. Importantly, although the introjected regulation process falls under extrinsic motivation, a person at this stage begins to consider internal rewards as a motivational source. This explains that, when an individual is internally motivated enough to consider personal interest and self-actualization as an impetus to achieve their goals [14], they can possibly transition extrinsic motivation into intrinsic motivation. Therefore, SDT seems to be highly relevant in the present context for the following reasons: While different motivation theories vary in their prediction of success, emotions, and perseverance in various contexts associated with competence and achievement, SDT distinguishes different types of self-regulation and motivation [14]; second, in its relatedness and competence components, SDT emphasizes the role of sociocultural context that is used as a key motivator for foreign-born, immigrant students’ pursuit of college achievement [13].

2.2 Cultural Mismatch Theory of Inequality Thus, this review also attempts to explain motivational factors from the perspective of the cultural mismatch theory of inequality. According to Stephens & Townsend [11], cultural mismatch on a U.S. campus happens when: “U.S. institutions tend to promote mainstream, independent cultural norms, and exclude interdependent cultural norms that are common among underrepresented groups; when institutions promote only mainstream norms, they inadvertently fuel inequality by creating barriers to the performance of underrepresented groups” (p. 129).

The cultural mismatch between the interdependent cultural norm of FGS and minority students and the independent cultural values of the U.S. mainstream culture creates many challenges for the non-traditional collegians, which may increase their level of stress and lowering their comfort level at college, leading to underperformance in both their academic and socialization experiences [4]. Furthermore, Batalova and Fledblum [8] acknowledged that while immigrants receive the same benefits and rights as their U.S.-born students with native parents, their “social and economic trajectories are tied to those of their immigrant families (p. 2)”. To deepen our understanding of FGS and immigrant students, it is meaningful to discuss the predictors their college success with relation to the cultural mismatch theory.

Motivational Factors for College Success: A Focus on First-Generation …

347

3 Literature Review Although a number of studies have examined motivators associated with college success [1, 15–19], surprisingly few explored the factors that are especially important to FGS and immigrant collegians. Schweinle and Helming [19] discovered three motivators important for undergraduate students: academic goals and motivation, challenging tasks, and previous experience with success. Mushtaq and Khan [18] acknowledged that student performance, communication, learning facilities, proper guidance, as well as family stress serve as factors for student motivation and success. Other studies that explore the linkages between gender and motivation for college student also considered race/ethnicity, SES, and classroom context [16, 17]. Additionally, Anderman [15] discusses the role of motivation concerning domain specificity and attempts to explain the differences in motivation of students across different subject-areas. While the aforementioned studies have applied quantitative, qualitative, and mixed-methods to explore the association between student motivation and achievement, the methodology of those studies will not be under discussion at great length in the current review paper, both due to limited space, as well as its primary focus on synthesizing the major motivational factors that are related to FGS and immigrant students’ success in college. The current review purports to explore important motivators for the targeted population on four general motivators: student, teacher, and learning content, as well as the home and school environment. For each general motivator, sub-factors relating to FGS and immigrant students’ learning motivation and college success will be discussed in detail.

3.1 General Motivators This review aims to synthesize FGS and immigrant college students’ motivational factors in four main domains: Student, teacher, content, as well as the home and/ or school environment [1]. Student-level factors include demographic factors such as gender, race/ethnicity, and SES; academic goal, student performance, and prior experience with success should also be considered [16–19]. Teacher-level motivators mainly consist of teacher’s professionality in their field of expertise, socialemotional competence, expectations for student achievement [1]. Content-level motivators mainly include domain specificity and the challenge of learning tasks [1, 15]. Environmental factor concerns both home and school environments, which concerns the level of support and resources they provide students to ensure their success in college learning.

348

M. Lu

3.2 Student-Level Motivators Meece and Askew [16] and Meece et al. [17] found that gender plays an important role in college students’ motivation in learning and success. Meece et al. [17] found that individuals seem to be influenced by gender stereotypes and behave accordingly at an early age. When factors such as race/ethnicity, SES, classroom environment, and ability are moderated, male students tend to report more confidence and stronger ability in learning math and science, whereas female students report higher selfefficacy and stronger performance in writing and language arts [17]. Both studies agreed that gender role is usually perpetuated by three parties — parents, educational institutions, and the society (e.g., sociocultural influence) — which deepen students understanding of their gender and form the belief that they should be learning or are expected to specialize in certain subjects [16, 17]. As such, SDT maintains that gender stereotypes from family, school, and the society externally motivates students in their choice of major field of study; due to existing gender norms, they may not fully explore their learning motivation if rely solely on extrinsic factors. Thus, outside forces should not impose additional stress on students’ choices, which may undermine their authentic performance in tasks or assessments. Previous studies also considered the influence of SES and racial/ethnicity on the accessibility and affordability to higher education [3, 6]: lowSES and immigrant students face many challenges in obtaining a college degree associated with academic-related factors, which include previous experience with success, academic performance and goals. Specifically, education goals and prior experience with academic success, serve as moderators that are linked with FGS and immigrant students’ college success [3, 6, 13, 16–18]. Blackwell and Pinder [3] especially explores the motivational factors on former immigrant (i.e., third-generation college students) and FGS who are African American women. Three causal conditions shared across all FGS participants: A passion for reading and learning at an early age, each respondent felt different from their siblings at an early age due to early passion in reading, and a pursuit for a higher quality of life (p. 50). Each FGS participant expressed that they see college diploma as a gateway to promote social mobility and help improve their family’s situation in the long run. According to SDT, the participants are internally motivated to obtain a higher education degree that will greatly enhance their sense of competence as the first in their family to fulfill this accomplishment. Urdan [6] essentially identifies several sociocultural factors concerning the cultural misalignment between the values of the immigrant family/culture with the host country culture: family resources and context; school resources and context; effects of acculturation; cultural identity and norms; host country/community’s attitude towards immigrants (p. 298). Concerning the cultural mismatch theory of inequality and its relation with FGS and immigrant students’ learning motivation in college, the aforementioned factors will be discussed in more detail in subsequent sections.

Motivational Factors for College Success: A Focus on First-Generation …

349

3.3 Teacher-Level Motivators Teacher’s expectation for students’ performance, professionality in their field of expertise, as well as their social-emotional competence are significantly linked with student motivation [1]. Teachers’ proper guidance, effective communication, and meaningful rapport with students are important to their academic success in college [18]. Thus, effective teachers need to be enthusiastic about teaching, be supportive on student learning, and hold high expectations for their students’ academic achievements [1, 18]. Generally, teachers can two types of support to their students, emotional support and academic support. Teachers’ support is crucial to students because their positive attitudes toward non-traditional attendees, expertise in their field of specialty, and their socio-emotional competence are associated with whether immigrant and FGS can integrate this positive external factor into intrinsic motivation when pursuing a higher degree in an overseas institution. Additionally, if an instructor is knowledgeable about the field and is highly motivated to teach, students will have higher motivation than instructors who lack professionality and enthusiasm in teaching. During the teaching process, teachers also need to apply differentiated, individualized, and/or personalized model of instruction that attends to all students’ needs. To reach this goal, they should build meaningful rapport with students to enable the class routines and activities go more “smoothly.” Where there is a close relationship between the teacher and student, it is easier for the teacher to implement instruction activities as planned. Consequently, it will promote student motivation in learning and will likely lead to provide better academic outcomes.

3.4 Content-Level Motivators Previous literature also explored student motivation across different subject areas, with a focus on the academic and instructional approaches of each domain; others also considered the level of challenge of specific learning tasks [1, 6, 15]. Anderman [15] pointed out that although several studies have focused on exploring motivation factors in more general terms, few especially focused on domain context concerning motivation. Thus, we need to consider motivation with regards to each academic domain’s “cultural” aspects, including “cultural norms for how different subjects are taught, what constitutes participation and learning, and how progress is evaluated (p. 284).” Likewise, Urdan [6] found that appropriate level of academic challenge is positively associated with students’ motivation in learning. However, neither too much nor too little challenge is helpful in enhancing student motivation because they would not benefit student’s learning incentive in healthy ways As such, moderate level of challenge may provide students with adequate amount of extrinsic motivation while

350

M. Lu

this challenge may not interfere with the student’s personal interest towards particular academic tasks. According to SDT, teachers should consider students’ basic psychological needs, such as always allowing them to feel a sense of control, competence, and relatedness to promote their motivation in the learning process [1]. While offering learning materials, it is important to offer a few alternatives for students to choose between, so that they will have a sense of control in their learning process. Therefore, teachers should provide choices for students as a way of differentiation. It is also important that the tasks assigned are not competent or punitive in nature; they should allow students to make immediate connections to real life and their prior experiences. Especially for ethnically diverse students such as immigrants and first-generation students, access to diversity-related content is an important instrument that would promote their learning motivation.

3.5 Environment-Level Motivators Family environment is another level of factor that is associated with FGS and immigrant students’ success in college. Sub-factors such as family involvement, parents’ education level, and family expectation are found to be significantly associated with undergraduates’ accessibility to college and realization of educational goals [4, 5, 18, 20]. In the family environment, parents’ accessibility to information about college is important because they can help their children prepare for college challenge in advance. For FGS, however, they generally lack this access at home because their parents did not have experience in college learning. Therefore, poor parental guidance or communication between child and parent may provide a source of stress for FGS, because they might come to college unprepared compared to their NFGS (non-firstgeneration student) counterparts [5, 18, 20]. Specifically, family expectations may be demonstrated at two levels - choice of school and choice of major. Compared with traditional college attendees, parents of minority students are more likely to expect their children to attend high-ranking colleges and specialize in specific fields mainly for the purpose for them to secure a socially-respectable occupation in their home countries upon graduation. Specific choice of major may include but not limited to: STEM majors, economic-related fields (e.g., financial economics), and medical-related fields (e.g., dentistry). Likewise, parental expectation on the choice of school mainly demonstrates on the ranking of the institution, as well as the reputation of the school and major in their home country based off the level of academic rigorousness and the graduates’ career potentials [4]. The school environment is also found important with relation to college students’ learning motivation [18]. Ideally, a high-quality educational environment should ensure safe commute to schooling, accessibility to diverse, state-of-the-art learning facilities, and a supportive classroom atmosphere are essential for FGS and immigrant

Motivational Factors for College Success: A Focus on First-Generation …

351

students to feel comfortable and motivated to learn. Moreover, all faculty and staff are responsible for co-creating a diversity-friendly environment for non-traditional college attendees [1]. On the students’ end, they will need to be active agents instead of passive recipients of knowledge in their process of learning. Essentially, the school environment can greatly enhance student motivation when they emphasize individual learning differences, engage students with teamwork, respect students’ opinions, and when they empower students with emotional support [15]. Teachers should also come to class prepared and be adept at applying the available tools in their instructional processes. In sum, schools’ positive and supportive attitudes toward immigrant and FGS are beneficial for their learning and academic performance [1]. Likewise, family environment can also create a positive impact for FGS and immigrant students’ learning motivation by respecting students’ choice of major and schools, and by actively engaging in learning about students’ college experience.

4 Synthesis of Findings Above all, previous studies identified motivators for FGS and immigrant college students at mainly four dimensions: student, teacher, content, and the home/school environment. Student factor mainly consists of students’ SES, race/ethnicity, and gender; tt also includes academic-related sub-factors such as students’ educational goals and academic expectations, as well as their previous experience with academic success. Next, teacher factor related to students’ learning motivation concerns their expertise in the subject, social-emotional competence while interacting with students, and their expectation for students’ academic performance. Furthermore, the content factor that related to students’ learning motivation regards the integration of sociocultural aspects within the learning materials, as well as the use of moderately challenging course materials. Lastly, the environmental factors such as the support and resources available from both the home environment and school environment, seem to be especially important for FGS and immigrant students’ motivation to learning.

5 Conclusions and Implications for Future Research For future studies on the college experience of non-traditional attendees, more indepth case studies should be conducted through inquiry methodologies such as the phenomenological or narrative approach. The primary focus should be on the “what” and “how” questions: a) What are the lived experiences of ethnically- diverse college attendees like the FGS and immigrant students? b) How would they like to improve their college experiences?

352

M. Lu

Gaining insights to these aspects would help us understand the perceptions and expectations of an ideal and effective college experience of the minority, immigrants, and FGS. While recording and interpreting students’ stories, researchers should discard possible presumptions or bias to remain objectivity and authenticity of the study findings. After all, our goal is to look into the differences between the college experience of the minority student population and their domestic peers, so that we can strive to cocreate a more inclusive intramural environment through a student-oriented approach. It is our hope that this collaborative effort will make all non-traditional college attendees feel safe and confident to survive and thrive in a learning environment that is oftentimes vastly different from their home environment.

References 1. Williams-Pierce, C.C., Williams-Pierce, C.K.: Five key ingredients for improving student motivation. Res. High. Educ. J. 11, 1–23 (2011) 2. Beffa-Nefrini, P.A., Cohen, N.L., Miller, B.: Strategies to motivate students in online learning environment. J. Nutrition Educ. Behav. 34(6), 334–340 (2002). https://doi.org/10.1016/S14994046(06)60116-4 3. Blackwell, E., Pinder, P.J.: What are the motivational factors of first-generation minority college students who overcome their family histories to pursue higher education? Coll. Stud. J. 48(1), 45–56 (2014) 4. Jones, L., Castellanos, J., Cole, D.: Examining the ethnic minority student experience at predominantly white institutions: a case study. J. Hisp. High. Educ. 1(1), 19–39 (2002). https:// doi.org/10.1177/1538192702001001003 5. Próspero, M., Vohra-Gupta, S.: First generation college students: motivation, integration, and academic achievement. Commun. Coll. J. Res. Pract. 31(12), 963–975 (2007). https://doi.org/ 10.1080/10668920600902051 6. Urdan, T.: Factors affecting the motivation and achievement of immigrant students. In: Harris, K.R., Graham, S., Urdan, T., Graham, S., Royer, J.M., Zeidner, M (eds.) APA E293–313. American Psychological Association (2012). https://doi.org/10.1037/13274-012 7. First generation college students: demographic characteristics and postsecondary enrollment. National Association of Student Personnel Administrators. Center for First-generation Student Success (2020). https://firstgen.naspa.org/ 8. Batalova, J., Feldblum, M.: Immigrant-origin students in U.S. higher education. Migration Policy Institute (2020) 9. Kigel, R.M., McElvany, N., Becker, M.: Effects of immigrant background on text comprehension, vocabulary, and reading motivation: a longitudinal study. Learn. Instr. 35, 73–84 (2014). https://doi.org/10.1016/j.learninstruc.2014.10.001 10. Phillips, L.T., Stephens, N.M., Townsend, S.S.M., Goudeau, S.: Access is not enough: Cultural mismatch persists to limit first-generation students’ opportunities for achievement throughout college. J. Pers. Soc. Psychol. 119(5), 1112–1131 (2020). https://doi.org/10.1037/pspi0000234. supp 11. Stephens, N.M., Townsend, S.S.M.: The norms that drive behavior: implications for cultural mismatch theory. J. Cross Cult. Psychol. 46(10), 1304–1306 (2015). https://doi.org/10.1177/ 0022022115600264 12. Cook, D.A., Artino, A.R.: Motivation to learn: an overview of contemporary theories. Med. Educ. 50(10), 997–1014 (2016). https://doi.org/10.1111/medu.13074

Motivational Factors for College Success: A Focus on First-Generation …

353

13. Ryan, R.M., Deci, E.L.: Intrinsic and extrinsic motivation from a self-determination theory perspective: Definitions, theory, practices, and future directions. Contemp. Educ. Psychol. 61, 101860 (2020). https://doi.org/10.1016/j.cedpsych.2020.101860 14. Ryan, R.M., Moller, A.C.: Competence as central, but not sufficient, for high-quality motivation: a self-determination theory perspective. In: Handbook of Competence and Motivation (2017) 15. Anderman, L.H.: Contemporary issues on motivation introduction: student motivation across subject-area domains. J. Educ. Res. 97(6), 283–286 (2004). https://doi.org/10.3200/JOER.97. 6.283-286 16. Meece, J.L., Askew, K.J.S.: Gender, motivation, and educational attainment. In: Harris, K.R., Graham, S., Urdan, T., Graham, S., Royer, J.M., Zeidner, M (eds.) APA Educational Psychology Handbook, vol. 2, pp. 139–162. American Psychological Association (2012). https://doi.org/ 10.1037/13274-006 17. Meece, J.L., Glienke, B.G., Brug, S.: Gender and motivation. J. Sch. Psychol. 44, 351–373 (2006). https://doi.org/10.1016/j.jsp.2006.04.004 18. Mushtaq, I., Khan, N.S.: Factors affecting students’ academic performance. Global J. Manag. Bus. Res. 12(9), 17–22 (2012) 19. Schweinle, A., Helming, L.M.: Success and motivation among college students. Soc. Psychol. Educ. 14(4), 529–546 (2011). https://doi.org/10.1007/s11218-011-9157-z 20. Grant, R.O.: The role of parental and family involvement in the persistence of freshman firstgeneration college students [ProQuest Information & Learning]. In: Dissertation Abstracts International Section A: Humanities and Social Sciences, vol. 82, no. 1–A (2021)

Perceptions of Online Strategies and Digital Readiness in the COVID-19 Environment: An Instrumental Case Study Pamela McCray, Valentina Rada, Steven A. Szeszko II, Norman St. Clair, and Kevin B. Vichcales

Abstract The COVID-19 pandemic has significantly impacted higher education, forcing institutions to rapidly transition from traditional face-to-face (F2F) learning to online learning environments. This sudden shift has presented significant challenges for graduate students, faculty, and administrators because of limited training and support for implementing innovative strategies and technologies. These challenges lead to an imbalance in the operational environment, disrupting classroom instruction and creating anxiety from the additional workload supporting complex and uncertain environments. The design for this research project was an instrumental case study focusing on one private University in South Central Texas designated as a Hispanic Serving Institution (HIS). Our population for this case included University graduate students, faculty, and administrators in the school of education from the Summer of 2021 through the Spring of 2022. Data were collected and analyzed using Miles and Huberman’s code-to-theory framework to identify categories, themes, and develop a Comprehensive Institutional Model (CIM) to support the transition to remote and blended modalities in graduate education. This instrumental case study also highlights the need for universities to invest in reliable and up-to-date Information Communication Technology (ICT) platforms, provide accessible training, and develop sustainability strategies and plans. Ultimately, the success of online learning depends on instructional support, instructor presence, and engagement. Universities must work to ensure that all stakeholders are ready for teaching and learning in this new environment. Keywords Digital readiness · Blended teaching modalities · Hy-flex modalities · Online learning · COVID-19 · Emergency remote teaching (ERT) · Instructional support

P. McCray (B) · V. Rada · S. A. Szeszko II · N. St. Clair · K. B. Vichcales University of the Incarnate Word, San Antonio, TX 78200, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_28

355

356

P. McCray et al.

1 Introduction The COVID-19 pandemic presented the opportunity for educational institutions to explore gaps in online strategies and digital readiness for graduate students, faculty, and administrators in higher education during COVID-19 Emergency Remote Training (ERT). The sudden shift from traditional face-to-face (F2F) teaching to online learning posed several pedagogical and technological challenges to higher education institutions [14, 20]. Most educational institutions had to adopt innovative technologies with limited faculty training and support [14, 20].

1.1 Challenges and Opportunities of Online Graduate Education The sudden shift to remote learning exposed the limitations of faculty and graduate student readiness and preparation for teaching and learning in digital environments [6, 7, 20]. Most universities were not equipped to support digital learning and blended modalities, leading to an imbalance in the operational environment that favored the F2F modality [15]. Many faculty members struggled with instructional technology proficiency and online communication practices, leading to a significant workload, additional training requirements, and anxiety [11, 17]. Faculty development and training programs were lacking to help the teaching community learn how to design and apply online teaching techniques and pedagogy to keep their learners engaged and supported [15, 21, 24]. The complexities and breadth of the changes necessitated by the pandemic call for further research to understand the educational redesign opportunities that have arisen with the transition to remote, blended, and Hy-flex modalities in graduate education [5, 20]. The impact on graduate students, faculty, and administrators is significant as educational institutions try to balance redesign strategies that meet the needs of all stakeholders while transitioning to a digital learning environment [19]. Graduate students also faced challenges adapting their learning behavior to the new digital platforms and technologies, including the lack of instructor and classmate social interaction and the need for self-discipline and time-management skills [3, 13]. The success of online learning hinged on instructional support, instructor presence, and engagement [13].

1.2 Modality Transition Challenges for Graduate Students Online courses require more self-discipline and time-management skills from a student perspective than traditional F2F courses [3, 13]. Current studies report both student advantages and disadvantages of online learning. The advantages include

Perceptions of Online Strategies and Digital Readiness in the COVID-19 …

357

efficiency, cost-effectiveness, and 24 hour access. The disadvantages include technical challenges, lack of clarity on course expectations and assessments, and the lack of instructor and classmate social interaction [3, 13]. Foerderer et al. [13] found instructional support had the biggest impact on student satisfaction with online learning during the COVID-19 ERT. Instructor presence and engagement were critical, and many graduate students looked to their instructors for emotional support and encouragement during this period [3, 13, 21]. Boški´c and Hausknecht [5] offer an exemplary case of a large Canadian university that restructured processes, reorganized its workforce, and invested in support resources to address institutional constraints in response to COVID-19. The University leveraged its pre-COVID-19 competencies of remote learning and the learning management system (LMS) usage while investing in course recording equipment, technology-enhanced classrooms, and technical resources [5]. The University used available staff to assist in support roles and hired five learning designers to assist faculty in building digitally enabled courses and hybrid offerings [5]. Therefore, universities must prepare before the next crisis by investing in reliable and up-to-date Information Communication Technology (ICT) platforms and technology-enabled classrooms, providing accessible training, and developing sustainability strategies and plans [5, 9, 20].

1.3 Online Learning Challenges, Opportunities, and Effectiveness The COVID-19 pandemic has forced educational institutions worldwide to adapt to new modes of teaching and learning. This event has brought forth several challenges and opportunities. Adedoyin and Soykan [1] explored these challenges and opportunities for online learning. The effectiveness of online learning depends on several factors, such as learning styles, course satisfaction, and course achievement. Al-Azawei et al. [2] investigated the effect of learning styles on a blended e-learning system and found that the Technology Acceptance Model (TAM) could be extended to incorporate learning styles. Cheng and Chau [8] examined the relationships between learning styles, online participation, learning achievement, and course satisfaction in a blended learning course. The study highlighted the importance of considering students’ learning styles when designing online courses. Beteille et al. [4] proposed three principles to ensure teacher effectiveness during the pandemic: prioritizing learning goals, ensuring equity, and supporting teacher professional development. Gurukkal [16] questioned whether the pandemic would turn higher education into another mode, emphasizing the need to consider the challenges and opportunities for effective implementation.

358

P. McCray et al.

2 Purpose, Scope, and Audience The purpose of the research was to explore the experiences and insights of key stakeholders (graduate students, faculty, and administrators) about integrating and relying on a combination of online, blended, and Hy-flex strategies for teaching and learning in the COVID-19 environment and beyond at one private University in South Central Texas. We believe our study will contribute insights to institutional leaders about best practices that support faculty and graduate student instructional and learning needs by providing necessary classroom infrastructure, instructional technology, and training. Our study focused on graduate and doctoral-level programs in the University’s school of education from the Summer of 2021 through the Spring of 2022. Our study aims to inform institutional leaders, such as administrators and faculty, who are interested in best practices for supporting faculty and graduate student instructional and learning needs in a blended and Hy-flex learning environment. The study’s findings can also benefit graduate students transitioning to digital learning modalities and requiring additional support.

2.1 Research Question The study’s primary research question was: How did graduate students, faculty, and administrators adjust to the rapid transition and impact of moving from F2F to remote teaching and learning during the COVID-19 ERT?

2.2 Design and Methodology The study design was an instrumental case study [10, 23] focused on one private University in South Texas designated as a Hispanic Serving Institution (HIS). We followed Miles and Huberman’s [18] code-to-theory approach for data collection, coding, and interpretation of our findings. Multiple data sources were explored. These sources include: semi-structured interviews with faculty and administrators, a doctoral student questionnaire, a doctoral student focus group, institutional archive documents, website metadata, and classroom sourcing using observation and content analysis (Fig. 1). All data collected were used in the data analysis process.

2.3 Participant Selection and Sample Size To select participants for our study, we used purposive sampling to ensure we recruited graduate students, faculty, and administrators from one private University

Perceptions of Online Strategies and Digital Readiness in the COVID-19 …

359

Fig. 1 The case study data collection and analysis process

in South Texas designated as a HIS. The study focused on the school of education and included students at the graduate and doctoral levels enrolled in remote and Hy-flex learning modalities during the 2021–2022 academic year. We also included faculty who taught graduate and doctoral courses in remote and Hy-flex modalities during the same academic year. The study also included administrators responsible for academic oversight, instructional technology design, and faculty professional development at the institutional level. Likewise, snowball sampling and recruiting were used to identify potential participants. The specific criteria for selecting participants were based on their experience with and knowledge of Hy-flex modalities and online learning. The sample sizes included seven students in a focus group, 15 students enrolled in a summer 2021 course, two students who responded to the questionnaire, three faculty members, and six administrators.

2.4 Data Collection and Instrumentation Data were collected through a variety of methods, including a questionnaire, classroom observations, discussion forum content analysis, semi-structured interviews, a focus group, and archival documentation. Classroom observations were conducted during a semester-long course, and researchers analyzed the classroom environment, student engagement, and technology use. All data collection methods were chosen to provide a comprehensive understanding of the participants’ experiences and perceptions.

360

P. McCray et al.

2.5 Data Analysis The data were analyzed using coding and categorizing techniques to identify related themes and theoretical insights. Saldana’s [22] coding strategies and Miles and Huberman’s [18] code-to-theory conceptual model were used to identify institutional digital readiness at the graduate level for teaching and learning using Hy-flex and remote modalities. In-Vivo coding strategies were used to identify language patterns within the data, and conceptual insights were identified through questionnaires and interviews. Emerging categories were synthesized into themes supported by theoretical sampling for online strategies and digital readiness requirements.

2.6 Data Validation The study’s data were validated using several methods, including theoretical sampling and member checking. Researchers continuously triangulated their findings with one another, and member checking was conducted with selected participants to clarify responses and ensure their input was accurately captured. Dworkin’s [12] framework was used to reach saturation of repeating patterns and concepts within the data. Independent research analysis and triangulation of findings supported quality assurance.

2.7 Ethical Considerations The study followed the institutional IRB process and CITI certification for each researcher. The University’s standardized IRB forms were used for recruiting participants and accessing institutional data. Consent forms were provided to each interviewee to ensure the study’s transparency, confidentiality, and opt-out options. The University’s IRB approved this study’s research design, protocols, and documents to protect the rights and welfare of research subjects, according to 45 CFR 46.111. The IRB approval was obtained on September 23, 2021, and the approval number was 21–09-003.

3 Findings Following Miles and Huberman’s framework [18], our analysis yielded categories and themes from each participant grouping of graduate students, faculty, and administrators. These categories and themes exposed gaps and support needs essential for effective teaching and learning within blended and diverse modalities. Faculty

Perceptions of Online Strategies and Digital Readiness in the COVID-19 …

361

and graduate student success necessitates support from institutional leadership. The prioritization of faculty training to understand how to use the best pedagogical and course design practices in conjunction with the institutional learning management system is vital. A student-centered approach emphasizing flexibility and support and investing in appropriate classroom instructional technology infrastructure is essential. In this section, we will focus on the four themes that resulted from our analysis of data which were: a) lack of faculty readiness, acceptance, and adaptability; b) need for more investment and training to ensure sustainable quality of classroom technology and instructor effectiveness in hybrid modalities; c) graduate students’ support needs for academic success; and d) graduate students’ expectations of faculty readiness.

3.1 Theme 1: Lack of Faculty Readiness, Acceptance, and Adaptability This theme represents an analysis of data collected from graduate students, faculty, and administrators. The results emphasized the importance of faculty receiving appropriate training in the move to Emergency Remote Training (ERT), which included using the LMS effectively, reliable classroom learning technology infrastructure, and administrators’ support to ensure faculty training provided continuity for quality graduate student instruction and learning. Categories emerged within this theme by triangulating data from graduate student focus group interviews, graduate student questionnaires, faculty interviews, administrators’ interviews, and institutional documents. These categories included: a) shifting teaching modalities and confusion; b) faculty technology acceptance and adoption; and c) educational infrastructure and operational support.

3.2 Theme 2: Quality of Classroom Technology and Instructor Effectiveness While this theme represents an analysis of all three groups, the graduate student group stressed the importance of reliable and quality instructional technology in the classroom, a consistent learning management system, and faculty readiness to deliver content effectively within Hy-flex and hybrid learning environments. Without the critical support from leadership to ensure and facilitate faculty readiness, graduate students expressed deep concern about the implications and impact of their learning. In addition, flexibility and choice were essential for graduate students and faculty to feel safe during this COVID-19 pandemic. The categories within this theme include: a) reliable and consistent classroom technology; b) prepared faculty; and c) vision and strategic plan.

362

P. McCray et al.

3.3 Theme 3: Graduate Students’ Support Needs for Academic Success This theme centers on addressing adult learners’ unique challenges and needs, who often require greater flexibility and support to balance academic responsibilities, work, and life. Graduate students, faculty, and administrators alike stress the importance of ensuring equal learning opportunities for graduate students of all ages, backgrounds, and technological savviness. This theme highlights the need for the University to recognize and accommodate differences among students, including those related to age, maturity levels, and contextual skills. The categories within this theme include: a) flexibility and balance for students; b) diverse adult challenges; and c) social interaction in online and F2F environments.

3.4 Theme 4: Graduate Students’ Expectations of Faculty Readiness This theme revealed the need for innovation in response to the challenges of a changing educational landscape. Graduate students, faculty, and administrators all recognize that changes in technology during the COVID-19 ERT required a shift in mindset and creative adaptation to better support graduate students. This shift includes training and resources for faculty to effectively use learning technologies and accommodate graduate students’ diverse technological skills. It also means setting clear expectations and guidelines for participating in hybrid environments to ensure continuity among instructors, effective communication, and a shared understanding of etiquette and presentation format. All stakeholders must be willing to embrace innovation and work collaboratively to find new ways of teaching and serving different segments. The categories within this theme include: a) faculty innovation mindset and b) hybrid environmental participation expectations.

4 Discussion, Conclusions, and Recommendations Our conclusions follow that universities must be proactive in emergency planning and invest in educational infrastructure and operational support to ensure faculty readiness and adaptability to new teaching modalities. Faculty should receive the necessary training and resources to teach effectively in online and hybrid environments. It is also crucial to ensure that classroom technology is reliable and consistent to support quality online and hybrid learning. Furthermore, universities should enhance faculty course design competencies, use innovative teaching techniques, and communicate effectively with graduate students to support engagement. To

Perceptions of Online Strategies and Digital Readiness in the COVID-19 …

363

promote graduate student success, universities should address diverse adult challenges, learning styles, and instructional technology support. Social interaction in both online and F2F environments should also be promoted to enhance the learning experience.

4.1 Recommendations Our recommendations and next steps for universities include prioritizing flexibility, engagement, accessibility, and support when designing a resilient and adaptable education system that meets the needs of both faculty and graduate students. Institutions should also consider the four guiding Hy-flex principles when designing, building, and implementing courses into their programs. These principles are flexibility, engagement, accessibility, and support. By implementing these recommendations, universities can create a supportive learning environment that addresses the diverse needs of graduate students and faculty. Therefore, we recommend the following Comprehensive Institutional Model (CIM) for universities to support online and hybrid/Hy-flex learning to ensure readiness for new emergencies and the aftermath of a “new normal” in higher education. Specifically, we propose a model that organizes how the alignment of administrators, faculty, and graduate students impact effective online and hybrid/Hy-flex teaching and learning environments. Administrators provide the infrastructure and operational support for graduate students and faculty readiness. Faculty skill development enables instructors to teach effectively in online and hybrid/Hy-flex environments. Graduate student support and assessable tools provide the ability to engage in flexible learning environments. The CIM offers a unique and comprehensive approach to synergizing all layers of the institution. When comparing our model to Boški´c and Hausknecht’s case study findings [5], we followed a more systematic approach involving all institutional stakeholders. We synthesized our findings into a model articulating a step-by-step process, aligning leadership and academic affairs elements that impact graduate education. Administrators’ Recommendations. The administrators’ role in this model is to develop proactive emergency planning for contingencies that require flexible modalities, faculty digital competencies, and pedagogical best practices. 1. Develop Proactive Emergency Planning. a. Develop a comprehensive emergency plan that outlines specific strategies and procedures to be followed in the event of contingencies that require flexible modalities, faculty digital competencies, and pedagogical best practices. b. Ensure the plan is regularly updated and communicated to all stakeholders, including faculty, staff, and graduate students. c. Conduct regular training and simulations to prepare faculty and staff for emergency situations.

364

P. McCray et al.

2. Invest in Educational Infrastructure and Operational Support. a. Ensure the institution has the necessary technological infrastructure to support online and hybrid/Hy-flex learning, such as reliable internet connectivity, a learning management system, and instructional technology tools. b. Provide operational support to faculty and staff, such as technical, pedagogical, and administrative support, to help them adapt to new teaching modalities. Encourage and support using digital tools for faculty peer mentoring to build skills and confidence. c. Ensure that faculty and staff have access to professional development opportunities to enhance their skills and competencies. 3. Prioritize the Needs of Graduate Students and Faculty. a. Develop a clear plan and vision for the future of education that considers the needs and priorities of graduate students and faculty. b. Solicit graduate students and faculty feedback through surveys, focus groups, and other means to inform institutional decision-making. c. Ensure the institution is responsive to graduate students and faculty’s evolving needs and expectations. 4. Promote Social Interaction and Address Diverse Challenges. a. Develop strategies to promote social interaction in all modalities, such as online discussion forums, virtual social events, and in-person meetups. b. Address diverse adult challenges, learning styles, and instructional technology support by providing access to resources and support services, such as tutoring, counseling, and disability services. c. Ensure the institution is inclusive and welcoming to graduate students from diverse backgrounds, cultures, and identities. Faculty Recommendations. The faculty’s role in this model is to teach effectively in online and hybrid environments and ensure that course content is accessible to all graduate students. 1. Provide Necessary Training and Resources. a. Develop training programs that provide faculty with the necessary skills and competencies to teach effectively in online and hybrid environments. b. Ensure faculty are trained to use learning management systems (LMS) and other online and hybrid teaching technology tools. c. Provide faculty access to instructional designers, technical support staff, and other resources to help them design and deliver effective online and hybrid/ Hy-flex courses. 2. Enhance Course Design Competencies. a. Provide faculty training on designing and developing effective online and hybrid courses, including course structure, content delivery, and assessment.

Perceptions of Online Strategies and Digital Readiness in the COVID-19 …

365

b. Encourage faculty to adopt best practices for online and hybrid teaching, such as using multimedia content, interactive activities, collaborative learning, and embedding best practices in digital pedagogy. 3. Use Innovative Teaching Techniques to Promote Engagement. a. Encourage faculty to use a variety of teaching techniques, such as online discussions, group work, and peer-to-peer feedback, to promote graduate student engagement and learning. b. Encourage faculty to use interactive tools like quizzes and polls to promote active participation and feedback from graduate students. c. Provide faculty access to educational technology and tools, such as virtual and augmented reality, to create engaging and immersive learning experiences. 4. Communicate Effectively with Graduate Students. a. Encourage faculty to establish clear expectations and communication protocols with graduate students, including course requirements, deadlines, and modes of communication. b. Provide faculty access to communication tools like discussion forums and video conferencing to facilitate effective communication and collaboration. c. Encourage faculty to provide timely feedback to graduate students on their progress and performance. Graduate Students’ Recommendations. The graduate students’ role in this model is to engage in their learning and take advantage of the resources and support offered by the University. To promote effective graduate student learning, universities should follow the four guiding Hy-flex principles when designing, building, and implementing courses into their programs. These principles are flexibility, engagement, accessibility, and support. 1. Flexibility. a. Offer a variety of course delivery options to accommodate graduate student needs and preferences, such as fully online, hybrid/Hy-flex, or F2F courses. b. Provide clear information about the course format, schedule, and expectations to help graduate students make informed decisions. c. Regularly review and update course delivery options based on graduate student feedback and changing needs. 2. Engagement. a. Use a range of teaching techniques to promote graduate student engagement, such as interactive lectures, group projects, and case studies. b. Provide graduate students with collaboration and discussion opportunities, such as online forums or discussion boards. c. Encourage graduate student feedback and input to improve the course and teaching effectiveness. 3. Accessibility.

366

P. McCray et al.

a. Ensure all course materials and technologies are accessible to graduate students with disabilities, such as providing closed captions for videos or alternate text for images. b. Train faculty to create accessible content and use accessible technologies. c. Provide accommodation to graduate students with disabilities, such as extended time on exams or assistive technology. 4. Support. a. Offer academic support services, such as tutoring, academic advising, and writing centers, to help graduate students succeed. b. Provide mental health counseling services to support the well-being of graduate students. c. Communicate regularly with graduate students to provide information and resources on support services.

5 Summary In summary, our proposed CIM organizes how the alignment of administrators, faculty, and graduate students impact effective online and hybrid/Hy-flex teaching and learning environments. Graduate students should embrace the four guiding Hyflex principles to engage in their learning successfully, while faculty should be provided with the necessary training and resources to teach effectively in online, hybrid, and Hy-Flex environments. Administrators should develop proactive emergency planning and invest in educational infrastructure and operational support to ensure faculty readiness and adaptability to new teaching modalities. They should also prioritize the needs of graduate students and faculty, promote social interaction, and address diverse adult challenges, learning styles, and instructional technology support to encourage graduate student success. While not generalizable, our findings and recommendations from our instrumental case study are transferable and could provide universities with the guidance they need to support quality online and hybrid learning. By implementing these recommendations, we believe that institutions can create a more resilient and adaptable education system that meets the needs of graduate students, faculty, and administrators.

References 1. Adeoyin, O.B., Soykan, E.: Covid-19 pandemic and online learning: the challenges and opportunities. Interact. Learn. Environ. 31, 863–875 (2020) 2. Al-Azawei, A., Parslow, P., Lundqvist, K.: Investigating the effect of learning styles in a blended e-learning system: an extension of the technology acceptance model (TAM). Aust. J. Educ. Technol. 33, 1–23 (2017)

Perceptions of Online Strategies and Digital Readiness in the COVID-19 …

367

3. Almahasees, Z., Mohsen, K., Amin, M.O.: Faculty’s and students’ perceptions of online learning during COVID-19. Front. Educ. 6, 1–10 (2021) 4. Beteille, T., Ding, E., Molina, E., Pushparatnam, A., Wilichowski, T.: Three principles to support teacher effectiveness during COVID-19. World Bank (2020) 5. Boški´c, N., Hausknecht, S.: Reshaping the educational landscape: during and after the COVIDˇ 19 pandemic. Inovacije U Nastavi: Casopis Za Savremenu Nastavu 34(4), 36–50 (2021) 6. Cavanaugh, C., Deweese, D.: Understanding the professional learning and support needs of educators during the initial weeks of pandemic school closures through search terms and content use. J. Technol. Teach. Educ. 28(2), 233–238 (2020) 7. Chen, V., Sandford, A., LaGrone, M., Charbonneau, K., Kong, J., Ragavaloo, S.: An exploration of instructors’ and students’ perspectives on remote delivery of courses during the COVID-19 pandemic. Br. J. Edu. Technol. 53(3), 512–533 (2022) 8. Cheng, G., Chau, J.: Exploring the relationships between learning styles, online participation, learning achievement and course satisfaction: an empirical study of a blended learning course. Br. J. Edu. Technol. 47, 257–278 (2016) 9. Conrad, C., Deng, Q., Caron, I., Shurska, O., Skerrett, P., Sundararajan, B.: How student perceptions about online learning difficulty influenced their satisfaction during Canada’s Covid19 response. Br. J. Educ. Technol. 53, 534–557 (2022) 10. Crowe, S., Cresswell, K., Robertson, A., Huby, G., Avery, A., Sheikh, A.: The case study approach. BMC Medical Research Methodology (2011) 11. Dam¸sa, C., Langford, M., Uehara, D., Scherer, R.: Teachers’ agency and online education in times of crisis. Comput. Hum. Behav. 121(106793), 1–16 (2021) 12. Dworkin, S.L.: Sample size policy for qualitative studies using in-depth interviews. Arch. Sex. Behav. 2012(41), 1319–1320 (2012) 13. Foerderer, M., Hoffman, S., Schneider, N., Prichard, J.R.: Predicting levels of student satisfaction during COVID-19. Educause Review (2021) 14. García-Morales, V.J., Garrido-Moreno, A., Martín-Rojas, R.: The transformation of higher education after the COVID disruption: Emerging challenges in an online learning scenario. Front. Psychol. 12, 616059 (2021) 15. Guppy, N., Verpoorten, D., Boud, D., Lin, L., Tai, J., Bartolic, S.: The post-COVID-19 future of digital learning in higher education: views from educators, students, and other professionals in six countries. Br. J. Educ. Technol. 53, 1750–1765 (2022) 16. Gurukkal, R.: Will COVID 19 turn higher education into another mode? High. Educ. Future 7, 89–96 (2020) 17. Karakose, T.: The impact of the COVID-19 epidemic on higher education: opportunities and implications for policy and practice. Educ. Process: Int. J. 10(1), 7–12 (2021) 18. Miles, M.B., Huberman, A.M., Saldana, J.: Qualitative Data Analysis: A Methods Sourcebook, 4th edn. Sage Publishing, New York (2020) 19. Montenegro-Rueda, M., Luque-de la Rosa, A., Sarasola Sánchez-Serrano, J.L., FernándezCerero, J.: Assessment in higher education during the COVID-19 pandemic: a systematic review. Sustainability 13(19), 10509 (2021) 20. Oliveira, G., Grenha Teixeira, J., Torres, A., Morais, C.: An exploratory study on the emergency remote education experience of higher education students and teachers during the COVID-19 pandemic. Br. J. Edu. Technol. 52(4), 1357–1376 (2021) 21. Paliwal, M., Singh, A.: Teacher readiness for online teaching-learning during COVID − 19 outbreak: a study of Indian institutions of higher education. Interact. Technol. Smart Educ. 18(3), 403–421 (2021) 22. Saldana, J., Omasta, M.: Qualitative Research Analyzing Life, 2nd edn. SAGE, New York (2022) 23. Stake, R. E.: Qualitative Research. Studying How Things Work. The Guilford Press, New York City (2010;1995) 24. Usher, M., Hershkovitz, A., Forkosh-Baruch, A.: From data to actions: Instructors’ decision making based on learners’ data in online emergency remote teaching. Br. J. Edu. Technol. 52(4), 1338–1356 (2021)

Using Polyphonic Storytelling Techniques for Skills Development Christina Merl

Abstract For a long time, the hero’s journey has served as a popular storytelling template for designing learning experiences, even more so with the rise of technology and its increased focus on character design, magic moments, and surprise elements. This paper suggests that it is about time learning experience designers looked more at what is going on at the periphery of stories. Lifting everyone’s — including AI’s — voice and engaging them in the co-creation of constructive alternative narratives can help learners to develop future skills, better understand complex problems, and drive behavioral change. To support this hypothesis, this paper showcases a series of pilot workshops on climate change that were conducted in the context of a global environmental initiative in the time period between November 2022 and February 2023. The workshops explored how polyphonic storytelling, community building, and artistic impulses ranging from classical to contemporary and indigenous poetry, sketches, multi-media collage, and video input pushes cross-disciplinary, cross-hierarchical, and cross-cultural groups of learners to use and develop their critical and connected thinking skills, their imagination, language skills, listening skills, empathy, and collaborative team skills to build environmental awareness and come up with a more constructive narrative about climate change. Qualitative data analysis indicates that learning experience design that moves away from the traditional concept of the hero’s journey towards co-creating polyphonic narratives can enable learners to leave current mental models and explore new pathways of thinking and doing. Keywords Learning experience design · Polyphonic storytelling · Narrative · Art · Community of practice · 2CG® multi-method approach · 21st century skills

C. Merl (B) TalkShop/2CG® , 1190 Vienna, Austria e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_29

369

370

C. Merl

1 The Role of Story in a Hyper-connected World 1.1 The Need for Creativity and Autonomy in Learning We live in a hyper-connected world, and societies everywhere are currently undergoing deep transformation [1]. Organizations, policy-makers, educational institutions and individuals are challenged to solve complex problems co-creatively and responsibly [2]. In other words, we are currently forced to re-think the way we lead, teach, work, learn, and live. Against this background, the goal of effective contemporary learning experience design must be to enable learners to transform their educational experience, apply what they have learned, and effect change in the real world. Unfortunately, education and training have seen increased emphasis on standardization and accountability, at the exclusion of creativity and autonomy, as pointed out in ref. [3]. And while these developments restrain students and teachers’ creative abilities to engage in the scientific discovery and cultural developments that are essential for hyper-connection, as stated in ref. [4], learners should actually be taught and inspired to start something new, to add something extra, or to adapt something old in whatever job they are [5].

1.2 Storytelling as an Effective Educational Tool Over the past decade, the ancient tradition of storytelling — and more recently data storytelling and digital storytelling — have been (re-)discovered as an effective tool for learning. Stories have become a popular instrument for creating meaningful and memorable, tech-enhanced learning experiences, as they can immerse learners and give them a sense of urgency, thereby driving team performance and engagement. Stories have always been a powerful way of communicating ideas; they signpost our experiences, make sense of what we know, and create continuity [6]. What is more, stories help us to contextualize information since we resonate with situations, characters, and their experiences, which can inform our thinking [7]. Technological Advances. With technological advances, such as digital cameras, editing software and authoring tools, the concept of multi-media, cross-channel stories has eventually conquered the educational sector [8]. Moreover, as pointed out in ref. [9], modern society is no longer exclusively dominated by the time-tested narrative media such as literature or film; new media such as videogames and social platforms have changed the way we understand, create, and replicate stories. Learners are challenged to make multiple decisions in their teams in short time, which should prepare them for real-life settings. Learning moments reside in in-video feedback and different communication and interaction channels as well as adventurous virtual experiences. This tech-enhanced way to storytelling scales well.

Using Polyphonic Storytelling Techniques for Skills Development

371

1.3 Harnessing the Power of Polyphonic Storytelling The current paper looks at how storytelling and narrative can be harnessed in combination with disruptive artistic impulses and community building to foster the development of twenty-first century skills or future skills in learners (see Fig. 1); to increase their understanding of complex topics; and to inspire cultural shift and drive behavioral change. More specifically, this paper explores the potential of polyphonic storytelling and co-created narrative as a timely approach in learning experience design. Definition of Polyphonic Storytelling. Polyphony is defined as a joint achievement that involves several independent participants that are collaboratively developing a time-lasting coherent framework starting from a given theme, even if transient deliberated dissonances may appear. Polyphony thereby enables expression of many voices or nuances, enhancing authenticity of the contextual representation [10]. While the hero’s journey, which goes back to John Campell [11] and puts emphasis on character design and surprise moments, has undoubtedly served as a popular story template in learning experience design, polyphonic storytelling and co-created narrative move away from this traditional concept of story where one protagonist solves all the problems and offers a more collaborative approach towards making meaning and solving complex problems. Polyphonic storytelling as applied in the context of the present study asks questions like: Who is the narrator? Who is the hero? What actions do take place at the periphery of the story? Whose voice do we hear and want to listen to? This non-traditional approach to story invites story contributors to

Fig. 1. 21st Century Skills or Future Skills Developed through Polyphonic Storytelling

372

C. Merl

move away from the idea that a hero can solve all the complex problems. They are encouraged to take a closer look at what else is going on in the story. Who is actively involved? What implications do individual actions have and how do they support the main plot?

2 2CG®: Combining Storytelling, Artistic Impulses and Community Building 2CG® stands for content- and context-specific generic competency coaching and is defined as an experiential, learner-centered teaching approach that hones human capabilities with a focus on twenty-first century skills [12]. The method targets individuals, executives, employees, workers, apprentices, students, and pupils across cultures and hierarchies, who need to develop their skills further so that they can deal with increasingly complex realities and meet new challenges creatively and responsibly. Grounded in learning science and the principles of communities of practice (CoP) as adapted from ref. [13], it connects learners across hierarchical, cultural, and disciplinary boundaries by immersing them in practice-based learning experiences. The multi-method approach is ideally applied in a hybrid hyper-structure for learning as discussed in ref. [2] and linked to a professional practice or issue. The method implies that teachers and learners have autonomy and to a certain extent can be in control of curriculum and content delivery, which may run counter to current tendencies of standardization and accountability.

2.1 Story as Applied by 2CG® 2CG® defines story as a work of art that learners can engage with, as discussed in ref. [14]. Counter to many educational approaches, story is not just treated as a form of self-expression or collective expression but as an object or artefact. This implies that the emphasis is not so much on the hero’s journey but on actions as well as the implications and consequences of these actions. In other words, plot plays a key role and comprises all events that support the main action. Emphasis is also placed on time and place and on what story contributors sense when they co-create the story. Disruptive artistic impulses, ranging from poetry, music, and dance to visual art, are used as a stimulus without distracting the senses too much. The approach aims at enabling story contributors to use their antennae and senses, rather than giving in to their craving for distraction as described in ref. [15]: a panorama of sights, sounds, thrills, and titillations that might distract story contributors too much, just like super-realistic toys may destroy a child’s imagination.

Using Polyphonic Storytelling Techniques for Skills Development

373

Fig. 2. 2CG® 3 Pillar Model: Co-Creating Narratives

Co-creating Narratives with the 3 Pillar Model. A major advantage of storytelling is that learners are encouraged to personalize what they learn and construct their own meaning and knowledge from the stories they hear and tell [16]. The 3 Pillar Model enables them to make meaning through sharing and reflecting; and they can find their purpose and develop a feeling of belonging (see Fig. 2).

3 The Co-creation and Story-Crafting Process The iterative polyphonic story-crafting process as applied in the pilot workshop series discussed in this paper involves 7 stages that may overlap (see Fig. 3). After an entry-level workshop where learners (story contributors) are familiarized with the topic and concept of polyphonic story, which helps them create shared understanding, they receive tailored prompts that include artistic impulses, such as poems, sketches, or video input. The overall process focuses on small daily interactions of all learners involved. Story contributors need to realize that the solution lies in their personal interactions, insights, and contributions, rather than the protagonist’s solution. Also, it is not enough to point out that we face complex problems. The facilitator needs to nourish the community and challenge contributors while enabling them to embark on a transformational learning journey. Transparency and collaborative spirit are key in the co-creation process. Data need to be collected throughout. The facilitator then needs to filter the data so that they can be analyzed and deliver relevant insights and results. Story contributors eventually need to select the most important moments of their journey and define concrete action steps together.

374

C. Merl

Fig. 3. 2CG® Iterative Polyphonic Story-Crafting Process

Coming in through the lens of polyphonic story allows the facilitator and the story contributors to leave the traditional top-down form of story that tries to enforce values and ideas and to move towards a co-creative way of crafting alternative narratives. These alternative narratives — and the conversations around them — may be considered the DNA of innovative solutions to complex problems.

3.1 Facilitating Polyphonic Storytelling Shaping stories and telling stories that live inside us is challenging. This process is supported by a learning environment of trust, shared passion, mutual respect, peer support, and reflection. It takes a keen facilitator — teacher, coach, and curator in one person — who is ready to dive deep and who can get story contributors to open up and share their inner thoughts and personal stories. To make polyphonic storytelling a rich and impactful learning experience for everyone involved, it is important that the facilitator steps out of the role of instructor and takes the role of interviewer. In the phase of onboarding (see Fig. 3), the facilitator needs to engage story contributors in trust-building activities. While providing a helpful framework and guidance, the facilitator needs to become a learner himself or herself and navigate together with story contributors. No less important, the facilitator needs to make what is called white space — room for thought and reflection that allows contributors to explore their inner thoughts and current mental models, to recognize patterns and

Using Polyphonic Storytelling Techniques for Skills Development

375

discover unknown pathways so that a shift in consciousness can take place, even if that happens later on. Most importantly, the facilitator needs to be as non-judgmental as possible in his or her approach.

4 2CG® Pilot Study: Crafting A Constructive Climate Narrative with Polyphonic Storytelling and Artistic Impulses Six hybrid, 2- to 4-h conversation-based workshops on environmental themes were conducted with the 2CG® multi-method approach between November 2022 and February 2023. Participants were aged between 21 and 63 and included engineers, IT experts, social workers, office workers, legal experts, journalists, public officials, company managers, researchers, creative designers, and part-time students from Austria, Germany, the Netherlands, Italy, and Bosnia. Group size varied between 6 and 30 participants. Five workshops took place via video-conferencing tool with geographically dispersed members; one workshop was held onsite in Vienna (A). Learners were taken on a journey where they had to deal with a variety of mostly environment-related ancient and contemporary poems, indigenous poetry and classical literature, video inputs, sketches, and images. The overall thought experiment was split into two parts. Part 1 aimed at defining the prevailing climate narrative as perceived and told by story contributors, referred to as Narrative 1. Part 2 aimed at coming up with a more constructive, alternative climate narrative, referred to as Narrative 2. Qualitative data analysis [17] has revealed that Narrative 1 is clearly overshadowed by negative feelings and emotions, such as sadness, guilt, anger, frustration, fear, fury, and a feeling of helplessness, while Narrative 2 is built on optimism, gratitude, love of life, respect for the other, the willingness to take risks, and the courage to fail in our effort to create a good situation for human kind. (see Table 1) Both narratives are built on 4 climate-related pillars or value systems that were identified during Part 1 of the thought experiment: 1. comfort and convenience, 2. health and safety, 3. expectations and social pressure, and 4. commercial interests. Overall, the pilot study demonstrates how polyphonic storytelling can enable story contributors to craft a narrative that also involves the players at the periphery and how artistic impulses can inspire story contributors to craft a more constructive alternative climate narrative that can help them better understand complex issues and drive behavioral change.

376

C. Merl

Table 1 Narratives crafted by story contributors with the help of artistic impulses Value Framework Narrative 1: Narrative 2 Current Climate Narrative Alternative Climate Narrative Comfort & Convenience

We know that we destroy the planet as long as we decide to stay in our comfort zone. However, as long as it’s convenient and stylish to drink from plastic bottles, we will continue to do so. As long as flights are cheaper than train tickets, we will take a plane rather than the train. Everybody does it. But yes, we know we should not.

Embarking on a journey with like-minded people can be such a hopeful undertaking. We may face challenges and even have to surrender, but being aware of it can help us create new perspectives. Doubt and tension are always inherent in adventure, so let’s take the risk and move on.

Expectations & Social Pressure

We watch “environmental pollution” in documentaries, on TV shows, we read it in the news. Expectations are high on us. What if we don’t manage? What if we don’t succeed? This puts a lot of pressure on us. We need better technologies. Time is running.

Health & Safety

We will not only kill nature, we will kill ourselves if we go on like this. Our health concerns are growing, more and more people are struggling — they are suffering from allergies, suffering from hot temperatures or storms and cold weather. Crops are destroyed due to the heat and dehydration. We can already feel and sense the consequences of climate change. We are in fear — what if we don’t survive. What if only the fittest survive?

We must not be naive. There are always the two sides of the coin: the good and the bad, the light and the dark. However, we need to have trust in ourselves. We are able to build community and make meaning through our emotions and our senses. This will help us to survive. If we want to, we can even adapt and change. If we know our purpose and have the willpower, we can move mountains. We must not only see ourselves and close off as we will get isolated. We have the duty to open up and communicate and connect with others and the other. Life is about communicating with all forms of life and with all things. By closing out the other, we will block ourselves. It is important that we find the balance of opening up and closing off: we need to recharge so that our inner voice does not fade out; we need to refill to be able to take new action.

(continued)

Using Polyphonic Storytelling Techniques for Skills Development Table 1 (continued) Habits

Commercial Interests

377

It’s hard to get rid of bad habits. Yes, we know that it would be better for us if we changed our habits but if not everyone does so, it won’t make a big difference if we do. What’s more, it’s an illusion that changing our habits will change the situation. It’s not as simple as that. Yet, we feel guilty.

We should not constrain ourselves by looking at things through a too soft filter. We’d rather face reality and perceive our vulnerability as something valuable. We can use the power of dreaming to bring about change. Also, our emotions will tell us how we can take care of ourselves and of the other(s).

There is a lack of better alternatives — we currently have no better material than plastic. Plastic is cheap, easy to handle, light-weight. After all, we have to look at scales, profits and finances. Innovation costs money and we often don’t have enough budget. Due to all the crises we suffer and to the austerity programmes that follow it’s hard to make progress here.

We cannot be the Uber-being. Rather, we need to get better at dealing with dichotomy and complexity. We also need to question our dreams and only go for them when our moral compass shows in the right direction. Only then we can undertake business with confidence. Yes, we should constantly enrich ourselves — but with human values, not financial ones. The pig stands for our constant search for sweet delights. Let’s make sure luck is on our side.

4.1 Including the Voice of AI in Polyphonic Storytelling An extra session was held with ChatGPT (Table 2) to see if and how AI could contribute to the narrative. The contribution of ChatGPT was created within a few seconds and, to some extent, reflects what story contributors have produced in their workshop settings. What was missing, though, were the small magical moments of human interaction, which could be called moments of true learning. These are the moments when learners share their experiences, make meaning, and reflect together. Story contributors need to have the time and space to process their own experiences and story moments, to make meaning, and to co-create and come up with solutions that are grounded in their capabilities in order to be able to effect change in the real world.

378

C. Merl

Table 2 Lifting the Voice of Chat GPT in Polyphonic Storytelling (March 2023) Instruction for ChatGPT: “We know that we destroy the planet with our lifestyle, but it is convenient and we have no urge to change this. What does that mean for us?” If we continue to destroy the planet with our lifestyle, it means that we are prioritizing short-term convenience and comfort over the long-term health of the planet and ourselves. It is understandable that changing our habits can be difficult, and we may not always feel an urge to do so. However, the consequences of not taking action are severe, as discussed in my previous answer It is important to recognize that our lifestyles are not only impacting the environment but also our own well-being. Environmental pollution, climate change, and the loss of biodiversity can have significant impacts on human health and quality of life. By taking action to protect the environment, we are also protecting our own future Changing our habits may require some sacrifices, but it can also bring significant benefits. For example, reducing our meat consumption can not only help the environment but also lead to better health outcomes. Using public transportation or biking instead of driving can help us stay active and reduce our stress levels In conclusion, if we prioritize short-term convenience over the long-term health of the planet and ourselves, we are risking severe consequences. However, by taking action to change our habits, we can create a more sustainable and fulfilling future for ourselves and future generations

4.2 How to Measure the Impact of Polyphonic Stories One key aspect of co-creating impactful learning experiences that build on polyphonic storytelling is placing emphasis on the small moments that happen in the daily lives of story contributors. If the facilitator manages to make them aware of these small connection points and to get them to open up and express themselves, story contributors can fully engage with the topic at hand and come up with relevant information and data. Critics of this approach may mention the lack of clear metrics, scalability, and statistics. Looking at the periphery of stories, however, will help us to paint a new picture, change the dynamics of narrative, and add to deep and embodied learning moments that can lead to cultural and behavioral shift — not only among story contributors but also when it comes to impact and transformation processes.

5 Findings and Discussion The results and insights gained from this 2CG® pilot study indicate that learning experience design that makes use of polyphonic storytelling, artistic impulses and community building has the potential to push future skills development in learners while driving cultural transformation and leading to behavioral change. Putting the periphery in the center and moving away from the traditional concept of the hero’s journey can allow learners — story contributors — to craft more constructive, alternative narratives, thereby paving the way for a much needed paradigm shift

Using Polyphonic Storytelling Techniques for Skills Development

379

from the current “what’s in it for us” attitude to a “what’s our purpose and identity on this planet” mindset. Co-creating stories — making tacit knowledge explicit through sharing experiences and anecdotes — can have a motivational effect on learners and provide them with the space to talk shop and reflect across cultures and disciplines while exploring new pathways of thinking and doing. The effect of this approach can also increase the satisfactory element that needs to be inherent in any effective learning experience aimed at transformation.

5.1 Shifting Perspective Professionally facilitated co-creation processes allow contributors to broaden their horizons and see a wider range of possibilities and opportunities, rather than problems and obstacles. Story contributors can engage in non-linear learning and are encouraged to leave their comfort zone while at the same time deepening their critical consciousness about relevant issues. Artistic impulses require learners to continually change perspective and reflect on what is different if they look at things differently or when they listen to diverse voices. Ideally, story contributors will emerge as change agents and see their role in crafting a more constructive, alternative narrative as they understand that their actions result from their thinking, and that they have consequences. Last but not least, learners start to feel more confident to express themselves, to take responsibility, and effect change in real life.

5.2 Professional Facilitation and Learner Engagement Professional facilitation plays a crucial role in co-creating narratives that can lead to conversations that reveal strategic opportunities for innovation and structural transformation. Crafting constructive alternative narratives implies moving away from the hero’s journey where the protagonist will have the solution. It means embracing non-linear storylines that are built on small moments of connection at the periphery as well as in the center, trusting that these are powerful enough to reach people and inspire impactful learning moments that can result in change of values and behaviors in real life. In professionally facilitated learning journeys, story contributors can experience each other as fully human in a rather short time, which increases their curiosity and the wish to dig deeper. This pilot project has shown that learner engagement is very high if the learning is about real-life stakes. When learners perform tasks that serve real-life issues, the lack of focus and short attention span that so many educators are currently complaining about are practically non-existent. The learners involved in this pilot study reported after the sessions that time was flying and that they did not feel any urge to get distraction from their mobile phones.

380

C. Merl

Good learning facilitators need to be teachers, coaches, content curators, learning designers, and facilitators at the same time. They need to lift everyone’s voice by providing guidance and an inspiring framework that challenges learners to leave their current mental models and explore new pathways of thinking. We could conclude that effective and impactful learning experiences require learning designers and facilitators who know how to nourish both, the traditional human network and the techenabled, hyper-connected network. Professional learning facilitators need to have the skills to support story contributors in navigating their personality, their strengths, their values, and their vision. This can only happen in a highly supportive, autonomous, and flexible learning culture where teachers and learners can still be in control of the curriculum and where diversity, democracy, and dialogue are key.

5.3 Including the Voice of ChatGPT in Polyphonic Stories While AI can contribute in a meaningful way to polyphonic stories, the facilitator needs to make sure that learners have enough time and space for fact check and critical reflection. AI-generated content can enhance the learning experience, provided that instructions are well formulated and to the point, and provided that results are considered as one additional voice in the process that should be taken into account and serve as inspiration. What AI cannot do is shut out the practice part, the conversational part, and the behavioral change part. The most relevant point when it comes to including AI in bottom-up storytelling experiences may be that the facilitator needs to make sure that the human dimension and embodiment of solutions do not get lost as it is in the human interaction where impactful learning takes place. Learning that results in behavior change and cultural shift is still social, all the more so when machines are involved.

5.4 Conclusion and Outlook One key aspect of designing and facilitating impactful learning experiences that build on polyphonic storytelling, artistic impulses, and community building is placing emphasis on the small magical moments of human interaction that happen in the daily lives of story contributors. Putting the periphery in the picture takes courage as it asks facilitators and story contributors to move away from a trope that has been played out many times, in different media, at different times. Overall, tech-inspired polyphonic co-creation of narratives proves a contemporary and timely educational practice that supports future skills development and can lead to cultural shift and change in real life. Curricula and training programs in higher education, schools, and organizations — if they want to create a lasting impact on learners and effect change in real life — need to build on story as an artefact and a means of expression.

Using Polyphonic Storytelling Techniques for Skills Development

381

Further applied research is needed to identify the skills and capabilities required to benefit from AI as a contributor to polyphonic stories.

References 1. Bethlehem, D.: The end of geography: the changing nature of the international system and the challenge to international law. Eur. J. Int. Law 25(1), 9–24 (2014) 2. Merl, C.: Lab 21 – a space for learning, sharing and innovating. In: Guralnick, David, Auer, Michael E., Poce, Antonella (eds.) Innovations in Learning and Technology for the Workplace and Higher Education, vol. 349, pp. 199–211. Springer, Cham (2022). https://doi.org/10.1007/ 978-3-030-90677-1_20 3. Giroux, H.A.: The Violence of Organized Forgetting: Thinking Beyond America’s Disimagination Machine. City Lights Books, San Francisco (2014) 4. White, E.S., Mistry, R.S., Chow, K.A.: How do teachers talk about economic inequality? The complexity of teaching at a socioeconomically integrated elementary school. Anal. Soc. Issues Public Policy 13(1), 370–394 (2013). https://doi.org/10.1111/asap.12024 5. Smith, P.: Accomplishing the goals of multicultural education. Curriculum Teach. Dialogue 15(1/2), 27–40 (2013) 6. Calvert, J., Hume, M.: Immersing learners in stories: a systematic literature review of educational narratives in virtual reality. Australas. J. Educ. Technol. 38(5), 45–61 (2022). https://doi. org/10.14742/ajet.7032 7. Kazanjian, C.J.: Hyper-curriculum: transcending borders of standardization in the cosmopolitan classroom (2016). ijci.wcci-international.org 8. Koh, A.W.L., Lee, S.C., Lim, S.W.H.: The learning benefits of teaching: a retrieval practice hypothesis. Appl. Cogn. Psychol. 32(3), 401–410 (2018) 9. Mih˘ae¸s, L.C., Andreescu, R., Anda, D.: Handbook of Research on Contemporary Storytelling Methods Across New Media and Disciplines (2021). https://doi.org/10.4018/978-17998-6605-3 10. Trausan-Matu, S.: The polyphonic model of hybrid and collaborative learning (2010). https://doi.org/10.4018/978-1-60566-380-7.ch028, https://www.igi-global.com/dictionary/pol yphonic-model-hybrid-collaborative-learning/22981. Accessed 19 Mar 2023 11. John Campell Rogers, B. et al: Seeing your life story as a hero’s journey increases meaning in life. J. Pers. Soc. Psychol. (2023) 12. Merl, C.: Human intelligence cultivation with the 2cg® poetry machine: how to boost future skills development and idea generation with artistic impulses in lab 21. Int. J. Adv. Corp. Learn. (iJAC) 15, 61–74 (2022) 13. Wenger-Trayner, E., Wenger-Trayner, B.: Learning to Make a Difference: Value Creation in Communities of Practice. Cambridge University Press, Cambridge (2020). https://doi.org/10. 1017/978110867743 14. Aristotle, L.J.: Potts, Aristotle on the Art of Fiction: An English Translation of Aristotle’s Poetics. Cambridge Univ. Press, Cambridge (1968) 15. Watts, A.: The Wisdom of Insecurity: A Message for an Age of Anxiety (2011) 16. Murray, M.: Narrative Psychology. Qualitative Psychology: A Practical Guide to Research Methods, pp 111–131 (2003) 17. Miles, M.B., Huberman, A.M., Saldana, J.: Qualitative Data Analysis: A Methods Sourcebook, 004 Edition. SAGE Publication, New York (2019). ISBN-10: 150635307X

The 50 + 10 Concept for the Development of Future Skills: A Pedagogical Framework Ana Francisca Monteiro , Soraia Gonçalves , and António H. J. Moreira

Abstract This paper presents the 50 + 10 concept, a pedagogical framework for the development of future skills across IPCA - Polytechnic Institute of Cávado and Ave’s educational offer. This framework was designed to address the challenge of integrating transversal competences with knowledge and disciplinary expertise in Higher Education courses. It offers an integrated approach that aims to respond to the needs of students, teachers, and employers, hence society at large, incorporating student-centered teaching and learning strategies, pedagogical development, transferable competences, and university-community-industry-society partnerships capable of delivering social transformations. The 50 + 10 approach is further expected to have a positive impact on student and teacher engagement, deliver high-quality learning, and prevent dropout. This work was developed at IPCA’s Future Advanced Skills Academy, created within the pedagogical development programme of the Regional University Network – European University. Keywords Pedagogical innovation · Future skills · Pedagogical framework · Pedagogical development · Collaborative partnerships · Social transformation

A. F. Monteiro (B) Research Centre On Education (CIEd), Institute of Education, University of Minho, Braga, Portugal e-mail: [email protected] A. F. Monteiro · S. Gonçalves Polytechnic Institute of Cávado and Ave, Barcelos, Portugal A. H. J. Moreira 2Ai – School of Technology, IPCA, Barcelos, Portugal © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_30

383

384

A. F. Monteiro et al.

1 Introduction 1.1 Pedagogical Innovation in Higher Education To prepare for high-uncertainty scenarios of both job creation and destruction, the Portuguese economy and labor market need to be empowered in a way that strengthens the Social State and attracts and retains qualified human resources in mobile and digital technologies and associated applications. This requires the stimulation of new digital and transversal skills among students at all education levels, as the society evolves and mutates continuously and individuals need to be better prepared for a future of growing uncertainty. Pedagogical innovation strategies are among the keys in higher education to address the skills gap in a multitude of areas while ensuring more success and less dropout; this can be evidenced by research, with a growing body of literature and examples of innovative practices. Here are some of the current state-of-the-art practices and trends in this field, along with references to support them: Micro-credentials. Micro-credentials, also known as badges, digital credentials, or nanodegrees, are short, stackable, and portable credentials that represent specific skills or competencies. Micro-credentials can help students acquire the skills and knowledge required for success in the workplace and demonstrate their mastery to employers. Recent studies have shown that micro-credentials can increase student motivation, engagement, and learning outcomes [1]. Project-Based Learning. Project-based learning emphasizes hands-on, collaborative learning through the completion of real-world projects that require the application of skills and knowledge. Project-based learning can help students develop the skills and competencies required in the workplace, such as critical thinking, problemsolving, and communication. Recent research has shown that project-based learning can improve student engagement, learning outcomes, and employability [2, 3]. Augmented and Virtual Reality. Augmented and virtual reality technologies can create immersive and interactive learning environments that simulate real-world situations and allow students to practice and apply skills and knowledge. Augmented and virtual reality can help students develop the skills and competencies required in the workplace, such as decision-making, problem-solving, and collaboration. Recent studies have shown that augmented and virtual reality can improve student engagement, motivation, and learning outcomes [3, 4] Work-Integrated Learning. Work-integrated learning involves the integration of academic learning with work-based learning experiences, such as internships, apprenticeships, and co-op programs. Work-integrated learning can help students develop the skills and competencies required in the workplace, such as teamwork, communication, and leadership. This approach also involves the integration of multiple disciplines and perspectives to address complex problems and develop

The 50 + 10 Concept for the Development of Future Skills …

385

innovative solutions. Recent research has shown that work-integrated learning can improve student employability, career readiness, and post-graduation outcomes [5].

1.2 RUN-EU and IPCA Future and Advanced Skills Academy (FASA) To prepare young people for an uncertain future and encourage job creation and entrepreneurial skills, the Portuguese government and Higher Education Institutions have developed national programs such as Digital Competencies e.2030 Portugal CoDigital 2030, Problem-Based Learning in Short Cycle Courses, LinkMeUP - 1000 ideas Co-creation projects by Portuguese Polytechnics, and the European Universities Initiative. IPCA is a founding member of the RUN-EU alliance, which promotes pedagogical innovation through the RUN-EU Future and Advanced Skills Academies (FASA) across institutions. FASA’s role is to design innovative programs and practices that offer students challenging and flexible learning experiences that provide cutting-edge knowledge and skills. However, unlike other institutions, IPCA does not have a school of education. This challenge led to the implementation of a broad learning-through-experience approach, where students, teachers, and partners take active roles and responsibility for educational change. To ensure that technical and transversal skills are learned in an engaging environment that promotes integration with real challenges and prepares students for future skills that are continuously learned, the 50 + 10 concept was devised based on previous knowledge gathered from active learning activities, new experiences with RUN-EU FASA members, and students’ and teachers’ learning objectives. “50 + 10” is a learning model developed by IPCA to ensure that students acquire both technical and transversal skills in an engaging environment that integrates real challenges, see Fig. 1. Fig. 1 IPCA 50 + 10 Concept and it’s relation with future skills

386

A. F. Monteiro et al.

The model consists of 50 hours of the time dedicated to traditional classroom learning, including active learning activities, such as problem-based learning, projectbased learning, and challenge-based learning. The “ + 10” time is dedicated to transversal skill development, such as communication, teamwork, creativity, and critical thinking, which are essential for students to succeed in the constantly changing job market. This model aims to prepare students for the future by providing them with the skills to adapt and thrive in uncertain and rapidly changing environments.

2 IPCA 50 + 10 Concept for Future Skills IPCA “50 + 10” is a teaching–learning framework based on “future skills” [6]. It aims to develop the competences that enable Higher Education students and teachers to deal and learn with real and complex problems, thus promoting actual and impactful social transformations. Teamwork, communication, collaboration, questioning, and creativity are examples of the key transversal competences that, through the active involvement of learners, the 50 + 10 aims to promote. Furthermore, it aims to address the challenge of integrating transversal competences with advanced disciplinary expertise, in Higher Education courses.

2.1 Methodology Distinctive features of this proposal are: introducing active learning methodologies early on (from the moment students enter Higher Education and are more open to change); providing learning experiences that promote students’ and teachers’ relational skills; involve external partners to set up and present real-world challenges at each academic turn; offering experiential learning activities, based on curricular flexibility, collaboration, and integrated learning; including students in different teams (within their own class or across classes) throughout the courses; and setting a timeframe, common to all courses, exclusively dedicated to addressing the challenge and presenting proposals (see Fig. 2). This proposal considers the following guidelines, which recognize both IPCA’s and RUN-EU networks’ needs, in what concerns the development of “future skills”: • Future skills encompass the development of hard and soft skills to address real world challenges. • Students are more likely to collaborate if they have confidence in their team/teams. • Students and teachers are willing to learn new methodologies in a safe environment. • Time is a limited resource and extracurricular work is not sustainable. • Heavy workloads and disconnected learning activities hinder deep approaches to learning.

The 50 + 10 Concept for the Development of Future Skills …

387

Fig. 2 Strategy of IPCA 50 + 10 Concept by ensuring multidisciplinary approach to embedded different stakeholders in the teaching environment, while ensuring that soft & hard skills are part of the curriculum with support of IPCA FASA

The concept owes its name to the reconfiguration of learning environments and inperson learning moments that support this flexible process. Built upon the idea that a curricular unit with 60 hours may be reorganized into moments of both hard and soft skill development, as well as x-based learning activities, it entails: 50 hours of fulltime dedication to disciplinary competences and 10 hours to cross-disciplinary, experimental learning. The set of hours made available for the development of transversal competences is allocated to three purposes/stages: 1. Kick off each semester’s 50 + 10 schedule with learning activities focused on relational skills. 2. Provide future skills learning activities, throughout the semester, one per curricular unit. 3. End each semester with a two-week of active learning activity dedicated to practical learning. With this integrated approach, the “50 + 10” concept aims to address the needs of students, teachers, organizations, and hence society at large. Integrating studentcentered teaching and learning strategies, pedagogical development, transversal competences, and university-community-industry-society partnerships, it is expected to enhance HEI’s capacity to deliver social transformations, thus promote student engagement, active and high-quality learning, as well as contribute to prevent dropout.

2.2 IPCA 50 + 10 Concept in Practice To achieve an overall balance between student expectations, teachers’ abilities and external entities needs, this concept rearranges the way lectures and learning moments

388

A. F. Monteiro et al.

Fig. 3 Overall description of IPCA 50 + 10 Concept for embedding transversal skills during the semester while fostering teamwork and projects with external entities in the last weeks

occurs during the semester, by creating 3 phases not alongside but with active engagement from the traditional lectures and teachers. The overall proposal can be analyzed in Fig. 3. Initial Preparation – Semester Reorganization. To promote the integration of hard/technical skills with soft/transversal skills, IPCA reorganized the pilot bachelors (see Fig. 1). The teachers participated in a FASA preparatory program, which taught them how to split the 60-h lecture time into 50 hours of technical knowledge and 10 hours of transversal activities that aligned with the semester’s objectives. During the FASA program, the teachers discussed the kickoff moment, relevant skills for new higher education students, which entity to invite for collaboration, and how to reflect the evaluation of the sprint project in the curricular units. Additionally, each teacher served as a mentor for one or two teams, monitoring their progress and addressing any questions or concerns. With the 50 + 10 concept, the courses did not need to be modified; instead, the focus was on how they interacted with each other and how the sprint week projects were reflected in the students’ grades. Phase 1 – Project Kickoff. For the kickoff moment, we devised a combined activity that engaged the students, teachers, and the invited challenge partner. The main objective of this activity, which lasts around 4 to 5 hours, is to break down barriers between all members and prepare the students for the challenge theme. FASA with the teachers promotes several moments during this activity: 1. Welcome & Icebreaking moment; 2. Challenge presentation; 3. Partner presentation; 4. Random Team creation; 5. Teambuilding time; and 6. Teams Pitch of first ideas/doubts. In the next subsequent semesters, the Welcome moment is also used to reflect in the pros/cons of the previous semester. Phase 2 – Skill Activities. This phase focus is on skill development through transversal activities that complement the technical lectures. These activities, lasting

The 50 + 10 Concept for the Development of Future Skills …

389

between 1 and 2 hours for each curricular unit, involve both students and teachers and aim to explore skills beyond the scope of the lecture. The phase is divided into three sections: 1. A brief icebreaker activity to form teams; 2. An introduction to a specific soft skill; and 3. A team activity that promotes discussions, presentations, and new experiences, centered around the specific skill. The soft skills promoted during these moments can include Trust, Creative Thinking, Conflict Resolution, Communication, Pitching, Interviews, and others that aid the students in tackling challenges in their course or professional life. While these moments can be related to the challenge presented in the kickoff, it is not mandatory. Mid-semester, one of these skill-building moments should be dedicated to revisiting and discussing the challenge, as well as any ideas or topics that require additional clarification. This review session should be promoted with the team mentors. Phase 3 – Sprint Weeks. The Sprint Weeks are a unique feature of the semester that allow students to put into practice the knowledge and skills they have gained throughout their coursework. These two weeks are reserved specifically for working on a transversal project or challenge that requires students to collaborate with their peers and teachers to come up with innovative solutions. To ensure that students are fully engaged in this process, no evaluations or assessments are permitted during these two weeks. All exams or reports are expected to be completed by the end of the thirteenth week, leaving the final two weeks free for the Sprint project. During the first day of the Sprint Weeks, the challenge is reviewed, and students are provided with various design thinking methodologies that can aid in their project development. However, following these tasks is optional, and students are encouraged to use whatever methods or strategies work best for them. Throughout the two weeks, students work together in teams to generate, create, and build prototypes of their ideas or solutions. Each team is expected to present their work on the final day of the Sprint Weeks, using a professional and convincing pitch or other form of presentation format. An external jury, which includes the invited entity, evaluates each team’s presentation and provides feedback. Since all bachelor courses follow a synchronous schedule, the Sprint Weeks generates an opportunity for multidisciplinary projects that tackle challenges involving two or more areas of knowledge (see Fig. 4). This cross-disciplinary collaboration allows students to gain an understanding of different fields and work with various people.

2.3 Preliminary Results Currently, IPCA has been piloting this proposal during the current scholar year of 2022/2023, in 5 bachelor courses (BSc) and 4 technical short cycle courses (CTeSP), with more than 280 students and 40 teachers involved (see Fig. 5). To measure the effectiveness and improve the quality of this methodology, we designed a data collection session (end of the first semester), followed by a feedback

390

A. F. Monteiro et al.

Fig. 4 IPCA 50 + 10 creates the possibility of multiple fields of studies working in collaborative challenges during the Sprint Weeks

Fig. 5 IPCA 50 + 10 activities in 2022/2023 (first-year students), from left to right, kickoff moments, Skills training and Sprint Week pitch and showcases

session (beginning of second semester), with students. Data collection included a qualitative approach, followed by a quantitative instrument: 1. brainstorm, facilitated with resource to a digital board, with 4 sections (what worked, need to improve, questions and ideas; see Fig. 6); and 2. survey, prepared in collaboration with IPCA’s Psychological Support Service, with the 4 dimensions (reasons to select the course, satisfaction, self-confidence, well-being). A feedback session with teachers was also organized, at the beginning of the second semester. The process of generating evidence from qualitative data followed a thematic analysis method, so far, on an exploratory basis [7]. Qualitative data from teachers and quantitative data are still under analysis.

The 50 + 10 Concept for the Development of Future Skills …

391

Fig. 6 Example of digital board for qualitative feedback collection

Positive aspects highlighted by the students were: getting to know each other; successful teamwork, even when teammates didn’t know each other; active and practical learning experiences, considered engaging, fun, and promoters of transversal skills, like creativity, self-confidence, and critical thinking; collaboration and mutual help; getting to know the region of Cávado and Ave; “making ideas happen”; being able to present their work in public (recognition); and direct contact with external partners. In need of improvement are: time available for projects; excessive workload; lack of clarity in defining objectives; random groups considered unfair, for some; redundant tasks; teachers’ lack of engagement; lack of time for other course assignments; and learning spaces. Thus, students expressed an interest in clarifying important aspects, for example: objectives, next steps, and reasons for selection of themes and learning timings. Overall, student feedback is positive, and projects developed were considered to overcome expectations. Nevertheless, key aspects need to be improved, to prevent students feeling disoriented. Initial feedback from the external entities, collected during and after the final presentations of the projects, is also very positive, challenging us to continue on this path, as they also envision opportunities for further collaboration. Further research must be taken to ensure effective results in the student’s engagement, success, and reduction of dropout.

392

A. F. Monteiro et al.

3 Conclusions Pedagogical innovation strategies such as micro-credentials, project-based learning, and technological and work-integrated learning are key to addressing the skills gap in higher education. The IPCA “50 + 10” model is a teaching and learning framework that aims to develop both technical/hard and transversal/soft skills through classroom learning to take on “future skills” development. By integrating real-world challenges and promoting skills as teamwork, communication, collaboration, and creativity, the 50 + 10 model attempts to prepare students for an uncertain and rapidly changing job market. This model provides a solution to the challenge of integrating transversal competencies with advanced disciplinary expertise in higher education courses, sometimes hard to rearrange because of Higher Education national quality guidelines. With 50 + 10, we expect to promote actual and impactful social transformations in students, teachers, and entities to promote creativity and collaboration. Preliminary results presented here are encouraging, in regard to student engagement and the development of “future skills,” but further research is needed to sustain these findings and analyze possible contributions for overall academic success and reduction of dropout. Acknowledgements This paper was funded the project Inter-Regional University Alliance (RUNEU), supported by the project the Erasmus+ Programme of the European Union, Grant Number: 101004068.

References 1. Reich, J., Ruipérez-Valiente, J.A.: The future of microcredentials. Harv. Bus. Rev. 99(1), 72 (2021) 2. Alshannag, Q., Ali, M.: Impact of project-based learning on students’ engagement and performance: evidence from a university in Jordan. Educ. Inf. Technol. 26(1), 1437–1458 (2021) 3. Lopatovska, I., Arnoud, J.: Enhancing students’ employability skills through project-based learning: a case study in higher education. Eur. J. Educ. Learn. 2(4), 1–14 (2020) 4. Lin, C.C., Chen, C.Y., Sung, Y.T.: Enhancing learning motivation and outcomes with an augmented reality simulation system in cardiac nursing education. J. Nurs. Educ. Pract. 10(7), 52–62 (2020) 5. Boud, D., Molloy, E.: Rethinking models of experiential learning in higher education. Routledge (2019) 6. Eskelinen, A., Kanervo, R.: Creativity and innovativeness as future skills. In: ICERI2019 Proceedings, pp. 696–701 (2019) 7. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3, 77–101 (2006)

Loose Parts: Creating Learning Opportunities Beyond the LMS Gary Natriello

and Hui Soo Chae

Abstract Software platforms and applications designed to facilitate administrative control and management dominate the higher education and workplace learning/ training landscapes. These systems are presented as simplifying online teaching for educators and streamlining access for learners/workers. In addition to the standardization within platforms, the competitive landscape for such products has evolved to reduce the number of providers and led remaining providers to converge on a set of features deemed to be essential. Once such platforms are adopted and installed in institutions, they are further configured to reduce staff workload. Standardized templates are developed and framed as consistent user experience for educators and learners. In this paper, we contend that these trends reduce opportunities for educator and student creativity and invention. We draw on Nicholson’s (1971) theory of loose parts to highlight the importance of variability in the learning environment. In this paper we describe our efforts to develop an online learning environment that leverages the theory of loose parts. We illustrate the ways in which an online environment can be made more educational and more flexible to support more creative learning experiences. Keywords Creativity · Learning management system · Learning environment

1 Introduction and Background In this section, we present a learning-centered technology approach that responds to the growth of online learning and the concomitant development of learning management systems as platforms intended to standardize the online learning experience for G. Natriello (B) Teachers College Columbia University, New York, NY 10027, USA e-mail: [email protected] H. S. Chae New York University, New York, NY 10003, USA © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_31

393

394

G. Natriello and H. S. Chae

students. We draw on Nicholson’s [23] theory of loose parts as a starting point for thinking about learning environments that support creativity and learner initiative.

1.1 Growth in Online Learning Over the past 25 years, the development of the internet and related communications and computing technologies has supported the growth of online learning programs in both traditional institutions and new institutions that are entirely online. During the same period, employers have developed online training programs for employee onboarding, development, and evaluation. Accompanying the growth of online learning offerings has been the growth of standard platforms to support such offerings. These platforms have been designed to standardize the process of creating online educational experiences, most typically in the course format. Such platforms do indeed streamline the course design process. Moreover, they provide important capacities for those in educational institutions, particularly registrars, for monitoring things like course sections, prerequisites, and enrollments. In the case of workplaces, such platforms allow organizations to monitor and ensure compliance with regulatory training requirements. However, these benefits are accompanied by some negative consequences such as constraints on experimentation and creativity for both educators and students.

1.2 Theory of Loose Parts To understand how to think about online learning environments and creativity, it is important to start by recognizing that online learning requires environments that are more than the digital analog of the classroom. Instead, the LMS presents us with an environment that often contains fewer opportunities for innovation and creativity than educators and learners would find in a physical classroom where they can leverage both online and physical tools. As LMSs have developed over time, they seem to have become more polished and finished, presenting as complete learning settings when, in fact, they are just management platforms. As we consider how to create online environments to support discovery and creativity for educators and learners, we are drawn to settings like playgrounds that are intentionally designed to support discovery and creativity. We resonate with the work of Patty Hill Smith at Teachers College, Danish landscape architect Carl Torenson, landscape architect Lady Allen of Hurtwood, and finally Simon Nicolson and Stuart Brand. Patty Hill Smith created wooden floor blocks to allow children to build at the own scale [21]. Carl Torenson opened a “junk playground” in Copenhagen using surplus construction materials. Lady Allen brought the concept of adventure playgrounds to the UK in the 1950s and to the US in the 1960s [13]. Simon Nicholson published a

Loose Parts: Creating Learning Opportunities Beyond the LMS

395

paper in Landscape Magazine in 1971 in which he set out his theory of loose parts [23]. In it, he argued that all children love to play, discover, and learn in the right settings. He saw the loose parts or variables in a setting as key to fostering creativity and learning. He summarized this approach with the following statement: “In any environment, both the degree of inventiveness and creativity, and the possibility of discovery are directly proportional to the number and kind of variables in it” [24, p. 30]. This sentiment is echoed by Stuart Brand [5], who observes the advantages of what he calls low road buildings — those spaces where no one cares what you do with them and where users get to build them out for their own purposes. The overall point argued by these authors is that, when users can shape their own environment, there is room for discovery, creativity, and learning [15]. With this perspective in mind, we reassess our various approaches to the online courses we have developed over the past 25 years.

2 Before the LMS We begin by considering our approach to courses that that we offered in the late 1990s and early 2000s. With the internet in the early stages of the transition from a research network to the commercially friendly infrastructure centered on the World Wide Web, and with Learning Management Systems still in a nascent phase, we took advantage of other online affordances to support courses.

2.1 Homemade Learning System Our attempt to create an online platform that would support discovery and creativity began with an ambitious effort to create an entire learning platform for the very first offering of a course where the topic was online learning. Because we were building a system for a single course, it did not have to include tools for the management and tracking of multiple courses. It was little more than a backend database, a page builder, and a user account module. This lightweight platform had several advantages for supporting creativity. First, the page builder offered flexibility in configuration and design that allowed pages to have distinct and different aesthetics that could take form based on the content to be featured on the page. Second, the system permitted pages to be added before and throughout the semester as a course developed. Third, and perhaps most directly in the spirit of theory of loose parts, the system permitted students to have the same rights to create “hypertexts” and to configure pages as they saw fit. This meant that every student in the class could build onto the class platform over the course of the semester. This was critical, since the major project assignment in the course called on students to develop an online learning opportunity using learning design principles.

396

G. Natriello and H. S. Chae

2.2 Journal Site Another early online learning experience allowed us to experiment with multiple platforms for a single course. Still in the late 1990s, we had occasion to offer a course in educational policy that was delivered in multiple formats. The course was offered in a face-to-face format with weekly sessions held on campus. These sessions were captured on video, and this material was made available through an online course site on the recently acquired Blackboard LMS and through a section of the online journal site connected to the Teachers College Record [20]. The variability in this approach was in the delivery system since students could choose the platform. An additional element was the possibility of embedding a learning experience in a broader context as we did when we featured the policy course in the section of the journal devoted to research on educational policy.

2.3 Library DocDel By 2002, we were managing an academic library where we had responsibility for supporting both campus-based and distance courses. The support responsibility included providing access to library resources through the traditional library reserves approach. However, with the need to support online learners who would never visit the library reserves room, we needed a delivery path for library materials in digital form. Our decision to create a stand-alone system to receive, store, and deliver digital documents wherever they might be needed allowed us to be consistent with our interest in systems of parts as opposed to more tightly integrated approaches. The Document Delivery System (DocDel) allowed users to upload materials directly. This included library staff, faculty, teaching assistants, and students. In addition, the system could connect to whatever learning platform the educator might be using for a course. Faculty could organize materials into collections for official courses or for other purposes such as research projects and informal reading groups.

2.4 Library Archive — PocketKnowledge By 2005, we were confronted with the need to move a library historical archive into digital form to support both preservation and access. Our approach was to create a highly modular system for organizing digital files of multiple types (text, image, audio, video) along with a user account system that allowed library staff, faculty, and students to upload materials directly into the archive and organize them in a variety of “pockets” (containers) with varying levels of access from private, to public to the community, to public to the entire Internet.

Loose Parts: Creating Learning Opportunities Beyond the LMS

397

The result was a popular institutional repository that met the needs of the library for historical archiving and the needs of contemporary users in the community for hosting and organizing digital materials. These collections could be accessed directly and just as easily accessed from course platforms at the item level or the collection level. Along with DocDel, PocketKnowledge became another content component available as a stand-alone resource or as an element in a learning platform. In both instances the application balanced institutional needs and individual needs and in both bases the prerogative of individual users were preserved [3, 18].

3 Dancing with Learning Platforms — Not During the 2000s, we had responsibility for supporting distance learning activities and it was during this time that we shifted from being LMS users to being LMS managers. This role gave us another perspective on the LMS approach.

3.1 Blackboard We operated the Blackboard Learning System for about 10 years at a time when several degree programs and a scattering of individual courses were offered online. The version of the system at the time included the kinds of user management and course management components that might be required at an institutional level. Our role as managers of the system gave us a bit more flexibility than users, but we soon discovered that there were limitations in the design of the actual course shells that even system managers could not overcome. Nonetheless, we did use the Blackboard system to meet the needs of numerous instructors, and it did have the advantage of being a widely used LMS, which meant that instructors joining the college from other institutions often had prior experience with the system. This, in addition to the limited feature set, made our support role easier. Our one effort to shape the use of this standard platform took the form of a utility to gather data on students in classrooms using the platform [19]. Although this component never achieved general use, it highlighted some possibilities for building capacities on top of learning management systems.

3.2 Moodle Although supporting Blackboard offered more opportunities for experimentation than the typical user enjoyed, we added the open source Moodle platform in the hopes of providing more flexibility. Since we hosted, configured, and managed the entire system, we had greater control, and we were able to support a broader range

398

G. Natriello and H. S. Chae

of use cases, particularly in the non-credit, informal education space. We also had the benefit of a global community of contributors to the system. Our most ambitious effort to shape the system allowed us to address the ongoing issue of instructors trying to monitor and assess student discussion posts. Discussion is a standard feature of learning management systems, but making effective use of student discussions can be a challenge for instructors. Our effort to address this challenge led to the design of the Discussion Forum Analyzer Tool (DFAT) to assist the analysis of student discussions by calling attention to “hot” areas in discussion boards where instructor intervention could be beneficial to student learning [4].

4 Creating Components We have vacillated between embracing and rejecting learning management systems. Although the benefits of the LMS for the management of large numbers of courses within institutions are substantial, their limitations as learning environments are also significant. Our interest in environments that facilitate learning led us to shy away from integrated systems to manage courses and devote attention to the creation of “parts” that might be combined by educators as needed to advance particular learning goals.

4.1 Vialogues With the rise of video as an important component of online learning offerings we decided to devote effort to the creation of a tool to leverage video for learning. We were intent on avoiding some of the poor practices for the use of video that had become common in face-to-face learning settings. Our approach was to develop an interactive video discussion application that could function as a stand-alone experience or be used in conjunction with any learning system [1]. The resulting tool was Vialogues, a site that hosted videos along with discussions of videos. The site enabled all account holders to create video-based dialogues, and was widely used by educators and their students. The design balanced openness to users with access control for educators using it with defined groups of learners. The application could be used as a stand-alone resource or in conjunction with online and face-to-face learning activities. While the majority of video resources were posted by the community of 60,000-plus users [9, 16], there were also curated video resources that highlighted major topics of interest, including advice on the most appropriate use of the site. Since we had created the site and managed it completely, we were able to engage in continuous research on use patterns [7, 11, 12, 14].

Loose Parts: Creating Learning Opportunities Beyond the LMS

399

4.2 Rhizr Although we avoided the creation of anything that resembled a course platform or LMS, we understood that learners, particularly self-directed learners, might need tools to support their learning through the curation of content. We decided to develop an online system that would allow learners to construct whatever model or metaphor for learning that suited their purposes. The application that we created was Rhizr, a system intended to support selfdirected learning and the crowdsourcing of learning content by a growing community of learners [9]. The application allowed users to create modules of content and to organize those modules in whatever way they chose for a particular learning opportunity. Users could create learning experiences for themselves by developing their own modules; they could share their learning resources with others, including their students if they were educators; and they could incorporate modules created by others in the system when they found them useful. The system incorporated the principles of self-directed, de-centralized, negotiated and social and personal knowledge associated with the concept of rhizomatic learning [8]. Although Rhizr was a content curation platform, we put it to use as a site for the work of students in several of our courses. It had the advantage of allowing for a wide range of learning content, both content from modules within the system and content from other applications such as Vialogues and PocketKnowlege as well as content from across the internet. Moreover, after many years since our initial homemade course platform, with Rhizr, we were able to give students access to the same content creation platform and tools that instructors used. Our students were thus able to create their own learning experiences within our course.

5 Our Own Loose Parts We have traveled a long way since our that first homemade learning platform from 1998, and we have had an opportunity to consider the LMS from all sides as managers, as support staff, as instructors, as developers, and as researchers. We maintain our position that the LMS approach to online learning carries substantial benefits and substantial limitations. And since the balance of strengths and drawbacks depends on the approach to learning, we begin this section with a discussion of the learning theories that undergird our current course design. In our current courses, we are aiming to help students progress as self-directed learners with abundant opportunities for social learning. We use the project method along with small learning teams to move through the course material. One goal is to have the demands of the project motivate students to consult the course materials and question us as instructors. In many respects, we are trying to encourage students to step away from their traditional approach to learning in courses where the instructor is in a directing role.

400

G. Natriello and H. S. Chae

5.1 Latrlab/WordPress To signal that we are taking an approach to learning that is different from typical institutionally based course experiences, we created an online site for all of our courses that is dedicated to the topics covered. For this site, we used WordPress and the Magazine theme. The overall site presents information in a variety of forms on topics related to the courses. This setting is one that is ancillary to but supportive of the course objectives. One important advantage of settling on WordPress for the online site is the WordPress architecture that includes plugin modules that provide a wide range of functions. The large user-base and developer community associated with WordPress have resulted in a large number of plugins. This allows customization beyond basic functionality as needed for particular needs. The combination of an established platform with many optional modules provides a reasonable balance of stability and variability.

5.2 LearnDash LMS Plugin Although we have emphasized the limitations of the LMS approach to organizing online learning, there are some basic functions of learning management systems that are important to retain. Among these are user account management, content access controls, and basic record keeping. Functions of this sort provide a solid foundation for a learning environment that can support variability and discovery within an environment somewhat protected from the larger network of users. To provide such basics, we use the LearnDash plugin for WordPress. LearnDash provides the user management and content access controls as well as a content schema and hierarchy that we used to organize materials for the projects in our courses. This plugin also has a variety of other functions such as collecting student data tracking and grading.

5.3 Slack To support the real-time communications that we view as important for the social learning opportunities in our courses, we searched for a chat or communications plugin, but we did not find one that worked well for our purposes. So we used Slack for communications with and among students. However, Slack proved less popular among our students than alternatives they identified, which we discuss further below.

Loose Parts: Creating Learning Opportunities Beyond the LMS

401

5.4 Asana Since we use the project method for our courses, we hoped to identify a suitable project management plugin for WordPress. Unfortunately, the other plugins to manage projects within WordPress were focused on managing WordPress development. Hence, we did not find any plugin that would allow students/users to manage projects within the platform. To help students structure the management of their projects, we used Asana [2] as a project management tool. Although some student teams made good use of this application, for others, it was less effective [17]. We continue to seek a project management tool that might provide supportive scaffolding for projects in our courses.

5.5 BuddyPress Plugin In each class, students are organized in small teams to work on the projects. Since we value social learning opportunities in our classes, we sought ways to make communications within the class and within the teams in the class easier. Although our efforts to find a real time chat application were not successful, we were able to identify the BuddyPress [6] Plugin as a tool to create discussions at both the whole class and team level.

5.6 Google Docs and Slides For real-time collaboration on project deliverables, we encouraged students to use Google Docs and Slides, and these have proven to be popular choices. Although there are other collaboration tools that we assess from time to time, it is challenging to compete with the Google Apps ecosystem since it is so integral to the student experience at the school (e.g., Gmail, Gchat).

5.7 Poll Maker Plugin From time to time, we have had class sessions where we would solicit student opinions on issues as a springboard for discussion. To support this practice, we used the Poll Maker [22] Plugin. Although there are many polling applications available for online use, because Poll Maker is a plugin, it gathers information on each student as they respond to the poll. This allows us to know both the final tally on a poll and then to pose additional questions to individual students based on their answers.

402

G. Natriello and H. S. Chae

5.8 Instructor Role Plugin One of the most important plugins for our goal of supporting students to become selfdirected is the Instructor Role Plugin [10]. This plugin allows us to give students the same privileges to create course materials as those held by instructors. For example, in a recent course, we designed a project in which the deliverable was for students to create an online learning experience using the LearnDash application. Students were able to build out their learning experience with the same tool set as the course instructors; this took us back to the very early course with the home-grown LMS. Twenty-five years and many iterations later, we had come full circle in our efforts to support student self-direction in an online course.

5.9 Student-Sourced Apps It is important to acknowledge that, in every course we have taught, the students in the class have brought their own apps to whatever tasks we assign for the course. These applications have changed over the years as new generations of students enroll. Such student-selected applications arise both in the context of individual student work and, often more visibly, in the context of their work in teams.

6 Conclusions Our 25 years of teaching and working online has provided us with a varied set of experiences to assess the LMS as an element in the online learning environment. It is easy to critique the LMS, and there are no shortage of critiques of learning management systems, both particular systems and the concept of such systems in general. In light of the constraints that such systems place on educators and students, the critiques are understandable and warranted. We have approached the question by beginning with no LMS and then creating one for a particular class. We have also approached the issue by beginning with an institutionally based LMS and then as managers of such systems. We have also created free-standing applications for functions such as video discussion and content management and organization. Finally, we have adopted a position of drawing on some basic functionality of a light weight LMS and augmenting and supplementing that functionality with other applications, sometimes integrated and sometimes freestanding. In thinking about the appropriate role for the LMS, we have considered the implications of Nicholson’s [23] theory of loose parts and the notion that creativity and discovery require educators and students to be in environments where there is sufficient variability and flexibility to engender such activities. Learning Management

Loose Parts: Creating Learning Opportunities Beyond the LMS

403

Systems are intended to limit variability in pursuit of the goals of simplicity and manageability. Extending learning management systems with greater functionality across the board will not solve the dilemma they pose for individual educators because the resulting systems will be neither simple not able to meet immediate needs of the educator’s approach to learning. Adding fully integrated components is likely to be disappointing because such a strategy will add more constraints. Educators and students may well want and need more “parts,” but they must be “loose” if they are to support creativity and discovery. Acknowledgements The authors wish to acknowledge and thank the many collaborators who have worked with us on the many projects and applications discussed in this paper. We also acknowledge the inspiration provided by the many students who have participated in our online and on campus learning activities.

References 1. Agarwala, M., Chae, H., Anwar, F., Natriello, G.: Vialogues: an asynchronous video discussion tool that supports adaptive instruction and social reflective learning. In: Paper Presented at the Fourth Annual International Symposium on Emerging Technologies for Online Learning, San Jose, CA (2011) 2. Asana.https://asana.com/. Accessed 13 Mar 2023 3. Asunka, S., Chae, H., Natriello, G.: Towards an understanding of the use of an institutional repository with integrated social networking tools: a case study of PocketKnowledge. Libr. Inf. Sci. Res. 33, 80–88 (2011) 4. Baid, P., Chae, H.S., Anwar, F., Natriello, G.: Moodle discussion forum analyzer tool (DFAT). In: Aleven, V., Kay, J., Mostow, J. (eds.) Intelligent Tutoring Systems. Lecture Notes in Computer Science, vol. 6095, pp. 209–211. Springer, Heidelberg (2010). https://doi.org/10. 1007/978-3-642-13437-1_21 5. Brand, S.: How Buildings Learn: What Happens After They’re Built. Penguin Books, New York (1995) 6. Buddy Press. https://buddypress.org/. Accessed 13 Mar 2023 7. Chen, Y., Behar, P., Liu, X., Chae, H., Natriello, G.: Measuring the Video-Driven Discussion Engagement Using Item Response Theory Model [Poster Session]. In: AERA Annual Meeting, San Francisco, CA (Conference Canceled), April 2020 8. Cormier, D.: Rhizomatic learning: community as curriculum. Innovate: J. Online Educ. 4(5), June-July 2008 9. Hsiao, I-H., Han, S., Chae, H., Natriello, G.: Constructing an educational online video discussion tool to effectively engage the crowd. In: Paper presented at the Annual Meeting of the American Educational Research Association, Philadelphia, PA, April 2014 10. Instructor Role Plugin.https://wisdmlabs.com/instructor-role-for-learndash/. Accessed 13 Mar 2023 11. Lee, S.Y., Chae, H., Natriello, G.: Identifying user engagement patterns in an online video discussion platform. In: Paper presented at Educational Data Mining 2018 (EDM), Buffalo, NY, July 2018 12. Liu, X., Zhou, Z., Chae, H., Natriello, G.: On the use of filtered linear discriminant analysis for automatic discussion rating. In: Presented at Annual Meeting of American Educational Research Association, Washington, D.C., April 2016 13. Lutyens, D.: Playing fare. Des. Week 25(36), 16–178 (2010)

404

G. Natriello and H. S. Chae

14. Malhotra, M., Zhou, Z., Garg, M., Agarwala, M., Chae, H., Natriello, G.: Understanding usage patterns of video-driven discussion using latent semantic analysis. In: Paper presented at EDMEDIA: World Conference on Educational Multimedia, Hypermedia, and Telecommunications, Denver, CO., June 2012 15. Natriello, G.: Creating Learning Settings: Physical, Digital, and Social Configurations for the Future of Education. Routledge, New York (2023) 16. Natriello, G., Chae, H.: Crowd sourcing video learning resources: patterns and possibilities. In: Discovery Session at the Eighth International Symposium on Emerging Technologies for Online Learning, Dallas, TX., April 2015 17. Natriello, G., Chae, H.S.: Taking project-based learning online. In: Guralnick, D., Auer, M.E., Poce, A. (eds.) Innovations in Learning and Technology for the Workplace and Higher Education. Lecture Notes in Networks and Systems, vol. 349, pp. 224–236. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-90677-1_22 18. Natriello, G., Hughes B., Cocciolo, A., Chae, H.: When MySpace meets d-space: building a personal/institutional repository. In: Paper Presented at the Educause Annual Conference, Orlando, FL., November 2008 19. Natriello, G., Pattinsky, M., Chae, H., Cocciolo. A.: E-learning systems as research platforms: results from the networked education database (NED) project. In: Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, April (2007) 20. Natriello, G., Rennick, M.: An online journal as a virtual learning environment. In: Weiss, J., Nolan, J., Hunsinger, J., Trifonas, P. (eds.) The International Handbook of Virtual Learning Environments, pp. 821–848. Springer, Dordrecht (2006). https://doi.org/10.1007/978-1-40203803-7_30 21. Pearlman, C.: What if…? The Architecture and Design of David Rockwell. Metropolis, New York (2014) 22. Poll Maker.https://ays-pro.com/WordPress/poll-maker. Accessed 13 Mar 2023 23. Nicholson, S.: How not to cheat children: the theory of loose parts. Landscape Archit. Mag. 62(1), 30–34 (1971)

ChatGPT: Should It Have a Role in Education? Birgit Oberer

and Alptekin Erkollar

Abstract Chatbots, which can answer questions using artificial intelligence, are increasingly finding their way into everyday school life. Artificial intelligence has long been part of our everyday lives, even if we are not always aware of it. When search engines provide us with certain results depending on our previous online behavior, navigation systems help us avoid traffic jams, or voice assistants fulfill our music wishes, there is ultimately always an AI behind it. The program ChatGPT from OpenAI, released in November 2022, is a chatbot that is able to provide human answers to all kinds of questions using AI. ChatGPT can be used in different languages and it can ideally mimic the user’s writing style, allowing a believable and natural interaction. This publication analyzes the strengths and weaknesses of ChatGPT from the users’ point of view and from the pedagogical point of view, and it focuses on threats and opportunities that an AI tool like ChatGPT can generally exhibit. It explores the questions of whether educational institutions should ban AI applications such as ChatGPT or whether educators should adapt their style of teaching and give ChatGPT a specific place in the classroom. Keywords Artificial intelligence · Chatbot · ChatGPT · Education

1 Literature Review The balanced scorecard (BSC) represents a performance measurement instrument, where objectives, measurands, and strategic actions are categorized according to specific dimensions. The balanced scorecard was designed by Kaplan and Norton, who focus on finance, customers, learning and growth, an on an internal dimension. For every dimension, main targets, objectives, measurands, and measures have to B. Oberer (B) International Society for Engineering Pedagogy IGIP, Vienna, Austria e-mail: [email protected]; [email protected] A. Erkollar ETCOP Institute for Interdisciplinary Research, Klagenfurt, Austria © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_32

405

406

B. Oberer and A. Erkollar

be defined. The balanced scorecard approach was further developed and researchers added different dimensions [1–3]. The term ‘bot’ is an abbreviation of ‘robot.’ It describes a program operating autonomously on the Internet [4]. Grimme defines the term chatbot, as a bot which can interact or chat with a human user in natural language. Chatbots are only as intelligent as their scripts and the respective databases behind [4, 5, 9, 10]. Artificial intelligence refers to systems with intelligent, self-learning behavior that analyze their environment and act with some degree of autonomy [10, 11]. Artificial intelligence (AI) has become a catch-all term for applications that perform complex tasks that in the past required human intervention [7]. The European Commission has published 2022 ethical guidelines for teachers on the use of artificial intelligence and data for teaching and learning purposes. The guidelines focus on how AI can be used in schools to support teachers and students in teaching and learning and to facilitate the handling of administrative tasks in educational institutions. The guidelines are part of the Digital Education Action Plan (2021–2027) and were developed by a dedicated Commission expert group [6]. ChatGPT is currently on everyone’s lips and is described as a revolution in marketing. GPT stands for ‘generative pre-trained transformer’ and is an artificial neural network. The chatbot is based on the GPT-3 language model from OpenAI. This language model relies on deep learning technology and is trained with algorithms fed with an enormous amount of data. The system is pre-trained and can interact with humans via a chat window. The growing importance and potential of artificial intelligence in the form of chatbots has been apparent for some time. The great benefits of artificial intelligence, for example in the area of customer service, have led to a very strong further development of the technology. The ChatGPT program from OpenAI is one of the latest developments [8, 11]. The result of a Chat GPT query is an equation with billions of variables, which express in which contexts certain words occur and then put them into logical relationships. This can, for example, far exceed the results of previous search engines. The first tests are currently taking place in conjunction with Microsoft’s Bing search engine. China is launching its own AI model almost simultaneously.

2 Methodology A deductive, qualitative research approach with an evaluative research design is applied with a focus on task accomplishment. A guideline-oriented evaluation method is used, in which the object of evaluation is the user interface of the software. The usability and the experience of the users are evaluated on the basis of a previously created guideline. Basic tasks in the areas of scientific work and at the high school level (exemplified in the subjects of history, geography and business studies) were designed to be done by ChatGPT. Based on the results of the tool, during evaluation adaptations of the tasks are made and it is tested how (type of formulation, level

ChatGPT: Should It Have a Role in Education?

407

of detail) tasks must be in order to obtain satisfactory solutions on the part of the software tool from the user’s point of view.

3 Results We have asked ChatGPT to perform a SWOT analysis for ChatGPT. After that, we wanted to find out if ChatGPT knows who he is. We asked him to make a SWOT analysis about himself. ChatGPT created a SWOT analysis for itself and explained the concept in the I form. The results for both tasks consist of the basic concept of a SWOT analysis on the one hand and the advantages and disadvantages of ChatGPT on the other hand. The statements are vague and apply broadly to AI systems. In a further step, we wanted to find out if and how well ChatGPT can connect multiple concepts and contextualize them. We therefore gave ChatGPT the task to create a balanced scorecard for ChatGPT, integrating a SWOT analysis for the tool. Figure 1 shows an excerpt of the outcome. ChatGPT applies the balanced scorecard framework in a rigid manner and provides content for each dimension. The addition of at least one more dimension was not executed. ChatGPT did not consider renaming dimensions, despite it being necessary in the given case. We had requested from ChatGPT to integrate a SWOT analysis for each dimension of the created balanced scorecard. The tool did not implement the requested integration of a SWOT analysis for each dimension. A balanced scorecard was created, and a SWOT analysis was added at the end, without any recognizable reference to the balanced scorecard. The tool performed poorly in completing the task. If given precise specifications, ChatGPT can produce a higher quality result based on the balanced scorecard in Fig. 2. This means that, if one is already well versed in a topic and manage to incorporate personal knowledge into the question, they will get a very satisfactory work from ChatGPT. If given a task on an unfamiliar topic, one should expect to receive a poor and inadequate result from ChatGPT. The more knowledge and detailed specifications the user provides, the more likely the result from ChatGPT will be useful. Subsequently, we asked ChatGPT to evaluate the balanced scorecard created by the tool itself. When ChatGPT was given the task of evaluating the balanced scorecard created by the tool itself, and making suggestions for improvement, the following question was posed to the user by ChatGPT instead of an analysis: ‘Can you provide a brief explanation of what natural language processing (NLP) is and how it is used in artificial intelligence (AI)?’ The question posed to the user was not related to the task. It was surprising that ChatGPT asked the user questions about its technology. The task was entered again. The result is shown in Fig. 3. ChatGPT tells the user that the balanced scorecard being analyzed is so bad that it is not worth analyzing. In the analysis in the scientific area, an analysis was carried out by the authors for a selected topic as an example. The focus of this publication is on education at the school and higher education levels, rather than the scientific area.

408

B. Oberer and A. Erkollar

Fig. 1 A balanced scorecard for ChatGPT with a requested in-dimension SWOT analysis

A guide was prepared for doing the analysis. In the case of the balanced scorecard, we queried in Fig. 2. We had expert knowledge. In the task presented in Fig. 1, we assumed a lack of knowledge about the subject. The task show in Fig. 3 was given to the tool 10 times in a row. Only with results 9 and 10 could ChatGPT establish a reference to the task and offer a solution. It should be noted that the analysis was superficial and did not provide any suggestions for improvement, contrary to the task definition. Figure 4 shows the result of the 10th query. According to the evaluation guide, the same request has now been made in different languages: German, Italian, Spanish, Turkish and Danish. As examples, the results are presented in German and Turkish. The German Balanced Scorecard created by ChatGPT is shown in excerpts in Fig. 5. The tool did not use technical terms available in the German language. Instead, the integrated AI translation tool apparently translated the text poorly from English (since the database consists of English texts) into German. For this reason, the result in German is not satisfactory. Furthermore, the German version is formulated more

ChatGPT: Should It Have a Role in Education?

409

Fig. 2 A balanced scorecard with concrete specifications for ChatGPT

Fig. 3 ChatGPT evaluating the balanced scorecard generated by ChatGPT, second attempt

superficially compared to the English Balanced Scorecard and does not present essential contents. The user rated the result as unsatisfactory. The Balanced Scorecard in Turkish presented a similar picture. In addition to technical terms that were not used correctly, terms were simply translated incorrectly (an excerpt is shown in Fig. 6). Already the first part, the headline is wrong. Instead of the term Balanced Scorecard, Corporate Scorecard is used, which is not a synonym. At the end of the ChatGPT analysis for scientific papers, the tool was given the task of translating terms from a balanced scorecard created by the tool itself in

410

B. Oberer and A. Erkollar

Fig. 4 ChatGPT evaluating the balanced scorecard generated by ChatGPT, 10th Attempt

Fig. 5 ChatGPT generated a balanced scorecard in German language

German into English in unchanged order. The results in Fig. 7 indicate that some of the terms in the German balanced scorecard created by ChatGPT were not translated into English, and some of the translated terms were not related to the intended task. Users found the result of the task to be inadequate. ChatGPT performs at a level comparable to a high school student. In the subjects geography, history, and business studies, according to the guide, various tasks from a student’s perspective as well as from a teacher’s perspective were given. Students, all English natives, at different school levels from different school types were asked to solve tasks from their current school life through ChatGPT.

ChatGPT: Should It Have a Role in Education?

411

Fig. 6 ChatGPT generated a balanced scorecard in Turkish language

Fig. 7 ChatGPT translation task

There were more than thirty students involved in this project. All tasks were passed to ChatGPT in English. It can be seen among the students that for all three subjects they were mostly satisfied with the result of ChatGPT. Teachers evaluated the results on average as moderate to high, which was lower than the students’ assessment. The student assignments were related to essay writing and preparing content for presentations. The results differed for teachers. Tasks were given to ChatGPT in the areas of quiz preparation, handouts, and test preparation. Teachers were generally satisfied with the quizzes created by ChatGPT when they revised the questions themselves. Unedited quizzes were rated low or moderate. Teachers rated the handouts created by ChatGPT as high or very high quality. On

412

B. Oberer and A. Erkollar

average, teachers evaluated ChatGPT’s ability to create test assignments as moderate. Refer to Table 1 for an overview. Table 1 is an overview. Table 1 ChatGPT performs High School tasks for students and teachers Subject: Geography Perspective

Type of Task

Number of tasks performed

Average User satisfaction*

Average Evaluation result by teacher*

Student

Essay writing

50

VERY HIGH

MODERATE to HIGH

Presentation

50

VERY HIGH

HIGH

Teacher

Preparation of quizzes (no change by teacher)

50

MODERATE

Preparation of quizzes (with manual adjustments by teacher)

50

HIGH

Preparation of handouts on selected topics

30

HIGH

Item creation for tests

15

MODERATE

Subject: History Student Teacher

Essay writing

50

VERY HIGH

MODERATE

Presentation

50

VERY HIGH

HIGH

Preparation of quizzes (no change by teacher)

50

LOW

Preparation of quizzes (with manual adjustments by teacher)

50

HIGH

Preparation of handouts on selected topics

30

HIGH

Item creation for tests

15

MODERATE

50

VERY HIGH

MODERATE HIGH

Subject: Business Studies Student

Essay writing Presentation

50

VERY HIGH

Teacher

Preparation of quizzes (no change by teacher)

50

MODERATE

Preparation of quizzes (with manual adjustments by teacher)

50

HIGH

Preparation of handouts on selected topics

30

VERY HIGH

Item creation for tests

15

MODERATE

*

VERY HIGH - HIGH – MODERATE – LOW

ChatGPT: Should It Have a Role in Education?

413

4 Discussion Should educational institutions ban AI applications or should get ChatGPT a specific place in the classroom? Google and other companies have already developed similar tools of their own. However, these have not been published so far with the reference to possible dangers. As with every new technology, the proponents and admonishers are once again divided. On the one hand, there are calls for a revolution in creativity and the increase in knowledge; on the other hand, there are fears of the demise of entire industries and the education system. Educators and plagiarism hunters still have their grief with ChatGPT and similar AI tools. Easily controllable copy and paste for homework or seminar papers is a thing of the past. Because AI has long been intelligent enough not to give the same answer twice to the same question, even if the content remains relatively constant. The writing style of ChatGPT when writing final papers as well as term papers is quite alright. It should also be possible by feeding ChatGPT with the previous texts of a human author to increasingly imitate and continue his style. ChatGPT is not new; it is a development stage in Natural Language Processing. What is new is not the technology behind it, but the improved quality of the results, the focus on the interaction between a user and the AI system, and for the user especially attractive; there is a web page where one can register and by the simple, rather inconspicuous input into a single text field dive into the infinite expanses of text generation. It should be noted that the ChatGPT database content for learning is largely from English sources. For the communication with users, an intermediate AI tool is used for translation. Thus, problems can arise on the one hand from queries made by users in non-English language on English-language content and, on the other hand, from poor interaction between tool and user due to immature AI translation tool. The advent of mostly freely available artificial intelligence will fundamentally change the lives of teachers and students. Students have the option to use ChatGPT as a tool for essay writing, as an aid in writing longer papers, in some cases for text problems in mathematics, as a translation tool, or as an ‘idea generator.’ Teachers can use Chat GPT as a tool for creating assignments and handouts, as a guide for methods to use, or for example as a creator for writing assignments and quizzes. The question arises as to how educational institutions should deal with ChatGPT and other AI applications. Below are some options. One could wait until the problem solves itself. This will not happen. Teachers could only accept handwritten homework from their students. This would not be effective as students could still use ChatGPT to generate the content. Blocking the ChatGPT website from the school’s Wi-Fi is also not effective, as many students can open hotspots themselves. One could actively use the tool in the classroom to generate added value. Teachers could take the tool as a given and adapt it for their lessons. ChatGPT is currently free, but there is already a paid plus option. If ChatGPT were to become paid in the future, there would be other comparable free applications and ChatGPT would also continue to be used by a certain group of students. There are many factors

414

B. Oberer and A. Erkollar

that need to be considered when it comes to an approach to using AI in education. It is essential that educational institutions develop policies to ensure that AI applications are used safely for students and provide value to education. Teachers need to teach students how to responsibly use ChatGPT and other AI tools. For this purpose, teachers need to learn about the tool in advance. Teachers should motivate students to be creative and trust in their abilities. They can give tasks in which students bring in personal commitment, in group work in the class skills related to critical thinking can be promoted, or multi genre projects conducted. For one genre assignments students might get support from ChatGPT. For multi-genre projects, tools like ChatGPT might not be useful at all. When students are empowered, they feel less stressed and feel that they can write essays on their own and do not need assistance from an AI tool. There are challenges associated with the introduction of AI in education.

5 Conclusions Science has long sought to recreate the complex human mind artificially. With the rise of digitization and new technologies, especially artificial intelligence (AI), we are seeing significant transformations in socio-economic developments. AI is increasingly important in education as well, with more and more learning platforms being developed to enhance student learning. AI systems are being used in classrooms to diagnose learning problems and improve learning outcomes. These systems also have the potential to make instruction more personalized and individualized. However, to establish user trust, AI applications must be engineered securely, reliably, and provide data sovereignty. Ethical and social responsibilities must also be considered, as well as data privacy concerns. While ChatGPT may be a passing hype, it is just one example of the ever-evolving tools that will follow as AI continues to usher in a paradigm shift in education. Teachers can use ChatGPT to support the development of learning materials, but it should be emphasized that it is only a helper and cannot replace communication skills and other methods necessary for learning. If teachers wish to avoid cheating, they should refrain from asking students for simple summaries and instead stimulate higher, more complex thought processes with assignments that emphasize analysis and interpretation. In conclusion, ChatGPT can be a valuable tool for approaching different content topics in education. However, it is important to recognize that it is only an enabler and not the end result. ChatGPT’s output should be seen as a source of information to be critically questioned and, if necessary, compared with other sources.

ChatGPT: Should It Have a Role in Education?

415

References 1. Erkollar, A., Oberer, B.: Balanced Scorecarding. Strategische Unternehmenssteuerung und Leistungsmessung, Tectum (2012) 2. Erkollar, A., Oberer, B.: Employee connection. The multidimensional team management scorecard. Procedia Econ. Finan. 26, 942–945 (2015) 3. Oberer, B.: Helping universities to be more competitive: the program design scorecard (PDS), shown in the example of a management information systems (MIS) PhD program. In: Lee, G (eds.) Advances in Education Sciences. Management and Sports Science Institute, vol. 1, pp. 84–89 (2013). ISBN: 978–981–07–5946–9 4. Oberer, B., Erkollar, A., Stein, A.: Social bots – act like a human, think like a bot. In: Stumpf, M. (eds.) Digitalisierung und Kommunikation, Europäische Kulturen in der Wirtschaftskommunikation, vol. 31. Springer VS, Wiesbaden (2019). https://doi.org/10.1007/978-3-658-261139_19 5. Grimme, C., Preuss, M., Adam, L., Trautmann, H.: Social bots: human-like by means of human control? Big Data 5(4), 279–293 (2017) 6. European Commission: Communication From The Commission to the European Parliament, The Council, The European Economic and Social Committee and The Committee of The Regions, Digital Education Action Plan 2021–2027, COM(2020) 624 final. https://eur-lex.eur opa.eu/legal-content/EN/ALL/?uri=CELEX:52020DC0624. Accessed 03 Mar 2023 7. Kitzmann, A.: Künstliche Intelligenz: Wie verändert sich unsere Zukunft? Springer, Cham (2022) 8. OpenAI. https://openai.com/blog/chatgpt. Accessed 03 Mar 2023 9. Okonkwo, C.W., Ade-Ibijola, A.: Chatbots applications in education: a systematic review. Comput. Educ. Artif. Intell. 2, 100033 (2021) 10. Yang, S., Evans, C.: Opportunities and challenges in using AI chatbots in higher education. In: Proceedings of the Proceedings of the 2019 3rd International Conference on Education and E-Learning, Barcelona, Spain, 5–7 November 2019, pp. 79–83 (2019) 11. Morrison, R.: OpenAI’s New Chatbot ChatGPT could be a Game-Changer for Businesses. https://techmonitor.ai/technology/ai-and-automation/chatgpt-openai-chatbot. Accessed 03 Mar 2023

Education 5.0: Design Thinking Goes ICT Birgit Oberer

and Alptekin Erkollar

Abstract In Education 5.0, students are protagonists rather than passive listeners. Classes are more collaborative, individualized, and focused on developing hard and soft skills. New technologies are used to provide instruction that focuses on the student, not the technology. Digital devices, infrastructure, and platforms remain critical, but they are tools that should support learning. Design thinking is a method and process for finding solutions to complex problems. This publication presents a project funded by the Austrian Research Promotion Agency as part of the Innovation Labs for Education program. The focus is on the design projects of students in the field of information and communication technology. It presents the newly developed approach of the project, in which the students apply the Design Thinking process from Stanford University to their design projects, mostly in the areas of social media, web design, audio and video editing, and animation. Keywords Creativity · Design thinking · Education 5.0

1 Introduction Working with the Design Thinking process is suitable for education. Young people are taught the skills they need to meet future challenges. Teachers play an essential role in the implementation of learning processes. For this reason only students and teachers will receive support in the form of guidelines to get to know Design Thinking and to use the process in their teaching. Design Thinking promotes independent work and helps to strengthen creative self-confidence. By working with Design Thinking, young people develop several key competencies, such as problem-solving skills, team skills, project skills, and creative confidence. B. Oberer (B) International Society for Engineering Pedagogy, Vienna, Austria e-mail: [email protected] A. Erkollar ETCOP Institute for Interdisciplinary Research, Klagenfurt, Austria © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_33

417

418

B. Oberer and A. Erkollar

Design thinking supports teachers in creating student-centered teaching concepts and materials, and provides an opportunity to shift their role from facilitator to learning guide. Working with design thinking motivates both students and teachers to approach problems constructively and to be curious and attentive in their search for solutions. This creates an optimistic work environment that supports both students and teachers to enjoy working on projects.

2 Literature Review Digital transformation is changing all aspects of life and work around the world. More and more people are leaving their home countries, mostly in search of a better life. This change places demands on the evolution of educational processes that go far beyond digital transformation. The COVID-19 pandemic once again highlighted the need for major changes in the school system. The COVID-19 pandemic forced schools to adapt to remote and hybrid learning models, highlighting the need for a more flexible and adaptable education system. It also exposed the digital divide, where many students lacked access to technology and Internet connectivity, exacerbating existing inequalities in education [1, 13] In response, governments and educational institutions have invested in digital infrastructure and tools to ensure that all students have equal access to learning opportunities. Digital transformation is also changing the nature of work, with automation and artificial intelligence replacing many routine and manual tasks. This requires a new set of skills and competencies, such as critical thinking, creativity, and problem solving, which are not easily automated. Education must therefore prepare students for a future in which the skills and knowledge required for success will continue to evolve rapidly. In summary, digital transformation and the COVID-19 pandemic have accelerated the need for major changes in the education system. Education 5.0 offers a new paradigm for teaching and learning that embraces technology, personalized learning, and the development of essential skills and competencies [1, 13]. Education 5.0 is a term used to describe the latest phase of educational transformation, characterized by the integration of advanced technologies such as virtual and augmented reality, Internet of Things (IoT), and artificial intelligence into teaching and learning practices. Education 5.0 aims to provide personalized and learnercentered education, where students have the autonomy to direct their own learning and teachers act as facilitators and mentors. In Education 5.0, students are expected to develop not only technical skills, but also soft skills such as critical thinking, creativity, collaboration, and communication, which are essential for success in the twenty-first century. In addition, Education 5.0 emphasizes the importance of lifelong learning and continuous professional development, where learners are equipped with the skills and competencies necessary to adapt to a rapidly changing world [1]. Education 5.0 has to be rethought, has to create individual learning opportunities, has to teach current and future relevant competencies, has to offer flexible spaces instead

Education 5.0: Design Thinking Goes ICT

419

Fig. 1 Stanford University Design Thinking Process

of rigid structures, creates changed requirement profiles for teachers and poses new challenges for school administrators [1]. Design thinking is a well-organized, step-by-step approach to solving challenging and multifaceted problems. Design thinking was first developed at Stanford University to foster creative ideas. Stanford Design Thinking has become a powerful tool for companies and organizations to solve complex problems with creative solutions. In contrast to other approaches in science and practice that focus on technical feasibility, Design Thinking focuses on user needs and user-centered invention. Design thinking is named after the working method of designers who rely on intuition, careful observation, and a strong focus on the user’s needs and preferences. Figure 1 shows the design thinking process developed by Stanford University [12]. Design thinkers look at a problem from the user’s point of view and put themselves in the user’s shoes. The method is all about inventive design. Stanford University’s design thinking process consists of five steps: empathize, define, ideate, prototype, and test. The empathize phase focuses on identifying the user’s wants (physical and emotional needs). In the define phase, it is decided what to focus on and in which direction to develop the process. Based on this, a formal problem description is developed. In the subsequent Ideate phase, the focus is on identifying solutions. Creative approaches are made possible to find user-oriented solutions. In the prototyping phase, the developed ideas are made tangible. Approaches are developed with which users can interact. In the testing phase, feedback is gathered from users to better understand the problem [9]. To implement design thinking, a constant feedback loop between solution developers and the target audience is essential. Design thinkers interact with end users, observe their processes and behaviors, and ask them questions to understand their needs and preferences. To ensure practical results, ideas and solutions are quickly transformed into prototypes that are communicable and visible to potential users, who can provide feedback long before the final product is completed or launched [10]. By prioritizing the human perspective, design thinking enables the creation of innovative and engaging products, services, or experiences that are not only attractive, but also viable and marketable [2–4]. There is a growing emphasis on fostering creativity in education. Educators need to create innovative educational frameworks, teach students how to be imaginative,

420

B. Oberer and A. Erkollar

and help them use their creativity throughout their academic and professional lives. To incorporate creative components into education, educators can focus on various creative aspects. Lego Serious Play (LSP) is an innovative method for problem solving, communication, and creativity development. It is based on the idea that any thought, experience or project can be visualized in the form of three-dimensional Lego models. Serious Play means that concrete topics and problems can be worked on in a moderated process. This takes place in a playful context (play), but the method ensures serious communication and goal-oriented topic processing (serious). A very wide range of topics can be addressed with LSP [5–7, 11]. By using an innovative process or specific facilitation techniques, complex problems can be solved with clear, viable, and sustainable solutions that can be implemented immediately. LSP enables the creation of a fresh, well-defined and shared vision for the future of a product, brand or company. It can also be used to reposition teams or divisions within an organization, foster a common language and identity, sustainably improve teamwork, and uncover effective solutions. [5–8].

3 Methodology The research question, “How can the outcome of student design projects be improved by combining the Stanford University Design Thinking process with the creative learning components of Lego Serious Play?” will be explored using a mixed methods approach that combines qualitative and quantitative research methods. LSP was integrated into the empathize, define and ideate phases of the Stanford University Design Thinking process. Figure 2 shows the process steps involved. Fig. 2 Design Thinking Process with integrated creative elements

Education 5.0: Design Thinking Goes ICT

421

The study aims to evaluate the effectiveness of both the design thinking process and the LSP approach in improving the outcomes of students’ design projects. Qualitative research methods, such as interviews and focus groups, will be used to gather data on students’ experiences and perceptions of the impact of the interventions on their learning and design outcomes. Quantitative research methods, such as surveys and assessments of design outcomes, will be used to measure the effectiveness of the interventions and compare them to a control group that does not receive the interventions. Overall, this mixed-methods research design allows for a more comprehensive understanding of the impact of the interventions on the outcomes of students’ design projects. It is important to note that this research is part of an ongoing project, and the results of this study will focus on measuring and evaluating the effectiveness of the interventions. The design of this new approach has been completed, but the results of the measurements will be available later. This study is innovative in its combination of the design thinking process and the creative learning elements of LSP in student design projects. The key success factors for the project are the sustainable application of the “Design Thinking with Creative Learning Elements” process developed in the project and the availability of the project results as an open educational resource. Project outputs will include guidelines for teachers and students, best practices for final projects, and a process for students to share their project experiences. Customer satisfaction will be measured and evaluated for students, capstone clients, client clients, and instructors, with appropriate indicators for each group. Ongoing expansion of the guidelines and the “students for students” process will contribute to the sustainability of the project.

4 Results and Discussion 4.1 Project Motivation In the 5th year, students in the ICT specialization complete projects for companies, associations or other institutions. Students are trained in general implementation in the areas of social media, web design, audio and video editing, and video editing and animation. The focus is on technical requirements and skills, such as the ability to create a website. The focus is not on the process that precedes this (technical) implementation. This upstream process includes requirements analysis. This is where functional requirements, non-functional requirements, design conditions, and process conditions are defined. User requirements analysis is an essential step in the design process. Requirements create a common understanding between the designer and the customer of what a digital product should be able to do. These requirements represent the “real” world. Different stakeholders (clients, customers, developers) are considered.

422

B. Oberer and A. Erkollar

This means that different ways of looking at the project, different conceptual images and vocabulary become visible, and the people (groups) involved need to agree on a common view. A requirement is an expression of a desired behavior of the product from the user’s point of view. Requirements management is a means of managing change needs. If designers, like students in corporate projects, do not perform a comprehensive requirements analysis, this can lead to multiple problems. Stakeholder requirements must also be prioritized into the categories of essential requirements (must-be criteria), desired requirements (not critical, but important), and optional requirements (nice-to-have criteria). After a specification is created, implementation planning takes place. Only then does the design of the product begin. In the context of education, a (digital) product is an output that is used in social media, web design, audio and video editing and animation, such as websites, social media presences for companies, images, videos, flyers, animated short films. In order to be able to analyze the requirements for the projects of the ICT students, the Design Thinking approach of Stanford University is implemented. This process consists of five steps: empathize, define, ideate, prototype, and test. The empathize phase is at the core of any human-centered design process. During this phase, designers strive to understand people’s needs, preferences, and behaviors in the context of the design task at hand. It is critical for designers to understand the ways, means, and reasons why stakeholders do what they do. In the Define phase, designers aim to construct a relevant and feasible problem statement along with actionable solutions. The Ideate phase focuses on brainstorming and generating innovative ideas that enable the transition from identifying problems to creating effective solutions for stakeholders. The Design Thinking process used in this project in combination with LSP focuses on the process phases Ideate, Prototyping, and Testing. In order to give an impression of how LSP can be used in this context, a task for the Ideate phase is considered as an example. Creating business ideas related to bicycles using individual Lego bricks as metaphors. The focus is on the Ideation phase, and the exercise involves setting a timer for 10 min, giving each participant a handful of individual Lego bricks, and encouraging them to use the bricks to create metaphors that represent different aspects of the bicycle and its components. At the end of the 10 min, each participant shares his or her metaphor with the group and explains how it relates to a business idea or concept, while the group discusses and expands on each other’s ideas. Below are some examples of the metaphor’s students created with Lego bricks in this assignment. A gear represents innovation and the need to constantly shift and adapt to changing markets. A spoke represents the importance of communication and collaboration within a team. A handlebar represents leadership and the need for a clear direction and vision for the company. A chain represents the interconnectedness of different parts of the business and the need for each part to work together smoothly. A seat represents the importance of comfort and customer experience in the business. A wheel represents the cyclical nature of the business and the need to constantly move forward and evolve. This exercise can help participants think creatively and generate new business ideas by using metaphors and visual representations. The use of individual Lego bricks allows for quick and easy manipulation of the metaphor,

Education 5.0: Design Thinking Goes ICT

423

Fig. 3 Metaphors created by students during a design thinking Ideate session

making it easy to experiment with different ideas and concepts. Figure 3 shows examples for metaphors. The prototyping and testing phases are concerned with turning the collected ideas into design products. This innovation project focuses on the empathize, define, and ideate of the design thinking process. In the fourth year, students are introduced to the concepts of the Design Thinking process and LSP and learn how to apply them to ICT application projects. Students then test these concepts in small projects using guidelines developed as part of this project. In their final projects (projects for companies, associations or other institutions), the students apply the skills they acquired in the 5th grade. LSP CLE are used in several phases of the Design Thinking process. Using the Design Thinking process and CLE, students learn to think globally and at a meta-level, to map a meta-level, and to create a meta model of the idea.

4.2 Innovation Aspects As part of the project, students will be introduced to the Design Thinking process and learn how to apply it to requirements analysis. This should significantly improve the quality of requirements analysis results. More accurate results in turn tend to lead to digital products that better meet the actual needs of stakeholders. It is an innovation that students in the ICT course use the design thinking process in their design projects. To further improve the results of the empathize, define, and ideate phases of the design thinking process, the creativity component is integrated into these process steps. Creative learning elements are used in the empathize, define, and ideate phases of the Design Thinking process. These represent a significant extension and improvement of the Design Thinking process. In particular, LSP is used. It can be used in strategy workshops, ideation workshops, or problem solving workshops or in collaborative learning. The use of LSP in Design Thinking has generally not taken

424

B. Oberer and A. Erkollar

place. The reason for this is that LSP works best when a complicated environment needs to be penetrated. The goal and the influencing factors are known, and the team has to develop a new strategy, for example. There is a rough idea of the problem, which must first be validated in the context of design thinking. Only after going through the phases of the Design Thinking process does one return to the sub-step of “defining the point of view,” understanding the problem and observing. Therefore, LSP is generally not used in Design Thinking. In this project, LSP is used in the empathize, define, and ideate phases of the Design Thinking process. This should improve the results of the requirements analysis.

4.3 Key Success Factors The first essential feature for the success of the project is that the “Design Thinking with Creative Learning Elements” process developed in the project is sustainably applied in ICT education and that the project results are made available as an open educational resource. The following outputs have been defined for the project. Guidelines for teachers who want to use Design Thinking with or without the addition of Creative Learning Elements (CLE) in their teaching of ICT or a similar subject. These guidelines explain the necessary steps for teachers to prepare themselves and their students for the use of Design Thinking and CLE. The guidelines are also available at other educational institutions with the same or similar educational focus, as well as — after appropriate adaptation — at other educational institutions (national and international) for teaching in the fields of IT, media management, digitalization, or similar. Guidelines for students when they start working on small projects in the 4th semester. They will find a summary of the Design Thinking process and help on how to use creative learning elements in projects. These guidelines will also be used in the design projects for companies in year 5. Students can contribute their experiences from the projects to the guidelines, which helps to expand and improve them. Best Practices of Final Projects. As part of the project, documentation of all final projects created by students using Design Thinking and CLE projects will be made publicly available in the form of best practices. By Students for Students — Exchange of Project Experiences. In order to promote sustainability within the educational focus, a process will be introduced in which students of the final year will be able to exchange with students of the fourth year and present their projects to students of the fourth year and explain the experiences gained for the respective project. The second essential feature for the success of the project is that, within the framework of the implemented process, the guidelines are continuously extended, e.g. due to changing customer requirements, and this contributes to the sustainability of the process. The “Students for Students” process implemented in the ICT area is also a success factor, as each graduating class

Education 5.0: Design Thinking Goes ICT

425

passes on its experience to the 4th class, thus contributing to the sustainability of the process. In the context of the developed process “Design Thinking with Creative Learning Elements,” the customers are the students, the clients of the final projects, if applicable, the clients of the clients, as well as the teachers. For each group of people the satisfaction is to be measured and evaluated with appropriate indicators.

5 Conclusions The purpose of this study is to examine the extent to which combining the Stanford University Design Thinking process with LSP can improve the quality of student design projects. In order to accomplish this goal, a mixed-methods research approach will be used, incorporating both qualitative and quantitative research methods. The qualitative research methods include interviews and focus groups, which will be used to collect data on students’ perceptions and experiences of the impact of the interventions on their learning and design outcomes. Quantitative research methods include surveys and assessments of design outcomes to measure the effectiveness of the interventions and compare them to a control group that does not receive the interventions. The key success factors of the study are the sustainable application of the “Design Thinking with Creative Learning Elements” process developed in the project and the availability of the project results as an open educational resource. The study is innovative in its combination of the Design Thinking process and the creative learning elements of LSP in student design projects. The study aims to create guidelines for faculty and students, best practices for capstone projects, and a process for students to share project experiences. The project will measure customer satisfaction for students, final project clients, client clients, and instructors, with appropriate indicators for each group. This ongoing project has made significant progress toward its goals. The guidelines for faculty and students are available, and in the current academic year, best practices are regularly documented and the studentto-student process has been implemented. Based on the current status of the project, it is expected to successfully achieve its goals and contribute to the advancement of the field.

References 1. Oberer, B., Erkollar, A.: Education 5.0: the effectiveness of game based learning strategies on post-pandemic educational competences. In: Durakbasa, N.M., Güne¸sGençyılmaz, M. (eds.) Towards Industry 5.0: Selected Papers from ISPR2022, pp. 243–254. Springer International Publishing, Cham (2023). https://doi.org/10.1007/978-3-031-24457-5_20

426

B. Oberer and A. Erkollar

2. Oberer, B., Erkollar, A., Stein, A.: Creative education shown in the example of a business course on innovation. In: Proceedings of the 2nd International Scientific Conference» Teaching Methods for Economics and Business Sciences, University of Maribor Press (2019) 3. Erkollar, A., Oberer, B., Ekren, G.: Advanced MIS and Digital Transformation for Increased Creativity and Innovation in Business (Advances in Business Strategy and Competitive Advantage), Business Science Reference (2019). 4. Oberer, B., Erkollar, A., Stein, A.: Back to stone age, creative education shown in the example of a business course on innovation. In: Proceedings of the 2nd International Conference on Teaching methods for economics and business sciences, Maribor University (2019) 5. Oberer, B.: Integrating creative learning elements in higher education shown in the example of a management information systems (MIS) courses. Education. 3(6), 319–324 (2013) 6. Oberer, B.: Creativity moves ahead: creative learning elements in higher education. The ML-LL approach. Int. J. Educ. Learn. 2(2), 1–14 (2013) 7. Beghetto, R.A., Kaufman, J.C.: Fundamentals of Creativiity. Creativity now! 70(5), 10–15 (2013) 8. Beghetto, R.A.: Creativity in the classroom. In: Kaufman, J.C., Sternberg, R.J. (eds.) The Cambridge handbook of creativity, pp. 447–466. Cambridge University Press, Cambridge (2010) 9. Vikas, T., Vinay, C., Habeeba, A., Rizwan, N.: Design thinking: a review paper. Int. J. Adv. Res. Sci. Commun. Technol. (IJARSCT). 2(2), 405–412 (2022) 10. Deepa, P.: A study on the concepts of design thinking. Int. J. Eng. Appl. Sci. Technol. 4(12), 269–272 (2020) 11. Heikkinen, S., Nemilentsev, M.: Lego serious play as an innovative method of learning. In: Innovative Teaching and Learning Methods in Multicultural Environments, pp.18–26 (2014) 12. Plattner, H. Institute of Design at Stanford: An Introduction to Design Thinking PROCESS GUIDE (2019). https://web.stanford.edu/~mshanks/MichaelShanks/files/509554.pdf 13. Togo, M., Gandidzanwa, CPi.: The role of Education 5.0 in accelerating the implementation of SDGs and challenges encountered at the University of Zimbabwe. Int. J. Sustain. Higher Educ. 22(7), 1520–1535 (2021). https://doi.org/10.1108/IJSHE-05-2020-0158

Redesigning Education Using Serious Games Jenny Pange , Liudmila Rupsiene, and Agostino Marengo

Abstract Technology is included in most educational environments. Gamification in education is experiencing a new era of application. We apply serious games either for learning or to redesign the teaching process. It is well documented that students attending online courses have greater engagement in the learning process when they use games during the learning process. So, in online courses, addressed to university students, many web applications use online games nowadays. Additionally, in online courses, self-regulated learning is considered an important constraint for learning by many researchers. Apart from that, another issue that is worth considering in an online learning process, is the correlation of self-regulated learning with the ability of self-assessment using gamification. Since the introduction of online learning in the universities during and soon after COVID-19 pandemic, the Lab of New Technologies, and Distance Learning of the University of Ioannina in cooperation with other higher educational Institutes, created MOOCs to support online courses concerning the Applications ICTs in education and entrepreneurship. In this audit study, a self-selected group of students participated in online learning activities for the course ‘Applications of ICT.‘ Students were highly motivated by the course, and they completed the course assignments in due time. All students reported that they had self-regulation ability, and they improved their learning experience using online serious games for self-assessment. To test their self-assessment ability, we applied the ‘Classcraft’ online game (https://www.classcraft.com/). This game creates learning pathways and supports personalized learning. Additionally, it enhances online learning and offers an innovative way for the self-assessment process. J. Pange (B) University of Ioannina, Ioannina, Greece e-mail: [email protected] L. Rupsiene (B) Klaipeda University, Klaipeda, Lithuania e-mail: [email protected] A. Marengo (B) University of Foggia, Foggia, Italy e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_34

427

428

J. Pange et al.

Keywords Redesign education · Games · Self-regulated learning · Self-assessment

1 Introduction The rapid growth of Information Computer Technologies (ICTs) has changed the teaching and learning methods in formal, informal, and non-formal education [1–3]. It is well documented that during the COVID-19 period and post-COVID period, students and teachers made extensive use of ICT, and several of them applied artificial intelligence and learning analytics. For better cognitive, emotional, and behavioral engagement of students in classroom activities, technology must be used. Students and teachers must use the opportunities offered by digital technologies [4]. Furthermore, ICTs, especially online games and gamification, have been widely used in classrooms for many years [1, 5, 6]. Recent studies report that teachers consider not only ICTs but also gamification as valuable teaching techniques because games offer flexible ways for learning and provide alternative ways to evaluate students’ progress [7, 8]. The advantage of gamification in educational settings is that it offers collaborative learning through a gaming process in non-game environments [9]. It seems that, in serious games, students not only upgrade their cooperation skills, but they also improve their critical thinking skills and problem-solving skills [1, 7]. Thus, serious online games motivate and upskill learners for dynamic interactive learning engagement [10, 11, 12]. Additionally, through online games, students can play, enjoy the learning process, and easily understand complex learning material [7, 8]. Nowadays, many traditional games used for outdoor activities are replaced by recognized digital games like Nintendo’s Wii [13] and many others. This way, digital games become very attractive and promising tools for learning because they include interactive learning material, games, colorful designs, plots, role characters, assessment tools, and learning tasks [3, 6]. Online games strengthen self-regulated learning [5] and include self-assessments for evaluation without stress, either in a simple gaming process or in a cyclical self-assessment process that has assessment criteria, self-directed feedback, and self-reflection [14, 15]. Self-regulated learning is a process for systematic learning that activates and sustains cognition and behavior towards pre-set learning goals and is a ‘studentcentered method of learning’ [5, 16, 17]. Self-regulated learning is applied to all types of learning, whether formal, informal, non-formal, or lifelong [5, 17, 18]. Lifelong learning is the continuing engagement of learners in the learning process. The development of ICT competencies and self-regulated learning improves the usability of lifelong learning courses [3, 14]. In self-regulated learning, the ability of the learner for self-evaluation makes the whole process a dynamic one [17], because it helps students to improve their academic performance, get better control of their learning, increase self-motivation, and empower them to use up-to-date technologyenhanced learning environments [5, 7, 17, 19, 20]. Other researchers [15, 21] focus

Redesigning Education Using Serious Games

429

on the skills the self-assessment process offers to learners, and they refer to the skills of collection of information and the ability of students to recognize the strengths and weaknesses of their learning process. Therefore, to reform education, educators must consider how to apply new teaching methods, including ICTs, gamification, personalized learning, and emotional learning. Higher educational institutions must prepare teachers for this reform from their undergraduate and postgraduate studies and familiarize students with online games like ‘Classcraft’, ‘Kahoot!’ or ‘Quizizz,’ which are widely used nowadays for learning and evaluating students’ performance [22–26]. Recent studies declare that students in primary education develop interests in educational courses using ‘Classcraft’ and improve their engagement and motivation in classroom activities. Additionally, students are motivated to learn when they participate in ‘Kahoot!’ sessions and are very responsive to online assessments using ‘Quizizz’ [27–29]. The ‘Classcraft’ online real-time game (https://www.classcraft.com/) [30] is a serious role-playing game that combines game tools for learning and self-assessment. The ‘Classcraft’ game is used for classroom management because it can foster the desired behavior of students pre-set by the teacher, it offers personalized learning inside or outside of the classroom, it includes up-to-date ICT tools, online discussions, self-tests, quests, and learning assignments, it supports entertaining structures, and it provides information from many online resources, like blogs [31]. Additionally, the ‘Classcraft’ game fosters teacher-to-student and student-to-student interactions, as well as student-to-technology interaction. Playing this game, students participate in classroom activities, follow the pre-set appropriate behavior set by the teacher, create teams for collaboration, and use online sources for self-learning or self-evaluation [2, 31, 32]. During the COVID-19 pandemic, the Lab of New Technologies and Distance Learning of the University of Ioannina, in cooperation with other higher educational institutes, created MOOCs to support online courses [33, 34]. Students who registered for these courses had to work in their own peace and apply self-regulating learning. At the end of these online courses, students had to present their views on online learning. According to the online discussions, students mentioned the importance of gamification in the learning process [33, 34]. This study aims to show how we can redesign education focusing to training students in using online games in classroom activities for self-regulated learning and self-evaluation.

2 Materials and Methods For our hypothesis, we included the ‘Classcraft’ online game in the learning process to collect information on how students react to learning via online games in an audit process. The online game ‘Classcraft’ includes gamification, material for selfregulated learning and online exercises for self-evaluation. A self-selected group of three undergraduates and two postgraduate students from the Department of Early

430

J. Pange et al.

Childhood Education at the University of Ioannina had to use ‘Classcraft’ for a semester. Postgraduate students had to create online games under the instructions and supervision of their teacher for the subject ‘Applications of ICTs in Education,’ and undergraduate students had to learn the material of this subject using the online games. The teacher in the ‘Classcraft’ game suggested the learning course material to be include in the game, the rules for appropriate behavior, the points should be earned for cooperation, and the time for accurate completion of the tasks. In the gamification process, postgraduate students had to create a game in ‘Classcraft,’ incorporate the learning material in many forms, set self-evaluation tests, track the academic progress of participants through quests and levels, show how to earn points and badges, offer the ability to communicate with other students, and develop a selfassessment process. This way, postgraduate students took on the role of teachers and created online games in ‘Classcraft’ for learning. The undergraduate students had to use the ‘Classcraft’ game to learn the course material. These students had to take an active role in the game provided by postgraduates and then apply the self-regulated learning process. Undergraduates had to learn the course material online through the online game in their own time, communicate with their classmates, evaluate their knowledge frequently, and track their progress toward their goals. Through anonymous online questionnaires, all students reported how they improved their knowledge using this game in a self-regulated learning process. The questions were open-ended based on the issues: What is your opinion of using online serious games in education? and What is the impact of the ‘Classcraft’ game in self-assessment? Student responses were analyzed using thematic analysis methodology. Thematic analysis is a flexible research methodology, involving a distinctive set of procedures, used to analyze social phenomena, where the research aims to gain insight into how participants experience these phenomena and, through the analysis of these experiences, to gain a deeper understanding of the social phenomena themselves [35, 36]. Thematic analysis focuses on the ideas in the research data, the researcher seeks to capture these ideas as they are latent and then deliberate on them, assigning each idea to a separate theme or sub-theme [35–37]. In applying thematic analysis in this study, we did thematic analysis followed the steps suggested by Braun & Clarke [35, 38] for conducting thematic analysis. In the first step, we familiarized with the course material in detail by actively reading and re-reading, noting down initial insights and ideas. In the second step, we coded the course material sequentially according to the ideas they discerned and grouped together the recurring codes into clusters. The third step involved working with the clusters of codes, looking for links between the codes and the emerging themes, and identifying the theme-sub-theme structure. In the fourth step, we looked at all the courses to make sure that the themes and sub-themes matched the collected data. To achieve internal homogeneity and external heterogeneity of the themes, we made some adjustments to the codes and their material and regrouped the themes and subthemes in this step. In the next step, the final names of the themes and sub-themes were decided. In the last step, the topics and subtopics were described.

Redesigning Education Using Serious Games

431

3 Results and Discussion Postgraduate students created gaming maps that contained a series of quest tasks for players to complete under the supervision of the teacher. They created the first quest (see Fig. 1) for teaching the ICTs concept. This quest consisted of five stages. The first stage was the introduction, where players would learn the educational value of ICTs. In the second, third, and fourth stages, there was information on ICTs and comprehension exercises. The fifth and final stage involved creating a story of using ICTs and earning a card as a reward. The second quest (see Fig. 2) was on the primary educational applications of ICTs. Players were asked to evaluate their knowledge by doing three types of tasks: a) completing a questionnaire created in ‘Kahoot!’ (https://kahoot.com/), b) creating digital stories through online comic tools, and c) cooperating with their classmates to create a list of questions about the educational use of ICTs. When completing each task, students received experience points (XP). The thematic analysis of the responses revealed how students learnt the subject using the Classcraft game. The characteristics of this learning method are described below according to the themes and sub-themes identified.

3.1 Theme: Efficient Learning According to the study, all students’ learning with ‘Classcraft’ was efficient. The following presentation of the research findings reveals the reasoning behind this statement.

Fig. 1 Screenshot from: Classcraft 1 game with nodes of the information of ICT in education

432

J. Pange et al.

Fig. 2 Screenshot from ‘Classcraft 2,, game and the nodes with the applications of ICTs in education

3.2 Sub-theme: Good Performance in the Subject At the end of this course, the teacher responsible for the course, using the game’s interface, evaluated the students’ progress and the ICT skills they gained. All postgraduate students had a high score on setting and using the ICT tools of the game. In other words, all students’ performance in the subject was good. Sub-theme: Successful Participation in All Tasks. Based on responses from teachers and students, the undergraduate students participated successfully in all tasks of games created by postgraduate students. Sub-theme: Digital and Other Skills Development. Undergraduate students reported that they have improved their digital skills. like one student wrote “As a result of working with ‘Classcraft’, I developed digital skills”. Sub-theme: Learning ‘Internet Security.’ Digital security was part of the introduction of learning material in this game and this provided an opportunity to learn about that type of security. This has been confirmed by student feedback such as “I learned ‘internet security’ through the quests”. Sub-theme: Development of Creativity. These games develop the creativity of learners because each figure represents one student. As one student said “I also consider that this game upgrades creativity because students can create their own figures and characters for the different games.

Redesigning Education Using Serious Games

433

3.3 Theme: Engaging Learning As can be seen from the students’ answers, all of them benefit from personalized learning, team learning experiences, and change in school experiences. It showed that students were more engaged in their learning compared to traditional learning methods - more motivated to learn because they found it interesting, enjoyable, entertaining, minimally stressing and at the same time challenging. The characteristics of their engaging learning are revealed in the sub-themes. Sub-theme: Motivating Learning. Students were influenced positively by the game and were involved in learning events through stories of the game. They stressed that they were willing to do the tasks without having to force themselves to do them. Motivation to learn remained high throughout the learning process. This is evidenced by student statements such as “I was motivated by the learning process.’ Sub-theme: Interesting Learning. All students stressed that learning using the game was interesting. They also noted that the game itself, its design, tasks and applications were interesting: “I consider ‘Classcraft’ to be a very interesting game for learning”, “’Classcraft’ is an interesting game.” “I was enthusiastic about the ‘Classcraft’ game because it is based on gamification and operates like a classroom.” “Games created on the ‘Classcraft’ web platform have an excellent modern design that provokes interest in using the platform and its features for learning.” “In my opinion, all tasks and applications of this game were interesting.” Sub-theme: Entertaining Learning. Students find ‘Classcraft’ not only an interesting and motivating learning tool. Learning with this game takes on even an entertaining character. This can be seen from student responses such as I also consider that this game offers entertainment”, “The ‘Classcraft’ is a great educational game. Participation in this game is an exciting and enjoyable learning experience”, ‘Classcraft’ was a game for learning and entertainment.” Sub-theme: Demanding and Challenging Learning. A very important observation made by students was the difficulty of learning with Classcraft. Students said, for example, “I found it difficult to create new games on ‘Classcraft’, “I worked hard to earn points and levels”. These responses suggest that learning with ‘Classcraft’ was a serious, demanding and challenging activity, not only interesting and enjoying entertainment. Sub-theme: Stress Free Learning. Students’ responses such as this:“This game minimizes stress” indicate that learning with ‘Classcraft’ reduces stress. This is despite the fact that this learning is challenging and demanding.

434

J. Pange et al.

3.4 Theme: Self-regulating Learning As the data shows, all students who studied with ‘Classcraft’ experienced the specifics of self-regulated learning. Postgraduate students when creating the learning material and the games spend time in a self-learning process and said that “When I was creating this course, I had to spend time in a self-learning process.“ “When I was collecting information to be included in the course material, I was reading articles and looking for other relevant information from many online sources.“ Additionally, undergraduate students reported that they improved their self-regulation ability and their self-learning experience using the ‘Classcraft’ online game. They said, “Learning through games means having the ability to self-regulate.” “’Classcraft’ is an interesting and useful platform for self-regulated learning.” “An advantage of using ‘Classcraft’ is the self-learning part.” The characteristics of self-regulated learning that emerged from the data analysis were the organization and management of one’s own learning process and self-assessment. Sub-theme: Organizing the Learning Process. Undergraduate students studied the material in their own time, tracked their progress (according to the levels of the game), and realized what they needed to improve in their ICT skills and how to set learning goals for themselves. Students said that “Additionally, it supports selfregulated learning because students, through this game, can organize their goals and plan how to achieve them”. Sub-theme: Management of Learning. Students realized the importance of the application of rules for appropriate behavior in classroom activities and said that “I liked this game because the teacher can give extra points or delete points depending on the way students behave in the classroom. I mean that when students participate in classroom activities, they get extra points, but when they are impolite, they lose points.” Sub-theme: Self-assessing Learning. This game includes self-assessment processes, and students said that “’Classcraft’ is an interesting and useful platform for self-assessment.” “The idea of having online questions at the end of each chapter is important and supports self-assessment.” “During the game, I went through many self-assessment procedures, which promoted my learning.” “‘Classcraft’ facilitates a self-assessment process through quests and points.” “‘Classcraft’ is useful, but I encourage its use only in universities and high schools since children in kindergarten will not be able to work independently or to understand the game without the help of teachers. Moreover, I think that primary school children will play the game as a game, and they will not find that in the picture with the beautiful design, there is an assessment tool, or self-assessment”.

Redesigning Education Using Serious Games

435

3.5 Theme: Collaborative Learning As undergraduates progressed through the game, they earned experience points and points for cooperation with their classmates. All students worked alone to complete quests and challenges, and they worked as a team to set questions and play the ‘quest’ part of the game. According to the responses of the undergraduate students, teamwork during the game was beneficial and promoted a sense of community and accountability among students. Sub-theme: Benefits of Learning in Teams. Students have found that working in teams to learn with ‘Classcraft’ has been beneficial for them. In particular, it was pointed out that this work helped them to understand the content of the course better as students said that “working in a team is also beneficial,” “working with classmates made me have better understanding of the course.” Sub-theme: Accountability to Team Members. When learning in a team, students are not only responsible for their own actions, but also for the actions of the team. Through relationships with team members, they develop responsibility towards others in shared team tasks. This was also observed by some students who wrote “we had a great relationship with team members to set up rules in our games and create the list of questions.”

3.6 Theme: Teaching Experience According to the views of postgraduate students, we can redesign education by including postgraduate students in the teaching process, as they can create and design online courses using up-to-date information. When postgraduates take on the role of teachers, they also evaluate their knowledge on this topic and make themselves part of a lifelong learning group where professors, postgraduates, and undergraduates constantly interact. Overall, postgraduate students considered this game as having a valuable experience during their studies. Postgraduate students said that “The ‘Classcraft’ is a great alternative to traditional games for teaching,” “the game is unique because the learning experience is not static,” “undergraduates, during the teaching process, realized to be aware of their choices and their replies all time because points were taken away,” “I realized how technology can support teaching and continuous self-assessment,” and “I developed a way about popularizing games for learning and teaching in my future professional practice.“ Overall, the use of ‘Classcraft’ game created significant effects in the learning environment and especially in developing the interaction between teachers and students. Additionally, through the different stages of the game motivation and self-regulated learning was developed through enjoyable activities.

436

J. Pange et al.

4 Conclusions In game-based teaching processes, students are engaged in joyful and challenging learning experiences, associated with collaboration and motivation. Education is provided in formal, non-formal, and informal forms and is addressed to everyone. Serious games encourage players to cooperate and learn in formal or non-formal settings. According to the findings of this study, serious games were well adapted by undergraduate and postgraduate students and have the potential to redesign education because they offer engaging and collaborative learning through interactivity and enjoyment. All students were highly motivated by the games in this course, and they were able to successfully complete all course assignments in due time. The interactive material in ‘Classcraft’ and the quizzes enhanced learning and offered an innovative way for self-assessment. Online games provide cost-effective education, and all educators can set up their own games accordingly. Additionally, online games can become agents to democratize education, making it accessible to all citizens without any consideration of learning difficulties, and approachable via open online courses to learners who might otherwise be excluded from traditional education. Moreover, online games help students and teachers set their own learning goals, develop soft skills, continuously evaluate their knowledge, and get simulated learning experiences, as other researchers have similarly reported [39]. The findings of this study are in line with the results of other studies [31, 32, 35] considering the importance of gamification for learning, and the application of an interactive, engaging, personalized learning approach. In conclusion, it becomes evident from this audit pilot study that redesigning education is a continuous process where students and teachers play an important role. For this reason, students must be well prepared for personalized learning, and teachers must consider the integration of technology in the classroom, like serious games, continuous training, continuous assessment, and lifelong learning. Although in this audit study we had a very small group of students and many factors were not fully examined, we got a clear indication from their responses of how online serious games can be applied for learning. This study will continue with a larger group of students and include more factors. Acknowledgements This research is co-financed by Greece and the European Union (European Social Fund- ESF) through the Operational Programme “Human Resources Development, Education and Lifelong Learning 2014-2020” in the context of the project “ICT in Education: Applications in Natural, Social and Health Sciences” (MIS 5162213).

References 1. Buckley, J., Colosimo, L., Kantar, R., McCall, M., Snow, E.: Game-based assessment for education. In: OECD Digital Education Outlook 2021: Pushing the Frontiers with Artificial

Redesigning Education Using Serious Games

2. 3. 4.

5.

6.

7. 8.

9. 10.

11.

12. 13. 14. 15. 16.

17.

18. 19. 20. 21.

22. 23.

437

Intelligence, Blockchain and Robots (2021). https://www.oecd-ilibrary.org/sites/9289cbfd-en/ index.html?itemId=/content/component/9289cbfd-en. Accessed 28 Feb 2023 Gerencer, T.: 5 Best Online Assessment Tools for Teachers. HP (2020). https://www.hp.com/ us-en/shop/tech-takes/best-online-assessment-tools-for-teachers. Accessed 28 Feb 2023 Khampirat, B.: Relationships between ICT competencies related to work, self-esteem, and self-regulated learning with engineering competencies. PLoS ONE 16(12), 1e0260659 (2021) Girdzijauskien˙e, R., Norvilien˙e, A., Šmitien˙e, G., Rupšien˙e, L.: Strengthening student engagement in learning through use of digital tools. Acta Paedagog. Vilnensia 49, 115–130 (2022). https://doi.org/10.15388/ActPaed.2022.49.8 Chu, S.-T., Hwang, G.-J., Chien, S.-Y., Chang, S.-C.: Incorporating teacher intelligence into digital games: An expert system-guided self-regulated learning approach to promoting EFL students’ performance in digital gaming contexts. Br. J. Educ. Technol. 54(2), 534–553 (2022) Hooshyar, D., Ahmad, R.B., Yousefi, M., Fathi, M., Horng, S., Lim, H.: Applying an online game-based formative assessment in a flowchart-based intelligent tutoring system for improving problem-solving skills. Comput. Educ. 94, 18–36 (2016) Pange, J.: Self-regulated learning strategies in groups of learners. Bridges/Tiltai 66(1), 169–181 (2014) Lekka, A., Sakellariou, M.: Computer games and ethical issues. In: 2014 International Conference on Interactive Mobile Communication Technologies and Learning (IMCL 2014), Thessaloniki, Greece, pp. 342–343 (2014). https://doi.org/10.1109/IMCTL.2014.7011160. Kapp, K.M.: The Gamification of Learning And Instruction: Game-Based Methods and Strategies for Training and Education. Pfeiffer (2012) Pange, J., Lekka, A., Katsigianni, S.: Serious games and motivation. In: Auer, M.E., Tsiatsos, T. (eds.) Interactive Mobile Communication Technologies and Learning, pp. 240–246. Springer International Publishing, Cham (2018). https://doi.org/10.1007/978-3-319-75175-7_25 Pagano, A., Marengo, A.: Training time optimization through adaptive learning strategy. In: 2021 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT), pp. 563–567 (2021). https://doi.org/10.1109/3ICT53449.2021.9582096 Lekka, A: The pedagogical exploitation of digital games in formal and informal education. Doctoral thesis. University of Ioannina, Greece (2017) Brad Meyer Courier staff: WiiFit: The digital workout (2008). https://www.chron.com/neighb orhood/article/Wii-Fit-The-digital-workout-9374746.php. Accessed 19 Apr 2023 Kitsantas, A.: Fostering college students’ self-regulated learning with learning technologies. Hellenic J. Psychol. 10, 235–252 (2013) Yan, Z., Brown, G.T.L.: A cyclical self-assessment process: towards a model of how students engage in self-assessment. Assess. Eval. Higher Educ. 42(8), 1247–1262 (2017) Zimmerman, B.J., Kitsantas, A.: The hidden dimension of personal competence: self-regulated learning and practice. In: Elliot, A.J., Dweck, C.S. (eds.) Handbook of Competence and Motivation, pp. 509–526. Guilford Publications (2005) Yan, Z.: Self-assessment in the process of self-regulated learning and its relationship with academic achievement. Assess. Eval. Higher Educ. 45(2), 224–238 (2020). https://doi.org/10. 1080/02602938.2019.1629390 Papanthymou, A., Darra, M.: The contribution of learner self-assessment for improvement of learning and teaching process: a review. J. Educ. Learn. 8(1), 48–64 (2019) Siegesmund, A.: Using self-assessment to develop metacognition and self-regulated learners. FEMS Microbiol. Lett. 364(11), fnx096 (2017). https://doi.org/10.1093/femsle/fnx096 Panadero, E., Jonsson, A., Botella, J.: Effects of self-assessment on self-regulated learning and self-efficacy: four meta-analyses. Educ. Res. Rev. 22, 74–98 (2017) Panadero, E., Alonso-Tapia, J.: Self-assessment: theoretical and practical connotations. When it Happens, how is it acquired and what to do to develop it in our students. Electron. J. Res. Educ. Psychol. 11(2), 551–576 (2013). https://doi.org/10.14204/ejrep.30.12200 Kahoot! https://kahoot.com/. Accessed 07 Mar 2023 Quizizz. https://quizizz.com/. Accessed 07 Mar 2023

438

J. Pange et al.

24. Dinsmore, D.L., Alexander, P.A., Loughlin, S.M.: Focusing the conceptual lens on metacognition, self-regulation, and self-regulated learning. Educ. Psychol. Rev. 20, 391–409 (2008). https://doi.org/10.1007/s10648-008-9083-6 25. Zhang, Q., Yu, L., Yu, Z.: A content analysis and meta-analysis on the effects of classcraft on gamification learning experiences in terms of learning achievement and motivation. Educ. Res. Int. 2021, 1–21 (2021). https://doi.org/10.1155/2021/9429112 26. Rivera-Trigueros, I., Sánchez-Pérez, M.: Conquering the iron throne: using ‘classcraft’ to foster students’ motivation in the EFL classroom. Teach. English Technol. 4, 3–22 (2020) 27. Sipone, S., Abella-García, V., Rojo, M., dell’Olio, L.: Using ‘classcraft’ to improve primary school students’ knowledge and interest in sustainable mobility. Sustainability 13, 9939 (2021) 28. Ismail, M.A.A., Ahmad, A., Mohammad, J.A.M., Fakri, N.M.R.M., Nor, M.Z.M., Pa, M.N.M.: Using Kahoot! as a formative assessment tool in medical education: a phenomenological study. BMC Med. Educ. 19, 230 (2019). https://doi.org/10.1186/s12909-019-1658-z 29. Darmawan, M.S., Daeni, F., Listiaji, P.: The Use of Quizizz as An Online Assessment Application for Science Learning in The Pandemic Era. Unnes Sci. Educ. J. 9(3), 144–150 (2020). https://doi.org/10.15294/USEJ.V9I3.41541 30. Classcraft official website. https://www.classcraft.com/. Accessed 28 Feb 2023 31. Sanchez, E., Young, S., Jouneau-Sion, C.: Classcraft: from gamification to ludicization of classroom management. Educ. Inf. Technol. 22, 497–513 (2017). https://doi.org/10.1007/s10 639-016-9489-6 32. Trigueros, I.R., MSanchez, M.: Classcraft as a resource to implement gamification in Englishmedium instruction. In: Sánchez, M. (ed.) Teacher Training for English-Medium Instruction in Higher Education, pp. 356–371. IGI Global (2020). https://doi.org/10.4018/978-1-7998-23186.ch017 33. Multiple Higher Educational Institutions Masters in Entrepreneurship – MHEI-ME rasmus+ Project. MHEI-ME. AdvenioeAcademy (2020) 34. Madleˇnák, R., D’Alessandro, S.P., Marengo, A., Pange, J., Neszmélyi, G.I.: Building on strategic eLearning initiatives of hybrid graduate education a case study approach: MHEI-ME Erasmus+ project. Sustainability 13, 7675 (2021). https://doi.org/10.3390/su13147675 35. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3, 77–101 (2006) 36. Guest, G., MacQueen, K.M., Namey, E.E.: Applied Thematic Analysis. Sage, Thousand oaks (2012) 37. Clarke, V., Braun, V.: Teaching thematic analysis: Overcoming challenges and developing strategies for effective learning. The Psychologist 26(2), 120–123 (2013) 38. Braun, V., Clarke, V.: Thematic analysis. In: Cooper, H., Camic, P.M., Long, D.L., Panter, A.T., Rindskopf, D., Sher, K.J. (Eds.) APA Handbook of Research Methods in Psychology, vol. 2, Research Designs: Quantitative, Qualitative, Neuropsychological, and Biological, pp.57–71. 1st Edn. American Psychological Association 39. Sánchez-Pérez, M.M., Galera Masegosa, A.: Gamification as a teaching resource for Englishmedium instruction and multilingual education at university. In: Andújar, A. (Ed.), Recent Tools for Computer- and Mobile-Assisted Foreign Language Learning, pp. 248–267). IGI-GLOBAL (2020). https://doi.org/10.4018/978-1-7998-1097-1.ch012

Gamifying Diabetes — An Education Game Teaching People with Diabetes About Physical Activity Maria Anna Rauchensteiner , Tim Colsman , Albina Fatykhova , Nilüfer Deniz Faizan, Matthias Christoph Utesch, Holger Wittges, and Helmut Krcmar

Abstract It has been found that sports is beneficial for individuals with type 1 diabetes. The risk for a severe hypoglycemia is increased by sports and needs to be managed carefully. Since health apps have a positive influence on the intended health behavior and clinical health outcomes, we decided to develop a health app for people with type 1 diabetes. We analyzed current apps for people with type 1 diabetes and found that there was no app focused on sports with type 1 diabetes. Thus, we decided to develop an app to teach people how to safely engage in sports with type 1 diabetes. For this, we worked together with the startup GLAICE which plans to bring a health app to the market that helps people with diabetes with their day to day lives and the management of their glucose levels. The startup’s focus topic is sports with diabetes. We developed a prototype of the app using the Design Thinking Method. Our solution is focused on the correct sports preparation and doing sports correctly with type 1 diabetes. We do this by using a game with multiple scenarios. Depending on how the user answers a series of questions, the player character can M. A. Rauchensteiner (B) · T. Colsman (B) · A. Fatykhova (B) · N. D. Faizan (B) · M. C. Utesch (B) · H. Wittges (B) · H. Krcmar (B) Technical University of Munich, 80333 Munich, Germany e-mail: [email protected] T. Colsman e-mail: [email protected] A. Fatykhova e-mail: [email protected] N. D. Faizan e-mail: [email protected] M. C. Utesch e-mail: [email protected] H. Wittges e-mail: [email protected] H. Krcmar e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_35

439

440

M. A. Rauchensteiner et al.

either have a successful sports day or can have a hypoglycemia during or after sports. Our targeted audience are people with type 1 diabetes between 18 and 35 years. We evaluated the prototype on people with diabetes, and 58 students visiting 12th grade tested the prototype for usability. For this, we used the System Usability Survey to receive the students’ feedback. The students graded the prototype’s usability with a B+. They also gave us further feedback by answering open questions. Our results show that this game is a successful first approach for teaching people with type 1 diabetes about sports with type 1 diabetes via app. Keywords Gamification · Type 1 diabetes · Education · Learning game · Exercise with Type 1 diabetes · Sports · Physical activity

1 Introduction Type 1 diabetes needs to be managed carefully to avoid diabetes-related complications [1]. Management of type 1 diabetes requires constant attention to many aspects such as blood glucose monitoring, meal planning and insulin administration [1]. Still, regular physical activity is an important part of type 1 diabetes management [2], yet many people with diabetes feel that they have inadequate knowledge around exercise management and fear a hypoglycemia caused by exercise [3]. Gamification can help improve learning outcomes [4]. Previous research has shown that health apps using gamification elements can help people with diabetes in improving their overall health [5]. People with diabetes need to receive education on sports with diabetes. Using a health app with gamification elements to teach about sports with diabetes seems sensible. We found that education on sports with type 1 diabetes and a health app using gamification elements has not been combined so far. Thus, the goal of our project is to close this research gap. To close this gap, we want to answer the following three research questions. 1. Which target groups and topics are addressed by often used diabetes apps currently on the market? 2. Which methods and solutions can we use to develop a learning app that motivates the user to engage continuously with the app and the topic sports with type 1 diabetes? 3. How can we develop, test and improve the prototype of the learning app containing the specific scenarios teaching about sports with type 1 diabetes? To answer our research questions, we first conduct a feature analysis of currently popular diabetes apps. Then we select which features we want to implement in our application and specify our design and explain the features and gamification elements we implement. Afterward, we present our app prototype and show the test results of our prototype and what steps we took to improve the prototype. We used the Design Thinking Method to develop our prototype over multiple iterations and tested the

Gamifying Diabetes — An Education Game Teaching People …

441

prototype at the end of each iteration. We tested the prototype on our target group and tested for usability with a larger group to achieve statistically significant results. We conducted the testing by using both the Thinking Aloud Method and the System Usability Scale. To test the prototype for usability with a larger user group, we use a questionnaire and hold an open discussion about merits and potential improvements of the prototype with the group.

2 Literature 2.1 Type 1 Diabetes and Physical Activity Type 1 diabetes accounts for 5%–10% of all cases of diabetes worldwide and its incidence continues to increase [1]. Type 1 diabetes has a strong genetic component, but the factors triggering the onset of the clinical disease are largely unknown [1]. The symptomatic onset of the disease usually occurs during childhood or adolescence [6]. Type 1 diabetes needs to be managed carefully to avoid diabetes related complications [1]. These complications are microvascular and macrovascular diseases, which account for most of the morbidity and mortality associated with type 1 diabetes [1]. Management of type 1 diabetes requires constant attention to many aspects such as blood glucose monitoring, meal planning and insulin administration [1]. People with type 1 diabetes need life-long insulin injections [6]. With better glycemic control, microvascular and macrovascular complications are less likely [6]. Thus, continuous engagement in type 1 diabetes self-management is critical to minimize the risk of diabetes related complications [7]. Regular physical activity is an important part of type 1 diabetes management [2]; it can even help individuals to achieve their glycemic goals [3]. But people with type 1 diabetes are on average at least as inactive as the overall population [3]. For a person with diabetes several more barriers to exercise exist [3]. Those barriers include inadequate knowledge around exercise management, fear of hypoglycemia and loss of glycemic control [3]. The risk of experiencing severe hypoglycemia, which is a potentially live threatening condition, can increase during exercise, as well as for up to 31 h after the exercise was conducted [2]. Also, different types of exercise have different effects on glucose levels and require different management strategies to ensure the blood sugar remains within safe boundaries [2]. For safe and effective exercise, people with type 1 diabetes need to have glucose targets for their chosen type of exercise and they need to adjust their nutrition and insulin doses to avoid exercise-related glucose highs or lows outside of the normal physiological range [2, 3].

442

M. A. Rauchensteiner et al.

2.2 Gamification Lately gamification — i.e., “using game attributes in a non-gaming context” [8] — is becoming increasingly relevant in education [8]. When talking about gamification, it is often defined by using the term “game.” Çeker and Özdamli define the term game as “an entertaining activity with certain rules that need be followed” [9]. While games have the aim to entertain the users [9], gamification tries to “change the attitude and behavior of the user” [9]. Gamification and Game Based Learning. Through game based learning, the users/ learners reach their learning goal by playing games [8]. In contrast to gamification “playing” has the major role during the learning process while gamification “helps to make learning a more participating activity” [9]. Another difference between “gamification” and “game based learning” is that they approach the learning process in a different way. While “gamification” changes a non-game oriented environment into a game environment, through “game based learning,” the users can learn any subject by using games [9].

3 Methodology 3.1 Description of Method Our primary goal is to develop and evaluate a prototype for an application that teaches people with type 1 diabetes how to engage in sports in a safe manner. Additionally, we want to ensure that our app has a modern design and at least one unique feature. Finally, our solution should motivate the user to engage continuously with the app and the topic sports with type 1 diabetes. We decided to cooperate with our industry partner GLAICE, whose focus topic is sports with diabetes [10]. We developed a prototype for an education app which helps people with diabetes to understand what they need to consider when exercising. Since this education app gives health recommendations that need to be correct, we requested the entire medical input on sports with diabetes from our industry partner who checked the prototype’s educational content for correctness periodically and before we started the first user tests. We selected the format in which to present this content, the design and the features of the prototype. To ensure that our app has at least one unique feature or a feature implemented in a new fashion, we conducted a target group analysis and a feature analysis on 23 solutions for people with diabetes. After talking to GLAICE about what they expected of the app design and looking at their previous design concept, we decided on the design of our prototype.

Gamifying Diabetes — An Education Game Teaching People …

443

What is notable in our implementation is that we used a modular build. By creating different characters for different types of sports and different chapters for a di-verse mix of situations in everyday life our application is indefinitely expandable. User Tests are conducted with two methods simultaneously. The Thinking Aloud Method is used to capture unfiltered first thoughts [11] the testers have about the prototype. We use the System Usability Scale to measure how easy it is to use our prototype. We do this, because the System Usability Scale is a versatile and highly robust tool for usability testing [13].

3.2 Design Thinking To find a way to integrate large amounts of scientific information into as simple and motivating materials as possible, we applied the Design Thinking Method. Design Thinking is a non-linear process with the purpose of understanding the user’s needs and to redefine problems to identify alternative strategies and solutions that may not be immediately obvious to our initial understanding. At the same time, design thinking offers a solution-oriented approach to problem solving [14].

3.3 Concurrent Thinking Aloud For our User Test, we used the Concurrent Thinking Aloud Method to see how a user would use our application, when opening it for the first time with no additional information. The Concurrent Thinking Aloud Method is one of the most widely used methods in usability testing [15]. The Concurrent Thinking Aloud is a usability test where the users express their thoughts loudly while using the test subject — in our case, the application [12]. By using this method, we can better understand which part of the app/screen the users are working on and what their thoughts are.

3.4 System Usability Scale We use the System Usability Scale to measure how people with diabetes and school students perceive the usability of our prototype. The System Usability Scale is used, since it offers a quick and easy way to measure perceived usability [13, 16]. It is also the most widely used standardized questionnaire for assessing perceived usability [16]. The System Usability Scale is a versatile and highly robust tool for usability testing [13]. The System Usability Scale consists of ten questions [17]. Those questions can be answered on a five point scale ranging from “strongly agree” to “strongly disagree” [17].

444

M. A. Rauchensteiner et al.

A drawback of the System Usability Scale is that it only gives a general overview about how usable the system is but does not answer which element of the system caused the user to struggle [17]. By adding open ended questions to the System Usability Scale, the elements causing the user to struggle might be identified [17]. To correctly calculate the scores of the System Usability Scale, odd questions are scored 0–4 based on the 1–5 scale, even question scored 4–0 on the 1–5 scale [17, 18]. The score over all ten questions is then multiplied by 2.5 to generate a System Usability Scale score between 0–100 points [17, 18]. The average result of the System Usability Scale is sixty-eight [19]; thus, previous researchers considered a score higher than sixty-eight as good [20]. Currently it is a common goal in industry to achieve a System Usability score of 80 or higher [21]. Since Finstad found that many non-native English speakers do not know the meaning of the word “cumbersome” [22], we use an adjusted version of the original System Usability Scale, where, for item 8, the word cumbersome from the original scale has been replaced by awkward [21]. Thus, we use the System Usability Scale depicted in the paper of Lewis and Sauro [21] because it is better suited for usability testing with non-native English speakers.

3.5 Sample and Data Collection for Prototype Testing We assess our prototype on three different groups which are focused on different aspects of the prototype. GLAICE. We conducted tests with GLAICE, where the focus of the testing was on the correctness of the content of the prototype and on further improvement of the prototype. Further improvement of the prototype could be adding a new functionality or reducing complexity of use. Due to the broad field of potential improvements, we conducted the tests via the Thinking Aloud Method as well as by receiving feedback via WhatsApp or E-Mail to allow them a longer testing period. People with Diabetes. We also evaluated an early version our prototype on two people with diabetes. The two people with diabetes were both female; while one has only received her diagnosis recently, the other has lived with her diagnosis for a very long time. We received the contact data of those people with diabetes from our industry partner. We contacted them and both agreed to assess our app during a Zoom meeting. With this test we wanted to receive first feedback from the targeted user group about what they like and dislike about our prototype. Since we wanted to receive as much feedback as possible, we used the Thinking Aloud Method while the testers clicked trough the prototype for the first time. Immediately after they evaluated the prototype, we asked both testers the same open-ended questions and let them fill out the System Usability Scale. School Lesson. We assessed the usability of the final version of our prototype with three school classes. The pupils are students at the FOSBOS Weilheim and are visiting

Gamifying Diabetes — An Education Game Teaching People …

445

12th grade. We had 90 min per class to present our prototype, let the students test the prototype and let them evaluate the prototype. The students evaluated the prototype by filling out a short questionnaire containing questions on their demographics, the System Usability Scale and few open-ended questions (see Appendix 3). Afterward, we held a discussion round with the students to allow for further feedback. Overall, 58 students tested our prototype.

4 Results 4.1 Analysis of Current Solutions for Diabetes Management To assess, what target groups and topics current diabetes apps address, we looked at 23 popular diabetes apps. Since we worked together with GLAICE, we wanted to find the best solution for them. Thus, we asked the startup to give us a list consisting of their main competitors. Shortly thereafter, we received a list consisting of 24 companies and their corresponding apps. This list defined the targeted market and the main competing apps themselves and delivered us the input needed to analyze the features of the most relevant competing apps in 2022. Since one of the 24 apps on the list is inactive by now, we reviewed the remaining 23 apps. Target Group. We found that the 23 apps we analyzed target not only people with type 1 diabetes, but also people with type 2 diabetes as well as people aiming to improve their own performance and health by keeping their blood sugar stable. The age of the targeted audience is mixed as well. Still, most apps seem to be targeting adults. Three notable exceptions are the apps eddii, SugarCoach and Xugarhero, which seem to target children with type 1 diabetes. We found that the apps seem to be targeted at men and women in equal measure. Offered Services. We analyzed which services those 23 apps offered to their customers. By conducting this analysis, we can see what the competitors are doing. Also, this analysis helps us to figure out which unique features we should implement in the app to give our app a unique selling point, since this feature or combination of features have not been implemented in a competing app according to our analysis. There are some features that are offered only by few apps. A “User Community” is offered by three apps. “Education on sports with diabetes” is offered by only two apps and no app offers a “Knowledge database for sports with diabetes”.

446

M. A. Rauchensteiner et al.

4.2 Selection of Features and Target Group for Our Application Selection of Features. We decided to develop a prototype for an app that included the following three features User Community, Education on sports with diabetes and Knowledge database on sports with diabetes. We decided to work primarily on the feature Education on sports with diabetes, because the two analyzed solutions offering the feature have big drawbacks. The DiabiLive App implements the feature Education on sports with diabetes, by sharing articles on the topics of diabetes, diet and bodily activity [23]. The website Glycomate also offers an article on sports with diabetes [24]. But this article is mostly about why sports are important for people with diabetes and does not explain in detail how to do sports in a save way as a person with diabetes [25]. Due to this lack of easily accessible information on how to do sports with diabetes in a safe manner, we decided to develop an app with a learning game using realistic scenarios on which users can test and broaden their knowledge on sports with diabetes. Selected Target Group. We decided to target male and female individuals with type 1 diabetes between 18 and 35 years of age. We did this because the symptomatic onset of type 1 diabetes occurs usually during childhood or adolescence [6]. Therefore, we want to educate people with type 1 diabetes on sports as early as possible. But we decided to educate them only after they reach legal age at 18, because this reduces legal issues.

4.3 Design In order to fit the high level of the market and to make our prototype visually and psychologically pleasing and even outstanding in comparison to the competitors, we thought about our design concept especially carefully. We received design references from our partner and decided to give our prototype a similar look. In coordination with the startup, we defined our concept as a modern dark minimalist design with interactive animations and an intuitive interface, using calm blue tones as a color palette. The minimalistic design was inspired by the current trend for minimalism [26]. Light blue tones were chosen as the most pleasing and inspiring color tones according to the color psychology theory [27]. Interactive animations improve user interaction with the application [28]. Interactive Graphical Tools. An important contribution of user interface development has been the creation of what has come to be called interface builders. These tools allow interactive components to be placed using a mouse to create windows and dialog boxes. We implemented this method with help of animated buttons. For example, in our game, you can only read health advice by clicking on them. We

Gamifying Diabetes — An Education Game Teaching People …

447

wanted to increase the interest of users to read health tips, which are an important part of educational process in our app. So we used these functional animations because they improve user interaction with the application [28]. Themed Images as a Full Screen Background. We are using themed images as a full screen background to help users get deeper into the game as you can see in Fig. 2. This helps to achieve better immersion, interaction and also a better imitation of the daily routine.

4.4 Features We worked a lot on the functionality of our application and thought about what features we could implement in order to simplify the way of assimilation and memorization of information about preparing for sports activities for our users. After analyzing current solutions for people with type 1 diabetes, we choose to implement the following three features for the app prototype: Education on Sports with Diabetes. The idea is to prepare a person for sports sessions, so that he or she gets diabetes specified advice at the end of the “game day” with the results and a list of the tips that had better be improved. These tips are considered as brief summary which is an efficient way to memorize the information [29]. Such a feature spares the user of reading a large text and provides more clarity in the process. All that turns the difficult process of implementing the new habits into a game. All information is taken from scientific sources and verified by experts from GLAICE, as well as evaluated and improved with the help of user tests from two users who have type 1 diabetes. The advice is generated in the following way: if the user made the wrong decision during the game, at the end he received a detailed explanation and advice on how he can improve. Although the advice has a scientific basis, they are written in easy and everyday language for better assimilation of information. Here is an example of a tip from our app Took Insulin in the morning: “TIP: Tim, it could be beneficial to think about the effect of the long acting insulin in accordance with the exercise that is planned during the day. So if you want to do sports later on, please think about adjusting your long-acting insulin accordingly.” Knowledge Database for Sports with Diabetes. Moreover, all the tips mentioned above are collected in a database. When the user plays one of the scenarios, all the tips he or she received based on his decisions during the game will be stored in a data base. The user has constant quick access to this database from the menu. Thus, the user does not have to play the scenario again to access to the tips. This feature also gives the user an opportunity to constantly check and improve his knowledge of diabetic specialized sports training preparation.

448

M. A. Rauchensteiner et al.

User Community. Another interesting feature in our application is the community. Here people can exchange their experiences and become trusted advisors for each other. This feature was implemented because the opportunity to communicate and compete amongst each other has a positive influence on the users educational process [30, 31].

4.5 Gamification Elements During the development of our application, we used different gamification elements. Three important ones are: Leveling. We established the leveling through using different characters and chapters. Each character represents a different knowledge level, they are dealing an increasing time with their illness type 1 diabetes and have a different experience level with sports. The chapters represent the different variations a day can have like a usual workday, a weekend day or even a weekend trip. Additionally, the leveling is implemented in a way that in the future chapters and levels can be added without having to worry about changing the basic application at all. Through this element, we ensure that users who just got their diagnosis as well as more experienced users can use our app and learn something. Stars and Ranking. Within our application, the users collect stars depending on which decisions they make and therefore receive instant feedback if their decision was correct. At the end of each scenario, the system displays to the user the number of received stars. For our ranking all the collected stars are added up in their individual scores. Through the ranking, the users can compare their learning progress with their friends and motivate each other to accumulate more knowledge about type 1 diabetes and sports and therefore live a healthier life.

4.6 Application Presentation Start Screens. After the users created their account, they can start into the game. First, they choose the character with whom they want to play. After the users have chosen the character, the system shows an introduction slide for the character, so the user knows all the valuable information about the character. Through the implemented chapters the application aims on educating users about distinct aspects of their days and how sports can have an impact (Fig. 1). Different Screen Types. During the development of our application, we used four different screen types.

Gamifying Diabetes — An Education Game Teaching People …

449

Fig. 1 Character and chapters

Fig. 2 Transition screen

The most important type in the application is the Decision Screen. With these screens the users are able to make choices on how they want to proceed and influence the outcome of the game as you can see in Fig. 2. The next screen is used to guide the users through the day. During the day, there are events where the users do not need to decide something but have an impact on decisions later in the game. Therefore, we have designed and implemented transitions screens (Fig. 3). Those screens just have an explanation about the event happening and an action button (the only action the user can take). Additionally, throughout every screen, we added a menu button through which the user can escape the game at all times and get to the menu. This way, if the users for example accidentally start the game but just wanted to get to the menu, they can escape the game through the menu button and get to the menu without finishing the game first. Through the progress bar at the bottom of the screens, the system shows

450

M. A. Rauchensteiner et al.

Fig. 3 Decision screen

the users their progress towards the end of the game. Such an indicator helps users to navigate in the game and also motivates them to finish the game to the end and sub-sequently gain new experience and knowledge [32]. In the end of every played story line, the system shows the users through the final screen how well they did by displaying the received stars. After the user has successfully completed a chapter, the next chapter is unlocked. After the final screen, the feedback screen opens. The feedback screen shows the user in which situations the user made the wrong decision and which decision was correct. From the feedback screen the user can get to the menu or back home to try the chapter again. If the user has completed the previous chapter, there is also an option to get directly to the next chapter. Menu. To have an easy access to our features around the game like the database, the ranking, or the chat/forum we designed a menu. Therefore, we implemented a link to the menu into each screen so it would be easily accessible.

4.7 User Tests GLAICE. Throughout our development phase of the application, we had continuous contact with the startup. Right before we entered the user testing phase with the people with diabetes, we had a first test with their team to primarily confirm that our content regarding type 1 diabetes was correct. Our second test with their team took place right before finishing our prototype. The feedback was overall a particularly good one with a few improvement tips. They were positive regarding the design and the

Gamifying Diabetes — An Education Game Teaching People …

451

concept of the prototype but had some suggestions for further improvement, which we implemented. People with Diabetes. Between our tests with the startup, we had the opportunity to talk to two people affected by type 1 diabetes. Our industry partner provided us with the contact information. Before we started the user tests, we designed a questionnaire with a few questions we had regarding the basic functions and features of our application. In addition to our questions, we conducted a System Usability Scale to analyze the functionality of the app. At the start of each user test the users had the opportunity to go through the application while applying the concurrent Thinking Aloud Method. Since the experience levels of our two test users were very different, their feedback variated quite a bit. The newly diagnosed diabetic identified more with our scenarios than the second because our implementation focuses more on an inexperienced diabetic. Another important improvement proposal we received was that a few of our screen transitions were not intuitive because the users had to double click on a button to get to the next screen. We fixed this aspect by changing the second “click” to a 1-s delay, thus enabling the users to switch screens by just clicking the button once. At the end of each user test, we went through the System Usability Scale. The average score was 72.5. As we only had two users test our variance is very high. School Lesson. On October 22, 2022, we had the opportunity to test our prototype’s usability with three 12th grade classes at FOS/BOS Weilheim. At the beginning, we shortly introduced our project. After our presentation, each student had the opportunity to test the prototype and then answer a questionnaire regarding the prototype’s usability. The questionnaire consisted of demographic questions, followed by the System Usability Scale and four open questions that could be answered via a text field. After all the students answered the questionnaire, we proceeded with a discussion round, during which the students could ask us questions and offer feedback and improvement proposals. Our total number of participants for the System Usability Scale was 58. Three participants did not answer all the questions in the System Usability Scale. Therefore, we had to remove their answers to receive a fully answered System Usability Scale. For the final data analysis, we reached 55 participants. The average score as well as all the three single scores from each group exceeded the score of 68 we reached during our first user test. The average score for the evaluation at FOS/BOS Weilheim is 78.36. According to Vlachogianni and Tselios, all of those 4 scores are seen as good scores [20]. The overall score of 78.36 corresponds with a B+ [21]. In fact, the first group gave us an A– [21]. Our worst score was 3.25 in the category Frequently Use, which the students explained was due to only one implemented scenario. Our best score was 4.82 in the category Not Need Support. This shows how intuitive our app is to use. During the feedback round the students produced a few very good improvements. As general feedback the students suggested to add multiple language options, night and day modus, and a possibility to connect the app with already installed health apps.

452

M. A. Rauchensteiner et al.

Regarding the design, the students mentioned that instant text feedback after each decision would improve the understanding. To improve our final feedback screen, the students suggested changing the buttons into continuous text. The feedback for our scenario implementation was very good. Nevertheless, the students suggested more customization options for example different jobs.

5 Discussion and Conclusion Our first goal was to find out what topics are and are not addressed by often used diabetes apps currently on the market. We found that the topic education on sports with diabetes is not covered well. We decided to solve this problem by developing an app prototype for a learning app and looked at suitable methods and solutions. We found that a solution using gamification elements is well suited for us. Thus, we decided to develop a learning app, where the player can decide how the diabetic player character acts while preparing for his sports day. Depending on the decisions the player makes for the player character the sports day ends with different scenarios. If the player makes good decisions in the game, the player character can have a successful sports day, ending with a relaxed evening at home. If the decisions made by the player were bad ones, the player character might end up with a severe hypoglycemia. At the end of each scenario, an explanation on what the player character could have done better is given. During the development of this prototype, we worked iteratively and consulted with GLAICE at the end of each iteration. We tested the prototype on GLAICE, people with diabetes and a group of students. For the testing, we used the Thinking Aloud Method, as well as the System Usability Scale. After the testing we identified the topics which we want to improve during the next iteration. The results of the tests with people with diabetes were fine; due to the early stage our prototype was in at the time, we were able to identify some issues and solve them before the next round of testing. Despite those issues, the average score on the System Usability Scale was 72.5 points, which is above the score of 68 points that have been considered good by previous researchers [20]. The average result of the usability tests over all three school classes is 78.36 points. According to the Curved Grading Scale for the System Usability Scale [21], this results in the school grade B+ . This scale grades the Usability with an A– when 78.9 points are reached [21]. Thus, our average usability is a very good B+ , which is achieved only by the best 20% of all systems [21]. A B+ corresponds to a percentile range between 80 and 84 [21], which shows that we developed an app prototype with a high usability. Limitations. One of our main limitations is the lack of time and resources to assess whether our game is effective in teaching people with type 1 diabetes about sports with diabetes. To check whether our game improves a person with diabetes’ selfcare regarding sports a study with a test and a control group of people with diabetes would

Gamifying Diabetes — An Education Game Teaching People …

453

be necessary. This could be tested by comparing the average hypoglycemia rates and their development over time for the two groups. Another limitation is that our prototype currently only educates the player about how to engage in running safely. Since for different types of sports a type 1 diabetic must prepare differently [2] explaining multiple types of sports would be important to motivate people with type 1 diabetes to engage with our app continuously. Outlook. Due to the modular design of our prototype, it will be easy to extend an app following the same modular structure indefinitely. Possible areas for further improvement include adding further scenarios for different types of sports and adding content to the different situations in everyday live, such as a workday, or a weekend trip, as well as giving diabetologists the opportunity to screen the content of the app and publicly declare their approval for the scenarios. Furthermore, the students testing the app prototype recommended that the app should be available in multiple languages. Acknowledgements We want to thank our supervisors from GLAICE Magomed Abdulaev, Leonard Rinser and Michelle Torres-Linke who gave us great feedback on our prototype.

References 1. Daneman, D.: Type 1 diabetes. The Lancet 367(9513), 847–858 (2006). https://doi.org/10. 1016/S0140-6736(06)68341-4 2. Guelfi, K.J., Jones, T.W., Fournier, P.A.: New insights into managing the risk of hypoglycaemia associated with intermittent high-intensity exercise in individuals with type 1 diabetes mellitus: implications for existing guidelines. Sports Med. 37(11), 937–946 (2007). https://doi.org/10. 2165/00007256-200737110-00002 3. Riddell, M.C., et al.: Exercise management in type 1 diabetes: a consensus statement. Lancet Diabetes Endocrinol. 5(5), 377–390 (2017). https://doi.org/10.1016/S2213-8587(17)30014-1 4. van Gaalen, A.E.J., Brouwer, J., Schönrock-Adema, J., Bouwkamp-Timmer, T., Jaarsma, A.D.C., Georgiadis, J.R.: Gamification of health professions education: a systematic review. Adv. Health Sci. Educ. 26(2), 683–711 (2021). https://doi.org/10.1007/s10459-020-10000-3 5. Cafazzo, J.A., Casselman, M., Hamming, N., Katzman, D.K., Palmert, M.R.: Design of an mHealth app for the self-management of adolescent type 1 diabetes: a pilot study. J. Med. Internet Res. 14(3), e70 (2012). https://doi.org/10.2196/jmir.2058 6. Katsarou, A., et al.: Type 1 diabetes mellitus Nat. Rev. Dis. Primers.3(1), 17016 (2017). https:// doi.org/10.1038/nrdp.2017.16 7. Hamilton, K., et al.: Sustained type 1 diabetes self-management: specifying the behaviours involved and their influences. Diabet. Med. 38(5), e14430 (2021). https://doi.org/10.1111/ dme.14430 8. Kim, B., Park, H., Baek, Y.: Not just fun, but serious strategies: using meta-cognitive strategies in game-based learning. Comput. Educ. 52(4), 800–810 (2009). https://doi.org/10.1016/J.COM PEDU.2008.12.004 9. Çeker, E., Özdamli, F.: What ‘gamification’ is and what it’s not. Eur. J. Contemp. Educ. 6(2), 221–228 (2017). https://doi.org/10.13187/ejced.2017.2.221 10. glaice: #1 Diabetes exercise-app. https://www.glaice.de/

454

M. A. Rauchensteiner et al.

11. Donevant, S.B., Estrada, R.D., Culley, J.M., Habing, B., Adams, S.A.: Exploring app features with outcomes in mHealth studies involving chronic respiratory diseases, diabetes, and hypertension: a targeted exploration of the literature. J. Am. Med. Inform. Assoc. 25(10), 1407–1418 (2018). https://doi.org/10.1093/jamia/ocy104 12. Eccles, D.W., Arsal, G.: The think aloud method: what is it and how do I use it? Qual. Res. Sport Exer. Health 9(4), 514–531 (2017). https://doi.org/10.1080/2159676X.2017.1331501 13. Bangor, A., Kortum, P.T., Miller, J.T.: An Empirical evaluation of the system usability scale. Int. J. Hum. Comput. Interact. 24(6), 574–594 (2008). https://doi.org/10.1080/104473108022 05776 14. Dam, R., Siang, T.: What is Design Thinking and Why Is It So Popular? (2018) 15. Boren, M.T., Ramey, J.: Thinking aloud: Reconciling theory and practice. IEEE Trans. Prof. Commun. 43(3), 261–278 (2000). https://doi.org/10.1109/47.867942 16. Lewis, J.R.: The system usability scale: past, present, and future. Int. J. Hum. Comput. Interact. 34(7), 577–590 (2018). https://doi.org/10.1080/10447318.2018.1455307 17. Klug, B.: An overview of the system usability scale in library website and system usability testing. Weave: J. Libr. User Exp. 1(6)https://doi.org/10.3998/weave.12535642.0001.602 18. J. Brooke, SUS: a retrospective, vol. 8, no. 2 (2013.) https://www.researchgate.net/profile/johnbrooke-6/publication/285811057_sus_a_retrospective 19. Sauro, J.: SUStisfied? Little-known system usability scale facts user experience magazine. User Exp. Mag. 10(3) (2011). https://uxpamagazine.org/sustified/ 20. Vlachogianni, P., Tselios, N.: Perceived usability evaluation of educational technology using the System Usability Scale (SUS): a systematic review. J. Res. Technol. Educ. 54(3), 392–409 (2021). https://doi.org/10.1080/15391523.2020.1867938 21. Lewis, J.R. Sauro, J.: Item benchmarks for the system usability scale. J. Usabil. Stud. 13(3), 1– 10 (2018). https://www.researchgate.net/profile/james-lewis-8/publication/330225055_item_ benchmarks_for_the_system_usability_scale 22. Finstad, K.: The system usability scale and non-native English speakers. J. Usabil. Stud. 1(4), 185–188 (2006). https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.365.2352&rep= rep1&type=pdf 23. avec DiabiLive: DiabiLive. https://www.diabilive.com/subscription 24. Glycomate: Glycomate - Transform Your Diabetes Control. Glycomate. https://www.glycom ate.com/ 25. Michaelides, C.: Physical Activity in Diabetes Improves Blood Glucose Control. Glycomate. https://www.glycomate.com/physical-activity-in-diabetes/ 26. Schenker, M.: The Minimalist Design Trend: Why Less Is More - Creative Market Blog (2021). https://creativemarket.com/blog/minimalist-design-trend. Accessed 13 Sep 2022 27. Cherry, K.: The Color Psychology of Blue (2020) https://www.verywellmind.com/the-colorpsychology-of-blue-2795815. Accessed 13 Sep 2022 28. How Functional Animation Helps Improve User Experience — Smashing Magazine. https:// www.smashingmagazine.com/2017/01/how-functional-animation-helps-improve-user-experi ence/. Accessed 27 Feb 2023 29. Zhao, J., Lin, L., Sun, J., Liao, Y.: Using the summarizing strategy to engage learners: empirical evidence in an immersive virtual reality environment. Asia-Pac. Educ. Res. 29(5), 473–482 (2020). https://doi.org/10.1007/s40299-020-00499-w 30. Verhoeff, T.: The role of competitions in education, December 1997. https://www.researchgate. net/publication/228714944_The_role_of_competitions_in_education. Accessed 13 Sep 2022 31. Baydillah, P.: Importance Of Communication In Education 32. Art Markman Ph.D.: Motivation and the middle | Psychology Today (2011). https://www.psy chologytoday.com/us/blog/ulterior-motives/201105/motivation-and-the-middle. Accessed 13 Sep 2022

Intelligent Digital Humans for Bias-Free Recruitment Interviews: A Diversity & Inclusion Training Program Fernando Salvetti, Barbara Bertagni, and Ianna Contardo

Abstract This article is about an intelligent digital human model enhanced by artificial intelligence, designed to meet the requirements from a multinational company in need of training for their human resources personnel on bias-free recruitment interviews. We have been creating a new generation of avatars with social intelligence, who are capable not only of presenting a wide variety of topics in a dynamic and engaging manner but also of interacting with the audience and communicating emotions and moods. We have been customizing avatars for role plays, building them as real interlocutors who facilitate training in how to handle difficult conversations by including aspects such as non-verbal communication, different communication styles, and diversity and inclusion. Practicing conversations with avatars accelerates learning from experience without the risks associated with learning in the field. At the end of each interview, timely feedback is provided so learners can determine how to improve their performance. These digital humans are able to perform like realistic human beings, challenging the interviewer both at a verbal and para-verbal level, as well on the cognitive and the emotional levels — making it easy for the interviewer to get trapped into biases and false assumptions. The key message is this: diversity and inclusion best practices are, first of all, about mindset. Keywords Bias-free interviews · Diversity & Inclusion · Digital humans

F. Salvetti (B) · B. Bertagni · I. Contardo e-REAL Labs at Logosnet, 10014 Turin, Italy e-mail: [email protected] e-REAL Labs at Logosnet, 6900 Lugano, Switzerland e-REAL Labs at Logosnet, Houston, TX 77008, USA e-REAL Labs at Logosnet, New York, NY 10013, USA © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_36

455

456

F. Salvetti et al.

1 The Problem of Biases in Recruitment Interviews and the Intelligent Digital Humans as a Solution This article explores the potential of using conversational digital humans enhanced by artificial intelligence for training human resources personnel to conduct bias-free recruitment interviews. Recruitment interviews are a critical part of the hiring process for any organization. Interviewers can be prone to biases, leading to a less diverse and inclusive workforce. Biases can stem from various factors, including unconscious biases, stereotypes, and personal preferences. Biases can also occur due to preconceived notions about gender, race, and age, leading to discrimination in the hiring process [1–5]. For example, an interviewer may prefer candidates who have similar educational backgrounds, leading to a less diverse workforce. Training programs using intelligent digital humans for interaction help reduce biases in recruitment interviews [6–14]. The problem of biases in recruitment interviews refers to the tendency of interviewers to make decisions based on irrelevant factors such as the candidate’s age, gender, ethnicity, appearance, and other personal characteristics, rather than their qualifications and skills. Bias can arise in recruitment interviews in several ways. For example, interviewers may have unconscious biases that affect their judgment, or they may be influenced by stereotypes and assumptions about certain groups of people. They may also be more likely to hire candidates who are similar to themselves or similar to those who have been successful in the past (affinity bias). To address the problem of biases in recruitment interviews, organizations can take several steps. One approach is to use structured interviews, which involve asking each candidate the same set of questions in the same order. This can help ensure that all candidates are evaluated on the same criteria and that interviewers are not influenced by irrelevant factors. Another approach is bringing diversity to interview panels to avoid hiring based on shared biases. Using technology can also reduce bias in recruitment. For example, using artificial intelligence to analyze resumes and job applications to identify the most qualified candidates, or using online assessments to evaluate candidates’ skills and abilities in a standardized and unbiased way. Blind screening is another way to improve the fairness of the entire hiring process. The program removes any unnecessary information from the candidate’s resume, such as their name, age, racial background, and gender. This helps to reduce the recruiter’s unconscious biases and ensure that every candidate is evaluated based on their qualifications and skills rather than personal biases. A further different approach — the one we developed with ENI, a multinational company employing over 31,000 people in nearly 70 countries around the world [15] — is based on interviewers’ training aimed at enhancing the awareness of their biases in order to take actions to mitigate them. In our case, this involves providing education and training on diversity and inclusion and encouraging interviewers to seek feedback and input from colleagues and other stakeholders. To do so, we provided a platform designed to enhance social interaction by sharing insights, knowledge, and ideas — a structure that is familiar to anyone who has used

Intelligent Digital Humans for Bias-Free Recruitment Interviews …

457

Fig. 1 The home page of the e-REAL Online Platform about bias-free recruiting interviews

typical social media platforms, allowing learners to add their content easily, whether it be text, image, or video. They also have control over where and how it is seen, through the use of categories and tags; furthermore, this solution fosters a learning motivation culture, supported by a custom-built gamification schema, where colleagues move through leaderboards and rankings according to their level of accomplishment, while active engagement is acknowledged and receives enterprise-wide visibility (Fig. 1). The key educational areas of the platform — known as e-REAL Online [16] — are as follows: • Simulated recruiting interviews with diverse conversational avatars that perform as candidates. • Tests and self-assessment tools to enable learners to measure their unconscious bias. • Evaluation exercises to reach a bias-free mindset. • Riddles and quizzes to further challenge implicit bias. At the core of the learning experience there are the conversational digital humans that are (artificially) intelligent enough to challenge the learners and to really enhance the target skills and competencies.

2 Conversational Digital Humans Enhanced by Artificial Intelligence At e-REAL Labs, we have been creating a new generation of avatars with social intelligence that are capable not only of presenting a wide variety of topics in a dynamic and engaging manner but also of interacting with the audience and communicating emotions and moods [17].

458

F. Salvetti et al.

We have been customizing avatars for role-plays, building them as real interlocutors who facilitate training in how to manage difficult conversations by including aspects such as non-verbal communication, different communication styles, and diversity. At the end of each interview, timely feed-back is provided by the tracking system embedded in the e-REAL platform, so learners can work out how to improve their performance. These digital humans are able to perform like realistic human beings, challenging their interviewers both at a verbal and para-verbal level, as well as on the cognitive and the emotional levels — making it easy for the interviewers to get trapped into biases and false assumptions (Fig. 2). Conversational digital humans enhanced by artificial intelligence are computergenerated characters, or avatars, that simulate human conversation in a natural and engaging way. These digital humans are created using advanced artificial intelligence techniques such as natural language processing, machine learning, and computer vision. They are programmed to interact with users in a variety of ways, such as answering and asking questions, sharing opinions, and other aspects of real recruiting interviews. They are customized to display different personalities, relational and communicative styles, and emotions, which can help to create a consistent and engaging user experience. They are also trained to learn from user interactions, allowing them to improve their responses. Overall, they are programmed to

Fig. 2 Representative conversational digital humans enhanced by artificial intelligence

Intelligent Digital Humans for Bias-Free Recruitment Interviews …

459

Fig. 3 A representative conversational digital human in action within a representative office dedicated to hiring interviews

attend recruitment interviews as candidates: Their responses, voices, gestures, and facial expressions are programmed to challenge the learners who have to perform as recruiters (Fig. 3).

3 Communication with Digital Humans Digital humans are designed to understand natural language and respond in a conversational manner. Every digital human is unique and has different capabilities and limitations. For example, using ambiguous language or complicated jargon may confuse the digital human; adding context for questions or requests helps the digital human understand the purpose of the conversation and provide more relevant and accurate responses. Avoiding interruptions while the digital human is speaking allows for a more fluid conversational experience. Digital humans are not at all human beings, even if they are designed in ways that are making them quite close to us. One of the key benefits of conversational digital humans is their ability to provide a human-like interaction without the need for a human agent. This can be particularly useful in situations where human agents may not be available 24/7 or where there is a high volume of interactions that need to be managed. Another benefit of using intelligent digital humans is that they allow learners to practice a range of situations and to interact with different kinds of personalities in a short amount of time, improving their expertise and their knowledge. Digital humans are an essential component of this training program at ENI, which is aimed at improving a standardized and consistent evaluation of candidates. By using digital humans first, educational outputs are showing that the training targets

460

F. Salvetti et al.

Fig. 4 By scanning the QR code, an avatar will appear and will perform a short self-introduction and provide an online calendar allowing you to book a meeting with a number of intelligent avatars ready to talk with real human beings

are met. Overall, we can add that conversational digital humans enhanced by artificial intelligence have the potential to transform the way that businesses and organizations deliver training in general, providing more engaging and personalized experiences that can drive business growth. Further research is needed to explore the potential of intelligent digital humans in reducing biases in recruitment interviews. At e-REAL Labs, we’re committed to this research because we envisage generative artificial intelligence as an interesting driver for education and training. A representative conversational digital human to interact with is available by scanning the QR code below and then scheduling a meeting online (Fig. 4): Regarding the educational program introduced here, we can say that biases in recruitment interviews can lead to a less diverse and inclusive workforce. The educational program co-developed with ENI provides a solution that reduces biases in recruitment interviews. The use of intelligent digital humans can help to improve the fairness of the hiring process, reduce the time and resources required for recruitment interviews, and increase the diversity and inclusivity of the workforce.

References 1. Burdick, A.: Unconscious Bias. Summersdale, London (2021) 2. Priest, H.: Biases & Heuristics. Independently Published (2019) 3. Behavioral Research Group: Bias: Encyclopaedia of Biases and Heuristics. Independently Published (2020) 4. Tversky, A., Kahneman, D.: Judgement under uncertainty: heuristics and biases. In: Science. New Series, vol. 185, N. 4157 (1974) 5. Gigerenzer, G., Brighton, H.: Homo heuristicus: why biased minds make better inferences. In: Topics in Cognitive Science, vol. 1 (2009) 6. Otugo, O., Alvarez, A., Brown, I., Landry, A.: Bias in recruitment: a focus on virtual interviews and holistic review to advance diversity. AEM Educ. Training 5(S1), S135–S139 (2021). https:// doi.org/10.1002/aet2.10661 7. Consul, N., Strax, R., DeBenedectis, C.M., Kagetsu, N.J.: Mitigating unconscious bias in recruitment and hiring. J. Am. College Radiol. 18(6), 769–773 (2021). https://doi.org/10.1016/ j.jacr.2021.04.006

Intelligent Digital Humans for Bias-Free Recruitment Interviews …

461

8. Beattie, G., Johnson, P.: Possible unconscious bias in recruitment and promotion and the need to promote equality. Perspect. Policy. Pract. Higher. Educ. 16, 1 (2012) 9. Vivek, R.: Is blind recruitment an effective recruitment method? Int. J. Appl. Res. Bus. Manage. 3, 3 (2022) 10. Reslan, F.: Understanding the impact of confessional diversity in the Lebanese public sector. Case study the Lebanese ministry of finance. Current Psychol. 11, 1–13 (2022) 11. Bhanot, A.: A novel review on the adaptation of artificial intelligence in human resources management by organizations in gulf countries. In: Handbook of Research on Innovative Management Using AI in Industry 5.0 (2022) 12. Imran, R.: Re-inventing human resource management through artificial intelligence. In: Fourth Industrial Revolution and Business Dynamics (2021) 13. Sidoruk, J., Ritter, H.: Künstliche Intelligenz – Kriegstreiber oder Friedensstifter im War for Talent? In: Altenburger, R., Schmidpeter, R. (eds.) CSR und Künstliche Intelligenz, pp. 327– 340. Springer, Berlin (2021). https://doi.org/10.1007/978-3-662-63223-9_17 14. Steed, K., De Nobile, J., Waniganayake, M.: Promotion to leadership, not just merit, but insider knowledge: What do school principals say? J. Educ. Leadersh. Policy Pract. 36, 1 (2021) 15. www.eni.com 16. www.e-real.net 17. Salvetti, F., Gardner, R., Minehart, R., Bertagni, B.: Effective extended reality: a mixed-reality simulation demonstration with digitized and holographic tools and intelligent avatars. In: Guralnick, D., Auer, M., Poce, A. (Eds.). Innovative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education. Proceedings of the Learning Ideas Conference 2022. Springer, Cham (2023)

Extended Reality and Medical Simulation: Cooperation into the Metaverse to Boost Diversity and Inclusion Fernando Salvetti, Roxane Gardner, Jenny Rudolph, Rebecca Minehart, Barbara Bertagni, and Ianna Contardo

Abstract Metaverse is a term used to describe a hypothetical shared virtual space where people can interact with a computer-generated environment and each other, and there are multiple metaverses currently being developed by various companies and organizations. Cooperation in the metaverse is at the core of the ongoing digital revolution that impacts the way we design and deliver overall education and training. Medical simulation is a powerful way to deliver education and training, based on the use of technology and other techniques to recreate clinical scenarios for the purpose of teaching and training healthcare professionals and students. This article is about how to involve learners in a metaverse within the medical simulation field. The key questions that we address are as follows: What is the metaverse today? What will it look like in a few years? How do we enhance medical simulation based on cooperation in the metaverse? How do we engage learners with diversity and inclusion? Keywords Extended Reality · Metaverse · Cooperation · Diversity & Inclusion

F. Salvetti (B) · B. Bertagni · I. Contardo e-REAL Labs at Logosnet, 10014 Turin, Italy e-mail: [email protected] F. Salvetti · R. Gardner · J. Rudolph · R. Minehart · B. Bertagni · I. Contardo e-REAL Labs at Logosnet, 6900 Lugano, Switzerland e-REAL Labs at Logosnet, Houston, TX 77008, USA e-REAL Labs at Logosnet, New York, NY 10013, USA R. Gardner · J. Rudolph · R. Minehart Center for Medical Simulation, Boston, MA 02129, USA R. Gardner Brigham and Women’s Hospital/Children’s Hospital Massachusetts General Hospital and Harvard Medical School, Boston, MA 02114, USA J. Rudolph · R. Minehart Massachusetts General Hospital and Harvard Medical School, Boston, MA 02114, USA © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_37

463

464

F. Salvetti et al.

1 Extended Reality and Metaverse for Education and Training Extended Reality (XR) is an umbrella term used to describe a combination of virtual reality, augmented reality, and mixed reality technologies [1]. XR provides users with immersive experiences that can be used for various purposes, including entertainment, education, and communication [2]. With the growing use of XR, there is an opportunity to leverage this technology to promote cooperation as well as diversity and inclusion. This paper explores the potential of XR to promote cooperation and boost diversity and inclusion in the metaverse based on the experiences developed using e-REAL® that is the enhanced reality system for immersive simulation that we have been developing at Logosnet since 2012. Primarily, e-REAL is the merging of real and virtual worlds, a mixed-reality environment for hybrid simulation where physical and digital objects are available for gesture shaping, visual, tactile, and vocal interaction both in a physical learning setting and online. The e-REAL immersive setting is fully interactive with 3D visualizations, avatars, electronically writable surfaces, and more. It is a glasses-free solution, where users experience full immersion without the need for Virtual Reality (VR) glasses or goggles [3] (Fig. 1). Metaverse is a term used to describe a hypothetical shared virtual space where people can interact with a computer-generated environment and each other, and there are multiple metaverses currently being developed by various companies and organizations. The metaverse is expected to be a virtual space where users can interact with each other in real time. Even if there are companies and organizations stating that the metaverse is already there, we assume that the metaverse is something under development — there will be more than the virtual place already available. A fully developed metaverse doesn’t really exist today. It is expected to be a collective virtual

Fig. 1 The immersive spectrum of XR with e-REAL at the core (red dot) as a hybrid reality solution enabling glasses-free learning experiences, both on site (Mixed Reality into a “phygital” environment or Hybrid Reality) and online (cloud-based Virtual Reality)

Extended Reality and Medical Simulation: Cooperation …

465

space, created by the convergence of virtually enhanced physical and digital reality, like the virtual objects and the avatars that we are interacting with inside the e-REAL labs. With the avatars to speak with and interact with, as well as the interactive digital objects and the multisensory 3D scenarios, we are currently working with some of the key components of a metaverse. And, let’s say, we are “e-really” expected to develop a metaverse in less than 5 years from now [4]. Let’s think of it as the next version of the Internet: We’re a few years away from the actualization of the Internet’s 3D layer of interoperable and interlinked immersive environments needed to have a metaverse. Media headlines bolster any extended reality experience as the metaverse, but it is misleading because today we only have some of its infrastructure components. The full metaverse will be used for various purposes, including gaming, socializing, and learning because the metaverse has the potential to provide a safe and inclusive space where users can engage with others regardless of their physical location, race, or gender [5, 6] (Figs. 2 and 3).

Fig. 2 The elements of a metaverse according to Gartner Group

466

F. Salvetti et al.

Fig. 3 The evolution spectrum for the metaverse according to Gartner Group

2 Designing an Immersive Learning Experience in the Metaverse Today, we can already deploy our training and simulation programs in a metaverselike environment such as e-REAL. We are doing so in order to explore and envisage what the metaverse looks like in terms of instructional design. Designing an immersive learning experience — like a medical simulation — in the metaverse requires a multi-disciplinary approach, incorporating aspects of education, game design, and virtual world development [7–9]. Moreover, the most effective and meaningful XR seems to be possible only when different expertise domains are represented in data and content (e.g., ophthalmology and neurology). Here are some general steps that can be taken to create such an experience: • Identify the learning objectives you want to achieve through the immersive experience. • Determine the target audience for the immersive experience, including their interests, preferences, and learning styles. • Choose a platform enabling a metaverse experience: it will be a platform enabling presence and shared experiences from anywhere — on any device — through mixed reality applications. e-REAL is an effective solution because it enables the merging of real and virtual dimensions. • Build a virtual environment that is immersive and engaging: This should include interactive objects, 3D models, audio, and other elements that create a realistic experience. Consider which elements will best support your learning objectives. Will it be a virtual trip, a game, a simulation, or something else? The elements

Extended Reality and Medical Simulation: Cooperation …

• • • •

•

•

467

you choose should be able to create a sense of presence and immersion for the learners. Create a story or scenario that the learners can become fully engaged in. This can be done through storytelling, role-playing, or — as in our case — simulation that is inclusive also of role-playing. Incorporate learning materials into the virtual environment, such as videos, audio recordings, text, and interactive activities. This should be done in a way that is fun and engaging but also facilitates learning. Provide interactive elements like quizzes, challenges, and games can help keep learners engaged and motivated. Interactive elements can also be used to reinforce key learning points. Design scenarios and challenges that require the learner to use critical thinking, problem-solving, and decision-making skills to progress through the experience. These should be designed to align with the learning objectives and to provide feedback to the learners. Use technology to enhance the experience: Technology can be used to create a more immersive learning experience. For example, virtual reality can be used to create a fully immersive environment, while augmented reality can be used to enhance the learner’s understanding of a concept. Provide feedback to the learner throughout the experience, including progress updates, achievements, and challenges. This helps to keep the learners engaged and motivated. Feedback and assessment are crucial elements of any learning experience. They help learners understand what they have learned and where they need to improve. Incorporate these elements into your immersive learning experience to make it more effective.

Finally, it’s important to continuously evaluate and improve your immersive learning experience. Gather feedback from learners and adjust as needed to ensure that your learning objectives are being met. Test the immersive experience with a small group of learners and gather feedback to refine and improve the experience. This process should be iterative, with feedback incorporated into the design to create a more effective and engaging experience. By following these steps, we can create an immersive learning experience that engages learners and helps them develop new skills and knowledge. e-REAL, as a Mixed Reality environment for hybrid simulation, can be a stand-alone solution or even networked between multiple locations, linked by a special videoconferencing system optimized to process operations without perceivable latency. This connectivity allows not only virtual object sharing (like medical imagery, infographics, etc.) in real time, but also remote cooperation between participants.

468

F. Salvetti et al.

3 Interprofessional Cooperation and Medical Simulation in the Metaverse Interprofessional cooperation into medical simulation refers to the use of simulationbased education to train healthcare professionals from different disciplines to work collaboratively as a team. The metaverse will be a collective virtual shared space created by the convergence of virtual reality, augmented reality, and other immersive technologies. Combining these two concepts can potentially provide a unique and effective way to train healthcare professionals in an immersive and collaborative environment. One possible application of interprofessional medical simulation cooperation in the metaverse is to create a virtual hospital setting where healthcare professionals from various disciplines can interact and learn to work together to provide patient care. The virtual hospital can be designed to simulate different clinical scenarios, including emergency situations, surgeries, and patient consultations. Healthcare professionals can use virtual tools and equipment to perform various procedures and communicate with each other to make decisions and coordinate care. The use of the metaverse in interprofessional medical simulation cooperation can offer several advantages. Firstly, it provides a safe and controlled environment for healthcare professionals to practice and learn new skills without putting patients at risk. Secondly, it can help break down communication barriers between healthcare professionals from different disciplines by providing a common virtual space for them in which to interact and collaborate. Lastly, it can enhance the learning experience by providing immersive and engaging simulations that can better prepare healthcare professionals for real-world situations. However, there are also some challenges that need to be addressed when implementing interprofessional medical simulation cooperation in the metaverse. These include technical issues related to the design and implementation of the virtual environment, the need for high-quality simulations that accurately replicate real-world scenarios (making them accessible at different locations, even if with different equipment), and the need for a robust and reliable communication platform to facilitate interprofessional collaboration. In summary, the use of the metaverse in interprofessional medical simulation cooperation has the potential to provide a unique and effective way to train healthcare professionals in an immersive and collaborative environment. However, it requires careful planning and implementation to ensure that it is effective and addresses the challenges associated with it. Enhancing medical simulation based on cooperation in the metaverse requires a comprehensive approach that involves designing the virtual environment (to deploy also into a hybrid or “phygital” environment), developing high-quality simulations, and providing effective communication and collaboration tools. Here are some ways to enhance medical simulation based on cooperation in the metaverse: • Design a realistic virtual environment: The virtual environment should closely mimic real-world hospital settings, including patient rooms, operating rooms,

Extended Reality and Medical Simulation: Cooperation …

•

•

•

•

•

469

emergency departments, and/or outdoor environments. The virtual environment should also be designed to include various equipment and tools necessary for healthcare professionals to perform their tasks. Develop high-quality simulations: The simulations should accurately replicate real-world clinical scenarios and should be challenging enough to provide a learning experience. The simulations should also allow for different scenarios, including rare or complex situations that healthcare professionals may not encounter often. Use multi-disciplinary team-based scenarios: The simulations should be designed to promote collaboration between healthcare professionals from different disciplines. The scenarios should encourage communication and coordination among team members to achieve common goals. Incorporate innovative technologies: The metaverse allows for the use of innovative technologies like virtual and augmented reality, which can help healthcare professionals experience a simulated environment in a more realistic and immersive way. Additionally, the use of haptic technology can provide tactile feedback and enhance the realism of the simulation. Provide effective communication and collaboration tools: The virtual environment should provide effective communication and collaboration tools that allow healthcare professionals to communicate with each other during simulations. These tools can include audio and video communication, text chat, and virtual whiteboards. Incorporate feedback mechanisms: The simulations should have mechanisms for providing feedback to healthcare professionals on their performance during the simulation. This feedback can help healthcare professionals identify areas where they need improvement and work on those areas.

Enhancing medical simulations based on cooperation in the metaverse requires careful planning and implementation. By designing a realistic virtual environment, developing high-quality simulations, using multi-disciplinary team-based scenarios, incorporating innovative technologies, providing effective communication and collaboration tools, and incorporating feedback mechanisms, healthcare professionals can benefit from an immersive and effective learning experience (Figs. 4, 5 and 6).

4 Boosting Diversity and Inclusion with the XR Technologies in the Metaverse XR technologies can be used to create immersive and engaging experiences that encourage cooperation and collaboration among learners. At the same time, XR can be used to boost diversity and inclusion in the metaverse in various ways. eREAL is a system designed with diversity, equity, and inclusion in mind. Firstly, XR can be used to create immersive experiences that represent diverse cultures and communities. This can help to promote cultural understanding and reduce prejudice

470

F. Salvetti et al.

Fig. 4 Representative e-REAL scenario designed for enhancing cooperation into a metaversebased simulation (for aviation and space medicine) where some people are performing in a hybrid environment, while others are in a fully virtualized dimension whose components are digital tools and conversational avatars

Fig. 5 Conversational avatars are representative components of an e-REAL cooperative metaverse: By scanning the QR code, an avatar will appear and will perform a short self-introduction (Augmented Reality) making available an online calendar in order to book a meeting with a number of intelligent avatars ready to talk with real human beings

and discrimination. Secondly, XR can be used to create inclusive environments that cater to the needs of users with disabilities. For example, XR can be used to simulate a wheelchair-friendly environment or provide audio descriptions for users with visual impairments. XR can also be used to promote diversity and inclusion by providing equal opportunities for users to participate in the metaverse. For example, XR technologies can be used to remove barriers to entry, such as physical location, by providing a virtual space where users can engage with each other. This can help to increase the diversity of users in the metaverse and promote inclusivity.

Extended Reality and Medical Simulation: Cooperation …

471

Fig. 6 Representative conversational avatars, that are digital humans enhanced by artificial intelligence from an e-REAL environment

Engaging learners with diversity and inclusion [10] is essential for promoting a positive learning experience and creating an inclusive environment. Here are some ways to engage learners with diversity and inclusion: • Creating a safe and inclusive environment is crucial to engaging learners with diversity and inclusion. Learners should feel that their contributions are valued and their experiences and perspectives are respected. Educators and trainers should establish ground rules that promote mutual respect and foster an environment that welcomes diversity. • Using examples and case studies that represent a range of experiences and perspectives can help learners appreciate the diversity of the world around them. Teachers can use examples from different cultures, genders, religions, and backgrounds to help learners develop a broader understanding of the world. • Encouraging open and honest communication can help learners share their experiences and perspectives. Teachers and trainers can facilitate group discussions that promote respectful dialogue and encourage learners to ask questions and share their thoughts. • Active listening is an important skill for engaging learners with diversity and inclusion. Teachers can encourage learners to listen actively to their peers and acknowledge their perspectives and experiences. • Creating opportunities for reflection and self-assessment can help learners identify their own biases and assumptions. Teachers can encourage learners to reflect on their experiences and perspectives and consider how those might affect their interactions with others. • Providing resources and support can help learners engage with diversity and inclusion more effectively. Teachers can provide reading materials, videos, and other resources that promote diversity and inclusion. They can also connect learners

472

F. Salvetti et al.

with support services that help them understand and navigate issues related to diversity and inclusion. Engaging learners with diversity and inclusion is essential for promoting a positive learning experience, like a medical simulation, and creating an inclusive environment by fostering a safe and inclusive environment, using diverse examples and case studies, encouraging open and honest communication, promoting active listening, creating opportunities for reflection and self-assessment, and providing resources and support, teachers and trainers can help learners engage with diversity and inclusion more effectively. One way to promote a safe and inclusive environment is by setting clear expectations for behavior and communication. Trainers should emphasize the importance of treating all virtual patients and colleagues with respect and encourage learners to ask questions and seek feedback in a respectful and non-judgmental manner. Using diverse examples and case studies that XR contributes enormously to create — for example, by changing gender, age, or ethnicity to the avatars that are populating a simulation session — can also help learners to see the value of diversity and inclusion in healthcare and understand the unique challenges faced by different patient populations. Active listening is another important skill that can be developed through extended reality-based simulation. Trainers can create opportunities for learners to practice active listening by incorporating role-playing scenarios where they must listen to a virtual patient’s concerns and respond with empathy and understanding (changing for example specific attributes related to gender, age, or ethnicity in a way that is not feasible during a standard simulation). Reflection and self-assessment can also be powerful tools for promoting equity and inclusion by encouraging learners to reflect on their own biases and assumptions and consider how they can improve their interactions with diverse patient populations that, in an XR setting, may vary more than in a traditional simulation. Finally, providing resources and support can help to close collaboration, equity, and inclusion gaps in medical simulation. This could include access to cultural competency training, mentorship opportunities, and resources for addressing implicit bias. By incorporating these elements into extended reality-based simulation scenarios, trainers and educators can help to create a more inclusive learning environment and prepare healthcare professionals to provide high-quality care to diverse patient populations. XR technologies, with e-REAL at the forefront, offer a unique opportunity to promote cooperation and boost diversity and inclusion in the metaverse. By creating immersive and inclusive experiences, XR can provide a safe and engaging space for users to interact with each other regardless of their physical location, race, age, or gender. The use of XR technologies can help to remove barriers to entry and promote equal opportunities for users to participate in the metaverse. Further research is needed to explore the potential of XR in promoting diversity and inclusion in the metaverse.

Extended Reality and Medical Simulation: Cooperation …

473

References 1. Salvetti, F., Gardner, R., Minehart, R., Bertagni, B.: Effective extended reality: a mixed-reality simulation demonstration with digitized and holographic tools and intelligent avatars. In: Guralnick, D., Auer, M., Poce, A. (Eds.) Innovative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education. Proceedings of the Learning Ideas Conference 2022. Springer, Cham (2023) 2. Marr, B.: Extended Reality in Practice. Wiley, Chichester (2021) 3. www.e-real.net. 15 May 2023 4. Salvetti, F., Gardner, R., Minehart, R., Bertagni, B.: Enhanced Reality for Healthcare Simulation. In: Brooks, A.L., Brenham, S., Kapralos, B., Nakajima, A., Tyerman, J., Jain, L. (Eds.) Recent Advances in Technologies for Inclusive Well-Being: Virtual Patients, Gamification and Simulation. Springer, Heidelberg (2021) 5. https://blogs.gartner.com/robert-hetu/metaverse-implications-for-retail-technology-and-ser vice-providers/. 15 May 2023 6. https://www.gartner.com/en/articles/metaverse-evolution-will-be-phased-here-s-what-itmeans-for-tech-product-strategy. 15 May 2023 7. Salvetti, F., Gardner, R., Minehart, R., Bertagni, B.: Medical simulation in the cloud: learning by doing within an online interactive setting. In: Guralnick, D., Auer, M., Poce, A. (Eds.). Innovative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education. Proceedings of the Learning Ideas Conference 2022. Springer, Cham (2023) 8. https://harvardmedsim.org/blog/an-interview-with-fernando-salvetti-and-barbara-bertagni-ofe-real/. 15 May 2023 9. https://www.bloomberg.com/press-releases/2022-10-10/e-real-launches-its-new-multi-sen sory-scenario-in-the-george-washington-university-immersive-mobile-learning-lab. 15 May 2023 10. Buchanan, D.T., RebeccaO’Connor, M.: integrating diversity, equity, and inclusion into a simulation program. Clin. Simul. Nurs. 49, 58–65 (2020). https://doi.org/10.1016/j.ecns.2020. 05.007

Using a Podcast to Foster Success Among Computer Science Students Sigrid Schefer-Wenzl and Igor Miladinovic

Abstract Over the past fifteen years, the use of podcasts has grown significantly in many areas, including education. In higher education, recording live lectures in face-to-face courses or offering additional learning materials in the form of podcast episodes is popular. However, podcasts can also serve other educational functions. In this paper, we introduce the concept of an educational podcast that we have launched for current and potential students in our computer science degree programs. The main goal of the podcast is to provide them with more information about our degree programs in order to increase student motivation, transparency, and success rates. In each episode, we focus on either a specific course or a specific study-related topic. In addition, we conduct expert interviews on current topics as well as interviews with our graduates. First statistics show a constant increase in the number of downloads. In future research, we plan a detailed evaluation among the listeners of our podcast. Keywords Computer Science Education · Educational Podcast · Podcast Design · Student Motivation

1 Introduction Over the past fifteen years, the popularity of podcasts has steadily increased. More and more organizations, including educational institutions, are using podcasts as a means to communicate with their customers [1, 2]. Educational podcasts appear in a variety of forms, such as lecture podcasts with short audio/video contributions on important topics [3] or podcasts to present scientific findings [4, 5]. We also launched a podcast two years ago, primarily to discuss courses in our computer science degree programs directly with the lecturers. Our goal was to S. Schefer-Wenzl (B) · I. Miladinovic Computer Science, University of Applied Sciences Campus Vienna, Vienna, Austria e-mail: [email protected] I. Miladinovic e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_38

475

476

S. Schefer-Wenzl and I. Miladinovic

increase the transparency of the courses for students and applicants, as the courses were only briefly described on our website. This caused students to have doubts about whether they could complete some courses and also to have false expectations. Over time, our podcast evolved beyond course presentations to discussions of current issues in computer science, insights and tips on study-related topics, and interviews with our graduates. The podcast is available over major podcast platforms as well as over our webpage as audio-only episodes. Therefore, it can be consumed worldwide and anytime, also during other activities. In this paper, we will share the design and purpose of our podcast, as well as the results of its popularity. This paper is structured as follows. In Sect. 2, we present related work on using podcasts in the field of higher education and science communication. Section 3 details the problem statement which encouraged us to start a podcast on our own. Section 4 introduces design variables for educational podcasts and outlines our podcast concept. In addition, we present some statistics regarding its popularity. Section 5 concludes the paper.

2 Related Work Podcasting has seen a growing interest over the past fifteen years. A podcast is a digital packet of audio that is usually part of a series, published at regular intervals, and downloadable to personal media devices [6]. In contrast to related media, such as radio and audiobooks, podcasts are distributed online without editing requirements by publishing houses or the need to fit into a broadcast timeslot. Consumers decide how and when they listen to podcast content. The lack of such restrictions means that podcast content is often seen as more authentic, in contrast to other radio formats [7]. Along these developments, podcasting in education has also spread rapidly in recent years, both inside and outside educational institutions [8]. Podcasts are widely viewed as positive learning tools that have the tangible benefit of bringing teachers and learners together, often across long distances. In most cases, the focus is on recording lectures that can be played back at a later point in time. Podcasts, then, are not seen as tools to improve students’ grades, but as a means to engage students in ways that motivate them to learn content and improve the relationship between teachers and students [9, 10]. Koehler [11] emphasizes that podcasts convey knowledge solely in an audio-based format. There is therefore no need to look at a display while consuming the content. Many authors point out that podcasting can’t replace the classroom, but it gives both students another way to learn new concepts and teachers to interact with their students. For example, in Polack-Wahl [12], the author points out that podcasting has proven to be a unique and successful method of disseminating information in two courses at the University of Mary Washington. The author applied podcasting in a course on software engineering as well as in a course on information systems, where students had to create a podcast within the lecture. She attributed the success to the

Using a Podcast to Foster Success Among Computer Science Students

477

fact that podcasting offers lecturers a method of communication based on technology that students have become accustomed to. Perez et al. [13] evaluated the effectiveness of two online teaching strategies, one with and one without podcasts. Results showed that the teaching strategy with podcasts resulted in better course evaluations and improved student performance compared to the other online teaching strategy. Similarly, Moura and Carvalho [14] developed a mobile learning-based concept including podcasts and evaluated that podcasts are a valuable tool to support the education process and to engage students in active learning. Another way to use podcasts in a university context is for science communication [15, 16]. The story-telling format of podcasts is argued to be very valuable to reach a broad audience and to also engage students in the research matters of a university. In contrast to our podcast, we only have found podcast concepts in the literature that are designed either for a specific lecture, a specific topic, or to communicate scientific findings. Our podcast has a much broader primary intention (i.e., to inform students about our degree programs), which we will explain in the next two sections.

3 Motivation for Starting a Podcast At our university, we are currently responsible for the Bachelor’s degree program “Computer Science and Digital Communication” and the two Master’s degree programs “Software Design and Engineering” and “Multilingual Technologies.” The main information point for applicants and students about the course contents of our degree programs is our website. However, here they can only find the course catalog as well as a short description of the contents and goals of each course. As a result, we frequently received questions about the details of our degree programs. In addition, students had very different expectations of what they will learn in their studies and were not sure whether they would be able to complete some courses. To address these issues, we considered the successful completion of a course as a goal for students. Then we defined these goals by using the well-known goal setting technique SMART [17, 18], which defines five characteristics of goals: specific, measurable, achievable, relevant, and timely (however, in the literature, assignable and realistic can also be found for A and R). It is expected that goals defined using this technique are more motivating and increase the likelihood of successful goal achievement. However, the lack of transparency in our courses meant that the goal (to successfully complete a course in the allocated time) was often not enough specific, achievable, and/or relevant. We therefore decided to launch a lightweight platform that would inform students about the learning objectives and contents of each course. A podcast proved to be the perfect format for our needs, as it can be consumed very easily, anywhere, anytime, and without further cost. In addition, our target audiences are daily consumers of various podcasts and are therefore very open to this format. Our podcast allowed

478

S. Schefer-Wenzl and I. Miladinovic

us to address the three SMART characteristics, specific, achievable, and relevant, as well as to improve clarity about our courses. However, there are also some disadvantages of this format that we need to mention. The biggest challenge is the time required to edit each episode, which is usually at least double or triple the time relative to the length of an episode. In addition, we do not want our podcast to be too much of a marketing tool. Rather, we want to convey a realistic picture of our degree programs in an interesting format. Besides informing our students about the courses in our degree programs in interview-style episodes with the respective course lecturer(s), the podcast format has several other positive side effects. First, it increases our attractiveness for potential students because we can present details about our courses together with the respective lecturers. Second, we can inspire people to engage with current topics related to our studies, as we often invite experts to discuss topics that are attractive to those interested in IT. Third, we can provide answers to frequently asked questions in a brief and yet detailed manner. Fourth, a podcast allows for edutainment in a simple way, as we can inform and entertain our listeners at the same time by bringing funny examples or side notes to each topic discussed. We launched our podcast at the beginning of 2021. Since then, it has continuously evolved and we have learned a lot in the process of creating this podcast. In the next section, we will discuss different podcast design variables and describe the concept of our podcast.

4 Design Concept for An Educational Podcast 4.1 Design Variables Several design variables can be used to categorize educational podcasts. The following five variables have been defined in other works [19–21] and are frequently used to describe podcasts: • Type of content: the main content type categories of educational podcasts include lecture recordings, review material, assessment feedback, administrative information, and interviews of experts. • Length: podcast length typically varies between short (1–5 min), medium (5– 15 min), and long (15 + minutes) • Author: Usually the podcast’s author is either a teacher, a student, a guest, or a combination of those. • Style: Formal podcasts are direct and often read from a prepared script. Informal podcasts can involve improvisational, humorous, and entertaining elements. • Purpose: Podcasts can be informative, reflective or motivating. The purpose can also be described using the cognitive verbs from Bloom’s taxonomy.

Using a Podcast to Foster Success Among Computer Science Students

479

The authors also suggest further categories to describe educational podcasts, such as voice and mode of address, fit with other learning materials, series structure, pedagogical approach behind the podcast as well as the subject area.

4.2 Podcast Concept Our podcast is called “10 nach 10 Podcast” (in English: “10 past 10 Podcast”). The name refers to the time 10 past 10, which is most often used to advertise analog watches. This has functional and aesthetic reasons, as the hands of the watch do not overlap, the brand logo and other elements such as the date display are usually visible, and the position is symmetrical, making the watch look more cheerful. However, with digital watches, all these advantages become less valid. Nevertheless, after the introduction of digital watches, the time display “10:10” was still used for advertisement. The name of our podcast is therefore a metaphor for being open to new developments and not repeating old patterns without reflection. In our roles as lecturers, researchers, and head of degree programs, we provide insights into the computer science programs at our university in our podcast. The target audiences are our students, applicants, and graduates, as well as others interested in these topics. Using the categories described in the previous Subsect. 4.1, we can set the following values for the main design variables: • Type: Our podcast provides an overview of the courses in our degree programs, but does not go into the details of each course. In addition, we have episodes providing study-related tips, interviews with experts on current topics and interviews with graduates. • Length: Each episode usually lasts between 15 and 25 min. • Author: The authors of the podcasts are the head of the degree program and one lecturer in these programs. • Style: The style is informal. We usually have a number of prepared questions, but we also add questions spontaneously. Our guests usually do not have prepared answers. • Purpose: The purpose of our podcast is to be informative and inspiring for students and applicants. In terms of content, we designed our podcast with a focus on the following four main categories of topics, which are also summarized in Fig. 1. The first category comprises all those episodes that introduce courses in our degree programs. In each episode we present one or more related courses together with the responsible lecturer(s). In addition to a discussion of the content, we also talk about the expected learning outcomes, their relevance to the job market, and the organization of the course. We also ask about the three most important lessons students should take away from this course. At the end of each episode in this category, we ask the lecturer(s) a few more personal questions to introduce them a bit outside of their

480

S. Schefer-Wenzl and I. Miladinovic

Courses

• Introduction of lectures and responsible lecturer(s) • Discussion of content, expected learning achievements, relevance for job market, organization of course(s) • Personal questions to the lecturer(s)

Insights

• Study-related insights and tips on studying • Answers to frequently asked questions • E.g. final examinations, bachelor's and master's theses, exchange semester, admission procedure

Hot topics

Graduates

• Current topics from the field of computer science and communication networks • Expert talks

• Insights from graduates of our degree programs • E.g. transfer from study to professional life, how to manage challenges during a study, tips for starting a career, factors toconsider when choosing a study

Fig. 1 Summary of our current podcast categories

lecturer role and make them more “human” to the students. Most of the currently published episodes can be assigned to this category. The second category is all about study-related insights and tips on studying. Here we talk about topics such as final exams, bachelor’s and master’s theses, exchange semesters, admission procedures, working alongside your studies, and how degree programs are structured at our university. When appropriate, we invite special guests to provide even more insights. Episodes in this section aim to answer frequently asked questions about important study-related topics. In the third category, we discuss current topics from the field of computer science and communication networks, such as design thinking, blockchain, DevOps, artificial intelligence or 5G/6G networks. For each episode, we invite an expert from the respective field and talk about the state of the art, possible future developments and the impact on the job market, especially in Austria. In addition to these categories, we also have a fourth category of episodes where we talk to our graduates. We discuss with them, for example, which topics or other aspects from their studies were particularly helpful for them in their later professional lives, how they dealt with particular challenges in their studies, what tips they can give for starting a career, and which factors they would consider when choosing a study today.

Using a Podcast to Foster Success Among Computer Science Students

481

4.3 Podcast Statistics Since the launch of our podcast two years ago, we have recorded a total of about 4,000 downloads of all episodes. At the time of writing this paper, we have published 38 episodes on our podcast. Thus, each episode has been listened to more than 100 times on average, even though the popularity of each episode fluctuates. We currently have about 400 students in our degree programs. Therefore, we can record an average of 10 downloads per student. In the most popular episode, we introduce our bachelor’s degree program. Listeners learn more about the main focus and highlights in the program, the structure of the program, the lecturers and how one can apply. The second most popular episode includes tips on writing bachelor’s and master’s theses. The third and fourth most popular downloaded episodes are on mathematics and programming courses in the first semester of our bachelor’s degree program. In fifth place is an elective module on artificial intelligence, which was recently introduced. About 60% of downloads were from mobile devices, 33% of downloads were from computers, and other downloads were from devices such as smart speakers or smart TVs. The most popular app used for listening to our podcast is Spotify, followed by the podcast website, Apple podcast and Amazon Music. More than 80% of the downloads originate from Europe, most of them from Austria (66%) and Germany (14%), which is not surprising since the language of our podcast is German. About 13% of the downloads come from Asia and 4% from North America.

5 Conclusion Podcasts offer an easy-accessible format to distribute information among a wide audience. In the education, podcasts are also used as a medium for communication between students and lecturers in both directions. Two years ago, we started an educational podcast with the goal of informing our students and applicants about our degree programs. We aimed to make learning objectives and content of our courses more transparent to increase success rates among students by reducing wrong expectations. Our podcast currently has four categories of topics: Overview of individual degree programs, study-related insights and tips, expert interviews on current topics, and interviews with graduates. In the near future, we plan to expand our podcast to include additional categories in which we report on our current research projects or events we are (co-)organizing. We are also planning a survey among students to get more structured feedback from our target group. In particular, we want to investigate size of groups of our audience (students, graduates, potential applicants, and other), what information is important for which group, whether we need more (or even less) categories, and how long our episodes should be. For that, we will use qualitative and quantitative interviews and promote them via our podcast.

482

S. Schefer-Wenzl and I. Miladinovic

We also plan to use podcasts as didactic tools, where students generate content to present outcomes of projects and group exercises. We plan to publish selected contributions out of this initiative as episodes in our podcast to promote students’ engagement among our audience.

References 1. Chan-Olmsted, S., Wang, R.: Understanding podcast users: consumption motives and behaviors. New Media Soc. 24(3), 684–704 (2022) 2. Rime, J., Pike, C., Collins, T.: What is a podcast? Considering innovations in podcasting through the six-tensions framework. Convergence 28(5), 1260–1282 (2022) 3. Andersen, R.H., Dau, S.: A review of podcasts as a learning medium in Higher Education. In ECEL 2021: 20th European Conference on e-Learning, pp. 34–41. Academic Conferences and Publishing International (2021) 4. Dantas-Queiroz, M.V., Wentzel, L.C., Queiroz, L.L.: Science communication podcasting in Brazil: the potential and challenges depicted by two podcasts. Anais da Academia Brasileira de Ciências 90, 1891–1901 (2018) 5. Quintana, D.S., Heathers, J.A.: How podcasts can benefit scientific communities. Trends Cogn. Sci. 25(1), 3–5 (2021) 6. Bonini, T.: The ‘second age’ of podcasting: Reframing podcasting as a new digital mass medium. Quaderns Del. CAC 41(18), 21–30 (2015) 7. Meserko, V.M.: The pursuit of authenticity on Marc Maron’s WTF podcast. Continuum 29(6), 796–810 (2015). https://doi.org/10.1080/10304312.2015.1073682 8. Celaya, I., Ramírez-Montoya, M.S., Naval, C., Arbués, E.: The educational potential of the podcast: an emerging communications medium educating outside the classroom. In: Proceedings of the Seventh International Conference on Technological Ecosystems for Enhancing Multiculturality, pp. 1040–1045 (2019) 9. Conroy, D., Kidd, W.: Using podcasts to cultivate learner–teacher rapport in higher education settings. Innov. Educ. Teach. Int. 1–11 (2022) 10. Carle, A.C., Jaffee, D., Miller, D.: Engaging college science students and changing academic achievement with technology: a quasi-experimental preliminary investigation. Comput. Educ. 52(2), 376–380 (2009) 11. Koehler, D., Serth, S., Meinel, C.: Consuming security: evaluating podcasts to promote online learning integrated with everyday life. In 2021 World Engineering Education Forum/Global Engineering Deans Council (WEEF/GEDC), pp. 476–481. IEEE (2021) 12. Polack-Wahl, J.A.: Work in progress—Using podcasting in engineering education. In: 2010 IEEE Frontiers in Education Conference (FIE), pp. F4C-1. IEEE (2010) 13. Perez, E.N., Vasquez, N.R., Roldan, R.E.V.: Integrating different teaching methods as a strategy to improve the students learning: a comparison study in an e-engineering course. In 2021 International e-Engineering Education Services Conference (e-Engineering), pp. 115–120. IEEE (2021) 14. Moura, A., Carvalho, A.A.: Mobile learning: teaching and learning with mobile phones and Podcasts. In: 2008 Eighth IEEE International Conference on Advanced Learning Technologies, pp. 631–633. IEEE (2008) 15. Husein, S., Saive, R., Jordan, M., Bertoni, M.I.: Podcasts: an under-utilized form of science communication. In: 2019 IEEE 46th Photovoltaic Specialists Conference (PVSC), pp. 2464– 2466. IEEE (2019) 16. Caratozzolo, P., Lara-Prieto, V., Hosseini, S., Membrillo-Hernández, J.: The use of video essays and podcasts to enhance creativity and critical thinking in engineering. Int. J. Interact. Des. Manuf. (IJIDeM) 16(3), 1231–1251 (2022)

Using a Podcast to Foster Success Among Computer Science Students

483

17. Conzemius, A., O’Neill, J.: The power of SMART goals: Using Goals to Improve Student Learning. Solution Tree Press (2009) 18. Morrison, M.: History of SMART objectives. Rapid Business Improvement (2022). http://rap idbi.com/management/history-of-smart-objectives/. Accessed 09 Mar 2022 19. Drew, C.: Edutaining audio: an exploration of education podcast design possibilities. Educ. Media Int. 54(1), 48–62 (2017) 20. Fernandez, V., Sallan, J., Simo, P.: Past, present, and future of podcasting in higher education. In: Li, M., Zhao, Y. (eds.) Exploring learning and teaching in higher education, pp. 305–330. Springer, Berlin (2015) 21. Carvalho, A., Aguilar, C., Carvalho, C., Cabecinhas, R.: Influence of podcasts characteristics on higher students’ acceptance. In: Bonk, C. (ed.) Proceedings of world Conference on elearning in Corporate, Government, Healthcare, and Higher Education 2008, pp. 3625–3633. Association for the Advancement of Computing in Education, Chesapeake (2009)

Research on Mental Health Training for Pre-service Teachers to Address Pupil Issues in Schools: An International Virtual Learning Experience Barbara Schwartz-Bechet

Abstract According to the World Health Organization (WHO), worldwide 10%20% of children and adolescents experience mental disorders. Half of all mental illnesses begin by the age of 14 and three-quarters by mid-20 s. According to the most recent analysis of 2016 National Survey of Children’s Health data published online in Journal of the American Medical Association (JAMA) Pediatrics indicated that as many as one in six U.S. children between the ages of 6 and 17 has a treatable mental health disorder such as depression, anxiety problems or attention deficit/hyperactivity disorder (ADHD). And in Canada, it is estimated that 10-to20% of children and youth in Canada experience mental illness and that only one in five children and youth who need mental health services receives them. (Alberta Health Services: Mental Health Capacity Building; http://www.albertahealthservi ces.ca/amh/Page2754.aspx). And, with the factors that may affect mental health of children and youth across the globe today, interrupted instruction, learning & accessing new platforms for learning, more social unrest, uptick in domestic violence, etc., there is now more need than ever to ensure that our soon to be educators are properly prepared to understand and reduce the stigma of mental health/illness with their pupils. A qualitative/quasi quantitative research-based study designed to examine the perspective of those who are currently instructing preservice teacher candidates across three countries in three institutions as well as the perspective of pre service teachers was conducted primarily virtually across three institutions in three countries. Keywords Mental Health · Virtual Learning Study · Mental Health Perspectives

B. Schwartz-Bechet (B) Misericordia University, Dallas, PA 18612, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_39

485

486

B. Schwartz-Bechet

1 Rationale 1.1 Introduction The aim of the current study was to assess if there is a need for greater instruction and understanding of mental health literacy and mental health understandings of pre-service teacher candidates in teacher education programs in three different countries. While it is true that there has been an increasing percentage of cases of children demonstrating mental health issues at a younger age — previously, most research has been conducted on adolescents — it has been suggested and there has been preliminary research indicating that the COVID pandemic exacerbated mental health issues. If this is the case, and it seemed to be based in some early studies to be true, it was an appropriate time to determine if teacher preparation programs were preparing their students to meet the needs of pupils in primary schools. Additionally, as many teacher education programs needed to go online during the COVID pandemic, it seemed that online instruction could be a variable in their own understandings and knowledge or the faculty ability to provide successful instruction effectively in meeting this possibly new need. Teacher education students were asked the same questions in each country within a survey to determine their knowledge on mental health literacy as well as on their perceptions of readiness to work with students in primary schools who have mental health concerns. Faculty members were asked if they felt that there students were prepared and also if their students were provided information specifically in mental health literacy in their programs of study. All research was conducted virtually.

1.2 Rationale for Questions in the Research Globally, teacher education programs often provide at least one course on special education topics within their curriculum if the program is in childhood education, without a specialization. However, there is little to no requirement that preservice teachers be taught about or have an understanding of mental health and/or mental illness in youth. As indicated above, one in five/six children in Canada and the US, respectively, will most likely be in need of some form of mental health services between the ages of 6–17. This is supported by the work of Brown, Phillippo, Weston, and Rodgers [3], who found that there were no specific mental health guidelines or preparation standards for teachers in most US states or provinces in Canada. Similarly, the Netherlands has numbers that match the United States and Canada in terms of cases. While the United States and Canada have much larger populations, it is more essential that the Netherlands also actively work to address this issue. It cannot be emphasized enough that it is imperative to educate those who work with children about mental health to prevent negative attitudes and to increase and

Research on Mental Health Training for Pre-service Teachers to Address …

487

Table 1 Questions asked of faculty in teacher education programs

Question 1:

Question 2: Question 3: Question 4:

Question 5:

Do you feel that teacher candidates, primarily of those who will teach young children, have enough knowledge to support the mental health of students? Are they prepared to support students? Did the students receive sufficient education to support mental health? Would students have sufficient time to support mental health of students once they are in the classroom teaching? Is there enough privacy to support an individual students' mental health in the classroom?

develop a greater understanding of mental health, to reduce stigma and discrimination. Having conducted research with colleagues in the Netherlands for over ten years on increasing global understandings in teacher education instruction and the incorporation of Universal Design for Learning, it appears it would be beneficial to include their teacher education program in this project to enable greater mutual collaboration and understandings across countries and people. Inviting a third country to work together supports the mission of Fulbright to bring understanding and mutual impact to those around the world. Working with an additional partner allowed for virtual dialogue throughout the program of study and on working collaboratively to design and implement a strategy following the results of the study. Working with colleagues in the Netherlands allowed for work to begin on this project virtually in September 2021, prior to the face to face work in Canada in January 2022 (Tables 1 and 2).

2 Project Description The project sought to determine if pre-service teacher candidates have been prepared and have knowledge of mental health literacy to determine if there may be a need to correct curricula and/or training in teacher education programs so as to be reflective of the current global issues related to youth mental health. All preservice teacher candidates involved in the project from each of the three institutions, University of Calgary, in Alberta, Canada, NHL/Stenden University of Applied Science in Leeuwarden, Netherlands, and Misericordia University, in Dallas, Pennsylvania, USA, were selected through purposeful sampling. The same holds true for the faculty members who were interviewed. Three variables across all teacher education programs were evaluated in order to assess mental health literacy of candidates, curricula, and programming:

488

B. Schwartz-Bechet

Table 2 Questions sent to students in the three countries

Question 1:

Question 2:

Question 3:

Question 4:

Question 5:

Question 6:

Question 7:

I have sufficient knowledge to support the mental health of my students (if not, what do you feel that you need to better help your students?) I am confident I can support student mental health (if not, why not and what would boost your confidence?) I feel prepared to support the mental health of my students (if so, how do you feel that you can support their mental health? If not, how could you be better prepared?) I have sufficient training and education to support student mental health (if not, what additional education do you believe would help you?) I feel I know how to competently support the mental health of my students (if not, what might you do to better feel that competent in supporting your students?) I will have sufficient time to support the mental health of my students (However you answer, why do you feel this way?) I will have sufficient privacy to support student mental health (do you believe that this is going to be true and if not, what might you do to rectify the situation?)

• Pre-service teacher candidates completed a survey based on the Mental Health Literacy Scale in English and took part in a structured interview questions as part of the survey. • Data collection of mental health and mental illness language use in teacher education curriculum and instructional materials such as handbooks in each university program. A utilization of appropriate terminology was guided from journals that focused on mental health, such as Advances in School Mental Health, to provide a thorough review of number of assignments, readings, etc. on the topic of mental health in all required courses. • Virtual interview questions were provided to faculty members purposely identified across the three institutions to survey their perceptions of mental health literacy instructional need and/or provision of instruction in mental health to preservice teacher candidates.

Research on Mental Health Training for Pre-service Teachers to Address …

489

3 Methods 3.1 Purposeful Sampling The source population was undergraduate education students. Criteria for inclusion: (a) about to begin practicum or student teaching undergraduate students; (b) aged 18 years or older. The size of the sample was determined by the practicalities of working with three universities which required delivering the intervention during a semester at a time convenient to university scheduling requirements and field placements for the students. It was best to recruit a minimum of 50 students in order to detect 5% level of significance with 90% power in order to detect medium effect sizes in the outcome variables. A total of 63 students was attempted to be recruited to account for 20% attrition. A total of 62 students responded across all three institutions. Purposeful, typical case sampling was used so as to narrow the range of variation and focus on similarities. This is similar to the use of quantitative central tendency measures (e.g., mean, median, and mode). Additionally, the faculty from each institution were sent emails with the same questions related to what they teach in the area of mental health literacy and preparation and then questions on their perceptions. The faculty were selected based upon the courses which they taught to the students in a particular major. All were told that a response to the emailed survey questions was not mandatory to complete, similarly to what the students were told regarding responses.

3.2 Participants The total n for responding students was 62 with 40 from the US, 20 from Canada and 2 from the Netherlands. All were preservice teacher education candidates. The total n for responding faculty who teach in the teacher education programs at the three institutions was 10, with 4 from the US, 5 from Canada, and 1 from the Netherlands. The number for faculty who responded to survey questions via email was a total of 10 with 6 respondents from the US, 3 from Canada and one from the Netherlands.

3.3 Ethics and Consent The study went through the IRB at both Misericordia University in the US and at the University of Calgary in Canada. Participants were first presented with an email outlining the purpose of the study which included exploring pre-service teachers’ attitudes towards children’s mental health literacy and their knowledge of mental health. The email also included the length of the study stating the survey, which was

490

B. Schwartz-Bechet

voluntary, would take between 5 and 10 min to complete. Once participants read the information, they clicked the survey link and were directed to the SurveyMonkey survey platform, sent through Misericordia University. Ethics approval was obtained through the University of Calgary and Misericordia University.

3.4 Study Design and Procedures A mixed methods study was utilized. The inclusion of qualitative methods was used to explore and obtain a greater depth of understanding related to the mental health literacy of pre-service teachers and to possibly identify strategies for facilitating implementation or change to programming for future pre-service teachers. The use of quantitative methods was used to test the hypotheses to determine if there was a quantity of evidence present to then be able to obtain an understanding of predictors of successful implementation based on identified trends. Purposeful, typical case sampling was used so as to narrow the range of variation and focus on similarities. This is similar to the use of quantitative central tendency measures (e.g., mean, median, and mode). An email was sent to students from the three institutions of higher education. with a description of the study and a survey link. Faculty members from the pre-service teacher education programs for primary school students were sent an email with an introductory letter and a separate page of questions regarding perceptions on the mental health knowledge taught in the programs of study to the pre-service teacher candidates as well whether they thought that those students were prepared to meet the challenges of pupils with mental health needs in the primary classroom upon graduation. The three sets of data were analyzed through triangulation to determine if there was any knowledge on the topic mental health and illness by pre-service candidates through teacher education program preparation, prior knowledge, and engagement in their practicum/student teaching placements.

3.5 Data Analysis A General Inductive Approach (GIA) proposed by Thomas [32] is a systematic set of procedures used to analyze qualitative data. This approach was used in the study study to condense the data provided by participants’ response to the survey questions in a straightforward, reliable and valid manner. The GIA was followed by reading the raw data from each question and clustering it into themes. The overall theme for each question was summarized, with quotes from the raw data in italics.

Research on Mental Health Training for Pre-service Teachers to Address …

491

4 Results The outcomes of the study began to shed light on whether pre-service teachers have knowledge of mental health literacy prior to attending a field placement. It also uncovered a general lack of purposeful instruction in mental health literacy across program instruction and materials within the three countries. The last expected out outcome identified a disconnect between how the students perceive their preparation and how the faculty perceive their preparation for the students. It was this last outcome that may have been impacted most heavily on this research occurring through online zoom meetings and online surveys. The results indicated that 52% of students from across all countries did not feel that they had sufficient knowledge to meet the needs children with mental health concerns and 56% felt that they could support the needs of the children. However, 61% felt that they were not provided sufficient information from their courses. 82% felt that they would not have time to meet with those students who needed help. The results of the review of the syllabi and handbooks provided to the students had less than 1% across all manuscripts of mental health verbiage anywhere in the materials unless in a course that specifically addressed mental health. There was one course in the US which students were required to take for one credit. There was one module in a health course that included discussion of mental health in a course in Canada. The of the interviews conducted with faculty via zoom or survey indicated that that those in Canada did not feel that there was a need to include information on mental health. The Canadian faculty who did feel it was needed were those in the educational psych program and not those who were teaching teacher education courses specifically. Most teacher education faculty in Canada did not want to take part in any interviews either on zoom or through a survey. US teacher education faculty were most happy to utilize a survey to respond to questions as was the one faculty member in the Netherlands (Table 3).

5 Discussion Overall, the results indicated that the students felt moderately prepared overall but could use more specific training and also did not truly see how they would have time in a class day to address issues. It appears the faculty members in Canada had the most difficulty accepting that there could be a need for greater instruction on mental health for the students to feel more prepared. It appears that the likelihood of greater preparation in mental health literacy would be more likely to occur in the US and perhaps the Netherlands. However, the sample size in the Netherlands is too small to draw any conclusions. The medium used to conduct this research may have had an effect on the faculty members asked to take part. There were no issues with any of the students who chose to take part, where they also provided individual feedback in boxes on the survey.

492

B. Schwartz-Bechet

Table 3 Themes and examples of comments relating to each question Question

Themes

Examples of Comments

1. How important is children’s mental health to you?

Children’s mental health is important, in Dallas, PA, Calgary, Alberta, Canada & Leeuwarden, Netherlands

“Extremely important”, “The most important”

2. I feel I know how to Most feel competent; Many “…Will need to seek other competently support the feel unsure of meeting all people or resources in order to mental health of my students mental health needs that may support the student(s) arise 3. I have sufficient training and Mixed responses to sufficient education to support student training – more positive in mental health the US and less so in Canada and the Netherlands

“I think a training course or class would be helpful” “An online training module would be helpful”

4. I will have sufficient time to support the mental health of my students

“It is hard to have sufficient time for all students” “Mental health is a priority, I will make time”

They will not have enough time to work with students

However, faculty in Canada, where they actually had the option to meet with the researcher face to face in March and April of 2022, mainly declined to be involved. The rationale for most was that they did not have time due to needing to teach virtually due to the pandemic or that they were too burned out. The faculty member in the Netherlands also agreed that there was a need to provide greater instruction in mental health literacy and admitted never recognizing the need until the pandemic occurred. The faculty member in the Netherlands was comfortable with virtual learning and had no issue with the online survey. Several faculty members in the US willingly took part and agreed that more training is necessary, but felt that there was not enough time to add another course to their programs. The faculty members in the US were also comfortable with the use of online platforms and surveys and teaching virtually. Another variable may be that the individual platforms used for instruction at the three universities could have caused some issues. The speculation regarding the platforms as an issue may be in that both the US and the institution in the Netherlands had been using several online platforms prior to the pandemic while the Canadian institution had not had their students nor their faculty involved in hybrid or online learning prior to the COVID. Moving forward, it might be worthwhile to compare the different platforms of instruction for implementation and outcomes across three countries, Canada, the Netherlands, and the United States, as well as the amount of training the faulty members and students had on the use of the LMS platforms and types of online surveys. Acknowledgements Canada and US Fulbright provided support for the research conducted with the University of Calgary faculty and students, in relation to work presented in this paper.

Research on Mental Health Training for Pre-service Teachers to Address …

493

References 1. Atladottir, H.O., et al.: The increasing prevalence of reported diagnoses of childhood psychiatric disorders: a descriptive multinational comparison. Eur. J. Child Adolesc. Psychiatry 24, 173– 183 (2015). https://doi.org/10.1007/s00787-014-0553-8 2. Bassuk, E.L., Richard, M.K., Tsertsvadze, A.: The prevalence of mental illness in homeless children: A systematic review and meta-analysis. J. Am. Acad. Child Adolesc. Psychiatry 54(2), 86–96 (2015). https://doi.org/10.1016/j.jaac.2014.11.008 3. Brown, E.L., Phillippo, K.L., Weston, K., Rodger, S.: United States and Canada pre-service teacher certification standards for student mental health: a comparative case study. Teach. Teach. Educ. 80, 71–82 (2019). https://doi.org/10.1016/j.tate.2018.12015 4. Brown, N.M., Green, J.C., Desai, M.M., Weitzman, C.C., Rosenthal, M.S.: Need and unmet need for care coordination among children with mental health conditions. Pediatrics 133(3), e530–e537 (2014). https://doi.org/10.1542/peds.2013-2590 5. Centers for Disease Control and Prevention: Web-based injury statistics query and reporting system (WISQARS). Atlanta, GA (2015). http://www.cdc.gov/injury/wisqars/index.html 6. Conley, C.S., Shapiro, J.B., Kirsch, A.C., Durlak, J.A.: A meta-analysis of indicated mental health prevention programs for at-risk higher education students. J. Couns. Psychol. 64(2), 121–140 (2017). https://doi.org/10.1037/cou0000190 7. Csillag, C., et al.: Early intervention services in psychosis: from evidence to wide implementation. Early Interv. Psychiatry 10(6), 540–546 (2016). https://doi.org/10.1111/eip.12279 8. DeLuca, J.S.: Conceptualizing adolescent mental illness stigma: Youth stigma development & stigma reduction programs. Adolescent Res. Rev. (2018).https://doi.org/10.1007/s40894-0180106-3 9. Desocio, J., Stember, L., Schrinsky, J.: Teaching children about mental health and illness: a school nurse health education program. J. School Nurs. Official Publ. Natl. Assoc. School Nurs. 22(2), 81–86 (2006) 10. Freeman, E.V.: School mental health sustainability: Funding strategies: Series 1–4. Washington, DC, Technical Assistance Partnership for Child and Family Mental Health (2011). http://www. tapartnership.org/content/education/default.php 11. Hamilton, P., Danby, G.: Addressing the ‘elephant in the room’, The role of the primary school practitioner in supporting children’s mental well-being. Pastor. Care Educ. 24(2), 90–103 (2016). https://doi.org/10.1080/02643944.2016.1167110 12. Harding, S., Morris, R., Gunnel, D., Ford, T., Hollingworth, W., et al.: Is teacher mental health and wellbeing associated with student mental health and wellbeing? J. Affect. Disord. 253, 460–466 (2019) 13. Hinz, A., Zenger, M., Brähler, E., Spitzer, S., Scheuch, K., Seibt, R.: Effort-reward imbalance and mental health problems in 1074 German teachers, compared with those in the general population: effort-reward imbalance in teachers. Stress Health 32(3), 224–230 (2014). https:// doi.org/10.1002/smi.2596 14. Jones, D.J., et al.: A review of the key considerations in mental health services research: a focus on low-income children and families. Couple Family Psychol. Res. Pract. 5(4), 240–257 (2016). https://doi.org/10.1037/cfp0000069 15. Kutcher, S., Wei, Y., Coniglio, C.: Mental health literacy: past, present, & future. The Canadian J. Psychiatry 61(13), 154–158 (2016) 16. Kidger, J., Brockman, R., Tilling, K., Campbell, R., Ford, A., Araya, R., et al.: Teachers’ wellbeing and depressive symptoms, and associated risk factors: a large cross sectional study in English secondary schools. J. Affect. Disord. 192, 76–82 (2016) 17. Link, B.G., DuPont-Reyes, M.J., Barkin, K., Villatoro, A.P., Phelan, J.C., Painter, K.: A schoolbased intervention for mental illness stigma: a cluster randomized trial. Pediatrics 145(6), e20190780 (2020). https://doi.org/10.1542/peds.2019-0780 18. Luby, J., et al.: The effects of poverty on childhood brain development: the mediating effect of caregiving and stressful life events. JAMA Pediatr. 167(12), 1135–1142 (2013). https://doi. org/10.1001/jamapediatrics.2013.3139

494

B. Schwartz-Bechet

19. McLaughlin, K.A., Green, J.G., Gruber, M.J., Sampson, N.A., Zaslavsky, A.M., Kessler, R.C.: Childhood adversities and adult psychiatric disorders in the National Comorbidity Survey Replication II: associations with persistence of DSM-IV disorders. Arch. Gen. Psychiatry 67(2), 124–132 (2010). https://doi.org/10.1001/archgenpsychiatry.2009.187 20. New York University & McSilver Institute for Poverty Policy and Research: Article brief: Mental Health and Poverty. New York University Silver School of Social Work (2017). http:// mcsilver.nyu.edu/sites/default/files/reports/Mental_Health_and_Poverty_one-sheet.pdf 21. Odgers, C.L.: Income inequality and the developing child: is it all relative? Am. Psychol. 70(8), 722–731 (2015). https://doi.org/10.1037/a0039836 22. Olfson, M., Druss, B.G., Marcus, S.C.: Trends in mental health care among children and adolescents. N. Engl. J. Med. 372, 2029–2038 (2015). https://doi.org/10.1056/NEJMsa141 3512 23. Osher, D., Cantor, P., Berg, J., Steyer, L., Rose, T.: Malleability, plasticity, and individuality: How children learn and develop in context. [manuscript under review]. American Institutes for Research, Washington, DC (2017) 24. Perou, R., et al.: Mental health surveillance among children—United States, 2005–2011. Morbidity Mortality Weekly Rep. 62, 1–35 (2013). http://www.cdc.gov/mmwr/preview/mmw rhtml/su6202a1.htm?s_cid=su6202a1_w 25. Pinto-Foltz, M.D., Logsdon, M.C., Myers, J.A.: Feasibility, acceptability, and initial efficacy of a knowledge-contact program to reduce mental illness stigma and improve mental health literacy in adolescents. Soc. Sci. Med. (1982), 72(12), 2011–2019 (2011). https://doi.org/10. 1016/j.socscimed.2011.04.006 26. Reinke, W.M., Stormont, M., Herman, K.C., Puri, R., Goel, N.: Supporting children’s mental health in schools: teacher perceptions of needs, roles, barriers. Sch. Psychol. Q. 26(1), 1–13 (2011). https://doi.org/10.1037/a0022714 27. Salerno, J.P.: Effectiveness of universal school-based mental health awareness programs among youth in the United States: a systematic review. J. School Health 86(12), 922–931 (2016). https://doi.org/10.1111/josh.12461 28. Schonfeld, I.S., Bianchi, R., Luehring-Jones, P.: Consequences of job stress for the mental health of teachers. In: McIntyre, T.M., McIntyre, S.E., Francis, D.J. (eds.) Educator Stress, pp. 55–75. Springer International Publishing, Cham (2017). https://doi.org/10.1007/978-3319-53053-6_3 29. Simon, A.E., Pastor, P.N., Reuben, C.A., Huang, L.N., Goldstrom, I.D.: Use of mental health services by children ages six to 11 with emotional or behavioral difficulties. Psychiatr. Serv. 66(9), 930–937 (2015). https://doi.org/10.1176/appi.ps.201400342 30. Stockings, E.A., Degenhardt, L., Dobbins, T., Lee, Y.Y., Erskine, H.E., Whiteford, H.A., Patton, G.: Preventing depression and anxiety in young people: a review of the joint efficacy of universal, selective and indicated prevention. Psychol. Med. 46(1), 11–26 (2016). https://doi. org/10.1017/S0033291715001725 31. Rockville, M.D.: Substance Abuse and Mental Health Services Administration (SAMHSA). Results from the 2014 National Survey on Drug Use and Health: Mental health detailed tables (2017a). https://www.samhsa.gov/data/sites/default/files/NSDUH-MHDetTabs 2014/NSDUH-MHDetTabs2014.htm#tab2-1a 32. Thomas, D.R.: A general inductive approach for analyzing qualitative evaluation data. Am. J. Eval. 27(2), 237–246 (2006). https://doi.org/10.1177/1098214005283748 33. Wahl, O.F., Susin, J., Kaplan, L., Lax, A., Zatina, D.: Changing knowledge and attitudes with a middle school mental health education curriculum. Stigma Res. Action 1(1), 44–53 (2011). https://doi.org/10.5463/sra.v1i1.17 34. Wahl, O., Rothman, J., Brister, T., Thompson, C.: Changing student attitudes about mental health conditions: NAMI ending the silence. Stigma Health 4(2), 188–195 (2019). https://doi. org/10.1037/sah0000135 35. Mental Health Literacy & Mental Health of Pre-Teachers/Teachers

Towards Inclusive Excellence: A Case Study in Engineering Hong Shaddy

Abstract The United States is projected to shift to a majority-minority country by 2050. The 2020 census shows rapid growth of the minority population, driving a diversity explosion according to noted demographer William Frey. College enrollment is reflecting the demographic trend with an increasingly diverse student population. Learners from different backgrounds and experience bring varied educational needs as well as strengths. This paper presents how an online graduate engineering course implemented strategies to meet different needs and harnessed diverse strengths to promote learner engagement and interaction towards inclusive excellence. The case study confirms the imperative to align instructional objectives with professional discipline outcomes and societal goals to be more effective in advancing student success as well as social cohesion. Keywords Inclusive education · Learner engagement and interaction · Higher education · STEM education

1 Introduction The United States is projected to shift to a majority-minority country by 2050 [9]. The 2020 Census shows more than 40% of Americans self-identify as people of color, from one or more racial or ethnic groups, accelerating a “diversity explosion” [8, pp.16–36; 10]. What are the implications of such a diversity explosion? How should we respond in higher education? Demographic transformation presents challenges as well as opportunities. Diverse learners bring different educational needs as well as assets. It’s critical to meet those needs and leverage the assets for student success and social cohesion. This paper presents a case study of an online graduate engineering course that implemented inclusive pedagogies to foster students’ academic and social growth. H. Shaddy (B) Johns Hopkins University, Baltimore, MD 21218, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_40

495

496

H. Shaddy

The course leveraged learner strengths arisen from diverse personal, academic, and professional backgrounds and experience to promote engagement and interaction towards inclusive excellence. It reviews relevant literature and discusses the findings as well as the impact of the intentional measures on student learning.

2 Context This section reviews relevant literature to set the context for the case study. The United Nations Sustainable Development Goal 4 highlights the importance of an inclusive and equitable quality education for all [22]. Inclusion is defined as a “process that helps overcome barriers limiting the presence, participation, and achievement of learners” whereas equity means “ensuring that there is a concern with fairness, such as the education of all learners is seen as being of equal importance” [23]. The goal of inclusive and equitable education is guided by various frameworks in practice. Among them is Universal Design for Learning (UDL), a set of widely adopted educational principles. UDL advocates the design of inclusive environments that can accommodate individuals with learning differences “to ensure that all learners can access and participate in meaningful, challenging learning opportunities” [6]. Meeting the needs of diverse learners requires a different approach to assessment, a critical component in the instructional process, Montenegro and Jankowski [15, 16] posit. The authors call for an assessment method that is equity-minded and attends to an increasingly diverse and global student population. An equity-minded approach centers on meaningful student involvement and transparency in the process. It provides students with “what is being assessed, how it is being assessed, and how well they achieved the assessment’s expectations” [16, p. 10]. Such an approach can reduce systemic inequities in education to “[bring] students to the same starting line with equal understanding and resources before the learning begins” [24, p. 167]. Equityminded assessment is “ultimately about being responsive, aware, and intentional in order to not perpetuate inequalities” [16, p.16]. Learner differences have been approached through different lenses. Diversity arising from individuals with different backgrounds can be a resource. Gurin et al. [12] explore the relationship between diversity and students’ academic and social growth in colleges and universities. The authors underscore the importance of interaction among students from diverse backgrounds in and out of college classrooms. Such interactions are crucial in fostering active thinking, perspective-taking, racial and cultural understanding, and other skills for achieving academic success and leading a heterogeneous and complex society. “Both the theory and findings indicate that individual students benefit when they are engaged with diverse peers” [12, p. 362].

Towards Inclusive Excellence: A Case Study in Engineering

497

3 Strategies How do you sustain active learner engagement and student success when a course is migrating to fully online? That was the familiar challenge facing a graduate engineering elective at a private institution in the United States in summer 2021. The course was launched in the Virtual Live modality (synchronous online) in the previous year. The change of instructional modality was used as an opportunity to re-assess strategies to fulfill course objectives and professional discipline outcomes as well as to connect with the broader dialogue on inclusive education within and beyond higher education.

3.1 Cultivating an Inclusive Online Learning Environment This course adopted a broad definition of inclusive education [3, p. 162; 4] as developing environments where all learners can flourish. Steps were taken to proactively reduce barriers and increase access and participation. Text alternatives were added to non-text items in course materials. All lecture videos were professionally captioned (with 99% fidelity) as part of the accessibility initiative. To accommodate student work schedules and time zone differences, extra office hours were added. Course modules were released all at once to support self-regulation and provide students with the flexibility to plan and work ahead. The course activities were designed to facilitate learner participation and engagement. The discussions, projects and assignments provided explicit encouragement, detailed instructions, and formal structures for learner to content, learner to instructor, and learner to learner interaction [17]. Quizzes were set up to allow multiple attempts. Projects were structured with multiple deliverables, which were spread out through the semester to allow time for feedback and modification. The continuous improvement process took into consideration diverse backgrounds including different levels of prior knowledge and motivation. It aimed to provide equitable learning opportunities and promote mastery of the course materials through practice, feedback, reflection, and revision.

3.2 Applying Transparency for Learner Empowerment The course attended to transparency as an important way to empower the students on the purpose of their learning and how to succeed. Each module started with specific, measurable learning objectives to convey expectations and set the alignment for the module activities and assessments. The discussion forums were graded with a rubric that explicitly encouraged and rewarded early participation, engaging with others as

498

H. Shaddy

well as linking course concepts to big ideas. Prior student work was show-cased to provide an example of excellence for the individual case study project. The group project was modeled on transparent assignments [21]. It is a common practice for assignments to list what students are required to do. A transparency assignment goes beyond and provides the purpose, task, and criteria for success. The group project was structured with an overview, learning objectives, instruction on how to get started and grouping instructions, multiple deliverables and their timeline, a grading rubric, etc. The project made clear the assignment goals and expectations, skills that can be applied in the real world, and the evaluation criteria in addition to specifying the requirements.

3.3 Leveraging Learner Differences for Engagement and Interaction Students in the course consisted of engineering professionals from different cultures and ethnicity (White, Hispanic, Asian, Non-resident Alien, and unidentified Race and Ethnicity), age group (20 s to 50 s), and discipline (computer science, electrical, mechanical, systems, space systems engineering, etc.). They each brought different perspectives, work experience and varied levels of prior knowledge. A core strategy focused on leveraging the strengths from such diversity to promote learner engagement and interaction to maximize learning for all. IMDB Style Peer Review. The course sought to enhance student engagement and collaborative learning via peer assessment. Previous research shows students perceive peer assessment beneficial to their learning through effective feedback, a supportive learning environment, and collaboration among learners [18]. A peer review component was added to the individual case study assignment where students were instructed to assess others’ project presentations. To ensure active participation from all students and promote critical thinking, a structured approach was developed, Internet Movie Database (IMDb) style. Students were to choose four of their peers’ work to critique as if they were movie critics. They were to summarize the topics presented, explain what others would learn, and share their thoughts for improvements. Discussion Leaderboard Experiment. In addition to common and best practices, the course turned to research and innovation. It used discussions as a major way to nurture an inclusive community and pro-mote exchange of diverse viewpoints and ideas. However, it can be challenging to make online discussion work effectively, often due to low student participation and engagement [2]. The challenge can be greater for an engineering course which, unlike those in some other disciplines, often does not lend itself naturally to discussions. A student-facing discussion leaderboard was piloted as an intervention. With two sections of the course offered for the first time in Summer 2021, a controlled experiment was conducted. A leaderboard was

Towards Inclusive Excellence: A Case Study in Engineering

499

implemented in Sect. 1 and Sect. 2 was used as the control group. The leaderboard utilized existing course analytics from the learning management system and was placed on the course home page. It displayed the top five participants in each module’s discussion forums. The hypothesis was that course learning analytics, when brought from behind the scenes to the student-facing front end, can impact learner motivation and engagement.

4 Findings 4.1 Course Evaluation Course evaluations use 5-point Likert scale questions (Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree). Student Interaction. Question G: My interaction with the other students in the course contributed to my learning. (Favorable = Strongly Agree or Agree; Neutral = Neutral; Unfavorable = Disagree or Strongly Disagree). Table 1 Course Evaluation Score for Question G on Student Interaction unfavorable

neutral

favorable

Summer 2020 (N = 15)

0%

13%

86%

Summer 2021 Section 1 (N = 14)

7%

0%

93%

Summer 2021 Section 2 (N = 11)

0%

0%

100%

Select Student Feedback. The following comments are from the Summer 2021 end of semester student evaluation, ranging from feedback on module discussions, module release, group projects, to real-life relevance of the course materials, etc. • The best parts about the course [were] the focus on student discussion. It was interesting seeing those with different backgrounds give answers regarding hardware trends. • I like the discussion modules the best. • Very interesting course, great discussions, reasonable [workload]. • … I also enjoyed the group and individual projects as it helped facilitate good discussion and research. • All modules were available from day 1, which provided flexibility for working professionals. • … I liked that all the mods were open day one, it help[ed] to be flexible with my work schedule.

500

H. Shaddy

• Lot of relevant and interesting material. Good real-life examples. • The module topics were logically organized, very relevant, and provided the opportunity for learning in areas directly applicable to my work. • I thoroughly enjoyed how practical this course was. … [It’s] been the most useful (and fun) class I’ve taken in the … program so far. • The course overall was practical … I learned more in this course than I have in almost any other course.

4.2 Discussion Leader board Experiment

Fig. 1 Module 3 Discussion Leaderboard (Summer 2021 Section 1) (Student names are blurred in the image)

User Activity in Discussion Forums and Overall Course Activity Table 2 Summer 2020 vs. Summer 2021 Section 1 vs. Summer 2021 Section 2 Control Group (SU20)

Control Group (SU21 Sec 2)

Study Group (SU21 Sec 1)

Class size

18

12

15

Number of online modules

12

12

12

Number of modules with discussion forums

12

12

12

Average number of posts by all participants

44.33

47.75

62.13

Course activities in hours (average)

129.04

127.78

264.49

5 Discussion Self-reports, behavioral measures [14] from the course evaluation and learning analytics as well as student performance show the achievement of the expected outcomes of learner engagement, interaction, and inclusive excellence. Comparing user

Towards Inclusive Excellence: A Case Study in Engineering

501

activity in discussion forums, the findings reveal a 30% increase by the study group over the control group from the same semester and 40% increase from the previous year (Table 2). Comparing overall course activity in terms of time spent within the course site, the results show a 107% increase by the study group over the control group from the same semester and 105% increase from the prior year (Table 2). An average of 96.5% of respondents in summer 2021 reported that their interaction with the other students contributed to their learning, compared to 86% the prior year (Table 1). Significantly, student participation in online discussions exceeded those when the course was offered synchronously the year before. “Learning is collaborative with meaning negotiated from multiple perspectives” [20, p. 129]. The course discussions valued individual differences and made it relevant to the learning process. The students, each bringing their unique perspectives and real-world experiences, expanded the shared knowledge base and enriched learning for all. Instead of low participation, a common challenge facing online discussions, students remarked in the course evaluation that discussions were the best parts, and “it was interesting seeing those with different backgrounds give answers regarding hardware trends.” Engaging with diverse perspectives not only enhances the students’ learning experience but also prepares them for professional success. ABET (Accreditation Board for Engineering and Technology) student outcomes expect engineering graduates to demonstrate abilities to lead in a collaborative and inclusive workplace environment [1]. The workplace is increasingly diverse and relies on collaborative efforts by individuals from different backgrounds for innovative problem-solving. Diverse teams are better positioned to tackle complex problems facing today’s world. Each member thinks differently. They bring cognitive diversity, which can produce “the diversity bonus,” which, in turn, can lead to creative solutions and faster rates of innovation [19, p. 216]. Previous literature points to positive educational, social, economic benefits of diversity [5, 7, 12, 13, 19]). The value of diversity, however, does not materialize on its own. Diversity needs to be approached as an institutional resource and leveraged in the curriculum and other learning venues so that students from different backgrounds actually interact with each other to realize educational benefits [11]. Group interaction does not flow automatically. It happens through planned activities and intentionally designed settings that encourage peer to peer engagement. The discussion forums, the peer review assignment, and the group project in this case study were examples of such planned activities. Quality peer interactions are particularly important in online learning. They afford the opportunities for crossgroup contact and social engagement for students, who otherwise would be largely isolated in an online environment. The intentionally designed activities exposed the students to different ways of thinking by people from different backgrounds, both personal and professional. They motivated intellectual engagement, facilitated understanding of individual differences and strengths, and promoted interpersonal connections. The structured activities contributed to realizing the educational and social benefits of interacting with diverse peers.

502

H. Shaddy

College enrollment is reflecting the demographic trend with an increasingly diverse student population. Learners from different cultural, linguistic, academic, or professional backgrounds may approach things differently. It takes more explicit communication to convey the expectations. The group project attended to such differences. Structured with purpose, requirements, and evaluation criteria, this authentic project provided transparency on the how and why as well as the what of learning. Transparency alleviates ambiguity, a potential barrier some students may face otherwise. It makes learning more purposeful and meaningful when connecting the classroom, regardless of modality, to the real world. Transparency empowers learner motivation and success, contributing to closing achievement gaps.

6 Conclusion This paper shows how a graduate engineering course cultivated student engagement and interaction towards an inclusive learning experience. The course strategies focused on the learning environment, student empowerment, and harnessing diverse learner strengthens via innovative interventions. The case study contributes to the discourse and practice of inclusive education, often narrowly linked to breaking physical or developmental barriers, which are crucial to address. In addition to overcoming barriers this course attended to learner assets stemming from different academic, personal and professional backgrounds, and experience. The course harnessed such diversity to enhance students’ intellectual and social engagement. Its environment and activities were designed for inclusion as well as accessibility for every student to excel. Prior literature presents the educational, social-economic benefits of engaging with diverse peers. This case study shows how to realize the value in practice. It confirms the imperative to align instructional objectives with professional discipline outcomes and societal goals to be more effective in promoting student success and social cohesion. The case study demonstrates that different learner assets can be effectively leveraged for academic excellence and social growth, which helps shape a diverse and connected world. Acknowledgements The author acknowledges Ann Darrin, the students, and everyone else involved in the course presented in this case study.

References 1. ABET: Criteria for Accrediting Engineering Programs, 2021 – 2022. Accreditation Board for Engineering and Technology (2022). https://www.abet.org/accreditation/accreditation-criteria/ criteria-for-accrediting-engineering-programs-2021-2022/.

Towards Inclusive Excellence: A Case Study in Engineering

503

2. Aloni, M., Harrington, C.: Research based practices for improving the effectiveness of asynchronous online discussion boards. Scholarsh. Teach. Learn. Psychol. 4(4), 271–289 (2018). https://doi.org/10.1037/stl0000121 3. Armstrong, A.C., Armstrong, D., Spandagou, I.: Inclusive Education: International Policy and Practice. Sage, London (2010). https://sk.sagepub.com/books/inclusive-education 4. Best, M., Corcoran, T., Slee, R.: Who’s in? Who’s out?: What to Do about Inclusive Education. BRILL (2018). https://doi.org/10.1163/9789004391000 5. Carnevale, A., Smith, N.: The Economic Value of Diversity. Edited by Nancy Cantor and Earl Lewis. Our Compelling Interests: The Value of Diversity for Democracy and a Prosperous Society. Princeton University Press, Princeton, NJ (2016) 6. CAST. n.d.: The UDL Guidelines. CAST. https://udlguidelines.cast.org/ 7. Formanek, K.: Introduction: A New Narrative for Diversity. Beyond D&I: Leading Diversity with Purpose and Inclusiveness. Springer International Publishing,Cham (2021).https://doi. org/10.1007/978-3-030-75336-8 8. Frey, W.H.: The “diversity explosion” is America’s twenty- first- century baby boom. In: Lewis, E., Cantor, N. (eds.) Our Compelling Interests: The Value of Diversity for Democracy and a Prosperous Society, pp. 16–36. Princeton University Press (2016). https://doi.org/10.1515/978 1400881260-004 9. Frey, W.H.: Diversity Explosion: How New Racial Demographics Are Remaking America. Brookings Institution Press, Washington, D.C. (2018) 10. Frey, W.H.: New 2020 Census Results Show Increased Diversity Countering DecadeLong Declines in America’s White and Youth Populations. Brookings Institute (2021). https://www.brookings.edu/research/new-2020-census-results-show-increased-divers ity-countering-decade-long-declines-in-americas-white-and-youth-populations/ 11. Gurin, P.: Group Interactions in Building a Connected Society. Edited by Nancy Cantor and Earl Lewis. Our Compelling Interests: The Value of Diversity for Democracy and a Prosperous Society. Princeton University Press, Princeton, NJ (2016) 12. Gurin, P., Dey, E., Hurtado, S., Gurin, G.: Diversity and higher education: theory and impact on educational outcomes. Harv. Educ. Rev. 72(3), 330–367 (2002). https://doi.org/10.17763/ haer.72.3.01151786u134n051 13. Herring, C.: “Does diversity pay? Race, gender, and the business case for diversity. Am. Sociol. Assoc. 74(2), 208–224 (2009). https://doi.org/10.1177/000312240907400203 14. Jones, B.D.: Motivating and engaging students using educational technologies. In: Bishop, M.J., Boling, E., Elen, J., Svihla, V. (eds.) Handbook of Research in Educational Communications and Technology: Learning Design, pp. 9–35. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-36119-8_2 15. Montenegro, E., Jankowski, N.A.: Equity and assessment: moving towards culturally responsive assessment (Occasional Paper No. 29). National Institute for Learning Outcomes Assessments (2017). https://files.eric.ed.gov/fulltext/ED574461.pdf 16. Montenegro, E., Jankowski, N.A.: A New decade for assessment: embedding equity into assessment praxis (Occasional Paper No.42). National Institute for Learning Outcomes Assessments (2022). https://www.learningoutcomesassessment.org/wp-content/uploads/2020/01/ANew-Decade-for-Assessment.pdf 17. Moore, M.G.: Three types of interaction. Am. J. Distance Educ. 3(2), 1–7 (1989). https://doi. org/10.1080/08923648909526659 18. Ndoye, A.: Peer/self assessment and student learning. Int. J. Teach. Learn. Higher Educ. 29(2), 255–269 (2017). http://www.isetl.org/ijtlhe/ 19. Page, S.E.: The Diversity Bonus: How Great Teams Pay Off in the Knowledge Economy. Princeton University Press, Princeton (2019).https://doi.org/10.2307/j.ctvc77fcq 20. Richey, R.C., Klein, J.D., Tracey, M.W.: The Instructional Design Knowledge Base. Routledge, London (2011) 21. TILT: TILT Higher Ed Examples and Resources. TILT Higher Ed. https://tilthighered.com/til texamplesandresources

504

H. Shaddy

22. UN: THE 17 GOALS | Sustainable Development. United Nations Sustainable Development. United Nations (2015). https://sdgs.un.org/goals#goals 23. UNESCO: A Guide for Ensuring Inclusion and Equity in Education. Premium Official News. Plus Media Solutions (2017). https://unesdoc.unesco.org/ark:/48223/pf0000248254 24. Winkelmes, M.-A.: Assessment in Class Meetings: Transparency Reduces Systemic Inequities. In: Henning, G.W., Baker, G.R., Jankowski, N.A., Lundquist, AE., Montenegro, E. (eds.) Reframing Assessment to Center Equity: Theories, Models, and Practices. Sterling, Stylus, Virginia (2022)

Massive Open Online Courses at Ukrainian Agrarian Universities: To Be or not to Be Bohdan Shunevych

Abstract Many Ukrainian universities successfully use MOOCs, YouTube channels, their own distance courses and other electronic materials together with the printed textbooks, dictionaries etc. in educational process. Before COVID-19, a new version of Moodle was installed at Lviv National Environmental University (LNEU) server and newer versions of distance courses were compiled for students of different specialties at LNEU five Faculties. Because of COVID-19 as well as the Russian-Ukrainian war, the rate of compiling the distance courses at the University became slower. That is why the teaching staff of LNEU decided to use online courses out of Ukrainian and foreign MOOC platforms as additional educational materials for distance courses, textbooks, lecture courses, etc. Our university students also analyzed different MOOC platforms and tried to search for the online courses and other electronic educational materials for their specialties in the form of research work. The results of the first-year students’ research were discussed at the Ukrainian and international student’s conferences and summed up in our report in 2022. The purposes of the present report are as follows: to describe the prospects of further search for educational material from MOOC platforms, other electronic resources for the second-year students and the materials analysis; to present the ways of spreading the results of students’ research work and participation at the scientific conferences for exchanging their experience in usage of the reviewed materials; and to represent some electronic education materials discussed by professors of different Ukrainian HEIs and implemented for their students as well as the software and other materials created by the teaching staff of LNEU for the first- and second-year students. Keywords MOOC platform · Online course · Students’ research work · Educational process

B. Shunevych (B) Lviv National Environmental University, Lviv Region, Dublyany 80381, Ukraine e-mail: [email protected]; [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_41

505

506

B. Shunevych

1 Introduction Ukrainian higher educational institutions (HEIs) began to use actively materials of MOOC platforms as additional educational materials for distance courses before the beginning of COVID-19 [1]. Now these materials are being used more effectively at HEIs. Administration of Lviv National Environmental University (LNEU) organized installing a new version of Moodle at its server, and newer versions of distance courses were being compiled for students of different specialties at our university’s five faculties just before the epidemic and during the full-scale Russian invasion of Ukraine. It was hard to organize distance learning at the beginning of the epidemic; that is why the simplest means of communication with university students were chosen by teaching staff of our university. For example, for providing English classes during the quarantine were used e-mail or Viber. The study materials were also available on the university website. All students had the opportunity to communicate constantly with their teachers by phone and e-mail, by Zoom video conferencing and textbooks, electronically, and by printed dictionaries. Then the teaching staff of the Department of Foreign languages at LNEU decided to use English online courses from Ukrainian Lingva.Skills and Prometheus platforms as well as American Coursera platform and other electronic materials as additional educational materials for distance courses, English textbooks, lecture courses, and more. Lecturers of LNEU usually ask students to research new foreign prospective technologies in different spheres of science and technology as well as other new trends of their development in Ukraine and abroad. But now because of COVID-19 and the war, many LNEU’s students began to use different educational materials of MOOC platforms during their studying at high schools and are using them successfully now at our university. Some of them tried to search for the online courses and other electronic educational materials for their specialties and analyzed them in the form of research work under the guidance of teachers. That’s why the most-active-in-this-trend students proposed to do their research in new additional electronic educational materials to improve their knowledge in different disciplines. The results of the first-year students’ research were discussed at the international student’s conferences [2] and teacher’s conferences [3] and were summed up in our report at the international conference in Fergana (Uzbekistan) [4] in 2022. The purposes of the article are as follows: to describe the prospects of further search for educational material from MOOC platforms, other electronic resources for the second-year students and the searched materials analysis; to present the ways of spreading the results of students’ research work as well as exchanging their experience in usage of the reviewed materials; and to represent some electronic education materials discussed by professors of different Ukrainian HEIs and implemented for

Massive Open Online Courses at Ukrainian Agrarian Universities: To …

507

their students as well as the software created by the teaching staff of LNEU for the first- and second-year students.

2 Methods For this research and the student’s work, a descriptive method and the method of comparative analysis were used. All of us described disciplines for different specialties for the first-year students at the LNEU faculties as well as analyzed education materials of distance courses proposed by professors of LNEU for different taught disciplines and educational materials from MOOC platforms that could be used as additional ones.

3 Results Students were asked to prepare materials for their research based on study materials for all disciplines taught to them by teachers at our university. In order to be able to compare the research results, first-year and now second-year students do their research according to the following scheme: traditional introduction, purpose and tasks of the research; use of a list of disciplines that students study during the first or second year; availability/absence of distance courses and other electronic and printed materials on these disciplines at the university; the presence/absence of online courses on the above-mentioned MOOC platforms, which can be used as additional materials for these disciplines; analysis of identified online courses that students use as additional materials to the electronic and printed educational materials offered by teachers; and conclusions.

3.1 The First-Year Students’ Research During 2020–2022, the first-year students of the Lviv National Environmental University completed their researches on the specialties “Automotive Transport,” “Accounting and Taxation,” “Information Systems and Technologies” and “Agronomy.“ They also made reports about the results of the research at the students’ and teacher’s international conferences that were held in Ukraine. The materials of their reports are published in the Proceedings of the conferences. For example, one of the students analyzed the materials of disciplines, taught by the first-year students majoring in “Automobile Transport” specialty, and alternative learning materials on the platform of massive open online courses “Prometheus” for the organization of blended learning at the Faculty of Mechanics and Power Engineering. He also described the experience and prospects of blended learning of

508

B. Shunevych

the English language for students majoring in this specialty who have English classes at the Department of Foreign Languages in Lviv National Environmental University. The second student analyzed the academic disciplines offered to first-year students of the “Accounting and Taxation” (AT) specialty (Faculty of Economics) and identified the presence of online courses at the Ukrainian “Prometheus” platform, which can be used as additional materials for training students of this specialty. In the process of research, it was found that, out of 16 disciplines offered to first-year students of the AT specialty, only 4 disciplines (Economic Theory, History of Ukraine, Philosophy, and English Language) are among the online courses of the Prometheus platform. Most online courses are related to the discipline “English language” (English for beginners, Elementary level A1 - A2, English for media literacy, English for career growth, and English for business and entrepreneurship). In addition to the mentioned online courses, the Ukrainian platform LinguaSkills was discovered. Its online courses can be used for learning English from A1 to B1 levels [4]. The third student analyzed the electronic educational materials offered by the teachers of the Department of Information Technologies and other departments of our University in the disciplines that are taught during the first year of the bachelor’s degree in the specialty “Information Systems and Technologies” (Faculty of Mechanics and Power Engineering). She also identified possible online courses that are provided by the “Prometheus” platform and in the Internet in general for further use them in teaching students of this specialty. At our university, first-year students majoring in “Information Systems and Technologies” study 13 disciplines, including: computer circuit engineering, business analytics, higher mathematics, English, and business in IT. In addition to traditional manuals and methodical recommendations, teachers of the above-mentioned disciplines offer distance courses in the Moodle virtual learning environment for the first-year students. In the process of analyzing the online educational platform “Prometheus,” online courses were revealed that can be used as the additional sources for studying the four disciplines mentioned above. This platform provides full courses for such disciplines as “History of Ukraine: Completed course of preparation for the ZNO,” “Philosophy,” “Y Combinator startup school,” “How to create a startup,” and “IT product from scratch: where to start and how to develop?”. A lot of information for studying the disciplines can be found on YouTube channels that provide an opportunity for better understanding the topic “HTML & CSS,” for example the course “Basics of HTML & CSS for beginners.“ This course can be used as an additional course for the first-year students while studying the “Fundamentals of Information Technologies” discipline. An interesting YouTube channel is English for IT from Brain TV. The channel offers English lessons for beginners in the field of information technologies [2]. The fourth student analyzed the materials of disciplines, taught by the first-year students majoring in “Agronomy” specialty, as well as alternative teaching materials on the platform “Prometheus” MOOC platform for the organization of blended learning at the Faculty of Agricultural Technologies and Ecology. [3].

Massive Open Online Courses at Ukrainian Agrarian Universities: To …

509

Educational materials studied by students are placed on the platform of the virtual learning environment of the Lviv National Environmental University. Additional educational materials from the above-mentioned disciplines were discovered by the student on the Prometheus platform. For example, the developers of educational materials of the Prometheus platform offer two intensive online courses in Botany and plant physiology, Agroengineering and Agronomy. These courses have topics that can be used as additional material for studying the disciplines of Plant Physiology, Plant Breeding and others. Among the general education disciplines, two online courses should be used in the educational process: “Modern History of Ukraine: from the beginning of the Second World War to the present days” and “New Physical Culture.” Students can also test their knowledge of these disciplines on the platforms mentioned above.

3.2 The Second-Year Students’ Research Now the second-year students specializing in “Construction and Civil Engineering” (Faculty of Construction and Architecture) as well as in “Information Systems and Technologies” (Faculty of Mechanics and Power Engineering) are doing their research according to the mentioned above scheme. They are going to expand the list of the investigated platforms because there are many special subjects in their curriculum as well as other electronic resources — for example, YouTube channels. Besides the mentioned above Coursera platform, there are also popular and used in Ukraine such foreign platforms as CanvasNetwork, edX, FUN, FutureLearn, htmldog, KhanAcademy, MIT, MyEducationKey, OpenCourseware, Open2Study, rubymonk, Udacity, Udemy, and XuetangX. Among Ukrainian platforms, there are well-known platforms such as EdEra (Educational Era), FreeMonline (Free University of Maidan online), and WiseCow (Free video lectures Wise Cow). The innovative form of learning by means of MOOC platforms gives a possibility to have a free access by means of the Internet simultaneously for many students to almost all or many educational materials of the above mentioned online platforms.

3.3 The Ukrainian Higher Education Teaching Staff Research The results of implementing online courses in educational process are discussed by professors of LNEU and other universities, including agrarian ones, at Ukrainian as well as International conferences that are held at Ukrainian universities [5–7] and abroad [4]. At the conferences, besides implementation of educational material of MOOC platforms, other electronic resources were discussed that can were used in educational

510

B. Shunevych

process. For example, at the International conference Topical Issues of Linguistics, Professional Linguodidactics, Psychology and Pedagogy of Higher Education [5] in Poltava (November 24 and 25, 2022) YouTube channels were discussed: Latinitas Animi Causa, Latintutorial, Polymathy (pp. 146–149) and other resources. At the conference Topical Issues of long-life education in information society [6] in Kyiv (May 29 and 30, 2020), social network (Facebook) and messengers (Viber, Telegram) for communication with students (pp. 99–100), geoinformation resources Google, online services Google Earth, Google Maps, Google Art Project (pp. 100–102), and others were discussed. At the conference Interaction of Language Units: Communicative and Cognitive, Sociocultural, Translation and Methodological Approaches [7] in Kyiv (April 22, 2020), mobile learning (iPods, cell phones, iPhones), blogs, online quizzes and tests, instant messenger, internet telephone Skype, teleconferences (pp. 91–92), Google Classroom platform (pp. 71–73), and collaborative projects, according to Telecollaboration for Intercultural Language Acquisition Consortium (pp. 30–32), were discussed. At Lviv National Environmental University the teaching staff of the Department of Management [8], Department of Energetics, Department of Information Systems and Technologies, and other departments also use different electronic resources at their lectures, practical classes and research work [9] as well as develop their own software [10] that is used for teaching students at our university.

4 Conclusion and Future Work The results of the students’ researches can be used, as additional materials, by the teaching staff for planning and improving the educational process at our university and other agrarian universities in the near future. Ukrainian and International students’ conferences as well as teachers’ conferences are good places for exchanging ideas of using educational materials of MOOC platforms among students and teachers of different HEIs in Ukraine and abroad. The teaching staff of LNEU uses licensed foreign electronic educational resources and actively develops their own electronic educational resources that can be used at Ukrainian and foreign higher educational institutions.

References 1. Shunevych, B.I., Drapalyuk, H.S., Pyndyk, N.I.: Innovative computer technologies in foreign languages instruction at Ukrainian higher schools. Ukr. J. Inf. Technol. 2(1), 73–78 (2020). https://doi.org/10.23939/ujit2020.02.073 2. Potsiluiko M.: Application of the Prometheus platform online courses, YouTube channels for studying disciplines in specialty “Information systems and technologies”. Students and scientific progress in agricultural and industrial complex. In: Proceedings of the students scientific forum (October 4–6, 2022), p. 569. Lviv (2022)

Massive Open Online Courses at Ukrainian Agrarian Universities: To …

511

3. Shunevych B., Parkhomuk O.: Modern possibilities of successful learning of educational disciplines by students of Lviv National Agrarian University. Current trends in the study and teaching of foreign languages. In: Proceedings of the 1-st International Scientific and Practical Online Conference (Poltava, 04 June 2021), pp. 252–255. Astraya, Poltava (2021). (in Ukrainian) 4. Shunevych B., Rak N.: Modern selection of educational materials for students of Ukrainian higher schools. International scientific conference «Information technologies and management in higher education and sciences»: Conference proceedings (November 28, 2022. Fergana, the Republic of Uzbekistan). Riga, Latvia : “Baltija Publishing”. Part 3, pp. 376–380 (2022). . https://doi.org/10.30525/978-9934-26-277-7-270. (in Ukrainian) 5. Topical Issues of Linguistics, Professional Linguodidactics, Psychology and Pedagogy of Higher Education. In: Proceedings of the 7th International Conference (Poltava, 24–25 November 2022), 275 p. Astraya, Poltava (2022). (in Ukrainian) 6. Topical Issues of long-life education in information society. In: Serhienko, V., Slabka, V., (eds.) Proceedings of conference (29–30 May 2020). 420 p. Mykhailo Drahomanov PH, Kyiv (2020). (in Ukrainian) 7. Interaction of Language Units: Communicative and cognitive, sociocultural, translation and methodological approaches. In: Proceedings of the 7th International scientific and practical conference (22 April 2020), 98p. Ihor Sikorskyi KPI, Politechnika Publishing House, Kyiv (2020). (in Ukrainian) 8. Voinycha L.: Disruptive technologies in the field of education. Theory and practice of agricultural and industrial complex as well as rural territories development. In: Proceedings of the 21st International scientific and practical forum (22–24 September 2020), pp. 225–228. Lviv: ATB PH. (2020). (in Ukrainian) 9. Syrotyuk S., Syrotyuk V., Halchak V.: Fuzzy-logic controller of management mode of operation wind power installation motrol. Commision of motorization and energetics in agriculture. Lublin – Rzeszow. 17(4), 39–46 (2015). (in Russian) 10. Tryhuba, A., Kondysiuk, I., Tryhuba, I., Lub, P.: Approach and software for risk assessment of stakeholders of hybrid projects of transport enterprice. In: CEUR Workshop Proceedings, vol. 3295, pp. 86–96 (2022). https://ceur-ws.org/Vol-3295/paper8.pdf

Participative Learning Experience Design Through Group Concept Mapping Slavi Stoyanov

Abstract Although there has been an increasing amount of literature on learning experience design in the past five years, it is not clear yet how to operationally integrate the process and methods of instructional design, learning design, userexperience design, design-based research and design thinking in a multi-step participative design process involving stakeholders with different perspectives. Our recent systematic literature review on relationships between learning design and learning analytics applying a critical interpretive synthesis and text analytics identified two issues rarely explicitly discussed in the literature: “evidence-informed instructional design approaches” and “design-based research.” Elaborating on these two concepts, we proposed that evidence-based practice promoted within the learning design paradigm and the need for applying research-based findings advanced in the instructional design field should be complementary to each other in a participatory learning experience design process built upon the tradition of design-based research and recent development of software engineering design and Design Thinking. Against this background, the position paper addresses the following research question: How can we facilitate the participative technology-enhanced learning design in an effective, efficient and appealing way? To this end, the paper introduces Group Concept Mapping (GCM), a mix-methods research methodology. GCM is a consensus-driven approach combining qualitative data collection with advanced statistical techniques to aggregate participants’ contributions and show their collective perspectives. Keywords Instructional Design · Learning Design · Learning-Experience Design · Group Concept Mapping

S. Stoyanov (B) Open University of The Netherlands, Valkenburgerweg 177, 6419 AT Heerlen, The Netherlands e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_42

513

514

S. Stoyanov

1 Introduction Instructional Design (ID), Learning Design (LD) and Learning Experience Design (LXD) are established epistemic traditions and specific problem-solving approaches that inform research and practice of technology-mediated learning and teaching. The concepts are closely related but for a long time the widespread discourse has considered them independent of each other. Instructional Design (ID) is concerned with research-based prescriptions for the design, development, evaluation and implementation of instructional methods for supporting learning in most effective, efficient and appealing ways [1, 2]. Learning design (LD) is referred to as a declarative framework for guiding, representing and sharing activities, resources and tools for achieving educational goals in a given context highlighting teachers’ experience and creativity [3, 4]. In an attempt to distinguish LD from ID, authors associated themselves with LD, present not always strong arguments. For example, ID is based on behaviourism, LD is grounded on constructivism [5]; ID adopts a content-centric approach while LD explores active pedagogies in learning and instruction [6]. The outcomes of ID research are more useful for implementation in large instructional projects, whereas LD works better in small-scale interventions [7]. ID is preferred concept in the United States, and LD is used more in Europe [2, 8]. The most straightforward and popular reference to Learning-experience design (LXD) is utilising User-Experience Design’s theories and methods for Learning Design and Instructional Design [9]. Some LXD scholars apply a relatively narrow definition of LXD, trying to contrast it to ID [10]. For example, LXD has a broader focus than instructional design. If Instructional Design asks, “How easy is a learning experience to use?” Learning Experience Design asks, “How useful is the learning experience?” According to them, traditional ID frameworks are based entirely on a process such as ADDIE’s (analysis, design, development, implementation, and evaluation). Instead focusing on the process, LXD focuses on the output. Schmidt et al. [9] define LXD as a humancentric, theoretically-grounded, and socio-culturally sensitive approach to learning design, intended to support learners towards identified learning goals, and informed by User-Experience Design (UXD) methods. Neelen and Kirschner [11] believe that learning cannot be designed. Learning experiences that support effective, efficient and enjoyable learning can. “What is nice about the word ‘experience’ is that it emphasises that learning is affected by a person’s involvement in or exposure to something. It acknowledges that learning can be experienced in many ways and in a wide variety of contexts and that learning is a journey, a process that takes place over a period of time” (11, p. 51). The term reflects any learning experience in any setting (classroom, museum, workplace). Malamed [12] formulate ten principles of LXD for workplace learning: LXD recognises that training is not always the solution; LXD is human-centered; LXD insists on inclusive Design; LXD seeks to create a positive and meaningful user experience; LXD emphasises that learning is a journey; LXD relies on research-based

Participative Learning Experience Design Through Group Concept …

515

findings to make design decisions; LXD seeks input from users and participants; LXD uses real-world metrics to measure performance improvement; LXD recognises the value of sharing and social engagement; and LXD is innovative and flexible. ID, LD and LXD emerged in different periods to tackle specific learning and instruction problems not addressed by the preceding approach. For example, LD enriched the systematic evidence-informed approach of ID with practice-oriented inquiry, context, teachers’ design community of practices, patterns of learning activities and specifications for the machine-readable language. Learning Experience Design promotes a human-centred approach and specific methods for designing software applications for educational and training purposes. It is more productive to emphasise the similarities between the design approaches so they can be mutually beneficial to each other rather than artificially magnifying their differences. The question is how “mutually beneficial” can be operationalised in concrete design activities and measures and how the different design approaches can be integrated. A more formal expression of it is as follows: How can we facilitate a participatory technology-enhanced learning design in an effective, efficient and appealing way? In the following sections the contributions of each design approach are described and then Group Concept Mapping—a mixed research methodology that could integrate these contributions is introduced.

2 Contributions of Instructional Design, Learning Design and Learning Experience Design In a systematic literature review using text analytics to conceptualise the role of ID, LD and LXD in designing Learning Analytics applications, two emerging— but shallowly discussed—approaches in literature themes were identified. They are: “Design-based research” and “Evidence-informed instructional design approaches” [13] (see Fig. 1). Design research, known initially as design experiment [14, 15] was an attempt to reconsider the complexity of ID interventions in a real-world context and the need to apply relevant measures that go beyond narrow laboratory tests. Design research was developed as formative research to refine educational designs based on prior laboratory research. At the same time, these foundational principles were further developed in generic ID models [16]. The most popular among them is Dick and Carrey’s model (ADDIE), which is often reduced to only its phases. The educational design research paradigm recognises the influences of ID models [17]. However, it emphasised the need for investigating a broader scope of educational problems instead of only instructional ones. Ill-structuredness is another feature to distinguish between educational problems [18]. Structure refers to the degree to which a problem has a single and most efficient solution path (well-structured) or many possible solution paths (ill-structured). Ill-structured problems are multifaceted and crossdisciplinary, with no clear-cut procedures and no simple right-or-wrong answers.

516

S. Stoyanov

Fig. 1 Text analytics educational design approaches and learning analytics

Design-based research (DBR) is a systematic, theory-driven, collaborative, and iterative process of analysis, design, and evaluation of research-based solutions for complex and even ill-structured problems in educational practice [14, 17]. Such a conceptualisation of DBR resonates with processes and methods of UserExperience Design (UxD) and Design Thinking (DT), which is an opportunity to borrow ideas from them. Apart from applying classical methods such as interviews, observations and surveys, educational design research would benefit from the UXD tradition of using techniques such as card sorting and affinity mapping, storytelling, personas, empathy mapping, experience mapping, job-to-be-done, contextual inquiry, prototyping, think aloud, A/B testing, analytics, eye-tracking, usability testing, and AI bots for usability testing. Non-traditional methods such as immersive ethnography, cultural probes, hypothetical digital dilemmas, discovery backlog, design-the-box pitch, the world cafe, and analysing memes could also be helpful. User-Experience Design practitioners clarify the relationships between quantitative and qualitative research by stressing the difference between the type of research questions they address. Quantitative research responds to questions such as “what” and “how much,” while qualitative research is best at answering questions about “why.” The field of UXD has also proposed some helpful ideas about user sampling and saturation. Researchers with academic backgrounds contributed to the field of UXD by promoting methods for quantitative analysis of qualitative data in addition to cognitive task analysis, systematic literature review and experimental design methods. Evidence-informed instructional design approaches (the other emerging theme from the case study, referred earlier) is the core idea of ID, LD and LXD. The interpretation of evidence-informed ID implies considering the following learning and instruction strategies and approaches: 1. First Principles of Instruction: learning is promoted when learners are engaged in solving real-world problems; learning is promoted when existing knowledge

Participative Learning Experience Design Through Group Concept …

2.

3. 4. 5.

6.

517

is activated as a foundation for new knowledge; learning is promoted when new knowledge is demonstrated to the learner; learning is promoted when new knowledge is applied by the learner; and learning is promoted when new knowledge is integrated into the learner’s world [19]. Instructional design heuristics: focusing on deep conceptual understanding, which includes factual, procedural and conceptual knowledge but also how and when to apply it; actively participating in learning processes; building on prior knowledge; including opportunities for reflection; and ensuring learning transfer [11]. Effective learning strategies: space (distributed) practice, practice test (retrieval practice), interleaving, questioning, and explaining to oneself [20]. Multimedia learning supported by cognitive load theory [21] and the cognitive theory of multimedia learning [22]. Holistic ID approaches: the Elaboration theory [23], the Four Component Instructional Design model [24], the Cognitive Apprenticeship approach [25], the Cognitive-Flexibility Theory [26], Problem-Based Learning [27], Epistemic Frames [28] and Instructional Design Model of Motivation [29]. Taxonomy of learning activities as an operationalisation of the LD notion of Evidence-based teacher practice: assimilative, finding and handling information, communication, productive, experiential, interactive, and assessment [6, 30]. Evidence-based practice means that teachers bring prior research to classroom, observe and measure its effects, adapt it if needed and share it with fellow teachers to build collective wisdom [5]. Design-Inquiry of Learning approach, which integrates ideas from inquiry-based learning, teaching inquiry, and design-based research [31], is also an operationalisation of the Evidence-informed practice concept.

One approach that should not be overlooked when discussing contributions to educational design research and practice is Design Thinking (DT). It is an overarching concept reflecting and extrapolating designers’ methods, tools and mindset for human-centred, creative, participative, interdisciplinary and iterative problemsolving of ill-structured problems. The problems that DT typically addresses are multifaceted, and cross-disciplinary, with no clear-cut procedures and no a simple right-or-wrong answer [32–36]. As a design-driven inquiry approach, DT is similar to the Educational Design Research and the User-Experience Design processes. On the one side, DT could be considered a rational approach, exploring the notion of ‘design as science’ [37] and operationalised through various data collection and analysis methods. From the other, DT implements creative techniques for divergent and convergent ideation. The design-thinking process iteratively facilitates active learning and experimentation through series of prototypes [32, 37, 38]. An important aspect of DT is cognitive abilities and strategies DT professionals employ (e.g., design heuristics, framing, analogical reasoning, abductive reasoning, mental simulation, pattern recognition, mental synthesis and transformation, structural imagination, and cognitive biases) [39–42]. DT also emphasises the need for

518

S. Stoyanov

participative design involving both designers and users [43]. The types of problems DT addresses require multidisciplinary, cross-functional teams, which makes managing diversity in teams essential [44, 45]. A human-centred mindset means not only a focus on users whose issues need to be addressed in a specific context but also a focus on the teams of designers.

3 Managing Diversity in Participative Educational Design Two generic problems must be considered and dealt with when a participative, educational design project is initiated. One is the issue that needs to be solved. The other is the differences between the participants (e.g., level of expertise, professional experience, and educational background). An individual difference characteristic, typically overlooked, is the cognitive style for problem-solving and decision-making. Cognitive style is a psychological dimension that represents consistencies in an individual’s manner of cognitive functioning, particularly with respect to thinking, decision making and problem-solving [44, 46]. Cognitive style could strongly affect educational design project outcomes because it operates across all other individual differences features [44, 47]. Different cognitive style could be found at different levels of expertise, experience and educational background. Cognitive style impacts data collection and analysis activities in the educational design process. Evidenceinformed cognitive style theories can explain the reasons for differences in the behaviour of people and suggest measures for managing diversity in groups [44, 47]. Such a theory is Kirton Adaption-Innovation (KAI) [44]. KAI is based on a wellestablished theory that has been used in different academic and business domains for more than 40 years. It has strong predictive power, high reliability and extensive validity evidence. Kirton’s Adaption-Innovation theory contrasts strongly level types of cognitive constructs, which typically address the question “how much” (e.g., intelligence, knowledge) and style cognitive constructs, which tackle the question “in what manner.“ The KAI Inventory, which is a valid and reliable operationalisation of the KAI theory, locates people on a continuum that ranges from highly adaptive to highly innovative, with large general populations exhibiting normal distributions. However, in concrete situations, it could be skewed towards one end, and individuals may fall on a particular group’s more innovative or more adaptive side, depending on that group’s overall style distribution. In general, the more adaptive one is, the more one has a positive regard for structure (e.g., theory, guidelines, policy) that is also agreed upon, and the more one will prefer to solve problems by refining, extending, and improving the current, generally accepted pattern, strategy, or paradigm. In contrast, the more innovative one is, the less tolerant it is of existing structure. The high adaptor tends to produce relatively few practical ideas within a particular framework. In contrast, the more innovative ones tend to generate many ideas as they try to think and work tangentially with a particular framework or even combine different ones.

Participative Learning Experience Design Through Group Concept …

519

The more cognitive style diversity, the broader the range of problems that the group can tackle, but the more difficult it is to manage. The less cognitive style diversity, the more limited the range of problems the group can tackle effectively, but the easier it is to manage. Awareness of style diversity is the first step in learning to become more tolerant and appreciative of its value in problem-solving [47]. The following section introduces the Group Concept Mapping approach that contributes to learning experience design by integrating findings from different methods for data collection and analysis to define the design problem, generate and structure the design ideas, create personas as part of user stories, and build and evaluate low and high fidelity prototypes.

4 Group Concept Mapping The first task in an educational design project is to frame the problem—to understand it as the users in a particular context experience it. To that end the team of designers needs to collect information by utilising a range of methods—interviews, observations, surveys, contextual inquiry, cognitive task analysis, and literature review. It is recommended to combine different sources of information as one aspect of research triangulation, but it is also challenging during synthesising and analysing the data. One approach often used is Affinity mapping. Briefly, a group of designers using sticky notes collects findings from various data collection methods and then classifies them under broader categories. Affinity mapping is a helpful exercise, but consensus, especially in a sizeable cognitive diversity case, appears to be a challenge. In addition, the synthesis of findings remains predominantly qualitative. Software tools associated with the card sorting method (e.g., Optimal workshop: https:// www.optimalworkshop.com) could optimise the procedure by introducing quantitative methods (e.g., hierarchical cluster analysis). These tools are intended primarily for designing a content architecture, and their preferred sorting method is through pre-structured categories. The information from the data collection could also be organised by constructing a classical collective concept map or building a cognitive map, which also allows some quantification, based on how the cognitive mapping constructs are arranged. Compared to affinity mapping, card sorting, concept mapping and cognitive mapping, Group Concept Mapping (GCM) provides more effective, efficient and appealing support for integrating the qualitative data collected through different methods, structuring that data and analysing it quantitatively. GCM adds another aspect of research triangulation by combining qualitative and quantitative data analysis. Group Concept Mapping is a mix-methods and a consensus-driven research approach that combines qualitative data collection with advanced statistical techniques (e.g., multidimensional scaling and hierarchical cluster analysis) to aggregate participants’ contributions and show their shared collective perspectives on what

520

S. Stoyanov

are the design ideas, how they are related, how they are grouped into more general categories and how they are valued [48, 49]. The resulting map is a group cognitive artefact that conveys the shared understanding objectified through a formalised representational schema derived statistically across individual contributions [50]. The map yields cognitive support for visual discourse and further elaboration on the findings to produce new outcomes. The cluster structure of GCM could be used as a research basis for creating personas, which could then facilitate the prototype design. Although GCM participants structure the statements in distinctive ways, the multivariate statistics implemented in the method make it possible to accommodate all individual perspectives, including cognitive styles for problem-solving, into a group’ shared vision [47]. This function of GCM is a unique contribution to cognitive style research and practice. GCM can also show differences between different groups based on how they rate the design ideas and how they grouped them [51]. The default option for idea generation of GCM is through brainstorming, addressing a question or, more often, a focus prompt (derived from the question as a sentence that needs to be completed). The ideas generated are in the format of statements/short phrases. In line with research [52, 53] the participants work individually and independently of each other, but they can see the statements that are being generated by the others. Statements extracted from interviews, contextual inquiry, observations and open survey questions could also be idea-generation inputs. Next, the participants are invited to structure the collected ideas by grouping them thematically into larger categories giving names and then rating each statement on some values (e.g., importance and feasibility). Sorting and rating are also individual activities supported by the GCM software [54]. In general, all the activities (managing participation, demographics, communication, idea generation, sorting, rating and analysis) can be carried out within the software. The following text presents some brief descriptions and examples of typical GCM outcomes from an unpublished study. The first output of the sorting data analysis is a point map (see Fig. 2), a two-dimensional model created by performing multidimensional scaling on the participants’ raw statements grouping. MDS produces a statistic called “stress index,” which indicates the extent to which the mathematical model in the point map represents the similarity matrix based on the participants’ statement sorting. The stress index is a generic indicator for the internal validity of the GCM study. The closer statements to each other are, the closer in meaning they are, which also means that more participants grouped these statements together. Statements that are further apart have been rarely grouped together, also suggesting a weak relationship. Thematic areas are outlined by applying hierarchical cluster analysis (HCA) on the coordinates of the point map ideas (see Fig. 3). The decision on the number of clusters is based on the suggestions made by HCA and the researchers’ judgments taking into account the purpose of the project and the need for a balance between the big picture and details. The next step is to label the clusters. The software makes suggestions based on the names given by the participants during the sorting. It is also helpful to check the

Participative Learning Experience Design Through Group Concept …

521

Fig. 2 Relationships between design ideas (A point map)

Fig. 3 Thematic areas (A cluster map)

bridging values (BVs; between 0 and 1) of individual statements in a cluster (MDS assigns a low BV when a statement is grouped with statements around it on the point map and a high BV when a statement is sorted with statements further apart). The statements with lower BV are considered a better representation of the content of a particular cluster. The final way to determine the name of the cluster is by reading through the statement to define the content’s meaning (see Fig. 4). Rating data provide some additional information. For example, so-called pattern match (see Fig. 5) compares clusters on some rating values.

522

S. Stoyanov

Fig. 4 Clusters labeled

Fig. 5 Rating values compared on a cluster level (Pattern match)

A visualisation called “go-zone” (see Fig. 6) divides a cluster into four quadrants based on the rating value averages to suggest shorter-term (upper right quadrant) or longer-term actions (the quadrant below). GCM can synthesize data from different learning design experience methods for data collection, such as contextual interviews, questionnaires, observation, systematic literature review and cognitive task analysis. While the participants structure individually and independently of each other the data by classifying and prioritizing it, some multivariate statistical methods aggregate the data across the participants with different individual characteristics to show their shared vision of the design problem as a clustered conceptual map. The clustered data map is the empirical basis

Participative Learning Experience Design Through Group Concept …

523

Fig. 6 Rating values compared at an individual cluster level (Go zone)

for creating learning experience design personas, which are involved in a series of prototypes contextually integrating design for learning strategies.

5 Limitations of the Study and Future Work The paper included references to several methods such as affinity diagram, storytelling, personas, empathy map, experience map, job-to-be-done, contextual inquiry, prototyping, think-aloud, A/B testing, analytics, eye-tracking and usability testing. In addition, it briefly discussed some ID approaches. It is out of the scope of the paper to describe each method and approach in detail. The intention for the future is to draft a learning experience design framework where to define, conceptualise and exemplify these methods and approaches by combining contributions of industry and academia. Some of the concepts are presented quite positively (e.g., Design Thinking and Persona), but there is also criticism, which has to be welcomed. However, the critiques often present the tools used in DT in a rather simplistic way (e.g., one-page template for persona or a short workshop limited to discussing the DT stages schematically). The problem is not in the tools but in how they are implemented. The use of GCM was focused in this paper on understanding the problem which determines the usefulness of technology-enhanced learning products (build the right product). However, there are no technical limitations to using GCM in generating alternatives for architecture and interaction design by implementing ideas from ID, LD and LXD, or for prototype evaluation. The purpose, stage, methods, tools for data collection and analysis and participants of a project could determine the decision when to use GCM. Experience does not recommend several GCM studies in one project. The participants could get tired and even bored.

524

S. Stoyanov

The GCM process inherently includes measures for managing potential threats to validity and reliability [55]. However, there are times when some methodological decisions need to be taken and it might affect the outcomes of a GCM study (e.g., selecting the number of clusters).

6 Conclusions ID, LD and LXD, as design approaches operating in technology-mediated learning, could be mutually beneficial to each other. ID, LD, and LXD uniquely contribute to designing technology-enhanced learning applications. ID is helpful with evidence-informed strategies and approaches. LD emphasises evidence-teacher practices, creativity and community of practice. LXD brings the notion of user experience and proposes methods for data collection and analysis and measures for managing multidisciplinary collaboration. GCM can integrate in a seamless way data collection and analysis of different educational design approaches.

References 1. Honebein, P.C., Reigeluth, C.M.: The instructional theory framework appears lost. Isn’t it time we find it again? Revista de Educación a Distancia 64(20), 1–24 (2019). https://doi.org/10. 6018/red.405871 2. Seel, N.M., Lehmann, T., Blumschein, P., Podolskiy, O.A.: Instructional Design for Learning: Theoretical foundations. Sense Publishers, Rotterdam (2017) 3. Dalziel, J., et al.: The larnaca declaration on learning design. J. Interact. Media Educ. 1(7), 1–24 (2016). https://doi.org/10.5334/jime.407 4. Mor, Y., Craft, B.: Learning design: reflections on a snapshot of the current landscape. Res. Learn. Technol. 20, 85–94 (2012) 5. Mor, Y., Ferguson, R., Wasson, B.: Editorial: learning design, teacher inquiry into student learning and learning analytics: a call for action. Br. J. Edu. Technol. 46(2), 221–229 (2015). https://doi.org/10.1111/bjet.12273 6. Holmes, W., Nguyen, Q., Zhang, J., Mavrikis, M., Rienties, B.: Learning analytics for learning design in online distance learning. Distance Educ. 40(3), 309–329 (2019). https://doi.org/10. 1080/01587919.2019.1637716 7. Persico, D., Pozzi, F.: Informing learning design with learning analytics to improve teacher inquiry. Br. J. Edu. Technol. 46(2), 230–248 (2015). https://doi.org/10.1111/bjet.12207 8. Wasson, B., Kirschner, P.A.: Learning design: European approaches. TechTrends 64(6), 815– 827 (2020). https://doi.org/10.1007/s11528-020-00498-0 9. Schmidt, M., Earnshaw, Y., Tawfik, A. A., Jahnke, I.: Methods of user centered design and evaluation for learning designers. In: Schmidt, M., Tawfik, A.A., Jahnke, I., Earnshaw, Y. (eds.) Learner and User Experience Research: An Introduction for the Field of Learning Design & Technology. EdTech Books (2020). https://edtechbooks.org/ux/ucd_meth 10. Thurber, D.: Designing learning experiences for the future of learning in the digital age: a proposed framework. Curr. Issues Educ. 22(1) (2021). http://cie.asu.edu/ojs/index.php/cie atasu/article/view/1890.

Participative Learning Experience Design Through Group Concept …

525

11. Neelen, M., Kirschner, P.A.: Evidence-Informed Learning Design. Creating Training to Improve Performance. Kogan Page, London (2020) 12. Malamed, C.: 10 Principles of learning experience design. The eLearning coach, August 2020. https://theelearningcoach.com/lxd/10-principles-of-lxd/ 13. Stoyanov, S., Kirschner, P.A.: Text analytics for uncovering untapped ideas at the intersection of learning design and learning analytics: critical interpretative synthesis. J. Comput. Assist. Learn. (2023). https://doi.org/10.1111/jcal.12775 14. Brown, A.: Design experiment. Theoretical and methodological challenges in creating complex interventions in classroom settings. J. Learn. Sci. 2(2), 141–178 (1992). https://doi.org/10.1207/ s15327809jls0202_2 15. Collins, A., Joseph, D., Bielaczyc, K.: Design research: theoretical and Methodological issues. J. Learn. Sci. 13(1), 15–42 (2004). https://doi.org/10.1207/s15327809jls1301_2 16. Gustafson, K.L., Branch, R.M.: Survey of Instructional development models. 3rd edn. ERIC (1997) 17. McKenney, S., Reeves, T. C.: Conducting Educational Design Research. Routledge, Milton Park (2018) 18. Jonassen, D.H.: Toward a design theory of problem solving. Educ. Tech. Res. Dev. 48(4), 63–85 (2000). https://doi.org/10.1007/BF02300500 19. Merrill, M.D.: First principles of instruction. Educ. Tech. Res. Dev. 50(3), 43–59 (2002) 20. Donoghue, G.M., Hattie, J.A.C.: A meta-analysis of ten learning techniques. Front. Educa. 6(581216) (2021). https://doi.org/10.3389/feduc.2021.581216 21. Sweller, J.: Cognitive load theory: recent theoretical advances. In: Plass, J.L., Moreno, R., Brünken, R. (eds.), Cognitive Load Theory, pp. 29–47. Cambridge University Press (2010). https://doi.org/10.1017/CBO9780511844744.004 22. Mayer, R.E. (ed.).: The Cambridge handbook of multimedia learning (2nd ed.). Cambridge University Press, Cambridge (2014) 23. Reigeluth, C.M.: Instructional design: what is it and why is it? In: C.M. Reigeluth, C.M. (ed.) Instructional design theories and models. An overview of their current status. LEA (1983) 24. Van Merriënboer, J.J.G., Kirschner, P.A.: Ten steps to complex learning. A systematic approach to four-component instructional design (3rd edition). Routledge (2018) 25. Brown, J.S., Collins, A., Duguid, P.: Situated cognition and the culture of learning. In: McLellen, H. (ed.) Situated learning perspectives, pp. 19–44. Educational Technology Publications (1996) 26. Spiro, R.J., Jehng, J.: Cognitive flexibility and hypertext: theory and technology for the nonlinear and multidimensional traversal of complex subject matter. In: Nix, D, Spiro, R.J. (eds.) Cognition, education and multimedia, pp. 163–205. Routledge (1990) 27. Hmelo-Silver, C.E.: Problem-based learning: what and how do students learn? Educ. Psychol. Rev. 16(3), 235–266 (2004). https://doi.org/10.1023/B:EDPR.0000034022.16470.f3 28. Shaffer, D.W.: Epistemic frames for epistemic games. Comput. Educ. 46(3), 223–234 (2006). https://doi.org/10.1016/j.compedu.2005.11.003 29. Keller, J.M.: Motivational design of instruction. In: C. M. Reigeluth, C.M. (ed.), Instructional design theories and models: An overview of their current status. Erlbaum (1983). 30. Laurillard, D., et al.: A constructionist learning environment for teachers to model learning designs.J. Comput. Assist. Learn. 29(1), 15-30 (2011). https://doi.org/10.1111/j.1365-2729. 2011.00458.x 31. Mor, Y., Mogilevsky, O.: The learning design studio: collaborative design inquiry as teachers’ professional development. Res. Learn. Technol. 21 (2013). https://doi.org/10.3402/rlt.v21i0. 22054 32. Brown, T.: Design thinking. Harv. Bus. Rev. 86, 84–92 (2008) 33. Brown, T., Martin, R.L.: Design for action. How to use design thinking to make great things actually happen. Harvard Business Review, September 2015. https://hbr.org/2015/09/designfor-action 34. Buchanan, R.: Wicked problems in design thinking. Des. Issues 8, 5–21 (1992). http://www. jstor.org/stable/1511637

526

S. Stoyanov

35. Henriksen, D., Richardson, C., Mehta, R.: Design thinking: a creative approach to educational problems of practice. Thinking Skills Creativity 26, 140–153 (2017). https://doi.org/10.1016/ j.tsc.2017.10.001 36. Glen, R., Suciu, C., Baughin, C.: The Need for design thinking in business schools. Acad. Manage. Learn. Educ. 13(4), 653–667 (2014). https://doi.org/10.5465/amle.2012.0308 37. Simon, H.A.: The Sciences of the Artificial, 3rd ed. MIT Press, Cambridge (1996) 38. Liedtka, J.: Why Design thinking Works. Harvard Business Review, September–October 2018 39. Carbuio, M., Lovalo, D., Dong, A., Lin, N., Tschang, T.: Demystifying the genius of entrepreneuriship: how design cognition can help the next generation of entrepreneurs. Acad. Manage. Learn. Educ. 17(1), 41–61 (2018). https://doi.org/10.5465/amle.2016.0040 40. Finke, R.A., Ward, T.B., Smith, S.M.: Creative cognition, MIT Press, Cambridge (1996) 41. Foster, M.K.: Design thinking: a creative approach to problem solving. Manage. Teach. Rev. 6(2), 123–140 (2021). https://doi.org/10.1177/2379298119871 42. Kahneman, D., Klein, G.: Conditions for intuitive expertise. Am. Psychol. 64(6), 515–526 (2009).https://doi.org/10.1037/a0016755 43. Sanders, E.B.N., Pieter Jan Stappers, P.J.: Co-creation and the new landscapes of design. Co-Des. 4(1), 5–18 (2008). https://doi.org/10.1080/15710880701875068 44. Kirton, M.J.: Adaptation-Innovation. Routledge, Milton Park (2006) 45. Wrigley, C., Straker, K.: Design thinking pedagogy: the educational design ladder. Innov. Educ. Teach. Int. 54(4), 374–385 (2017). https://doi.org/10.1080/14703297.2015.1108214 46. Armstrong, S.J., Peterson, E.R., Rayner, S.G.: Understanding and defining cognitive style and learning style: a Delphi study in the context of educational psychology. Educ. Stud. 38, 449–455 (2011). https://doi.org/10.1080/03055698.2011.643110 47. Stoyanov, S., Jablokow, K., Rosas, S., Wopereis, I., Kirschner, P.: Concept mapping – an effective method for identifying diversity and congruity in cognitive style. Eval. Program Plann. Concept Mapping at 25 (special issue) 60, 238–244 (2017). https://doi.org/10.1016/j. evalprogplan.2016.08.015 48. Kane, M., Rosas, S.: Conversations about Group Concept Mapping. Applications, Examples, and Enhancements. Sage, Newcastle upon Tyne (2018) 49. Trochim, W.M., McLinden, D.: Introduction to a special issue on concept mapping. Eval. Program Plann. 60, 166–175 (2017). https://doi.org/10.1016/j.evalprogplan.2016.10.006 50. Stahl, G.: Computer Support for Building Collaborative Knowledge. Acting with Technology Series. MIT Press, Milton Park (2006) 51. Rosas, S.R.: Multi-map comparison for group concept mapping: an approach for examining conceptual congruence through spatial correspondence. Q. Q. Int. J. Method. 51(6), 2421–2439 (2017). https://doi.org/10.1007/s11135-016-0399-x 52. Korde, R., Paulus, P.B.: Alternating individual and group idea generation: finding the elusive synergy. J. Exp. Soc. Psychol. 177–190 (2017). https://doi.org/10.1016/j.jesp.2016.11.002 53. Rietzschel, E.F., Nijstad, B.A., Stroebe, W.: Productivity is not enough: a comparison of interactive and nominal brainstorming groups on idea generation and selection. J. Exp. Soc. Psychol. 42, 244–251 (2006). https://doi.org/10.1016/j.jesp.2005.04.005 54. The concept system® groupwisdom™. Build 2022.30.10. Web-based Platform. Concept Systems, Incorporated, 2022 Browser based statistical platform. https://www.groupwisdom. tech 55. Schophuizen, M., Kreijns, K., Stoyanov, S., Rosas, S., Kalz, M.: Does project focus influence challenges and opportunities of open online education? A sub-group analysis of group-concept mapping data. J. Comput. High. Educ. (2020). https://doi.org/10.1007/s12528-020-09264-w

Using Design Thinking to Understand Student (Dis)engagement in Higher Education: Involving Students in the Co-creation of Their Own Learning Experiences Elaine Tan

Abstract This paper documents the development of a design-thinking workshop to collaborate with students to understand disengagement from the physical campus. Student disengagement has become increasingly pronounced since a return from enforced online learning posing questions regarding underlying long-term problems faced by students. A design-thinking exercise was undertaken with 3 groups of students within an academic module, with the dual aims of facilitating student learning of the Design Thinking process, whilst simultaneously seeking to understand and develop solutions to the problem of student disengagement. The factors influencing student lack of attendance were examined with students working collaboratively to design a solution to promote engagement. Keywords Student Engagement · Design-Thinking · Student Experience · Student Partnerships

1 Introduction A return to campus for many students has been an underwhelming experience. Among colleagues, many are anecdotally reporting a greater level of apathy, disengagement and absence among the students. Further research is required to understand how the longer-term impact of pandemic has shaped and influenced student behaviours. Student transition into higher education was undertaken into one that was primarily technologically negotiated and independently undertaken. The outcome of this is a misalignment between staff and students of expected patterns of engagement and whether a return to “normal” is even possible.

E. Tan (B) Newcastle University Business School, Newcastle University, Newcastle, UK e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_43

527

528

E. Tan

1.1 Factors Influencing Attendance Decisions Previous studies have focused on identifying the factors which cause students not to attend lectures, with a number of conclusions drawn from influence of part time jobs [1, 2], gender [3], commute time and weather conditions [4], associated assessment [5], or student perceptions of poor quality of lectures [6]. With the introduction of learning technologies in particular, lecture capture has been the subject of much research in relation to attendance. Whilst previously reporting mixed findings [7–9], more recent results have indicated a significant negative impact on attendance [10– 12]. More recent studies by Baillie et al. [13] indicate that the use of lecture capture by students was also linked to lower attainment; the researchers suggest that a reliance on the tool had a more concerning impact of limiting the active engagement and behaviors of students in class when they did attend. What is clear from the literature, however, is that student attendance is a cause for concern, and for many students, there is a continual “decision process that involves weighing the benefits and costs of attending” [14, p. 23] and involves a number of factors. After the pandemic, little has been studied regarding the impact of these tools in relation to engagement and attendance as a longer-term outcome of changes in student practices and the cost-benefit decision-making behaviours. They note that lecture capture creates a “contested” [12] space, something that, in light of recent events, is even more significant as the impact of COVID-19 has expedited an existing trend and evolving culture of digital practices of students [15].

2 Methodology This study took place within an academic module as part of a combined business program. As a learning outcome of the module, students would develop an understanding of some of the tools used in Innovation practice, including Design Thinking. The project undertaken here served double duty; firstly as a means of developing understanding of the student experience (expressed as jobs, along with associated pains and gains), and secondly as a learning activity for students to experience and understand the process of Design Thinking as an activity and process in itself.

2.1 Design Thinking Design Thinking was selected as the approach as it affords the opportunity to build empathy with students as active participants [16] and seeks to understand the jobs, pains and gains from their perspective. Adopting a solutions-orientated approach, it involves students in the discussion and design of ways forward to create a vision of

Using Design Thinking to Understand Student (Dis)engagement …

529

higher education that is suitable for them in light of the contemporary higher education context and to reconcile some differences with staff. As the activity (unassessed) was undertaken within a stage 3 academic module studying Lean Innovation, a separate outcome of the process was also used to provide students with the opportunity to engage with and understand Design Thinking as a tool in itself, using a relevant and relatable problem. During the exercise, students also participated in focus groups to gather their thoughts and experiences. Design thinking has grown in momentum and adoption in recent years, with the flexibility afforded by the process ensuring a wide and broad application to a range of problems but with increasing use in education design [17–19]. At its heart and ethos is a human-centered approach to orient conversations towards a solutions based on participants insights [20]. There are 5 distinct steps of design thinking;: Empathize, Define, Ideate, Prototype and Test, arranged as a “double diamond” of divergence followed by convergence to obtain an in-depth exploration followed by steps towards defining a solution to a broad problem. In this study, the operational problem students were presented with was “Students are not engaging.”

3 Results The first stage, Empathize, sought to understand the student experience “asking the right questions, challenging assumptions” [16, p. 60], as to more fully understand the immediate experience of students in higher education post-pandemic. Empathy maps [21] were used to unpack the student perspective and serve to identify the gains and pains [22] of learners. Students in their groups completed a provided empathy map (Fig. 1) as collaborative activity, addressing each of the areas in turn.

Fig. 1 Provided Template

530

E. Tan

What was interesting was that students placed a value on attendance and engagement. The main gains and benefits they expected from these activities were in relation to improved well-being, establishing a positive “routine,” providing a sense of purpose and belonging and as means for them to stay in control of their studies (Table 1 and 2). Table 1 Empathizing Map Results Think and Feel

See

Say or Do

Hear

• Intimidating • Boring • Sense of Achievement • Disconnected • Apathetic • Broke • Tired • Busy • Overwhelmed • Stressed • Coasting • Fear of Missing out • Fun • Cold • Off Track • Disconnected • No Point • Demotivated • Lack of validation • Might have other priorities • Anxious • Pressures (Work and Parents) • Uncared for • Lack of reward • Friendship • Lazy

• Airpods (Sense of other students ignoring each other) • People • Full lecture theatres • Boring • Other people in groups not engaging • Silence • Strangers • Workplace • Friends • Library • Home • Empty Chairs • Modern • Huge Building

• Work (Employment part time jobs) • Work on the assessment • Prioritize the assessment • Go to the library • Don’t ask questions • Sports • Hungover • Decide to not walk into the lecture • Scan and Scram • Catch up online – I’ll just watch recap • “I’m ill” • Nothing –Inertia • Work to the assignment brief

• Fake promises • ‘It was boring’ • ‘You didn’t miss anything anyway’ • Nothing – silence • Formal warning • Disappointed comments • Phone alarms • Pressure from jobs • Excuses • Attendance emails – automated • Friends dissuading attendance • You’re wasting your money • Why didn’t you go? • Watch it online • Nobody else turned up

Using Design Thinking to Understand Student (Dis)engagement …

531

Table 2 Empathy Map Pains and Gains Responses Gains

Pains

• • • • • • • • • •

• • • • • • • • • • •

Get steps in Change of scenery and socialising Confidence Some challenge Out of bed and done something positive Beneficial Routine Meeting and seeing people Sense of control Guidance Lecture capture quality is variable

9 a.m Weather Commute Lack of support Lack of gain Speed Transport Inconvenience Hangovers Inconvenient timetabling Disruption of work

4 The Current Student Experience – Defining the Problem(s) 4.1 Disengagement and Isolation Some positive experiences such as the development of friendships and sense of purpose and challenge were shared by the students; however, the overwhelming picture given by students was one of disengagement. They shared an experience where students were disconnected, demotivated and apathetic towards their education. A key factor in this disengagement was that students identified that they had not had the opportunity build friendships within their course: “I didn’t even know your name until today.” Even when sessions such as seminars were held, they identified that students were disinclined to participate, stating that “no one really speaks.” In relation to their engagement with the school and attendance, they were unlikely to remain in the space to socialize, stating that “you come in for a lecture; then you leave again.” Lectures themselves were viewed and evaluated through the lens of lecture capture rather than as events themselves. Students did not simply evaluate a lecture on the basis of enjoyment or engagement. Rather they discussed how lecture capture provision should be considered when planning and delivering lectures. Their attendance decisions were based on whether the material would be best viewed “at 1.5 speed.” They also noted that “they [lecturers] will spend ages at the start of the lecture going over and recapping what they did last week. Like what’s the point? If I need that, I can just recap [watch the lecture capture] it.” Students reported that, at times, they felt ill equipped to undertake the actions required for active engagement and attendance with the experience being overwhelming. Lacking a secure social network within their cohorts, they shared how embarrassment would frequently deter students from attending lectures if they were late—“they’ll see the lecture has already started and won’t go in”—or they would experience a sense of isolation and unfamiliarity with peers. Students mentioned

532

E. Tan

how the lack of connection and community between students was frequently even visible: “You come into the building and everyone is just … by themselves with airpods [headphones] in, ignoring everyone.” They pointed to the lack of engagement by others in the cohort during lectures, and whilst acknowledging that staff attempted to engage students and to promote conversation, this was frequently unsuccessful and led to sessions lacking in interactivity and complaints that “other people” in sessions don’t engage.

4.2 Strategic Engagement and Established Practices Identifying competing pressures of part-time work, studying and socialising meant that, for many, they experienced a degree of stress. For several of the students, parttime work was an increasing necessity, due to cost of living. Several of the jobs listed included daytime work where students were required to negotiate with employers to determine shifts in advance, which severely impacted upon their ability to attend. As for others, who worked in night time economy jobs, they were frequently “too tired” from work the previous night to attend or make earlier morning lectures. Several students indicated that similarly they found the cost presented by using public transport another factor that they considered when making decisions regarding attendance. Whilst relatively low they stated that “if you’re doing it every day, it all adds up.” Overall, regarding transportation, they found that what they perceived as inconvenient or unreliable public transportation presented a significant deterrent to attendance. (It should be noted that, for the majority of students, the campus was approximately a forty-minute walk from their accommodation, or a fifteen-minute commute via public transport). Assessment was frequently listed by students as one of the main factors when making attendance decisions. Students, when making the cost-benefit analysis, frequently position assessment completion over attendance: “I’ll not go to a lecture because I’ve prioritized working on the assessment.” An extremely strategic approach was shared regarding assessment and engagement. Students reported that they “just worked to the assessment brief ” going through materials provided online. Unless there was an explicit link to the assessment to be completed, lectures and seminars were seemingly unrelated entities.

4.3 Trust, Relationship and Space in the University Elements of mistrust between the institution and students were mentioned in reference to their previous years’ learning experience of remote learning and what they saw as “Fake Promises.” Due to the changing nature of regulations regarding COVID19, provision of in-person teaching for students was unable to be provided. This

Using Design Thinking to Understand Student (Dis)engagement …

533

distrust of the institution was also reflected when asked about how they interpreted communication they received. Students did not believe that the messages they received were personalized—“the automated emails you get about attendance? I just ignore them”—and that any warnings, formal or otherwise, were seen meaningless as “nothing ever happens.” They also pointed out that they perceived there was an inconsistency in messaging: “Last year, we were told to stay home and watch it online, and this year, we’re told we have to go in” which also added to this distrust. They reported that they “got used to just watching lecture capture” and that many people “preferred” this method of engaging and lacked “self-regulation,” having never established a routine of attendance and engagement. A lack of what they perceived as compulsion was also a primary element in the problem, with automated communication from the university regarding attendance interpreted as a further reflection of the lack of care and support provided. Students themselves cited that they found the physical space of the institution difficult to navigate, at times feeling intimidating, and shared that they had “no space of our own” despite there being a cafe. The expense of the catering available in relation to supermarket prices—“it’s really expensive”—and the space used for this purpose to them suggested that “there’s nowhere that encourages you to stay.” Students unable to use this facility were frequently discouraged from remaining in the building, and once they had left, they were unlikely to return. Large gaps between sessions had an immediate impact upon attendance during later sessions as well.

5 Determining the Solution Students were tasked with identifying ways in which the problem could be addressed using divergent thinking (Table 3) and encouraged to be creative. An analysis of the solutions provided indicated that they fell into three distinct categories: 1) Creating value for students when attending, 2) Valuing students as individuals, and 3) Implementing a system of structure and support.

6 Discussion 6.1 Proposed Solution 1: Increased Support to Promote Engagement Students identified that a more robust support system regarding attendance was required to address the issue of a lack of self-regulation. Related to their assertions that students themselves had not developed the routines or habits to support attendance since studying remotely, students in this group identified that further structures of support were required, following up on lower attendances earlier and

534

E. Tan

Table 3 Solutions Proposed Value for Students Engaging

Value of Students

Support and Structure

• Nicer and bigger social areas • Sessions to be more engaging • Incentives to go to lectures (grade % point reward) • A choice of assessment method • Practical lectures • Ownership of spaces • Smaller groups • More of a campus • Broken up assessment periods • No compulsory group projects • More seminars less lectures • A greater number of lower stake assessments • More collaboration between students

• • • •

• Consequences for missing lectures • Remove lecture capture • Enforced attendance • Block teaching • Intensive periods • Compressed Timetabling

• • • • • • •

•

Free tea and coffee Napping Pods Free shuttle bus Choosing seminar times flexibly A 4-day week Free washing machines Closer- free travel Bar on the student floor Same timetable each week Pairing students together Incentives to stay in the building (less expensive catering) Options for online participation

more personally. A key finding of this study was that students who provided this solution indicated that they strongly believed a lack of attendance had what they termed a “domino effect.” They clearly linked a lack of attendance and engagement with consequences regarding a students’ ability to make social connections with peers and consequently deteriorating mental health and overall well-being as they “lost control,” “went off course” and became increasingly “disconnected” from peers. Secondly, in line with previous findings [10], they identified that lecture capture was a severe impediment to attendance. They understood the value of attendance as a fundamental component of their student experience but identified that they were easily dissuaded from engaging and placed too high a value and reliance on the provision for habitual rather than occasional use of lecture recordings. They indicated that it was frequently used by students as the rationale for nonattendance and confirmed that, even when a decision regarding attendance was made on the availability of the lecture recording, seldom did students really engage with this material. What was interesting was that, as Edwards and Clinton [11] noted, it had now for some students become the “preferred way” of attending lectures.

Using Design Thinking to Understand Student (Dis)engagement …

535

6.2 Proposed Solution 2: Create a More Flexible and Engaging Environment Students determined that the cost-benefit analysis undertaken when making attendance decisions was exacting. What was clear was that students have established using lecture capture as the habitual means of engagement. So that they sought more interactivity when determining the added “value” of attending sessions. At times, this was a financial calculation, determined by part time working [1, 2] and commute costs. Students wished for intimacy of experience during learning activities [23], with a desire for smaller groups [24] and a greater level of interaction and interactivity during sessions. A student comment that was particularly illuminated was that they “wished my [lecturers] knew my name”; others noted that they “missed school” referring to the opportunity to build relationships with frequent engagement with peers and staff.

7 Conclusion From the discussion with students, post-pandemic understanding of higher education presents questions for all involved regarding value. Technology—previously a peripheral component—has adopted a central role, with student practices having now evolved to the extent where this is the preferred mode of engagement. How we can meet student expectations and the feasibility of creating the intimacy of experience that is desired, and how trust and relationships can be rebuilt, is a conversation in which students must be active and engaged participants. Acknowledgements Thanks to the students who participated in this study and for so honestly and generously sharing their thoughts and experiences.

References 1. Kelly, G.E.: Lecture attendance rates at university and related factors. J. Furth. High. Educ. 36(1), 17–40 (2012). https://doi.org/10.1080/0309877X.2011.596196 2. Humphrey, R.: Pulling structured inequality into higher education: the impact of part-time working on English university students. High. Educ. Q. 60(3), 270–286 (2006). https://doi.org/ 10.1111/j.1468-2273.2006.00317.x 3. Woodfield, R., Jessop, D., McMillan, L.: Gender differences in undergraduate attendance rates. Stud. High. Educ. 31(1), 1–22 (2006). https://doi.org/10.1080/03075070500340127 4. Longhurst, R.J.: Why aren’t they here? Student absenteeism in a further education college. J. Furth. High. Educ. 23(1), 61–80 (1999). https://doi.org/10.1080/0309877990230106 5. Sloan, D., Manns, H., Mellor, A., Jeffries, M.: Factors influencing student non-attendance at formal teaching sessions. Stud. High. Educ. 1–14 (2019). https://doi.org/10.1080/03075079. 2019.1599849

536

E. Tan

6. Dolnicar, S., Kaiser, S., Matus, K., Vialle, W.: Can Australian universities take measures to increase the lecture attendance of marketing students? J. Mark. Educ. 31(3), 203–211 (2009). https://doi.org/10.1177/0273475309345202 7. Karnad, A.: Student use of recorded lectures: a report reviewing recent research into the use of lecture capture technology in higher education, and its impact on teaching methods and attendance, London School of Economics and Political Science, London (2013). http://eprints. lse.ac.uk/50929/1/Karnad_Student_use_recorded_2013_author.pdf 8. von Konsky, B.R., Ivins, J., Gribble, S.J.: Lecture attendance and web based lecture technologies: a comparison of student perceptions and usage patterns. Australasian J. Educ. Technol. 25(4) (2009). https://doi.org/10.14742/ajet.1130 9. Brooks, C., Epp, C.D., Logan, G., Greer, J.: The who, what, when, and why of lecture capture. In: Proceedings of the 1st International Conference on Learning Analytics and Knowledge, in LAK 2011. New York, NY, USA, pp. 86–92. ACM (2011). https://doi.org/10.1145/2090116. 2090128 10. Howard, E., Meehan, M., Parnell, A.: Live lectures or online videos: students’ resource choices in a first-year university mathematics module. Int. J. Math. Educ. Sci. Technol. 49(4), 530–553 (2018). https://doi.org/10.1080/0020739X.2017.1387943 11. Edwards, M.R., Clinton, M.E.: A study exploring the impact of lecture capture availability and lecture capture usage on student attendance and attainment. High Educ 77(3), 403–421 (2019). https://doi.org/10.1007/s10734-018-0275-9 12. Morris, N.P., Swinnerton, B., Coop, T.: Lecture recordings to support learning: a contested space between students and teachers. Comput. Educ. 140, 103604 (2019). https://doi.org/10. 1016/j.compedu.2019.103604 13. Baillie, L.D., Banow, R., Botterill, J.J.: The impact of lecture capture availability on academic performance in a large biomedical science course. Educ. Inf. Technol. 27(5), 7183–7203 (2022). https://doi.org/10.1007/s10639-022-10903-1 14. Moore, S., Armstrong, C., Pearson, J.: Lecture absenteeism among students in higher education: a valuable route to understanding student motivation. J. High. Educ. Policy Manage. 30(1), 15–24 (2008). https://doi.org/10.1080/13600800701457848 15. Henderson, M., Selwyn, N., Finger, G., Aston, R.: Students’ everyday engagement with digital technology in university: exploring patterns of use and “usefulness.” J. High. Educ. Policy Manage. 37(3), 308–319 (2015). https://doi.org/10.1080/1360080X.2015.1034424 16. Carroll, M.: Stretch, dream, and do - a 21st century design thinking & STEM journey. J. Res. STEM Educ. 1(1) Art. no. 1, (2015). https://doi.org/10.51355/jstem.2015.9 17. Noel, L.-A., Liu, T.L.: Using design thinking to create a new education paradigm for elementary level children for higher student engagement and success, presented at the Design Research Society Conference 2016, June 2016. https://doi.org/10.21606/drs.2016.200 18. Gunter, G.A., Kenny, R.F.: Using design thinking and formative assessment to create an experience economy in online classrooms. J. Form Des. Learn. 5(2), 79–88 (2021). https://doi.org/ 10.1007/s41686-021-00059-5 19. Bustard, J.R.T., Hsu, D.H., Fergie, R.: Design thinking innovation within the quadruple helix approach: a proposed framework to enhance student engagement through active learning in digital marketing pedagogy. J. Knowl. Econ. (2022). https://doi.org/10.1007/s13132-022-009 84-1 20. Brown, T., Katz, B.: Change by design. J. Prod. Innov. Manage. 28(3), 381–383 (2011). https:// doi.org/10.1111/j.1540-5885.2011.00806.x 21. Gray, D., Brown, S., Macanufo, J.: Gamestorming: A Playbook for Innovators, Rulebreakers, and Changemakers. O’Reilly Media, Inc., Sebastopol (2010) 22. Osterwalder, A., Pigneur, Y., Bernarda, G., Smith, A.: Value Proposition Design: How to Create Products and Services Customers Want. Wiley, Hoboken (2015) 23. Simpson, N.: Asynchronous access to conventional course delivery: a pilot project. Br. J. Edu. Technol. 37(4), 527–537 (2006). https://doi.org/10.1111/j.1467-8535.2006.00534.x 24. Oldfield, J., Rodwell, J., Curry, L., Marks, G.: Psychological and demographic predictors of undergraduate non-attendance at university lectures and seminars. J. Furth. High. Educ. 42(4), 509–523 (2018). https://doi.org/10.1080/0309877X.2017.1301404

So You Want to Be a Tutor? Professional Development and Scenario-Based Training for Adult Tutors Danielle R. Thomas , Shivang Gupta , Erin Gatz, Cindy Tipper, and Kenneth R. Koedinger

Abstract Tutoring is one of the most powerful academic interventions used to increase student achievement. In response, organizations are developing tutoring programs and discovering a common problem – a shortage of qualified and experienced adult tutors. We introduce Personalized Learning Squared (PLUS), a holistic tutoring platform designed to improve tutoring efficiency and workplace training. PLUS combines human tutors and artificial intelligence (AI)-powered math software to double math learning gains for low-income middle school students. In order to provide differentiated student support, tutors need to be trained in supporting math content and student socio-motivation. Currently, there is a wide range of experience among adult tutors, especially younger, part-time tutors. Not only do these tutors have varied initial skill levels, but there is also high turnover (40% annually) among such youth workers. PLUS specializes in providing efficient and low-cost tutor training while delivering situational experiences to inexperienced tutors. Through previous research, we isolated key competencies of successful tutoring, Social-Emotional Learning, Mastering Content, Advocacy, Relationships, Technology Tools (called our SMART framework) and developed synchronous, interactive training and, PLUShoused, asynchronous lessons. We have documented evidence of tutor learning gain with typical tutors performing ~20% better on the posttest compared to the pretest simulations and scenarios. Currently, we are optimizing asynchronous lesson design using learner sourced data to create real-life scenarios and authentically challenging multiple-choice tasks. In addition, this present work discusses (a) determining the required training components and lessons completed by tutors to demonstrate mastery and (b) designing a personalized pathway to PLUS tutor certification. Keywords Tutor Training · Mentor Training · Scenario-Based · Equitable Tutoring · Professional Development · Rapid Upskilling · E-learning

D. R. Thomas (B) · S. Gupta · E. Gatz · C. Tipper · K. R. Koedinger Carnegie Mellon University, Pittsburgh, PA 15213, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_44

537

538

D. R. Thomas et al.

1 Introduction Individualized tutoring is one of the most powerful academic interventions known to increase student achievement [8]. In response, organizations are developing tutoring programs and discovering a common problem—a shortage of qualified and experienced adult tutors. There is increasing demand for experienced and qualified tutors in public education both during the school day and to participate in after school programs. The post-pandemic educational landscape is barren of licensed teachers and qualified tutors ready to instruct and empower today’s students. Not only is there a shortage of qualified tutors, but there is also a wide range of experience among them, especially younger, part-time tutors, with a turnover rate of 40% annually [6]. PLUS—Personalized Learning Squared—specializes in providing efficient and lowcost tutor training while delivering situational experiences to inexperienced tutors. Historically marginalized students—such as those who are Black, Hispanic, experiencing poverty, or first-generation college students—are at the highest risk of not meeting annual growth gains compared to their peers, with the greatest declines in overall performance occurring in math, a subject students are struggling the most to demonstrate growth gains [12]. In recent years, the United States has lost twenty years of student progress in math among middle school students [12], with historically marginalized students being impacted the hardest. Tutors, in the context of this work, are defined as uncertificated paraprofessionals who provide academic, socio-motivational support, and guidance to students. This work introduces PLUS, a holistic tutoring platform designed to improve tutoring efficiency and workplace training [7]. The PLUS human-computer teaming platform provides three solutions to increase tutoring performance and efficiency which will be covered in detail: (a) Training, focusing on personalized professional development for tutors of all ages, experiences, and backgrounds; (b) Toolkit, engaging tutors and students in tandem through the use of math software to provide progress monitoring and goal setting; (c) Tutoring, developing a central corps of trained and certified tutors ready to be deployed to schools, both in-person and remotely. In addition, we discuss the evidence of tutor learning from our scenariobased training program, which employs a hybrid approach of using asynchronous lessons and synchronous live training designed to meet the needs of adult tutors of varying skills and experiences. Future work focuses on how to determine tutor demonstration of mastery and the process of designing a personalized pathway to PLUS tutor certification.

2 The PLUS Approach PLUS is an award-winning math tutoring program developed by Carnegie Mellon University in collaboration with Carnegie Learning and Stanford University. PLUS combines human tutoring with cutting edge technology to personalize learning for

So You Want to Be a Tutor? Professional Development …

539

each student. The PLUS mission is to double the rate of math learning for participating students. PLUS does this by increasing the number of learning opportunities students have by increasing access to quality learning opportunities through AI-assisting technology. In addition, PLUS attends to both students’ math content and socio-motivational needs, through tutor training based on our research-driven SMART framework, which encompasses strategic areas of successful competencies of tutoring: Social-Emotional Learning; Mastering Content; Advocacy; Relationships; Technology. The combination of high-quality tutor training and AI-assisted technology to determine students’ immediate needs, makes PLUS an efficient and effective tutoring platform to increase student learning and self-efficacy [1–3].

2.1 History and Transformation of PLUS Introduced in a previous work [10], the PLUS system was originally referred to as PL2 , referencing its ability to improve the learning outcomes for marginalized students by leveraging, or “squaring,” the power of both human and computer tutoring. Early iterations of the system possessed two key differences from how it is being used today. First, the term “mentor” was used instead of “tutor” to emphasize the role of providing socio-motivational support through attention to advocacy and relationship building, in addition to, academic support evident when referencing tutoring. Second, early iterations of the platform involved its ability to improve mentoring efficiency by primarily focusing on its role in providing resource recommendations based on student math performance (determined from math software and educational technology) and tutor inputs and feedback. PL2 was mainly used as an AI-assisted, resource recommender system for mentors [10]. Since then, PLUS has undergone significant development, transforming into a comprehensive tutoring system, leveraging a three-pronged approach to increase tutoring efficiency. This approach includes three solutions to provide quality personalized instruction for students: (a) tutor training, (b) personalization using math software, and (c) and the creation of an in-house tutoring corps. While this work focuses on the development and impact of PLUS Training, we will briefly discuss PLUS Toolkit and PLUS Tutoring as well. By adopting this multi-pronged approach, PLUS has transformed from an AI-assisted resource recommender system for mentors to a versatile tutoring platform intended to meet the diverse needs of schools and organizations.

540

D. R. Thomas et al.

2.2 PLUS Training The PLUS Training program is based on research by scientists and researchers at Carnegie Mellon University. The evidence-based program covers 15 different competencies, across five strategic areas that investigations showed are needed for educators and tutors to be successful. Initially presented in [1], the five research-identified areas of tutor competency, include: Social-Emotional Learning, Mastering Content, Advocacy, Relationships, Technology Tools. The SMART framework was developed by surveying PLUS partner members, comprising tutors and tutor supervisors, to determine the most important skills of successful tutors [1, 2]. The most important tutoring skills were found to be the ability to Engage and Motivate Students, Demonstrate Awareness of Bias, and Build Relationships with Students, spawning the schematic categorization of the SMART framework [1, 2]. The PLUS training program consists of asynchronous, scenario-based lessons and synchronous, interactive sessions to reach the diverse experiences and needs of all tutors. Asynchronous lessons are brief, taking the typical tutor less than fifteen minutes to complete. Applying an interactive design process, tutors practice responding to a training scenario, that doubles as a pretest, asking tutors to predict and explain the best approach for responding to a student in a given scenario (ex: a student struggling to stay motivated). Tutors receive feedback on their responses and observe the research-recommended approach to reply to a student in the given situation. Then tutors respond to a transfer scenario, also known as a posttest. The change in tutor performance from the training (pretest) to the transfer (posttest) scenario is then analyzed to determine tutor learning gain [11]. Figure 1 below provides an example of a training scenario for the Giving Effective Praise lesson illustrating a constructed-response question, followed by a selected-response question. PLUS asynchronous lessons require tutors to predict and explain the best approach, then observe the research-recommended response, followed by prompting tutors to explain their thoughts and agreements with the research-supported approach. This lesson design is modified from the Gibbs’ Reflective cycle [4] and is thoroughly explained in [11]. Presently, PLUS contains sixteen asynchronous lessons on recent, in-demand topics, such as Narrowing Opportunity Gaps, Addressing Microaggressions, and Exploring Implicit Bias which fall within Advocacy in the SMART framework. PLUS contains five synchronous lessons aligned with each area of the SMART framework. Each lesson comprises similar session components: synchronous scenario-based training; role-playing and situational-based activities; socialemotional learning activities tutors can directly use with students; and reflection, with all lessons concluding with group share out and collaborative discussion. Live, interactive synchronous lessons are strategically designed to prepare tutors and educators with a toolkit of strategies aligned with each SMART competency area. For example, within the Social-Emotional Learning synchronous lesson, tutors practice applying strategies to assist students facing three types of motivational challenges within the following scenarios: a student not able to maintain focus, concentrate, or “sit still”

So You Want to Be a Tutor? Professional Development …

541

Fig. 1 An example of a training scenario from the Giving Effective Praise lesson asking tutors to best respond to a student struggling with staying motivated. Notice question 1 prompting a tutor to predict how to respond via constructed response, and question 2 asking a tutor to select the best approach among multiple choice responses

(passive disengagement); a student who is bored with tasks completed, suggestive of exceeding academic goals (passive disengagement); and a student acting out by being disruptive or refusing to participate (active disengagement). Tutors practice responding to these common tutoring situations dealing with varying levels of student engagement. Both synchronous and asynchronous lessons can be used by tutor supervisors and organization administrators in tandem to ensure effective training and preparation of tutors. Often asynchronous lessons pertaining to topics covered during the live, synchronous lessons are assigned by tutor supervisors either before a synchronous training to allow for frontloading of material or foster group discussion during the related synchronous training. Other times, asynchronous lessons are assigned after the corresponding synchronous lesson to enrich and expand on tutor learning. PLUS Training also features tools for tutoring program administrators to allow them to get advanced data on how their tutors are performing on the lessons in the app. By looking at tutors scores and ratings, administrative staff can determine areas of improvement for groups and individual tutors. The PLUS tutor training program personalizes learning for tutors by allowing administrators and tutor supervisors to then assign specific lessons based on the data observed as well as the needs they identify from observation. To support further improvement of these asynchronous lessons, and to enable ongoing research by experts and learning scientists, the PLUS app also records a real-time log of all tutor interactions within a lesson such as buttons clicked and time spent on different components of the lesson. This log is then stored in DataShop, CMU’s open learning data analysis service.

542

D. R. Thomas et al.

In addition to synchronous and asynchronous training materials, the PLUS research team has also curated resources from around the web covering the entire range of 16 identified competencies. These resources include free videos, online courses, specially designed growth mindset and motivational resources and much more. The resources and strategies are provided to tutors in a central repository, allowing them quickly find what they need. PLUS Toolkit data is also used to suggest resources to tutors based on the individual needs of their specific students.

2.3 PLUS Toolkit In early iterations, the PLUS toolkit primarily focused on AI-assisted resource recommendations for tutors. For example, a tutor provides feedback in the app after a tutoring session and states a student is struggling with staying motivated. The tutor then receives a suggested resource related to motivation and engagement, such as a video of Michael Jordan discussing how he missed more basketball shots than he made. The video resource would contain question prompts for the tutor and student and be aligned with students’ personalized needs. Although the AI-assisted resource recommender system is still being used, PLUS has segued to tutor training personalization to create a multidimensional tutor support system. The administrative dashboard (partially shown in Fig. 2) displays individual and collective data on which tutors have completed which lesson, how much time was spent, lesson performance metrics, and tutors’ personal ratings and feedback from the lessons. Administrators can assign specific lessons to individual tutors or to the whole group based on tutor performance metrics, lesson performance, or observation while tutoring.

2.4 PLUS Tutoring Recent changes due to the ramifications of the COVID-19 pandemic have caused a nationwide shortage of qualified teachers and tutors [9]. PLUS has approached this obstacle by developing a strategic plan in recruiting quality tutors. Based on a model proposed by [8], we designed a tutor recruitment plan targeting undergraduate college students, those interested in pursuing a career in education, and part-time workers. Recruitment of college students is often easier because of their need for volunteer hours for graduate applications, internships, and resume building. As a benefit to organizations, many college students work as volunteers and receive mentoring certifications, participate in training, gain valuable experience working with children, and contribute to the community. College-level tutors can also benefit from the tutoring process by gaining stronger content knowledge and developing social connectedness and confidence [5]. Necessary tutor qualifications include: English speaking, reading, and writing skills; a willingness to work with children; and basic math skills. Fortunately, many college students today are already competent in math

So You Want to Be a Tutor? Professional Development …

543

Fig. 2 A partial view of the administrator dashboard displaying: tutor needs based on the SMART framework for a group of tutors; average lesson completion and accuracy rates; and average time spent on lessons. Specific lessons are shown under each competency with tutors’ collective performance metrics (i.e., average accuracy, rating, and time spent). Notice administrators can assign lessons to individual tutors or the group based on performance metrics

and possess the ability to tutor through the middle school level, oftentimes matching the math skills of professionally-trained tutors.

3 Evidence of Tutor Learning Evidence of Tutor Learning. We have promising evidence of tutor learning from our scenario-based, asynchronous lessons with tutors predicted to perform significantly better at posttest compared to pretest [11]. Using a mixed methods approach we focused on the impact of short (~15 min.), online lessons in which tutors participate in situational judgment tests based on everyday tutoring scenarios. We created three lessons on strategies for supporting student engagement and motivation related

544

D. R. Thomas et al.

to the prior findings of [1], which found tutors valued engaging and motivating students as an important competency. The three lessons used in the study to determine tutor learning outcomes and their descriptions are described below: • Giving Effective Praise- Tutors practice responding to students to increase their engagement and motivation to learn. Tutors apply strategies by responding to students by giving effective feedback and praise. • Learning What Students Know- Tutors practice meeting students where they are in their learning by determining what they know and what they need to learn. Tutors apply strategies to assess student’s prior knowledge. • Reacting to Errors- Tutors practice responding to a student making an error to increase motivation and engagement. Tutors apply strategies by effectively responding to a student making an error. A total of 80 tutors from a national, online tutoring organization completed between one to three of the lessons, and we analyzed their pretest to posttest performance. In [11], a mixed-effects logistic regression model was used to investigate the learning effect on tutors. Across all three lessons, tutors improved in their ability to respond to common tutoring scenarios. Figure 3 illustrates the pre- to post-instruction learning gains for tutors when challenged with applying three tutoring skills in the respective lessons: giving a student effective praise to encourage motivation (Giving Effective Praise); determining student prior knowledge or what they already know (Determining What Students Know); and reacting to a student who has recently made an error (Reacting to Errors). The results showed that there was a statistically significant increase in tutor performance post-instruction, with a 20% improvement compared to pre-instruction (β = 0.811, p < 0.01). In the posttest, tutors scored ~30% higher on selected-response questions compared to constructed response questions. It was noted that tutors are able to

Fig. 3 Pre- and post-instruction learning gains for tutors on three PLUS lessons illustrating substantial tutor improvement in applying learned strategies on how to best respond to common tutoring scenarios

So You Want to Be a Tutor? Professional Development …

545

learn effectively from selected-response questions alone. The study used learning analytics and qualitative feedback from tutors and determined several design modifications: create scenarios that are more challenging and authentic; use learner sourced data, or prior tutor response data, to identify common misconceptions; and vary the modality of scenario delivery, such as using video or audio, instead of tutors reading text scenarios [11]. We are interested in determining tutor transfer of learning by analyzing tutors situational responses to students in given situations as a function of participating in our PLUS asynchronous lessons and, in particular, far transfer of tutor learning. In other words, if tutors are applying what they learned from our lessons, how long does the impact last? This information will assist PLUS researchers with determining tutor training requirements and the level of need for on-going training.

4 Discussion The prescriptive combination of asynchronous, scenario-based lessons and synchronous training comprising the PLUS Training is strategically designed to allow for personalization by tutors. Recent advancements of the administrator dashboard within the PLUS Toolkit leverage tutor individual and collective performance metrics to give tutor supervisors and program administrators’ autonomy and allow for personalization across organizations. As we expand our comprehensive tutor training program, there are two key areas of ongoing improvement and future work: optimization of asynchronous lesson design to increase tutor efficiency and improve learning gains; and the development of a pathway to accreditation, which personalizes tutor training to ensure tutors receive enrichment of skills they show competence and refinement of skills they may need additional support. Both further endeavors are discussed in detail below. Asynchronous Lesson Optimization. Currently, we are optimizing asynchronous lesson design using learner sourced data to create real-life scenarios and authentically challenging multiple-choice tasks [11]. In other words, we are taking incorrectly constructed response statements provided by tutors and repurposing them as multiplechoice options. For example, a common incorrect tutor response on how to respond to students who have recently made an error (from Responding to Errors lesson) is to call attention to the student’s error, sometimes pointing out the mishap by calling it a “small mistake” or “common error.” We have taken these incorrect real tutor responses and made them multiple-choice options [11]. The use of learner sourcing can be leveraged to optimize the asynchronous lessons, making them more authentically challenging by capturing real tutor misconceptions and common errors [13]. We are also optimizing lessons by: varying the modality of scenario delivery by providing video- and audio-delivered scenarios; providing corrective feedback, explaining why a tutor’s response is correct or incorrect; and ensuring scenarios deal with math content knowledge that all tutors have the ability of understanding [11].

546

D. R. Thomas et al.

4.1 Mastery Development and Pathway Accreditation Presently, we are developing a method of determining tutor competency completion for purposes of gauging mastery. Effective methods of assessing mastery are an important component to developing a pathway to tutor accreditation. The new administrator view or dashboard within PLUS allows tutor supervisors and educational leaders to view data regarding their tutors’ performance on the PLUS lessons allowing for personalization towards a pathway to accreditation. Future research and development exploration includes: What level of on-the-job competency is needed to conclude mastery? Subsequently, does tutor proficiency level or completion accuracy within our asynchronous lessons align with preserved mastery in the field (see Fig. 2)? Are some areas of the SMART framework more important or certain asynchronous lessons more crucial than other lessons when best preparing and training tutors? What does it mean to be SMART certified? Research is being conducted to further determine and answer these questions to ensure the most impactful, scenario-based training for adult tutors.

5 Conclusion In this work, we discuss the importance and urgency to increase the number of qualified and experienced tutors to assist today’s students with learning math— a subject students are struggling to make significant learning gains in since the COVID19 pandemic. We introduce PLUS, a comprehensive tutoring system designed to improve tutoring efficiency and on-the-job training for tutors. Leveraging AI-based software and human tutors, PLUS aims to double student learning gains through the use of three solutions which are explained in detail: (a) Training, focusing on personalized professional development for tutors of all ages, experiences, and backgrounds; (b) Toolkit, engaging tutors and students in tandem through the use of the multidimensional support system via a tutor and administrator dashboard to allow for individual tutor- and organization-based training personalization; and (c) Tutoring, developing our own corps of trained and certified tutors ready to be deployed to schools, both in-person and remote. We discuss interest in determining transfer of learning by analyzing tutors modifying strategies when working with students as a function of participating in our lessons. Other future work involves building on the administrator dashboard and tutor performance metrics for the research and development of a pathway to accreditation and assessment of tutor mastery. PLUS believes anyone can be a tutor with the right personalized training and opportunities to gain authentic tutoring practice and experience. Acknowledgements This work is supported with funding from the Chan Zuckerberg Initiative (Grant #2018-193694), Richard King Mellon Foundation (Grant #10851), Bill and Melinda Gates Foundation, and the Heinz Endowments (E6291). Any opinions, findings, and conclusions expressed in this material are those of the authors.

So You Want to Be a Tutor? Professional Development …

547

References 1. Chhabra, P., Chine, D.R., Adeniran, A., Gupta, S., Koedinger, K.R.: An evaluation of perceptions regarding mentor competencies for technology-based personalized learning. In E. Langran (ed.), Proceedings of Society for Information Technology & Teacher Education International Conference, pp. 1812–1817. San Diego, CA, United States, Association for the Advancement of Computing in Education (AACE) (2022) 2. Chine, D.R., Chhabra, P., Adeniran, A., Gupta, S., Koedinger, K.R.: Development of scenariobased mentor lessons: an iterative design process for training at scale. In: Proceedings of the Ninth ACM Conference on Learning@Scale (2022) 3. Chine, D.R., et al.: Scenario-based training and on-the-job support for equitable mentoring. In: Guralnick, D., Auer, M.E., Poce, A. (eds.) Innovative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education. TLIC 2022. LNNS, vol. 581, pp. 581–592. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-21569-8_54 4. Gibbs, G.: Learning by doing: A guide to teaching and learning methods. Further Education Unit (1998) 5. Fuligni, A.J.: The need to contribute during adolescence. Perspect. Psychol. Sci. 14(3), 331–343 (2019) 6. Next Generation Youth Work Coalition. The case for investing in America’s youth workers. National Institute on Out-of-School time (NIOST). http://www.niost.org/pdf/Youth_Worker_ Case_Statement_March_2010.pdf (2010) 7. PLUS-personalized learning squared. Accessed 22 May 2023. https://tutors.plus/ (2023) 8. Robinson, C.D., Kraft, M.A., Loeb, S., Schueler, B.E.: Accelerating student learning with highdosage tutoring. EdResearch for Recovery Design Principles Series. EdResearch for Recovery Project (2021) 9. Rosenberg, M.S., Mason-Williams, L., Kimmel, L., Sindelar, P.T.: Addressing teacher shortages in the COVID-19 landscape: viewing teacher candidates as assets. Excelsior: Leadersh. Teach. Learn. 13(2), 86–95 (2021) 10. Schaldenbrand, P., et al.:. Computer-supported human mentoring for personalized and equitable math learning. In: Roll, I., McNamara, D., Sosnovsky, S., Luckin, R., Dimitrova, V. (eds.) Artificial Intelligence in Education. AIED 2021. LNCS, vol. 12749, pp. 308–313. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-78270-2_55 11. Thomas, D.R., Yang, X., Gupta, S., Adeniran, A., McLaughlin, E.A., Koedinger, K.R.: When the student becomes the student: design and evaluation of efficient scenario-based lessons for tutors. In: LAK23: 13th International Learning Analytics and Knowledge Conference (LAK 2023), 13–17 March 2023, Arlington, TX, USA. ACM, New York, NY, USA (2023) 12. U.S. Department of Education. Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2022 Math Assessment (2022) 13. Wang, X., Talluri, S.T., Rose, C., Koedinger, K. R.: UpGrade: sourcing student open-ended solutions to create scalable learning opportunities. In: Proceedings of the Sixth ACM Conference on Learning@Scale, pp. 1–10 (2019)

Fostering Inclusion and Well-Being Through Digital Language Learning in Museum Contexts Maria Tolaini

Abstract Inclusive foreign language learning is one of the key elements for promoting the inclusion and well-being of the individual and the community. The European Union suggests different strategies to improve the general levels of linguistic competences of all learners, including operating in informal contexts such as museums. Thanks to their informal, flexible, and situated learning and their role as inclusion and well-being facilitators, museums can enhance learning efficiency. However, given that inclusive education is at the core of the general improvement of society’s inclusion and well-being, it appears that the issue of inclusion is still scarcely addressed in the field of museum language learning. A possible way to design more inclusive museum paths aimed at enhancing linguistic skills is to introduce digital tools. After presenting the benefits in terms of the inclusion of such means and examining some examples of their non-inclusive application in museum language learning contexts to highlight their unexpressed potential, future implications based on the theoretical framework outlined will be discussed. Keywords Museum · Language Learning · Inclusion

1 Multilingual Competence: Its Importance and How to Enhance It 1.1 Education and Multilingual Competence for the Improvement of Society “Everyone has the right to quality and inclusive education, training, and life-long learning in order to maintain and acquire skills that enable them to participate fully in society and manage successfully transitions in the labor market” [1]. M. Tolaini (B) University of Genoa, Via Balbi 5, 16126 Genova, Italy e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_45

549

550

M. Tolaini

The first article of the European Pillar of Social Rights stresses the importance of providing adequate education and training to everyone. Access to educational opportunities is fundamental to developing the individual’s potential and allowing them to fully integrate into society, which leads to the improvement of the single and of the community’s inclusion and well-being. At the core of this idea lies the need of ensuring that everyone can benefit from proper education and training, particularly individuals with special educational needs, those with disabilities, or those from disadvantaged backgrounds [2]. To concretize the right to quality and inclusive education, the European Union published several recommendations, providing indications, strategies, and solutions to achieve this mission [3–6]. One of the most crucial documents is the Council’s Recommendation on Key Competences for Lifelong Learning [4]. Published in 2006, the document has been updated in 2018 in order to face today society’s challenges. By identifying eight competences that need to be developed through the course of a person’s life, and by proposing ways in which these skills can be acquired, the European Commission defines a framework to facilitate and achieve universal education and thus improve society. Assuming that all eight competences are equally important and are at the same time related to each other [4], the present research will focus on the issues around the development of multilingual competence, specifically the first foreign language competence. Similarly to the other competences, language learning competence enables people to take an active part in society, thus improving their and their society’s well-being and inclusion. In particular, multilingual competence is essential for this twofold purpose: on one side, it increases the satisfaction of the individual and expands the possibilities of traveling, learning, training, and working [4]; on the other side, it fosters intercultural understanding, enables the development of democratic competences of active citizenship, increases social cohesion and promotes greater social participation [5].

1.2 The Unsatisfactory Level of Multilingual Competence Given the fundamental contribution of multilingual competence to the individual and to society’s inclusion and well-being, the low levels of English proficiency in upper secondary school students at the end of their educational cycle are alarming. In 2005, the European Commission planned an inquiry on language competences, which was conducted in 2011 and published in 2012. Despite its outdated release date and its partial cover (14 European countries responded to the survey), it is the most complete analysis of language competence at a European level. Based on the data collected, it comes to the conclusion that “language competences need to be considerably enhanced” [7]. The data show that the level of independent users, corresponding to B1 and B2 levels [8], of first foreign language, which for the majority of students is English [7], was only attained by 42% of the students. Regarding the Italian context, data gathered nationally present a similar picture. The INVALSI, a

Fostering Inclusion and Well-Being Through Digital …

551

yearly report conducted by the National Institute for the Evaluation of the Education and Training System that evaluates students’ Italian, Mathematics, and English skills, revealed that the average Italian upper secondary student does not reach the required B2 level at the end of the educational cycle [9]. These individuals are in fact those who will soon continue their training or enter the job market, and thus need to be equipped with the adequate tools to reach their full potential.

1.3 Possible Solutions to Bridge the Gap In its role as a guiding reference for social issues, the European Union suggests numerous strategies and indicates some directions that can be taken to improve the general levels of language skills. A fundamental concept at the core of all European Union recommendations is that learning should be tailor-made and inclusive. According to the principle of universal inclusive education, one of the strategies indicated by the European Union is the employment of multiple and diverse approaches. Language skills can be enhanced by exploiting different methods, such as arts education, inquiry-based learning, and games, to increase motivation and engagement in learning [4]. Another quite beneficial approach to language learning suggested by the European Council is Content and Language Integrated Learning (CLIL) [5]. CLIL is a methodology that emerged in the 1990s that consists of teaching a subject in a foreign language, for example teaching Chemistry in English in an Italian school [10]. It is now extensively used as it was proven to greatly improve foreign language skills: it increases students’ motivation and challenges their cognitive skills as they are required to learn not only the target language but information about a different subject as well [11]. Great attention is also devoted to informal learning. Blending and integrating classroom learning with other non-formal and informal learning environments give the students the opportunity to experience more engaging learning contexts, enrich the uptake of languages, and improve and innovate teaching practice [12]. Partnerships between formal, informal, and non-formal institutions foster lifelong learning and skills development and facilitate the transition from school to work. Based on the UE propositions, this paper intends to examine the benefits of museums’ informal learning environments to promote inclusive language learning.

2 Language Learning in the Museums 2.1 Museum Learning The collaboration between museums and formal institutions, such as schools, can provide great benefits to the students’ progress in learning languages.

552

M. Tolaini

Indeed, museums are considered informal education settings as they do not present a fixed curriculum that progresses by degrees of difficulty, do not require compulsory attendance, and do not deliver a certificate of knowledge or competence at the end of the experience [13]. Several advantages characterize the experience of informal learning in the museum. Firstly, the absence of control facilitates the learning process [14]. Secondly, numerous researchers produced extensive evidence of the positive impact of museums on the development of transversal competences [15– 17], such as collaboration, [18] critical thinking, [19] creativity, and intercultural skills [20]. Thirdly, the presence of objects facilitates a multisensory experience [20] and increases the attention and motivation of the students [21]. Objects indeed offer a connection to the real world and favor the creation of a link between learning content and daily life [22]. The connection with the real world and with society is not only obtained by the materiality of the objects. In recent times museums have taken the role of social inclusion and well-being facilitators of a more accessible, open, and inclusive society, and by leveraging their unique settings and their collections, they promote well-being for the individual and the society [23, 24].

2.2 Language Learning in the Museum Given the evident benefits of learning in the museum, research on language learning in museum contexts has been conducted since the late 1990s. In the Anglo-Saxon world, studies investigated the impact of museum education programs on the language competence of English for Speakers of Other Languages (ESOL). In the United States of America [25] and Australia [26], English writing and English discussion workshops have been implemented in museums for students whose first language was not English. In England, the University of Nottingham has produced self-access learning materials for ESOL visitors to use during their visits to Lincolnshire’s flagship Museum of Archaeology in Lincoln, UK [27]. Certainly, slightly different is the case of countries where English is the first or second foreign language. Research has been carried out worldwide, from Kuwait [28] to Singapore [29], from Korea [19] to Spain [30] and Italy [31], and many other countries. The methodologies vary; however, usually, a visit to the museum is followed by a workshop or a post-activity. The workshop might consist of different activities: compiling a narrative journal, completing a treasure hunt with a mobile application, writing a comparative essay, compiling postcards in groups, or creating a virtual museum with a software. Various evaluation tools have been used: questionnaires, focus groups, semi-structured interviews, and the evaluation of the materials produced by the attendants. Although the experiences are characterized by a great variety of elements, the outcomes are similar: after attending such workshops, language skills, especially speaking and writing competences, but also transversal skills, such as collaboration, creativity, and digital skills, are improved. In order to highlight the advantages and point out the limits of examples in which the museum environment is used to promote language skills, two studies are examined.

Fostering Inclusion and Well-Being Through Digital …

553

2.3 Case Studies of Language Learning in Museum Contexts In Italy, a partnership between University, the Natural History Museum in Venice, and scientific, linguistic, and artistic high schools investigated “secondary school students’ attitudes towards the integration of CLIL and museum-based pedagogies” [31]. 284 Italian high schoolers of A2-B1 English level attended an in-class activity (3 h), then a CLIL museum visit (2 h), and finally a post-activity in class (4 h). During the first encounter, the activities were presented and topics and vocabulary concerning the museum visit were discussed. In the follow-up stage, students worked on a poster that they presented in class. Students compiled a survey on their learning attitudes and perceived learning outcomes and 40 selected students took part in focus groups (7 sessions). Students showed a positive attitude towards the use of CLIL in museums and the use of English outside school. The artistic high schoolers showed a less positive attitude: according to the researchers, this was probably due to the scientific focus of the activity, which was perceived as less adherent to their scholastic curriculum. The second example of a language learning experience in the museum is set in Australia [26]. In total, 240 ESL students of different provenance (such as Pakistan, Vietnam, India, and other countries) attended museum laboratories aimed at improving their language skills, which were already of intermediate and advanced levels. Participants were invited to look at art objects at the Ian Potter Museum of Arts and use Visual Thinking Strategies to analyze the works of art. While they were interpreting the artworks, the guide would provide contextual information. After that, students had to discuss the artwork in pairs, or groups using topic cards, or write a text from the perspective of a character in the artwork. Two questionnaires were administered: one for the students and one for the teachers. In the self-evaluation survey, students felt they made progress in terms of English language mastery: 87% of the respondents indicated the worksheets, the material provided and used at the museum, helped them improve their English. These two case studies were considered the most relevant examples of language learning in museums targeting students (and not only educators as in other cases [30]). Numerous museums nowadays offer language learning activities, but as is often the case for museum learning actions, those workshops are singular events, involving a small number of attendants and generally lack scientific foundations [32]. Even in published research in this field, it is common to encounter reports of carried out laboratories that are not accompanied by empirical data supporting the validity of their outcomes. The first case study presented a scientific structure, involved a great number of students, reported the methodology adopted, and provided accurate statistical analysis of quantitative and qualitative data. The second study also involved a great number of learners, described the phases of the activities, and focused more on qualitative data as it analyzed the replies of the students and of the teachers. Both papers confirmed the advantages of learning languages in the museum. However, from the discussion of those two case studies, two urgent aspects still need to be

554

M. Tolaini

addressed. The first aspect is the actual improvement of language skills. Both papers asked students to self-evaluate their skills. Although the results were encouraging in both cases, objective criteria to assess substantial language advancement are needed. The second element is that neither of these studies considered inclusive learning tools from the perspective of educational universal design. As previously stated, one of the main goals of education is to provide every person with the adequate means to become an active and conscious citizen and contribute to the inclusion and well-being of society. To do so, it is vital that everyone, regardless of their educational needs, their disabilities, and their socio-economic backgrounds, is put in the condition to have free access to learning and training opportunities. While museums promote social equality, and while growing attention is paid to inclusiveness in language education, inclusive language learning in museums needs to be addressed. And in this field, digitalization has a key role to play.

3 The Road to Inclusion 3.1 Digital Tools for Inclusion Education needs to be shaped in a way to ensure that all groups and individuals receive actual equal access to quality learning opportunities in order to thrive in the world [2]. The European Union has profusely spoken on this subject, suggesting various possible approaches to address this issue [2, 33–35]. This paper intends to discuss the use of digital tools to foster inclusive learning and guarantee accessible education to everyone with the prospect of improving society’s inclusion and wellbeing [35–38]. Before describing the main uses and advantages of digital tools, it is useful to discuss a few concepts. Firstly, it is fundamental to acknowledge that “Information and communication technology (ICT) is a tool rather than an end in itself or a universal solution” [36]. When crafting digital inclusive tools, the approach should be led by pedagogical strategies that focus on the needs and characteristics of the learners rather than on the innovative features of digital technology. Secondly, as technology is “no universal solution,” it can be considered a possible element of the universal design approach [36]. By “universal design,” it is meant the design of products, environments, programs, and services that can be enjoyed and used by every person, without the need to adapt them and without excluding the possibility of integrating assistive devices [39]. Thirdly, to maximize the results of the universal design approach, digital gap issues, in terms of “digital infrastructure, connectivity, and access to digital devices, equipment, resources and to digital skills, as well as their accessibility for those with disabilities” must be addressed in parallel to avoid the exclusion of certain learner categories [2]. The advantages of digital tools in terms of inclusion are numerous and welldocumented in the literature. Thanks to their adaptable, personalized, and suitable characteristics [2], they foster learners’ motivation and deepen their engagement in

Fostering Inclusion and Well-Being Through Digital …

555

the learning process [40], increase collaboration, promote more on-task behavior and better conceptual understanding [41], and enhance creativity [42]. The Inclusive Digital Education report [43] describes the current digital technologies adopted in the education field and presents the benefits of their use in terms of inclusion. For example, Augmented Reality and Virtual Reality are proven to help develop reading comprehension skills, acquire vocabulary [44], and foster social, emotional, and affective skills [45]. In the case of Artificial Intelligence, an adaptive e-learning system that provides personalized learning services and study materials for all learners can be designed [46]. Robotics in inclusive education can increase motivation in ASD learners [47] and reduce loneliness and isolation in students that are hospitalized [48]. Learning analytics (LA) may also support inclusive learning environments: by collecting learners’ data, LA can support inclusive education as it allows creating personalized learning profiles [49]. Gamification and mobile learning present numerous advantages as well [43]. These technologies are widely adopted in diverse settings and for different purposes, in museum language learning as well. To understand the actual and the unexpressed potential of these tools in such a context, two case studies are discussed below.

3.2 Digital Language Learning in Museums In Singapore, an empirical study on the design of a virtual museum within the context of an English language curriculum was conducted [29]. The aims of the research were to enhance students’ multimodal awareness and stimulate imagination and creativity. In total, 158 12- to 13-year-old students attended the project for a period of 10 to 12 weeks. Firstly, they took part in a “real world” museum visit. Then they conceptualized and designed a virtual museum: they created the virtual rooms, inserted, and cataloged objects, and wrote captions and labels. Students then presented their work and discussed it with their peers. Regarding language skills, they expanded their vocabulary and trained their oral skills. Data were collected through semi-structured interviews, reflections, notes of classroom observation and student-generated products, surveys, and questionnaires. Although language learning goals were not the main focus of the research, attitudinal and motivational changes to EL lessons were reported by teachers. From the questionnaire, it emerged that students were keener and felt more confident in speaking out loud. In general, the development of collaborative learning skills and the increase of language learning motivation were evident. In Italy, a study on the use of a mobile application to improve language and key competences in class and in the museum has been carried out [50]. French students learning Italian (A2–B2 levels) attended an Italian language course and took part in the project. In class, the teacher contextualized the topic and introduced linguistic aspects. During the visits to three different museums (Galleria Internazionale di Arte Moderna, Museo del Settecento Veneziano e Collezione Peggy Guggenheim),

556

M. Tolaini

students were engaged in experiential tasks and interacted with museum resources. Then, in class, language competences were reinforced, and the instructions to elaborate products with the izi.TRAVEL application were illustrated. Using the mobile app, students created multimedia tours with a digital-storytelling approach. According to the researcher, the museum environment allowed the learners to use the foreign language in a creative, meaningful, and authentic way. Informal feedback revealed that students enjoyed sharing knowledge, acquiring, and reinforcing competences, and feeling active users of cultural heritage. As previously pointed out, only a few examples of scientific-based research in language learning in museums are to be found in the literature. Although the case studies mentioned do not present all the elements that characterize empirical research (for example, the second case lacks supporting empirical evidence), they are particularly significant in that they allow highlighting certain elements. They do not investigate the actual improvement of linguistic competence, but rather let the participants self-evaluate their skills or report informal feedback. Furthermore, the employment of technology is not designed from a universal design perspective that ensures inclusive education for all. Although the benefits of digital technology are listed (for instance, the personalization and creativity they allow [50]), their inclusive potential is not explicitly disclosed. Thus, further investigation should be carried out into the use of digital inclusive tools to allow everyone to fully benefit from language learning in museums.

4 Implications for Future Research The development of inclusive multilingual competence is crucial to provide equal and accessible education to every person, in order to increase the individual and society’s inclusion and well-being. Thanks to their informal, flexible, and engaging environment museums can help enhance the insufficient levels of language competence photographed by reports. The integration of inclusive digital tools in language learning museum experiences can ensure that the different needs of learners are met, and obstacles are removed. Although the theoretical framework provides encouraging evidence, field research that can assess the actual improvement of language through digital tools in museum contexts should be conducted. To design and carry out such inquiries, a few concepts that emerged from the literature review should be considered. The universal design approach seems almost mandatory. When designing digital inclusive didactic museum paths suitable for every person, one must consider the individual and the group’s learning needs and characteristics, the advantages and limitations of the contexts, and the target competences and skills. The tools selected must be aligned with these elements and with the objectives. If digital means are chosen, the possible consequences of the digital divide must be prevented. The design of the activity should follow the structure proposed by Hooper-Greenhill [51] and implemented by Fazzi with the linguistic dimension [50, 52]. It should thus consist of a first encounter where the activity is presented and linguistic elements, such as

Fostering Inclusion and Well-Being Through Digital …

557

grammar and vocabulary, are introduced. The second phase should take place in the museum and foresee a tour in the target language and the completion of tasks. Lastly, in the third encounter, linguistic learning should be reinforced, and products should be created and discussed. In addition, at the beginning and at the end of the experience, pre, and post-test should be carried out to assess the actual improvement of language competences [53], and such experiences should be carried out for a significant amount of time in order to have actual impact on the learners’ competence [52]. A final element to consider in the designing of a digital language learning museum workshop is the partnerships between museum professionals, schoolteachers, and educators, and university researchers [53]. It was highlighted that one of the disincentives to design museum language learning workshops is the feeling of schoolteachers to be lacking proper background and skills [19, 52, 53]. Collaboration between these institutions is fundamental to completing and integrating each partners’ expertise. Universities can offer the scientific method and approach as a foundation for the research; museum professionals can contribute with their knowledge of museum didactic strategies and of the museum collections; and teachers can integrate the content and competences and adapt them to the school curriculum. Planned design and collaboration are central to guaranteeing inclusive education that can lead to greater inclusion and well-being of society.

References 1. European Commission: European pillar of social rights (2018) 2. European council: council conclusions on equity and inclusion in education and training in order to promote educational success for all 2021/C 221/02 (2021) 3. European Council: council conclusions on the role of youth work in supporting young people’s development of essential life skills that facilitate their successful transition to adulthood, active citizenship, and working life 2017/C 189/06 (2017) 4. European council: recommendation on key competences for lifelong learning 2018/C 189/01 (2018) 5. European council: recommendation on a comprehensive approach to the teaching and learning of languages 2019/C 189/03 (2019) 6. European council: recommendation on high-quality early childhood education and care systems 2019/C 189/02 (2019) 7. European commission: Europeans and their language (2012) 8. European council. Common European framework of reference for languages: learning, teaching, assessment. Cambridge University Press, New York (2001) 9. INVALSI: Rapporto Invalsi. (2022) 10. Marsh, D., Langé, G.: Using languages to learn and learning to use language. University of Jyväskyulä, Jyväskylä (2000) 11. Wolff, D.: CLIL: bridging the gap between school and working life. In: Marsh, D., Wolff, D. (eds.) Diverse Contexts-Converging Goals. CLIL in Europe, pp. 15–25. Peter Lang, Frankfurt (2007) 12. Cedefop: European guidelines for validating non-formal and informal learning. Cedefop reference series, num 104. Publications Office, Luxembourg (2015) 13. Hein, G.E.: Learning in Museums. Routledge, London (1998) 14. Lewis, B.N.: The museum as an educational facility. Mus. J. 80(3), 151–155 (1980)

558

M. Tolaini

15. Sandell, R.: Museums as agents of social inclusion. Mus. Manage. Curatorship 17(4), 401–418 (2002) 16. Nardi, E.: Musei e pubblico. Franco Angeli, Milano (2014) 17. Poce, A.: Il patrimonio culturale per lo sviluppo delle competenze nella scuola primaria. Cultural Heritage and the Development of XXI century skills in Primary Education. Franco Angeli, Milano (2018) 18. Gottlieb, M.: Assessing English Language Learners: Bridges from Language Proficiency to Academic Achievement. : Corwin Press, Thousand Oaks (2006) 19. Dziedzic Wright, K.: The museum as classroom: expanding the boundaries of English language curriculum in Korea. In: Costache, I.D., Kunny, C. (eds.) Academics, Artists, and Museums: 21st-Century Partnerships, pp. 57–68. Routledge, London (2019) 20. Fazzi, F., Meneghetti, C.: ‘Migrare’ la classe: i musei come spazi innovativi di apprendimento linguistico e interculturale. In: Caruana, S., Chircop, K., Gauci, P., Pace, M. (eds.) Politiche e pratiche per l’educazione linguistica, il multilinguismo e la comunicazione interculturale, Atti del VI convegno della Società di Didattica delle Lingue e Linguistica Educativa (DILLE), pp. 519–537. Università Ca’ Foscari, Venezia (2021) 21. Hooper-Greenhill, E.: Museum and their visitors. Routledge, London (1994) 22. Evans, M., et al.: The authentic object? A Child’s Eye View. In: Paris, S.G. (ed.) Perspective on Object-Centered Learning in Museums, pp. 55–77. Lawrence Erlbaum, Mawah (2002) 23. ICOM: museum definition. https://icom.museum/en/resources/standards-guidelines/museumdefinition/ (2022) 24. Falk, J.H.: The Value of Museums. Enhancing Societal Well-being. Rowman & Littlefield Lanham (2022) 25. Shoemaker, M.K:Art is a wonderful place to be: ESL students as museum learners. Art Educ. 51(2), 40–45 (1998) 26. Ruanglertbutr, P.: Utilising art museums as learning and teaching resources for adult English language learners: the strategies and benefits. English Aust. J. 31(2), 3–22 (2016) 27. Cooker, L., Pemberton, R: Self-access language learning in museums: a materials development project. Stud. Self-Access Learn. J. 1(2), 87–99 (2010) 28. AlAjlan, M.: Museums as learning spaces: a case study of enhancing ESP students’ language skills in Kuwait University. English Lang. Teach. 14(2), 1–8 (2021) 29. Ho, C.L.M., Nelson, M.E., Müeller-Wittig, W.: Design and implementation of a studentgenerated virtual museum in a language curriculum to enhance collaborative multimodal meaning-making. Comput. Educ. 57, 1083–1097 (2011) 30. Garcìa-Sampedro, M.: Five sense in the museum: una experiencia multidisciplinar en la formaciòn del profesorado bilingue (espanol-inglés). Didacticae 5, 145–159 (2018) 31. Fazzi, F., Lasagabaster, D.: Learning beyond the classroom: students’ attitudes towards the integration of CLIL and museum-based pedagogies. Innov. Lang. Learn. Teach. 15(2), 156–168 (2021) 32. Poce, A.: Educational research in museum settings: methodologies, tools and functions. Edizioni Scientifiche Italiane (2020) 33. European commission: communication from the commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions: Strengthening European Identity through Education and Culture: The European Commission’s contribution to the Leaders’ meeting in Gothenburg, 17 November 2017COM(2017) 673 final (2017) 34. European council: council resolution on a strategic framework for European cooperation in education and training towards the European Education Area and beyond (2021–2030) (2021) 35. European council: council conclusions on supporting well-being in digital education 2022/C 469/04 (2022) 36. Hersch, M.: Technology for Inclusion. Background paper prepared for the 2020 Global Education Monitoring Report. Paper commissioned for the 2020 Global Education Monitoring Report, Inclusion and education. ED/GEMR/MRT/2020/P1/ (2020)

Fostering Inclusion and Well-Being Through Digital …

559

37. UNESCO: Qingdao declaration. In: International Conference on ICT and post-2015 Education. Seize digital opportunities, lead education transformation. 23–25 May 2015, Qingdao, the People’s Republic of China (2015) 38. European council: recommendation on promoting common values, inclusive education, and the European dimension of teaching 2018/C 195/01 (2018) 39. United Nations: convention on the rights of persons with disabilities (2006) 40. Panesi, S., Bocconi, S, Ferlino, L: Promoting students’ well-being and inclusion in schools through digital technologies: perceptions of students, teachers, and school leaders in Italy expressed through SELFIE piloting activities. Front. Psychol. 11, 1563 (2020) 41. BECTA: the impact of ICT in schools – a landscape review (2007) 42. BECTA: an exploration of digital creativity used to engage and motivate ‘hard-to- reach’ learners in behavioural, emotional, and social difficulties (BESD) schools (2007a) 43. European agency for special needs and inclusive education: inclusive digital education. Weber, H., Elsner, A., Wolf, D., Rohs, M., Turner-Cmuchal, M. (eds.). Odense, Denmark (2022) 44. Ioannou, A., Constantinou, V.: Augmented reality supporting deaf students in mainstream schools: two case studies of practical utility of the technology. In: Auer, M.E., Tsiatsos, T. (eds.). Interactive Mobile Communication Technologies and Learning. IMCL 2017. Advances in Intelligent Systems and Computing, vol. 275, Springer, Cham (2017). 45. Baragash, R.S., Al-Samarraie, H., Alzahrani, A.I., Alfarraj, O.: Augmented reality in special education: a meta-analysis of single-subject design studies. Eur. J. Spec. Needs Educ. 35(3), 382–397 (2020) 46. Almohammadi, K., Hagras, H., Alghazzawi, D., Aldabbagh, G.: A survey of artificial intelligence techniques employed for adaptive educational systems within e-learning platforms. J. Artif. Intell. Soft Comput. Res. 7(1), 47–64 (2017) 47. Paillacho Chiluiza, D.F., Solorzano Alcivar, N.I., Paillacho Corredores, J.S.: LOLY 1.0: a proposed human-robot-game platform architecture for the engagement of children with autism in the learning process. In Botto-Tobar, M., Zamora, W., Larrea Plúa, J., Bazurto Roldan, J., Santamaría Philco, A. (eds.). Systems and Information Sciences Proceedings of ICCIS 2020. Advances in Intelligent Systems and Computing vol. 1273. Springer, Cham (2021) 48. Chubb, L.A., Fouché, C.B., Agee, M., Thompson, A.: Being there: technology to reduce isolation for young people with significant illness. International Journal of Inclusive Education, pp. 1–18 (2021) 49. Cooper, M., Ferguson, R., Wolff, A.: What can analytics contribute to accessibility in e-learning systems and to disabled students‘ learning?. In Gaševi´c, D., Lynch, G., Dawson, S., Drachsler, H., Penstein Rosé, C. (eds.). LAK ’16: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. Association for Computing Machinery, New York (2016) 50. Fazzi, F.: izi.TRAVEL: una piattaforma digitale per promuovere l’apprendimento linguistico dentro e fuori la classe. Bollettino Itals, 19(89), 104–113 (2021) 51. Hooper-Greenhill, E.: Gallery and Museum Education. Leicester University Press, London (1994) 52. Fazzi, F.: CLIL dalla scuola al museo: potenzialità, criticità e implicazioni glottodidattiche. Italiano LinguaDue 12(2), 627–649 (2020) 53. Rymarczyk, J.: Foreign language learning with new technologies in the museum. In: Rymarczyk, J. (eds.). Foreign Language Learning Outside School. Places to see, learn and enjoy, pp. 159–169. Peter Lang, Frankfurt (2013)

Proposing a Hybrid Campus: A Community Engagement Framework for Online Learners Roxana Toma and Matthew Berge

Abstract We expand on the work of Redmond et al. ( P. Redmond, A. Heffernan, L. Abawi, A. Brown, and R. Henderson, “An online engagement framework for higher education,” Online Learning, vol. 22, no. 1, 2018.) and their proposal of an Online Engagement Framework for Higher Education, setting the stage for an investigation into online course designs and pedagogical methods likely to foster increased student social engagement, confidence, and resilience in the online learning process. Our research builds on the student-engagement themes proposed by Redmond et al.: cognitive, behavioral, collaborative, and emotional engagement; we see these as forms of students’ capital that can be facilitated by online course design and instruction. This capital, in turn, helps foster students’ social engagement, which is linked to better learning outcomes, increased confidence, and resilience in the online learning process. Following a meta-analysis of contemporary literature on distance education, we draw on the community of inquiry and community of practice frameworks, identifying ways to measure these constructs. As such, we propose a new model — the Community Engagement Framework for Online Learners, which can be tested with survey data to identify elements of online course design and instruction most likely to influence students’ engagement. In doing so, we also theorize that a hybrid campus can be achieved for online learners through increased social engagement, which should likely, in turn, increase their confidence and resilience in the online learning process. Keywords Online learning · Student engagement · Social engagement · Hybrid campus

R. Toma (B) · M. Berge SUNY Empire State University, Saratoga Springs, NY 12866, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_46

561

562

R. Toma and M. Berge

1 Introduction The COVID-19 pandemic has allowed instructors, academicians, and administrators working at higher education institutions to re-evaluate and re-envision teaching and learning processes. While literature surrounding issues associated with the transition to online learning and students’ satisfaction with online courses started to emerge, there is a scarcity of work addressing the research gap — the importance of course design, teaching, and learning presence in online classes. It is thus essential to think of ways to build a strong presence and use designs to create meaningful learning experiences for students, leading to increased engagement, confidence, and resilience in online learning environments. Our research builds on Redmond et al.’s [4] proposal of an Online Engagement Framework for Higher Education and the four student-engagement themes: cognitive, behavioral, collaborative, and emotional engagement; in our model, we see these as varying forms of capital that can be facilitated by online course designs and pedagogical methods. This capital, in turn, helps foster students’ social engagement, which is linked to higher student confidence levels, and students with high social engagement can more easily recover from setbacks, displaying greater resilience. Furthermore, higher student confidence levels correlate with learning outcomes, as the “learners’ performance” in online settings is facilitated by skills like Self-Regulated Learning, which we will discuss at length [1]. Moreover, social engagement is a manifestation of social capital, and the “development of social capital…is critical for [students’] resilience” [2, p. 5]; additionally, Beals et al. note that social capital helps “foster confidence…and resilience” through increased “peer and faculty support” [3, p. 2]. As such, it is worth exploring how these relationships correlate in online learning settings to achieve the best academic outcomes for online learners. Following a meta-analysis of contemporary literature on distance education, we utilize concepts from the community of inquiry and the community of practice frameworks to identify meaningful ways to measure these constructs. As such, we propose a new theoretical model, which we shall call the Community Engagement Framework for Online Learners. This novel model aims to identify elements of online course designs and pedagogy that are likely to foster students’ cognitive, behavioral, collaborative, and emotional capital — our latent-independent variables, which influence students’ social engagement — our latent-dependent variable. In doing so, we also theorize that a hybrid campus can be achieved for online learners through increased social engagement, which should likely, in turn, increase their confidence and resilience in the online learning process. Our work contributes to the literature on distance learning by suggesting a model that is more theoretically complex and analytically sound than previous proposals related to online student engagement and which renders applicability through testing with real-world data. By looking at students’ cognitive, behavioral, collaborative, and emotional capital as latent-independent variables, which influence students’ social engagement — our latent-dependent variable, we are able to survey online students for cross-validation using path analysis and structural equation modeling. As such,

Proposing a Hybrid Campus: A Community Engagement Framework …

563

this model is of value to faculty, administrators, and instructional designers teaching or facilitating academic processes online or in a hybrid format during the pandemic, post-vaccine stage, and in the post-pandemic world. Future studies can also survey online instructors to identify practical uses of our proposed engagement variables in their course designs and pedagogies, as those are likely to correlate with increased student engagement, confidence, and resilience in the learning process.

2 Theorizing Engagement in Online Learners In the “Online Engagement Framework for Higher Education,” Redmond et al. call for follow-up research to expand on their framework [4]. The constructs they present focus on various forms of engagement, including cognitive, behavioral, collaborative, and emotional engagement [4]; however, we view these terms as varying forms of capital for our proposal. Additionally, we are building upon the insights gleaned from prior research into “The Power of Synchronous Sessions in Distance Education: Building Community and Resilience in the Age of COVID-19” by Toma and Mhamed and “Online Teaching in a Time of Crisis: Social Capital and Community Building Tools” by Toma and Berge [2, 5]. Ultimately, our focus is on social engagement, which we see as a latent-dependent variable, with the other four engagement themes serving as latent-independent variables. Shea, Vickers and Hayes’ research provides the foundational “Community of Inquiry” (CoI) theoretical framework, originally “developed by Garrison (2000)…based on a model of critical thinking and practical inquiry” [6, p. 128]. Moreover, the identified constructs were frequently examined in the existing literature on the CoI framework [1, 7, 8]. Additional literature examining the CoI framework comes from [6], other journal articles by Shea et al. [9, 10], and a more recent “conceptual paper” that examines the CoI framework from a “distributed perspective” by Piera Biccard from 2021 [11]. We also look to Wenger’s “Community of Practice” (CoP) theoretical framework by examining a “critical review” from 2014 of the CoP framework in “online and blended learning research” between 2000– 2014 by Smith, Hayes, and Shea [12]. It is worth noting that emotional engagement serves as an indicator in the Redmond et al. framework; however, there is little to no mention of this in the other literature on the CoI framework; as such, we turn to an article from 2016 by Cottingham titled “Theorizing emotional capital” [13]. We are looking to build upon these frameworks with a novel community engagement framework with the potential to increase student social engagement, which will likely correlate with increased student learning outcomes, confidence, and resilience in the learning process.

564

R. Toma and M. Berge

2.1 Cognitive Capital Reviewing the literature on the CoI framework presented many examples of what we should look for when identifying cognitive capital. Redmond et al. use the term “cognitive engagement” [4]; meanwhile, Shea and Bidjerano, as well as Biccard, use terminology like “cognitive presence” and “epistemic engagement” [5, 7–11] in describing the construct we are examining as cognitive capital. The literature provides indicators of what to look for to measure cognitive capital through survey research methods. Redmond et al. plainly states that cognitive engagement is the “active” learning process [4]. Meanwhile, Shea and Bidjerano provide the following definition for cognitive presence: the ability of learners “to construct and confirm meaning through sustained reflection and discourse” [7]. Redmond et al. identified six indicators for cognitive engagement: “thinking critically, activating metacognition, integrating ideas, justifying decisions, developing deep discipline understandings, [and] distributing expertise” [4, p. 190]. At this stage, we have identified four traits as indicators to measure students’ self-reported level of cognitive capital: Activating Metacognition, Epistemic Engagement, Strategic Learning, and Confidence in Online Discussions. We used Redmond et al.’s indicators to define cognitive capital by coupling them with the principles Shea uses when defining “cognitive presence” [7]. Activating Metacognition is part of cognitive engagement and “the active process of learning” [4], making this an inherent component of cognitive capital. Tanner notes that a concrete definition for metacognition remains elusive because the term “is used in different disciplines in different ways” [14, p. 113]. However, the most clear-cut overarching practical definition for active metacognition is “emphasis on planning, monitoring and evaluating one’s…learning processes” [14, p.114]. In this light, metacognition is akin to the reflective learning process, which can be seen as an identity construction feature in community online learning settings. Students with activated metacognition will go through a process of transformative learning “because learning transforms who we are and what we can do, [making this] an experience of identity” [12, p. 213]. Epistemic Engagement is identified within the CoP framework by “the modes of thinking and acting that…help individuals learn how to participate meaningfully” [12, p. 224]. In contrast, in the CoI framework, epistemic engagement can be the reflective “processes of participatory practice” featuring course design elements to aid students in developing the “skills of a disciplinary discourse community” [7, p. 544]. As such, Epistemic Engagement can be considered part of the process identified by Redmond et al. of “developing deep discipline understandings” [4, p. 193]. As students engage with course content, especially when taking the further step to engage with supplemental reading materials provided by instructors, they show increased rates of cognitive capital through their increased epistemic engagement. Strategic Learning indicates cognitive capital because it is highly correlated with the self-accountability often found in online learners; given the “self-directed” nature of this modality, students who develop strategic learning tend to be more successful

Proposing a Hybrid Campus: A Community Engagement Framework …

565

[8, p. 1723]. The indicators from Redmond et al. that most closely fit within this category are integrating learned concepts or ideas and “distributing expertise” [4, p 190]. Further, Biccard expands on this when describing cognitive presence and the benefits of a “distributed design [for] research sharing,” allowing for “research finding [with] a distributive approach” [12, p. 7]. Similarly, Biccard describes one function of cognitive presence as the student’s ability to “select the appropriate content to meet the [desired] outcomes” [11, p. 6] fitting the mold of integrating learned concepts and distributing expertise developed by the student through their ability to learn while strategically engaging with course content. Confidence in Online Discussions is an inherent quality in individuals with high cognitive presence. Shea and Bidjerano indicate this in their study, as students who strongly agreed with the statement “I felt comfortable participating in the course discussions” displayed “significantly higher levels of cognitive presence” than those who were neutral or disagreed with the statement [7, p. 549]. Redmond et al. present “justifying decisions” as an indicator of cognitive engagement, noting that students displaying “deep cognitive engagement” will “justify or compare ideas and solutions” by integrating concepts from “multiple sources, providing new information,” supporting their arguments in online discussions [4, p. 192]. These students bring novel concepts to the discussion space because they are confident in their ability to justify their position with supporting evidence, making this a clear indicator of cognitive capital.

2.2 Behavioral Capital Redmond et al. use the term “behavioral engagement,” which they describe as “doing the work and following the rules” [4, p. 193]. Further noting that this principle is referred to in various terms, like “learning presence” or “self-regulating behaviors,” both of which are frequently found in the literature from Shea and Bidjerano [1, 8, 10]. Similarly, Biccard refers to learning presence as an extension of the CoI framework relating to the “distribution of teaching presence” [11, p. 6], which makes sense as many qualities we associate with behavioral capital relate to students engaging with course content provided by instructors. The indicators of behavioral capital we have identified fall under what Shea and Bidjerano call Self-Regulated Learning and Self-Efficacy, coupled with Redmond et al.’s indicators of behavioral engagement. Self-Regulated Learning (SRL) falls under the behavioral engagement indicators identified by Redmond et al., with concepts like “developing academic skills,” “developing agency,” and one’s ability to uphold “online learning norms” [4, p. 193]. Shea and Bidjerano hold that SRL is given particular importance in online learning environments and “personally directed forms of learning,” especially when “seeking information from electronic sources;” moreover, they hold that SRL is potentially part of the “larger construct” of learning presence [8, p. 1723]. Further, they postulate that self-efficacy also falls under this veil, most closely resembling the development of agency described by Redmond et al., as learners with “adaptive self-efficacy beliefs”

566

R. Toma and M. Berge

are more likely to see failure as providing them an opportunity to put in the “effort to achieve better” [8, p. 1723]. Shea and Bidjerano explain that learners with high levels of SRL will set “proximal attainable goals” based on their perceptions of their abilities concerning the “complexity of the learning task” and will create a learning environment that works best for them; further, they will “constantly monitor” their progress and evaluate how well their goals are being met [1, p. 317]. Furthermore, a quintessential component of SRL is the ability to identify opportunities and challenges and engage with appropriate institutional resources associated with these opportunities or challenges. For example, suppose a student is struggling with a particular course, but they have a high level of self-efficacy. In that case, they will be more likely to admit they are having challenges and seek appropriate academic support opportunities offered by the institution. Moreover, Fensie notes that mere “participation in online learning” does not improve one’s SRL skills; however, “these skills can be taught,” as improved SRL skills are associated with better learning outcomes, particularly for adults in online learning [15, p. 142]. Additionally, Fensie notes that executive functioning skills are fostered in traditional classroom learning settings but are less frequent in “asynchronous online learning” [15, p. 142]. From a pedagogical standpoint, it is thus worth exploring how synchronous learning sessions in online learning can influence executive functioning skills. Peer support can be considered for both behavioral and collaborative capital. Nevertheless, under the Redmond et al. model, students’ willingness to support and encourage their peers indicates behavioral engagement [4, p. 193]. Furthermore, they note that students with high behavioral engagement display “high effort and persistence” through participation, maintaining “positive attitudes” with high levels of SRL [4, p. 193]. Similarly, Shea and Bidjerano note that Self-Regulated Learners (SRLs) often hold “more positive perceptions of online courses” [1, p. 318]; furthermore, “positive self-efficacy beliefs” are associated with the ability to recognize challenges and even “failure as an occasion to be informed” [8, p. 1723]. Shea et al. note that social presence, which relates to social engagement, is found in SRL through “online discourse that promotes positive affect, interaction, and cohesion” [10, p. 90]. Ultimately, one of the most critical indicators of behavioral capital in assessing social engagement is students’ ability to build relationships, be it with peers or faculty; likewise, relationship building is also crucial for our next indicator, collaborative capital.

2.3 Collaborative Capital Redmond et al. describe collaborative engagement as the “development of different relationships and networks that support learning;” furthermore, they identify indicators of collaborative engagement as follows: “learning with peers, relating to faculty members, connecting to institutional opportunities, and developing professional networks” [4, p. 194]. Similar themes arise in Smith, Hayes and Shea’s review of Wenger’s CoP, where they identify indicators of what we have defined

Proposing a Hybrid Campus: A Community Engagement Framework …

567

as collaborative capital. For instance, “joint enterprise” occurs when a community develops a “collective understanding” concerning the purpose of said community, or “mutual engagement,” which is inherently collaborative as this relies upon interacting with peers to establish “norms, expectations, and relationships;” additionally, they identify the development of a “shared repertoire” through “using the communal resources” afforded to them by the course instructor, or institution [12, p. 212]. Epistemic Engagement goes beyond merely indicating cognitive capital, as the “deep discipline understandings” [4, p. 193] can also be accomplished through interaction among students with peers and faculty. As such, interaction leads to “co-construction of knowledge,” which affords learners with the opportunity to “build upon each others contributions, defend and argue positions, challenge and criticize each other…and ask and answer each others questions” [1, p. 324]. Ultimately, the social aspect of collaborative learning is witnessed as an element of Shea and Bidjerano’s concept of learning presence; furthermore, collaboration with peers should correlate with increased cognitive and social engagement [1, 4, 8, 10, 11]. Indicators within the realm of collaborative capital beyond those mentioned by Redmond et al. for collaborative engagement include other elements of the CoI framework conception of learning presence (see Table 1). Across the literature, the concept of learning presence arises in correlation to teaching presence. Both are associated with learners’ self-efficacy, which is “stronger for students in blended learning environments” [8, p. 1727]. Indeed, the literature makes it clear that students have higher success rates when the instructor is actively involved in the learning process. Further, prior research by Toma and Mhamed found that the addition of “regular synchronous sessions to otherwise fully asynchronous courses” seemingly results in better learning outcomes for online learners, as this creates an opportunity to “empower students and provide them with an effective way to build community and social capital” [2, p. 11]. As such, this indicates that the “blended classroom” described by Shea and Bidjerano over a decade ago can be achieved in a fully-online setting through modern technology and online meeting spaces by implementing synchronous learning sessions where students can collaborate in otherwise asynchronous settings.

2.4 Emotional Capital Redmond et al. note that emotional engagement is the “emotional reaction to learning,” associated with “feelings and attitudes towards learning,” further noting that emotional engagement is activated by “both negative and positive emotions” [4, p. 195]. Moreover, Redmond et al. state that emotional engagement is observable through students’ “attitude, enthusiasm, interest, anxiety or enjoyment in the learning process,” additionally noting the following indicators for emotional engagement: “managing expectations, articulating assumptions, recognizing motivations, [and] committing to learning” [4, p. 195]. Furthermore, these indicators of emotional engagement directly correlate with the principles of emotional capital,

568

R. Toma and M. Berge

Table 1 Features to be Considered in Online Course Design and Pedagogy that Facilitate Different Forms of Student Capital/Student Engagement Cognitive capital

Behavioral capital

Collaborative capital

Emotional capital

Social engagement

- Activating metacognition - Epistemic engagement - Strategic learning - Confidence in online discussions

- Self-Regulated Learning (SRL) - Engaging with Course Resources - Identifying Opportunities and Challenges - Supporting and Encouraging Peers

- Learning with peers - Building rapport with faculty members - Engaging with institutional opportunities - Interacting with peers to establish trust - Developing professional networks

- Managing expectations - Articulating assumptions - Recognizing motivations - Committing to learning - Peer support systems - Academic self-esteem - The ability to cope or adapt to stressors - Perceptions of inclusion or exclusion within the college community

- Building community - Creating a sense of belonging - Developing purposeful relationships with others - Establishing trust and rapport

which Cottingham calls “a form of cultural capital” utilizing individuals’ “emotionspecific…resources that [they] activate and embody in distinct fields” [13, p. 451]. Under our framework, principles of emotional capital are observed in the distinct field of online learning. Additionally, our indicators for measuring emotional capital include peer support systems, academic self-esteem, the ability to cope or adapt to stressors, and perceptions of inclusion or exclusion within the college community. Cottingham views emotional capital through a “framework of emotion-aspractice” by combining the concepts of “social practice with emotional management theory;” under this framework, emotional capital and social engagement highly correlate [13, p. 453]. When defining emotional capital, Cottingham points to the “capacity to reinvest emotional capital” [13, p. 453] in the most productive way, which can easily be applied to the concept of “committing to learning,” as presented by Redmond et al. [4, p. 195]. Furthermore, Cottingham notes that Nowotny is generally credited for coining the term emotional capital, which Nowotny defines as “knowledge, contacts, and relations as well as access to emotionally valued skills and assets” [13, p. 453–454]. Subsequently, Cottingham notes how Froyum built upon the definition to view emotional capital as a form of capital that “treats emotions and their management as skills…that translate into social opportunities,” which aligns with our conception of emotional capital and how a higher capacity should also raise an individual’s social engagement [13, p. 454].

Proposing a Hybrid Campus: A Community Engagement Framework …

569

2.5 Social Engagement Redmond et al. view social engagement in higher education as “students’ social investment in the collegiate experience,” including “participation in academic [and] non-academic activities,” happening beyond the “virtual classroom” [4, p. 191]. Furthermore, social engagement is a means of “developing relationships” and a “sense of belonging” and community in online learning; most importantly, social engagement helps students establish trust, and all of these factors serve as indicators of social engagement [4, p. 191]. Moreover, Fensie holds that “learning is socially contextualized” as students subjectively interpret their lived experiences with “social relationships [and] cognitive opportunities as…emotionally experienced by the learner” [15, p. 142]. This contextualization of online learning shows how social engagement relates to cognitive, collaborative, and emotional capital. Furthermore, behavioral capital is an inherent underlying feature as the executive function of SRL is needed for online learners to utilize their cognitive, collaborative, and emotional capital effectively. In other words, in order for students to be socially engaged at their college/university, the other forms of student capital must be present at work. Social presence is a common theme in the literature by Shea and Bidjerano [1, 7, 8], Shea, Hayes and Vickers [6], Shea et al. [9, 10], and Biccard [11]. Shea and Bidjerano use social presence in the CoI model, emphasizing the necessity for students in online learning to portray themselves as “real people” [3, p. 545]. Furthermore, they define social presence as “a supportive collegial online setting” [4, p. 1722], which inherently “promotes positive affect, interaction, and cohesion” in online learning and supports “productive participation” [1, p. 317]. Moreover, Shea et al. found that when courses had higher levels of teaching and social presence, the result was “higher levels of student cognitive presence” [9, p. 15], which shows how different elements within the CoI framework work together. Shea et al. also note that within the CoI framework, social presence supports “a functional collaborative environment” that allows for discourse in online learning settings that are affected by elements of behavioral capital, like “positive affect, interaction, and cohesion” [10, p. 90]. Biccard notes that social presence may be “set in motion by the instructor,” which can be direct through teaching presence or “as part of the design of the course” [11, p. 2]. Indeed, Fensie notes that “motivation is an important consideration” when examining online learning environments, as “lower levels of motivation” highly correlate to “course performance” in online learners [15, p. 143]. Fensie identifies motivation as an “emotional factor” but further notes that self-efficacy serves as a “relevant component of motivational theories” [15, p. 143]; furthermore, the concept of motivation falls under our constructs of both behavioral and emotional capital, which influences student social engagement. Nevertheless, motivation is one of many factors with overlapping variables for consideration; indeed, others include the accessibility and use of resources by students, which involves elements of cognitive capital through epistemic engagement, behavioral capital through SRL, and emotional capital through

570

R. Toma and M. Berge

emotional management theory. Synchronous learning sessions in otherwise asynchronous courses should help foster cognitive capital as students display their confidence in discussions, collaborative capital as these sessions inherently promote working with the instructor and your peers, and behavioral capital through active participation in the learning process. Students’ social engagement can be measured in several ways. For instance, students can be asked about their experiences with college community building or their sense of belonging to the college. Furthermore, the previously identified latentindependent variables of cognitive capital, behavioral capital, collaborative capital, and emotional capital will most certainly play a role in this assessment. After all, one must collaborate with peers when building a community, which also takes cognitive investment, and the ability to follow through directly relates to the principles that define behavioral capital. Moreover, it can be expected that students who develop a sense of belonging to the college will have higher levels of emotional capital invested in their studies and community-building practices. Learning how and if students develop meaningful relationships with faculty and peers in online learning communities is worth studying. If these relationships develop, it is likely from a sense of mutual trust established by building rapport with faculty and peers. Elements of course design, like synchronous learning sessions in otherwise asynchronous courses, fit this description and provide the means for students to utilize their collaborative capital through open discourse by providing their lived experience through emotional capital. Nevertheless, social engagement is also shaped by the learners’ cognitive capital, through which they devote themselves to learning, and the behavioral capital to commit to self-regulated learning.

3 Our Proposal Thus far, we have expanded upon Redmond et al.’s framework to operationalize the constructs of cognitive, behavioral, collaborative, and emotional capital. We also utilized the CoI and CoP frameworks to identify and further describe the social engagement construct as it pertains to the online learning environment. In doing so, we showed how many of the indicator variables can measure these constructs, and indeed, the latent-independent variables themselves are likely to influence each other/covary. Our work is summarized in Table 1 as a checklist of activities that can be measured in order to gauge students’ cognitive, behavioral, collaborative, emotional, and social engagement in online courses. We assume that we can exercise some control over these features through online course design and pedagogy. We also theorize that cognitive, behavioral, collaborative, and emotional engagement are likely to influence students’ social engagement, which, in turn, is linked to better learning outcomes, increased confidence in the learning process, and greater resilience. Surveys of students in online courses can identify which elements of course design and pedagogical methods foster various types of student capital and correlate with

Proposing a Hybrid Campus: A Community Engagement Framework …

571

higher social engagement on their part. Additionally, instructor surveys can also identify their practice in terms of each of the engagement indicators; more specifically, which of the online engagement elements, and their associated indicators, they considered in designing and delivering their online courses by asking them to give relevant examples of activities that demonstrate indicators of various engagement. It is worth mentioning that some of these indicator variables are likely to affect multiple types of capital. For example, instructor engagement with students is critical for various reasons. Redmond et al. noted that engagement initiated by college faculty or staff “is essential before students can engage” [4, p. 184]. Instructors taking the initiative to engage with the student first can be likened to teaching presence, which the literature notes is correlated with student cognitive presence, learning presence, and social presence in the CoI framework [1, 6–11]. Moreover, Shea and Bidjerano suggest that teaching presence impacts both social and learning presence in students; additionally, each of these forms of presence help to foster overall cognitive presence [8, p. 1727]. Ultimately, they then suggest that SRL, which is indicative of behavioral capital, serves as an “important mediator” for evaluating the “links between teaching presence, social presence, and cognitive presence” [1, p. 318]. Further, it is also worth noting that underlying factors might give the student an increased chance of scoring higher or lower on social engagement compared to the ordinary online learner. For example, participation in Esports programs is shown to help students “develop different social and behavioral skills” [16], and participation in student clubs or team-based activities fosters a “sense of belonging” among students [17]. On the other hand, first-generation college students are predisposed to have less social engagement than their peers whose parents went to college and shared their lived experiences with their children, helping prepare them for higher education. Indeed, Moschetti and Hudley note that the “pre-college social networks” found in students whose parents attended college grant them “insights on how to seek help when needed and how to seek access to campus support are resources,” while first-generation students coming from working-class backgrounds are “less likely to access institutional” supports [18, p. 29]. However, socioeconomic status (SES) seems to break this mold, as Cottingham notes that the lived experience of the “working-class and poor” potentially fosters the generation of the emotional capital “needed to confront economic adversities” [13, p. 456]. Finally, employment status is worth noting because full-time employed students are less likely to have the time to stay engaged in the college community, resulting in a limited sense of belonging to the college.

4 Discussion We expanded upon Redmond et al.’s Online Engagement Framework for Higher Education to build a model that focuses on student social engagement in online courses. This model can functionally measure students’ self-reported cognitive, behavioral, collaborative, and emotional capital levels in online courses and their

572

R. Toma and M. Berge

perceptions of social engagement. The latter can be evaluated through students’ selfreported sense of belonging to the college, sense of community within the college, development of purposeful relationships, and establishment of a sense of trust and rapport with faculty and peers. We postulate that, in turn, social engagement fosters higher student confidence levels, and students with high levels of social engagement can more easily recover from setbacks, displaying greater resilience. As such, it is worth exploring how these relationships correlate in online learning settings to achieve the best academic outcomes for online learners. Future research can utilize this model to survey online students and cross-validate it with path analysis and structural equation modeling. Future studies can also survey online instructors to identify practical uses of our proposed engagement variables in their course designs and pedagogy, as those are likely to correlate with increased student social engagement, confidence, and resilience in the learning process.

References 1. Shea, P., Bidjerano, T.: Learning presence as a moderator in the community of inquiry model. Comput. Educ. 59(2), 316–326 (2012) 2. Toma, R., Mhamed, A.A.S.: The power of synchronous sessions in distance education: building community and resilience in the age of COVID-19. In: Guralnick, D., Auer, M.E., Poce, A. (eds.) Innovative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education. TLIC 2022. Lecture Notes in Networks and Systems, vol. 581, pp. 433–445. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-21569-8_41 3. Beals, R., Zimny, S., Lyons, F., Bobbitt, O.: Activating social capital: how peer and socioemotional mentoring facilitate resilience and success for community college students. In: Frontiers in Education, vol. 6 (2021) 4. Redmond, P., Heffernan, A., Abawi, L., Brown, A., Henderson, R.: An online engagement framework for higher education. Online Learn. 22(1), 183–204 (2018) 5. Toma, R., Berge, M.: Online teaching in a time of crisis: social capital and community building tools. Int. J. Adv. Corp. Learn. 16(1), 65–77 (2023) 6. Shea, P., Vickers, J., Hayes, S.: Online instructional effort measured through the lens of teaching presence in the community of inquiry framework: a re-examination of measures and approach. Int. Rev. Res. Open Distribut. Learn. 11(3), 127 (2010) 7. Shea, P., Bidjerano, T.: Community of inquiry as a theoretical framework to foster ‘epistemic engagement’ and ‘cognitive presence’ in online education. Comput. Educ. 52(3), 543–553 (2009) 8. Shea, P., Bidjerano, T.: Learning presence: towards a theory of self-efficacy, self-regulation, and the development of a communities of inquiry in online and blended learning environments. Comput. Educ. 55(4), 1721–1731 (2010) 9. Shea, P., et al.: A re-examination of the community of inquiry framework: social network and content analysis. Internet High. Educ. 13(1–2), 10–21 (2010) 10. Shea, P., et al.: Learning presence: Additional research on a new conceptual element within the community of inquiry (COI) framework. Internet High. Educ. 15(2), 89–95 (2012) 11. Biccard, P.: A distributed perspective on the community-of-inquiry framework for distance education. UnisaRxiv (Preprint) (2021) 12. Smith, S.U., Hayes, S., Shea, P.: A critical review of the use of Wenger’s community of practice (COP) theoretical framework in online and blended learning research, 2000–2014. Online Learn. 21(1) (2017)

Proposing a Hybrid Campus: A Community Engagement Framework …

573

13. Cottingham, M.D.: Theorizing emotional capital. Theory Soc. 45(5), 451–470 (2016) 14. Tanner, K.D.: Promoting student metacognition. CBE—Life Sci. Educ. 11(2), 113–120 (2012) 15. Fensie, A.: A conceptual model for meeting the needs of adult learners in distance education. In: Guralnick, D., Auer, M.E., Poce, A. (eds.) Innovative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education. TLIC 2022. Lecture Notes in Networks and Systems, vol. 581, pp. 136–149. Springer, Cham (2022). https://doi.org/10.1007/978-3-03121569-8_13 16. CDW, “Benefits of Esports in schools,” CDW, 24-Jan-2020. https://www.cdw.com/content/ cdw/en/articles/hardware/the-benefits-of-an-esports-team-in-k12-and-higher-education.html. Accessed 24 Jan 2023 17. Mannion, A.: Participation in student activities linked to academic success. Chicago Tribune, 22-May-2019. https://www.chicagotribune.com/suburbs/la-grange/ct-dlg-student-act ivities-tl-0324-20160317-story.html. Accessed 24 Jan 2023 18. Moschetti, R., Hudley, C.: Measuring social capital among first-generation and non-firstgeneration, working-class, white males. J. College Admission (2007). https://eric.ed.gov/?id= EJ829418. Accessed 25 Jan 2023

Narrating the Museum: Enhancing Cultural Heritage Through User Profiling and Individualized Content Eliana Maria Torre

Abstract Museums contribute to creating collective memory and identity, provided that their communication strategies reach all social groups. Good stories bring the most disparate and complex content closer to the “feeling” of nonspecialists, staying in people’s emotional and cognitive memory. This project aims to promote mental well-being, social inclusion, and active citizenship by stimulating participants’ 4 C’s (Communication, Collaboration, Critical Thinking, Creativity) and digital skills through individualized paths that prompt effective dialogue between users and museums. The study is developed at the National Roman Museum through three research questions. What digital narrative forms promote the social inclusion of various museum audiences? How do you design educational paths that promote social inclusion and active citizenship by stimulating the 4 C’s and digital skills? How do narratives in museums facilitate the participation of different categories of users, especially those at risk of marginalization? The study is based on empirical research and co-participatory design. Facilitated tours and workshops are conducted through inclusive methodologies — i.e., Object-based Learning, Visual Thinking Strategy, and Digital Storytelling. Evaluation measures the levels of the 4 C’s and digital skills, mental well-being, and perceived social inclusion through skills assessment grids, the UCL Museum Wellbeing Measures Toolkit, and focus groups. User profiling, carried out through eye-tracking applied to Digital Storytelling products and a brief questionnaire, identifies the characteristics of groups of users to offer individualized pathways. The expected outcomes are a valuable resource for the inclusiveness of museums, the community’s participation, and all educators who wish to innovate their educational practices. Keywords Transverse competences · Digital storytelling · Social inclusion

E. M. Torre (B) Università Degli Studi La Sapienza, 00185 Rome, Italy e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_47

575

576

E. M. Torre

1 Good Stories Always Work: Museum Education for Mental Well-Being and Social Inclusion In the last decades, there has been growing recognition of the vital links between place-identity theory [1], sense of belonging [2], and the role played by museums. Indeed, museums are intrinsically connected to the territory where they are placed and its history. Furthermore, in recent years, museums have been seen as agents of social change [3]. As such, they can play a pivotal role in creating collective memory and identity, provided that their communication strategies reach all social groups. As stated by Vaglio [4] in a book on storytelling in museum settings, “good stories always work.” A skillfully designed narrative makes it possible to mediate the most disparate and complex contents, to bring them closer to the “feeling” of non-specialists, and to remain in the emotional and cognitive memory [5]. Imitation as intended by Hickok [6] is fundamental in this process and is strictly linked to mirror neurons. Even though this is not the place to explore Hickok’s thesis in depth, it is worth noting that, among the purposes of imitation, there are social learning, cultural transmission, and empathy. Besides, compelling stories promote empathy, which, according to Zaki [7], encompasses cognitive empathy (i.e., the identification of others’ feelings), emotional empathy (i.e., sharing others’ emotions), and empathic concern (i.e., wanting to improve others’ experiences). Effective narratives make people active readers, listeners, watchers, and storytellers, in a lifelong learning perspective that considers education as a wide, multifaceted, and social process [8, 9]. The European Council adopted a Recommendation on Key Competences for Lifelong Learning [10], in which transverse competences are relevant to society insofar as they are needed for personal fulfillment, a healthy and sustainable lifestyle, employability, active citizenship, and social inclusion. In today’s society, digital skills can rightfully be listed among transverse competences. As stated in the Pietrelcina Chart [11], digital technologies provide a tool to both enhance cultural heritage and overall reconfigure cultural entities and places as communal heritage. They can foster strategic opportunities for the reorganization of knowledge, openness to entities and content, and access to the very forms of the contemporary. According to the new definition of museum, approved in 2022 in Prague, museums are “open to the public, accessible and inclusive, [and] foster diversity and sustainability.” To do so, a participatory shift in museums’ policies is required, so that “visitors can create, share, and connect with each other around content” [12]. Participatory designs may, in fact, draw in audiences at risk of marginalization for sociocultural, economic, or disability-related reasons by providing emotional and cognitive engagement through creative activities and social connection, while creating new value for museums, visitors, and non-participating audience members [12]. Therefore, alongside more technical, yet necessary, descriptions well suited to experts, it is fundamental to put in place communication strategies and individualized content that allow diverse museum targets, and then the community as a whole, to re-appropriate their cultural heritage. The need for a shift towards co-participatory designs is also evident in the Faro Convention [13], finally ratified in Italy by L. 133/2020 [14]. The

Narrating the Museum: Enhancing Cultural Heritage Through User …

577

Convention emphasizes the right to cultural heritage as a human right fundamental to democracy. Cultural heritage is relevant because of the meanings and uses that people attach to it and the values it represents. From this perspective, civic initiatives enable institutions and communities to develop decision-making capacities and manage their development processes, ensuring that heritage contributes to the social, cultural, and economic dynamics of the communities. The 2030 Agenda for Sustainable Development [15] highlights the importance of culture in both Goal 3 Ensure Healthy Lives and Promote Well-being for All at all Ages and Goal 4 Ensure Inclusive and Equitable Quality Education and Promote Lifelong Learning Opportunities for All. Well-being, in particular, is a complex concept that can be approached from different disciplines and through diverse lenses. However, for this study’s purpose, definitions provided by the WHO are adopted [16]. Consequently, well-being is a “positive state experienced by individuals and societies. Similar to health, it is a resource for daily life and is determined by social, economic, and environmental conditions. Well-being encompasses quality of life, as well as the ability of people and societies to contribute to the world in accordance with a sense of meaning and purpose. Focusing on well-being supports the tracking of the equitable distribution of resources, overall thriving, and sustainability. A society’s well-being can be observed by the extent to which they are resilient, build capacity for action, and are prepared to transcend challenges” [16]. Mental health is intended as “a state of well-being in which the individual realizes his or her abilities, can cope with the normal stresses of life, can work productively and fruitfully, and contribute to his or her community […]” [16]. Furthermore, mental well-being is regarded as a “dynamic state, in which the individual is able to develop their potential, work productively and creatively, build strong and positive relationships with others, and contribute to their community” [17].

2 The Research Project Narrating the Museum: Enhancing Cultural Heritage through User Profiling and Individualized Content is a PhD research project started in November 2022. It focuses on the function of storytelling in museum settings as a means to enhance cultural heritage and on user profiling as a means of providing individualized content. Because of its early stage, this paper pinpoints the theoretical and methodological framework. In the next subsections, the following aspects will be addressed: museum setting; research hypothesis along with research questions and objectives; methodology with regard to teaching strategies, evaluation tools, and user profiling; and expected outcomes at the end of the project.

578

E. M. Torre

2.1 The Museum Setting In order to carry out the project, an agreement between Sapienza University (Rome) and the National Roman Museum (MNR hereafter), also in Rome, was established. According to DL 83/2014 as modified by L 106/2014, the MNR operates with scientific, financial, organizational, and accounting autonomy. It is one of the most representative museums of the Italian capital and comprises four different locations in the city center that display the complexity and variety of Rome throughout millennia. Furthermore, the MNR is interconnected with other national and international institutions: going beyond the concept of a traditional physical perimeter to be an aggregator of different places, memories, and experiences is part of its mission. Regarding the visitors’ engagement, the MNR delivers a wide range of services, experiences, workshops, etc., searchable through the institutional website of the museum [18], achieving remarkable results in terms of inclusion and cultural accessibility. To name but a few of those initiatives closest to the ratio of this project, consider the activities of L’abilità, a non-profit organization, within the project Museo per tutti, devoted to people with cognitive disabilities [19], the book by Carlotta Caruso 101 Storie Svelate, a wonderful example of storytelling scientifically accurate yet compelling and easily understandable by non-experts, inspired by the epigraphs of the MNR [20], or its digital counterpart — i.e., the series of videos Vedi alla voce… [21]. What is missing, however, is a co-participatory design that involves citizens in creating new interpretations of the collections. Currently, there are no dedicated activities where non-experts cooperate among themselves with the support of museum staff or professionals in the field. Furthermore, the MNR expressed the need for more inclusive and cognitively accessible captions, which need to be designed homogeneously for all four locations. The here presented research project aims at bridging these gaps.

2.2 Research Hypothesis, Research Questions, and Objectives The core research hypothesis of the project is that it is possible to promote mental well-being, social inclusion, and active citizenship. To do so, participants’ transverse and digital skills are stimulated through individualized pathways, and effective dialogue between the user and the museum is fostered. In order to verify the hypothesis, three main research questions are set. They investigate the following subjects: i) the narrative forms, including digital ones, that promote the social inclusion of various museum audiences; ii) the design of coparticipatory educational paths that promote social inclusion and active citizenship by stimulating the so-called 4 C’s skills (Communication, Collaboration, Critical

Narrating the Museum: Enhancing Cultural Heritage Through User …

579

Table 1 Research Questions and Objectives Research Questions

Objectives

What narrative forms, including digital ones, promote the social inclusion of different audiences in museum education settings?

To analyze narrative forms, including digital ones, to promote social inclusion through the activation of transverse and digital skills;

How to design co-participatory educational pathways to promote social inclusion and active citizenship by stimulating the 4 C’s skills and digital skills?

To identify museum objects to create paths on topics that are interesting in today’s world; To implement and offer museum education paths centered on the use of narrative texts that are inclusive, individualized, technologically advanced, and participatory; To stimulate participants’ soft and digital skills to promote mental well-being and social inclusion;

What kind of function can narrative strategies have in the museum’s communication and educational solutions to facilitate the participation of different categories of users, especially those at risk of marginalization?

To carry out an evaluation to build personas and analyze any increase in participants’ skills, mental well-being, and social inclusion; To monitor and evaluate the function of various types of narrative texts to understand how they stimulate transverse skills, citizenship, and social participation; To develop inclusive and cognitively accessible captions and narrative text models for the museum.

Thinking, Creativity) and digital competences; and iii) museum narratives that facilitate the participation of different categories of users, especially those at risk of marginalization. The operationalization of the research hypothesis is formulated into sets of objectives for each research question. They are both summarized in Table 1.

2.3 Methodology The study is conducted through quasi-experimental research and employs an evaluation apparatus that monitors the achievement of the objectives. Figure 1 shows the methodological structure adopted for this project. Potentially, many different target groups will be involved over a whole year: from all school grade levels to university students, including those with specific learning disorders (SLD) and special educational needs (SEN); from children to young adults, adults, and senior citizens, especially those at risk of social marginalization and cultural poverty; and from people with physical and cognitive disabilities to those with health conditions.

580

E. M. Torre

Fig. 1 Teaching methodologies for facilitated tours, Digital Storytelling workshops, and evaluation tools adopted for the quasi-experiment

Participants will be recruited through contact with schools, associations working in the cultural heritage sector to promote well-being and inclusion, and the museum’s channels. If necessary, their caregivers will accompany them. Participants will be informed of the project’s intent and how data will be gathered and analyzed. By declaring their willingness to join the project, they will commit to participating in the evaluation phase. They will also be informed of the ethical aspects concerning the sharing of their digital stories on the institutional channels of the MNR. All these aspects will be determined according to Sapienza’s and MNR’s guidelines and codes of ethics. Participants will engage in a series of tours and workshops facilitated by museum cultural mediators, who are the fundamental joining link with the Servizio Educativo’s staff (see Fig. 2). Co-participatory designs can have a top-down, center-out, or bottom-up approach. Regardless of this, they are realized when everyone involved is an active participant and has a voice to share; when everyone’s contribution, each according to his or her competences, is valued and is precious for the realization of services accessible to all. The Servizio Educativo’s staff indicates in which direction to realize the museum’s mission. Based on this, cultural mediators design facilitated tours and workshops and offer them to audiences. Participants give feedback on the effectiveness of the museum experiences and provide important inputs on how to improve physical and cultural accessibility, thanks to their reactions and engagement in evaluation; they

Fig. 2 Interactions among stakeholders

Narrating the Museum: Enhancing Cultural Heritage Through User …

581

also reinterpret museum collections starting from the emotions felt during tours, through their Digital Storytelling products. These, after validation by the Servizio Educativo’s staff, are made available to all on the museum’s institutional channels. They are free of technicalities and therefore understandable to non-expert audiences; they are also inclusive insofar as they combine audio, video, and text in such a way as to be usable by blind or deaf people. Having completed the testing and evaluation phases, museum cultural mediators implement the educational pathways in accordance with the Servizio Educativo’s staff and develop new models of inclusive captions and explanatory texts. Both tours and workshops are designed according to a socio-constructivist approach [22, 23] that promotes lifelong learning and co-participation. The former are conducted through Object-based Learning (OBL hereafter) and Visual Thinking Strategy (VTS hereafter); the latter are Digital Storytelling (DST hereafter) workshops, devoted to the realization of new products. OBL, VTS, and DST have been selected due to their inclusiveness, ability to involve participants emotionally, make them protagonists in their learning process, and promote transverse and digital skills. OBL is an active learning methodology [24], increasingly widespread in formal and informal educational settings, that uses objects to promote deeper levels of learning. By facilitating the involvement of learners, including neurodivergent people, those who show difficulties with written texts, have relational issues, have cognitive disabilities, or have autism spectrum disorders, it fosters emotional wellbeing [25–30]. In fact, OBL provides participants with a haptic experience, during which they are encouraged by facilitators and through questions to make observations on the object’s shape, derive meaning from it, discuss its function, and compare it to other objects while collaborating with their peers [31], thus promoting a more communal and democratic process of knowledge creation within the learning environment. Besides the 4 C’s skills, the exploratory process started through OBL stimulates observation and analysis skills, metacognition, and problem-solving abilities [32, 33], which all are necessary prerequisites for learning and the development of the XXI century society [34]. Furthermore, as objects are signifiers of culture, framing questions through a critical lens can foster learners’ awareness and help them become more informed citizens [35]. VTS is an inquiry-based teaching method that fosters student-centered learning by stimulating the ability to describe, analyze, and interpret imagery and information through thoughtful observations and inclusive discussions of visual art [32, 36]. By giving “permission to wonder” and learning independence, VTS empowers audiences at large regardless of their age [37]. VTS promotes the 4 C’s skills and aesthetic thinking, especially among learners who struggle with traditional text-based or lecture-based learning environments [37–40]. The method has proved to be useful also for hearing-impaired learners [36]. VTS is prompted through three main questions: i) what’s going on in this picture? ii) what do you see that makes you say that? iii) what more can you find? Furthermore, many aspects of VTS fall into the OBL practice [34], making this methodology a valid alternative to OBL whenever it is not possible to manipulate museum objects.

582

E. M. Torre

Storytelling is an intrinsically human need and facilitates the sharing of experiences and knowledge. It has the potential to bring new voices into the public debate, address social issues, and include groups at risk of marginalization by breaking down cultural and social barriers [5, 41], promoting the self-confidence needed to break out of social exclusion and access new opportunities [29]. Storytelling has a century-long history in the museum and cultural communication field [42] and has proved to be effective also with people with cognitive disabilities [29] or autism spectrum disorders [30]. Therefore, it is not only a tool for personal and creative expression but also an inclusive teaching and learning method. Despite this huge potential, only recently the proliferation of new and different technologies has led to the emergence of numerous forms of storytelling [43, 44]. In particular, the DST model created by Dana Atchley and Joe Lambert [5] is the most suitable to achieve this project’s purpose. This is because of its seven steps structure, the use of digital technologies — thus providing an occasion for digital literacy — and, first and foremost, its reliance on people interactions. The role played by emotion in the DST approach is pivotal to the promotion of “embodiment, which is a form of knowledge that exists in the telling of stories with emotional meaning” [41]. The DST process encompasses the four ways of knowing in the “extended epistemology”: experiential knowing, presentational knowing, propositional knowing, and practical knowing [45]. The application of DST to heritage education settings promotes the 4 C’s skills [41] and a number of competences required for successfully operating in a “knowledge society” — i.e., communication in the native language; digital skills; learning; and cultural awareness and expression [10]. DST is a co-creative media useful for developing participatory culture through a process that takes into account the agency of non-experts, professionals, organizations, and technology [46]. Evaluation is pivotal to determine whether the objectives are met. Levels of the 4 C’s and digital skills are measured throughout the educational path, also with reference to the DST products [34, 47, 48]. Observation grids validated by Poce and her research team will be filled out by facilitators [34, 49, 50]. Increases in mental well-being are assessed by administering the UCL Museum Well-being Measures Toolkit to participants in the pre-, mid-, and final stages of the educational path [51]. The toolkit consists of scales of measurement that assess levels of well-being arising from participation in museum activities. It measures psychological well-being as an indicator of the mental state of the individual by focusing on levels of self-reported changes in mood and emotion, both positive and negative. The choice fell on this particular tool because of its ease of use and versatility that make it appropriate to museum targets of diverse ages and needs. Focus groups will be carried out for deep data collection, especially when children, senior citizens, and people who prefer spoken interaction within a group are involved. They will be designed to investigate perceived levels of social inclusion and whether these are increased as a result of joining the educational path. User profiling through artificial intelligence processes is crucial to verify the research hypothesis. It aims at identifying the user’s starting point with respect to the objectives and the characteristics of a given group in order to offer individualized paths. Two approaches will be adopted, which blend together both “you are what you

Narrating the Museum: Enhancing Cultural Heritage Through User …

583

do” and “aspirational” profiles [12]. The implicit approach of eye-tracking applied to narrative texts identifies the parts that have been read the most and the longest [52, 53], to determine whether the stories are comprehensible or not. The explicit indirect approach consists of a brief questionnaire adapted from research conducted by Poce and her team. Through questions indirectly related to the content of the museum, it is possible to infer characteristics of the profiles of the individuals involved [54]. Data analysis of the profiling activity will also provide the basis for designing new inclusive models of captions and explanatory apparatuses. In particular, the “language preferences” section of the adapted questionnaire is especially useful for designing new caption models, while the “narrative preferences” section tests the DST products developed by the participants and is useful for new explanatory text models. In addition, both captions and explanatory text models will also be designed based on ministerial guidelines [55].

2.4 Expected Outcomes The research project has three main expected results: • delivery of activities to promote the 4 C’s and digital skills, mental well-being, and social inclusion for a wide range of museum targets, which can be adopted by any museum professional; • delivery of co-participatory DST pathways on the MNR’s institutional channels as a means of engaging the community, including those who do not physically visit museum venues for various reasons; and • design of new models of culturally accessible captions and explanatory texts applicable in different museum settings. The above-mentioned expected results are a valuable resource not only for the inclusiveness of museums and the conscious and active participation of the community but also for all educators who, regardless of the educational context in which they work, wish to innovate their educational practices in the name of inclusiveness. In particular, the second expected result might provide a way to attract at least some types of non-visitors, potentially intrigued by compelling narratives comfortably enjoyable from home.

3 Next Steps The present contribution illustrated the theoretical and methodological framework of the PhD research project Narrating the Museum: Enhancing Cultural Heritage through User Profiling and Individualized Content. The first section provides an overview of the function of narratives in museum settings and a general framework for lifelong learning competences, well-being, and social inclusion at the EU level.

584

E. M. Torre

The second section addresses the methodology adopted in this project. It encompasses several subsections devoted to the description of i) the museum setting; ii) the research hypothesis, the research questions, and the objectives; iii) the teaching and learning methodologies that will be employed during facilitated tours and workshops, the evaluation tools, and the user profiling apparatus; and iv) the expected outcomes. In recent years, Italian museums have been achieving milestones in terms of inclusivity and accessibility; nonetheless, co-participatory designs are still a rare occurrence. The main contribution of this research project is to promote, in the context of the MNR, a form of democratization of museum practices that involves museum staff, museum cultural mediators, and non-experts in a participatory process leading to the co-creation of new interpretations of the collections. Promoting accessibility and cultural participation is a viable way to foster mental well-being and social inclusion as well as active citizenship at a community level; however, much work is yet to be done. Therefore, research in the field ought to continue to investigate suitable methodologies and indicators to detect levels of quality of life, social cohesion, and community development, with particular attention to both social groups at risk of marginalization and non-visitors. Acknowledgements This research project is being carried out within the University La Sapienza National PhD Programme in Heritage Science and funded by the University of Modena and Reggio Emilia. I wish to thank the Director of the National Roman Museum, Prof. Stéphane Verger, for his availability and the warm welcome I have received since the beginning of this research project. Last but not least, I am sincerely grateful to my mentor and PhD tutor, Prof. Antonella Poce, for her valuable teachings and constant support.

References 1. Proshansky, H.M., Fabian, A.K., Kaminoff, R.: Place-identity: physical world socialization of the self. J. Environ. Psychol. 3(1), 57–83 (1983). https://doi.org/10.1016/S0272-4944(83)800 21-8 2. McMillan, D.W., Chavis, D.M.: Sense of community: a definition and theory. J. Community Psychol. 14(1), 6–23 (1986). https://doi.org/10.1002/1520-6629(198601)14:1%3c6::AID-JCO P2290140103%3e3.0.CO;2-I 3. Chynoweth, A., Lynch, B., Petersen, K., Smed, S.: Museums and Social Change: Challenging the Unhelpful Museum, 1st edn. Routledge, London (2020). https://doi.org/10.4324/978042 9276903 4. Vaglio, M.G.: Lo storytelling per i beni culturali: il racconto. In: Dal Maso, C. (ed.) Racconti da museo. Storytelling d’autore per il museo 4.0, pp. 27–52. Edipuglia, Bari (2018) 5. Lambert, J., Hessler, B.: Digital Storytelling: Capturing Lives, Creating Community. Routledge, New York-London (2018) 6. Hickok, G.: The Myth of Mirror Neurons: The Real Neuroscience of Communication and Cognition. WW Norton & Company, New York-London (2014) 7. Zaki, J.: The war for Kindness: Building Empathy in a Fractured World. Broadway Books, New York (2020) 8. Kolb, D.A., Plovnick, M.S.: The experiential learning theory of career development. Working paper (Sloan School of Management), pp. 705–74 (1974)

Narrating the Museum: Enhancing Cultural Heritage Through User …

585

9. Poce, A. (ed.): Memory, inclusion and learning experiences in cultural heritage. First results of the Inclusive Memory project by Roma Tre University. Edizioni Scientifiche Italiane, Napoli (2020) 10. EUR-Lex Access to European Union Law. https://eur-lex.europa.eu/legal-content/EN/TXT/? uri=uriserv:OJ.C_.2018.189.01.0001.01.ENG&toc=OJ:C:2018:189:TOC. Accessed 04 Mar 2023 11. Pietrelcina Chart on Digital Cultural Heritage Education. https://www.diculther.it/blog/2020/ 01/01/carta-di-pietrelcina-sulleducazione-alleredita-culturale-digitale/. Accessed 10 Mar 2023 12. Simon, N.: The Participatory Museum. Museum 2.0, Santa Cruz (2010) 13. Faro Convention Homepage. https://www.coe.int/en/web/culture-and-heritage/faro-actionplan. Accessed 12 Mar 2023 14. LEGGE 1 ottobre 2020, n. 133: Ratifica ed esecuzione della Convenzione quadro del Consiglio d’Europa sul valore del patrimonio culturale per la società, fatta a Faro il 27 ottobre 2005. (20G00152). https://www.gazzettaufficiale.it/eli/gu/2020/10/23/263/sg/pdf. Accessed 12 Mar 2023 15. United Nations Homepage. https://sdgs.un.org/2030agenda. Accessed 04 Mar 2023 16. Health promotion glossary of terms 2021. World Health Organization, Geneva (2021). License: CC BY-NC-SA 3.0IGO. https://sdgs.un.org/2030agenda 17. Foresight Mental Capital and Wellbeing Project. Final Project Report – Executive summary. The Government Office for Science, London (2008) 18. Museo Nazionale Romano Homepage. https://museonazionaleromano.beniculturali.it/. Accessed 04 Mar 2023 19. Museo Nazionale Romano Website. https://museonazionaleromano.beniculturali.it/attivitaeducative/museo-per-tutti/. Accessed 04 Mar 2023 20. Caruso, C.: 101 Storie Svelate. Le iscrizioni del Museo Nazionale Romano raccontano Roma. Dielle Editore, Roma (2022) 21. Museo Nazionale Romano Website. https://museonazionaleromano.beniculturali.it/vedi-allavoce/. Accessed 04 Mar 2023 22. Bruner, J.: Life as narrative. Social research: an international quarterly 71(3), 691-710 (2004) 23. Rounds, J.: Meaning making: a new paradigm for museum exhibits. Exhibitionist 18(2), 5–8 (1999) 24. Pollak, D. (ed.): Neurodiversity in Higher Education: Positive Responses to Specific Learning Differences. Wiley-Blackwell, Chichester (2009) 25. Freeman, S., et al.: Active learning increases student performance in science, engineering, and mathematics. Proc. Natl. Acad. Sci. U.S.A. 111, 8410–8415 (2014) 26. Hannan, L., Chatterjee, H., Duhs, R.: Object based learning: a powerful pedagogy for higher education. In: Boddington, A., Boys, J., Speight, C. (eds.) Museums and Higher Education Working Together: Challenges and Opportunities, pp. 159–168. Ashgate Publishing, Farnham (2013) 27. Kador, T., Chatterjee, H.: Object-Based Learning and Well-Being: Exploring Material Connections. Routledge, London (2020) 28. Chatterjee, H., Noble, G.: Museums Health and Well-Being. Routledge, London (2017) 29. Re, M.R.: Digital Storytelling e Object-based Learning per la promozione dell’inclusione sociale e del benessere al museo. In: Poce, A. (ed.) Studi empirici di educazione museale 2, pp. 79-94. Edizioni Scientifiche Italiane, Napoli (2022) 30. Poce, A., Re, M.R., Swift, P., Vander Stichele, A., Rodrigo San Juan, C., Chichy, B.: How to make museums autism-friendly. The Erasmus+ Spektrum Project. In: Poce, A. (ed.) Pensiero critico tra scuola, università e mondo del lavoro. Esperienze innovative di formazione, pp. 123– 140. Edizioni Scientifiche Italiane, Napoli (2022) 31. Romanek, D., Lynch, B.: Touch and the value of object handling: final conclusions for a new sensory museology. In: Chatterjee, H.J. (ed.) Touch in Museums: Policy and Practice in Object Handling. Berg, Oxford and New York (2008) 32. Hubard, O.: Rethinking critical thinking and its role in art museum education. J. Aesthetic Educ. 45(3), 15–21 (2011)

586

E. M. Torre

33. Poce, A., Re, M.R., Valente, M., De Medio, C.: Musica e stampa 3D. Promuovere le competenze trasversali in alunni di scuola primaria tramite la riproduzione di strumenti musicali etruschi. In: Antonella Poce (curated by), Veicolare l’inclusione attraverso il patrimonio. Alcuni risultati del progetto Inclusive Memory dell’Universit‘a Roma Tre, pp. 13-31. Edizioni Scientifiche Italiane, Napoli (2019) 34. Poce, A.: Il patrimonio culturale per lo sviluppo delle competenze nella scuola primaria. Franco Angeli, Milano (2018) 35. Lelkes, J.: How inclusive is object-based learning? Spark: UAL Creat. Teach. Learn. J. 4(1), 76–82 (2019) 36. liyah, A., Syarif, F.: Integrating Visual Thinking of Hearing Impaired Students in Inclusive Classroom. Studia Religia: Jurnal Pemikiran dan Pendidikan Islam 4(1), 108–130 (2020) 37. Choi, J., et al.: Integration of visual thinking strategies to undergraduate health assessment course: a mixed-method feasibility study. Nurse Educ. Today 113, 105374 (2022) 38. Albert, C.N., Mihai, M., Mudure-Iacob, I.: Visual thinking strategies—theory and applied areas of insertion. Sustainability 14(12), 7195. MDPI AG. (2022). https://doi.org/10.3390/su1 4127195 39. Housen, A.C.: Aesthetic thought, critical thinking and transfer. Arts Learn. Res. 18(1), 99–132 (2001–2022) 40. Moeller, M., Cutler, K., Fiedler, D., Weier, L.: Visual thinking strategies = creative and critical thinking. Phi Delta Kappan 95(3), 56–60 (2013). https://doi.org/10.1177/003172171309 500312 41. Liguori, A., Bakewell, L.: Digital storytelling in culture and heritage education: a pilot study as part of the DICHE project. In: Poce, A. (ed.) Advanced Studies in Museum Education. Lectures, pp. 63–78. ESI, Napoli (2019) 42. Cataldo, L.: Dal Museum Theatre al Digital Storytelling. Nuove forme della comunicazione museale fra teatro, multimedialità e narrazione. Franco Angeli, Milano (2011) 43. Lughi, G.: Interactive Storytelling. In: Arcagni, S. (ed.). I media digitali e l’interazione uomomacchina, pp. 169–190. Aracne Editrice, Roma (2015) 44. Bonacini, E.: I musei e le forme dello storytelling digitale. Aracne, Canterano (2020) 45. Liguori, A.: Unlocking contested stories and grassroots knowledge. In: Trifonas, P.P. (ed.) Handbook of Theory and Research in Cultural Studies and Education. Springer International Handbooks of Education, pp. 465–479. Springer, Cham (2020). https://doi.org/10.1007/9783-319-56988-8_35 46. Spurgeon, C., Burgess, J., Klaebe, H., McWilliam, K., Tacchi, J.A., Tsai, Y.-H.: Co-creative media: Theorising digital storytelling as a platform for researching and developing participatory culture. In: Flew, T (ed.) Communication, Creativity and Global Citizenship: Refereed Proceedings of the Australian and New Zealand Communication Association Conference 2009. Australian and New Zealand Communication Association, Online, pp. 274–286 (2009) 47. Poce, A.: Verba Sequentur. Pensiero e scrittura per uno sviluppo critico delle competenze nella scuola secondaria. Franco Angeli, Milano (2017) 48. Poce, A., Amenduni, F., De Medio, C., Re M.R.: Road to critical thinking automatic assessment a pilot study. Form@re – Open Journal per la formazione in rete 19(3), 60–72 (2019) 49. Poce, A. (ed.): Tecnologia critica, Creatività e Didattica della Scienza. Franco Angeli, Milano (2015) 50. Re, M.R.: Digital storytelling in museum education context: how to promote critical thinking skills through digital tools within secondary school pupils. In: EDEN Proceedings 2021 Annual Conference, pp. 135–144 (2021) 51. Thomson, L., Chatterjee, H.J.: UCL Museum Wellbeing Measures Toolkit. UCL, London (2013) 52. Di Giovanni, E.: Museum through the visitor’s eyes: eye tracking and museum fruition. In: Poce, A. (ed.) Studi avanzati di educazione museale. Lezioni, pp. 79–92. Edizioni Scientifiche Italiane, Napoli (2019) 53. Di Giovanni, E.: Eye tracking and the museum experience in Italy. Altre Modernità. Saggi 24, 10-24 (2020)

Narrating the Museum: Enhancing Cultural Heritage Through User …

587

54. Poce, A., Amenduni, F., Re, M.R., De Medio, C., Valente, M.: Raccogliere informazioni sui visitatori per fornire percorsi personalizzati e inclusivi all’interno dei musei: il questionario e la web app Inclusive Memory. In: Poce A. (ed.) Memoria, inclusione e fruizione del patrimonio culturale. Primi risultati del progetto Inclusive Memory dell’Università Roma Tre, pp. 25–42. Edizioni Scientifiche Italiane, Napoli (2020) 55. Da Milano, C., Sciacchitano, E.: Linee guida per la comunicazione nei musei: segnaletica interna, didascalie e pannelli. Quaderni della valorizzazione – NS 1 (2015)

Closing the Gender Gap in STEM MOOCs Through Brief, Novel Interventions Alexandra D. Urban

Abstract Stemming from societal inequities, women have consistently lower persistence rates in Science, Technology, Engineering, and Mathematics (STEM). This study used previous insights to design in-course prompts for STEM Massive Open Online Courses (MOOCs) to support women and counteract external factors negatively affecting their learning progress. The self-determination theory of intrinsic motivation was used to align the novel intervention prompts with women’s sense of competency, autonomy, and relatedness and compensate for external drivers of women’s lower retention. The intervention study deployed to learners in 150 STEM MOOCs to boost confidence, improve planning, and emphasize individuals’ values to counteract gender inequality. The Coursera platform was used to conduct a randomized controlled trial (RCT) experiment, allowing causal quantitative data analysis of these interventions’ impact on learners’ persistence. This RCT study included 324,457 total active learners with identified gender. The four treatment groups (three variant types, plus the combined treatment) each resulted in a significant increase in first-week completion rates for women compared with the control. The value relevance treatment group retained this significant increase, successfully eliminating the gender gap in STEM MOOC completion. The self-efficacy treatment significantly raised the number of women completing the course by 50% in the youngest age tier. Given the double-blind RCT research design, this study goes beyond correlational evidence and suggests that moving all of the active learners in this sample from the control group to the value relevance treatment would result in approximately 1,400 additional women completing their STEM course. Implications for future research and practice are explored, including personalized message deployment given differences in impact by age, gender, and geography. Keywords STEM · MOOC · Gender gap · Intrinsic motivation · Completion · Randomized experiment · Empirical study

A. D. Urban (B) Coursera, Mountain View, CA 94041, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_48

589

590

A. D. Urban

1 Introduction In Science, Technology, Engineering, and Mathematics (STEM) subjects, men complete Massive Open Online Courses (MOOCs) on Coursera at a rate more than 30% higher than their peers, even after controlling for women’s lower enrollment rates [6, 18]. The COVID-19 pandemic brought disproportionately more women to the Coursera platform than in previous years and led to elevated female enrollment numbers, with 37% of STEM enrollments in 2021 from women compared to 31% in 2019 [10]. This increased share of women enrolling in scientific and technical courses makes in-course interventions to improve their retention even more needed. To ad-dress the pervasive and persistent gender gap in STEM retention, the researcher created light-touch interventions to counteract challenges ranging from national-level pat-terns of bias and inequality [11, 21] to individual differences in confidence and connection to the material [3, 14, 36, 45]. This study provides insights on how brief, text-based messages can significantly impact retention at scale. In the remainder of this article, the theoretical framework and relevant research literature are explored before outlining the specific methods of this experiment. Given learners’ random assignments to the control or treatment groups, this experiment provided the opportunity to draw causal conclusions on any observed outcome differences, which are explored in the Results section. Broader analysis and implications are presented in the Discussion section. Future directions are also considered to further this research and continue empowering women in online courses worldwide.

2 Theoretical Framework Using the Self-Determination Theory (SDT) framework, the researcher expected these prompts to increase intrinsic motivation and further engagement with the course con-tent by fulfilling the psychological needs of competency, autonomy, and relatedness [47]. Largely the result of cultural conditioning and external factors beyond their control, women have consistently shown lower levels of competency, autonomy, and relatedness in STEM courses [29, 38, 40]. With societies and individuals consistently undermining women’s confidence, freedom, and sense of belonging in STEM, women show understandably lower rates of motivation, on average, to continue [30, 38, 40]. This induced lower motivation can cause women to drop out of higher education STEM courses, often at far higher rates than their peers who identify as men [23, 29, 38]. While it may seem counterintuitive to tackle these external factors through ad-dressing intrinsic motivation, interventions based on the SDT framework have suc-cessfully elevated women’s persistence and performance in STEM courses and offer a method for better supporting women while compensating for societal inequities [7, 17].

Closing the Gender Gap in STEM MOOCs Through Brief, Novel …

591

3 Synthesis of Relevant Research Literature The most critical factors from the recent needs assessment study included women’s lower self-efficacy, immense time constraints, and challenges from genderimbalanced societies. The researcher used regression models, hypothesis tests, and qualitative coding by theme to assess why learners drop out [43]. Many high correlation values for behavioral relationships were found [24, 41]. National gender inequality, one factor tested in the recent needs assessment [43], showed a significant correlation with the average number of completed STEM MOOCs per women by country (R2 = 0.31), indicating how societal level factors can significantly impact individual’s learning behavior. In addition, women’s completion rate was linked more strongly to the average course completion time (R2 = 0.25) than the completion rate for men was (R2 = 0.21). This result indicates how schedules may be less flexible for women, and course length alone can explain one-quarter of their variation in completion. Two of the most frequent themes from women’s self-reported reasons for dropping out of a STEM MOOC before completing were “no time” (21%) and “not confident” (14%). While women are not inherently less confident in their abilities, societies’ stereotypes and treatment of women, especially within STEM disciplines, makes them feel less like they belong and can achieve, despite the evidence of women’s strong math and science abilities [7, 30, 38, 40]. Thus, this intervention included four versions of each course with prompts to (a) boost confidence, (b) improve planning, (c) align the content with individuals’ values, and (d) a combination of these three interventions into an extra-strength variant. While these intervention approaches have been shown to help women in varying contexts, each was also designed to help a specific group of learners often needing greater support. The first intervention option may most assist younger students and those without a background in the topic [4, 22, 26, 33]. The second option could es-pecially aid those with more family, home, and job responsibilities [1, 12, 31]. Finally, the third option was poised to most benefit those joining online courses from develop-ing countries dealing with the challenges of societies with greater gender inequality [19, 20]. Combining the three prompt types may be most beneficial by providing a variety, may fail to emphasize any single message, or may overwhelm learners with too many pop-up messages [28], which is why the stronger treatment option was tested separately. These four variants, tested alongside a fifth control version, provid-ed a systematic experiment to assess the effectiveness of lightweight messages for empowering women in STEM MOOCs.

4 Research Questions This study included both process and outcome evaluation methods, each with related research questions. These three questions measured concrete implementation and impact indicators, creating a holistic view of this intervention’s success.

592

A. D. Urban

RQ1. To what extent did the intervention reach the target learner group? RQ2. To what extent did learners find the prompt helpful? RQ3. What differences in impact did each intervention have on week one and course completions?

5 Methods The intervention research emphasized quantitative analysis, aligning with best practices for analyzing the data from a large-scale randomized controlled trial (RCT). This quantitative approach allowed the researcher to assess any statistically significant findings [5].

5.1 Participants Learners’ RCT participant assignment, behavior in the courses, and grades were saved automatically in the back-end of the Coursera system. The researcher had access to these actions and answers only in aggregate without any personally identifiable information (PII). This data collection plan falls within Coursera’s Terms con-cerning Educational Research. The United Nations’ Gender Inequality Index from 2018 provided a quantitative summary of each country’s gender inequality level in the analyses. When executed successfully, RCTs mitigate most threats to internal validity by removing the human selection process through a double-blind, randomized process and isolating treatment effects as the only experience changes across groups [37]. This online double-blind RCT also avoided the potential issues of participant reactivity from knowledge of group assignment, experimenter’s expectations, and any compensatory treatment provided to the control group since the experimenter herself did not have any direct contact with the learners [37].

5.2 Intervention This implementation focused on integrating research-based interventions within the online course learning experience. This researcher divided newly enrolled learners into five intervention groups within each STEM MOOC. The first four groups (selfefficacy, planning, value connection, and a combination of the first three) aimed to assist the main areas identified in the recent needs assessment. The fifth acted as the control group with no new enhancements added. These intervention variants were designed to be as light-touch as possible to avoid interrupting each course’s learning

Closing the Gender Gap in STEM MOOCs Through Brief, Novel …

593

materials and activities. Across all five groups in the sample, the coursework looked the same for each given STEM MOOC. After learners enrolled, they were shown prompted messages, each based on selfdetermination theory (SDT) literature, or nothing if they were in the control group. In each of the first three intervention designs, four text-based, in-course messages were surfaced directly to learners at similar milestones throughout the first three weeks of the online course in which they had enrolled. In the fourth version, the combination of the previous three intervention approaches purposely offered an increased treatment strength with six total in-course messages. Each pop-up included a title, a text-based message, a clickable link to the research when appropriate, and a quick helpfulness question. These milestones were broad enough that all 150 STEM MOOCs included in the study had each of the following trigger moments for the planned messages. Prompt Design. The first intervention was focused on boosting self-efficacy. This treatment applied learnings from Huang and Mayer’s [17] experiment to test the utility of self-efficacy-boosting messages in online statistics courses. Consistent with these researchers’ findings, this intervention’s messaging emphasized students’ already demonstrated efforts and accomplishments while also offering encouragement in the form of praise to increase their self-confidence and persistence [17]. When learners enrolled, they were congratulated on taking the first step to increase their learning. Then, after each of the first three graded assessments, students received automated messages depending on their performance. The second intervention aimed to provide planning support. The researcher aimed to combine insights from other settings and previous Coursera trials into the proposed design. For example, highlighting data on how certain practices can increase their likelihood of success has been beneficial in other in-course interventions previously tested on Coursera [15, 42, 43]. This intervention variant also congratulated learners on their successful planning as they progressed through the course material. This praising message leveraged other researchers’ insights on how adding planning support can increase learners’ feelings of confidence and competence [25, 35]. The third intervention group learners received in-course messages prompting reflection on their values and goals. Just as previous researchers have tested in other course settings, the researcher started by surfacing specific value areas and asking learners to reflect on which is most important to them [21, 27, 32]. As learners continued to progress in the MOOC, they received encouragement and reminders related to their values and goals. Reminding learners of their values during the course journey has proven helpful in STEM content, especially for women [32] and those from less developed countries [21]. The fourth treatment group combined prompts from the first three intervention groups to test both a stronger overall treatment (i.e., more messages) and if these interventions could be beneficial when presented together. For this fourth version, the course had six in-course messages with two prompts taken directly from each of the first three variant conditions. Cognitive scientists warn educational researchers about overwhelming students with too much to process [28], and some investigators

594

A. D. Urban

have even found additional items to process linked with lower self-efficacy in STEM undergraduate courses [9], which would be counterproductive to the goals of this study. However, just-in-time nudges can be useful to learners, particularly in online settings [15, 46]. Thus, this combination group allowed the testing of a stronger treatment of messages to assess if the benefits outweigh the negatives when surfacing these prompts in a single course experience. The fifth group was the control. Learners in the control group saw the same course experience as learners before this experiment was implemented. These learners in the control sample did not see any of the prompted messages designed for this interven-tion. Additionally, all other in-course messages from other Coursera experiments were turned off during this experiment to isolate these SDT messages as the only systemat-ic change between intervention groups. Each STEM MOOC included the same coursework across all treatment and control groups as well as the typical functionality within the Coursera learning experience. As is recommended for RCT designs, the control group provided an identical experience besides the intervention variants de-scribed above for the four treatment groups, enabling a more successful causal im-pact analysis [34, 37]. Thus, learners in the control sample saw the same STEM MOOC experience on Coursera as before this experiment began, without any of the above intervention messages shown.

5.3 Data Collection and Analysis Data were collected across 150 of the highest enrollment STEM MOOCs on the Coursera platform. The full sample for this experiment consisted of 242,847 women and 365,949 men, representing all learners with identifiable gender who newly enrolled in one of the 150 MOOCs during the experiment’s open period, from December 8, 2021, to March 20, 2022. Learners were included in either the women or men group given their own self-selection of gender within their Coursera profile or in an account linked to their Coursera account. This full sample was used for the first two research questions to assess the implementation process. Separately, the outcome question was designed to examine only the learners who had successfully started the in-course learning activities, requiring a narrowing of the sample pool to active learners: 131,804 women and 192,653 men. For the outcome question quantitative analysis, two-way analysis of variance (ANOVA) inferential tests were used to assess the causal impact of this RCT across treatments and genders. Given the repeated statistical tests planned and the large sample size of more than 300,000 active learners, the researcher corrected the alpha to a 0.01 level to increase statistical validity.

Closing the Gender Gap in STEM MOOCs Through Brief, Novel …

595

6 Results 6.1 Research Question 1: Learner Audience Reached The percentage of women in each intervention is similar over time and among groups. The final distribution of women ranged from 39.6% to 39.9% across the intervention groups. The back-end randomization process occurred as designed to create an RCT experiment with approximately equal-sized groups. Since this back-end system was randomizing at the learner level and blind to gender during group assignment, the researcher wanted to ensure approximate consistency in the women-to-men balance across the intervention sub-groups. A two-way ANOVA test did not detect any meaningful differences in the gender balance of these groups (p = 0.093), indicating insufficient evidence to suggest any meaningful differences in these groups’ gender ratio or size. There was, as expected, a statistically significant difference in the number of men versus women in each treatment group (p < 0.0001). However, the overall sample portion of women (nearly 40%) was a higher percentage than in previous research, where women comprised less than 30% of STEM enrollments [6, 18]. This larger proportion in the intervention study shows how women were especially interested in the 150 top-performing STEM MOOCs included in the experiment.

6.2 Research Question 2: Prompts’ Helpfulness Rating Each treatment group demonstrated a helpfulness average across messages of 86%– 92%, a significantly higher average rating (t – test, p < 0.0001) than the average helpfulness of 70%–75% observed for previous messages shown through random assignment to users on the Coursera platform [16]. As expected, the average helpfulness indicated by women was nearly equal to or higher than the average of the overall learner sample. The two-way ANOVA analysis revealed a significant difference between treatment group messages (p < 0.0001), resulting from the self-efficacy and value relevance treatments having consistently higher helpfulness ratings than the planning and combined groups. The difference between the gender columns was not significant (p = 0.146), meaning there is insufficient evidence to suggest any systematic differences between men’s and women’s perceived helpfulness of the messages across treatment groups. This intervention demonstrated an overall response rate of 17.7%, with 17.4% for the self-identifying women in the sample, meaning that more than one in six learners responded to the “yes” or “no” helpfulness question across all treatment groups. Lastly, learners who responded to the helpfulness question showed no difference in completion rate. The active learners who identified as women and indicated “yes” or “no” to the helpfulness question had an average completion rate across the four

596

A. D. Urban

active treatment groups of 16.8% as compared with the overall completion rate for these active learners across the four groups of 17.1% (t = –1.73, p = 0.083). This similarity demonstrates the representativeness of this responsive subsample.

6.3 Research Question 3: Impact on First Week and Course Completions The two-way ANOVA test revealed a significant difference in how gender affected learners’ first-week completion rates in the experiment ( p = .0097, see Fig. 1). All four treatment groups demonstrated significantly higher persistence than the control when examining only women (t − test, p < .001 for each pair). Additionally, the learners who identified as women showed significantly higher first-week completion rates than the self-identifying men in the self-efficacy boost, value relevance connection, and combined intervention groups (t − test, p < .01 for each pair by gender). The error bars in Fig. 1 show the intervals for the true value of each subgroup at the 99% confidence level and do not overlap within each treatment pair, except for the planning support group, demonstrating the statistically significant differences between the values at the 0.01 alpha level for the other four groups including the

Fig. 1 Graph of learners’ first-week completion rates by intervention group and gender

Closing the Gender Gap in STEM MOOCs Through Brief, Novel …

597

control. Looking only at the women’s completion rates, the confidence intervals also do not overlap between the treatment groups, as viewed together, and the control group. This increase from 41 to 44% of learners finishing the first week represents an additional 500 total women who completed this first week of materials and assessments in the self-efficacy, value relevance, and combined intervention groups than would have if all these women were in the control group. Also of note, the first-week completion rates of men and women in the control group were substantially closer than in previous research [6, 18]. This control finding is likely the result of a stronger pedagogical design: the STEM courses with the highest enrollment also tend to have the highest ratings and best teaching strategies with clear, useful assessments. However, women in the control group still demonstrated a significantly lower first-week completion rate than the men in the control group (t − test, p = .0085). When examining the effect on the full course, there is a clear interaction between gender and treatment group, with only men showing remarkably consistent completion rates across groups (ANOVA, p = 0.0054, see Fig. 2). Although the impact of the self-efficacy boost and planning support waned over the duration of the course, the value relevance emphasis and combined intervention groups showed sustained benefits for the self-identified women. Notably, the women in the value relevance group demonstrated a completion rate indistinguishable from the men in the sample (t − test, p = 0.076). Thus, the value relevance intervention successfully closed the gender gap in learners’ STEM MOOC completion rates. Both the value relevance emphasis and combined intervention groups increased women’s course completion rates significantly above the women’s control group completion rate (t – test, p < 0.001 for each pair). While the increase in women’s course completion rate from the control to value relevance groups may not appear meaningful, this difference resulted in 7% more women completing the course. With the double-blind RCT design, these experimental results go beyond correlational findings and suggest that moving all the women in this experimental sample from the control group to the value relevance condition would result in approximately 1,400 additional women completing the STEM MOOC in which they voluntarily enrolled. Young women (ages 18–24) who were active learners in the self-efficacy-boost group demonstrated a course completion rate of 13.21% compared to only 8.97% in the control group (t – test, p < 0.0001, n = 315). While still lower than the overall completion rate for women active learners, this finding demonstrates the importance of investigating the impact by varying demographic groups. With this intervention sample, the researcher mapped countries according to their 2018 United Nations Gender Inequality Index (UN GII) score, the same country score used during the earlier needs assessment study [43]. After narrowing to countries with at least 35 unique women who became active learners in the RCT, this researcher conducted linear regressions to analyze the impact of the value relevance treatment group on completion rate by county’s UN GII score. This model found a negative correlation between the course completion rate in the control group and gender inequality level, where a lower number equates

598

A. D. Urban

Fig. 2 Graph of learners’ course completion rates by intervention group and gender

to greater gender equality (β = −0.04, R 2 = 0.02) across the 82 countries. For the value relevance treatment group, the plotting of women active learners’ course completion rate by country’s UN GII resulted in a nearly flat trend line (β = −0.007, R 2 = 0.0006; n = 82countries). While not a significant correlation ( p = 0.202forcontroland p = 0.831forvaluerelevance), this flattening of the correlation to almost zero suggests the value relevance intervention potentially helped counterbalance the effect of a nation’s gender inequality on women’s likelihood to complete the MOOC. See the Appendix for a graphical representation of this finding.

7 Discussion These research-based, light-touch interventions significantly increased women persistence and completion in the most popular STEM MOOCs on Coursera. All four treatments raised women’s first-week completion rate significantly above that of women in the control group. The self-efficacy-boost, value relevance emphasis, and combination treatment groups demonstrated significantly higher first-week completion rates than the men in the first week. This result was more conclusive than expected and showed how potent even brief interventions could be in asynchronous

Closing the Gender Gap in STEM MOOCs Through Brief, Novel …

599

online courses. Minor encouragements designed to increase women’s intrinsic motivation, drawing on their competence, autonomy, and relatedness, yielded promising results. The impact of these prompts on course completion rate was also encouraging. While the 4% gender gap witnessed during the needs assessment [43] would be challenging to close with only a few text-based prompts, the baseline difference in the completion rate witnessed for these top 150 STEM MOOCs during the RCT was 1.2%. The most popular courses on the Coursera platform tend to have the most thoughtful designs and highest-quality pedagogical approaches, which previous authors have found can provide the most assistance to women [15, 39, 44]. With this reduced baseline gender gap and the impact of these novel prompts, the value relevance group resulted in men and women course completion rates that were indistinguishable. This finding demonstrates that women’s completion rate significantly in-creased when their values were highlighted and linked to their learning journey, successfully closing the gender gap in STEM MOOC completion. Differences were observed across subgroups of women. For younger women, the self-efficacy intervention was especially helpful. Specific subgroups of women are most likely to drop out before course completion, such as younger learners and those joining from developing countries, so it is encouraging to observe the self-efficacy boost increasing the number of young women completing their course by 50%. The self-efficacy boost and value relevance treatments displayed the highest helpfulness ratings from women. In particular, the “This learning is for you” message, as part of the value relevance treatment condition, was the most highly rated message across the entire experiment for all learners and women, with 94.2% of the latter group indicating it as “yes” helpful. Ultimately, this message was part of the treatment that had the largest benefits for women’s course completion rate.

7.1 Limitations Operationalizing the retention metrics in various ways to include both qualitative and quantitative indicators would have strengthened the construct validity of this study [37]. Furthermore, any significant findings are only applicable to MOOCs. While there may be lessons that could transfer to smaller online courses or inperson, lecture-based classes, the results of this experiment are from large-scale, online, asynchronous courses. Maintaining a setting consistent with the context of this experiment is the best method for ensuring external validity [37].

7.2 Implications for Research and Practice Greater personalization and intentional deployment would amplify the impact of these interventions. Researchers and practitioners could match learners with the

600

A. D. Urban

types of in-course messages that would provide them the greatest benefit. Given the successful increase in women’s first-week completion rates across all four treatments, extending the messages into later weeks of the course may also be fruitful. Finally, while not possible within this intervention study, more resource-heavy interventions, such as creating personalized schedules given learners’ time demands, would be useful to explore.

8 Conclusion This RCT intervention experiment successfully improved women’s retention, erasing the gender gap in STEM MOOC completion rate in the value relevance treatment group. Different subgroups demonstrated the nonuniform impact of these treatments, with the self-efficacy boost most benefiting the youngest women in the sample. While completions are not the end goal, they are often the prerequisite to having a MOOC lead to the greatest impact in a learner’s life. In particular, MOOC completers can experience increases in job opportunities and salary [13]. In the few months of this intervention, hundreds more women completed their STEM MOOC than would have if all were in the control group, increasing these learners’ potential for career gains and mobility. At the national level, countries observe increased technical innovations and entrepreneurship activity when more women work in STEM fields [2, 8], highlighting the cascading benefits of helping learners complete STEM MOOCs on Coursera. Empowering women to be more successful in online science and technology courses benefits these individuals, their families, and the surrounding communities. Acknowledgements The author would like to thank Dr. Linda Lohr, Dr. Ranjini JohnBull, and Dr. Alexandra Murtaugh for their continued support of this research.

Appendix Women’s Control Course Completion Rate by Country’s Gender Inequality

Closing the Gender Gap in STEM MOOCs Through Brief, Novel …

601

Note. Each bubble represents the active learners who identify as women from a single country. The vertical axis is the percentage of these learners who completed the STEM MOOC they enrolled in as part of the control group of this intervention experiment, and the horizontal axis is the country’s UN GII score, with higher scores indicating greater inequality. The bubble’s diameter represents the relative number of woman-identifying active learners from that country enrolled in the RCT. Women’s Value Relevance Course Completion Rate by Country’s Gender Inequality

602

A. D. Urban

Note. Each bubble represents the active learners who identify as women from a single country. The vertical axis is the percentage of these learners who completed the STEM MOOC they enrolled in as part of the value relevance treatment group of this intervention experiment, and the horizontal axis is the country’s UN GII score, with higher scores indicating greater inequality. The bubble’s diameter represents the relative number of woman-identifying active learners from that country enrolled in the RCT.

References 1. Allione, G., Stein, R.M.: Mass attrition: an analysis of drop out from principles of microeconomics MOOC. J. Econ. Educ. 47(2), 174–186 (2016). https://doi.org/10.1080/00220485. 2016.1146096 2. Beede, D.N., Julian, T.A., Langdon, D., McKittrick, G., Khan, B., Doms, M.E.: Women in STEM: a gender gap to innovation. SSRN Electron. J. (2011). https://doi.org/10.2139/ssrn.196 4782 3. Charleston, L.J., Lang, N.M., Adserias, R.P., Jackson, J.F.L.: Intersectionality and STEM: The role of race and gender in the academic pursuits of African American women in STEM. J. Progr. Policy Pract. 2(3), 273–293 (2014) 4. Chyung, S.Y.: Age and gender differences in online behavior, self-efficacy, and academic performance. Q. Rev. Distance Educ. 8(3), 213–222 (2007). https://www.infoagepub.com/qrdeissue.html

Closing the Gender Gap in STEM MOOCs Through Brief, Novel …

603

5. Bedard, K.K., Marks, A.K.: Mixed methods. In: Loue, S., Sajatovic, M. (eds.) Encyclopedia of Immigrant Health, pp. 1093–1096. Springer, New York (2012). https://doi.org/10.1007/978-14419-5659-0_519 6. Crues, R.W., Henricks, G.M., Perry, M., Bhat, S., Anderson, C.J., Shaik, N., Angrave, L.: How do gender, learning goals, and forum participation predict persistence in a computer science MOOC? ACM Trans. Comput Educ. 18(4), 1–14 (2018). https://doi.org/10.1145/3152892 7. Dell, E.M., Verhoeven, Y., Christman, J.W., Garrick, R.D.: Using self-determination theory to build communities of support to aid in the retention of women in engineering. Eur. J. Eng. Educ. 43(3), 344–359 (2018). https://doi.org/10.1080/03043797.2017.1410522 8. Dilli, S., Westerhuis, G.: How institutions and gender differences in education shape entrepreneurial activity: a cross-national perspective. Small Bus. Econ. 51(2), 371–392 (2018). https://doi.org/10.1007/s11187-018-0004-x 9. Feldon, D.F., Franco, J., Chao, J., Peugh, J., Maahs-Fladung, C.: Self-efficacy change associated with a cognitive load-based intervention in an undergraduate biology course. Learn. Instruct. 56, 64–72 (2018). https://doi.org/10.1016/j.learninstruc.2018.04.007 10. Glassberg Sands, E., Reddick, R., Karsten, E.: Women and skills report: Addressing gender gaps through online learning (2021). https://about.coursera.org/press/wp-content/uploads/2021/09/ Coursera-Women-and-Skills-Report-2021.pdf 11. Guiso, L., Monte, F., Sapienza, P., Zingales, L.: Culture, gender, and math. Science 320(5880), 1164–1165 (2008). https://doi.org/10.1126/science.1154094 12. Gütl, C., Rizzardini, R.H., Chang, V., Morales, M.: Attrition in MOOC: lessons learned from drop-out students. In: Uden, L., Sinclair, J., Tao, Y.-H., Liberona, D. (eds.) Learning Technology for Education in Cloud. MOOC and Big Data, pp. 37–48. Springer International Publishing, Cham (2014). https://doi.org/10.1007/978-3-319-10671-7_4 13. Hadavand, A., Gooding, I., Leek, J.: Can MOOC programs improve student employment prospects? (2018). https://ssrn.com/abstract=3260695 14. Handoko, E., Gronseth, S.L., McNeil, S.G., Bonk, C.J., Robin, B.R.: Goal setting and MOOC completion. Int. Rev. Res. Open Distrib. Learn. 20(3), 39–58 (2019). https://doi.org/10.19173/ irrodl.v20i4.4270 15. Hickey, A., Bakthavachalam, V., Urban, A., Kolodny, T.: A growth mindset can reduce the gender gap in STEM. Coursera Blog (2018). https://blog.coursera.org/ 16. Hickey, A., Urban, A.: The machine learning advantage (2019) 17. Huang, X., Mayer, R.E.: Adding self-efficacy features to an online statistics lesson. J. Educ. Comput. Res. 57(4), 1003–1037 (2019). https://doi.org/10.1177/0735633118771085 18. Kizilcec, R.F., Cohen, G.L.: Eight-minute self-regulation intervention raises educational attainment at scale in individualist but not collectivist cultures. Proc. Natl. Acad. Sci. U.S.A. 114(17), 4348–4353 (2017). https://doi.org/10.1073/pnas.1611898114 19. Kizilcec, R.F., Davis, G.M., Cohen, G.L.: Towards equal opportunities in MOOCs. In: Proceedings of the Fourth ACM Conference on Learning @ Scale - L@S 2017, pp. 121–130 (2017). https://doi.org/10.1145/3051457.3051460 20. Kizilcec, R.F., et al.: Scaling up behavioral science interventions in online education. Proc. Natl. Acad. Sci. 117(26), 14900–14905 (2020). https://doi.org/10.1073/pnas.1921417117 21. Kizilcec, R.F., Saltarelli, A.J., Reich, J., Cohen, G.L.: Closing global achievement gaps in MOOCs. Science 355(6322), 16–18 (2017). https://doi.org/10.1126/science.aag2063 22. Lambert, S.R.: Do MOOCs contribute to student equity and social inclusion? A systematic review 2014–18. Comput. Educ. 145, 1–17 (2020). https://doi.org/10.1016/j.compedu.2019. 103693 23. León, J., Núñez, J.L., Liew, J.: Self-determination and STEM education: effects of autonomy, motivation, and self-regulated learning on high school math achievement. Learn. Individ. Differ. 43, 156–163 (2015). https://doi.org/10.1016/j.lindif.2015.08.017 24. Lochmiller, C.R., Lester, J.N.: An introduction to educational research. In: Connecting Methods to Practice. SAGE (2017) 25. Lung-Guang, N.: Decision-making determinants of students participating in MOOCs: merging the theory of planned behavior and self-regulated learning model. Comput. Educ. 134, 50–62 (2019). https://doi.org/10.1016/j.compedu.2019.02.004

604

A. D. Urban

26. Macphee, D., Farro, S., Canetto, S.S.: Academic self-efficacy and performance of underrepresented STEM majors: gender, ethnic, and social class patterns. Anal. Soc. Issues Public Policy 13(1), 347–369 (2013). https://doi.org/10.1111/asap.12033 27. Miyake, A., Kost-Smith, L.E., Finkelstein, N.D., Pollock, S.J., Cohen, G.L., Ito, T.A.: Reducing the gender achievement gap in college science: a classroom study of values affirmation. Science 330(6008), 1234–1237 (2010). https://doi.org/10.1126/science.1195996 28. Moulton, S.T.: Applying psychological science to higher education: Key findings and open questions (2014). http://hilt.harvard.edu/wp-content/uploads/2018/09/moulton_2014_ applying_psychological_science_to_higher_education_april16.pdf 29. Murphy, S., MacDonald, A., Wang, C.A., Danaia, L.: Towards an understanding of STEM engagement: a review of the literature on motivation and academic emotions. Can. J. Sci. Math. Technol. Educ. 19(3), 304–320 (2019). https://doi.org/10.1007/s42330-019-00054-w 30. OECD: The ABC of gender equality in education: aptitude, behaviour, confidence (2015). https://doi.org/10.1787/9789264229945-en 31. Perez, C.C.: Invisible women: Data bias in a world designed for men. Abrams (2019) 32. Peters, E., et al.: Improving numeracy through values affirmation enhances decision and STEM outcomes. PLoS ONE 12(7), 1–19 (2017). https://doi.org/10.1371/journal.pone.0180674 33. Rabin, E., Henderikx, M., Yoram, M.K., Kalz, M.: What are the barriers to learners’ satisfaction in MOOCs and what predicts them? The role of age, intention, self-regulation, self-efficacy and motivation. Australas. J. Educ. Technol. 36(3), 119–131 (2020). https://doi.org/10.14742/ ajet.5919 34. Rossi, P. H., Lipsey, M. W., Henry, G.T.: Evaluation: A Systematic Approach (8th Ed.). SAGE Publications Inc. (2019) 35. Sambe, G., Bouchet, F., Labat, J.-M.: Towards a conceptual framework to scaffold selfregulation in a MOOC Sixième Colloque National Sur La Recherche En Informatique et SES Applications, pp. 245–256 (2017). https://doi.org/10.1007/978-3-319-72965-7_23 36. Sax, L.J., et al.: Anatomy of an enduring gender gap: the evolution of women’s participation in computer science. J. Higher Educ. 88(2), 258–293 (2017). https://doi.org/10.1080/00221546. 2016.1257306 37. Shadish, W., Cook, T., Campbell, D.: Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin (2002) 38. Simon, R.A., Aulls, M.W., Dedic, H., Hall, N.C.: Exploring student persistence in STEM programs: a motivational model. Can. J. Educ. 38(1), 1–27 (2015) 39. Stolk, J.D., Jacobs, J., Girard, C., Pudvan, L.: Learners’ needs satisfaction, classroom climate, and situational motivations: evaluating self-determination theory in an engineering context. In: Proceedings of Frontiers in Education Conference, FIE, pp. 1–5 (2018). https://doi.org/10. 1109/FIE.2018.8658880 40. Stolk, J.D., Zastavker, Y.V., Gross, M.D.: Gender, motivation, and pedagogy in the STEM classroom: a quantitative characterization. In: ASEE Annual Conference and Exposition, Conference Proceedings (2018). https://www.asee.org/public/conferences/106/papers/22784/ view 41. Thompson, B.: “Statistical”, “practical”, and “clinical”: how many kinds of significance do counselors need to consider? J. Counsel. Develop. 80, 64–71 (2002). https://doi.org/10.1002/ j.1556-6678.2002.tb00167.x 42. Urban, A.: Nail it, then scale it: Doubling down on data to achieve platform success. Coursera Blog (2019). https://blog.coursera.org/doubling-down-on-data-to-achieve-platform-success/ 43. Urban, A.D.: Addressing the gender gap in STEM MOOCs: How brief, in-course messages can increase females’ motivation and online learning success (2022). http://jhir.library.jhu.edu/ handle/1774.2/67442 44. Vennix, J., den Brok, P., Taconis, R.: Do outreach activities in secondary STEM education motivate students and improve their attitudes towards STEM? Int. J. Sci. Educ. 40(11), 1263– 1283 (2018). https://doi.org/10.1080/09500693.2018.1473659 45. Walton, G.M., Logel, C., Peach, J.M., Spencer, S.J., Zanna, M.P.: Two brief interventions to mitigate a “chilly climate” transform women’s experience, relationships, and achievement in engineering. J. Educ. Psychol. 107(2), 468–485 (2015). https://doi.org/10.1037/a0037461

Closing the Gender Gap in STEM MOOCs Through Brief, Novel …

605

46. Yeomans, M., Reich, J.: Planning prompts increase and forecast course completion in massive open online courses. In: ACM International Conference Proceeding Series, March 2017, pp. 464–473 (2017). https://doi.org/10.1145/3027385.3027416 47. Deci, E.L., Ryan, R.M.: The “what” and “why” of goal pursuits: human needs and the selfdetermination of behavior. Psychol. Inquiry 11(4), 227–268 (2000). https://doi.org/10.1360/ 982004-431

Peer Assessment in the University Context for the Development of Transversal and Digital Skills Mara Valente

Abstract Goal 4 of Agenda 2030 states the need to ensure education based on “quality” and “equity,” with a view to global economic, social, and environmental development. To improve the quality of teaching and learning in the university context, there is an increasing shift towards didactic approaches that prioritize the processes implemented instead of the content ( Fabbri, L., & Romano, A.: Metodi per l’apprendimento trasformativo. Casi, modelli, teorie. Roma: Carocci Editore (2019).). Placing students at the center of the teaching and learning process, aiming for strategies and methodologies that promote their engagement and active participation using technology, and creating new learning environments are current challenges. One of the learning methodologies that successfully combines new ways of student participation with the possibility of soliciting transversal skills is peer assessment, which Topping (Topping in Rev. Educ. Res. 68:249–276, 1998) defined as “a system in which individuals consider the quantity, level, value, quality, or success of the learning products or outcomes of peers of the same level.” In fact, this strategy is essential to identify and promote the activated mechanisms that concern the stimulation of transversal skills ( Arter, J. A., & Bond, L.: Why is assessment changing. In R. E. Blum, & J. A. Arter (Eds.), A handbook for student performance assessment in an era of restructuring, (I3: 1–4), Alexandria, VA: Association for Supervision and Curriculum Development (1996).). Starting from these brief theoretical assumptions, this contribution aims to illustrate the educational experiences carried out within the workshop of Educational Measurement at the Faculty of Primary Education — University of Modena and Reggio Emilia to test peer assessment in a university context using a digital platform with the direct engagement of students. Educational activities, learning and assessment tools, and the online platform to review and share feedback will be described. The collected data and the related results will also be discussed to underline the impact of the experience in terms of soliciting transversal skills, self-assessment, and self-reflection. Keywords Peer assessment · Transversal skills · Digital skills · Feedback M. Valente (B) University of Roma Tre, Rome, Italy e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_49

607

608

M. Valente

1 Introduction Goal 4 of the 2030 Agenda aims to ensure equitable and inclusive quality education and focuses on the connection between basic education and vocational training. It also emphasises equity and quality in education in a lifelong learning perspective [4]. Today, it is essential to recognize that successful inclusion in society also depends on digital literacy. According to Recommendation (EC) 2006/962 of the European Parliament, as well as subsequent updates, the acquisition of digital skills by students is a key objective of compulsory education to ensure better integration into today’s society [5]. Having digital skills and training allows individuals to relate to the current global society and avoid exclusion. It is noteworthy that digital literacy is a transversal goal that concerns not only SDG 4 but also SDG 1 (No Poverty), SDG 5 (Gender Equality), and SDG 8 (Decent Work and Economic Growth) with reference to the 2030 Agenda [5, 6]. The advent of the SARS pandemic COVID-19 has accelerated the move towards digital technologies, prompting educational institutions and beyond to rethink learning environments by introducing new technologies on a massive scale [7]. These technologies have enabled educational sustainability, even at a distance, during this prolonged period of global health crisis. Moreover, the new teaching approaches place increasing emphasis on engaging the student at the center of the teaching and learning process, aiming for techniques and methodologies that encourage their involvement and active participation, including the use of technology, and establishing new learning environments. One of the most successful methods in this regard is peer assessment. This type of evaluation is intended to assist students in organizing their learning, determining their areas of strength and weakness, locating opportunities for corrective interventions, and building metacognitive and other transferable abilities on a personal and professional level [2, 8]. Giving students the chance to actively participate in their assessment shifts the power dynamic between teacher and student and promotes appropriate control over their learning since, according to Vickerman [9]; any form of peer interaction engages students in improving their learning from various perspectives, including academic, cognitive, emotional, and social perspectives [9]. In addition, Bloxham and Boyd’s 2007 [10] study, which examined peer evaluation in a university setting, discovered several advantages for students who participate in the assessment process. Peer evaluation aids students in realizing the academic standards of the course, comprehending the assessment criteria in-depth, understanding how they relate to their performance, realizing alternative methods for academic tasks, developing the capacity to make judgments and defend their points of view, encouraging the ability to provide constructive criticism to peers, and advancing students’ ability to study independently by supporting this skill [10]. Falchikov [11] emphasises the importance of adopting peer assessment strategies for their ability to stimulate transversal skills. In this regard, Bloxham and West [12] highlight how peer assessment helped students learn, develop critical thinking skills, and improve their understanding of assessment standards. Kearney [13] also supports this idea, stating that educators should design assessments that

Peer Assessment in the University Context for the Development …

609

engage, motivate, and encourage students to develop essential skills, such as critical thinking and independent learning, while stimulating innovation and creativity. Although international educational research encourages and promotes the adoption of peer assessment strategies, Italy still relies on traditional forms of assessment, especially in the university environment where the assessment of individual learning appears to be the end goal. University students often focus solely on obtaining their degree, disregarding the learning process, according to Grion et al. [14]. This seems to be supported by the assessment practices that students become accustomed to, as noted by Pastore [15], Ricchiardi [16], and Grion et al. [14]. Brown [17] argues that restoring the assessment practices currently used in academia could foster the improvement of university education. According to Van der Pol, Van den Berg, and Admiraal [18], one practical advantage of using peer feedback is that it becomes available earlier in the learning process, and there are more options available than those that a teacher could provide alone. Topping [2] highlights the educational advantages of hearing other students’ appreciation of different elements of each student’s understanding as well as being exposed to other approaches and solutions based on the literature. This encourages the capacity to critique different kinds of texts, provide, and receive peer feedback. Peer feedback can also be used as an intermediary stage in self-assessment. According to Nicol [19, 20], it is precisely through peer review that students, acting as evaluators, play an active role. Through the process of evaluating and reviewing their peers’ work, students reflect on their own work, enabling them to deepen their understanding of aspects of the subject or topic that they may have previously treated superficially. This process also allows them to examine different ways of expressing the same concept and to stimulate their self-assessment skills. Additionally, according to Amhag [21], students can use their reflections as input for self-assessment and to support their next learning task after completing an assignment. Assessing learning is not solely to verify memorized knowledge but also to identify and promote activated mechanisms related to critical thinking, problem-solving, metacognition, efficient testing, teamwork, reasoning, and lifelong learning competencies [3]. Only through this approach can assessment be intended as assigning and/or identifying the value of learning within a framework of meaning that contributes to the actual improvement, growth, and integral development of the person [22].

2 The Workshop to Experiment with Peer Evaluation Peer assessment contributes to meaningful learning and the development of soft skills [23–25]. Various studies have emphasized the importance of peer assessment in promoting cross-competencies, such as increasing engagement [12], encouraging metacognition [9], prompting critical thinking [23], enhancing self-regulation [19, 26], and motivation to learn [27]. Considering the theoretical premises mentioned above, educational strategies aimed at promoting peer assessment were designed and implemented as part of an Educational Measurement Workshop in May 2022.

610

M. Valente

This workshop was planned for the fourth year of the Primary Education degree course at the University of Modena and Reggio Emilia to encourage and apply the practice of peer assessment and feedback in the university context. The decision to design a workshop with peer evaluation goals and feedback comparison was made to achieve several objectives: to promote and implement assessment practices not yet widely used in an academic setting, particularly for future preschool and primary school teachers who will be involved in assessment processes daily, and to test the impact of these methodologies in terms of self-reflection, self-assessment, and the development of cross-competencies among students participating in the workshop, including the use of digital technologies.

2.1 The Workshop: Structure and Organization Forty-six students participated in the activities (44 females and 2 males). The 12h workshop was carried out fully in person. The activities included three different meetings: the table below summarises the structure and content of the activities (Table 1). The first meeting of the workshop focused on competence assessment. To provide students with hands-on experience in constructing assessment tools for this purpose, they were asked to design an authentic task using the GASPS matrix [28]. In order to encourage self-assessment of their work, a checklist was provided for assessing the new format design used in completing the authentic task [29]. The second meeting was specifically dedicated to the theoretical and legislative introduction of the new learning assessment introduced in primary schools [30]. The new approach moved away from assigning numerical values to using descriptive judgment, which emphasizes the formative value of assessment. This process is increasingly seen as necessary to attribute value to the progressive construction of knowledge achieved by pupils, encourage the deployment of each individual’s potential based on their actual levels of learning, and support and enhance motivation for continuous improvement as a guarantee of educational and scholastic success [30]. For this reason, the students were engaged in the construction of an assessment rubric related to the authentic task carried out during the previous meeting. Table 1 Workshop phases Phases

Activities

Duration

Phase I

Realisation of an authentic task*

4h

Phase II

Design of the assessment rubric for the authentic task*

4h

Phase III

Peer review of papers and feedback*

4h

Self-assessment questionnaire *

Students worked in pairs on the proposed activities; the pairs formed during the first phase were the same throughout the entire

Peer Assessment in the University Context for the Development …

611

Fig. 1 The peer review, feedback, and self-assessment process

The third and final meeting (Fig. 1) was dedicated to the peer assessment process and the exchange of feedback after the peer review phase. Students were invited to upload their assignments onto the Peergrade platform. The platform automatically and anonymously distributed two assignments to each pair of students. They read and reviewed their colleagues’ assignments by filling in a specific assessment rubric consisting of multiple-choice and open-ended questions. Once the revision process was completed, the platform sent the collected reviews to the students, allowing them to read the responses and, if necessary, send feedback to the reviewers on the assessments they had received. At the end of the peer review and feedback process, the students completed a questionnaire to express their perceptions of the peer review activity and the transversal competencies it solicited. All the proposed activities were carried out in pairs; the pairs were formed spontaneously during the first meeting and continued to work together during the second and third meetings. Pair work aimed to solicit collaboration, communication, and critical-thinking skills. From several studies in the field, it is evident how collaboration is increasingly understood as an important goal of education in general [31]. Griffin et al. [32] consider it as the “ability to work together towards a common goal,” and Kuhn [33] defines it as a process that leads to the effective realization of desired individual and group outcomes. Vygotskij [35] himself founded many of his well-known theories on collaborative learning by emphasizing the fundamental significance of the social role of education. Collaborative activities facilitate learning in the zone of proximal development, enabling the internalization of theories and concepts, in the process in which the individual also assimilates and learns through the support of others.

2.2 The Platform for Peer Assessment The platform used by the students for peer assessment activities is Peergrade. Peergrade1 is a multifunctional online platform developed to enhance peer review. Its structure is meant to promote conversation and communication between evaluators and evaluated, in order to effectively direct learning. The platform allows teachers 1

https://www.peergrade.io/.

612

M. Valente

to freely use different types of tasks based on the goals they set for their students, to choose the digital format and characteristics of the task they need to share, to create a customized evaluation rubric for peer review, or to use those already made available online by other teachers. Teachers can also choose whether to allow individual or group reviews and customize the number of tasks to assign to each evaluator. Additionally, teachers can constantly monitor their students’ activity through a broad live overview feature. During the assessment step, in addition to filling out the rubric, students can use the “Flags” option to recall remarks from colleagues and seek further information. Peergrade also invites evaluators to convey their thoughts on the feedback received. They can indicate the usefulness of the input received using five descriptors and can provide remarks on final grades using a free-text field. The feedback release phase is the one in which students focus the most on the evaluations received and provide motivated and correct remarks, sparking processes of reflection, self-reflection, and critical thinking. The platform also includes a peer assessment method that allows users to begin by assessing the work of others and then move on to self-evaluation and feedback. This technique is consistent with findings from field studies, which show that students learn more by offering feedback on peers’ work than by receiving input from peers [34].

3 Data Collection and Data Analysis 3.1 Peer Review Tools: Rubric and Feedbacks In order to provide guidelines to facilitate the most valid and reliable review of the tasks delivered by colleagues, an evaluation rubric consisting of closed and openended questions has been designed and made available to students on Peergrade. The four closed-ended questions were aimed at evaluating the quality of the evaluated tasks in terms of ease of understanding, adherence to the proposed contents and objectives, clarity and structure of the contents, and completeness of contents, using a Likert scale (scoring from 1 [minimum quality] to 5[maximum quality]). A closed-stimulus question was included to indicate, always referring to a scale of measurement, whether the indications of the task were respected or not. The students were then asked to motivate with open-ended questions the strengths of the revised papers, provide suggestions on any aspects to be improved, highlight a content/aspect they learned thanks to the review of their colleagues’ works, and provide concluding space for further observations and comments for colleagues. The feedback section is automatically predisposed by the platform and is formed by two questions: one with a closed answer that asked to define, according to a scale of measurement composed of five options, the usefulness of the feedback received by the evaluators, and a further space with free text to provide the possibility for the evaluated to argue in a more exhaustive way the usefulness of the feedback.

Peer Assessment in the University Context for the Development …

613

Fig. 2 Evaluate the quality of your colleague’s work in terms of (scoring from 1 [minimum quality] to 5 [highest quality]). Clarity of structure and content

3.2 Peer Task Evaluation: Some Results The findings on the assessment of the quality of peer work, using six different indicators as illustrated above, are generally very encouraging. The students evaluated a total of 44 works in pairs (2 works per pair). Based on a Likert scale from 1 to 5, the students expressed their first judgment on the ease of comprehension of the analyzed tasks, which had an average score of 4.4 points (std. dev. = 0.661). The papers were judged to be almost fully aligned with the proposed contents and objectives (M: 4.5/ 5; std. dev. = 0.731). The structure and content used in the elaboration of the tasks were assessed as almost completely exhaustive, receiving an average score of 4.4 out of a maximum of 5 points (std. dev. = 0.784) (Fig. 2). The feedback regarding the comprehensiveness of content about the elaboration of the tasks had an average score of 4.3/5 (std. dev. = 0.707). Using a Likert scale from 1 to 4 (from 1 [never] to 4 [fully]), the students gave an average score of 3.7 out of 4 (std. dev. = 0.423) to the indicator related to the coherence of the task in comparison to the given indications. The last indicator of the rubric was used to measure the usefulness of the feedback received in response to the peer evaluations. On a Likert scale of 1 to 5 (from 1 [not at all useful] to 5 [extremely useful]), the students gave an average score of 4.4 (std. dev. = 0.830), confirming the validity of the practice adopted and the feedback received.

3.3 Students’ Perception of the Usefulness of Peer Assessment A specific open-ended question was proposed to explore students’ perceptions of the usefulness of the conducted peer review activity. The question asked was “Do you think peer review is useful? Argue your reasons, whether they are positive or negative.”

614 Table 2 Recurring words used in the evaluation of the usefulness of peer-assessment

M. Valente

Word

Abs. Frequency

Comparison

8

Self-reflection

7

Self-assessment

6

Constructive

6

Reflection

5

Twenty-two pairs of respondents provided their answers. The table below (Table 2) displays a list of terms frequently used by the students to express their opinions on the usefulness of peer assessment, along with the number of times each word was used in the 22 responses. No negative judgments were found in the answers given by the students. Summarizing the overall opinions, students agree that they find peer assessment useful for positive and constructive discussions among peers facing the same educational journey. The peer review activity encourages processes of self-reflection and selfassessment on the work done, granting an opportunity to revise and rework their ideas considering the feedback received. According to most, peer assessment involves students applying success criteria related to a learning objective, reflecting on their efforts, identifying improvements, and adjusting the “quality” of their work. It represents an opportunity to learn about colleagues’ ideas and ways of working, promoting personal and professional development and growth. Peer assessment also shows a way to experiment with making judgments, learning from others, and getting used to accepting constructive criticism. It allows for comments and observations to broaden one’s cultural background, and it is considered an excellent way to compare and identify critical issues and strengths together (Fig. 3).

3.4 The Self-evaluation Questionnaire Results After reviewing the papers and providing feedback using the virtual platform, the students were asked to reflect on their experience by filling out a questionnaire. A total of 45 students completed the questionnaire (43 = F; 2 = M). These students were enrolled in the fourth year of the Faculty of Primary Education Science at the University of Modena and Reggio Emilia, and their average age was 27 years old. The first section of the questionnaire focused on assessing the knowledge gained and learning methods related to peer assessment. Based on a Likert scale from 1 to 5, the results showed that students’ expectations were almost completely met regarding the knowledge gained (M = 4.33/5; std. dev. = 0.564), the learning was perceived as progressive (M = 4.44/5; std. dev. = 0.693), and the performance of the activity was easily understood. In fact, almost all students expressed an average score of 4 out of 5 based on the Likert scale (M = 4.31/5 std. dev. = 0.668). The students were asked if some information was taken for granted without appropriate explanations using a

Peer Assessment in the University Context for the Development …

615

Fig. 3 Words-cloud on the usefulness of peer assessment according to students

scale from 1 to 5 (1 = nothing; 5 = a lot of things). The average score was 1.82 out of 5, confirming their good understanding of the given information. Similarly, the students were asked if they would prefer more information, and the average score was M = 2.27 out of 5, indicating that the explanations provided for the activity were considered exhaustive by the students, which supported their performance. The positive results that emerged from the analysis of the first section of the questionnaire are further confirmed in the last question, where the students stated their interest in investigating further topics after the workshop (M = 4.13 out of 5; std. dev. = 0.968). This result is encouraging for further progress and highlights the benefits of the study that may be applied by the students in their classes. Upon analyzing the second section of the questionnaire regarding the use of the Peergrade platform and its influence on collaborative educational methods associated with technology, it is evident, based on the Likert scale (from 1 [no incentive] to 5 [highest incentive]) that the average score for the growing motivation and participation of students is 4.09 out of 5 (Fig. 4). This significant result confirms what specialist scientific research has underlined: peer assessment, when combined with critical use of technological tools, increases motivation in learning and tends to create high levels of participation. The data analysis on the peer review activity conducted through the Peergrade platform shows that it encouraged collaborative learning, receiving an average score of 4.47 out of 5 points. The online activity also led to a perceived improvement in the quality of teaching, with a score of 4.11 out of 5. The students’ familiarity with technological tools was confirmed to be positive with an average score of 3.96 out of 5, as well as their willingness to use new e-learning tools (M = 4.11/5; std. dev.

616

M. Valente

Fig. 4 Growth in motivation due to students’ use of the Peergrade platform for peer assessment activities

= 0.885). Moreover, the activity conducted on Peergrade facilitated the sharing of materials among students, with an average score of 4.33 out of 5. In the last section of the questionnaire, which aimed to measure the level of students’ perception of the soft skills stimulated by the peer review activity (Fig. 5), it is noticeable that the most triggered competencies were Collaboration (M = 4.64/ 5; std. dev. = 0.609) and Critical Thinking (M = 4.24/5; std. dev. = 0.857). The task was performed in pairs and solicited collaboration among students, triggering comparisons, different points of view, teamwork to reach shared goals, respect, active participation, and support among colleagues. The peer assessment activity also encouraged Critical Thinking skills by involving students in the processes of making inferences, linking several pieces of information, self-correction, conceptualization, good communication, and objective reasoning and assessments. All the remaining competencies analyzed are prompt enough; however, Working Memory was the only one that obtained an average value lower than the others (M = 3.27/5). By examining the correlations among the benchmarks in the questionnaire, some interesting results were found. There was a correlation between the growing personal Collaboration Research attitude Working Memory Problem Solving Critical Thinking Communication Innovation Creativity 0

1

2

3

Fig. 5 Competencies solicited by the peer assessment activity

4

5

Peer Assessment in the University Context for the Development …

617

motivation related to collaborative education methods using Peergrade and the peer review activity (r = 0.427; p = 0.001). The use of new tools in e-learning activities was also correlated with the solicitation of Creativity (r = 0.461; p = 0.001) and Collaboration (r = 0.413; p = 0.005), which confirms the results from previous research in the field. Another interesting result to focus on is the Collaboration skill, which is correlated to the improvement of the quality of education methods related to the peer review activity on the Peergrade platform (r = 0.600; p < 0.001). Furthermore, the results show that Critical Thinking competence is connected to the Peergrade platform (r = 0.563; p < 0.001). The ability to reason critically, make inferences, and connect information is correlated with the use of a new digital tool for peer assessment, leading to a positive stimulation of both Critical Thinking and digital skills in the students.

4 Conclusions This contribution discusses a pilot activity that aims to describe the use of peer assessment in a university context to foster the development of transversal and digital skills, as well as processes of self-reflection and self-evaluation. The goal is to disseminate good practices that are not yet widely used in Italy, especially in the university context. The Educational Measurement Workshop took place in May 2022 and was part of the degree course in Primary Education. It involved 46 learners who participated in three meetings aimed at creating an authentic task and its evaluation rubric. These were then peer-reviewed and compared through feedback from the evaluated. Overall, the evaluations assigned by the students themselves using the rubric recorded a very good average of judgments about the six indicators considered: quality of work in terms of easiness of understanding, adherence to proposed content and objectives, clarity of structure and content, exhaustiveness of content, compliance with the delivery instructions, and usefulness of the feedback received. At the end of the activity, a questionnaire was issued to analyze participants’ perceptions of the peer assessment activity and the soft skills it elicited as well as to gather opinions regarding the conduct of the peer review on the Peergrade platform and the delivery of feedback. The data indicate a positive reaction from the students towards the proposed activity, which was being experienced by them for the first time, as revealed in classroom discussions. Expectations were broadly met (M = 4.33/5), learning was gradual (M = 4.44/5), and the information received about the activity was considered comprehensive (M = 4.31/5). Students were very interested in what was done in the classroom with the intention of pursuing the strategy further (M = 4.13/5). The data on the solicitations triggered by the peer review activity on the Peergrade platform show that it encouraged forms of collaborative learning, scoring an average of 4.47 out of 5 points. The conduct of the online activity also registered a perceived improvement in the quality of teaching, obtaining 4.11 out of 5 points. Encouraging results also came from the questions regarding perceptions of the skills solicited by the activity: among all of them, Collaboration (M = 4.64/5) and Critical Thinking (M = 4.24/

618

M. Valente

5) scored the highest. The least stimulated, according to the students’ opinion, was Working Memory (M = 3.27/5). Positive feedback was also received in an analysis of opinions given about the efficacy of peer review and the release of feedback from the assessed. These initial and non-generalizable findings provide a first methodological and pedagogical overview of using a digital tool for peer review in a university context. The study could be further improved beyond current practice. For example, students from different years of study could participate in the activities to allow for objective evaluations and to increase the number of participants. In fact, involving a larger number of students over a longer period could be advantageous to analyze the data more precisely on the effectiveness of peer assessment in the university context and determine its relative impact in terms of soliciting transversal and digital skills.

References 1. Fabbri, L., Romano, A.: Metodi per l’apprendimento trasformativo. Casi, modelli, teorie. Carocci Editore, Roma (2019) 2. Topping, K.J.: Peer assessment between students in colleges and universities. Rev. Educ. Res. 68(3), 249–276 (1998) 3. Arter, J.A., Bond, L.: Why is assessment changing. In: Blum, R.E., Arter, J.A. (eds.) A Handbook for Student Performance Assessment in an Era Of Restructuring, vol. I3, pp. 1–4. Association for Supervision and Curriculum Development, Alexandria, VA (1996) 4. ONU: Assemblea generale. Agenda 2030, A/RES/70/, Distr.: Generale 21 ottobre 2015 (2015) 5. Méndez, D., Méndez, M., Anguita, J.M.: Digital teaching competence in teacher training as an element to attain SDG 4 of the 2030 Agenda. Sustainability 14, 11387 (2022). https://doi.org/ 10.3390/su141811387 6. Rangel-Pérez, C., Gato-Bermúdez, M.-J., Musicco-Nombela, D., Ruiz-Alberdi, C.: The massive implementation of ICT in universities and its implications for ensuring SDG 4: challenges and difficulties for professors. Sustainability 13, 12871 (2021) 7. Secundo, G., Mele, G., Del Vecchio, P., Elia, G., Margherita, A., Ndou, V.: Threat or opportunity? A case study of digital-enabled redesign of entrepreneurship education in the COVID-19 emergency. Technol. Forecast. Soc. Change 166, 120565 (2021) 8. Brown, S., Rust, C., Gibbs, G.: Strategies for Diversifying Assessment in Higher Education. Oxford Centre for Staff Development, Oxford (1994) 9. Vickerman, P.: Student perspectives on formative peer assessment: an attempt to deepen learning? Assess. Eval. High. Educ. 34(2), 221–230 (2009) 10. Bloxham, S., Boyd, P.: Developing Effective Assessment in Higher Education: A practical guide. Open University Press, Berkshire (2007) 11. Falchikov, N.: Peer feedback marking: Developing peer assessment. Program. Learn. 32(2), 175–187 (1995). https://doi.org/10.1080/1355800950320212 12. Bloxham, S., West, A.: Understanding the rules of the game: Marking peer assessment as a medium for developing students’ conceptions of assessment. Assess. Eval. High. Educ. 29(6), 721–733 (2004). https://doi.org/10.1080/0260293042000227254 13. Kearney, S.: Improving engagement: the use of “Authentic self- and peer-assessment for learning” to enhance the student learning experience. Assess. Eval. High. Educ. 38(7), 875–891 (2013) 14. Grion V., Serbati A., Tino C., Nicol D.: Ripensare la teoria della valutazione e dell’apprendimento all’università: un modello per implementare pratiche di peer review. Giornale Italiano della Ricerca Educativa X(19), 209-229 (2017)

Peer Assessment in the University Context for the Development …

619

15. Pastore S.: Silent assessment? Cosa pensano della valutazione gli studenti universi- tari. Giornale Italiano della Ricerca Educativa, V, Numero speciale, 62–73 (2012) 16. Ricchiardi P.: Sviluppo di strategie di apprendimento in contesti didattici differen- ziati: un’indagine. In: Coggi, C. (ed.) Per migliorare la didattica universitaria, pp. 305–356. Pensa MultiMedia, Lecce (2005b) 17. Brown, G., Harris, L.: Student self-assessment. In: McMillan, J.H. (ed.) The SAGE Handbook of Research on Classroom Assessment, pp.367–393. Sage (2014). https://doi.org/10.4135/978 1452218649 18. Van der Pol, J., Van den Berg, B.A.M., Admiraal, W.F., Simons, P.R.J.: The nature, reception, and use of online peer feedback in higher education. Comput. Educ. 51, 1804–1817 (2008) 19. Nicol, D.J., Macfarlane-Dick, D.: Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Stud. High. Educ. 31(2), 199–218 (2006) 20. Nicol, D.: From monologue to dialogue: improving written feedback processes in mass higher education. Assess. Eval. High. Educ. 35(5), 501–517 (2010) 21. Amhag, L.: Creativity in and between collaborative peer assessment processes in higher distance education. Creat. Educ. 4, 94–104 (2013). https://doi.org/10.4236/ce.2013.47A2011 22. Tessaro, F.: Compiti autentici o prove di realtà? Formazione & Insegnamento, XII 3, 77–88 (2014). ISSN 1973-4778. Print – 2279-7505. https://doi.org/107346/-fei-XII-03-14_07, https:// ojs.pensamultimedia.it/index.php/siref/article/download/1119/1085/3900 23. Lynch, R., McNamara, P.M., Seery, N.: Promoting deep learning in a teacher education programme through self- and peer-assessment and feedback. Eur. J. Teach. Educ. 35(2), 179–197 (2012). https://doi.org/10.1080/02619768.2011.643396 24. Sluijsmans, D.M.A., Brand-Gruwel, S., van Merriënboer, J.J.G.: Peer assessment training in teacher education: effects on performance and perceptions. Assess. Eval. High. Educ. 27(5), 443–454 (2002) 25. Foschi, L.C., Cecchinato, G., Say, F.: Quis iudicabit ipsos iudices? Analisi dello sviluppo di competenze in un percorso di formazione per insegnanti tramite la valutazione tra pari e l’autovalutazione. Ital. J. Educ. Technol. 27(1), 49-64 (2019). https://doi.org/10.17471/24994324/101 26. Panadero, E., Jönsson, A., Botella, J.: Effects of self-assessment on self-regulated learning and self-efficacy: four meta-analyses. Educ. Res. Rev. 22, 74–98 (2017). https://doi.org/10.1016/j. edurev.2017.08.004 27. Topping, K.: Trends in peer learning. Educ. Psychol. 25(6), 631–645 (2005). https://doi.org/ 10.1080/01443410500345172 28. Wiggins, G., McTighe, J.: Understanding by Design. Association for Supervision and Curriculum Development, Alexandria (1998) 29. Comoglio, M.: Insegnare e apprendere con il Portfolio. Milano, Fabbri (2004) 30. MIUR. Ordinanza ministeriale n°172 del 4 dicembre (2020) 31. Poce A.: Il patrimonio culturale per lo sviluppo delle competenze nella scuola primaria. Franco Angeli s.r.l., Milano (2018) 32. Griffin, P., Care, E.: Assessment and Teaching of 21st Century Skills Methods and Approaches. Springer, Dordrecht (2015) 33. Kuhn, D.: Thinking together and alone. Educ. Res. 44(1), 46–53 (2015). https://doi.org/10. 3102/0013189X15569530 34. Nicol, D., Thomson, A., Breslin, C.: Rethinking feedback practices in higher education: a peer review perspective. Assess. Eval. High. Educ. 39(1), 102–122 (2014). https://doi.org/10.1080/ 02602938.2013.795518 35. Vygotskij, L.: Storia dello sviluppo delle funzioni psichiche superiori e altri scritti. GiuntiBarbera, Firenze (ed. 2010)

The Coherence Between Innovative Teaching Methods and Formative Assessment in Higher Education Vilmos Vass

Abstract Under the umbrella of VUCA-world and growing international competition, the human factors are playing more important role in higher education (De Witt, H., Gacel-Ávila, J., Jones, E. & Jooste, N. (eds.): The globalization of internationalisation, Routledge, London. (2017);Maringe and Foskett in Maringe and Foskett (eds), Globalization and Internationalisation in Higher Education, Continuum International Publishing Group, London, 2012;). Innovative teaching methods and formative assessment are significant transformational part of this process (Smith M.K; Vass,V: The relationship between internationalisation, creativity and transformation: A case study of higher education in Hungary TRANSFORMATION IN HIGHER EDUCATION 2: 1 pp. 1–9. (2017)). No doubt, teaching and learning methodology and assessment has strong coherency ( Marzano, R.J.: The Arts and Science of Teaching. ASCD, Alexandria, VA (2007)). This presentation focuses on the relationship between innovative teaching methods and formative assessment. In the first part of the presentation, the philosophical phenomena of this process comes from John Dewey ‘learning by doing’ principle (Dewey in Experience and Education, Macmillan Company, New York, 1938). Thus, innovation teaching methods have strong impact of different types of interactions and broader meaning of learning, especially problem-, projectand inquiry-based learning (Hattie in Visible learning: A synthesis of over 800 meta analyses related to achievement, Routledge, New York, 2008). Formative assessment focuses on following students’ progression and continuous feedback changing feedback culture in the teaching and learning process [Wiliam in Stud. Educ. Eval. 37:3– 14, March 2011; Wiliam, D., Thompson, M.: Integrating assessment with learning: What will it take to make it work? In: Dwyer, C.A. (ed.) The future of assessment: shaping teaching andlearning, pp. 53–82.Lawrence Erlbaum Associates, Mahwah, NJ (2007);]. Obviously, there are strong coherence between innovative teaching methods and formative assessment. In the second part of the presentation, the case

V. Vass (B) Budapest Metropolitan University, Budapest Nagy Lajos Király Útja 1-9, Budapest 11148, Hungary e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_50

621

622

V. Vass

study from Budapest Metropolitan University gives evidences to this required relationship giving best practises on innovation and formative assessment. Finally, at the end of the presentation, opened conclusion has dilemmas and questions. Keywords Skill gap · Innovative teaching methods · Formative assessment · Diagnostic assessment · Feedback culture

1 Context No doubt, innovation is the fundamental engine of rethinking and restructuralization on higher education. Under the umbrella of, on the one hand, VUCA-world (Volatile, Uncertain, Complex, Ambiguous) and growing international competition, the human factors are playing more important role in higher education [9, 12]. On the other hand, knowledge economy and skill gap, the challenge of innovation in higher education is a relevant hot topic, especially from the needs of renewing teaching methods and assessment functions. In the first approach, the meaning of innovation is technological progress under the context of science and technology. In the second approach, which is more relevant of our topic, pedagogical innovation is “characterized by an intentional action that aims to improve university students’ learning in a sustainable manner” [19, p.197]. In this sense, the traditional teacher- and teachingcentered higher education has changed to the learner and learning-centered approach. No doubt, the horizontal strategy of higher education in the twenty-first century is teaching and assessment for learning. This is the contextual frame, where innovative teaching methods and formative assessment work in practice. No doubt, teaching and learning methodology and assessment has strong coherency [13]. The philosophical phenomena of this process comes from John Dewey ‘learning by doing’ principle [8]. Thus, innovation teaching methods have strong impact of different types of interactions and broader meaning of learning, especially problem-, projectand inquiry-based learning [11]. In other words, innovative teaching methods means new ways of teaching, which are go beyond lectures. Thus, growing needs of innovation has economic and pedagogical pillars. Turning back the growing needs of closing skill gap, Olson states: “In 2012, McKinsey & Company forecasted a troubling outlook on the labor market through the year 2020. The report highlighted three talent shortages across the globe: nearly 40 million too few college educated workers in the global labor market; a 45 million shortfall of workers with secondary and vocational education in developing countries; and up to 95 million workers that lack the skills needed for employment in advanced economies” [14, p.1]. As labour markets are dynamic and focusing on raising the output of educational systems, so companies would continue to access the talent they need to sustain growth and create opportunities” [10]. The World Economic Forum and Tata Consultancy Services also analysed the data, opportunities and actions of closing the skill gap, focusing on the needs of skilling, reskilling and upskilling [5, 6, 17]. In this sense,

The Coherence Between Innovative Teaching Methods and Formative …

623

the answer of higher education is significant, namely strengthening competencybased education via mastery of skills and adaptive teaching methods. As the abovementioned White Paper concluded from the educational point: “Current education policies should be reformed to place greater emphasis on mastery of skills over gaining qualifications. This includes an evolution in the curriculum as well as the modes of instruction to meet the dynamic demands of the new economy” [5, p. 16]. The key question is: Can higher education systems change and transform dynamic and adaptive ways as labour market? The needs and enormous challenge is given, but the traditional culture of higher education has huge discrepancy. Obviously, that revision of curriculum is a significant step in order to close the skill gap. Additionally, in the first approach, this is a significant step to increase dynamism and adaptability of higher education. Especially when the strategy focuses on learner- and learning-centered education, the curricula have changed from the former aim-orientation, input-oriented, content-based approach. Output-focused, especially learning outcomes, competency-based curriculum design comes to the front. Worthy of note, that from the needs of dynamism and adaptability of higher education, the most important changing parts of curriculum from the pragmatic point are (i) learning strategies and activities promoting self-directed learning, learning to learn and collaborative learning, (ii) innovative teaching methods, and (iii) diagnostic and formative assessment. As a result curriculum is a process and not product, de facto planning for learning. In other words, curriculum is not a noun, it is a verb: “We have reconceived the curriculum; it is no longer only a noun. It is also a verb: currere” [15]. Planning for learning, the process-oriented curriculum, is a broader meaning of curriculum is related to transformation and complexity of higher education. The vertical transformation of the curriculum, from the first perspective, is based on some philosophical foundations and ideologies of education and the curriculum. Evident, that pragmatism and existentialism can handle the changes and the dynamic, adaptive process with the interaction of the individual and the changing social environment. In other words: “The curriculum reflects on the interaction between the learner and the environment, thus it results in the construction of meaning” [18, p. 8]. Last, but not least, curriculum planning is a consistent part of a continuous circle, which is an innovative and creative process. Just imagine an innovation-based triangle, which has three relevant peaks: curriculum design, learning and teaching development and assessment. Frequently, rethinking and restructuralization of higher education is starting with revision of curriculum planning, but in my opinion the significant and effective starting steps of this process to change the functions of assessment, especially from summative to formative function than turning to the innovative teaching methods. As Black and Wiliam defined, “…formative assessment when the evidence is actually used to adapt the teaching to meet student needs” [1].

624

V. Vass

In fact, formative assessment is for learning under the umbrella of “raising the standards of learning” [1]. The other frequent used short definition is assessment for learning [1–3, 20, 21]. No doubt, formative assessment promoting learning with following students’ progression using rubrics, personalized learning diary and learning portfolio. But turning back to the original definition of formative assessment, we need to see the required coherence between assessment and teaching. “Teachers need to know about their pupils’ progress and difficulties with learning so that they can adapt their own work to meet pupils’ needs - needs that are often unpredictable and that vary from one pupil to another. Teachers can find out what they need to know in a variety of ways, including observation and discussion in the classroom and the reading of pupils’ written work” [2]. Focusing on higher education, from the above-mentioned process, formative assessment and innovative teaching methods have enormous impact to students’ learning, last but not least, closing the skill gap. Innovative teaching methods and formative assessment are significant transformational part of this process [16]. The feasible and required coherency between formative assessment and innovative teaching methods is based on, among others, continuous response and feedback [4]. In other words, higher education institutions have different feedback culture, using responses and feedback for improving and innovation of learning and teaching. Parallel to this feedback circle, using evidences via adaptive teaching “to meet student needs” is starting with diagnostic assessment. “There is no sharp distinction between formative and diagnostic assessment, nor does a universal definition for diagnostic assessment exist. However, it is usually described as a kind of assessment which focuses on problems, explores possible difficulties, assesses if students are prepared for a learning task, and thus may measure prerequisite knowledge as well” [7, p. 3]. In this sense, turning back to the above-mentioned innovation-based triangle, which has three relevant peaks: curriculum design, learning and teaching development and assessment. Sharing the aims and expectation in order to “adapt the teaching to meet student needs,” mapping students’ prior knowledge, using selfevaluation diagnosing students’ strengths and weaknesses and making pre-test (then at the end of the semester post-test) measuring personal components. Later following the progression is the significant starting point in the beginning of the semester. In this perspective, we have many diagnostic data for improving learning via innovative teaching methods such as brainstorming, making mind maps, using place mat, organizing projects, working in cooperative teams, debating and questioning.

2 Practice In Budapest Metropolitan University at BA and MA levels, there are several innovation-based courses, which have evidences on the above-mentioned required coherency between innovative teaching methods and formative assessment. For

The Coherence Between Innovative Teaching Methods and Formative …

625

instance the project-based course Social Studies, which is focusing on problem-, project- and inquiry-based learning, starts with provocative questions to the students: Why are you here? The students need to formulate their aims and expectations of the course. No doubt, this process is unusual to the students, but from the point of evaluation, it gives many evidences on students’ attitudes and way of thinking. From the formative assessment view, these aims and expectations are going back at the end of the semester in order to collect some students’ feedback. Under the umbrella of diagnostic assessment, the next activity is an introductory game, where students need to know each other. Firstly, they need to introduce themselves using first name’s letters (not each) to tell each other some personal characteristics. For instance, my name is Vili, I am very intelligent and innovative. The next part of the game is introduction your classmates, how can you concentrate other student’s characteristics. It is important; do not use each letter on your first man for introduction. Flexibility can promote making creative ideas and respect each other’s as well. From pedagogical point of view, it is valuable to face to different ways of thinking and to know your students’ characteristics. At the end of the semester, it is important, because of following the progression, turning back to these personal characteristics via selfevaluation checking the changes or potential developmental areas. The next part of the course is mapping students’ prior knowledge using brainstorming, place mat and mind map on project or project-based mindset. Brainstorming is based on collecting first ideas about the project, then using these ideas organizing groups making place mat. The key question is: Why is project-based mindset important in the twenty-first century? Students (4 of each group) need to answer, firstly, individually this question; then they discuss together about the common point. Finally, students need to make a mind map on project answering three questions: (i) What kind of knowledge do you use in the project work? (ii) What kind of skills are relevant of your project work? (iii) What kind of attitudes do you have in the project work? We are making a collaborating mind map making a competency structure of the project-based work using later these three pillars for following the progression at the of the semester. On the base of the results of the introductory game, brainstorming, place mat and mind map, students need to fill in a short self-evaluation for in order to indicate their strengths and weaknesses of the project work. It is a useful evidence following their progressions at the end of the semester as well. In order to strengthen different feedback culture, we negotiate with the students using feedback cups during the semester. The red cup is for questioning, discussing and evaluating about the relevant, pragmatic, opened questions. Basically, questions are welcome and appreciate during the course. The yellow cup is for understanding, students need to indicate immediately if they do not understand anything. This process requires, on the one hand, continuous feedback. On the other hand some differentiated teaching and methods and students’ tasks. The green cup represents “flow” feeling, when we are making, presenting, evaluating and celebrating the projects under the creative atmosphere. Following the progression has two types: first, using diagnostic and formative assessment, students’ progression comes to the front. Second, writing Progress Report is focusing on project progression answering some questions at the middle part of the semester: what we did; what we are going to do; any risk

626

V. Vass

factors; how you see success criteria. Writing Progress Report requires collaboration and this process is the middle-term of the project work. Turning to the end of the semester, the project teams present their work on the shared evaluation criteria using collaborative assessment and feedback culture. The teams need to make a reflective report summarizing the main outcomes of the projects, advantages and difficulties of the collaborative work. Last but not least, individual feedback with summarization of the evidences on progression and feedback window is closing the semester. Feedback window has four parts: What did I learn? What about my feeling? Any questions? What about usefulness of the course? To sum, the list of best practices in terms of assessment in university settings where innovative methods are used are introductory game, questioning, mapping prior knowledge via brainstorming, place mat and mindmapping, self-evaluation, reflective report and feedback window.

3 Conclusion No doubt, innovative teaching methods and formative assessment has strong coherency. Basically, assessment has strong, significant feedback to the learning and teaching process, thus changing the evaluation and feedback culture have enormous impact of the quality on learning and teaching. In this sense, diagnostic and formative assessment are the pragmatic and useful function change planning curricula and learning and teaching process. In spite of the pedagogical added value, the required coherency between innovative teaching method and formative assessment, at the systematic point, can closing the skill gap via turning to the competency- and portfolio-based education. Pragmatism and “learning by doing” philosophy are the fundamental educational strategy in order to create collaborative feedback culture and promoting learning using diagnostic and formative assessment effectively.

References 1. Black, P., Wiliam, D.: Inside the black box raising standards through classroom assessment. Phi Delta Kappan 80, 139–148 (1998) 2. Black, P., Wiliam, D.: Assessment and classroom learning. Assess. Educ. Princ. Policy Pract. 5, 7–74 (1998) 3. Black, P., Harrison, C., Lee, C., Marshall, B., William, D.: Assessment for Learning. Putting it Into Practice. Open University Press, Berkshire (2003) 4. Black, P., Harrison, C., Lee, C., Marshall, B., Wiliam, D.: Working inside the black box: assessment for learning in the classroom. Phi Delta Kappan 86, 8–21 (2004) 5. Closing the Skills Gap: Key Insights and Success Metrics. White Paper. World Economic Forum, November 2020 6. http://www3.weforum.org/docs/WEF_GSC_NES_White_Paper_2020.pdf

The Coherence Between Innovative Teaching Methods and Formative …

627

7. Csapó, B., Molnár, G.: Online diagnostic assessment in support of personalized teaching and learning: the eDia system. Front Psychol 10, 1522 (2019). https://doi.org/10.3389/fpsyg.2019. 01522/full 8. Dewey, J.: Experience and Education. Macmillan Company, New York (1938) 9. De Witt, H., Gacel-Ávila, J., Jones, E., Jooste, N.: The Globalization of Internationalisation. Routledge, London (2017) 10. Dobbs, R., Lund, S., Madgavkar, A.: Talent tensions ahead: a CEO briefing. McKinsey Q. (2012). https://www.mckinsey.com/featured-insights/employment-and-growth/talent-ten sions-ahead-a-ceo-briefing 11. Hattie, J.: Visible Learning: A Synthesis of Over 800 Meta Analyses Related to Achievement. Routledge, New York (2008) 12. Maringe, F., Foskett, N.: Introduction: Globalization and Universities. In: Maringe, F., Foskett, N. (eds.) Globalization and Internationalisation in Higher Education, pp. 1–15. Continuum International Publishing Group, London (2012) 13. Marzano, R.J.: The Arts and Science of Teaching. ASCD, Alexandria (2007) 14. Olson, M.P.: A multilateral approach to bridging the global skills gap. Cornell HR Review. Retrieved [insert date] from Cornell University, ILR School site, vol. 74, 8 May 2015. http:// digitalcommons.ilr.cornell.edu/chrr/ 15. Pinar, W. F.: What is Curriculum Theory? Routledge, Taylor& Francis, New York and London (2012) 16. Smith, M.K., Vass, V.: The relationship between internationalisation, creativity and transformation: a case study of higher education in hungary. Transform. High. Educ. 2(1), 1–9 (2017) 17. The Future of Jobs Report 2020. World Economic Forum, October 2020. http://www3.wef orum.org/docs/WEF_Future_of_Jobs_2020.pdf 18. Vass, V: The transformation and complexity of the curriculum. The curriculum as a product and/or a process? Eruditio - Educatio 13(3), 5–12 (2018). http://e-eruditio.ujs.sk/archive/201811-26_EE_2018_3_NYOMDAKESZ_Belivek.pdf 19. Walder, A.M.: The concept of pedagogical innovation in higher education. Educ. J. 3(3), 195– 202 (2014) 20. Wiliam, D.: What is assessment is for learning? Stud. Educ. Eval. 37(1), 3–14 (2011) 21. Wiliam, D., Thompson, M.: Integrating assessment with learning: What will it take to make it work? In: Dwyer, C.A. (ed.) The future of assessment: shaping teaching andlearning, pp. 53– 82.Lawrence Erlbaum Associates, Mahwah, NJ (2007)

The Potential of Tele-Assessment and Virtual Treatment for the Future of Lifelong Learning: A Literature Review A. Jordan Wright

Abstract Lifelong learning includes the traditional education of nontraditional learners (often meaning adult learners who have delayed their higher education and returned to the formal educational setting) as well as continuing professional development and continuing education for those in their careers. Undiagnosed and untreated learning and thinking differences can wreak havoc on those earnestly attempting to learn throughout their lives. However, there are so many barriers to traditional assessment and treatment that many lifelong learners do not have easy access to these services. Tele-assessment (and tele-diagnosis) and virtual treatment hold a potential to reduce barriers, increase access and ease, and provide seamless supports for those whose brains are mismatched to the typical demands of lifelong learning. By reducing time constraints (including requiring rigid, in-office hours as well as travel time), increasing supply (by more easily distributing resources especially to rural and dense, urban areas), and providing valid and research-supported diagnosis and treatment, tele-assessment and virtual treatment can help lift some of the roadblocks that lifelong learners face in their education. Keywords Telehealth · Tele-Assessment · Teletherapy

1 The Problem There is a great deal of literature that shows there are significant unmet educational and psychological needs in children and adolescents [23, 37], and the COVID-19 pandemic highlighted and exacerbated these unmet needs [9, 25]. Further, adults with dyslexia—especially undiagnosed—are often set up to fail as adults at work and beyond [44]. This is especially true for lifelong learning, which includes both the traditional education of nontraditional learners (often meaning adult learners who have delayed their higher education and returned to the formal educational setting) A. J. Wright (B) Parallel Learning & New York University, New York, NY 10003, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_51

629

630

A. J. Wright

as well as continuing professional development and continuing education for those in their careers. Those who are either returning to formal higher education settings or working to improve their knowledge and skill base at work must navigate learning environments that are most often mismatched with their brains if they have learning or thinking differences, such as dyslexia or attention-deficit/hyperactivity disorder (ADHD). This can serve as an extreme roadblock for them to be successful in their educational endeavors. But many needs and diagnoses go unidentified and untreated, for many reasons. One of the primary culprits identified in perpetuating these unmet needs is the lack of appropriate identification of needs, with fewer than a quarter of those with learning and thinking differences being identified [4, 21]. Access to services that could help identify these needs is generally poor, as well as structurally inequitable in a way that reinforces traditional marginalization [22, 42, 53]. This includes more problematic access in rural areas as well as denser, urban areas [3, 31], with children and adults of color being especially lacking in access [28, 29]. Further, logistical barriers bar adults from seeking out assessments and treatment, including time constraints, needing to take off work to engage in a comprehensive evaluation, physical access and the need to travel a great distance for services, and of course cost. School psychologists and related professionals are in short supply in rural areas [2, 11, 12] and overwhelmed in other areas [7, 38], on average working well over twice caseload recommended by the National Association of School Psychologists [32]. Of course, individuals are not getting the evaluations (identification) and services they need. Tele-assessment—the administration of psychological and educational tests typically traditionally given in person via a telehealth or virtual platform [52]—has the potential to redistribute resources, change the way students are identified, and ultimately impact higher education and the broader workforce. Tele-diagnosis, as well as virtual treatment, can interrupt the trajectory from unidentified and unmet educational needs to ultimate failure in higher education or in the workforce. Nontraditional students (often the label for adult learners) generally have a higher incidence of learning differences and disabilities than the general population of students in higher education [26]. Unfortunately, the overwhelming majority of these go unassessed and unidentified [40]. This means that those nontraditional students in higher education are not getting the supports and services that would likely improve their chances of succeeding in college. While many reasons for nontraditional students not succeeding in college focus on intrinsic traits (like motivation, cognitive ability, etc.), some other reasons posited are more structural, including schools simply not having the right resources for these students’ needs [15]. Beyond nontraditional students formally enrolled in higher education, there has been a greater emphasis in recent years on lifelong learning within the workforce itself, including things like certificates, credentials, and badges [36, 41]. While historically much learning after a job or career had been established was informal and largely on-the-job, newer models are emerging to formalize the drive toward lifelong learning [27, 41]. However, these formal programs are in their infancy, as competition for individual workers’ resources is steep. That is, engaging in formal education once on a full-time and demanding job requires a great deal of extra resources on the

The Potential of Tele-Assessment and Virtual Treatment for the Future …

631

part of the workers, many of whom are likely to have been out of a formal academic environment for some time.

2 Tele-Assessment and Virtual Treatment Evidence Tele-assessment represents a significant shift in the way school psychologists (and other psychologists) work, and the research evidence behind its use is really in its infancy. However, there is quite a bit of promising evidence about the use of teleassessment procedures in schools, ranging from a great deal of evidence that review of records, interviews, and observations are relatively unscathed by the change in modality [18, 52] to actual research showing relative equivalency between traditional, in-person testing (cognitive and academic) and remote, tele-assessment procedures [37, 46–48, 52]. While rigorous research has not been replicated (given obvious constraints of the COVID-19 pandemic), many of the larger professional organizations (not to mention test publishers themselves) have supported the use of teleassessment procedures, given the current state of the literature. This has included both the National Association of School Psychologists [33] and the American Psychological Association [50]. Ultimately, this means providers are able to diagnose learning and mental health conditions remotely quite effectively, without the need for formal, in-person, traditional assessment methods. When it comes to virtual treatment models, evidence for tele-mental health, teleoccupational therapy, tele-speech therapy, and virtual academic tutoring and remediation has been amassed for decades [6, 8, 10, 30], long before the COVID-19 pandemic. Online educational and treatment models have been generally widely accepted at this point (depending on the type of education and/or problems being treated), with a great deal of aggregated evidence about effectiveness. Although some barriers have been defined, such as state and local regulatory limitations, overall the benefits of telehealth and virtual education services has the potential to redistribute resources, cut down on significant overhead and travel costs, improve access, and save money [1, 13, 20, 39]. Telehealth and tele-educational models certainly proliferated during the COVID-19 pandemic, though the modalities and processes themselves remained fairly stable from established methods (though with improved technology). That is, telehealth procedures generally followed a standard protocol of video-conferencing that mirrors in-person sessions. Online education typically balanced synchronous (most often through video-conferencing group sessions) and asynchronous components. These are models that have been used for decades.

632

A. J. Wright

3 Tele-Assessment and Virtual Treatment’s Potential Unmet learning and mental health needs in school, higher education, and in the workforce obviously affects productivity, success, and contribution. As such, a preventative and promotional model has the potential to be economically savvy, saving society and industry a great deal of money on intervention [24]. Prevention and promotion models, though, have a significant potential barrier, as the up-front cost can be jarring compared to the unknown (and vague) future costs associated with lost productivity and necessary intervention for learning and mental health problems. That is, although the cost of prevention and promotion models is invariably lower than the longer-term costs of not engaging in them and needing intervention later on, these latter costs are less visible and predictable, and so investing in prevention and promotion feels more expensive in the short term. Employee assistance programs (EAPs) have generally embraced virtual care for over a decade, though uptake of such services has been slow [5]. Despite the fact that it has been found to be more convenient, time-saving, and as effective as in-person counseling, college counseling centers have been slower to engage in teletherapy practice, at least up until the COVID-19 pandemic [35]. Both EAPs and college counseling centers further typically engage in extremely little psychological testing and assessment [34, 45], even in traditional, face-to-face models. This means that the primary model in higher education and in the workforce is one of intervention, rather than prevention and promotion, even though rapid screening (which could trigger deeper assessment, when needed) might be more cost effective and help fewer students and workers ‘slip through the cracks.’ This is especially true of adults, who have to navigate the autonomy of adult living at work, in relationships, and across myriad different contexts of their lives where learning disabilities and other neurodivergences—especially undiagnosed—can wreak havoc [14, 16, 43]. Ultimately, the cost-effectiveness, time-savings, and effectiveness of both teleassessment (and diagnosis) and virtual treatment (educational and mental health) have the potential to support students in school, higher education, and at work in ways that will decrease struggling and suffering. These practices can offer students and adults the supports they need to be successful in higher education and at work and decrease problems with productivity and the need for longer term treatment. Further, holistic and integrated treatment planning and intervention—beginning with tele-assessment and continuing on to targeted interventions and supports—fulfills the promise of psychological assessment as a tool for improving functioning in a knowledge-rich and seamless process [51].

The Potential of Tele-Assessment and Virtual Treatment for the Future …

633

4 Conclusion Learning and thinking differences (like dyslexia and ADHD) bring with them a great deal of strength [17, 19], but they also pose a significant mismatch between the way an individual’s brain works and what is generally, typically expected of them in higher education and workplace contexts [49]. Identifying these kinds of struggles as early as possible (through assessment and diagnosis) and intervening seamlessly (through supports, accommodations, and interventions) can mitigate larger problems, both for individuals themselves and for those they work with. Tele-assessment, tele-diagnosis, and virtual treatment options offer a valid, effective, and cost- and time-effective method for supporting this preventative and promotional process in higher education and the workforce.

References 1. Brophy, P.D.: Overview on the challenges and benefits of using telehealth tools in a pediatric population. Adv. Chronic Kidney Dis. 24(1), 17–21 (2017) 2. Castillo, J.M., Curtis, M.J., Tan, S.Y.: Personnel needs in school psychology: a 10-year followup study on predicted personnel shortages. Psychol. Sch. 51, 832–849 (2014). https://doi.org/ 10.1002/pits.21786 3. Clopton, K.L., Knesting, K.: Rural school psychology: reopening the discussion. J. Res. Rural. Educ. 21, 21–25 (2006) 4. Cortiella, C., Horowitz, S.H.: The State of Learning Disabilities: Facts, Trends and Emerging Issues. National Center for Learning Disabilities, New York (2014). www.ncld.org/wp-con tent/uploads/2014/11/2014-State-of-LD.pdf 5. Couser, G.P., Nation, J.L., Hyde, M.A.: Employee assistance program response and evolution in light of covid-19 pandemic. J. Work. Behav. Health 36(3), 197–212 (2021) 6. Crowe, M., Inder, M., Manuel, J., Carlyle, D.: Characteristics of effective teletherapy for major depression: a systematic review. J. Affect. Disord. 327, 175–182 (2023) 7. Curtis, M.J., Chesno-Grier, J.E., Abshier, D.W., Sutton, N.T., Hunley, S.: School psychology: turning the corner into the twenty-first century. Communique 30, 5–6 (2002) 8. Dehghani, S., Mirzakhany Araghi, N., Dehghani, S., Pashmdarfard, M.: The use of teleoccupational therapy for children and adolescents with different disabilities: systematic review of RCT articles. Med. J. Islamic Republ. Iran (MJIRI) 37(1), 118–129 (2023) 9. Duan, L., Zhu, G.: Psychological interventions for people affected by the COVID-19 epidemic. Lancet Psychiatr. 7, 300–302 (2020). https://doi.org/10.1016/S2215-0366(20)30073-0 10. Eslami Jahromi, M., Ahmadian, L., Bahaadinbeigy, K.: The effect of tele-speech therapy on treatment of stuttering. Disabil. Rehabil. Assist. Technol. 17(1), 34–39 (2022) 11. Fagan, T.K.: School psychology’s significant discrepancy: historical perspectives on personnel shortages. Psychol. Sch. 41, 419–430 (2004). https://doi.org/10.1002/pits.10185 12. Gadke, D.L., Valley-Gray, S., Rossen, E.: NASP annual report of graduate education in school psychology: 2014–2015. NASP Res. Rep. 1, 2 (2016) 13. Gajarawala, S.N., Pelkowski, J.N.: Telehealth benefits and barriers. J. Nurse Pract. 17(2), 218–221 (2021) 14. Gerber, P.J.: The impact of learning disabilities on adulthood: a review of the evidenced-based literature for research and practice in adult education. J. Learn. Disabil. 45(1), 31–46 (2012) 15. Goncalves, S.A., Trunk, D.: Obstacles to success for the nontraditional student in higher education. Psi Chi J. Psychol. Res. 19(4), 164–171 (2014)

634

A. J. Wright

16. Gordon, C.T., Fabiano, G.A.: The transition of youth with ADHD into the workforce: review and future directions. Clin. Child. Fam. Psychol. Rev. 22, 316–347 (2019) 17. Grigorenko, E.L., Compton, D.L., Fuchs, L.S., Wagner, R.K., Willcutt, E.G., Fletcher, J.M.: Understanding, educating, and supporting children with specific learning disabilities: 50 years of science and practice. Am. Psychol. 75(1), 37–51 (2020). https://doi.org/10.1037/amp000 0452 18. Hass, M.R., Leung, B.P.: When you can’t RIOT, RIO: tele-assessment for school psychologists. Contemp. Sch. Psychol. 25, 33–39 (2021) 19. Hatak, I., Chang, M., Harms, R., Wiklund, J.: ADHD symptoms, entrepreneurial passion, and entrepreneurial performance. Small Bus. Econ. 57, 1693–1713 (2021) 20. Jennett, P.A., Hall, L.A., Hailey, D., Ohinmaa, A., Anderson, C., Thomas, R.: The socioeconomic impact of telehealth: a systematic review. J. Telemed. Telecare 9(6), 311–320 (2003) 21. Jensen, P.S., Goldman, E., Offord, D., Costello, E.J., Friedman, R., Huff, B.: Overlooked and underserved: “Action signs” for identifying children with unmet mental health needs. Pediatrics 128, 970–979 (2011). https://doi.org/10.1542/peds.2009-0367 22. Kastrup, M.C.: Inequity in mental health: an issue of increasing public health concern. World Soc. Psychiatr. 1, 36–38 (2019) 23. Kieling, C., Baker-Henningham, H., Belfer, M., Conti, G., Ertem, I., Omigbodun, O.: Child and adolescent mental health worldwide: evidence for action. Lancet 378, 1515–1525 (2011). https://doi.org/10.1016/S0140-6736(11)60827-1 24. Le, L.K.D., Esturas, A.C., Mihalopoulos, C., Chiotelis, O., Bucholc, J., Chatterton, M.L.: Costeffectiveness evidence of mental health prevention and promotion interventions: a systematic review of economic evaluations. PLoS Med. 18(5), e1003606 (2021) 25. Lee, J.: Mental health effects of school closures during COVID-19. The Lancet: Child & Adolescent Health. Advance Online Publication (2020). https://doi.org/10.1016/S2352-464 2(20)30109-7 26. Leggins, S.: The “new” nontraditional students. J. Coll. Admiss. 251, 34–39 (2021) 27. Martin, W., Gutierrez, J., Muldoon, M.: Digital badges forging connections between informal and higher education. Aftersch. Matt. 33, 16–24 (2020) 28. Morgan, P.L., Farkas, G., Hillemeier, M.M., Maczuga, S.: Are minority children disproportionately represented in early intervention and early childhood special education? Educ. Res. 41, 339–351 (2012). https://doi.org/10.3102/0013189X12459678 29. Morgan, P.L., Farkas, G., Hillemeier, M.M., Mattison, R., Maczuga, S., Li, H., Cook, M.: Minorities are disproportionately underrepresented in special education: longitudinal evidence across five disability conditions. Educ. Res. 44, 278–292 (2015). https://doi.org/10.3102/001 3189X15591157 30. Mutezo, A., Maré, S.: The effectiveness of e-tutoring in an open and distance e-learning environment: evidence from the University of South Africa. Open Learn. J. Open Dist. e-Learn. 36(2), 164–180 (2021). https://doi.org/10.1080/02680513.2020.1717941 31. Myers, K.M., Vander Stoep, A., McCarty, C.A., Klein, J.B., Palmer, N.B., Geyer, J.R.: Child and adolescent telepsychiatry: variations in utilization, referral patterns and practice trends. J. Telemed. Telecare 16, 128–133 (2010). https://doi.org/10.1258/jtt.2009.090712 32. National Association of School Psychologists. Model for comprehensive and integrated school psychological services. Author (2010) 33. National Association of School Psychologists. Guidance for delivery of school psychological telehealth services. Author (2017). https://www.nasponline.org/assets/documents/Guidance_ Telehealth_Virtual_Service_%20Delivery_Final%20(2).pdf 34. Newhart, S., Pohto, P., Mullen, P.R.: A survey of demographic, professional, and clinical characteristics of clinicians in university counseling centers. J. Coll. Couns. 24(2), 98–114 (2021) 35. Nobleza, D., Hagenbaugh, J., Blue, S., Stepchin, A., Vergare, M., Pohl, C.A.: The use of telehealth by medical and other health professional students at a college counseling center. J. Coll. Stud. Psychother. 33(4), 275–289 (2019)

The Potential of Tele-Assessment and Virtual Treatment for the Future …

635

36. O’Banion, T.U.: A brief history of workforce education in community colleges Community College. J. Res. Pract. 43(3), 216–223 (2019) 37. Patel, V., Kieling, C., Maulik, P.K., Divan, G.: Improving access to care for children with mental disorders: a global perspective. Arch. Dis. Child. 98, 323–327 (2013). https://doi.org/ 10.1136/archdischild-2012-302079 38. Reschly, D.J.: The present and future status of school psychology in the United States. Sch. Psychol. Rev. 29, 507–522 (2000) 39. Reynolds, C.A., Maughan, E.D.: Telehealth in the school setting: an integrative review. J. Sch. Nurs. 31(1), 44–53 (2015) 40. Reynolds, S.L., Johnson, J.D., Salzman, J.A.: Screening for learning disabilities in adult basic education students. J. Postsec. Educ. Disabil. 25(2), 179–195 (2012) 41. Roumell, E.A., Todoran, C., Salajan, F.D.: A framework for capacity building in adult and workforce education programming. Adult Liter. Educ. 2(2), 16–32 (2020) 42. Saxena, S., Thornicroft, G., Knapp, M., Whiteford, H.: Resources for mental health: scarcity, inequity, and inefficiency. Lancet 370, 878–889 (2007). https://doi.org/10.1016/S0140-673 6(07)61239-2 43. Shifrin, J.G., Proctor, B.E., Prevatt, F.F.: Work performance differences between college students with and without ADHD. J. Atten. Disord. 13(5), 489–496 (2010) 44. Tanner, K.: Adult dyslexia and the ‘conundrum of failure.’ Disabil. Soc. 24(6), 785–797 (2009) 45. Weis, D.: Employee assistance programs and behavioral health disability. In: Warren, P.A. (ed.) Handbook of Behavioral Health Disability Management, pp. 289–318. Springer (2018) 46. Wright, A.J.: Equivalence of remote, online administration and traditional, face-toface administration of the Reynolds Intellectual Assessment Scales-Second Edition [White Paper] (2018a). https://www.parinc.com/Portals/0/Webuploads/samplerpts/RIAS2% 20White%20Paper%20FINAL.docx. Accessed 13 Mar 2023 47. Wright, A.J.: Equivalence of remote, online administration and traditional, face-to-face administration of the Woodcock-Johnson IV cognitive and achievement tests. Arch. Assess. Psychol. 8(1), 23–35 (2018b) 48. Wright, A.J.: Equivalence of remote, digital administration and traditional, in-person administration of the WISC-V. Psychol. Assess. 32(9), 809–817 (2020). https://psycnet.apa.org/ful ltext/2020-54568-001.pdf 49. Wright, A.J.: Deliberate context-driven conceptualization in psychological assessment. J. Pers. Assess. 104(5), 700–709 (2022) 50. Wright, A.J., Mihura, J.L., Pade, H., McCord, D.M.: Guidance on Psychological TeleAssessment During the COVID-19 Crisis. American Psychological Association (2020). https:// www.apaservices.org/practice/reimbursement/health-codes/testing/tele-assessment-covid-19 51. Wright, A.J., Pade, H., Gottfried, E.D., Arbisi, P.A., McCord, D.M., Wygant, D.B.: Evidencebased clinical psychological assessment (EBCPA): review of current state of the literature and best practices. Prof. Psychol. Res. Pract. 53(4), 372–386 (2022). https://doi.org/10.1037/pro 0000447 52. Wright, A.J., Raiford, S.E.: Essentials of Psychological Tele-Assessment. Wiley (2021) 53. Yun, K., Lurie, N., Hyde, P.S.: Moving mental health into the disaster-preparedness spotlight. N. Engl. J. Med. 363, 1193–1195 (2010). https://doi.org/10.1056/NEJMp1008304

ALICE (Adaptive Learning Via Interactive, Collaborative and Emotional Approaches) Special Track

Explainable Prediction of Student Performance in Online Courses Nicola Capuano, Diego Rossi, Victor Ströele, and Santi Caballé

Abstract Student Performance Prediction (SPP) models and tools are useful for quickly identifying at-risk students in online courses and enable the provision of personalized learning plans and assistance. Additionally, they give educators and course managers the information they need to recognize the programs that require improvement. High accuracy is essential for such tools, but understanding the reasons of their predictions is equally important to ensure fairness and build trust in their adoption. Although many SPP models and tools have been proposed so far by different researchers, very few of them take explainability into account. This research proposes an SPP approach that is both effective and explainable. Based on demographic, administrative, engagement, and intra-course outcome data, it enables the prediction of student performance in terms of success/failure and final grade. It supports multiple machine learning models and includes post-hoc techniques for explainability capable of justifying the behavior of the whole system as well as its individual predictions. Keywords Learning analytics · Educational data mining · Student performance prediction · Explainable artificial intelligence

N. Capuano (B) School of Engineering, University of Basilicata, Potenza, Italy e-mail: [email protected] D. Rossi · V. Ströele Federal University of Juiz de Fora, Juiz de Fora, Brazil e-mail: [email protected] V. Ströele e-mail: [email protected] S. Caballé Open University of Catalonia, Barcelona, Spain e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_52

639

640

N. Capuano et al.

1 Introduction The increased availability of learning data from online tools and Virtual Learning Environments (VLEs) has given Learning Analytics (LA) a growing importance in understanding and optimizing the learning process and the environments in which it occurs [1]. The COVID-19 pandemic has further highlighted the need of LA. In fact, at the end of April 2020, higher education institutions in around 180 countries closed their campuses and quickly switched to online education. As a matter of fact, even though the global health emergency is subsiding, many of these educational institutions still include online learning as a component of their offerings [2]. Student Performance Prediction (SPP) is among the most prominent challenges addressed by LA. It allows students who may fail the final exam to be identified in time to provide them with further assistance or personalized learning paths. In addition, it could provide relevant information to educators to identify courses and programs that need improvement or additional resources [3]. Applications of SPP include the identification of at-risk students and the prediction of dropout. It can also be used to build early warning tools [4] and customized recommendation systems [5]. High prediction accuracy is crucial for SPP models, but their effectiveness depends on other factors as well. Indeed, more and more researchers recognize the importance for stakeholders to trust the SPP model by understanding the reasons of its predictions [6]. In line with the principles of Explainable Artificial Intelligence (XAI), both the accuracy and the interpretability of model predictions are equally important to build trust and confidence in their adoption. This also aims to contrast the so-called black box phenomenon, which refers to the difficulty for users, even domain experts, to understand the internal mechanisms of the model and the generated outcomes [7]. Explainable SPP would add value over standard SPP to all stakeholders. Students would benefit if the model justified the factors behind a decision — e.g., if a student is expected to fail due to certain factors, he or she can work on those factors to increase the chances of being successful. Educational institutions can use SPP to recommend courses to students based on their strengths: explanations will allow them to trust the model and then follow its recommendations. The obtained information would also help management improve courses and programs and would be fundamental for developers to verify the correctness and fairness of AI models. Unfortunately, according to ref. [6], although the adoption of XAI techniques is increasing in many fields, there are still very few applications for SPP. Moreover, the existing applications only use simple models that are inherently explainable as decision trees and rules-based systems. SPP tools based on more complex and performing models, on the other hand, seem not to consider explainability. To fill this gap, this research proposes an SPP approach that aims to be both performing and explainable, based on state-of-the-art Machine Learning (ML) models and XAI techniques.

Explainable Prediction of Student Performance in Online Courses

641

The defined approach can predict student performance in terms of success/failure and final grade (in case of success) based on demographic, administrative, engagement, and intra-course performance information. It supports several ML models that have been trained and tested on different parts of the Open University Learning Analytics Dataset (OULAD) dataset [8] and have been compared with respect to obtained performance. Finally, it incorporates XAI techniques that can provide global and local explanations — i.e., explanations for both the behavior of the model as a whole and for individual predictions. The remainder of this paper is structured as follows: the related work on explainable SPP is discussed in Sect. 2; the defined SPP approach and the related ML models are described in Sect. 3; the applied XAI techniques and their integration within the SPP approach are reported in Sect. 4; and conclusions are drawn in Sect. 5.

2 Related Work Artificial intelligence (AI) has been widely adopted in recent years in a variety of application domains, including LA and Educational Data Mining (EDM). Indeed, AI and ML techniques are increasingly used to generate automatic predictions and recommendations regarding learning and teaching [9–11]. Learning and content analysis, knowledge monitoring, the reinforcement of instructional materials, and SPP are some of the most popular applications [12]. The purpose of SPP is to use relevant information of students to predict their future learning performance [13]. It can be shaped as a classification task (e.g., predict if the student will pass or fail the exam) or as a regression one (e.g., predict the student final grade). In ref. [3], SPP approaches are broadly classified into three categories: similarity-based (predictions consider the performance of similar students), modelbased (predictions are based on a model built on the implicit correlations between the analyzed data samples), and probabilistic (predictions are based on the characteristics of the probability distribution observed in the dataset). Model-based approaches, often based on ML techniques, have proven to be the most accurate. Among these, in ref. [14], a set of features collected from the first six weeks of a course was used to implement an SPP model to predict student outcome. As part of this work, the authors compared five ML models and found that support vector machines outperformed other models for this task. In ref. [15], three feedforward neural networks were used to progressively predict the final grades of students enrolled in an e-learning course. In ref. [16], students were clustered based on learning behaviors and a convolutional neural network was used to predict whether students could complete the course based on personal learning behaviors and that of other students in the same cluster. Among the most comprehensive and popular datasets for LA, widely used to train and test SPP models, is the OULAD, which was also adopted in this research. In ref. [17], 18 OULAD-based research papers for predicting students at risk, early dropout, and final exam success, published from 2017 to 2021, were described and compared.

642

N. Capuano et al.

Most of them use ML models to generate predictions, but none of them apply XAI techniques to provide explanations. Indeed, the application of XAI techniques to LA is increasingly in demand by researchers and practitioners. In ref. [18], the new research line of Explainable Learning Analytics (XLA) was shaped, merging XAI and LA. The authors argue that, while visualization and ML techniques can provide accurate recommendations, they are not always in themselves trusted by the user. Opening the black box to the user is considered the first step towards obtaining transparency, reliability, persuasion, and effectiveness. On the other hand, a recent review on XAI techniques applied to SPP highlights the lack of studies in this area [6]. The most used algorithms are inherently explainable like decision trees and rule-based algorithms. When more complex prediction models are adopted, explainability is no longer considered. Furthermore, none of the analyzed works use evaluation metrics to assess the explainability of the proposed models. Only recently, some preliminary research in this field has been published. For example, in ref. [19], several classifiers were tested on a selected set of features to predict student achievement across seven university courses. The classifier predictions were analyzed both locally and globally using Shapley values (see Sect. 4) to provide explanations to at-risk students. In ref. [20], several ML models were experimented to predict the performance of secondary school students and Local Interpretable Model-Agnostic Explanations (LIME) was used to obtain local explanations. This research goes in the direction traced by these seminal works and, applying the principles of XLA, aims to define an explainable approach for SPP. Unlike the last cited works, a widely adopted open dataset (OULAD) has been used rather than a closed one to make the obtained results broadly comparable.

3 Student Performance Prediction The defined approach to explainable SPP relies on state-of-the-art ML models capable of predicting student performance in terms of success/failure and final grade. Six classification models and six regressors have been selected for the two tasks. Models training was performed on the OULAD dataset, whose features have been preliminarily engineered, as explained in Subsect. 3.1. Models’ performance in terms of accuracy and mean absolute error is then presented and discussed in Subsect. 3.2.

3.1 Preprocessing and Features Engineering The OULAD [8], which was released in 2017, contains information on 22 editions of seven courses of the UK Open University held between 2013 and 2014. It includes anonymized demographic data of 32,593 students, the courses taken (referred to as

Explainable Prediction of Student Performance in Online Courses

643

modules), the time of year the courses began (referred to as presentations), data on students’ academic success as measured by grades on their homework assignments and final exams (173,912 entries), and the log of the interactions with the university’s VLE (more than 10 million entries). The OULAD data scheme consists of seven related tables released as separate CSV files. A single data view was obtained by joining and aggregating such tables on the shared fields. In particular, for each triple (student, module, presentation), the following features were extracted and grouped in five categories: – demographic: gender, region (the region where the student lived during the course), highestEducation (the student’s education level before entry into the course), imdBand (the Index of Multiple Deprivation, or IMD, of the place where the student lived during the course), ageBand (0–35, 35–55 or 55+ ), disability (yes/no); – administrative: dateRegistration (the day of enrollment in the course with respect to the start date of the course), dateUnregistration (the day of possible cancellation from the course with respect to the start date of the course), modulePresentationLength (the course duration in days), studiedCredits (the number of credits gained by the student before the course), numOfPrevAttempts (the number of times the student attempted this course before); – engagement: totalClicks (the number of times the student interacted with course material within the VLE); – performance: assessmentsDue (the total number of homework required by the course), assessmentsCount (the number of homework submitted by the student), assessmentsLate (the number of homework submitted by the student after the deadline), weightedScore (the overall weighted score obtained by the student for homework — i.e., the sum of each score multiplied by the weight of the assignment); – outcome: grade (the score obtained by the student in the final exam of the course, from 0 to 100), finalResult (the final result of the student between distinction, pass, fail, withdrawn); Some inconsistencies in the dataset were corrected during preprocessing. A uniform weight distribution was associated with some courses with unweighted assignments, some missing IMD bands were filled with the most frequent IMD bands for each region, and missing registration dates were filled with the course start day. About 10% of the students did not register any interaction with the VLE; for these students, we considered 0 total clicks. Moreover, about 18% of students have no registered homework score; for these students, we considered 0 assessments with a weighted score of 0. It should be noted that only 4,959 students out of 32,593 have a registered grade for the final exam, and this contrasts with the fact that, for 22,437 students, the registered finalResult is either distinction, pass, or fail, meaning that the student has at least attempted the final exam. Evidently not all the results of the final exam were recorded, as confirmed also in ref. [10]. Therefore, only the subset of students with the final exam grade were considered for the final grade prediction task.

644

N. Capuano et al.

Table 1 Description of the selected, engineered features Category

Feature

Value description

demographic

gender

Binary: 0 (male) or 1 (female)

ageBand

Ordinal: 0 (0–35), 1 (35–55), 2 (more than 55)

disability

Binary: 0 (no) or 1 (yes)

imdBand

Ordinal: from 0 (0–10%) to 9 (90–100%)

highestEducation

Ordinal: 0 (no formal qualifications), 1 (lower than A level), 2 (A level or equivalent), 3 (higher education), 4 (postgraduate)

administrative dateRegistration

studiedCredits

Scaled so that 90% of the distribution is in [-1, 1], negative values are registrations prior to the course start (often this is the case but not always) Scaled so that 90% of the distribution is in [0, 1]

numOfPrevAttempts

Scaled so that 90% of the distribution is in [0, 1]

engagement

totalClicks

Scaled so that 80% of the distribution is in [0, 1]

performance

assessmentsRate

Ratio between the number of homework submitted by the student and those required by the course

assessmentsLateRate Ratio between the number of homework submitted late by the student and those submitted outcome

weightedScore

Weighted score divided by 100 (maximum score)

finalResult

Binary: 0 (failed or withdrawn) or 1 (passed with or without distinction)

grade

Grade obtained in the final exam divided by 100

The selected data was then engineered for use by the ML models described in the next subsection. Ordinal encoders were applied to categorical data, while quantilebased transformers were used with the fields belonging to the administrative and engagement groups, to scale data robustly with respect to outliers. Table 1 reports the structure of the selected features and how they have been prepared. In particular, the features belonging to the outcome group were used as targets for model predictions.

3.2 Estimators Definition and Evaluation Six binary classifiers were trained with different subsets of the features described in the 3.1 for the task of predicting the finalResult field. Trained models include a logistic regressor with LBFGS optimizer and L2 regularization; a k-nearest neighbors’ classifier with k = 10; a decision tree classifier based on the Gini impurity measure; a random forest model with 20 estimators; a support vector machine classifier with RBF kernel; and a multilayer perceptron with two hidden layers of 32 and 16 nodes, ReLU activation, and Adam optimizer.

Explainable Prediction of Student Performance in Online Courses

645

The selected classifiers were trained to solve seven problems related to different feature groups including demographic and administrative data (DA), engagement (E), performance (P), demographic, administrative and engagement (DAE), demographic, administrative and performance (DAP), engagement and performance (EP), or the complete dataset (DAEP). The estimation of student performance based on DA features is obviously the most difficult task and can be seen as pre-course outcome prediction — i.e., predicting the performance before the course start. Classification performance was evaluated through a stratified k-fold crossvalidation process [21] in which each dataset was divided into 4 groups containing approximately the same percentage of samples of each target class as the complete dataset (however, the dataset is quite balanced with 53% pass and 47% fail). Then, each group was used as test-set for a model trained on the remaining part. The model performance, measured in terms of accuracy (proportion of correct predictions on the total number of cases tested), was then averaged among the 4 folds. Results are reported in Table 2. The random forest classifier shows the best performance in almost any task with greater than 92% accuracy on P, EP, and DAEP. It appears that the most significant feature set is P while the least significant is DA. It should be noted that both the sets E and P consider the entire duration of the course. A future work, it would also be interesting to evaluate the performance of the classifiers at specific times of the course (e.g., after each month) to see how accuracy evolves over the feature sets. Six regressors were also trained with the same feature subsets for the task of predicting the grade field — i.e., the final score obtained by the student at the end of the course. In addition to the models already described, a linear regression model was used instead of the logistic regressor. A k-fold cross validation process was applied also in this case, but the performance was measured in terms of Mean Absolute Error (MAE) — i.e., the arithmetic average of the absolute errors |yi − xi | between each prediction yi and the corresponding true value xi . Results are reported in Table 3. Unlike the previous case, in this case, the dataset is made up of only 4,959 students out of a total of 32,593 — i.e., the only ones to have a score registered for the final Table 2 Classifier performance (accuracy) for the student outcome prediction task Model

DA

E

P

DAE

DAP

EP

DAEP

Logistic Regression

0.607

0.762

0.914

0.785

0.913

0.914

0.913

k-Nearest Neighbors

0.575

0.764

0.920

0.775

0.896

0.919

0.897

Decision Tree

0.585

0.787

0.911

0.787

0.911

0.911

0.911

Random Forest

0.603

0.782

0.924

0.810

0.924

0.924

0.924

Support Vector Machine

0.602

0.790

0.920

0.799

0.911

0.920

0.912

Multilayer Perceptron

0.602

0.790

0.921

0.810

0.919

0.924

0.922

646

N. Capuano et al.

Table 3 Regressor performance (MAE) for the student grade prediction task E

P

DAE

DAP

EP

DAEP

Linear Regression 0.169

0.169

0.124

0.164

0.123

0.124

0.123

k-Nearest Neighbors

0.173

0.172

0.119

0.167

0.161

0.127

0.159

Decision Tree

0.169

0.167

0.137

0.166

0.137

0.137

0.137

Model

DA

Random Forest

0.169

0.169

0.116

0.161

0.115

0.116

0.115

Support Vector Machine

0.168

0.167

0.116

0.160

0.123

0.116

0.123

Multilayer Perceptron

0.170

0.167

0.121

0.165

0.123

0.125

0.126

exam. Both the support vector machine and the random forest regressors show good performance in most of the tasks with the latter scoring an average error of 11.5% on the final grade on DAP and DAEP. The most significant feature set is P also in this case while the least significant is DA. The experiments confirm that the performance of students on homework is a good estimator of the grade achieved in the final exam.

4 Predictions Explanation After training several models to solve both the classification and regression problem for SPP, this section describes the application of XAI techniques for interpreting and explaining the predictions of these models. In particular, we adopt a post-hoc technique named SHAP, based on game theory, which is aimed at approximating a black box ML model by generating simpler surrogate models. Subsection 4.1 describes the adopted technique, while Subsects. 4.2 and 4.3 explain how it has been used for generating global and local explanations. In fact, while global explanations provide insights into the trained ML model by identifying the most important features, local explanations identify the contribution of every feature’s value to each individual prediction.

4.1 Shapley Additive Explanations A trained ML model defines a (generally complex and difficult to interpret) function f that maps the instance space X to the label space Y . Without any internal knowledge of the model, a post-hoc local explainer is aimed at approximating f locally near a given input with a simpler function g that can be easily interpreted. SHAP, acronym for SHapley Additive exPlanations, is a post-hoc local explanation technique based on Shapley values from coalitional game theory. Given an instance

Explainable Prediction of Student Performance in Online Courses

647

x = (x1 , . . . xn ), the model prediction y = f (x) is explained in terms of the marginal contribution of the value of each feature xi to the overall prediction. In other words, for each instance x, the prediction f (x) is approximated as: g(x) = φ0 +

n i=1

φi

(1)

where φi is the contribution of the i-th feature with i ∈ {1, . . . , n} and φ0 is a contribution that is independent of the features [22]. Shapley values are consistent, in the sense that if a feature xi has more influence on the model than another feature x j then φi > φ j . Moreover, it has been demonstrated that they are unique, in the sense that they are the only possible solutions to Eq. 1 that are also consistent. The equation for obtaining Shapley values is: φi =

 S⊆Ni

(n − |S| − 1)!|S|! [ f x (S ∪ {i}) − f x (S)] n!

(2)

where n is the number of features, Ni is the set of all features excluding the i-th, f x (S) is the output of the model on the instance x using only the subset S of features and, thus, f x (S ∪ {i}) − f x (S) is the contribution of the i-th feature to the output [23]. The number of possible subsets of Ni is 2n−1 and increases exponentially with the number of features so, in practical applications, a number of subsets of Ni are randomly sampled. Furthermore, the prediction on an instance in the absence of some features is obtained by replacing the values of those features with the values of the same features in randomly selected instances from the training set. The Shapley values can be used for local explanations. Given a set of feature values, they represent the additive contribution of each feature value to the difference between the actual prediction and the average prediction. Moreover, by averaging the absolute Shapley values calculated on a sample of instances from a test set, a global explanation of a model can be obtained in terms of the importance of each feature.

4.2 Global Explanation Through SHAP, it is possible to calculate the importance of each feature for both the student outcome prediction (classification) and the student score prediction (regression) tasks described in Sect. 3. To this end, the best model for both tasks (random forest) was selected and, for each feature set (among DA, E, P, DAE, DAP, EP and DAEP), 200 random instances from the test set were used to compute Shapley values as described in Subsect. 4.1. Table 4 shows the absolute Shapley values averaged over the whole samples for the classification task. They can be seen as a measure of the importance of each feature for the model trained on each feature set. Note that, as the feature set changes, the

648

N. Capuano et al.

feature importance also changes. For example, the highestEducation feature is the most important for the model trained on DA (i.e., without any information on student engagement and intra-course performance) but is negligible if information on student intra-course performance is also available (e.g., for DAP or DAEP feature sets). A similar table can be obtained for the regression task, but we omit it for the sake of brevity. Several graphs can be used to make Shapley values more usable by end users; for example, the pie charts shown in Fig. 1 for DA and DAEP feature sets only. The global explanations can be used by teachers to understand which aspects should be revised first to improve students’ overall performance. For example, from the graph of Fig. 1 (right side), it can be argued that a possible strategy to improve Table 4 Shapley values for the random forest model trained for student outcome prediction Feature

DA

E

P

DAE

DAP

EP

DAEP

gender

0,014

0,044

0,004

0,005

ageBand

0,019

0,004

0,004

0,003

disability

0,007

0,003

0,001

0,001

imdBand

0,037

0,016

0,003

0,003

highestEducation

0,066

0,037

0,005

0,009

dateRegistration

0,017

0,008

0,004

0,003

studiedCredits

0,045

0,025

0,004

0,004

numOfPrevAttempts

0,029

0,010

0,003

0,003

totalClicks assessmentsRate

0,297

0,285 0,199

0,209

0,034

0,039

0,221

0,182

assessmentsLateRate

0,013

0,015

0,018

0,012

weightedScore

0,224

0,212

0,188

0,202

Fig. 1 Feature importance for the SPP classification task with different feature sets

Explainable Prediction of Student Performance in Online Courses

649

performance is to encourage students to do their homework because the assessmentRate feature affects students’ outcome by 39%. Interaction with the VLE should also be stimulated because the totalClicks feature affects the outcome by 8%. The global explanations can also be used by course managers to generate recommendations or understand how to structure course access criteria based on pre-course features (Fig. 1, left side) that mostly affect student performance. Such criteria would be selected from those with the greatest impact if they are ethically acceptable. In this sense, the global explanations will also help the model developer to avoid discriminatory effects — for example, by excluding those features which, while showing a high relevance — are ethically unacceptable (e.g., gender, age, or IMD). It is worth noting that the selection of ethically acceptable features should also consider the indirect discrimination possibly introduced by features that, while not directly ethically sensitive, act as proxy fields masking ethically sensitive information. While this does not seem to apply to OULAD, it could apply to larger datasets and lead to what is known as a “red-lining” effect — i.e., the biased practice of excluding individuals from opportunities based on their race, gender, or other protected features.

4.3 Local Explanations Local explanations can be used to provide personalized advice to individual students. To this end, starting from Shapley values of a single prediction, several user-friendly graphs can be generated. Figure 2 shows two examples of waterfall plots, which are designed to see how the Shapley value of each feature shifts the model output from the expected value (the average model output) to the specific output for the individual instance. The features are sorted by the magnitude of their contribution with the smallest grouped at the bottom of the graph. In both examples (referring to the DAEP feature set), the expected output of the model is E[ f (X )] = 0.493 that is the average probability to pass the exam. The prediction for the first student is f (x) = 0.213 meaning that he or she is expected to fail the exam. According to the graph, even though the student’s interaction with the VLE and the number of completed assignments were satisfactory (although many were sent late), his assignment score was too low, which suggests that the exam will be unsuccessful. The other features are essentially irrelevant to the prediction. The prediction for the second student is f (x) = 0.703 meaning that he or she is expected to pass the exam. According to the graph, the most relevant feature that affect the prediction is to have completed all the homework, even with an average score (in fact the weightedScore contribution is modest) together with an acceptable interaction with the VLE. The contribution of other features is residual.

650

N. Capuano et al.

Fig. 2 Local features contribution in two individual predictions

5 Conclusions In this paper, an effective and explainable SPP approach has been proposed to predict student performance in online learning systems in terms of both success/failure and grade in the final exam. Regarding effectiveness, the defined classifier for binary outcome prediction achieves an accuracy greater than 92%, while the regressor defined for final grade prediction shows an average absolute error of 11.5%. Regarding explainability, SHAP has been used to determine the additive contribution of each feature value to individual model predictions thus explaining the reasons behind each (positive or negative) output. Then, by averaging the absolute Shapley values calculated on a sample of instances from the test set, a global explanation of a model is obtained in terms of the importance of each feature. It should be noted that trained models consider different feature sets, some including only information available at the beginning of the course (e.g., demographic, and administrative) some including all information available at the end of the course, before the final exam (e.g., engagement and inter-course performance). As future work, we plan to evaluate models’ performance at specific times of the course (e.g., after each month) to see how accuracy and explanations evolves with

Explainable Prediction of Student Performance in Online Courses

651

partial information on engagement and inter-course performance. This would be also useful to provide timely advice to students and teachers before course ends. In previous works [24, 25], we used machine learning and ontology to identify the best pedagogical intervention to be made by teachers and tutors, considering messages posted by students in discussion forums. The model can classify students’ messages from three perspectives: sentiment, urgency, and confusion. As a future work, we plan to apply XAI techniques described in this paper for interpreting and explaining the predictions of this model, aiming to improve pedagogical interventions. Moreover, we also plan to experiment with additional XAI models such as those based on counterfactual explanations that seek to identify the smallest necessary change in feature values that modify the prediction towards the desired performance. This would be a further step towards generating real actionable feedback.

References 1. Viberg, O., Hatakka, M., Bälter, O., Anna, M.: The current landscape of learning analytics in higher education. Comput. Hum. Behav. 89, 98–110 (2018) 2. Celik, I., Gedrimiene, E., Silvola, A., Muukkonen, H.: Response of learning analytics to the online education challenges during pandemic: opportunities and key examples in higher education. Policy Fut. Educ. (in press, 2022) 3. Tomasevic, N., Gvozdenovic, N., Vranes, S.: An overview and comparison of supervised data mining techniques for student exam performance prediction. Comput. Educ. 143, 1–18 (2020) 4. Du, X., Yang, J., Hung, J.: An integrated framework based on latent variational autoencoder for providing early warning of at-risk students. IEEE Access 8, 10110–10122 (2020) 5. Capuano, N., Lomasto, L., Pozzi, A., Toti, D.: Natural language understanding for the recommendation of learning resources within student collaboration tools. In: Proceedings of the Learning Ideas Conference, New York (2022) 6. Alamri, R., Alharbi, B.: Explainable student performance prediction models: a systematic review. IEEE Access 9, 1–12 (2021) 7. Adadi, A., Berrada, M.: Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 6, 52138–52160 (2018) 8. Kuzilek, J., Hlosta, M., Zdrahal, Z.: open university learning analytics dataset. Sci. Data 4, 1–9 (2017) 9. Capuano, N., Dell’Angelo, L., Orciuoli, F., Miranda e, S., Zurolo, F.: Ontology extraction from existing educational content to improve personalized e-Learning experiences. In: Proceedings of the 3rd IEEE International Conference on Semantic Computing (ICSC-2009), Berkeley, CA, USA (2009) 10. Capuano, N., Gaeta, M., Salerno e G., S., Mangione, R.: An ontology-based approach for context-aware e-learning. In: Proceedings of the 3rd IEEE International Conference on Intelligent Networking and Collaborative Systems (INCoS-2011), Fukuoka, Japan (2011) 11. Capuano, N., Gaeta, M., Miranda, S., Orciuoli, F., Ritrovato, P.: LIA: an intelligent advisor for e-learning. In: Lytras, M.D., Carroll, J.M., Damiani, E., Tennyson, R.D. (eds.) Emerging Technologies and Information Systems for the Knowledge Society. Lecture Notes in Computer Science (Lecture Notes in Artificial Intelligence), vol. 5288, pp. 187–196. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-87781-3_21 12. Bakhshinategh, B., Zaïane, O., ElAtia, S., Ipperciel, D.: Educational data mining applications and tasks: a survey of the last 10 years. Educ. Inf. Technol. 23, 537–553 (2018)

652

N. Capuano et al.

13. Meier, Y., Xu, J., Atan, O., van der Schaar, M.: Personalized grade prediction: a data mining approach. In: Proceedings of International Conference on Data Mining, Atlantic City, NJ, USA (2015) 14. Li, H., Lynch, C., Barnes, T.: Early prediction of course grades: models and feature selection. arXiv.org.vol.1812.00843 (2018) 15. Lykourentzou, I., Giannoukos, I., Mpardis, G., Nikolopoulos, V., Loumos, V.: Early and dynamic student achievement prediction in e-learning courses using neural networks. J. Am. Soc. Inf. Sci. 60, 372–380 (2009) 16. Feng, W., Tang, J., Liu, T.: Understanding dropouts in moocs. In: Proceedings of the 33rd AAAI Conference on Artifcial Intelligence (2019) 17. Alhakbani, H., Alnassar, F.: Open learning analytics: a systematic review of benchmark studies using open university learning analytics dataset (OULAD). In: Proceedings of the 7th International Conference on Machine Learning Technologies, Rome, Italy (2022) 18. De Laet, T., Millecamp, M., Broos, T., De Croon, R., Verbert, K.: Explainable learning analytics: challenges and opportunities. In: Proceedings of the 10th International Conference on Learning Analytics & Knowledge, Frankfurt, Germany (2020) 19. Jang, Y., Choi, S., Jung, H., Kim, H.: Practical early prediction of students’ performance using machine learning and explainable AI. Educ. Inf. Technol. (in press, 2022) 20. Hasib, K., Rahman, F., Hasnat, R., Alam, A.: A Machine learning and explainable AI approach for predicting secondary school student performance. In: Proceedings of the IEEE 12th Annual Computing and Communication Workshop and Conference (2022) 21. Ojala, M., Garriga, G.: Permutation tests for studying classifier performance. J. Mach. Learn. Res. 11, 1833–1863 (2010) 22. Lundberg, S., Lee, S.: A unified approach to interpreting model predictions. In: Proceedings of the 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA (2017) 23. Gianfagna, L., Di Cecco, A.: Explainable AI with Python, Springer, London (2021) 24. Rossi, D., et al.: CAERS: a conversational agent for intervention in MOOCs’ learning processes. In: Proceedings of the Learning Ideas Conference, New York (2021) 25. Rossi, D., et al.: An architectural system for automatic pedagogical interventions in massive online learning environments. In: Proceedings of the 37th International Conference on Advanced Information Networking and Applications (AINA), Juiz de Fora, Brazil (2023)

The Magic of Games: Creating a Pull-Based Learning System Through Serious Games Ritika Datta, Ajay Gupta, and Bob Philips

Abstract Learning practitioners spend significant time and effort incentivizing or pushing employees to engage with online learning content. In this paper, the authors use a mixed method approach to explore the feasibility of using serious games to create an effective pull-based learning system instead of a push-based one, by analyzing first-hand data from roughly 10,000 corporate learners. Quantitative data on the number of games played and replayed provided insights on pull while quantitative and qualitative feedback from learners provided insights on the extent of satisfaction, loyalty and perceived usefulness — strong indicators of effectiveness. The study found significant evidence of pull and effectiveness through unique learner behaviors like game binging (playing multiple games in one sitting) and game replays (repeating a game multiple times even after completion). Both these behaviors were observed in the absence of a pass criteria or organizational mandates on how many games one was required to play. Game buying behavior also provided interesting insights on the extent of pull. Internal marketplaces were set up to allow learners to choose and purchase the learning games that they wanted. It was observed that a number of games were purchased late at night, on weekends, and during public holidays, challenging the assumption that employees do not want to spend their personal time learning workplace skills. Furthermore, the data shows that traditional learning metrics thrived under this pull-based approach, with learning gains of about 21% and completion rates of roughly 90%. The implications of this are relevant for learning game designers and practitioners alike. By moving learners to the center of the learning agenda, the overall learning experience is likely to become more engaging and fruitful for everyone involved. R. Datta (B) · A. Gupta (B) Transcendix Partners LLP, Bengaluru, Karnataka, India e-mail: [email protected] A. Gupta e-mail: [email protected] B. Philips (B) Transcendix Partners LLP, Dubai, UAE e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_53

653

654

R. Datta et al.

Keywords Serious games · Gamification · E-learning

1 Introduction According to a recent study, about 70% of corporate employees prefer online, selfpaced courses to address their training needs [22]. Research suggests that organizations that excel at implementing corporate e-learning, experience improvements in employee efficiency, productivity, lifelong learning and morale boost. These, in turn, have been shown to have a positive impact on long term organizational success [13]. For these reasons and more, organizations spend significant effort ensuring the effectiveness of the e-learning made available to their employees.

1.1 Content-Centric Versus Learner-Centric Measures of Effectiveness The effectiveness of e-learning systems is typically measured through content-centric metrics like completion or abandonment rates — a measure of how many learners finished what they started. With adequate usage of the content at the center of the learning agenda, organizations use various strategies to push employees to engage with this content as much as possible. Some strategies include linking high usage to financial rewards and promotions, and low usage to penalties, reprimands, or financial disincentives. Certain organizations also encourage managers to take accountability for the completion rates of their team members [17]. Despite these efforts, completion rates tend to average around 15% [14], with abandonment rates often being as high as 96% — numbers that have stayed more or less constant through the years [21]. Given this reality, learning practitioners have recognized the need to put learners at the center of the learning agenda. While still keeping an eye on traditional metrics like completion rates, Vargas Souza, Motoki, Mainardas, and Azzari [23] recommend tracking metrics like user satisfaction, loyalty, and perceived usefulness to evaluate the effectiveness of an e-learning system. An e-learning system is considered effective if learners have a positive emotional response after the e-learning activity [24], if learners choose to reuse the e-learning service multiple times [5] and if they believe that the knowledge gained from the e-learning system will bring about a future improvement in their job performance [16]. Under this approach, it is not the learners who are judged for how much they have used the content, but the content which is judged on its ability to pull learners towards it. Several approaches have been utilized to build impactful, engaging learning experiences that create such a pull. This paper explores the impact of two of these approaches in greater detail — serious games and gamification.

The Magic of Games: Creating a Pull-Based Learning System Through …

655

1.2 The Impact of Serious Games on E-learning Effectiveness Serious games are entertaining applications, used to achieve a serious goal [2]. They combine story, art, and technology with pedagogy to impart knowledge or skills [25]. Over the years, serious games have been used in various sectors including military, healthcare, education and business [6]. In the professional context, serious games have been found to make the learning process more enjoyable and engaging [9, 18]. In fact, research suggests that employees trained using serious games show a higher level of satisfaction, better results and an overall more positive experience as compared to those undergoing paper-based training [4]. Studies also show that those playing games display a stronger ability to multi-task, make decisions and effectively evaluate risks. They tend to be more flexible in the face of change and more inclined to treat setbacks as opportunities to try again [1, 8]. As with other approaches, the effectiveness of serious games is dependent on how well the game has been built. According to Przybylsk, Rigby, and Ryan [20], the extent of challenge, autonomy, curiosity, and imagination provided by a game impact how engaging and effective it will be.

1.3 The Impact of Gamification on E-learning Effectiveness Gamification is ‘the use of game-design elements in a non-game context’ [10]. These elements may be reward-driven, like awards, points and badges, competition or public recognition driven like leaderboards, or achievement-driven, like levels that one must progress through to reach a goal. The primary objective of gamification is to enhance motivation, and drive desired user behavior [7]. It is important to note that gamification is not the same as serious games. While serious games are used as an alternative to conventional training methods, gamification involves using game-like elements on top of existing training programs to make them more engaging and fun [3]. Early research shows that gamification is most useful in situations where the learning content itself is not perceived as interesting or valuable [11, 19]. In a more recent study by Hussain et al. [15], it was found that motivation, engagement, commitment and loyalty all increased in post-test gamified environments, while motivation and retention levels reduced in post-test non-gamified environments. While some fear that the extrinsic rewards available through gamification may reduce intrinsic learner motivation, there is evidence to suggest that if built well, these gamification elements can lead to intrinsic motivation and positively change attitudes towards tasks [12]. Impactful gamification systems are those where the rewards commensurate with the level of effort involved, are meaningful and are not too frequent or too scarce [11].

656

R. Datta et al.

1.4 Serious Games and Gamification in a Single Learning Product Combining the principles of games and gamification, the authors have developed a learning product called Flogames. Flogames is a suite of competency-mapped learning games and simulations to help learners master critical behavioral skills. It was developed in 2020 by Transcendix Partners LLP, an India based corporate training organization, of which the authors are a part. The games are short, about 10–15 min each, with immersive story arcs and complex decision structures. They provide a safe, sandbox environment for learners to develop new skills, solve complex problems and receive immediate feedback, as they play. The games are delivered through the Flogames Learning Experience Platform, which includes gamification elements like badges, rewards, leaderboards, bonus points and achievement levels that unlock as more and more games are played. In the present study, first-hand usage data from Flogames is analyzed to explore whether such game-based products can create a pull for learners and provide an effective e-learning experience.

2 The Present Study This section highlights the research questions being explored and the methodology used to collect and analyze the data.

2.1 The Research Questions The following two research questions are being explored in this paper: RQ 1: Is there evidence to suggest that game-based products like Flogames can create a pull-based learning system? In this context, a pull-based learning system is defined as one that actively engages learners and encourages them to keep coming back after their first use. RQ 2: Is there evidence to suggest that game-based Products like Flogames provide an effective e-learning experience? An effective e-learning experience is defined as one that shows a positive impact on learner-centric metrics like satisfaction, loyalty and perceived usefulness, on traditional engagement metrics like completion and abandonment and on proficiency metrics like learning gains.

The Magic of Games: Creating a Pull-Based Learning System Through …

657

2.2 The Data Set Over the last two years, Flogames has sold around 10,000 user licenses to client organizations. Client organizations choose from three implementation approaches to deliver Flogames to their employees. The first is as an open library implementation, where employees get unrestricted access to all the games. They can play them as and when they please. The second is as a part of an existing program or event. In this case, select games are launched, often to specific audiences, to drive a particular learning need or organizational agenda. The third is as an internal marketplace. Much like the open library, all the games are made available to the employees, some in the form of 45- to 60-min game bundles. However, in this case, employees must use their own money to purchase the game bundles that they are interested in and have it reimbursed by the organization later on. The marketplace may also have modules from other learning providers for the employees to choose from. Out of the 10,000 licenses sold, 6,632 licenses have been utilized so far. A license is considered utilized by a learner if they play at least one game. For the purpose of this study, only data from learners who have utilized their licenses has been considered.

3 Data Analysis The study uses a mixed methods approach to analyze the data of 6,632 unique learners. In certain cases, data from the marketplace implementation has been separately analyzed. Given the easy availability of other options in the marketplace and the need to make a conscious purchase decision, the authors felt that this data set could lead to interesting insights on both pull and overall effectiveness.

3.1 Analysis of Quantitative Data The following quantitative metrics have been analyzed, as means and percentages, to provide insights on the two research questions. Number of Games Played per Learner. The number of unique games a learner chose to play at least once. This metric has been analyzed for evidence of pull (RQ 1) and loyalty (RQ 2). Number of Games Replayed per Learner. The number of times a learner chose to replay a game that they had already played before. This metric has been analyzed for evidence of pull (RQ 1) and loyalty (RQ 2). Learning Gains. The difference in proficiency scores from one attempt to the next. This metric has been analyzed for evidence of effectiveness (RQ 2).

658

R. Datta et al.

Completion and Abandonment Rates. Completion rates refer to the percentage of game bundles completed, where a game bundle is a collection of games delivered together as a single 45–60 min journey. Abandonment rates refer to the percentage of game bundles left incomplete after 60 days. Both completion and abandonment rates have been analyzed for evidence of effectiveness (RQ 2). Feedback Ratings. These rating were provided by learners at the end of each game. Learners were asked to rate each game on a 5 point rating scale, where 1 meant very poor and 5 meant excellent. Providing ratings was optional and did not impact completion in any way. In total, 59,227 feedback ratings were received and analyzed for evidence of learner satisfaction (RQ 2).

3.2 Analysis of Qualitative Data Along with the feedback rating at the end of each game, learners were provided an open text box to share their thoughts on the game. This was optional and did not impact their completion in any way. The qualitative responses received were further analyzed, as follows: Step 1: Data Cleaning. All junk responses were removed from the data set. The 4,250 responses that remained were used for the analysis. Step 2: Creating Categories. Four categories were defined to better organize the data — positive, negative, useful and not relevant. Positive responses were those that indicated that learners had a positive experience, or those having positive keywords like ‘fun’, ‘engaging’, ‘amazing’, ‘wonderful’, ‘good’, ‘great’, or ‘fantastic’. Negative responses were those that indicated that learners had a negative experience, or those having negative keywords like ‘bad’, ‘terrible’, ‘boring’, ‘too easy’, or ‘too difficult’. Useful responses were those that indicated that learners found the content useful and applicable back at work, or those with keywords like ‘helpful’ or ‘useful’. Finally, responses that did not fall under any of these categories were categorized as ‘not relevant’ for the purpose of this study. Step 3: Data Tagging. Based on the definitions, each response was tagged to one or more categories. Tagging was done independently by each of the three authors, through a keyword search and manual reading of the responses. Differences in tags between authors were discussed and modifications made to arrive at a final list of tagged responses. Step 4: Using Tags as Evidence. The percentage of positive responses, as compared to negative responses, has been analyzed for evidence of learner satisfaction (RQ 2). The percentage of useful responses has been analyzed for evidence of perceived usefulness (RQ 2).

The Magic of Games: Creating a Pull-Based Learning System Through …

659

Qualitative feedback responses have also been used across the results section to substantiate the quantitative findings and provide insights on the learners’ experiences, in their own words.

4 Results This section has been broken into two parts one focusing on each of the research questions.

4.1 Flogames as a Pull-Based Learning System (RQ 1) This section explores quantitative data on number of games played, number of games replayed and learner purchase behavior. Number of Games Played per Learner. It was found that 72% of learners chose to play more than 1 game. The mean number of games played by a single learner was 10, with 4% playing over 65 games each. In total, 8% of learners played 10 games or more within a single week. With limited mandates on how many games to play, this pattern seems to suggest evidence of pull. Number of Games Replayed. It was found that 28.5% of learners chose to replay 1 or more games. The mean number of times learners chose to replay the games was 17. The mean number of times each game was replayed was 3. With no minimum score requirement or pass criteria, the choice to replay a game even after completion, appears to be a strong indicator of pull. Learner Purchase Behavior. Data from the internal marketplace implementation was analyzed to explore further evidence of pull. Based on 1,052 game bundle purchases and 304 unique learners, it was found that 61% of learners purchased more than 1 game bundle. The mean number of game bundles purchased per learner was 4. Additionally, it was found that 8.3% of the purchases were made on nonworking days (weekends and public holidays) and 26.7% of the purchases were made after work hours (after 6 PM). Learners choosing to purchase games in their personal time, in the absence of organizational pressure, may be indicative of the presence of pull. Qualitative Feedback. Qualitative feedback from learners provides additional evidence of pull. One learner said that they chose to replay a game to see whether they had learnt from the mistakes they made earlier. One said “Amazing interactivity. The concepts were explained so well through a story. It’s encouraged me to play more and more games like this.” Resonating with this thought, another said, “I loved this

660

R. Datta et al.

game and would love to explore more games like this. A very easy way to learn.” Yet another learner said, “I learn something different every time I play,” providing some insight as to why learners may keep coming back.

4.2 Flogames as an Effective E-Learning Tool (RQ 2) To understand the effectiveness of Flogames as an e-learning tool, findings from the qualitative feedback analysis has been presented, along with quantitative data on feedback ratings, completion rates and learning gains. Learner Satisfaction. Satisfaction is defined as the emotional response occurring after the e-learning activity [24]. Evidence of learner satisfaction was explored through mean feedback ratings and the percentage of qualitative feedback responses tagged as positive. A mean feedback rating of 4.6 on 5 was found, with 70% of learners rating the games 5 on 5. Additionally, 67.1% of the qualitative responses were tagged as positive, as compared to 17.1% tagged as negative. As one learner eloquently put it, “Brain wrecking dialogue trees and quite a situation to be in! Loved the feedback given at each stage. Got to learn a lot about myself along the way. Definitely looking forward to more such games.” Both the feedback ratings and the percentage of positive responses are suggestive of high levels of learner satisfaction. Loyalty. Loyalty is defined as the intention to reuse the e-learning service [5]. As the data required to make a case for loyalty is identical to that used to evaluate the extent of pull, the reader is requested to refer back to the previous section on Flogames as a Pull-Based Learning System for evidence on this metric. Perceived Usefulness. Perceived usefulness refers to the belief that a certain knowledge learned will bring about a future improvement in their job performance [16]. The percentage of qualitative feedback responses tagged as useful was used as evidence for this metric. It was found that 45.4% of the responses were tagged as useful. Speaking about one particular game, a learner said “Certain aspects of the challenge were similar to challenges we face in the office from time to time. It was relatable and a bit tough because every option felt like the right one! It definitely helped validate some of my existing leadership methods.” Another said, “These games help me learn skills that I will need for the next level.” Given this data, there appears to be evidence to indicate perceived usefulness of the games. Learning Gains. It was found that learners achieved a mean learning gain of 21% when replaying games.

The Magic of Games: Creating a Pull-Based Learning System Through …

661

Completion and Abandonment Rates. Analyzing the data from 1,052 game bundles, a completion rate of 92% was found, with most learners finishing the games in 20 days or less. An abandonment rate of 8% was found, where abandonment was defined as game bundles left incomplete for more than 60 days. The next section delves deeper into the relevance of the results in the context of each of the research questions.

5 Discussion In this paper, the authors set out to explore two things — whether there was evidence to suggest that game-based products like Flogames that combine serious games and gamification, could create a pull, and whether there was evidence to suggest that such products provided an effective e-learning experience. With regard to the first question of Flogames creating a pull-based system, the answer can be found in certain unique learner behaviors highlighted below. 1. Game Binging. The data showed that over 70% of learners chose to play more than 1 game, with several playing 10 games or more. Experiential evidence has shown that when an additional layer of gamification in the form of competitions was added, this number often increased to learners playing over 65 games in just two weeks. This behavior provides evidence to suggest that both games and gamification work together to create a pull. 2. Replays. While the number of games attempted could be attributed, in some cases, to passing instructions on how many games to play, there appears to be no explicit reason for learners replaying a game. There was no pass or fail criteria impacting completion or organizational instructions on trying to improve one’s scores. Replays appeared to be purely motivated by a drive to improve one’s score, explore different outcomes of the game, or get higher on the leaderboard. The authors, therefore, see the number of replays as strong indicator of games and gamification creating a pull-based system for learners. 3. Learner Purchase Behavior. The purchase of multiple game bundles by a single learner, in the presence of other alternatives, is in itself, a good indicator of pull. However, an even stronger case can be made when looking at the number of purchases made after working hours or on non-working days, as this implies that learners chose to spend their personal time, investing in professional learning activities. The second question of effectiveness draws its answer in part from the first, as learners would not choose to go back multiple times if they did not find it worth their time. Along with this, qualitative and quantitative feedback, completion rates significantly higher than industry averages and notable learning gains, further substantiate these findings.

662

R. Datta et al.

Findings in this study are also in line with earlier ones with regard to the benefits of games and gamification in learning. Similar to what Hussain et al. [15] found, there does seem to be evidence to suggest that learners are engaged, satisfied and committed in a gamified environment. The implications of these findings and possible next steps have been explored in the final section below.

6 Conclusion, Implications and Future Scope Ever since serious games have been used for training, there has been a lot of discussion around whether these games are “too fun” to be effective [11], even while traditional e-learning approaches saw little success. Luckily, the growing body of research in this area, including the findings from the present study, seem to paint a positive picture. While games are fun and engaging, it has been found that, if developed well, they also appear to provide an effective learning experience. The authors hope that these findings encourage more learning practitioners and organizations to explore the possibility of using serious games and gamification to deliver impactful learning experiences to employees. Given the evidence of pull, there may also be scope for learning practitioners to start using serious games in MOOCs, distance education and other direct-to-customers course offerings, typically known for their low completion rates. However, if used outside the context of organizations, further research may need to be done to better understand how games and gamification impact learner purchase behavior. In general, scope for future research includes evaluating the impact of games and gamification under more regulated conditions — a limitation of the present study. For example, while client organizations using Flogames did not have any explicit mandates to push the product onto learners, the authors recognize that implicit pushes and expectations may still exist. A controlled experiment in the absence of explicit or implicit pushes may help better understand the extent to which such game-based products are able to create pull. Additionally, while the authors drew insights and patterns from learner behavior like number of games played and replayed, using more formal, research validated tools for measurement of learner-centric metrics may be an interesting area for future research. Another area worth exploring may be the impact of games and gamification on enrolment rates. While data from the present study was able to show how many learners came back to the platform after the first use, it would be interesting to understand whether these approaches are able to create an initial pull and get more learners onto the platform in the first place. High enrolment rates in conjunction with high completion rates and overall effectiveness would make a much stronger case for using games and gamification in corporate training. Finally, experiential evidence seems to suggest that in-game elements like autonomy, challenge and curiosity, and gamification elements like competitions and leaderboards helped make Flogames an effective learning product. Extending this

The Magic of Games: Creating a Pull-Based Learning System Through …

663

further, comparative studies on the effect of different gamified and non-gamified learning approaches on learner-centric metrics like satisfaction, loyalty and perceived usefulness, and traditional metrics like completion rates and learning gains, would go a long way in helping learning practitioners find the recipe for success, as they work towards designing more engaging and impactful learning experiences.

References 1. Allal-Chérif, O., Lombardo, E., Jaotombo, F.: Serious games for managers: creating cognitive, financial, technological, social, and emotional value in in-service training. J. Bus. Res. 146, 166–175 (2022) 2. Allal-Chérif, O., Makhlouf, M.: Using serious games to manage knowledge: the SECI model perspective. J. Bus. Res. 69(5), 1539–1543 (2016) 3. Almeida, F., Simoes, P.: The role of serious games, gamification and industry 4.0 tools in the education 4.0 paradigm. Contemp. Educ. Technol. 10(2), 120–136 (2019) 4. Aydan, U., Yilmaz, M., Clarke, P., O’Connor, R.: Teaching ISO/IEC 12207 software lifecycle processes: a serious game approach. Comput. Stand. Interf. 54(1), 129–138 (2017) 5. Ayuni, D., Mulyana, A.: Applying service quality model as a determinant of success in Elearning: the role of institutional support and outcome value. Rev. Integr. Bus. Econ. Res. 8(1), 145–159 (2019) 6. Azadegan, A., Riedel, J.C.K.H., Baalsrud Hauge, J.: Serious Games Adoption in Corporate Training. In: Ma, M., Oliveira, M.F., Hauge, J.B., Duin, H., Thoben, KD. (eds.) Serious Games Development and Applications. SGDA 2012. LNCS, vol. 7528, pp. 74–85. Springer, Berlin (2012). https://doi.org/10.1007/978-3-642-33687-4_6 7. Barata, G., Gama, S., Gonçalves, D.: Studying student differentiation in gamified education: a long-term study. Comput. Hum. Behav. 71, 550–585 (2017) 8. Beck, J., Wade, M.: Got Game Shows How Growing Up Immersed in Video Games Has Profoundly Shaped the Attitudes and Abilities of This New Generation. Harvard Business Press, Boston (2004) 9. Cohard, P.: Evaluation of serious game user experience: the role of emotions. Electr. J. Inf. Sys. Eval. 22(2), 128–141 (2019) 10. Deterding, S., Sicart, M., Nacke, L., O’Hara, K., Dixon, D.: Gamification. Using gamedesign elements in non-gaming contexts. In: CHI’11 Extended Abstracts on Human Factors in Computing Systems, pp. 2425–2428. ACM (2011) 11. Donovan, L.: The use of serious games in the corporate sector (A State of the Art Report). Learnovate Center (2012) 12. Eisenberger, R., Rhoades, L., Cameron, J.: Does pay for performance increase or decrease self-determination and intrinsic motivation? J. Pers. Soc. Psychol. 77, 1026–1040 (1999) 13. Gabelaia, I., Bucovetchi, O.: The relevance of corporate E-learning/Etraining for job development: crafting culture and evolving yourself [Conference paper]. In: The 16th International Scientific Conference eLearning and Software for Education. Bucharest, Romania (2020) 14. Hollands, F., Kazi, A.: Benefits and Costs of MOOC Based Alternative Credentials. Columbia University, Teachers College (2018) 15. Hussain, S., Qazi, S., Ahmed, R., Streimikiene, D., Vveinhardt, J.: Employees management: evidence from gamification techniques. Monteneg. J. Econ. 14(4), 97–107 (2018) 16. Joo, Y.J., So, H.J., Kim, N.H.: Examination of relationships among students’ self-determination, technology acceptance, satisfaction, and continuance intention to use K-MOOCs. Comput. Educ. 122, 260–272 (2018) 17. Jun, J.: Understanding E-dropout? Int. J. E-Learn. 4(2), 229–240 (2005)

664

R. Datta et al.

18. Larson, K.: Serious games and gamification in the corporate training environment: a literature review. Tech Trends 64, 319–328 (2020) 19. Lepper, M.R.: Motivational considerations in the study of instruction. Cogn. Instr. 5, 289–310 (1988) 20. Przybylski, A.K., Rigby, C.S., Ryan, R.M.: A motivational model of video game engagement. Rev. Gen. Psychol. 14(2), 154 (2010) 21. Reich, J., Ruipérez-Valiente, A.J.: The MOOC pivot. Science 363, 130–131 (2019) 22. Society for Human Resource Management. 2022 Workplace Learning and Development Trends Report, https://www.shrm.org/hr-today/trends-and-forecasting/research-and-surveys/ Documents/2022%20Workplace%20Learning%20and%20Development%20Trends%20R eport.pdf. Accessed 13 Mar 2023 23. Vargas Souza, F., Motoki, F.Y.S., Mainardes, E.W., Azzari, V.: Public corporate e-learning: antecedents and results. Public Organ. Rev. 22(4), 1139–1156 (2022) 24. Wang, Y.S.: Assessment of learner satisfaction with asynchronous electronic learning systems. Inf. Manag. 41(1), 75–86 (2003) 25. Zyda, M.: From visual simulation to virtual reality to games. Computer 38(9), 25–32 (2005)

Predicting Students’ Academic Success Based on Various Course Activities: A Case Study ˇ c Poturi´c , Sanja Candrli´ ˇ Vanja Coti´ c , and Ivan Draži´c

Abstract In all areas of learning, teachers strive to improve the educational process. In the field of information and software engineering, activities usually combine theory and practical training. By using various assessment activities during the course, students’ motivation can be improved and with proper feedback from the teacher, students can manage their learning process to achieve better results. The research presented in this paper was conducted in the course “Business Process Analysis,” which is part of the undergraduate computer science curriculum. The course was designed with two quizzes, two project assignments, and two midterm exams followed by the final exam. The course requires a project-based approach and completion of practical assignments using theory. Our dataset contains outcome data from 93 students who took the course in the 2021/2022 academic year. Our research question was: can we predict students’ academic success on the final exam based solely on the activities completed during the semester—i.e., quizzes and project activities as a combination of theoretical and practical knowledge. Logistic Regression, a statistical model commonly used for classification and predictive analysis estimates the probability of an event occurring based on a given data set of independent variables. It is used to predict a binary outcome (0 or 1) based on a set of independent variables. In this paper, we use it to predict whether a student will pass or fail the course, and we use the scores obtained in two quizzes and two project assignments as independent variables. The results have shown that students’ success in these activities can predict their overall success, with accuracy of 0.9474. We can conclude that some of the activities used in the course are good indicators of students’ final success—i.e., consistent with the final goal of the learning outcomes. Accordingly, we can use this method to assess whether we use appropriate activities in different topics. With this method, we can evaluate and then adjust the teaching process. ˇ Poturi´c · S. Candrli´ ˇ V. C. c (B) Faculty of Informatics and Digital Technologies, University of Rijeka, 51000 Rijeka, Croatia e-mail: [email protected] ˇ Poturi´c · I. Draži´c V. C. Faculty of Engineering, University of Rijeka, 51000 Rijeka, Croatia © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_54

665

666

ˇ Poturi´c et al. V. C.

Keywords Prediction of Students’ Success · Logistic Regression · Indicators

1 Introduction In the third semester, students of the Faculty of Informatics and Digital Technologies at the University of Rijeka listen to the course Business Process Analysis. The objective of the course is to teach students how to independently perform analysis, interview users, collect user requirements and design process models, as well as to develop the designer’s mindset, with a high level of critical attitude towards the results of the analysis and models obtained. The course requires a project-based approach and completion of practical assignments using theory. The course assessment consists of two quizzes (5 points each), the first project work (12 points, no threshold), the second project work (8 points, no threshold), two midterm exams (20 points each, threshold 50% for each), and the final exam (30 points, threshold 50%). On the midterm exam, students solve both closed-ended questions (multiple choice questions, yes/no questions) and open-ended questions (short answer and essays). Students who earn at least 35 points during the semester may take the final exam. In the final exam, students design process models based on a given business description. Final exam points are added to those that the student collected during the semester. If the total is more than 50 (out of a possible total of 100 points), the student has passed the course. Our research question is: Can we predict students’ academic success in the final exam based solely on the activities completed during the semester—i.e., quizzes and project activities as a combination of theoretical and practical knowledge. Prediction is one of the common tasks in the fields of Learning Analytics (LA) and Educational Data Mining (EDM) [1]. LA and EDM both aim to improve the teaching and learning process by improving the way assessments are conducted, identifying educational problems, and developing effective interventions [2]. They use comparable methods and techniques, such as classification, grouping, regression, and visualization. In this paper, we use Logistic Regression classifier to predict whether a student will pass or fail this course. Logistic Regression is a statistical model commonly used for classification and predictive analysis that estimates the probability of an event occurring based on a given data set of independent variables. It is used to predict a binary outcome (0 or 1) based on a set of independent variables. As independent variables we use the points obtained in two quizzes and two project tasks—i.e., activities in which it is possible to obtain 30 points out of a total of 70 points during the semester. The course lasts 15 weeks, and assessment activities during the semester are distributed as follows: first quiz in the 5th week, first midterm in the 7th week, first project assignment in the 8th week, second quiz in the 10th week, second project assignment in the 11th week, and second midterm in the 14th week. If the selected

Predicting Students’ Academic Success Based on Various Course …

667

input variables are good predictors, there is sufficient time in the 12th and 13th weeks for interventions to ensure that at-risk students do well on the second midterm. In the following, we first review the previous research literature on the use of Linear Regression to predict students at risk of failing the course in Sect. 2. Then, Sect. 3 presents the methodology for this case study, which is based on predictive modeling. Section 4 discusses the results of the predictive modelling considering the defined research question. Finally, Sect. 5 concludes this paper and provides an outlook for future work.

2 Related Studies Exploring Prediction of At Risk Students Using Logistic Regression The main purpose of predicting students’ academic performance is to identify those at risk of falling behind and provide them with tailored support before it is too late. Some studies have focused primarily on predicting students’ final grades after they complete their courses, which does not allow enough time for timely interventions, while others have attempted to predict students’ outcomes earlier in their academic careers to provide timely and effective support. In our previous research [3], we presented a method for identifying predictive factors of student failure in STEM oriented course using Logistics Regression achieving an accuracy of 0.8571. We discovered that several predictors of passing the course were significant, including previous course success, regular participation in weekly self-tests, downloading course materials, and watching video lessons. Based on odds ratios, prior course success had the strongest positive effect on passing the course, followed by regular work such as completing weekly tests. As for possible pedagogical interventions, the most important factors are tests, lectures, and instruction since we can influence them during class. Regular monitoring of these three variables could allow for personalized pedagogical interventions that are likely to increase the success rate of the course. The research paper titled “On Developing Generic Models for Predicting Student Outcomes in Educational Data Mining” [4] shows the development of a generic predictive model to identify students at risk of falling behind in different courses. The study involved conducting experiments with different algorithms, and the results showed that the generic model has a high level of accuracy. The researchers found that the CatBoost algorithm had the highest accuracy of 75% ± 2.1%, while Logistic Regression achieved an accuracy of 67% ± 3%. The authors used the Shapely Additive Explanations method (SHAP) to evaluate the significance of features and model behavior. They found that both current and past assignment scores were critical predictors in their model, which is not surprising. In the paper by Verma et al. [5], five individual supervised machine learning techniques, including Logistic Regression, were used for early prediction of student

668

ˇ Poturi´c et al. V. C.

academic achievement. To analyze the effects of an imbalanced data set, the performance of the algorithms was tested with and without different resampling methods. The results of the study show that Logistic Regression performs best with any balanced data set generated with all four resampling methods and achieves the highest accuracy of 94.54% with the Synthetic Minority Oversampling Technique (SMOTE). In the paper “Comparing Different Resampling Methods in Predicting Students’ Performance Using Machine Learning Techniques” [6], the authors conducted a study using several machine learning classifiers, including Logistic Regression. They used a dataset with information about 650 students and 19 different attributes that could potentially influence their academic performance. Based on the results of various evaluation metrics, the authors found that the models with fewer classes and nominal attributes performed better. Specifically, Logistic Regression achieved the highest accuracy of 77.69% among all classifiers for binary classification. Ramaswami and colleagues conducted a study [7] to determine the earliest possible time in a course at which it is possible to identify students at risk of failing. The researchers used a combination of data from learning management systems, demographic data, and grades on assignments. They tested four different classification algorithms, including Naïve Bayes, Random Forest, Logistic Regression, and k-Nearest Neighbors. The study included two experiments, one using all available features and the other using only the features that had the highest accuracy in predicting risk. The results showed that Logistic Regression achieved the best accuracy of 83% at week 11. The authors recommended that feature selection methods should be used rather than selecting all features for prediction because selecting all features may lead to overfitting. Compared with the aforementioned related studies, we use a very small number of predictors in this study. It is important to emphasize that we have achieved the same or better accuracy with a small number of predictors than some other works with a larger number of predictors. It is also important to mention that learning analytics and prediction of academic success are still in their infancy in Croatia and are rarely or never used in university teaching. With this study, we show the possibility of implementing LA in university courses in countries with a similar educational system as in Croatia.

3 Methodology Machine learning is an active and exciting field of computer science that encompasses three primary areas: supervised, unsupervised, and reinforcement learning. Supervised learning involves using input data to predict output variable values, with the model being developed from training data where both input and output variable values are known. The model then generalizes the relationship between input and output variables to predict output values for other data sets with only input data. The two primary supervised learning models are classification, where the output variable has discrete values, and regression, where the output variable has continuous values.

Predicting Students’ Academic Success Based on Various Course …

669

To answer the research question, we use a supervised machine learning classification model and a predictive modeling approach, which is described in more detail below.

3.1 Predictive Modeling Predictive modeling has become a core practice of researchers in the fields of Learning Analytics and Educational Data Mining, with a significant emphasis on predicting student success [8]. This approach operates under the belief that by analyzing a specific set of existing data, it is possible to make accurate predictions about the value or outcome of new data set. It involves several key stages, including identifying the problem to be solved, gathering the relevant data, selecting the most important features, building a model, and evaluating the model’s performance. Problem Identification. To effectively apply data analysis techniques, it is important to choose a problem that is likely to recur in the future, has measurable characteristics, has a clearly defined outcome, and has the potential for intervention. We have faced the problem of identifying those students who are at risk of failing the Business Process Analysis course. Authors will continuously monitor this course in the following years, and they intend to take appropriate pedagogical measures to prevent the failure of this course, having identified the students at risk early on. Data Collection. The researcher must determine both the output variable (such as the final grade or level of achievement) and the potential input variables. Our dataset contains outcome data from 93 students who took the Business Process Analysis course in the 2021/2022 academic year. Sixty-four students passed the course, meaning they earned a total of 50 or more assessment points out of a possible 100. These students were assigned a value of 1 (passed) for the out-come variable, while the other 29 students were assigned a value of 0 (failed). Possible input variables (predictors) include the number of points earned on two quizzes, two project assignments, and two midterm exams during the semester. Feature Selection. To develop and use a predictive model, it’s important to choose input variables (predictor variables) that are correlated with the output variable. Quizzes are used within the course as a self-assessment tool for students to take prior to theory exams to assess their readiness for the exams. The questions used in the quizzes are mostly of the closed-ended type. By solving them, students master the material covered in several weeks of theory classes. The results are given to the students immediately after participating in this activity. After solving the quizzes, students have enough time to prepare for the midterm exams according to their results. Project assignments are used to check the level of implementation of practical methods. These assignments are simple and help students achieve the practical outcomes of the course—i.e., they prepare them to successfully pass the final exam.

ˇ Poturi´c et al. V. C.

670

While the midterm exams test theoretical knowledge, the final exam tests deep understanding of practical knowledge. The feedback from these assignments helps students to better prepare for the final exam. Although the quizzes and practical assignments chosen as input variables are much simpler and only partially cover the subject matter, the authors of the paper believe that they are a necessary foundation for full adoption of the course outcomes and a basis for further learning. The summary statistics of chosen input variables is listed in Table 1. The data were then standardized to ensure that all features contribute equally to the model. Model Building. After collecting a dataset and performing attribute selection, a predictive model can be built. A number of different algorithms exist for building predictive models. White-box machine learning algorithms, unlike black-box algorithms, provide not only a result but also clearly readable rules. We tried several white-box classification algorithms, Decision Tree, Logistic Regression, and k-Nearest Neighbors, and selected the one that gave the best results. For our data set we used Logistic Regression model and package Caret in software environment R [9]. Logistic Regression is a statistical model commonly used for classification and predictive analysis that estimates the probability of an event occurring based on a given data set of independent variables. It is used to predict a binary outcome (0 or 1) based on a set of independent variables. Model Evaluation. To assess the accuracy of a predictive model, it is necessary to have a separate dataset with known labels (also called the test set) that the model has not seen before. The model’s predictions on this test set can then be compared to the actual true labels to evaluate the model’s performance. Our dataset was split 80:20 and 19 instances were used as a test set for model evaluation. We used tenfold repeated five times cross-validation resampling method to optimize the bias variance trade-off. The confusion matrix for our model is shown in Table 2. There are 18 correctly classified instances and one misclassified instance. The Logistic Regression model incorrectly predicted for one student that he would fail Table 1 Summary statistics of input variables Mean

1st quiz

2nd quiz

1st project task

2nd project task

3,61

3,47

7,83

3,93

Standard Error

0,11

0,16

0,44

0,32

Median

3,75

3,95

9

5

Mode

4,63

0

0

0

Standard Dev

1,02

1,57

4,26

3,06

Minimum

0

0

0

0

Maximum

5

5

12

8

Predicting Students’ Academic Success Based on Various Course … Table 2 Confusion matrix for Logistic Regression model

671

Predicted 0

1

0

6

0

1

1

12

Actual

the course—i.e., that he would score a total of less than 50 out of a possible 100 points in the course and assigned him a value of 0 instead of the actual value, which is 1.

4 Findings and Discussion To answer the research question, we used a predictive modeling approach with Logistic Regression classifier. Evaluation measures for comparing true labels and predicted labels are shown in Table 3. Accuracy (correct predictions over total predictions) of our model is 0.9474, Sensitivity or Recall (the true positive rate) is 0.9231, and Specificity (the true negative rate) is 1.0000. The classifier has high Accuracy, high Specificity and high Sensitivity meaning that it can accurately predict both negative and positive class. To further confirm this statement, we plot the Receiver Operating Characteristic (ROC) Curve. It compares the false positive rate to the true positive rate. For our model area under the curve (AUC) is 0.962 (see Fig. 1). The higher the AUC is, the more accurately model can predict outcomes. Table 3 Evaluation measures for comparing true labels and predicted labels

Fig. 1 ROC Curve for Logistic Regression model

Accuracy

Specificity

Sensitivity/Recall

0.9474

1.0000

0.9231

ˇ Poturi´c et al. V. C.

672

Thus, we conclude that we can accurately predict students’ passing on the final exam based solely on the activities completed during the semester—i.e., the quizzes and project activities as a combination of theoretical and practical knowledge. In addition, we can take a closer look at our model. The coefficients for the Logistic Regression model are shown in Table 4. Looking at the z-statistics and the associated p-values, we find that only the coefficient of variable 2nd quiz is not significant. If we want to obtain the odds, we must exponentiate the coefficients (Table 5). For each point increase in 1st project task (one point more of total of 12 points), the odds for passing Business Process Analysis course increase by a factor of 1.48. For each point increase in 2nd project task (maximum is 8 points), the odds for passing the course increase by a factor of 1.54. For each point increase in 1st quiz (maximum is 5 points), the odds for passing the course increase by a factor of 4.30. Finally, the importance of the variables for our model is shown in Fig. 2. The most important variable is the number of points for the first project task, followed by the number of points for the first quiz and the second project task. In summary, compared to our previous study [3] where the accuracy was 0.8571, this model has an even higher accuracy of 0.9474, which is almost the same result Table 4 Coefficients for Logistic Regression model Variable 1st

Coefficient

Std. Error

z value

Pr(>|z|)

quiz

1.4588

0.4906

2.974

0.002941

2nd quiz

−0.1546

0.2656

−0.582

0.560477

1st project task

0.3920

0.1216

3.224

0.001265

2nd project task

0.4335

0.1884

2.301

0.021388

−7.7728

2.3500

−3.308

0.000941

Intercept

Table 5 Odds ratios for Logistic Regression model Odds ratios

Fig. 2 Variable importance

1st project task

2nd project task

1st quiz

1.48

1.54

4.30

Predicting Students’ Academic Success Based on Various Course …

673

as the work of Verma et al. [5]. Unlike a previous study [4], where the authors experimented with different algorithms, we conducted our study using only one model. Although our study was conducted with a much smaller sample of 93 students, the results of the Logistic Regression model are better than in a previous study [6], even though the authors of that study used a larger sample of 650 students and a larger number of various classifiers for binary classification. Like the authors of another paper [7], we have shown that it is possible to predict students at risk of failing a course early enough in the semester so that timely action can be taken to prevent this from happening. It should be emphasized that in education, particularly in higher education, there is no one-size-fits-all method for making predictions. The approach to prediction largely depends on the specific variable being predicted and the data available to make the prediction. This means that the choice of variables and modeling techniques used will vary based on the unique circumstances of each prediction task.

5 Conclusion The potential of predictive modeling in predicting student academic performance is immense and invaluable. With its help, educators can identify those who are slipping and provide them with the support they need to ensure their success. In addition, teachers can tailor their instruction to meet the specific needs of individual students. Early warning systems based on machine learning algorithms can be used to identify students at risk of dropping out and initiate preventive measures. Finally, predictions of academic success from predictive modeling can help schools and education policymakers make wise decisions when allocating resources to maximize educational outcomes. Our Logistic Regression classifier proved highly successful, with an amazing 92.31% sensitivity and excellent specificity. This allowed us to identify students who were at risk of failing the course before the final exam and to intervene with specific pedagogical and didactic strategies. Our analysis also uncovered which factors are most important in predicting whether a student will successfully complete the course; the first project task proved to be the most influential factor. Consequently, we can now take targeted pedagogical and instructional measures to significantly increase the pass rate. It is worth noting that the results obtained should also be considered at the methodological level, as a clear, reliable, and straightforward methodology has been developed for predicting the completion of any course, regardless of whether it is offered in a conventional, hybrid, or online environment. Continuation of this research should incorporate a number of potential variables from the range of learning analytics available in the e-learning system. It is important to focus on those variables that allow earlier and more accurate detection of students at risk of failing the course as soon as possible.

674

ˇ Poturi´c et al. V. C.

The immediate priority is to develop a more general predictive model that can be applied to a larger number of courses. To this end, further research needs to be conducted, particularly to increase the sample size for both the total number of students and the number of courses studied. It is also important to analyze additional factors that could have an impact on students’ academic success. These additional factors could include pre-course scores on other tests, demographic data, or student characteristics (e.g., their prior academic performance). Finally, it is important to include factors that can better identify the characteristics and needs of individual students (e.g., their prior academic performance). As school systems develop and implement courses that are offered primarily or entirely online, the number of students taking these courses will increase dramatically. Educational institutions need to be equipped with adequate tools to identify students who need extra help before it is too late. This research may be particularly useful for institutions that are revising their course catalogs and moving from a traditional college curriculum to a more flexible and open approach that provides more options for hybrid learning. In summary, predictive modeling is an invaluable tool for improving the academic experience of students. It can identify those who need extra help, tailor learning to the student, set up early warning systems, and enable data-driven decisions for education leaders. Therefore, predictive modeling is essential for educators and policymakers to improve the quality of education and increase student success. Acknowledgements This work has been fully supported by the University of Rijeka under the project number uniri-drustv-18-140.

References 1. Romero, C., Ventura, S.: Data mining in education. Wiley Interdisc. Rev. Data Min. Knowl. Discov. 3, 12–27 (2013) 2. Siemens, G., Baker, R.: Learning analytics and educational data mining: towards communication and collaboration. In: ACM International Conference Proceeding Series, pp. 252–254. Association for Computing Machinery, New York (2012) ˇ c Poturi´c, V., Draži´c, I., Candrli´ ˇ 3. Coti´ c, S.: Identification of predictive factors for student failure in STEM oriented course. In: ICERI2022 Proceedings, pp. 5831–5837 (2022) 4. Ramaswami, G., Susnjak, T., Mathrani, A.: On developing generic models for predicting student outcomes in educational data mining. Big Data Cogn. Comput. 6, 6 (2022) 5. Verma, S., Yadav, R., Kholiya, K.: A scalable machine learning-based ensemble approach to enhance the prediction accuracy for identifying students at risk. Int. J. Adv. Comput. Sci. Appl. 13, 185–192 (2022) 6. Ghorbani, R., Ghousi, R.: Comparing different resampling methods in predicting students’ performance using machine learning techniques. IEEE Access 8, 67899–67911 (2020) 7. Ramaswami, G.S., Susnjak, T., Mathrani, A., Umer, R.: Predicting students final academic performance using feature selection approaches. In: Proceedings of the 2020 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE), Gold Coast, Australia (2020)

Predicting Students’ Academic Success Based on Various Course …

675

8. Brooks, C., Thompson, C.: Predictive modelling in teaching and learning. In: Lang, C., Siemens, G., Friend Wise, A., Gaševi´c, D., Merceron, A. (eds.) Handbook of Learning Analytics, 2nd edn., pp. 29–37. SoLAR (2022) 9. The Caret Package. https://topepo.github.io/caret/. Accessed 05 Mar 2023

Augmented Reality in STEM Education: Mapping Out the Future Sarantos Psycharis , Konstantina Sdravopoulou , and Evi Botsari

Abstract While several review papers have proven the applicability of augmented reality (AR) in various domains of education, we are still short of an in-depth quantitative and qualitative assessment of the usefulness of AR in STEM education. To address this problem, this paper presents the results of a meta-analysis of 42 peerreviewed papers in which AR has been implemented in STEM education. Cohen’s d measure of effect was calculated from the reported statistics, and the content of these papers was analyzed. It was thus shown that: a) AR is useful in STEM education regardless of the number or age of participants (although its effectiveness tends to diminish with age in mathematics education; b) it is considerably more effective in the science and mathematics components of STEM education rather than technology and engineering. It was also found that, although some negative aspects have been expressed, student’s attitudes were quite positive about it, as were the reported advantages it was shown to have in the entire range of STEM education. Keywords AR in education · STEM · AR in STEM · Cohen’s d · Content analysis

1 Introduction and Related Works Augmented reality (AR) has been implemented in various educational settings and for a large variety of educational aims, with STEM education prominent among them. The usefulness of AR in STEM education has been documented on several occasions; in formal educational settings (in which most of the studies have been carried out), outdoors—i.e. in field trips [1]—as well as in temporary installations in out of school educational settings [2]. It may aim at fostering the students’ conceptual understanding [3] as well as problem-solving [4]. AR can be used in the laboratory or in the classroom as well as by each student separately or in collaborative learning processes [5, 6]. Interestingly, the positive impact of AR in STEM education is not S. Psycharis · K. Sdravopoulou (B) · E. Botsari School of Pedagogical and Technological Education (ASPETE), Athens, Greece e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_55

677

678

S. Psycharis et al.

restricted to outcomes related to learning only, for it extends to changing students’ attitudes towards science also [4, 7]. However, it is intriguing to consider that different systematic reviews of the literature of STEM in education yield different results. For instance, a systematic review [3] has stressed the usefulness in knowledge discovery through exploration and simulation and suggested that future research may focus on inquiry-based activities. Another systematic review [8] found that motivation and learning gains were the prominent advantages reported from the literature. Hence, the findings in these reviews prompt us to carry out content analyses of what the authors report (quoting their expressions and characterizations verbatim) as main advantages (or even disadvantages) of AR in STEM education. Furthermore, while some authors have reported the educational effect (as measured by Cohen’s measure of effect d), many other authors have not. Consequently, there are two outstanding problems related to the use of AR in STEM education, which this study aims to tackle: a) how to measure (quantitatively) the effectiveness of AR in STEM education from as many research results that have been carried out around the world and have been published in peer-reviewed papers, and b) how to qualitatively assess its advantages, negative aspects (if any) and students’ attitudes towards it from those research results.

2 Methods To identify the impact of AR on students’ learning effectiveness, the value of Cohen’s d was calculated as an indication of the effect of the use of AR in STEM education, for each one component of STEM (science, technology, engineering, mathematics). The calculation was performed on the basis of the values (mean ± s.d.) of pre- and post-AR intervention, as these values were reported for the experimental groups in each paper. In cases where several mean scores and standard deviations were reported in the same study, they were averaged, and the averages were used to calculate the mean d (the mean effect size). This is an indicated practice when several d values are reported from within the same research project [9]. Table 1 shows the studies considered in this meta-analysis. Furthermore, the papers were analyzed as per their content, in order to identify characteristic expressions towards AR in STEM education.

3 Results 3.1 Values of Cohen’s d Per Component of STEM Education The values of Cohen’s d for each one of the topics of the “S” (Science) and “T” (Technology) components of STEM) are given in Fig. 1 and for the components “E” (Engineering) and “M” (Mathematics) in Fig. 2.

Augmented Reality in STEM Education: Mapping Out the Future

679

Table 1 The studies considered in this meta-analysis Author/s

Field

Sample size

Akçayır et al. [7]

Ph.

76 first-year university students, aged 18–20

Cascales-Martínez et al. [10]

M.

22 students with Special Education Needs aged 6–12

Chen and Wang [11]

E.S.

144 7th-grade students from six classes in 2 junior high schools

Gutiérrez deRavé et al. [12]

M.

50 full-time mechanical engineering students (aged 18–20 years)

Yoon et al. [13]

Ph.

58 students 11 years old to 14 years old (41% male, 59% female)

Sommerauer and Müller [14]

M.

101 participants museum visitors aged 14–79

Fonseca et al. [5]

En.

57 students mean age = 19.45 years

Coimbra et al. [15]

M.

students of the Polytechnic Higher Education of Leiria (Portugal)

Bursztyn et al. [1]

E.S.

Nearly college 1000 students

Enyedy et al. [16]

Ph.

43 students aged 6–8 years

Martín-Gutiérrez et al. [17]

M.

First year mechanical engineering students 49 participants

Dünser et al. [18]

Ph.

Ten secondary school students, aged 13 to 15 years

Estapa and Nadolny [19]

M.

61 students participated in the study, with 56% female students

Salmi et al. [2]

T.

146 pupils average 12.3 years old

Wang et al. [6]

Ph.

A total of 40 university students,

Barrow et al. [20]

L.S.

90 students (out of a class of 150), 95% were 19–24 years old

Guntur et al. [21]

M.

75 Mathematics teachers

Sung et al. [22]

T.

10 students

Bazarov et al. [23]

En.

24 engineering students of electrical and technological specialties

Martin-Gutierrez et al. [24]

En.

first year electrical engineering students (49 participants)

Criollo-C et al. [25]

En.

80 participants: 60 aged 18–19 old and 20 aged 20–21

Vaughan et al. [26]

E.S.

45, from freshmen-level through senior-level

Demitriadou et al. [27]

M.

Thirty primary education pupils aged 9 to 11 years

Suprapto et al. [28]

Ph.

34 10th grade students

Abdul Hanid et al. [29]

M.

124 Secondary students

Fidan and Tuncel. [4]

Ph.

91 seventh graders students (continued)

680

S. Psycharis et al.

Table 1 (continued) Author/s

Field

Sample size

Volioti et al. [30]

Ph.

314: teachers (15), pupils (189), computer science students (110)

Kaur et al. [31]

En.

34 undergraduate students from and electrical engineering

Aldalalah et al. [32]

M.

86 school students

Schutera et al. [33]

M.

15 twelfth-graders (16–17 years old)

Erbas and Demirer [34]

L.S.

40 (22 female and 18 male) ninth-grade biology course students

Kularbphettong and Limphoemsuk [35]

Ph.

Experimental group 29, control group 30, primary school level

Gün and Atasoy [36]

M.

88 students (spatial ability), 81 academic achievement measures)

Arifuddin et al. [37]

M.

49 5th-grade students in elementary schools

Lubis et al. [38]

M.

60 students of fifth grade elementary school

Lozada-Yánez et al. [39]

M

29 third-grade children (13 girls and 16 boys)

Thornton and Lammi [40]

En.

50 university students were enrolled

Ismail et al. [41]

T.

30 high school students, 15–16 years

Sommerauer and Müller [42]

M.

26 pupils (K-20) and their mathematics teacher

Cai et al. [43]

T.

8th-grade: experimental group (24 students), control group (26)

Cai et al. [44]

Ph.

98 high school students aged between 16 and 18

Cr˘aciun and Bunoiu [45]

Ph.

22 pre-service science teachers, between the years 2014 and 2016

Ph. = Physics, M. = Mathematics, E.S. = Earth Sciences, En. = Engineering, L.S. = Life Sciences, T = Technology

Fig. 1 Values of Cohen’s d for each topic of the components “S” (Science) and “T” (Technology) of STEM education

Augmented Reality in STEM Education: Mapping Out the Future

681

Fig. 2 Values of Cohen’s d for each topic of the components “E” (Engineering) and “M” (Mathematics) of STEM education

Table 2 Mean d, s.d., confidence level and confidence interval for all the components of STEM education, indicating the effect of the use of AR in STEM education Mean d

s.d.

Confidence level

Confidence interval

S

1.598

1.755

95%

0.4513 to 2.745

T

1.491

1.513

95%

0.0086 to 2.973

E

0.670

0.693

95%

−0.2904 to 1.630

M

2.991

2.209

95%

1.4600 to 4.522

The mean effect calculated from the studies belonging to the category of Science was d = 1.598 with a 95% confidence interval from 0.45 to 2.745. For the category of Technology, the mean effect was d = 1.491 with 95% confidence interval from 0.0086 to 2.973; for the category of Engineering, the mean effect was d = 0.67 with 95% confidence interval of −0.2904 to 1.630; and for the category of Mathematics, the mean effect was d = 2.991 with 95% confidence interval from 1.4600 to 4.522. All these values correspond to a large effect as indicated by Cohen’s d, which supposes that AR has a positive impact on learning gains for all the categories (Table 2), given that any d value larger than 0.6 is considered a large effect.

3.2 Statistical Correlations of Values of Cohen’s d and Student Age No statistically significant correlation was found between the students’ mean age and the average Cohen’s d (Pearson’s R = −0.348, P = 0.1037, which is not significant at p < 0.10). This means that the positive effect of AR in STEM education (as documented by the values of Cohen’s d) is independent of age. Also insignificant (R = −0.25, P = 0.2499, which is not significant at p < 0.10) is the correlation between Cohen’s d values and number of participants in each

682

S. Psycharis et al.

experiment, therefore indicating that the positive impact of AR in STEM education is also independent of the number of participants. However, after examining the correlations between student age and Cohen’s d for each component independently, no correlation was found for any component with the exception of M (mathematics), in which case there is a moderately negative correlation between the two (R = −0.679, P = 0.64, which is significant at p < 0.10 but not at p < 0.05). This indicates that AR is probably more effective in educating younger students of mathematics rather than elder ones; the relationship between age and Cohen’s d can be approximated (with R2 = 0.83) by the relationship d = 44.0955e−0.233058t (where t = students’ age).

3.3 Results of the Qualitative Assessment of Using AR in STEM Qualitative Description of the Advantages of Using AR in STEM. A summary of the advantages of AR in STEM is shown in Fig. 3. It is important to clarify that these are only some of the advantages that have been more commonly reported in the studies, whereas most studies reported more than one advantage. Specifically, the reported positive impacts of AR in STEM education are manifold and in the following domains:

Fig. 3 An overview of the main advantages of AR in STEM education, as reported in the 44 selected studies

Augmented Reality in STEM Education: Mapping Out the Future

683

i) Visualization has been reported as one of the domains that AR improves. A study [13] presented the results in which it was demonstrated that students “in the AR condition” demonstrated significantly greater gains in knowledge over students in the non-AR condition. Through interview responses, it was shown that the use of AR affords greater ability to visualize details and hidden information to help students learn science courses. AR provides contextual visualizations as well as vision-haptic visualizations that favor engagement and interaction in the learning process. [5, 15, 29, 40]. Likewise, a study [32] conducted et al. Manar School in Abu Dhabi by seventh grade primary students presented a quasi-experimental type of methodology indicating that AR was effective in promoting visual thinking skills. ii) Spatial ability was also signalled as a major field in which AR has positive impact. A research [36] expressed the opinion that “it is known that spatial ability is important and can be improved for solving problems experienced in mathematics learning”. Another [12] indicated that a mobile Augmented Reality system, called DiedricAR improved students’ spatial ability. iii) Creating more realistic representations is another important domain in which AR has a positive impact. AR creates possibilities of providing more realistic representations which will be helpful for education [22]. Promoting learning performance is the second most common advantage reported in the selected studies when using AR: i) AR has been associated with higher learning achievement [4, 11]. ii) AR promotes learning mathematical knowledge: characteristically, a study conducted in primary education in a public school in Alicante (Spain) [10] demonstrated through a tabletop system that AR is a feasible technology that can be successfully applied in special educational needs contexts. iii) Another research [12] presented “DiedricAR” which offers the advantage of being specifically developed for mobile devices giving the students the possibility of using an ubiquitous learning technology. iv) Furthermore, the implementation of AR in mathematics education improved both student interest and interactivity, contributing to more efficient learning [27]. v) It also improved “higher learner perception” and cognitive skills and selfefficiency in learning [44]. Improvement of students’ ability is yet another common advantage reported in the selected studies: Studies stated that, when using AR systems, students improved their academic performance in different domains such as i) ii) iii) iv) v)

laboratory skills [7], money dealing skills [10] computational thinking [29], interactivity [27] mathematical creative thinking skills [37] and reducing students’ mathematical anxiety [38].

684

S. Psycharis et al.

In general, studies mentioned that students guided through AR obtained better scores than those who were guided through traditional approaches. Improving attitudes is another important advantage described in the selected studies. The combination of real and virtual worlds increases the satisfaction of students and the willingness of their teachers to adopt such technologies since the proposed system provides a variety of benefits, resulting in the improvement of the educational process. [7, 30, 45]. Negative Aspects of Using AR in STEM. Certain of the selected studies reported some disadvantages or problems when using AR in STEM education. Figure 4 summarizes the main disadvantages reported in the selected studies. Some teachers that participated were not accustomed to using supporting software to develop AR, especially for technical problems that required precision and accuracy [21]. Another reported issue related to AR was that positive effects vanished in the long run [14]. Another reported disadvantage was that though AR is an effective ICT tool that most students would like to learn with it, there was no significant effect of either learning styles or ICT competence on learning achievement [11]. Occasionally, the possibility that AR applications may lead to physical disorders among some of the students should not be precluded [4]. Also, AR simulations provide only a limited learning environment [22]. Finally, the difficulty of generating content increases the complexity of the application of this technology for educational purposes [5].

Fig. 4 Some negative aspects of AR in STEM education, as reported in the 42 selected studies

Augmented Reality in STEM Education: Mapping Out the Future

685

4 Discussion This study has focused on the statistical data reported by the authors who had published their empirical researches on AR in STEM education. Unfortunately, not all authors who had published on AR in STEM reported their full statistics in their papers, so this study exploited only those particular data that were published. It nevertheless became possible to calculate the values of Cohen’s d for each experimental group from the means ± s.d. values that the authors reported. Had more authors reported such basic descriptive statistics, we would have acquired a more complete and accurate view of the usefulness and applicability of AR in STEM education. Previous review papers had focused on the identification of particular advantages of AR in education, be it in general education [8] or specifically in STEM [3]. But unless the reported positive impacts of AR in STEM education are measured by a measure that applies to all the examined empirical research papers, the magnitude of the overall effectiveness in education will remain unknown. With the quantitative part of our research however, it was revealed that the effectiveness of AR varied significantly among the different components of STEM. Specifically, it was shown that the effectiveness of AR in mathematical education is considerably more important in mathematics (primarily) and the sciences (secondarily), rather than in technology and engineering (mean Cohen’s d values 3, 1.6, 1.5, 0.67 respectively). These values are higher compared to what has been previously reported for sciences and mathematics [3], but it might be an issue of concern for engineering and technology educators. Surprisingly perhaps, although AR is a technology itself, it apparently has a less significant effect in technological education (the “T” component) in comparison to mathematics and sciences. As concerns the qualitative analysis, the results of the content analysis corroborated those of the quantitative analysis, in the sense that the significant effects of Cohen’s d that were measured are supported by the expressions of acceptance of AR by those who participated in the projects (researchers, teachers, students).

5 Conclusions Although the multiple positive effects of AR in STEM education have been documented repeatedly in numerous studies, it is important to carry out content analyses and to calculate Cohen’s d in order to gain deeper insight into findings. By such analyses, it was shown that, while AR is desirable, effective, useful, pleasant, mind-stretching tool in STEM education, it has been considerably more exploited (and also more effective) in the context of science and mathematics courses (STEM components for which Cohen’s d values were higher) than in other components. Furthermore, AR has a positive effect in STEM education regardless of the students’ ages or numbers.

686

S. Psycharis et al.

References 1. Bursztyn, N., Walker, A., Shelton, B., Pederson, J.: Assessment of student learning using augmented reality Grand Canyon field trips for mobile smart devices. Geosphere 13(2), 260– 268 (2017) 2. Salmi, H., Thuneberg, H., Vainikainen, M.P.: Making the invisible observable by Augmented Reality in informal science education context. Int. J. Sci. Educ. Part B 7(3), 253–268 (2017) 3. Ibáñez, M.B., Delgado-Kloos, C.: Augmented reality for STEM learning: a systematic review. Comput. Educ. 123, 109–123 (2018) 4. Fidan, M., Tuncel, M.: Integrating augmented reality into problem based learning: the effects on learning achievement and attitude in physics education. Comput. Educ. 142, 103635 (2019) 5. Fonseca, D., Martí, N., Redondo, E., Navarro, I., Sánchez, A.: Relationship between student profile, tool use, participation, and academic performance with the use of Augmented Reality technology for visualized architecture models. Comput. Hum. Behav. 31, 434–445 (2014) 6. Wang, H.Y., Duh, H.B.L., Li, N., Lin, T.J., Tsai, C.C.: An investigation of university students’ collaborative inquiry learning behaviors in an augmented reality simulation and a traditional simulation. J. Sci. Educ. Technol. 23(5), 682–691 (2014) 7. Akçayır, M., Akçayır, G., Pekta¸s, H.M., Ocak, M.A.: Augmented reality in science laboratories: the effects of augmented reality on university students’ laboratory skills and attitudes toward science laboratories. Comput. Hum. Behav. 57, 334–342 (2016) 8. Garzón, J., Pavón, J., Baldiris, S.: Systematic review and meta-analysis of augmented reality in educational settings. Virtual Real. 23(4), 447–459 (2019) 9. Bernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M., Huang, B.: How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Rev. Educ. Res. 74(3), 379–439 (2004) 10. Cascales-Martínez, A., Martínez-Segura, M.J., Pérez-López, D., Contero, M.: Using an augmented reality enhanced tabletop system to promote learning of mathematics: a case study with students with special educational needs. Eurasia J. Math. Sci. Technol. Educ. 13(2), 355–380 (2016) 11. Chen, C.P., Wang, C.H.: Employing augmented-reality-embedded instruction to disperse the imparities of individual differences in earth science learning. J. Sci. Educ. Technol. 24(6), 835–847 (2015) 12. de Ravé, E.G., Jiménez-Hornero, F.J., Ariza-Villaverde, A.B., Taguas-Ruiz, J.: DiedricAR: a mobile augmented reality system designed for the ubiquitous descriptive geometry learning. Multimedia Tools Appl. 75(16), 9641–9663 (2016) 13. Yoon, S., Anderson, E., Lin, J., Elinich, K.: How augmented reality enables conceptual understanding of challenging science content. J. Educ. Technol. Soc. 20(1), 156–168 (2017) 14. Sommerauer, P., Müller, O.: Augmented reality in informal learning environments: a field experiment in a mathematics exhibition. Comput. Educ. 79, 59–68 (2014) 15. Coimbra, M.T., Cardoso, T., Mateus, A.: Augmented reality: an enhancer for higher education students in math’s learning? Procedia Comput. Sci. 67, 332–339 (2015) 16. Enyedy, N., Danish, J.A., Delacruz, G., Kumar, M.: Learning physics through play in an augmented reality environment. Int. J. Comput.-Support. Collab. Learn. 7(3), 347–378 (2012) 17. Martín-Gutiérrez, J., Saorín, J.L., Contero, M., Alcañiz, M., Pérez-López, D.C., Ortega, M.: Design and validation of an augmented book for spatial abilities development in engineering students. Comput. Graph. 34(1), 77–91 (2010) 18. Dünser, A., Walker, L., Horner, H., Bentall, D.: Creating interactive physics education books with augmented reality. In: Proceedings of the 24th Australian Computer-Human Interaction Conference, OzCHI 2012, Melbourne, Victoria, Australia, pp.107–114 (2012) 19. Estapa, A., Nadolny, L.: The effect of an augmented reality enhanced mathematics lesson on student achievement and motivation. J. STEM Educ. 16(3), 40–48 (2015) 20. Barrow, J., Forker, C., Sands, A., O’Hare, D., Hurst, W.: Augmented reality for enhancing life science education. In: VISUAL 2019-The Fourth International Conference on Applications and Systems of Visual Paradigms, Visual 2019, Rome, Italy (2019)

Augmented Reality in STEM Education: Mapping Out the Future

687

21. Guntur, M.I.S., Setyaningrum, W., Retnawati, H., Marsigit, M., Saragih, N. A., bin Noordin, M. K.: Developing augmented reality in mathematics learning: the challenges and strategies. Jurnal Riset Pendidikan Matematika 6(2), 211–221 (2019) 22. Sung, N.J., Ma, J., Choi, Y.J., Hong, M.: Real-time augmented reality physics simulator for education. Appl. Sci. 9(19), 4019 (2019) 23. Bazarov, S.E., Kholodilin, I.Y., Nesterov, A.S., Sokhina, A.V.: Applying Augmented Reality in practical classes for engineering students. IOP Conf. Ser. Earth Environ. Sci. 87(3) (2017). IOP Publishing, Gothenburg, Sweden 24. Martin-Gutierrez, J., Guinters, E., Perez-Lopez, D.: Improving strategy of self-learning in engineering: laboratories with augmented reality. Procedia Soc. Behav. Sci. 51, 832–839 (2012) 25. Criollo-C, S., Abad-Vásquez, D., Martic-Nieto, M., Velásquez-G, F.A., Pérez-Medina, J.L., Luján-Mora, S.: Towards a new learning experience through a mobile application with augmented reality in engineering education. Appl. Sci. 11(11), 4921 (2021) 26. Vaughan, K.L., Vaughan, R.E., Seeley, J.M.: Experiential learning in soil science: use of an augmented reality sandbox. Nat. Sci. Educ. 46(1), 1–5 (2017) 27. Demitriadou, E., Stavroulia, K.E., Lanitis, A.: Comparative evaluation of virtual and augmented reality for teaching mathematics in primary education. Educ. Inf. Technol. 25(1), 381–401 (2020) 28. Suprapto, N., Nandyansah, W., Mubarok, H.: An evaluation of the “PicsAR” research project: an augmented reality in physics learning. Int. J. Emerg. Technol. Learn. (IJET) 15(10), 113–125 (2020) 29. Abdul Hanid, M.F., Mohamad Said, M.N.H., Yahaya, N., Abdullah, Z.: Effects of augmented reality application integration with computational thinking in geometry topics. Educ. Inf. Technol. 27(7), 9485–9521 (2022) 30. Volioti, C., Keramopoulos, E., Sapounidis, T., Melisidis, K., Zafeiropoulou, M., Sotiriou, C., Spiridis, V.: Using augmented reality in K-12 education: an indicative platform for teaching physics. Information 13(7), 336 (2022) 31. Kaur, D.P., Mantri, A., Horan, B.: Enhancing student motivation with use of augmented reality for interactive learning in engineering education. Procedia Comput. Sci. 172, 881–885 (2020) 32. Aldalalah, O., Ababneh, Z., Bawaneh, A., Alzubi, W.: Effect of augmented reality and simulation on the achievement of mathematics and visual thinking among students. Int. J. Emerg. Technol. Learn. (iJET) 14(18), 164–185 (2019) 33. Schutera, S., Schnierle, M., Wu, M., Pertzel, T., Seybold, J., Bauer, P., Teutscher, D., Raedle, M., Heß-Mohr, N., Röck, S., Krause, M.J.: On the potential of augmented reality for mathematics teaching with the application cleARmaths. Educ. Sci. 11(8), 368 (2021) 34. Erbas, C., Demirer, V.: The effects of augmented reality on students’ academic achievement and motivation in a biology course. J. Comput. Assist. Learn. 35(3), 450–458 (2019) 35. Kularbphettong, K., Limphoemsuk, N.: The effective of learning by augmented reality on Android platform. In: E-Learning, E-Education, and Online Training. Springer, Cham (2017) 36. Gün, E.T., Atasoy, B.: The effects of augmented reality on elementary school students’ spatial ability and academic achievement. Egitim ve Bilim 42(191), 31–51 (2017) 37. Arifuddin, A., Wahyudin, W., Prabawanto, S., Yasin, M., Elizanti, D.: The effectiveness of augmented reality-assisted scientific approach to improve mathematical creative thinking ability of elementary school students. Al Ibtida Jurnal Pendidikan Guru MI 9(2), 444–455 (2022) 38. Lubis, M., Oktarina Handayani, D., Ridho Lubis, A., Jafar Adrian, Q.: Using augmented reality (AR) to educate the student: bridging the communication for internet addiction. In: The 2022 5th International Conference on Electronics, Communications and Control Engineering, pp. 27–34. Publication History, New York United States (2022) 39. Lozada-Yánez, R., La-Serna-Palomino, N., Molina-Granja, F.: Augmented reality and MSkinect in the learning of basic mathematics: KARMLS case. Int. Educ. Stud. 12(9), 54–69 (2019) 40. Thornton, T., Lammi, M.: An Exploration of augmented reality in an introductory engineering graphics course. J. Technol. Educ. 32(2), 38–55 (2021)

688

S. Psycharis et al.

41. Ismail, A., Festiana, I., Hartini, T.I., Yusal, Y., Malik, A.: Enhancing students’ conceptual understanding of electricity using learning media-based augmented reality. J. Phys. Conf. Ser. 1157(3), 032049 (2019) 42. Sommerauer, P., Müller, O.: Augmented reality for teaching and learning-a literature review on theoretical and empirical foundations. In: Twenty-Sixth European Conference on Information Systems (ECIS2018), Portsmouth, UK, p. 31 (2018) 43. Cai, S., Chiang, F.K., Wang, X.: Using the augmented reality 3D technique for a convex imaging experiment in a physics course. Int. J. Eng. Educ. 29(4), 856–865 (2013) 44. Cai, S., Liu, C., Wang, T., Liu, E., Liang, J.C.: Effects of learning physics using Augmented Reality on students’ self-efficacy and conceptions of learning. Br. J. Edu. Technol. 52(1), 235–251 (2021) 45. Cr˘aciun, D., Bunoiu, M.: Boosting physics education through mobile augmented reality. In: AIP Conference Proceedings, vol. 1916, no. 1, p. 050003. AIP Publishing LLC, College Park (2017)

Certainty-Based Self-Assessment: A Chance for Enhanced Learning Engagement in Higher Education. An Experience at the University of Barcelona Ana Remesal, María José Corral, Patricio García-Mínguez, Judit Domínguez, Iria SanMiguel, Tomas Macsotay, and Ernesto Suárez

Abstract In this exploratory study, we inquire about the potential of the certaintybased marking (CBM) strategy for the activation of metacognitive self-assessment processes by means of the questioning of the degrees of certitude comparing four different disciplinary areas in Higher Education: Primary Teacher Education, Secondary Teacher Education, Physio-psychology and Microeconomics. In all these areas, the instructors implement one same instructional design with a double diagnostic and formative purpose. Out of 1,325 enrolled students, 951 eventually participate in this first experience of certainty-based self-assessment responding to a 10-multiple-choice-item test with questions presented for (self-)diagnosis at the beginning of the semester for each course in the project. Results show significant yet small differences regarding three research variables: disciplinary area (psychology, economics, teacher education), educational level (either pre-graduate or post-graduate), and instructional value of the questions (prior formal knowledge versus prior informal knowledge). The most salient results refer to the degree of certitude in case of mistaken answer, which points to useless or pernicious knowledge. The area of economics, in particular, presents contrasting results with respect to a high degree of certainty. Keywords Self-Assessment · Formative Assessment · Metacognition · Degrees of Certitude · Certainty-Based Marking A. Remesal (B) · M. J. Corral · J. Domínguez · I. SanMiguel Faculty of Psychology, Universtat de Barcelona, Barcelona, Spain e-mail: [email protected] P. García-Mínguez Faculty of Economics, Universitat de Barcelona, Barcelona, Spain T. Macsotay Faculty of Humanities, Universitat Pompeu Fabra, Barcelona, Spain E. Suárez Faculty of Law, Universitat Pompeu Fabra, Barcelona, Spain © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_56

689

690

A. Remesal et al.

1 Introduction This study presents partial and preliminary results from a very first experience at a multidisciplinary and multi-level exploration of the impact of using certainty-based assessment at higher education. The data and results presented here belong to a broader project which is still in progress and will be finished by June 2023. In this paper, we share some ideas and empirical results that should challenge higher education instructors consider new forms of assessment for the sake of improving students’ engagement and metacognitive activation towards a life-long learning professional competence.

1.1 Why Certainty-Based Self-Assessment? Back in the decade of 1980, D. Leclercq proposed an alternative psychometric algorithm to be used with multiple-choice quizzes (MCQ) that should contribute to a more valid and reliable assessment of knowledge [1]. In more recent studies, he refers to degrees of certitude [2, 3]. This idea was later picked by Gardner-Medwin [4, 5] and denominated Certainty-based Marking or CBM. Leclercq and GardnerMedwin developed subsequently independent research programs, both contributing to the improvement of knowledge assessment. This notion of certainty sometimes has been replaced by confidence. Both terms, as a matter of fact, point to the same idea of metacognitive reflection on the answers one is providing and the possibility of becoming aware of our knowledge; thus, they increase engagement in the learning process [6]. This exercise of self-consciousness is extremely important in the context of higher education and also vocational education, as a life-long learning competence becomes crucial. In fact, as another study [3] states, in the professional or real-life context, there is little interest in knowing something with a great amount of doubt or uncertainty. Such knowledge is rather useless for professional decision taking. On the other hand, a great certitude of mistaken answers neither is desirable. Such knowledge would be pernicious or nocuous, since wrong and misfortunate decisions could be inspired by such mislead certitude. Eventually, it is desirable and expectable that professionals construct correct knowledge and develop a positive certitude of it, along with a basic awareness of moments of doubt, and a sensible attitude to confront and solve it. These would be important qualities of a competent professional, regardless the particular discipline. Technically speaking, the system consists of applying an alternative algorithm to the grading of MCQ, after the learner declared a particular degree of certainty (out of three options) at each single answer. Depending on the answer being correct or false, the student gets: −6 points (wrong answer, high certainty), −2 points (wrong answer, middle certainty), 0 points (wrong answer low certainty), +1 point (right answer, low certainty), +2 points (right answer, middle certainty), or +3 points

Certainty-Based Self-Assessment: A Chance for Enhanced Learning …

691

(right answer, high certainty). Currently, the LMS Moodle is the only one — to our knowledge — that has incorporated the algorithm to the assessment applications, as an optional feature.

1.2 The Missing Transdisciplinary Look at HE Traditionally, our knowledge about learning processes at HE mainly comes from Education and Psychology students, as the own teaching context of educational researchers provides convenient sampling options. Only just recently, there are some scholar voices preoccupied for the improvement of teaching and learning at HE at any other disciplinary area, recognizing the curricular and discursive peculiarities of each branch of knowledge [7–9]. Hence, research is needed here to better know how students from different areas might approach learning in different ways, according to different learning traditions and cultures, in need of a variety of changes and strategies of improvement. Attending this call, previous research on CBM or degrees of certitude provides us with many different results from a diversity of disciplinary areas, such as: Chiropractic Education [10], Medical Education [11–14], Aviation Education [15], Foreign Language Education [16], Business Administration and Management [17], Engineering [18], and Law Education [19]. All of these studies, nevertheless, always concentrate on just one curricular area. We still miss comparative studies that take differences between areas into account.

1.3 The Missing CBM-Research in Comparing Different Levels of Education As novel as research about CBM in different disciplinary area is, another lack of the current state of the art refers to the study of students’ reactions and perceptions at different educational levels — hence age — regardless of the curricular area. In our literature search, we found only one conference paper focusing on adults [20] and two previous studies with secondary education students [21, 22]. Once again, a comparative approach is missing.

1.4 The Missing Look at the Instructional Value of the Learning to Be Assessed Assessment serves a variety of purposes at different moments of the educational process. From its very beginning, CBM is proposed as a tool for formative goals,

692

A. Remesal et al.

rather than summative ones [1, 3, 4]. However, previous literature is opaque regarding the instructional value of the contents to be assessed by CBM. We contend that the nature of the questions the students are exposed to could also have different impact on their answers. In our study, students answered questions with different instructional implications: either referring to informal prior knowledge (folkloric misconceptions) or to formal prior knowledge (knowledge supposedly acquired in a previous semester course).

2 Goals and Research Questions After framing our project in previous literature, we define our research goal as follows: We intend to analyze potential benefits and risks of the “certainty-based” evaluation system at different disciplinary areas in HE and with a formative purpose. This goal takes empirical form in the subsequent research questions as a first step in a longer research path: • RQ-1. Are there differences related to disciplinary areas? • RQ-2. Are there differences related to the educational level? • RQ-3. Are there differences related to the instructional value of the tests within the teaching program?

3 Method We carried out this study at the University of Barcelona in the academic year 2022– 2023 with instructors of four disciplinary areas in collaboration. The full project contemplates participation of a second institution at the same city of Barcelona, which has data collection still in progress. It is an exploratory study; the natural instructional context does not guarantee the control of variables that would be required in an experimental design, and, most importantly, our sociocultural comprehension of teaching and learning processes sets certain conditions of attending subjects’ perspectives as they occur in the natural context. Although the full study contemplates a broader data collection regarding precisely these subjective experiences, for the sake of time and space, we present here only results about direct students’ answers to CBM-tests to the first testing (out of three) in the semester. We throw, thus, a transversal look on the data. Table 1 presents the instructional characteristics of the sample of participants. Table 2 presents complementary demographic data.

Certainty-Based Self-Assessment: A Chance for Enhanced Learning …

693

Table 1 Sample of participants Discipline (course)

Students Participating Groups, Educational Instructional enrolled students (%) year level value of tests (semester)

Economics (Microeconomics II) (ECO)

209

111 (53.11%)

3 groups, second year (4th )

Bachelor

Formal prior knowledge

Psychology (Physio-psychology) (PSY)

607

371 (61.12%)

7 groups, first year (2nd )

Bachelor

Formal prior knowledge

50 (94.33%) 1 group, first year (1st )

Bachelor

Informal prior knowledge

Primary Education Teacher (Learning Psychology) (PTE) Secondary Education Teacher (Developmental and Instructional Psychology) (STE) TOTAL

53

454

421 (92.73%)

1325

951 (71.77%)

15 groups, Master one year only, (1st )

Informal prior knowledge

Table 2 Participants’ demographics N = 951

Students %

Sex Women Men Other

70.42 27.51 2.08

Age Under 20y 20-24y 25-29y 30-34y 35y or more

28.71 40.31 18.17 6.57 6.23

Prior studies High school Vocational studies Graduate, Bachelor Postgraduate/Masters/PhD

35.64 8.82 39.97 15.57

Job and studies No job Part-time job Full-time job

50.87 34.60 14.53

Family load (children or elders) No Yes

89.6 10.4

Students %

3.1 Instructional Design Five instructors intervened in total, distributed in the four disciplinary areas: one instructor at economics; one instructor at teacher education, both primary and secondary; and three instructors at psychology. All instructors agreed upon the same instructional design. In all courses, the students would answer a 10-item multiple choice test with four answer options at the beginning of each of three content topics along the semester. The instructors were in charge of the design of these items for each discipline to ensure contextual validity of the items. The tests had a diagnostic

694

A. Remesal et al.

purpose; it was presented to the students as a self-diagnostic tool for the course. The items referred in all cases to required prior knowledge for each course, and they could have two different instructional values: they could refer to informal or to formal prior knowledge. For the case of informal prior knowledge, the questions tackled folkloric ideas and typical misconceptions concerning the course program. For the case of formal prior knowledge, the questions referred to contents covered in a previous course in the semester immediately before the one where the study develops. The results of the students would not have in any of the participating courses other than a formative value, so even if getting negative results, students did not have to fear academic consequences. Table 1 presents the instructional features of the samples. The students were conveniently informed about the conditions of the project and informed consent was collected from participants, according to the ethical requirements.

3.2 Data Collection The students responded to the tests either during the lecture or at home (depending on the instructor’s planning) through the institutional virtual campus. Right after responding the tests, they received automatic numerical feedback from the system, according to the CBM grading algorithm. Students received an interpretation guide from the research-team with a qualitative scale in order to help them understand their results. The virtual campus, based on the LMS Moodle, automatically registered students’ answers concerning hit/error and degree of certitude. Again, depending on each instructor’s planning, the students would receive a general, collective feedback at different moments of the course. Students’ Demographics. Table 2 presents the samples’ demographics. The predominant sex was female (over 70%), nearly 40% of the students already had a bachelor’s degree and were between 20 and 24 years old. Nearly 90% of the sample had no family responsibilities, and about half of the sample had no job (ca. 51%). These two latter data (family and job responsibilities) are considered to indirectly inform about personal maturity and time available for studying.

4 Analysis and Results The Chi-square test of independence was carried out and the Phi coefficient calculated in order to identify differences in the data and their effect size. Table 3 presents the full set of results. Table 4 to Table 7 present frequency results of hits/errors regarding degrees of certitude, relative to each of the research questions.

Certainty-Based Self-Assessment: A Chance for Enhanced Learning …

695

Table 3 Differences and effect size regarding hits/errors and degrees of certainty Significant Effect size difference χ2 φ Complete sample 556.902 Errors versus Hits (df = 2, χc 2 = 9.21)

Significant Effect size difference χ2 φ

0.242*

Analysis of Hits

Analysis of Errors

Inter-disciplinary 127.001 (four areas) (df = 6, χc 2 = 16.812)

0.147

Inter-disciplinary 179.343 (four areas) (df = 6, χc 2 = 16.812)

0.222*

ECO v. other areas

95.066 (df = 2, χc 2 = 9.21)

0.127

ECO v. other areas

481.761 (df = 2, χc 2 = 9.21)

0.36**

PSY v. other areas 17.744 (df = 2, χc 2 = 9.21)

0.055

PSY v. other areas 150.875 (df = 2, χc 2 = 9.21)

0.204*

PTE v. other areas 2.464 (df = 2, χc 2 = 9.21)

–

PTE v. other areas 4.161 (df = 2, χc 2 = 9.21)

–

STE v. other areas 50.782 (df = 2, χc 2 = 9.21)

0.093

STE v. other areas 74.217 (df = 2, χc 2 = 9.21)

0.143

ECO v. PSY

28.093 (df = 2, χc 2 = 9.21)

0.179

ECO v. PSY

115.707 (df = 2, χc 2 = 9.21)

0.254*

ECO v. PTE

0.385 (df = 2, χc 2 = 9.21)

–

ECO v. PTE

20.331 (df = 2, χc 2 = 9.21)

0.169

ECO v. STE

103.792 (df = 2, χc 2 = 9.21)

0.182

ECO v. STE

8.132 (df = 2, χc 2 = 9.21)

–

PSY v. PTE

0.385 (df = 2, χc 2 = 9.21)

–

PSY v. PTE

4.182 (df = 2, χc 2 = 9.21)

–

PSY v. STE

27.943 (df = 2, χc 2 = 9.21)

0.075

PSY v. STE

137.901 (df = 2, χc 2 = 9.21)

0.217*

PTE v. STE

8.813 (df = 2, χc 2 = 9.21)

–

PTE v. STE

15.622 (df = 2, χc 2 = 9.21)

0.092

74.217 (df = 2, χc 2 = 9.21)

0.143

Educational level Pregraduate v. Postgraduate

Educational level 50.782 (df = 2, χc 2 = 9.21)

0.093

Pregraduate v. Postgraduate

(continued)

696

A. Remesal et al.

Table 3 (continued) Significant Effect size difference χ2 φ Instructional value of questions Formal v. Informal prior knowledge

Significant Effect size difference χ2 φ Instructional value of questions

42.548 (df = 2, χc 2 = 9.21)

0.085

Formal v. Informal prior knowledge

58.828 (df = 2, χc 2 = 9.21)

0.127

Note: * effect size over 0.20; ** effect size over 0.30

Table 4 Frequency of hits/errors and degrees of certainty in all participants Hit/Error

Errors %

Hits %

Knowledge quality

Nocuous knowledge

Useless knowledge

Useful knowledge

Degrees of certainty

High

Middle

Low

Low

Middle

High

All participants

16.77

14.49

7.05

15.55

21.90

24.24

4.1 General Differences in the Whole Sample Table 3 presents the results of the Chi-square test and Phi coefficient calculation. As we can observe, there are significant differences indicating a connection between the degrees of certainty and the correctness of students’ answers; nevertheless, these differences are mostly of a very small or small size, with the exception of the economics students and their cases of mistaken answer, who reach a medium effect (φ = 0.36). Table 4 offers frequency results regarding the full sample of students, referring to the quality of their knowledge, either nocuous (31.26%), useless (22.60%), or useful (46.14%). Unfortunately, there is high frequency of nocuous knowledge (nearly one third of the students present errors with middle or high certainty). These results receive a closer look in the following sections addressing each of the research questions.

4.2 RQ-1: Differences Regarding Disciplinary Area There are, as pointed before, some noticeable results regarding the disciplinary area (Table 5): for all areas the frequency of useful knowledge oscillates between 43.20% (PTE) and 47.20% (PSY), students of physio-psychology present the most percentage of useful knowledge; however, there is a prominent high certainty among students of Microeconomics II, which also presents significant difference of medium effect size (φ = 0.36). When looking at the useless knowledge, in all four areas, we find frequency between 21.47% (ECO) and 23.87% (STE), whilst nocuous knowledge oscillates between 30.31% (STE) and 34.60% (PTE), which is not a negligible result.

Certainty-Based Self-Assessment: A Chance for Enhanced Learning …

697

Table 5 Frequency of hits/errors and degrees of certainty in disciplinary areas Hit/Error

Errors %

Knowledge quality

Nocuous knowledge

Useless knowledge

Hits % Useful knowledge

Degrees of certainty

High

Middle

Low

Middle

High

ECO

16.42

16.97

13.30

8.17

14.81

30.83

PSY

19.33

11.94

3.40

18.14

21.59

25.61

PTE

20.00

14.60

5.60

16.60

18.80

24.40

STE

14.23

16.08

8.81

15.06

24.51

21.31

Low

Disciplinary areas

4.3 RQ-2: Differences Regarding Educational Level With respect to the educational level, the participants of this study must be redistributed in the following way: all STE students were enrolled in a masters course, while the other groups correspond to bachelor pre-graduate courses. Table 6 presents the corresponding results. There is indeed some significance in the differing results, but with a very small effect size (see Table 3: STE v. other areas; φ = 0.09 regarding hits; φ = 0.143 regarding errors). Interestingly, the proportions of nocuous, useless and useful knowledge are similar as related to disciplinary areas: nocuous (30%–32%), useless (21%–24%), and useful (45%–46%). Table 6 Frequency of hits/errors and degrees of certainty on an educational level Hit/Error

Errors %

Knowledge quality

Nocuous knowledge

Useless knowledge

Hits % Useful knowledge

Degrees of certainty

High

Middle

Low

Low

Middle

High

Bachelor (pre-graduate)

18.79

13.23

5.64

15.94

19.83

26.57

Master (post-graduate)

14.23

16.08

8.81

15.06

24.51

21.31

Educational level

Table 7 Frequency of hits/errors and degrees of certainty in the instructional value of tests Hit/Error

Errors %

Knowledge quality

Nocuous knowledge

Useless knowledge

Useful knowledge

Degrees of certainty

High

Middle

Low

Low

Middle

High

Informal prior knowledge

14.84

15.92

8.47

15.22

23.91

21.63

Formal prior knowledge

18.67

13.08

5.65

15.88

19.94

26.79

Hits %

Instructional value of tests

698

A. Remesal et al.

4.4 RQ-3. Differences Regarding Instructional Value of the Items Depending on the instructors’ teaching plan, the students were presented with alternative instructional questions, referring to formal or informal prior knowledge. Our hypothesis was that we would find differences in students’ relation to these two kinds of questions/knowledge. The analysis leads us to confirm these differences, although there is a very small effect (see Table 3). Once again, we can observe that the proportion of the three qualities of knowledge remain stable: nocuous (30%–31%), useless (21%–23%), and useful (45%–46%) (see Table 7).

5 Discussion of the Preliminary Results Our first reflection on the results of this study considers the unequal participation rate of students from the four different areas invited to participate. Altogether, just over 70% of the enrolled students volunteered to participate in the study. Albeit a high percentage, there was a great variety of participation ratio, with a maximal ratio (over 90%) among teacher education students, an important decrease among psychology students (ca. 60%) and a minimal participation among Economy students (ca. 50%). This decreasing participation should be related, in our view, to the lack of habit of engaging in reflection over the learning process among students in these noneducational areas, which by itself is a very important result to take notice of. Despite this decreased participation, previous studies report about only a 16.7% response in the area of Business and Administration Management [17], which would be the closest to our economy course. The results allow us to give a positive answer to all three research questions, since differences were found regarding all three aspects: discipline area, educational level, and the questions’ instructional value. Nevertheless, in most cases the differences had a small impact. Economy students offer different behavior regarding their certainty — hence, their self-competence — both regarding cases of mistakes and hits. Regarding hits, economy students presented the highest values at high-certainty with correct answers; however, when looking at the cases of low certainty, these students’ hits ratio is the opposite from the other three areas (with a higher load of social interaction). Students who declared a low level of certainty in the area of economy presented a proportion of 13.30% mistake and 8.17% hit, while all the other areas present more hits than mistakes when declaring low certainty. These results cannot be contrasted with previous studies, since previous reports do not offer data on these specific aspects. Instead, they usually report on general means of grades, which we do not take into account precisely for the transdisciplinary nature of our study. A last reflection concerning our results is that whichever variable we approach (disciplinary area, educational level or questions’ instructional value), the proportion

Certainty-Based Self-Assessment: A Chance for Enhanced Learning …

699

of nocuous, useless, and useful knowledge [3] remains similar: nocuous knowledge (ca. 30%), useless knowledge (ca. 25%), and useful knowledge (ca. 45%).

6 Conclusions and Practical Implications At this point, part of the data collection is still in progress, so we need to be cautious about our conclusions. The main conclusions of our study, up to now, refer both to the similarities and the differences found in our data, since both affect our daily praxis, as HE educators. We found similarities regarding a relatively low proportion — less than half — of what we could consider useful knowledge: correct answers with medium or high certainty or confidence. At the same time, there is almost one third of nocuous knowledge, that is mistaken answers with a high or medium confidence. This is in itself a negative result, as it shall be our aspiration to contribute to the education of future highly competent professionals and researchers. Regarding the differences manifested in our results, the main point we want to underline refers to the minor participation of economy students but also to their difference concerning confidence and correctness of answers, in comparison with the classmates from the other areas with a greater social-interaction basis. However, we would like to be prudent here and wait for the missing data belonging to two additional disciplinary areas, namely history of art and law, in order to deepen the analysis. Also, the finalization of our data collection shall enable us to address further research questions referred to other individual variables, such as specific age, sex, job or family load, previous studies, and conceptions of assessment. Acknowledgements This study received funding from the Universitat de Barcelona (ref. REDICE22-3060).

References 1. Leclercq, D.: Confidence marking, its use in testing. In: Postlethwaite & Choppin, Evaluation in Education, vol. 6, pp. 161–287. Pergamon Press, Oxford (1982). http://hdl.handle.net/2268/ 9482 2. Leclercq, D., Gilles, J.L.: GUESS, Un logiciel pour s’entraîner à l’auto-estimation de sa compétence cognitive. In: Weber, J., Dumont, B. (ed.), QCM et Questionnaires Fermés, actes du 3° colloque international de l’ESIEE, Marne-la-Vallée, décembre, pp. 137–158 (1994). http://hdl. handle.net/2268/8953 3. Leclercq, D. (ed): Diagnostic cognitif et métacognitif au seuil de l’université. Le projet MOHICAN mené par les 9 universités de la Communauté Française Wallonie Bruxelles. Editions de l’université de Liège: Liège (2003). http://hdl.handle.net/2268/17837 4. Gardner-Medwin, T., Curtin, N.: Certainty-based marking (CBM) for reflective learning and proper knowledge assessment. In: Proceedings of the REAP International Online Conference: Assessment Design for Learner Responsibility. Glasgow: University of Strathclyde, pp. 29–31 (2006)

700

A. Remesal et al.

5. Gardner-Medwin, T.: Certainty-based marking: stimulating thinking and improving objective tests. In: Bryan, C., Clegg, K. (eds.). Innovative Assessment in Higher Education, pp. 141–150. Routledge, NY (2019) 6. Bembenutty, H., Kitsantas, A., Cleary, T.J. (eds.). Applications of self-regulated learning across diverse disciplines: a tribute to Barry J. Zimmerman. IAP, Charlotte, NC (2013) 7. Quinlan, K.M., Pitt, E.: Towards signature assessment and feedback practices: a taxonomy of discipline-specific elements of assessment for learning. Assess. Educ. Principles Policy Pract. 28(2), 191–207 (2021). https://doi.org/10.1080/0969594X.2021.1930447 8. Pitt, E., Quinlan, K.M.: Signature assessment and feedback practices in the disciplines. Assess. Educ. Principles Policy Pract. 28(2), 97–100 (2021). https://doi.org/10.1080/0969594X.2021. 1930444 9. Quinlan, K.M.: Leadership of teaching for student learning in higher education: what is needed? High. Educ. Res. Dev. 33(1), 32–45 (2014). https://doi.org/10.1080/07294360.2013.864609 10. Barr, D.A., Burke, J.R.: Using confidence-based marking in a laboratory setting: a tool for student self-assessment and learning. J. Chiropr. Educ. 27(1), 21–26 (2013). https://doi.org/10. 7899/JCE-12-018 11. Chamberland, M., St-Onge, C.: Back to basics: keeping students cognitively active between the classroom and the examination. BMC Med. Educ. 47(7), 641–643 (2013). https://doi.org/ 10.1111/medu.12218 12. Luetsch, K., Burrows, J.: Certainty rating in pre-and post-tests of study modules in an online clinical pharmacy course-a pilot study to evaluate teaching and learning. BMC Med. Educ. 16(1), 1–9 (2016) 13. Smrkolj, S., Bancov, E., Smrkoli, V.: The reliability and medical students’ appreciation of certainty-based marking. Int. J. Environ. Res. Public Health 19, 1706 (2022). https://doi.org/ 10.3390/ijerph19031706 14. Schoendorfer, N., Emmett, D.: Use of certainty-based marking in a second-year medical student cohort: a pilot study. Adv. Med. Educ. Pract. 3, 139–143 (2012). https://doi.org/10.2147/AMEP. S35972 15. Novacek, P.F.: Exploration of a confidence-based assessment tool within an aviation training program. J. Aviation/Aerospace Educ. Res. 26(1), 65–88 (2017) 16. Salehi, M., Sadighi, F., Bagheri, M.S.: Comparing confidence-based and conventional scoring methods: the case of an English grammar class. J. Teach. Lang. Skills 33(4), 123–152 (2015). https://doi.org/10.22099/JTLS.2015.3105 17. Serradell, E., Lara, P., Castillo, D., González-González, I.: Confidence-based learning in investment analysis. In: International Conference on Technology Enhanced Learning, Springer, Berlin, pp. 28–35 (2010) 18. Yuen-Reed, G., Reed, K.B.: Engineering Student self-assessment through confidence-based scoring. Adv. Eng. Educ. 4(4), 4 (2015) 19. Wong, S., Rojas-Mora, J.: Aplicación del método de evaluación basada en la certeza en la enseñanza del derecho: Un estudio exploratorio en alumnos de primer año de la carrera de Derecho. Revista de Pedagogía Universitaria y Didáctica del Derecho 7(1), 43–62 (2020). https://doi.org/10.5354/0719-5885.2020.54762 20. Novacek, P.: Confidence-based assessments within an adult learning environment. In: IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2013), pp. 403–406 (2013) 21. Foster, C.: Confidence and competence with mathematical procedures. Educ. Stud. Math. 91, 271–288 (2016). 10/1007/s10649-015-9660-9 22. Clark, C.: The impact of confidence-based marking on unit exam achievement in a high school physical science course. Graduate Research Papers. p. 1449 (2020). https://scholarworks.uni. edu/grp/1449

Curated Recommendations of Teaching and Learning Videos on YouTube with the Help of a Chatbot Theresa Zobel, Hendrik Steinbeck, and Christoph Meinel

Abstract In February 2020, 500 h of video were uploaded to YouTube every minute, including numerous videos from the education sector [19, 20]. If you are looking for suitable teaching or learning videos, for example, this mass of videos makes it much more difficult to select. In the course of an implementation project with Bachelor students, teaching videos in the German language on the topic of mathematics on YouTube were examined and how they can be found for teachers and learners. From the results of this research, a database of selected videos and the corresponding search terms and difficulty levels was created in an exploratory manner. In addition, a chatbot was implemented to support users in their search for learning and teaching videos. The initial analysis showed that there are only a dozen channels that provide the majority of learning content for the subject of mathematics. The technical solution shown is able to extend this result and suggest existing but less known educational videos. The present study thus provides both a theoretical and practical complement to user-centred teaching–learning scenarios. Keywords Chatbot · Video teaching · Recommendation systems

1 Introduction The use of videos for learning at school and university is widespread. Either to support solving specific tasks, for use in class or for exam preparation—the use of audiovisual media can be broadly defined. With increasing upload numbers—approx. 720,000 min per day—YouTube is a central video platform for learners and teachers. In the context of the present study, the existing offer was analysed and explorative evaluated based on the school subject mathematics. As a compulsory and major subject, it is represented in all federal states in Germany up to high school graduation. At the same time, some narratives within school and society can be identified for T. Zobel (B) · H. Steinbeck · C. Meinel Hasso Plattner Institute, Potsdam, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 D. Guralnick et al. (eds.), Creative Approaches to Technology-Enhanced Learning for the Workplace and Higher Education, Lecture Notes in Networks and Systems 767, https://doi.org/10.1007/978-3-031-41637-8_57

701

702

T. Zobel et al.

mathematics that are worth highlighting; the auto-completion of relevant search engines add either expletives or simply “a problem subject” to the search term “Maths is.” The clue in crossword puzzles for the solution word “math” can sometimes be “dreaded school subject” [3]. Combined with learners seeking help online, this results in a very realistic scenario: when a learner gets stuck, he or she consults YouTube. No matter whether it is calculate zeros or understand expected value, numerous video results are displayed. A technical solution was developed to systematically process this oversupply. It classifies the growing amount of video material, evaluates it and returns videos to the user. Using a dialogue-based system (chatbot), suitable videos can be suggested according to the level of education (primary/secondary and university). On the one hand, this solution can aggregate several video and educational platforms, but on the other hand, it can also expand the result space that the first page of a YouTube search does not display. Against this context, the formalised research questions are: 1. How can the video offer in German language on YouTube for the school subject mathematics be outlined? 2. In what ways can a dialog-based system support the access to these recommendations? Embedded in the relevant scientific literature, the first question is answered by means of an explorative data analysis. Derived from this, the second research question is presented with a practical implementation project.

2 Relevant Literature This section presents existing research and projects in the field of creating and acquiring educational resources specifically for teachers and learners. Furthermore, dialogue-based systems in the educational context and for the acquisition of digital content are dealt with. While educational content on YouTube serves as a research object [11], the process of educational content creation has also been outlined and analysed [4]. Whether and how teachers in Europe use audio-visual media in concrete terms has been investigated as well [6]. In this survey, the group of authors also shows the dominant position of YouTube among teachers. At the same time, only one in three teachers has created their own videos, and only just under 7% state that they regularly produce their own learning content (ibid.). According to ref. [17], mathematics is the school subject with the most viewed explainer videos and the most requested subject for commercial tutoring. They also see greater use of videos for catching up on the subject matter among learners with lower grades. Dialogue-based systems can be used in a variety of ways [2]. In addition to customer support [18], ordering bots and FAQ bots [14, 15, 22], chatbots can also be used for acquiring and evaluating information, such as the TruthBot [7]. This chatbot assists in the search for information and messages, and checks the results for accuracy.

Curated Recommendations of Teaching and Learning Videos …

703

In addition, the use of chatbots as recommendation systems is popular in various fields. For online courses (MOOCs), for example, the chatbot “MOOCBuddy” [8] was researched, which suggests online courses and learning materials to users with the help of social media profiles. In order to support both learners and teachers in MOOCs, a concept for chatbots was created [21]. The chatbot offers several features that are often absent in online teaching, unlike traditional teaching. These features consist of a library-like recommendation system for additional learning resources and a study desk-like recommendation system for courses.

3 Methodology In March 2022, 15 topics were retrieved via the YouTube API. Due to the limitation of the daily retrievals (“quota”), a pre-selection of the search terms had to be made. Based on German math reference curricula [12, 13], 13 typical search terms were identified that occur in the upper school. The research team agreed to incorporate the most frequently used search terms expected to be covered in any school semester in the other 14 federal states of Germany, as per the standard curriculum guidelines of two distinct regions. However, they decided to exclude any uncommon or region-specific terms. Two search terms (taking roots and shortening fractions) are exemplary topics from lower secondary school. Pure keywords (cosine theorem) and combinations with action verbs (solve, form) were deliberately chosen to simulate various, realistic search behaviour by learners. From the analysis conducted, insights can be derived into the availability of German-language instructional videos on the topic of mathematics on YouTube. For each search term, the YouTube channels and the number of search results were evaluated, and for each video, the number of comments and number of views and likes were evaluated. The latter three attributes were additionally collected for the 15 most frequent channels. The results are displayed in Table 1. The Comments, Views and Likes of the respective videos were averaged as multiple results were yielded from YouTube for each keyword.

4 Analysis and Implementation In the following, the results of the quantitative data analysis, as well as the architecture of the technical implementation and the chatbot are presented.

704

T. Zobel et al.

Table 1 The average attributes of the analysed keywords from the curricula, including video length, comments, views, likes, and the total number of videos available Search term

Length

Comments

Views

Likes

Videos total

Solve linear system of equations (Lineares Gleichungssystem lösen)

07:28

520

593,054

9,956

31,826

Integral calculus (Integralrechnung)

11:26

209

394,686

5,401

28,611

Calculate zero (Nullstelle berechnen)

06:33

281

390,924

6,726

21,172

pq-formula (pq-Formel)

07:22

421

358,078

8,091

340,668

Reduce fractions (Brüche kürzen)

05:58

304

226,923

3,777

84,536

Root extraction (Wurzel ziehen)

06:27

261

205,439

3,952

240,644

Bernoulli probability (Bernoulli Wahrscheinlichkeit)

07:34

90

197,670

2,292

59,699

Form primitive function (Stammfunktion bilden)

06:39

104

197,310

2,825

42,485

Derive e function (e Funktion ableiten)

06:32

127

193,405

2,784

100,489

Expected value and standard deviation (Erwartungswert und Standardabweichung)

06:30

100

173,533

1,736

20,129

Gaussian bell curve 09:23 (gaußsche Glockenkurve)

225

162,047

2,050

746

Calculate scalar product (Skalarprodukt berechnen)

05:16

57

154,794

1,726

14,440

Cosine theorem (Kosinussatz)

07:45

162

148,373

3,201

5,852

Invert matrix (Matrix invertieren)

07:56

60

109,429

955

11,042

Horner scheme (Horner Schema)

08:06

50

42,650

556

66,420

4.1 Quantitative Analysis of YouTube The four column attributes (video length in minutes, number of comments, video views and likes) of Table 1 are each the average of the first 20 search results about the topic in question. The average mathematics learning video on YouTube is 7 min

Curated Recommendations of Teaching and Learning Videos …

705

and 24 s long, is commented on 200 times and liked over 3,700 times. With an average of over 70,000 videos found per search term, the quantity of results initially appears very large. The analysis of the channels puts this impression into perspective: the 300 videos (15 terms * 20 results) produce a list of the 15 most popular German-language mathematics channels on YouTube. “Popular” is defined purely quantitatively, based on the appearance in the list of results. It can be assumed that a visitor will also click on one of the first search results and use it for learning. As shown in Table 2, the percentage distribution of this exploratory analysis shows that four channels are already responsible for 54% of video views for the keywords analysed. In addition to the “top 15” shown above, a total of 73 channels were found. Of these, 63 have fewer than three videos among the first 20 search results and thus on the “first page.” The origin and the mechanism for placing this list of results is not transparent. YouTube only applies the filter “Sorted by: Relevance.” Looking at the size of the channels, a correlation can be seen, which is also reflected in the correlations. There is a strong correlation between the number of comments and the number of “likes” (0.92). Moreover, “likes” exhibit a very high correlation with video views (0.95). In order to suggest videos that are not on this “best list” and to consider smaller channels with fewer videos, the technical implementation is presented below. Table 2 Evaluation of the largest German-language YouTube channels with a mathematics focus. Sorted by share of the found list of results Channel name

Videos

Subscriptions

Views

Share of result list

mathebydanieljung

2,802

845,000

336,591,904

72 (24%)

mathematrick

875

196,000

25,887,806

34 (11.33%)

mathesimpleclub

417

815,000

139,991,642

29 (9.67%)

lehrerschmidt

1,738

1,320,000

168,451,638

27 (9%)

mathepeter

594

67,200

11,925,362

20 (6.67%)

einfachmathebyjenny

694

32,600

5,034,307

10 (3.33%)

mathehoch13

464

10,800

3,104,105

8 (2.67%)

koonysschule

987

39,400

12,592,736

7 (2.33%)

matheleichtgemacht

736

17,200

2,907,129

6 (2%)

dorfuchs

163

249,000

43,594,067

4 (1.33%)

howtomathe

157

7,750

1,374,375

3 (1%)

Matheretter

95

58,900

12,300,606

3 (1%)

oberprima

3,382

8,510

5,669,684

3 (1%)

matheonlinevideos

220

2,450

485,586

3 (1%)

matheseitede

2,711

12,300

6,328,922

3 (1%)

706

T. Zobel et al.

Fig. 1 The architecture consists of three main components: the backend, database and chatbot

4.2 Architecture The architecture, illustrated in Fig. 1, consists of three main components, a backend, a database and the chatbot. The backend collects and processes the data that the chatbot uses for user interaction. First, keywords are extracted from curricula of different schools and universities in Germany. These are used to search for educational videos on the YouTube video platform. Visual and acoustic information in text form is then extracted from the videos using two AI models, and subsequently analysed. In the next section, these processes will be discussed in more detail. The results are stored in an SQL database. This completes the preparation and, with access to the database, the chatbot can suggest suitable videos to users on different topics and levels of difficulty based on the level of education (primary/ secondary and university).

4.3 Video Analysis To assign videos to topics and educational levels, and determine their level of detail, more information is needed than what is provided by their titles and descriptions. Therefore, the videos are downloaded and converted to text using a machine learning model called Scribosermo, which requires an audio signal with a sampling rate of

Curated Recommendations of Teaching and Learning Videos …

707

16 kHz [5]. To enable optical text recognition, a binary image is generated from the video every three seconds, and edge detection is performed on it [10]. The identified key frames are then converted to binary images using OpenCV [1], and text is extracted using the Tesseract OCR engine via the Python library Pytesseract [16]. Scribosermo was chosen because of its open-source nature, Python library, and good results [5]. However, generating a binary image for every frame of the video would be too costly and have little added value, so edge detection is used to decide which frames to examine for text. At present, errors have not been examined due to the dual system of video analysis with OCR and the use of Speech-to-text (Scribosermo), which permits quality control. Moreover, while truth-and-fact checking of the content could be possible, it was not the focus of this paper. The used words of the respective content are attempted to be related to a difficulty scale by the current approach. However, it is more of a range of education, considering that some students may encounter higher algebraic functions earlier or later in their formal education, although there is broad consensus that fractions are not a part of a high school curriculum.

4.4 Content Classification The video’s text is extracted, and informative words are identified using spaCy. These words are compared to keywords extracted from curricula of different schools and universities in Germany. The set of words that appear in both are referred to as scientific terms, and each word is stored in the database with its absolute frequency. If the number of scientific terms exceeds a threshold, the video is assigned to the relevant topic. The video’s educational level is determined by taking the rounded average of the educational levels assigned to the scientific terms. In addition to scientific terms, the lexical density of the video text is determined by calculating the proportion of informative words, including nouns, proper nouns, verbs, adjectives, and adverbs [9]. From this, the video length can then be used to determine the difficulty level of the video.

4.5 Chatbot A dialogue-based system, namely a chatbot in German, is used to support users in their search. In a dialogue in natural language, the chatbot “CASPER” (project name) asks the users some questions about their search: Which educational institution the users are currently attending, in which subject support is needed and then on which topic exactly a video is being searched for. The selection of the educational institution and the subject is not made by a textual user input, but via a selection menu. This is to ensure that a selection is made that the chatbot can also process. It also reduces the risk of possible spelling mistakes and misunderstandings. With the help of the

708

T. Zobel et al.

Fig. 2 Sample dialogue of a learner with the chatbot “CASPER” (Chatbot Assisted Service for Providing Educational Resources)

answers, a suitable video is then determined, which the chatbot suggests to the users. The users then have the option of accepting the video or asking the chatbot for a new suggestion if the video is not satisfactory. It is an intentional design decision that the users will only ever see one video that best matches their search query. Otherwise, the users would have to click through a selection of videos again, which in turn would require more effort and not produce a clear result. Such an exemplary dialog for the search for square roots is shown in Fig. 2. The advantage of using a dialogue-based system instead of a dashboard, for example, is the guided conversation from the beginning of the search to the result at the end. In addition, users can restart the search at any time or ask questions if they do not understand something. This saves confusing help windows and explanations at the edge of the screen. Another benefit of a chatbot is its constant accessibility and cost-effective use, as no real person has to lead the conversation. Nevertheless, the users could have the feeling of talking to a competent, real person, as the dialogue is conducted in natural language. The chatbot was implemented using the open-source AI-supported framework Rasa. Rasa offers the possibility to run the chatbot instance on its own server, which means that data can be processed and stored in a way that is protected from access and knowledge by third parties. Rasa consists of two main components: Rasa NLU (“natural language unit”) and Rasa Core. Rasa NLU is a language processing tool for classifying the intent and entities of a conversation. Rasa Core is the dialogue engine that determines the next steps (“action”) of the chatbot based on the detected topic, entities and some specifications. Next steps can be a response but can also be a request for data via an API. The next steps have a selection probability, determined among other things based on the previous conversation, the topic, and the entities.

Curated Recommendations of Teaching and Learning Videos …

709

Fig. 3 The open-source framework Rasa consists of Rasa NLU and Rasa Core, which interpret the user’s message and determine a meaningful next action

The action with the highest probability is executed. Figure 3 shows a simplified illustration of the interaction between Rasa NLU and Rasa Core with an example conversation.

5 Discussion and Conclusion To answer the first research question (“How can the video offer in German language on YouTube for the school subject mathematics be outlined?”), educational videos on the video platform YouTube were analysed according to various criteria. This data analysis has shown that educational content on various topics is available and that, in purely quantitative terms, numerous users access it. However, the distribution of “market shares” within the explorative data analysis showed that there is an uneven distribution of search results and that only four channels share more than half of the results on the first page. Sixty-three other channels of varying sizes appear here only in the low single digits. To technically support a democratisation and accessibility of learning content, the presented chatbot architecture can be a low-threshold solution. By pre-sorting and evaluating the audio and video track using AI-supported frameworks, the popularity of YouTube as a video platform can continue to be used, and the “curation power” of the platform can be expanded by an external (educational) actor. Unfortunately, we were unable to conduct an initial evaluation of the video platform YouTube, as the process of applying for API Developer Access proved to be excessively lengthy and complicated. The API is essential for indexing videos in

710

T. Zobel et al.

the database. Therefore, for our first experiment, we had to rely on videos from a different source, namely a MOOC platform. Our assessment revealed that the variety of learning videos available on mathematics outside of YouTube is limited. Additionally, the chatbot’s outcomes can still be expanded in terms of content. We intend to revise the dialog’s nature in the upcoming phases to give it a more conversational tone, resembling that of a learning assistant rather than a filtering dashboard. Nevertheless, our initial inquiry demonstrated that a chatbot could provide educational video recommendations by asking some preliminary questions. This approach shows the potential of a chatbot in this domain, and it has uncovered opportunities to answer our second research question: “In what ways can a dialog-based system support the access to these recommendations.” We believe that with improvements to the chatbot’s conversational abilities and the benefit of the YouTube API, the chatbot can offer satisfactory recommendations to students through a well-structured conversation. The subsequent stage involves using these findings to revise the chatbot while also gaining access to the YouTube API. The system will then be tested by the intended audience, including high school and university students as well as teachers. Feedback obtained will be utilized to enhance the chatbot’s usability and functionality and test its real-world applicability. Additionally, we intend to use other video platforms and more public educational resources (i.e., “open educational resources”). Furthermore, before storing the videos in the database, we plan to evaluate them using quality scales. This will ensure that users receive only high-quality videos while also allowing us to better identify videos appropriate for different educational levels (elementary/ high school and university). These are just a few of the many ways in which we intend to continue and shape the project in the future.

References 1. About - OpenCV, p. 10. https://opencv.org/about/. Accessed 28 June 2022 2. Adamopoulou, E., Moussiades, L.: An overview of chatbot technology. In: IFIP International Conference on Artificial Intelligence Applications and Innovations, pp. 373–383. Springer, Cham (2020) 3. Alibek, A.: Online. In: Twitter. https://twitter.com/AidaAlibek/status/1505601831375974407. Accessed 14 Apr 2022 4. Beautemps, J., Bresges, A.: What comprises a successful educational science youtube video? A five-thousand user survey on viewing behaviors and self-perceived importance of various variables controlled by content creators. Front. Commun. 5, 600595 (2021) 5. Bermuth, D., Poeppel, A., Reif, W.: Scribosermo: Fast Speech-to- Text models for German and other Languages, p. 10 (2021). https://doi.org/10.48550/ARXIV.2110.07982 6. Espino, M., Suárez, M., González-Henríquez, J.: Video for teaching: classroom use, instructor self-production and teachers’ preferences in presentation format. Technol. Pedagogy Educ. 29(2), 147–162 (2020) 7. Gupta, A., et al.: TruthBot: An Automated Conversational Tool for Intent Learning, Curated Information Presenting, and Fake News Alerting (2021) 8. Holotescu, C.: MOOCBuddy: a Chatbot for personalized learning with MOOCs. In: RoCHI, pp. 91–94 (2016)

Curated Recommendations of Teaching and Learning Videos …

711

9. Ure, J.: Lexical density and register differentiation. In: von Perren, G.E., Trim, J.L.M. (Hrsg) Applications of Linguistics, vol. 11, pp. 443–452. Cambridge University Press, London (1971) 10. Jeong, H., et al.: Automatic detection of slide transitions in lecture videos. Multimedia Tools Appl. 74, 7537–7554 (2015) 11. Jones, T., Cuthrell, K.: YouTube: educational potentials and pitfalls, Comput. Sch. 28(1), 75–85 (2011) 12. Kerncurriculum für die gymnasiale Oberstufe der Deutschen Schulen im Ausland. Accessed 25 Mar 2022. https://www.kmk.org/fileadmin/Dateien/veroeffentlichungen_beschluesse/2010/ 2010_04_29-Kerncurriculum-Mathematik.pdf 13. Ministerium für Schule und Weiterbildung des Landes Nordrhein-Westfalen. Kernlehrplan für die Sekundarstufe II Gymnasium/Gesamtschule in Nordrhein-Westfalen. Zuletzt abgerufen am 25.03.2022. https://www.schulentwicklung.nrw.de/lehrplaene/ 14. Massaro, A., Maritati, V., Galiano, A.: Automated self-learning chatbot initially build as a FAQs database information retrieval system: multi-level and intelligent universal virtual front-office implementing neural network. Informatica 42(4) (2018) 15. Shawar, A., Atwell, E., Roberts, A.: Faqchat as in information retrieval system. In: Human Language Technologies as a Challenge for Computer Science and Linguistics: Proceedings of the 2nd Language and Technology Conference. Wydawnictwo Pozna´nskie: with co-operation of Fundacja Uniwersytetu im. A. Mickiewicza, pp. 274–278 (2005) 16. Smith, R.: An overview of the tesseract OCR engine. In: Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), vol. 2, p. 10. IEEE, (2007). https://doi. org/10.1109/icdar.2007.4376991 17. Wolf, K.D., et al.: Leistungsoptimierung von Schülerinnen und Schülern durch schulbezogene Erklärvideonutzung auf YouTube. Entschulungsstrategie oder Selbsthilfe? In: MedienPädagogik Heft, vol. 42, pp. 380–408 (2021) 18. Xu, A., et al.: A new chatbot for customer service on social media. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 3506–3510 (2017) 19. YouTube For Press. https://blog.youtube/press/. Accessed 25 Apr 2023 20. YouTube: hours of video uploaded every minute 2022. https://www.statista.com/statistics/259 477/hours-of-video-uploaded-to-youtube-every-minute/. Accessed 25 Apr 2023 21. Zobel, T., Meinel, C.: Towards personalized, dialogue-based system supported learning for MOOCs. In: The Learning Ideas Conference, pp. 425–435. Springer, Cham (2021) 22. Zobel, T., Renz, J., Meinel, C.: Improving the scalability of MOOC platforms with automated, dialogue-based systems. In: 2020 IEEE Learning With MOOCS (LWMOOCS), pp. 42–46. IEEE (2020)