Handbook of Multiple Source Use 9781138646599, 9781138646605, 9781315627496

The Handbook of Multiple Source Use draws on theory and research within cognitive and educational psychology, the learni

197 75 5MB

English Pages [561] Year 2018

Report DMCA / Copyright

DOWNLOAD PDF FILE

Table of contents :
Handbook of Multiple Source Use- Front Cover
Handbook of Multiple Source Use
Title Page
Copyright Page
Dedication
Contents
Acknowledgments
Chapter 1: Introduction to Research on Multiple Source Use
INTRODUCTION
RESEARCH ON MULTIPLE SOURCE USE
GOALS OF THIS HANDBOOK
THEMES OF THIS HANDBOOK
OVERVIEW OF THE SECTIONS AND CHAPTERS
REFERENCES
Section I: THEORETICAL FRAMEWORKS
Chapter 2: Representations and Processes in Multiple Source Use
READING SINGLE VS. MULTIPLE TEXTS: SAME OR DIFFERENT PROCESSES?
OVERVIEW OF REPRESENTATIONS AND PROCESSES
USING RESOLV TO EXAMINE TASK AND TEXT INTERACTIONS
DISCUSSION
REFERENCES
Chapter 3: Cold and Warm Perspectives on the Cognitive Affective
Engagement Model of Multiple Source Use
COLD PERSPECTIVES ON MULTIPLE SOURCE USE
COLD FACTORS IN MSU
WARM PERSPECTIVES ON MULTIPLE SOURCE USE
DEVELOPMENT OF THE COGNITIVE AFFECTIVE ENGAGEMENT MODEL
MULTIPLE SOURCE PROCESSING ACCORDING TO THE CAEM
INITIATING TASK
DEFAULT STANCE
MSU PROCESSES AND BEHAVIORS
CAEM-LIKE PROFILES IN PRIOR RESEARCH
FUTURE DIRECTIONS
INSTRUCTIONAL IMPLICATIONS
CONCLUSION
REFERENCES
Chapter 4: Toward a New Literacies Perspective of Synthesis:
Multiple Source Meaning Construction
BORROWING FROM NEW LITERACIES THEORY
PURPOSE
TEXT
READERS
TECHNOLOGIES
CONTEXT
UNDERLYING PRINCIPLES FOR SYNTHESIS AS MULTIPLE SOURCE MEANING CONSTRUCTION
THE JOURNEY FORWARD
NOTE
REFERENCES
Chapter 5: A Social Psychological Perspective on Multiple Source
Use: Elaboration and Persuasion
THEORETICAL BACKGROUND
RESEARCH LINKING MULTIPLE SOURCES TO ELABORATION AND ELABORATION TO RESISTANCE TO CHANGE
CONCLUSIONS AND IMPLICATIONS FOR FUTURE RESEARCH
REFERENCES
Section II: INDIVIDUAL DIFFERENCES, COGNITIVE
MECHANISMS, AND CONTEXTUAL FACTORS
IN MULTIPLE SOURCE USE
Chapter 6: Individual Differences in Multiple Document Comprehension
THEORETICAL BACKGROUND
COGNITIVE DIFFERENCES
METACOGNITIVE DIFFERENCES
MOTIVATIONAL AND AFFECTIVE DIFFERENCES
SOCIO-CULTURAL DIFFERENCES
INDIVIDUAL DIFFERENCES INTERACT WITH EACH OTHER AND WITH THE TASK CONTEXT
DEVELOPMENT OF INDIVIDUAL DIFFERENCES
EDUCATIONAL IMPLICATIONS
FUTURE DIRECTIONS
REFERENCES
Chapter 7: Potential Processing Challenges of Internet Use Among
Readers with Dyslexia
PURPOSE OF THE CHAPTER
OVERVIEW OF THE CHAPTER
SINGLE SOURCE READING AND DYSLEXIA
MULTIPLE SOURCE USE IN A DIGITAL CONTEXT AND POTENTIAL CHALLENGES FOR READERS WITH DYSLEXIA
EMPIRICAL RESEARCH ON DYSLEXIA AND INTERNET READING
CONCLUSION AND FUTURE RESEARCH
REFERENCES
Chapter 8: Strategic Processing in Accessing, Comprehending, and
Using Multiple Sources Online
GOAL FOR THE CHAPTER
(RE)CONCEPTUALIZING READING WITH MULTIPLE ONLINE SOURCES
STRATEGIC PROCESSING OF MULTIPLE SOURCES ONLINE
HOW STRATEGIC PROCESSING COMPLEMENTS ACCESSING, COMPREHENDING, AND USING MULTIPLE ONLINE SOURCES
IMPLICATIONS FOR THEORY, RESEARCH, AND PRACTICE
CONCLUSION
REFERENCES
Chapter 9: The Role of Validation in Multiple-Document
Comprehension
INTRODUCTION AND PURPOSE
COMPREHENSION AND VALIDATION OF TEXT INFORMATION
A TWO-STEP MODEL OF VALIDATION IN MULTIPLE TEXTS
STEP 1: BELIEF-BASED EPISTEMIC MONITORING OF CONFLICTING INFORMATION
STEP 2: ELABORATIVE PROCESSING OF CONFLICTING INFORMATION
CONSEQUENCES OF VALIDATION FOR COMPREHENSION OUTCOMES
CONTINUED INFLUENCE OF MISINFORMATION, FALSE KNOWLEDGE, AND BELIEFS
BELIEF CONSISTENCY EFFECTS IN MULTIPLE-DOCUMENT COMPREHENSION
IMPLICATIONS FOR THEORY, RESEARCH, AND PRACTICE
CONCLUSION
NOTE
REFERENCES
Chapter 10: Text Relevance and Multiple-Source Use
TEXT RELEVANCE AND GOAL-FOCUSING
TEXT RELEVANCE DIFFERS FROM TEXT IMPORTANCE
TEXT RELEVANCE AND SOURCE CREDIBILITY
TEXT RELEVANCE, SOURCE CREDIBILITY, AND USEFULNESS
CONCLUSION AND FUTURE DIRECTIONS
REFERENCES
Chapter 11: The Role of Conflict in Multiple Source Use
INTRODUCTION
THEORETICAL BACKGROUND
EMPIRICAL WORK
CONCLUSIONS, IMPLICATIONS, AND FUTURE DIRECTIONS
REFERENCES
Section III: MULTIPLE SOURCE USE IN SPECIFIC
CONTENT AREAS
Chapter 12: Multiple Source Use in History
OVERVIEW OF MULTIPLE SOURCE USE IN HISTORY
RESEARCH ON MULTIPLE SOURCE USE IN HISTORY: HISTORIANS
TRANSLATING THIS TO STUDENTS
RESEARCH ON MULTIPLE SOURCE USE IN HISTORY: K-12 STUDENTS
CONCLUSIONS
REFERENCES
Chapter 13: Functional Scientific Literacy: Disciplinary Literacy
Meets Multiple Source Use
HOW SCIENTISTS READ SCIENTIFIC ARTICLES
A FRAMEWORK FOR MULTIPLE SOURCE USE
SCIENTIFIC CONSIDERATIONS IN MULTIPLE SOURCE USE IN SCIENCE
IMPLICATIONS AND FUTURE DIRECTIONS
AUTHOR’S NOTE
NOTE
REFERENCES
Chapter 14: The Role of Sourcing in Mathematics
HOW DO WE KNOW WHICH MATHEMATICAL STATEMENTS ARE TRUE
SOURCING IN MATHEMATICAL PRACTICE
DISCUSSION
NOTES
REFERENCES
Chapter 15: Multiple Source Use When Reading and Writing in
Literature and Language Arts in Classroom Contexts
INTRODUCTION
A READER/WRITER-TEXTS FRAMEWORK OF MULTIPLE SOURCE USE IN READING AND WRITING IN LITERATURE AND LANGUAGE ARTS EDUCATION
A SOCIAL-INTERACTIVE-TEXTS FRAMEWORK OF MULTIPLE SOURCE USE IN READING ANDWRITING IN LITERATURE AND LANGUAGE ARTS EDUCATION
FINAL COMMENTS
NOTES
REFERENCES
Section IV: MULTIPLE SOURCE USE BEYOND THE CLASSROOM
Chapter 16: The Provenance of Certainty: Multiple Source Use and the
Public Engagement with Science
CONCEPTUAL BUILDING BLOCKS FOR A THEORY OF PROVENANCE PROCESSING
WHY SOURCING IS CRUCIAL FOR CITIZENS’ ENGAGEMENT WITH SCIENCE
CHALLENGES FOR CITIZENS’ ENGAGEMENTS WITH SCIENCE
SHOULD I BELIEVE THIS? TWO TYPES OF STRATEGIES FOR JUDGING THE QUALITY OF SCIENTIFIC KNOWLEDGE CLAIMS
THE FUNCTIONS OF PROVENANCE INFORMATION FOR THE PUBLIC ENGAGEMENT WITH SCIENCE: AN EMPIRICAL OVERVIEW
THE ROLE OF SOURCE INFORMATION IN LAYPEOPLE’S EVALUATION OF PLAUSIBILITY
THE ROLE OF SOURCE INFORMATION IN LAYPEOPLE’S UNDERSTANDING OF SCIENTIFIC INFORMATION
FUTURE DIRECTIONS
AUTHORS’ NOTE
REFERENCES
Chapter 17: Non-Academic Multiple Source Use on the Internet
INTRODUCTION
THEORETICAL BACKGROUND
EMPIRICAL FINDINGS
CONCLUSIONS
REFERENCES
Chapter 18: Updating of Character Information When Reading Multiple
Texts for Pleasure
THE CURRENT CHAPTER
MENTAL REPRESENTATIONS OF NARRATIVES
CONSTRUCTING CHARACTER MODELS
UPDATING CHARACTER INFORMATION
INFLUENCES ON UPDATING
THE COMPLEXITY OF CHARACTERS
TO BE CONTINUED . . .
REFERENCES
Chapter 19: Self-Regulated Learning Processes and Multiple Source Use
In and Out of School
THEORETICAL BACKGROUND
REVIEW OF LITERATURE ON SRL AND MULTIPLE SOURCE USE
BEFORE PROCESSES
DURING PROCESSES
AFTER PROCESSES
INTERVENTIONS ON SRL DURING MULTIPLE SOURCE USE
IMPLICATIONS FOR THEORY, RESEARCH, AND PRACTICE
CONCLUSION
REFERENCES
Section V: MULTIPLE SOURCE USE INTERVENTIONS
Chapter 20: Effects of Instructional Conditions on Comprehension from Multiple Sources in History and Science
THEORETICAL BACKGROUND
SUMMARY OF EMPIRICAL RESEARCH
SUMMARY OF RESEARCH FINDINGS, CURRENT CHALLENGES, AND PRACTICAL IMPLICATIONS
AUTHOR NOTE
REFERENCES
Chapter 21: Learning to Read While Reading to Learn: The Central Role
of Multiple Documents in Two Instructional Programs
CONTEXT
WORD GENERATION’S USE OF MULTIPLE TEXTS
STARI’S USE OF MULTIPLE TEXTS
EFFECTIVENESS
IMPLICATIONS FOR INSTRUCTION AND INTERVENTION
ACKNOWLEDGMENTS
REFERENCES
Chapter 22: Promoting Multiple-Text Comprehension Through
Motivation in the Classroom
INTRODUCTION
DIFFERENTIATED COGNITIVE PROCESSES OF READING
ATTRIBUTES OF READING PROCESSES
TEXTUAL CHARACTERISTICS
DIFFERENTIATED MOTIVATIONS RELATED TO READING
ATTRIBUTES OF MOTIVATION PROCESSES
AFFECTIVE SALIENCE
COGNITION–MOTIVATION ALIGNMENT HYPOTHESIS
ASSESSING THE COGNITIVE–MOTIVATION ALIGNMENT HYPOTHESIS
INSTRUCTIONAL PRACTICES RELATED TO THE ALIGNMENT HYPOTHESIS
INSTRUCTION IN LOWER-ORDER READING PROCESSES
INSTRUCTION IN A BALANCE OF LOWER AND HIGHER-ORDER READING PROCESSES
INSTRUCTION FOR HIGHER-ORDER READING PROCESSES
RESEARCH NEEDS
REFERENCES
Chapter 23: Instruction to Promote Information Problem Solving
on the Internet in Primary and Secondary Education:
A Systematic Literature Review
INTRODUCTION
INFORMATION-PROBLEM SOLVING
DESIGN PRINCIPLES TO FOSTER INFORMATION-PROBLEM SOLVING
METHOD
RESULTS
DISCUSSION
REFERENCES
Section VI: ASSESSMENT OF MULTIPLE SOURCE USE
Chapter 24: Complementary Methods for Assessing Online Processing
of Multiple Sources
INTRODUCTION
PROCESSING MULTIPLE SOURCES: WHAT IS INVOLVED?
NOTE-TAKING
THINKING ALOUD
READING TIME
EYE MOVEMENTS
A NOVEL APPROACH: PHYSIOLOGICAL MEASURES
IMPLICATIONS FOR THEORY, RESEARCH, AND PRACTICE
ACKNOWLEDGMENTS
REFERENCES
Chapter 25: Scenario-Based Assessment of Multiple Source Use
PROLOGUE
OVERVIEW OF THE CHAPTER
MULTIPLE SOURCE USE CONSTRUCTS: WHAT IS IT AND HOW CAN WE ASSESS IT?
MD-TRACE MODEL: DECONSTRUCTING THE CONSTRUCT OF MULTIPLE SOURCE USE
FROM COGNITIVE MODEL TO ASSESSMENT DESIGN
THE ASSESSMENT PARADOX: HOW TO MEASURE AND SUPPORT MULTIPLE SOURCE USE
THE GLOBAL, INTEGRATED SCENARIO-BASED ASSESSMENT (GISA) APPROACH
KEY FEATURES OF GISA
HOW THE FEATURES ARE ASSEMBLED IN AN ASSESSMENT CONTEXT
ILLUSTRATING MULTIPLE SOURCE USE THROUGH A GISA EXAMPLE: CONNECTING THE MD-TRACE MODEL TO ASSESSMENT
SECTION 1: SETTING UP THE TASK MODEL – WHAT ARE STUDENTS SUPPOSED TO DO AND PRODUCE?
SECTION 2: MEASURING BACKGROUND KNOWLEDGE – WHAT DO STUDENTS KNOW ABOUT THE TOPIC?
SECTION 3: BUILDING UP STUDENTS’ UNDERSTANDING – SINGLE-SOURCE COMPREHENSION
SECTION 4: EVALUATING WEB LINKS – ASSESS ITEM RELEVANCE
SECTION 5: GAINING A DEEPER UNDERSTANDING OF THE TOPIC – UPDATE TASK MODEL AND CREATE A DOCUMENTS MODEL
SECTION 6: OPPOSING VIEWPOINTS AND COUNTERARGUMENT– UPDATE DOCUMENTS MODEL
SECTION 7: PRODUCE A FLYER – CREATE A TASK PRODUCT
PROPERTIES OF GISA
APPLICATIONS OF GISA
OTHER SCENARIO-BASED ASSESSMENT RESEARCH PROGRAMS
IMPLICATIONS FOR RESEARCH AND PRACTICE
SUMMARY
ACKNOWLEDGMENTS
REFERENCES
Chapter 26: Assessment of Multiple Resource Comprehension and
Information Problem Solving
ASSESSMENT: A PROCESS OF REASONING FROM EVIDENCE
WHAT TO ASSESS AND HOW TO ASSESS IT
IMPLICATIONS FOR THE DESIGN OF ASSESSMENTS
FURTHER READING
REFERENCES
Chapter 27: Assessing Online Collaborative Inquiry and Social
Deliberation Skills as Learners Navigate Multiple
Sources and Perspectives
THEORETICAL BACKGROUND
COLLABORATIVE LEARNING
SOCIAL DELIBERATION
ASSESSMENTS OF ONLINE INQUIRY, COLLABORATION, AND SOCIAL DELIBERATION
ASSESSMENTS OF ONLINE INQUIRY
ASSESSMENTS OF COLLABORATIVE PROBLEM SOLVING
ASSESSMENTS OF ONLINE COLLABORATIVE INQUIRY AND SOCIAL DELIBERATION
IMPLICATIONS FOR THEORY, RESEARCH, AND PRACTICE
REFERENCES
Chapter 28: Computer-Based Assessment of Essays Based on
Multiple Documents: Evaluating the Use of Sources
THEORETICAL PERSPECTIVES ON WHAT SHOULD BE ASSESSED
DIMENSIONS OF ESSAYS THAT SHOULD BE EVALUATED
WHAT IS NEEDED TO EVALUATE ESSAYS BASED ON MULTIPLE TEXTS
PROMISING APPROACHES IN AUTOMATED ESSAY ASSESSMENT
COMPUTATIONAL TOOLS FOR ANALYZING ESSAY CONTENT
EXAMPLE STUDIES USING NLP TO STUDY MULTIPLE DOCUMENTS PROCESSING
CHALLENGES IN DEVELOPING AUTOMATED ESSAY EVALUATION SYSTEMS
CONCLUSIONS AND FUTURE DIRECTIONS
REFERENCES
Chapter 29: Reflections and Future Directions
REFLECTING ON CORE QUESTIONS POSED IN CHAPTER 1
DIRECTIONS FOR FUTURE RESEARCH
CONCLUSION
REFERENCES
Index
Recommend Papers

Handbook of Multiple Source Use
 9781138646599, 9781138646605, 9781315627496

  • 0 0 0
  • Like this paper and download? You can publish your own PDF file online for free in a few minutes! Sign Up
File loading please wait...
Citation preview

Handbook of Multiple Source Use

The Handbook of Multiple Source Use draws on theory and research within cognitive and educational psychology, the learning sciences, disciplinary education, information literacy, reading psychology, and social psychology, to present the first comprehensive research volume on this topic. Many learners both in and out of school have almost instantaneous access to an enormous range of information sources at present. In this book, broken into six sections, international scholars come together toward understanding factors that influence how individuals cope with the challenge of building knowledge from diverse, often conflicting, information. Jason L. G. Braasch is Assistant Professor of Psychology and Affiliate of the Institute for Intelligent Systems at the University of Memphis, USA. Ivar Bråten is Professor of Educational Psychology at the University of Oslo, Norway. Matthew T. McCrudden is Associate Professor of Educational Psychology at Victoria University of Wellington, New Zealand.

Educational Psychology Handbook Series Series Editor: Patricia A. Alexander

International Handbook of Emotions in Education Edited by Reinhard Pekrun and Lisa Linnenbrink-Garcia International Handbook of Research on Teachers’ Beliefs Edited by Helenrose Fives and Michelle Gregoire Gill Handbook of Test Development, 2nd Edition Edited by Suzanne Lane, Mark R. Raymond, and Thomas M. Haladyna Handbook of Social Influences in School Contexts: Social-Emotional, Motivation, and Cognitive Outcomes Edited by Kathryn R. Wentzel and Geetha B. Ramani Handbook of Epistemic Cognition Edited by Jeffrey A. Greene, William A. Sandoval, and Ivar Bråten Handbook of Motivation at School, 2nd Edition Edited by Kathryn R. Wentzel and David B. Miele Handbook of Human and Social Conditions in Assessment Edited by Gavin T. L. Brown and Lois R. Harris Handbook of Quantitative Methods for Detecting Cheating on Tests Edited by Gregory J. Cizek and James A. Wollack Handbook of Research on Learning and Instruction, 2nd Edition Edited by Patricia A. Alexander and Richard E. Mayer Handbook of Self-Regulation of Learning and Performance, 2nd Edition Edited by Dale H. Schunk and Jeffrey A. Greene Handbook of Multiple Source Use Edited by Jason L. G. Braasch, Ivar Bråten, and Matthew T. McCrudden

Handbook of Multiple Source Use

Edited by Jason L. G. Braasch, Ivar Bråten, and Matthew T. McCrudden

First published 2018 by Routledge 711 Third Avenue, New York, NY 10017 and by Routledge 2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN Routledge is an imprint of the Taylor & Francis Group, an informa business  2018 Taylor & Francis The right of Jason L. G. Braasch, Ivar Bråten, and Matthew T. McCrudden to be identified as the authors of the editorial material, and of the authors for their individual chapters, has been asserted in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988. All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. Trademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Library of Congress Cataloging-in-Publication Data A catalog record for this book has been requested ISBN: 978-1-138-64659-9 (hbk) ISBN: 978-1-138-64660-5 (pbk) ISBN: 978-1-315-62749-6 (ebk) Typeset in Minion Pro by Swales & Willis Ltd, Exeter, Devon, UK

Dedication Jason L. G. Braasch dedicates his work on this volume to his wife Claire Gaynor. Many hours were necessary to complete this work; her love, support, and understanding made it possible. •• Ivar Bråten dedicates his work on this volume to his dear partner, Ann Kristin Heller, and his son, Jørgen Bråten, both reliable sources of a meaningful life. •• Matthew T. McCrudden dedicates his work on this volume to his wife, Sally, and two sons, Arthur and Hugh, for their ongoing support. ••

CONTENTS

Acknowledgments xi Chapter 1

Introduction to Research on Multiple Source Use

1

JASON L. G. BRAASCH, IVAR BRÅTEN, AND MATTHEW T. MCCRUDDEN

Section I

THEORETICAL FRAMEWORKS

15

Chapter 2

Representations and Processes in Multiple Source Use

17

M. ANNE BRITT, JEAN-FRANÇOIS ROUET, AND AMANDA DURIK

Chapter 3

Cold and Warm Perspectives on the Cognitive Affective Engagement Model of Multiple Source Use

34

ALEXANDRA LIST AND PATRICIA A. ALEXANDER

Chapter 4

Toward a New Literacies Perspective of Synthesis: Multiple Source Meaning Construction

55

DOUGLAS K. HARTMAN, MICHELLE SCHIRA HAGERMAN, AND DONALD J. LEU

Chapter 5

A Social Psychological Perspective on Multiple Source Use: Elaboration and Persuasion

79

DUANE T. WEGENER, KATHLEEN M. PATTON, AND CURTIS P. HAUGTVEDT

Section II

Chapter 6

INDIVIDUAL DIFFERENCES, COGNITIVE MECHANISMS, AND CONTEXTUAL FACTORS IN MULTIPLE SOURCE USE

97

Individual Differences in Multiple Document Comprehension

99

SARIT BARZILAI AND HELGE I. STRØMSØ

vii

viii  • Contents

Chapter 7

Potential Processing Challenges of Internet Use Among Readers with Dyslexia

117

ØISTEIN ANMARKRUD, EVA WENNÅS BRANTE, AND ANETTE ANDRESEN

Chapter 8

Strategic Processing in Accessing, Comprehending, and Using Multiple Sources Online

133

BYEONG-YOUNG CHO, PETER AFFLERBACH, AND HYEJU HAN

Chapter 9

The Role of Validation in Multiple-Document Comprehension 151 TOBIAS RICHTER AND JOHANNA MAIER

Chapter 10 Text Relevance and Multiple-Source Use

168

MATTHEW T. MCCRUDDEN

Chapter 11 The Role of Conflict in Multiple Source Use

184

IVAR BRÅTEN AND JASON L. G. BRAASCH

Section III MULTIPLE SOURCE USE IN SPECIFIC CONTENT AREAS

203

Chapter 12 Multiple Source Use in History

205

EMILY FOX AND LILIANA MAGGIONI

Chapter 13 Functional Scientific Literacy: Disciplinary Literacy Meets Multiple Source Use

221

IRIS TABAK

Chapter 14 The Role of Sourcing in Mathematics

238

KEITH WEBER

Chapter 15 Multiple Source Use When Reading and Writing in Literature and Language Arts in Classroom Contexts

254

DAVID BLOOME, MINJEONG KIM, HUILI HONG, AND JOHN BRADY

Section IV MULTIPLE SOURCE USE BEYOND THE CLASSROOM

267

Chapter 16 The Provenance of Certainty: Multiple Source Use and the Public Engagement with Science

269

RAINER BROMME, MARC STADTLER, AND LISA SCHARRER

Chapter 17 Non-Academic Multiple Source Use on the Internet LADISLAO SALMERÓN, YVONNE KAMMERER, AND PABLO DELGADO

285

Contents  •  ix

Chapter 18 Updating of Character Information When Reading Multiple Texts for Pleasure

303

AMALIA M. DONOVAN AND DAVID N. RAPP

Chapter 19 Self-Regulated Learning Processes and Multiple Source Use In and Out of School

320

JEFFREY ALAN GREENE, DANA Z. COPELAND, VICTOR M. DEEKENS, AND REBEKAH FREED

Section V

MULTIPLE SOURCE USE INTERVENTIONS

Chapter 20 Effects of Instructional Conditions on Comprehension from Multiple Sources in History and Science

339 341

JENNIFER WILEY, ALLISON J. JAEGER, AND THOMAS D. GRIFFIN

Chapter 21 Learning to Read While Reading to Learn: The Central Role of Multiple Documents in Two Instructional Programs

362

LOWRY HEMPHILL AND CATHERINE SNOW

Chapter 22 Promoting Multiple-Text Comprehension Through Motivation in the Classroom

382

JOHN T. GUTHRIE

Chapter 23 Instruction to Promote Information Problem Solving on the Internet in Primary and Secondary Education: A Systematic Literature Review

401

SASKIA BRAND-GRUWEL AND JOHAN L.H. VAN STRIEN



Appendix 23.1

415

Section VI ASSESSMENT OF MULTIPLE SOURCE USE

423

Chapter 24 Complementary Methods for Assessing Online Processing of Multiple Sources

425

LUCIA MASON AND ELENA FLORIT

Chapter 25 Scenario-Based Assessment of Multiple Source Use

447

JOHN SABATINI, TENAHA O’REILLY, ZUOWEI WANG, AND KELSEY DREIER

Chapter 26 Assessment of Multiple Resource Comprehension and Information Problem Solving SUSAN R. GOLDMAN, ALYSSA BLAIR, AND CANDICE M. BURKETT

466

x  • Contents

Chapter 27 Assessing Online Collaborative Inquiry and Social Deliberation Skills as Learners Navigate Multiple Sources and Perspectives

485

JULIE COIRO, JESSE R. SPARKS, AND JONNA M. KULIKOWICH

Chapter 28 Computer-Based Assessment of Essays Based on Multiple Documents: Evaluating the Use of Sources

502

JOSEPH P. MAGLIANO, PETER HASTINGS, KRISTOPHER KOPP, DYLAN BLAUM, AND SIMON HUGHES

Chapter 29 Reflections and Future Directions

527

JASON L. G. BRAASCH, MATTHEW T. MCCRUDDEN, AND IVAR BRÅTEN

Index 539

ACKNOWLEDGMENTS

The Handbook of Multiple Source Use is the result of the vision and dedication of many people. We give special thanks to Patricia Alexander, as Editor for the Educational Psychology Handbook Series by Routledge, for inviting us to edit this volume. We are grateful for her enthusiasm and guidance in creating the Handbook. We would also like to thank Routledge Editor Daniel Schwartz for his advice, support, and encouragement throughout the process. We extend our gratitude to the many scholars who served as reviewers for the chapters including: Øistein Anmarkrud, Sarit Barzilai, M. Anne Britt, Inmaculada Fajardo, Leila Ferguson, Emily Fox, Jeffrey Greene, CarolAnne Kardash, Carita Kiili, Alex List, Doug Lombardi, Mônica Macedo-Rouet, Lucia Mason, Katie McCarthy, Jeffery Nokes, Edward O’Brien, Yasuhiro Ozuru, Alina Reznitskaya, Tobias Richter, Ladislao Salmerón, Marc Stadtler, Helge I. Strømsø, Gregory Trevors, Meng-Jung Tsai, Peggy van Meter, Johan van Strien, and Mike Wolfe. We would like to thank the many scholars who served as chapter authors. The success of this volume is owed to their commitment to a critical analysis of the field of research on multiple source use, while also benefitting greatly from their suggestions for future research and practice in the years to come. Finally, thanks go to the University of Memphis (USA), University of Oslo (Norway), and Victoria University of Wellington (New Zealand) for affording the time and resources for developing and producing the current volume.

xi

1

INTRODUCTION TO RESEARCH ON MULTIPLE SOURCE USE Jason L. G. Braasch university of memphis, usa

Ivar Bråten university of oslo, norway

Matthew T. McCrudden victoria university of wellington, new zealand

INTRODUCTION As consumers of information, we live in unprecedented times, especially given the centrality of the Internet in our daily lives. We have rapid access to a staggering amount of information from diverse information sources. This is in sharp contrast to previous generations that had to physically seek out information from a relatively narrow set of source types to be informed about different topics (e.g., reading an excerpt from an encyclopedia at a library to learn about a historical event, purchasing a newspaper from a local store to keep up with recent news), or perhaps rely on the evening news to stay informed. The Information Age has – of course – changed how people access, use, and understand information, which provides us with important opportunities. We can easily and almost instantaneously retrieve up-to-date information from a broad spectrum of sources that provide us with a rich body of information covering the gamut of what we would want or need to know about anything, from key information about important issues to subtle minutiae. At the same time, information consumers must face and overcome many new challenges. Readers in the current Information Age need traditional reading skills, such as the ability to decode the meaning of words and understand the gist of a sentence. However, they also need to be able to acquire

1

2  •  Braasch et al.

and flexibly use a diverse set of knowledge, skills, and dispositions to guide their selection, processing, and use of information from multiple information sources (Alexander & The Disciplined Reading and Learning Research Laboratory, 2012; Brand-Gruwel & Stadtler, 2011; Bråten, Braasch, & Salmerón, in press; Goldman & Scardamalia, 2013; OECD, 2013). This chapter provides an introduction to research on multiple source use. It is important to note that, across the chapters in the Handbook, there is some variability in how different authors use the term source, and – as such – what is meant by multiple source use. In a broader sense, the term source is considered synonymous and used interchangeably with terms like information resource, text, document, or multimedia resource (e.g., those including text alongside photos or videos). In this way, the term source reflects a body of information that is distinct and demarcated from other bodies of information (Wiley, Jaeger, & Griffin, this volume). In a more restricted and different sense, others use the term source with reference to metadata information embedded within or provided outside the body of information, including its origin, context, and purpose, to name but a few (Barzilai & Strømsø, this volume; Britt, Rouet, & Braasch, 2013). Accordingly, what is meant by multiple source use necessarily differs depending on how one is defining source. In the former, broader sense, multiple source use refers to an individual’s ability to construct meaning from multiple bodies of information through engagement in a broad range of processes. These can include: interpreting the task, locating, selecting, analyzing, evaluating, comprehending, and transforming information, while also corroborating, integrating, and constructing ideas within and across sources (Barzilai & Strømsø, this volume; Gil, Bråten, Vidal-Abarca, & Strømsø, 2010; Goldman & Scardamalia, 2013). In the latter, more narrow sense, multiple source use refers to an individual’s propensity to attend to, represent, evaluate, and apply available or accessible metadata features embedded within or provided outside bodies of information, e.g., reading and evaluating “About us” information on a website to determine whether the author is biased in making a specific claim (Bråten & Braasch, this volume; Bråten, Stadtler, & Salmerón, in press). All told, the next section presents research on multiple source use in both senses of the term, from the more general to the more specific. Importantly, several authors in this area of research argue that multiple source use – in both senses of the term – interact in constructing a more complete understanding of a situation, topic, or phenomenon (e.g., Britt, Rouet, & Braasch, 2013).

RESEARCH ON MULTIPLE SOURCE USE As information has proliferated and as information sources have diversified over the past several decades, so too has research on multiple source use. Seminal, early studies on multiple source use in academic contexts (Hartman, 1995; Perfetti, Britt, & Georgi, 1995; Rouet, Britt, Mason, & Perfetti, 1996; Stahl, Hynd, Britton, McNish, & Bosquet, 1996; Wineburg, 1991) have paved the way for contemporary investigations focused on understanding multiple source use within both academic and non-academic settings. This ever-expanding research literature has provided theories of multiple source use, rich descriptions and explanations of readers’ engagement in activities that promote or constrain their use of multiple sources, and prescriptions for designing educational interventions that target multiple source use.

Introduction to Multiple Source Use  •  3

The remaining chapters in this volume provide an extensive review of theory and research on multiple source use; in this chapter we introduce several competencies that seem particularly important in today’s information-rich society, many of which interact in complex ways. For example, when individuals initiate an inquiry on the Internet, they must be able to locate and evaluate information returned by search engines. Such activities include planning what search terms will be used, scanning and skimming the search results to identify texts that may prove useful, monitoring the status of what useful texts were found relative to the reader’s goals and threshold for task completion, and revising search terms to get a better set of results (Braasch et al., 2009; Brand-Gruwel, Kammerer, van Meeuwen, & van Gog, in press; Greene, Yu, & Copeland, 2014; Kammerer, Bråten, Gerjets, & Strømsø, 2013; Salmerón, Kammerer, & García-Carrión, 2013). When accessing full texts, more effective multiple source users evaluate information based on several dimensions including quality, authenticity, relevance, and reliability, all through the lens of an individual’s current reading goals (Bråten, McCrudden, Stang Lund, Brante, & Strømsø, in press; McCrudden & Schraw, 2007). For example, people can validate information they come across based on what they already believe and know, including what they may have just read (Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012; O’Brien & Cook, 2016; Richter & Maier, 2017; Stadtler & Bromme, 2014). This validation process could result in confirmation if similar or complementary ideas were previously encountered, or in noticing that new information directly conflicts with prior beliefs or knowledge (Braasch & Bråten, 2017; Braasch, McCabe, & Daniel, 2016; Stadtler, Scharrer, Brummernhenrich, & Bromme, 2013). Furthermore, we often interact with multiple sources that contain a mixture of information, only some of which is relevant given our current inquiry. In this sense, readers must also be equipped with the skills to determine the extent to which information is relevant to their current reading goals (Anmarkrud, McCrudden, Bråten, & Strømsø, 2013; Bråten et al., in press; McCrudden, Stenseth, Bråten, & Strømsø, 2016). People must also consider text features that can help them to determine the reliability of information they encounter (Britt, Rouet, & Braasch, 2013; Thomm & Bromme, 2012). If readers use information sources from the Internet, for example, they must be aware that such information may not have undergone any explicit review policies or quality control compared to most paper-based publications. Prior to the arrival of the Internet, readers could defer to information providers, such as editors and publishing companies, in reducing the amount of unreliable information they may have encountered. However, the current information boom has shifted the responsibility of evaluating the reliability of information to the readers themselves, including the credibility of the sources. Thus, effective, efficient multiple source use requires that individuals incorporate sourcing strategies into their daily reading routines. These strategies help people to focus information processing efforts on information that is reliable and from more credible sources, and to more actively scrutinize, disregard, or actively refute information that is less reliable or from less credible sources. Thus, reliability evaluations may stem from two separate, but related facets of texts. First, one can establish reliability based on a more critical analysis of the semantic content provided by information sources. For example, a more critical person could determine a text’s content is less reliable if it offers a claim

4  •  Braasch et al.

with no supporting evidence, or the claim outstrips the available data (e.g., a strong causal claim using correlational data as support). Second, one can establish reliability based on a more critical analysis of the source features that are available. For example, a more critical person might use source features, such as an author’s credentials, to infer bias in presenting a claim. Similarly, an individual might deduce that the publication venue has ulterior motives in posting some information on their website. Thus, sourcing, in particular, reflects a complex set of competencies that include attending to, representing, evaluating, and using features of information sources (Bråten, Stadtler, & Salmerón, in press). Source features can include information about the author’s credentials (or lack thereof), the type and date of publication, and publication genre (e.g., a personal blog versus an online magazine article), to name but a few. Whereas these features surely help effective readers determine the reliability of content presented within multiple sources, the Internet provides for additional sourcing challenges. Source features that are typically available in printed texts may be masked, unavailable, or – at times – extremely hard to interpret on many websites (Britt & Gabrys, 2000; Flanagin & Metzger, 2008). It is also common that the complete answer to an individual’s question requires an integration of relevant information that is distributed across multiple sources (Cho & Afflerbach, 2017). In this sense, a number of processes may be involved including elaboration of complementary ideas, and noticing and rectifying experiences of cognitive conflicts, to name but a few. Related to this point, effective readers tend to monitor the status of a mental model they are gradually constructing during reading (Goldman et  al., 2012). When the learner has successfully met their subjective threshold for acceptability (List & Alexander, 2017) and satisfied their goals for reading (McCrudden & Schraw, 2007), the task is considered completed. But what if a reader doesn’t have the requisite knowledge, skills, and dispositions to effectively engage with multiple sources? Alternatively, what if they lack motivation to do so? If this is the case, several problems can arise. For example, readers may experience “information overload” if they are unable or unwilling to distinguish between more and less relevant information to satisfy their purposes for reading. Similarly, if readers are unable or unwilling to focus their information processing efforts toward more reliable, higher-quality information provided by credible sources, they may continue to endorse misconceptions, or create new ones. Moreover, if readers do not strategically integrate relevant concepts into a coherent mental representation, they may come away with an assortment of unconnected facts and concepts (Goldman, 2004). This brief introduction offers an overview of the complexities involved in multiple source use. Coordinating these component processes is no small feat. Multiple source use demands a great deal of cognitive resources and adeptness in knowing when, how, and why one should engage in these different mental activities at the various stages of inquiry outlined above. To date, a significant body of research has emerged that reflects major advances toward theory development and empirical investigations of multiple source use, as well as tests of educational practices designed to support it. The following chapters, and the larger literature base from which they are drawn, were gathered to connect ideas offered by the different disciplines studying multiple source use.

Introduction to Multiple Source Use  •  5

GOALS OF THIS HANDBOOK We had two main goals for this Handbook of Multiple Source Use. One goal was to provide a comprehensive survey of research to describe when, how, and why readers use multiple sources. For the first time, theoretical advancements and empirical investigations of multiple source use in both formal and informal contexts are covered in a single volume. Author contributions focus on the theoretical and practical implications of the work reviewed, describe current challenges in the area, and offer recommendations for future research. Thus, each chapter provides substantial depth that readers may seek within their particular topics of interest. In addition to depth, our second goal was to organize the Handbook in a way that affords opportunities for cross-fertilization across disciplines, broadening the readers’ knowledge base with respect to key issues that cut across the various themes. Colloquially speaking, researchers interested in multiple source use tend to, for the most part, “stay in their lane.” That is, we tend to investigate our respective research questions through use of our preferred terminology, use our favorite paradigms, work with a small set of similarly minded collaborators, attend the same conferences and publish in similar journals, and so forth. This Handbook represents an authentic opportunity to increase conceptual clarity, helping to identify what we collectively mean when we talk about multiple source use. Similarly, the timing is right for a volume to provide more coherent understandings of how different lines of research meaningfully relate to – and potentially inform – one another. To address these various goals, the Handbook of Multiple Source Use is necessarily interdisciplinary in nature, drawing on theory and research within cognitive and educational psychology, the learning sciences, disciplinary education (i.e., history, science, mathematics, and language arts education), information literacy, reading psychology, and social psychology. All of these areas of research are undeniably important in understanding multiple source use. It should also be noted that the Handbook’s contributors represent a large group of international scholars from several continents and many different countries. The volume is broadly focused on multiple source use research from various regions around the world. Thus, solicited chapters represent broad, international perspectives of multiple source use as a field of inquiry. As such, we believe this Handbook serves as a means to promote dialogue between scholars from different disciplines and perspectives relevant to multiple source use. The hope is that such a focus will not only inspire more integrated research efforts among established scholars in the field, but will also help newcomers launch innovative and integrative research projects that cut across previous divides, contributing to greater consilience across various disciplines. All told, the Handbook provides both seasoned scholars and novices in the field with invaluable new insights into multiple source use, with the expectation that it will inspire future research for years to come.

THEMES OF THIS HANDBOOK Several themes permeate the current state of the field on multiple source use. Below, we outline core questions as a means of summarizing some overarching themes that are woven into the fabric of this Handbook. Later, we address these questions in the final chapter.

6  •  Braasch et al.

1. How do theories describe macro- and micro-level processes of multiple source use? What particular cognitive and non-cognitive activities guide individuals as they construct understandings from multiple sources? 2. What individual characteristics support (or constrain) multiple source use? 3. Are there discipline-specific (e.g., history versus science learning) and domaingeneral multiple source use processes? 4. How do people use multiple sources in non-academic contexts? In what ways are the processes similar to or different from those used in academic contexts? 5. What types of tasks and interventions promote successful multiple source use? 6. Finally, how can different kinds of assessments provide insight into the component processes and products of multiple source use?

OVERVIEW OF THE SECTIONS AND CHAPTERS Informed by these overarching core questions, we solicited contributions from a team of scholars that were organized into six broad themes. Section I: Theoretical Frameworks The first section of the Handbook includes formalized presentations and discussions of theoretical frameworks, models, and perspectives relevant to the field of multiple source use. The chapters in this section explore multiple source use from diverse perspectives. All four theories, however, describe macro- as well as micro-level processes underlying multiple source use. In Chapter 2, Britt, Rouet, and Durik, writing from a cognitive psychological perspective, focus on mental representations and psychological processes that underlie multiple source use. They describe their recent REading as Problem SOLVing (RESOLV) model of reading comprehension (see also Rouet, Britt, & Durik, 2017), which is derived from and extends their previous Multiple-Document Task-based Relevance Assessment and Content Extraction (MD-TRACE) model (Rouet & Britt, 2011). The RESOLV model highlights the kinds of cognitive representations people construct, and the processes in which they engage to support multiple source reading experiences. They then use the model to explain key results from two illustrative cases of multiple source use. In Chapter 3, List and Alexander introduce the Cognitive Affective Engagement Model of multiple source use (CAEM), which is grounded in a cognitive perspective. Their model conceptualizes multiple source use as a function of the cognitive processes in which readers engage, and their motivational and affective experiences during multiple source use. Thus, they use their model as a framework to relate cognitive and affective factors to a range of multiple source use behaviors including text access, processing and evaluation, and task cessation. In Chapter 4, Hartman, Hagerman, and Leu explore how readers construct meaning from multiple online sources adopting a “New Literacies” perspective. At the heart of this literacy-focused perspective is the idea that online contexts present special and unique informational spaces for meaning construction. Their review is organized around five elements that are viewed as important determinants of readers’ synthesis of information from multiple sources including purpose, context, texts, readers, and technologies.

Introduction to Multiple Source Use  •  7

In Chapter 5, Wegener, Patton, and Haugtvedt provide a detailed review of multiple source use from a social psychological perspective. Their focus on elaboration and persuasion provides insight into the extent to which (and the processes by which) multiple sources influence people’s pre-existing attitudes and beliefs on a topic. They use two primary social psychological theories to organize the research on persuasion: the Elaboration Likelihood Model (Petty & Cacioppo, 1986) and the Discrepancy Motives Model (Clark & Wegener, 2013). Section II: Individual Differences, Cognitive Mechanisms, and Contextual Factors in Multiple Source Use The second section includes reviews of research and connections among three areas. First, several chapters provide updated, comprehensive overviews of the contributions of individual differences to multiple source use. These include constructs that are believed to be static (e.g., working memory) as well as more dynamic factors (e.g., reading strategies). Additional chapters provide descriptions and explanations of cognitive mechanisms and contextual factors that drive multiple source use. In Chapter 6, Barzilai and Strømsø comprehensively review the current state of research on individual differences that contribute to multiple source use. These include cognitive and metacognitive, as well as motivational, affective, and socio-cultural differences. The chapter additionally discusses the implications of these individual difference characteristics for future educational research and practice. In Chapter 7, Anmarkrud, Brante, and Andresen review research on multiple sources for readers diagnosed with dyslexia. The authors describe how readers with dyslexia differ from typically developing readers, and the additional challenges such readers face in the context of multiple source use. Then, the authors review relevant literature that illustrates some of the challenges and affordances of Internet reading and multiple source use for readers with dyslexia. Given the dearth of research focusing on readers with dyslexia, the authors provide several avenues of future research for supporting multiple source use with these readers. In Chapter 8, Cho, Afflerbach, and Han discuss the role of strategic processing when readers access, comprehend, and use multiple sources for some purpose. The authors conceptualize reading from multiple sources as actively constructing meaning by creating links between sources, evaluating the contribution of each source to one’s understanding of the topic, and considering the credibility of sources when determining the value of the information conveyed. They propose a framework that consists of three layers of strategic processing across which readers’ cognitive and metacognitive strategies mutually interact during the construction of meaning. The framework is illustrated with empirical examples. In Chapter 9, Richter and Maier focus on validation (Singer, 1993) or epistemic monitoring (Isberner & Richter, 2014) (terms are used interchangeably) as a cognitive mechanism underlying multiple source use. In particular, the authors propose a two-stage model. In a first stage, people routinely make judgments about information plausibility when dealing with multiple sources. In doing so, belief-consistent information has a processing advantage over belief-inconsistent information in comprehension and memory. Controversies experienced during the initial validation stage may enact a second stage involving more elaborative processing, and – as a result – a more balanced mental representation of the controversial information.

8  •  Braasch et al.

In Chapter 10, McCrudden extends a discussion about the construct of text relevance, the perceived instrumental value of text information for meeting a goal for reading, to the context of multiple source use. The chapter begins by describing how text relevance has historically been investigated in the context of single-author text with little or no focus on the source. Then, research on text relevance in multiple source use is described with a particular emphasis on the joint roles that text relevance and source credibility play in the selection, processing, and use of information from different sources and how these constructs may affect the perceived usefulness of information. In Chapter 11, Bråten and Braasch focus on noticing (and attempting to resolve) discrepancies as a cognitive mechanism underlying multiple source use. Two relevant frameworks for understanding the role of conflict in multiple source use are discussed: the Discrepancy-Induced Source Comprehension assumption (Braasch & Bråten, 2017) and the Plausibility-Induced Source Focusing assumption (de Pereyra, Britt, Braasch, & Rouet, 2014). They review important empirical work framed by these assumptions, and research addressing the roles that individual and contextual factors play when dealing with multiple conflicting sources. Section III: Multiple Source Use in Specific Content Areas The third section addresses multiple source use from the perspectives of specific content areas. The authors in this section demonstrate that multiple source use is highly relevant in the content areas of history, science, mathematics, and language arts. Educational implications of research on multiple source use in disciplinary contexts are also highlighted in this section. In Chapter 12, Fox and Maggioni focus on the nature of history and the use of multiple sources in history by historians and students. In particular, Fox and Maggioni review the empirical literature on K-12 students’ multiple source use in history, both when students work on their own and in the contexts of instructional interventions. These authors also discuss the educational implications of the existing body of research and offer directions for future research. In Chapter 13, Tabak focuses on how scientists read science and the challenges laypeople typically encounter when using multiple scientific sources. In particular, how laypeople deal with online science news and scientific informational texts is discussed in light of the more sophisticated strategies displayed by scientists using multiple scientific texts. Based on this discussion, Tabak argues that efforts to enhance functional scientific literacy among laypeople should pay attention to how scientists use multiple scientific sources. In Chapter 14, Weber describes how attention to source information, such as authors and publication venues, may come into play when mathematicians judge mathematical statements and proofs. In particular, he discusses how the status of the author, the reputation of the publication, and the fame or importance of the finding may influence mathematicians’ confidence in statements and proofs. Weber also compares the role of sourcing in mathematics and other disciplines and discusses the implications of his analyses for mathematics education. In Chapter 15, Bloome, Kim, Hong, and Brady aim to advance the theorizing of multiple source use when reading and writing in literature and language arts

Introduction to Multiple Source Use  •  9

classroom contexts. The authors distinguish between two theoretical frameworks; one primarily focusing on individual, decontextualized cognitive and linguistic processes involved in multiple source use, and one primarily focusing on multiple source use as social and cultural practices. They present cases to illustrate each framework and highlight differences between the two frameworks and their implications. Section IV: Multiple Source Use Beyond the Classroom Section IV addresses issues of multiple source use beyond academic settings. The authors of these chapters discuss how novices deal with multiple sources on scientific issues, the challenges of comprehension in digital environments, and characters in fictional works, as well as the role self-regulation plays when using multiple sources in and out of school settings. Further, these authors describe how research in formal settings can be extended to inform multiple source use outside of the classroom context. In Chapter 16, Bromme, Stadtler, and Scharrer invite researchers to view multiple source use through a wider lens. In much the same way that painting is evaluated on the basis of its full history such as who painted it and when, sources and the information they provide should be evaluated more broadly to include information about a source and how that broader context facilitates establishing meaning. Against this backdrop, the authors discuss how non-experts rely upon and use source information when evaluating science. In Chapter 17, Salmerón, Kammerer, and Delgado focus on readers’ use of multiple online sources to seek information to make informed decisions that are important to everyday life, or reading scenarios that occur outside of formal educational settings, such as personal and medical issues. They discuss models related to understanding non-academic use of multiple sources on the Internet. Then, they review empirical research on how readers locate and access information, how they approach reading when the goal is to be informed, and how they engage in reading for making decisions. In Chapter 18, Donovan and Rapp focus on the processes that affect readers’ experiences with story characters when they read multiple fictitious texts, as well as the products that emerge as a function of those processes. A crucial process involved in reading multiple fictitious texts is updating, which refers to constructing, adding to, or revising existing character representations that are encoded and retained in memory. Not only does this chapter provide meaningful insights into how readers process and remember information across multiple fictional texts, it also offers a unique lens for reflecting on the role that updating may play in multiple source use with non-fiction. In Chapter 19, Greene, Copeland, Deekens, and Freed discuss the role of selfregulated learning in the context of multiple source use, both in and out of formal educational settings. Importantly, models of self-regulation provide a framework for investigating and understanding not only how readers engage with multiple sources during reading, but also what readers do before and after reading. The authors focus on how processes that occur before, during, and after reading can affect multiple source use and review research on self-regulated learning interventions in multiple source use.

10  •  Braasch et al.

Section V: Multiple Source Use Interventions The fifth section discusses interventions to promote efficient, effective multiple source use. These authors review intervention work designed to promote student success in coping with the challenges of multiple information sources in different content areas and when using the Internet. Across the four chapters, this body of work testifies to the educational applicability of theory and research in the field of multiple source use. In Chapter 20, Wiley, Jaeger, and Griffin provide an extensive review of research on the effects of the inquiry task, the task environment, and the instructional context on multiple source comprehension in history and science. In doing this, the authors also discuss similarities and differences between the two content areas. The intervention work reviewed in this chapter represents a solid basis for future efforts to promote adaptive use of multiple sources in educational contexts. In Chapter 21, Hemphill and Snow describe multiple source use in two instructional programs designed to promote literacy development in primary/elementary school children and younger adolescents. In particular, these authors discuss the curricular materials developed for the two programs in great detail, as well as how those materials were used in reading tasks embedded in discussions about issues relevant to students’ lives. In conclusion, they discuss the effectiveness of the two literacy programs and reflect on their implications for instruction and intervention to promote literacy development. In Chapter 22, Guthrie focuses on the relationships between motivational and cognitive processes in reading. Guthrie argues that some forms of reading motivation are better aligned with complex reading comprehension, including multiple-text comprehension, than others. Accordingly, attempts to promote students’ multiple-text comprehension through motivation should consider the forms of motivation that can be assumed to underlie multiple-text comprehension in classroom contexts. In Chapter 23, Brand-Gruwel and van Strien review intervention work that aims to promote information-problem solving among elementary and secondary school students, especially when students use the Internet to search for information. In particular, they examine the effects of these interventions and the extent to which researchers have used instructional design principles when designing their interventions. Similarities and differences between intervention studies conducted in this area are identified. Section VI: Assessment of Multiple Source Use The sixth and final section addresses issues concerning the assessment of multiple source use. These authors present and discuss innovative approaches to measuring the processes and products when individuals comprehend and learn from multiple information sources. In doing this, they also clarify that multiple source use cannot be adequately captured by traditional assessment tools focusing on single-text comprehension. In Chapter 24, Mason and Florit review methods for assessing online processing of multiple source use, including note taking, verbal protocols, reading time, and eye movements. In addition, Mason and Florit argue that physiological measures could be used to assess affective engagement during multiple source use. These authors judge the

Introduction to Multiple Source Use  •  11

suitability of the various methods for assessing different subprocesses of multiple source use, and also assess the overall strengths and weaknesses of the methods they review. In Chapter 25, Sabatini, O’Reilly, Wang, and Dreier discuss the use of scenariobased assessments as an approach to measuring multiple source use. In particular, these authors describe and illustrate how their Global, Integrated, Scenario-based Assessment (GISA) approach can be used to assess different subprocesses and products of multiple source use. This innovative approach to measuring reading ability, with an emphasis on multiple source use, is also compared and contrasted with more traditional testing paradigms focusing on reading. In Chapter 26, Goldman, Blair, and Burkett discuss Evidence-Centered Design (ECD) as an approach to designing assessments of multiple source comprehension. However, these authors emphasize that the ECD process must be guided by a consideration of theoretical and empirical research in the domain of multiple source comprehension and information-problem solving. In reviewing this body of research, they explain how the ECD process can function as a means of going from normative theory and research to the design of assessments. In Chapter 27, Coiro, Sparks, and Kulikowich focus on the assessment of online collaborative inquiry and social deliberation in digital learning environments. These authors review and illustrate current and emergent assessments that target these constructs, especially emergent work that attempts to design a broad approach to assessment that captures collaborative inquiry and social deliberation simultaneously. They lay out the challenges as well as the promises of ongoing work in this area. In Chapter 28, Magliano, Hastings, Kopp, Blaum, and Hughes provide a fascinating glimpse into the world of computer-based, automatic assessment of multiple source use. Based on a theoretical model of writing essays from multiple documents, these authors present and discuss promising approaches to automatic assessment of essay content, exemplifying such approaches by four studies that have used natural language processing systems to analyze multiple document processing and comprehension. They discuss a range of challenges associated with automatic analyses of essays based on multiple documents, and offer specific directions for future research in this area. Chapter 29: Reflections and Future Directions In this final chapter, we revisit and address the six core questions posed above. In doing so, we synthesize concepts presented within each of the six sections. After, we reflect on themes that emerge when synthesizing and analyzing ideas presented across the six sections. In alignment with these themes, we offer several tractable directions for future research on multiple source use. Finally, we complete the chapter with a general conclusion.

REFERENCES Alexander, P. A., & The Disciplined Reading and Learning Research Laboratory. (2012). Reading into the future: Competence for the 21st century. Educational Psychologist, 47, 259–280. Anmarkrud, Ø., McCrudden, M. T., Bråten, I., & Strømsø, H. I. (2013). Task-oriented reading of multiple documents: Online comprehension processes and offline products. Instructional Science, 41, 873–894. Braasch, J. L. G., & Bråten, I. (2017). The discrepancy-induced source comprehension (D-ISC) model: Basic assumptions and preliminary evidence. Educational Psychologist, 52, 167–181.

12  •  Braasch et al. Braasch, J. L. G., Lawless, K. A., Goldman, S. R., Manning, F., Gomez, K. W., & MacLeod, S. (2009). Evaluating search results: An empirical analysis of middle school students’ use of source attributes to select useful sources. Journal of Educational Computing Research, 41, 63–82. Braasch, J. L. G., McCabe, R. M., & Daniel, F. (2016). Content integration across multiple documents reduces memory for sources. Reading and Writing, 29, 1571–1598. Brand-Gruwel, S., Kammerer, Y., van Meeuwen, L., & van Gog, T. (in press). Source evaluation of domain experts and novices during Web search. Journal of Computer Assisted Learning. Brand-Gruwel, S., & Stadtler, M. (2011). Solving information-based problems: Evaluating sources and information. Learning and Instruction, 21, 175–179. Bråten, I., Braasch, J. L. G., & Salmerón, L. (in press). Reading multiple and non-traditional texts: New opportunities and new challenges. In E. B. Moje, P. Afflerbach, P. Enciso, & N. K. Lesaux (Eds.), Handbook of research in reading (Vol. V). New York: Routledge. Bråten, I., McCrudden, M. T., Stang Lund, E., Brante, E. W., & Strømsø, H. I. (in press). Task-oriented learning with multiple documents: Effects of topic familiarity, author expertise, and content relevance on document selection, processing, and use. Reading Research Quarterly. Bråten, I., Stadtler, M., & Salmerón, L. (in press). The role of sourcing in discourse comprehension. In M. F. Schober, M. A. Britt, & D. N. Rapp (Eds.), Handbook of discourse processes (2nd ed.). New York: Routledge. Britt, M. A., & Gabrys, G. L. (2000). Teaching advanced literacy skills for the World Wide Web. In C. R. Wolfe (Ed.), Learning and teaching on the World Wide Web (pp. 73–90). San Diego, CA: Academic Press. Britt, M. A., Rouet, J.-F., & Braasch, J. L. G. (2013). Documents experienced as entities: Extending the situation model theory of comprehension. In M. A. Britt, S. R. Goldman, & J.-F. Rouet (Eds.), Reading from words to multiple texts (pp. 160–179). New York: Routledge. Cho, B.-Y., & Afflerbach, P. (2017). An evolving perspective of constructively responsive reading comprehension strategies in multilayered digital text environments. In S. E. Israel (Ed.), Handbook of research on reading comprehension (2nd ed.; pp. 109–134). New York: Guilford. Clark, J. K., & Wegener, D. T. (2013). Message position, information processing, and persuasion: The Discrepancy Motives Model. In P. Devine & A. Plant (Eds.), Advances in experimental social psychology (Vol. 47, pp. 189–232). Burlington, UK: Academic Press. de Pereyra, G., Britt, M. A., Braasch, J. L. G., & Rouet, J.-F. (2014). Readers’ memory for information sources in simple news stories: Effects of text and task features. Journal of Cognitive Psychology, 26, 187–204. Flanagin, A. J., & Metzger, M. J. (2008). Digital media and youth: Unparalleled opportunity and unprecedented responsibility. In M. J. Metzger & A. J. Flanagin (Eds.), Digital media, youth, and credibility (pp. 5–27). Cambridge, MA: The MIT Press. Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H. I. (2010). Summary versus argument tasks when working with multiple documents: Which is better for whom? Contemporary Educational Psychology, 35, 157–173. Goldman, S. R. (2004). Cognitive aspects of constructing meaning through and across multiple texts. In N. Shuart-Faris & D. Bloome (Eds.), Uses of intertextuality in classroom and educational research (pp. 317–352). Charlotte, NC: Information Age Publishing. Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381. Goldman, S. R., & Scardamalia, M. (2013). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities. Cognition and Instruction, 31(2), 255–269. Greene, J. A., Yu, S., & Copeland, D. Z. (2014). Measuring critical components of digital literacy and their relationships with learning. Computers & Education, 76, 55–69. Hartman, D. K. (1995). Eight readers reading: The intertextual links of proficient readers reading multiple passages. Reading Research Quarterly, 30, 520–561. Isberner, M.-B., & Richter, T. (2014). Comprehension and validation: Separable stages of information processing? A case for epistemic monitoring in language comprehension. In D. N. Rapp & J. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 245–276). Boston, MA: MIT Press. Kammerer, Y., Bråten, I., Gerjets, P., & Strømsø, H. I. (2013). The role of Internet-specific epistemic beliefs in laypersons’ source evaluations and decisions during Web search on a medical issue. Computers in Human Behavior, 29, 1193–1203.

Introduction to Multiple Source Use  •  13 List, A., & Alexander, P. A. (2017). Text navigation in multiple source use. Computers in Human Behavior, 75, 364–375. McCrudden, M. T., & Schraw, G. (2007). Relevance and goal-focusing in text processing. Educational Psychology Review, 19, 113–139. McCrudden, M. T., Stenseth, T., Bråten, I., & Strømsø, H. I. (2016). The effects of author expertise and content relevance on document selection: A mixed methods study. Journal of Educational Psychology, 108, 147–162. O’Brien, E. J., & Cook, A. E. (2016). Coherence threshold and the continuity of processing: The RI-Val model of comprehension. Discourse Processes, 53, 326–338. Organisation for Economic Co-operation and Development [OECD]. (2013, March). PISA 2015 Draft Science Framework. Retrieved from www.oecd.org/pisa/pisaproducts/pisa2015draftframeworks.htm. Perfetti, C. A., Britt, M. A., & Georgi, M. C. (1995). Text-based learning and reasoning: Studies in history. New York: Routledge. Petty, R. E., & Cacioppo, J. T. (1986). Communication and persuasion: Central and peripheral routes to attitude change. New York: Springer/Verlag. Richter, T., & Maier, J. (2017). Comprehension of multiple documents with conflicting information: A two-step model of validation. Educational Psychologist, 52, 148–166. Rouet, J.-F., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. To appear in M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Relevance instructions and goal-focusing in text learning (pp. 19–52). Greenwich, CT: Information Age Publishing. Rouet, J.-F., Britt, M. A., & Durik, A. (2017). RESOLV: Readers’ representation of reading contexts and tasks. Educational Psychologist, 52, 200–215. Rouet, J. F., Britt, M. A., Mason, R. A., & Perfetti, C. A. (1996). Using multiple sources of evidence to reason about history. Journal of Educational Psychology, 88, 478–493. Salmerón, L., Kammerer, Y., & García-Carrión, P. (2013). Searching the Web for conflicting topics: Page and user factors. Computers in Human Behavior, 29, 2161–2171. Singer, M. (1993). Causal bridging inferences: Validating consistent and inconsistent sequences. Canadian Journal of Experimental Psychology, 47, 340–359. Stadtler, M., & Bromme, R. (2014). The content-source integration model: A taxonomic description of how readers comprehend conflicting scientific information. In D. N. Rapp & J. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 379–402). Cambridge, MA: MIT Press. Stadtler, M., Scharrer, L., Brummernhenrich, B., & Bromme, R. (2013). Dealing with uncertainty: Readers’ memory for and use of conflicting information from science texts as function of presentation format and source expertise. Cognition and Instruction, 31, 130–150. Stahl, S. A., Hynd, C. R., Britton, B. K., McNish, M. M., & Bosquet, D. (1996). What happens when students read multiple source documents in history? Reading Research Quarterly, 31, 430–456. Thomm, E., & Bromme, R. (2012). “It should at least seem scientific!”: Textual features of “scientificness” and their impact on lay assessments of online information. Science Education, 96, 187–211. Wineburg, S. S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87.

Section I

Theoretical Frameworks

2

REPRESENTATIONS AND PROCESSES IN MULTIPLE SOURCE USE M. Anne Britt northern illinois university, usa

Jean-François Rouet cnrs, université de poitiers, france

Amanda Durik northern illinois university, usa

In this chapter, we provide an overview of the cognitive representations and processes involved in multiple source use. As illustrated in Chapter 1 of this Handbook, readers frequently read multiple sources for a purpose other than to serially represent the meaning of the texts in the manner intended by the authors of those sources. This places the burden of creating meaning, determining relevance, and evaluating sources on the reader. To push the field forward, we need to better understand the representations and processes involved in this type of reading activity.

READING SINGLE VS. MULTIPLE TEXTS: SAME OR DIFFERENT PROCESSES? Theories of single-text comprehension have given us an understanding of how readers progress from visual scanning of words to comprehending the meaning of a whole text. Readers decode words, parse the clauses and sentences, retrieve word meanings, and integrate partial semantic representations with prior context and knowledge (Perfetti, Landi, & Oakhill, 2005) to create a representation of the situation that is something like what is described in the text (Kintsch, 1998; McNamara & Magliano, 2009). To accomplish this, readers rely on many cognitive processes. For example, to create local coherence among successive sentences or phrases, readers have to map

17

18  •  Britt et al.

the concepts presented in the sentence with recently activated concepts from the prior text or knowledge (Albrecht & O’Brien, 1993; Kintsch, 1998; McKoon & Ratcliff, 1998). Readers then use selected activated knowledge to generate inferences in order to fill in missing causal, spatial, and temporal relationships (Magliano, Trabasso, & Graesser, 1999). Finally, readers can deliberately search their memory and make inferences based on active, goal-driven processes and knowledge structures to create global coherence (e.g., Kintsch & van Dijk, 1983; Graesser, Singer, & Trabasso, 1994; Meyer & Freedle, 1984). Using such processes enables readers to create both a temporary representation of what the text says, the Textbase, and a more lasting representation of what the text means, the Situation Model. Undoubtedly, these passive and strategic processes are also at work when reading multiple sources. However, comprehending multiple sources also requires processing that goes beyond the construction of a model of a single author’s description of a situation. When reading multiple documents, it is rarely sufficient to just serially read and represent each author’s text to create a single, globally coherent representation. Instead, readers of multiple documents are often in a situation where text representations must be kept somewhat distinct but their inter-relationships noted. Furthermore, multiple-document reading is often undertaken for purposes that do not necessarily align with those of the document authors. In such cases, the reader may need to search for and/or select documents to read and to then select and represent textual information in a manner that is conducive to the reader’s goal. The reader can read all or some of any document, in any order, including rereading documents. Since writers have different voices and language choices, the reader also has to decide whether particular lexical or semantic phrases are co-referential across sources. The documents will not necessarily all be of the same genre (e.g., all narrative, all arguments) and the optimal genre schema for completing the task may not even be the same as that of the documents available or selected (e.g., Wiley & Voss, 1999). Therefore, reading a document for global and local coherence may actually work against successful completion of the goal. In multiple-document situations the reader will also encounter multiple perspectives and multiple interpretations of the same situation or phenomena, which may include discrepancies or even direct conflicts (Braasch, Rouet, Vibert, & Britt, 2012; Stadtler & Bromme, 2014). The reader who simply tries to add to a single representation is at risk of ending up with an inconsistent Situation Model. In these cases, the reader has to decide whether to integrate some current information without qualification, to integrate it with some kind of qualification, or to reject the information (Rouet, Le Bigot, de Pereyra, & Britt, 2016). Qualification can include attending to and interpreting source information and use of qualifiers in the integrated representation. Selection of what to believe or what to reject can include evaluating source information, other evidence, or plausibility (Bråten, Stadtler, & Salmerón, 2017; Richter, Schroeder, & Wöhrmann, 2009). Thus, there are several additional processes that, though less relevant in the single-document reading situation, are critical and less optional in multiple-document reading. In this chapter, we present an overview of the RESOLV (REading as Problem SOLVing) model (Britt, Rouet, & Durik, 2018). RESOLV extends our earlier models of multiple-document reading, TRACE (Rouet, 2006) and MD-TRACE (Rouet & Britt, 2011), to include two additional representational structures: The Context Model,

Representations and Processes  •  19

a representation of the task and reading situation, and the Task Model, a representation of the goals, plans, and associated values that control the general reading process. Like our earlier models, RESOLV also includes a representation of the Integrated Model of the content and an Intertext Model of the sources and links to the content (Documents Model; Britt, Perfetti, Sandak, & Rouet, 1999; Perfetti, Rouet, & Britt, 1999; Rouet, 2006). Finally, it includes routine (e.g., “Is this information relevant?”) and non-routine (e.g., “What is my next goal?”) processing decisions that arise during reading. These decisions both derive from one’s Task Model and can alter that model depending on their outcome. The remainder of this chapter is divided into two main sections. The first section presents an overview of the representations involved in reading multiple documents. We then propose several “processing decisions” that we see as central to strategies for comprehending multiple documents. The second section contrasts two case studies to illustrate the conditions that can lead to routine and non-routine processing decisions.

OVERVIEW OF REPRESENTATIONS AND PROCESSES When reading multiple documents, one has to decide what to read and how to read, given one’s goals and available resources (McCrudden & Schraw, 2007; Rouet & Britt, 2011; Snow & the RAND Reading Study Group, 2002). Previous models have proposed that reader goals determine what is attended to and represented during reading of single texts (McCrudden & Schraw, 2007) and multiple texts (Rouet & Britt, 2011). MD-TRACE (Rouet & Britt, 2011) is a framework for understanding the representations and processing decisions involved in reading multiple documents. MD-TRACE posits that a broad range of external and internal resources are required to fully understand reading processes and outcomes. External resources include task statements, hints, documents, and tools. Internal resources include the range of skills, knowledge, and attitudes the reader brings to the reading task. Examples of important skills include decoding, managing working memory, and self-monitoring. Important knowledge includes vocabulary, genre schemata, and general world knowledge. Attitudes and values include self-concept, motivation, and interest in task and topic. According to MD-TRACE, reading involves the construction of several internal representations: Task Models, Situation Models, and Document Models. Reading also involves several recurrent decisions such as “Is external information needed?”, “Is the current information relevant for one or more active goals?”, and “Did the immediately processed information satisfy one or more goals?”. While the MD-TRACE framework highlighted the contextualized and decisionbased nature of reading, there are several limitations that led us to propose RESOLV as a more comprehensive model of complex reading. First, the processing decisions are limited to three routine decisions, hypothesized to occur mostly at idealized processing occasions and viewed as simple “Yes” or “No” decisions. A second limitation is the lack of specificity in how readers represent the situation and task itself. Finally, the roles of motivation, interest, and value are not integrated into the MD-TRACE framework. RESOLV is a more general model that extends MD-TRACE to address the interpretive, problem-solving nature of reading. It is not just for multiple-document reading, but for reading for a purpose (e.g., find a fact or document, come to an opinion, write an evidence-based argument). RESOLV rests on a set of assumptions regarding cognitive

20  •  Britt et al.

processes and representations (see also Britt et al., 2018, and Chapter 6 for a more detailed presentation of these and other underlying assumptions). The first assumption is that reading is an adaptive goal-directed activity embedded in a social context. Reading behaviors will begin if reading is perceived as necessary for achieving goals and will stop when those goals are achieved or abandoned. This assumption leads to recognition of goals and values for those goals. The second assumption, limited processing resources, acknowledges that at any point in the reading episode, some information is active, some less active, and there is a mechanism for controlling attentional resources. These limits constrain the amount of information that can be actively represented about the situation at one time and the number of goals that can be pursued. For multiple-document reading situations, this limit will impact what can be considered at various points when making processing decisions (e.g., Is the current sentence relevant to the goal?, Are all subgoals satisfied?) and one’s evaluation of the benefits and costs of an action (e.g., having to turn pages or click links to reread a prompt vs. looking up at the top of the screen). The third assumption, feeling of knowing evaluation, assumes that readers assess whether information can be retrieved from memory. The reader can use their feeling of knowing evaluation when making decisions about how to achieve their goals (e.g., read, reread, or simply answer). The fourth assumption, benefit–cost analysis, is that readers evaluate the physical, cognitive, and emotional cost of pursuing goals and actions based on their skills, knowledge, interest, and motivation. For example, if a text is below reading level and easily accessed, then the cost of reading it would be lower compared to a text that is above reading level and requires a long walk across campus to access. This benefit–cost analysis is taken into account in nearly all processing decisions. The fifth processing assumption concerns decision thresholds. RESOLV assumes that decisions have threshold levels and that readers can modulate their decision-making thresholds. Thresholds are tightly connected to benefit–cost analysis. Readers’ Mental Models of the Reading Context, the Task, and the Document(s) We consider reading to begin when a person adopts a goal for which reading a text is either essential or sufficiently useful to pursue. Reading goals are rooted in the physical and social context and may be prompted by explicit or implicit requests. When this occurs, readers encode the request, relevant features of the physical and social reading context, and the external and internal resources available. They also activate prior knowledge and experiences related to those dimensions of the situation. In RESOLV, we refer to this knowledge as the Context Model. A Context Model can include information about the request (task statement), the requester (person/authority making the request), the audience (person intended to receive the completed request), supports or obstacles (external resources available potentially or explicitly and how they can aid or inhibit request completion), and the self (assessment of one’s own skill, knowledge, and interest and an assessment of the perceived costs and benefits of the activity). The construct of a Context Model is also a way to understand the range of possible interpretations that different readers will derive from the same set of instructions. We contend that much of this variability derives from readers’ encoding of nonlinguistic cues from the task and the situation in which it is proposed.

Representations and Processes  •  21

Based on the Context Model, the reader uses their encoding and interpretation of the request by applying relevant task schemata to create a Task Model. This representation includes the goal (goal state for the final product, subgoals, constraints), how to accomplish it (plans, strategies), and values associated with certain subgoals (importance of desired states and methods). To create a Task Model, the reader needs to create a representation of what the final goal state entails (i.e., what a “good enough” task product looks like) which can be determined by using their knowledge of question and task schemata to interpret the task statement (see our goal-structures hypothesis in Britt et al., 2018 for more details). This schematic knowledge will also guide goal decomposition and plan creation as well as inform the selection of alternative goals/actions at points of impasse. One’s Task Model, therefore, is partially limited by whether the reader has a complete and accurate task schema that is activated by the task statement and situation. These goals and plans are also expected to be constrained by materials and tools one perceives to be available; one’s perceived skill in executing possible plans, actions, and operators; one’s available cognitive resources; and the value one places on the goals and plans (see the personal-goal hypothesis in Britt et al., 2018 for more details). RESOLV proposes a significant role for motivation, interest, and value that MD-TRACE did not have. How much one values the task will be represented in the Task Model (e.g., one might place a high value on stating an opinion about some topic, but a low value on reading an opposing perspective) and one’s benefit–cost considerations are represented in the Context Model (e.g., high cost to reading graphs and evidence; high benefit to getting strong pro reasons for an argument) which will affect one’s Task Model and decision thresholds. The ultimate purpose of all of this cognitive and physical reading activity is the construction of a Documents Model, a representation of the meaning of the textual content that can help achieve the reader’s goal. As we have proposed (Britt et al., 1999; Perfetti et al., 1999), readers of multiple documents create two types of interconnected representations: an Integrated Model and an Intertext Model. The Integrated Model is often the primary target of reading. Like the Kintsch and Van Dijk (1983) Situation Model, readers represent the ideas from the text (Textbase) and then create a coherent, elaborated model of the described situation by making knowledge or text-based inferences. We use the term Integrated Model to indicate the inclusion of representational structures across text boundaries, with agents and events drawn from the documents according to the plan established by the Task Model (Britt & Rouet, 2012; Wiley, Britt, Griffin, Steffens, & Project READi, 2012). The Intertext Model is a representation of source information (e.g., who wrote the document and in what context) and connections to content and other sources. Source-to-Content links allow the reader to connect information about the source to specific facts, events, and interpretations from a particular document. Source-to-Source links allow the reader to connect sources from multiple documents by way of rhetorical relations such as supporting (corroboration or evidence) and oppositional relationships (disagree, contradict). There is a growing literature showing that readers do represent source information and attempt to create an Integrated Model of multiple documents (Britt & Aglinskas, 2002; Kobayashi, 2009; Rouet, Favart, Britt, & Perfetti, 1997; Saux et al., 2017; Stadtler & Bromme, 2007; Strømsø, Bråten, & Britt, 2010). Bråten et al. (2017) present an up-to-date review of the literature on sourcing. To the extent that a reader views the document as “an entity” (Britt, Rouet, & Braasch, 2013), they will be prompted to create an intertext model.

22  •  Britt et al.

Reading Decisions and Actions The RESOLV model proposes that reading multiple sources unfolds as a function of reader decisions and actions. During reading, people are confronted with routine and non-routine decisions. Routine decisions are based on the person’s prior experience with similar reading contexts. Consistent with previous models like MD-TRACE (Rouet & Britt, 2011), RESOLV includes three types of routine decisions: – Do I need external information? – Is this information relevant? – Am I done? In RESOLV, these decisions are considered to be along a YES/NO continuum, but based on thresholds that the reader can evaluate. At the time when the reader tests the value against the threshold, a binary response occurs (or is held off). Thus, readers use their Task Model to move toward their goal state by routine decisions or they encounter impasses and need to update their Task Model through non-routine decisions. The non-routine decisions occur when a reader encounters an unexpected obstacle or impasse and doesn’t know what to do next. Such impasses include: when the action or goal component of the Task Model is not sufficiently elaborated to know what to do, when the current action sequence is not working, or when there are multiple competing options from which the reader must select. When these impasses arise, readers must decide whether to retrieve, change, or select among competing actions or goals. As a result, such decisions can affect reading behaviors and thereby the ultimate representation of the information from the texts (Documents Model). Although RESOLV may not capture all processing decisions with these categories, they at least serve to emphasize the fact that processing steps are more dynamic and there is a need to update one’s task representation and regulate reading behaviors throughout the reading process. Any model that outlines a predominately linear set of processes will be less accurate and helpful to researchers. Routine Processing As previously mentioned, RESOLV proposes three routine processing decisions to guide reading that are a regular part of general reading behavior. These include: “Does my purpose require that I seek information from an external resource?” (“Need external?”), “Is the information that I am currently processing relevant to my purpose?” (“Relevant?”), and “Have I achieved my purpose?” (“Done?”). At any point in the process, the reader can answer these questions in a binary “Yes”/“No” answer or they can hold off until a threshold is reached to achieve a decision. The first routine question – “Need external?” – occurs when the reader asks whether their goal requires them to seek information from an external resource. This decision is based on a comparison at any point between the current state of active knowledge and the requirements in one’s Task Model. To answer this question, the reader asks whether a good enough and complete enough answer can be retrieved from memory or whether one needs to find information from an external source. Such sources might include locating a document or contacting a person. Assessment of information needs almost certainly occurs at the beginning of the task but can occur at other times such

Representations and Processes  •  23

as when a new subgoal is pursued. The question may be constrained or overridden, however, if there are explicit indicators that the requestor wants an answer from an external source and the person responding translates that into their Task Model constraints. Several factors can increase the probability of a “Yes” decision to seek out external information. Reader factors include: when the reader has a low feeling of knowing evaluation, when they believe they can succeed in gaining information from external resources, and when they value the goal or task. Situational factors include: when the request directs the reader to use external sources, when resources are immediately available and clearly useful (low effort, high benefit), and when the requester is an authority. When designing experiments or classroom activities, the routine question of “Need external?” can be obscured or overridden when the primary task is to read each sentence and each document. The second routine question – “Relevant?” – can be asked at various times by the reader, at different levels (e.g., about a document as a whole or about a segment within a document) and based on source or content information. To the extent that the relevance decision is more toward the “Yes, this action is helping to satisfy the current goal” decision, the reader will select it, read it more carefully and fully, and add the information to their integrated model. To the extent that the relevance decision is more toward the “No, it is not relevant for the current goal” decision, the reader will skip it, skim it, or not include information from it in their integrated model. Reader factors can affect this relevance decision. Readers’ skill in monitoring goals and subgoals, expertise with the genre structure that is required, and interest or motivation in achieving the goals can all affect whether someone will ascertain the significance of the information to their goal. Likewise, situational factors can affect or even change the nature of the decision. For instance, the task might interfere with controlling one’s processing (e.g., requiring processing of each sentence, not allowing look backs or direct comparisons), or the task might explicitly or implicitly override the need for the decision (e.g., authority-provided text(s)). Finally, a text might directly answer the goal question or be written in such a difficult manner as to be deemed not worth the effort of reading. The third routine question – “Done?” – can be asked about the task as a whole or about any subgoal. It is a Yes/No decision about whether a goal is satisfied. If yes, then the reader can stop. If the answer is “no”, the reader has several options ranging from working longer or harder to achieve the goal with the current action to revising their Task Model to create a new goal or action (see non-routine decisions below). This decision can arise when the current action reaches a point of segment completion, such as when the reader is done reading a particular text or chapter or a certain amount of time has elapsed. This decision will be most strongly affected by situational factors such as whether there is high overlap between the task prompt and a text segment and whether there is a time constraint or other resource limitations. Non-Routine Processing Reading in multiple-document situations does not always progress smoothly. There are many situational factors that can lead to an impasse and, therefore, to nonroutine decisions and an updated Task Model. In general, non-routine decisions relate to either actions (methods of pursuing a goal) or the goal/subgoal itself. Consider first that an impasse could arise because the reader has a goal but there is a problem with the action. That problem might be one of not knowing what to do and

24  •  Britt et al.

needing to “get an action”. For instance, the reader may have the goal to write an argument but it may take work to figure out how to go about doing that. This will occur most often for readers who are not experts with the specific task schema (e.g., argument schema or explanation schema). Alternatively, the reader could decide that the current action is not working and therefore has to make a “change action” decision. For instance, if one is searching for a counterargument in a document set and there isn’t one, then the reader has to find an alternative method of getting a counterargument. Finally, the reader may have multiple competing options and therefore they will have a “select action” decision. For instance, when writing an argument, the reader could have several methods of selecting a position or claim (e.g., select claim because it is the first encountered, has the most supporting evidence, or is most consistent with personal values). The action or method they select can be affected by how important the answer is to the reader. Consider next that an impasse could arise because there is a problem with the goal. As with actions, the reader may have no clear goal and therefore has to make a “get goal” decision. This can occur because the reader does not know how to decompose a high-level goal into subgoals. For example, recognizing that “accurate” information is needed but not having subgoals such as locating unbiased sources. Alternatively, the reader could have a goal that they are pursuing but it is not working. This type of impasse can be resolved by a “change goal” decision. For example, the reader may be trying to get a particular fact to support the claim that is being made. If that fact cannot be found, the reader can change the goal to one of making a more supportable claim based on the information that is available. Finally, the reader may have multiple competing goals and need to make a “select goal” decision. The reader could use the relative benefit and cost of each subgoal to make this decision. Unlike routine processing decisions, non-routine decisions typically result in changes to the task model. In this way, readers in multiple-document tasks can adapt to both their information environment and their own limitations, values, and purposes. In the next section, we illustrate the explanatory value of the RESOLV model by examining two key multiple-document studies: Wiley and Voss (1999) and Gil, Bråten, Vidal-Abarca, and Strømsø (2010). We will use features of these studies to illustrate RESOLV decisions, focusing on conditions that switch the reader from more routine to non-routine processing.

USING RESOLV TO EXAMINE TASK AND TEXT INTERACTIONS Several studies, beginning with Wiley and Voss (1999; see also Le Bigot & Rouet, 2007; Wiley et al., 2009), found that giving students an argument task improved their learning in a multiple-document reading situation as compared with a summary task. Other similar studies, however, found either no difference or a benefit for the summary task (Gil et al., 2010; Le Bigot & Rouet, 2007). In this section, we present an analysis of these tasks to explain how an “argument” task could lead to different reading goals and actions which could explain the pattern of learning results found in two of these studies (Gil et al., 2010; Wiley & Voss, 1999). Wiley and Voss (1999) found that students given an argument-writing task were more accurate at recognizing potential inferences and performed better on an analogy transfer task than those writing to summarize. In contrast, Gil et al. (2010) found that students asked to write a summary learned more in terms of memory for individual sentences and

Representations and Processes  •  25

accurate recognition of potential within- and across-document inferences inquiry task than did those asked to “express and justify your personal opinion.” Table 2.1 summarizes key characteristics of these studies with respect to (1) the domain (e.g., history vs. science); (2) the task (e.g., the exact wording of prompt, hints, and instructions, the structure of the reading and writing tasks, and how much time was given); (3) the document set (e.g., type of documents available and features of their type and interrelations to each other and to the prompt); and (4) the environment (e.g., modality for reading and writing, tools and social supports, setting). We consider now how the context, as described by these dimensions,

Table 2.1  Characteristics of Document Sets and Prompts for Two Key Multiple-Document Studies. Wiley and Voss, 1999

Gil et al., 2010

Domain Prompt

History: Irish potato famine “Your task is take the role of historian and develop {an argument / a summary / a narrative} about what produced the significant changes in Ireland’s population between 1846 and 1850”.

Hints, instructions

“Historians work from sources including newspaper articles, autobiographies and government documents like census reports to create histories.” Students were instructed to use both windows. Participants were asked to read through all before writing. Documents available during writing. 30 mins Excerpts from authentic primary descriptive documents of various types but not directly answering prompt. No summary of answer to read. No internal citing of documents in set. All documents relevant but low semantic overlap across documents and with prompt. Source information presented but not helpful in selecting (not clearly untrustworthy or unknowledgeable). Mock web environment. The overview menu was accessible through an icon, browser had two side-by-side windows, each excerpt viewable without scrolling. Computer?

Science: climate change “Imagine that you have to write a brief report to other students {that summarizes / where you express and justify your personal opinion about} how climate changes may influence life on Earth and what are the causes of climate changes”. “Base your report on information included in the following seven texts. Use the most relevant information, and try to express yourself clearly and to elaborate the information – preferably in your own words”. Read first. Then write without documents available.

Structure of tasks Time Document set

Modality for reading Modality for writing Support and obstacles? Setting/ place

Supports: Documents available for writing

Unknown

35 mins to read; 15 mins to write Argument and descriptive texts directly answering conflicting perspectives on prompt. Textbook sets up the main controversy and overviews two sides. No internal citing of documents in set. All documents relevant and high semantic overlap across documents and with prompt. Source information presented but not helpful in selecting (not clearly untrustworthy or unknowledgeable). Read using Read&Answer (see obstacles below). Computer without documents. Supports: N/A Obstacles: Could not see more than one segment at the time, requires effort to unmask, could not view two docs at same time, answer questions from memory. In groups of 10 at university lab.

26  •  Britt et al.

might explain the findings. While it is beyond the scope of this chapter to discuss them all, we will focus on explaining the situational elements and then we will show how varying a couple of important dimensions – the Documents and the Prompt – can influence how readers engage with the content and therefore represent the content differently. Because the two studies (Gil et al., 2010 and Wiley & Voss, 1999) used different topics, domains, and prompts and hints, we will use Revised Prompt 1 for our illustration of how the match between the structure of the documents in a set and the structure of the target product from the prompt can influence processing decisions. Revised Prompt 1: “Write an argument about the causes of climate change. Use the most relevant information from the documents provided to support your position”. Document Structure Aligned to Target Integrated Model Let’s consider the case in which the documents themselves have the structure of an argument (i.e., supported claims) addressing the specific prompt so the material can be selected and accumulated easily into the structure of an Integrated Model for this argument prompt. (Note this is one of the key features of the argument condition from Gil et al. (2010) as shown in the final column of Table 2.1. However, their actual argument condition had a prompt asking for an opinion, not an argument.) Figure 2.1 shows the series of states and decisions the reader could go through in order to achieve the goal. What might students’ Context and Task Models look like in this situation? RESOLV proposes a minimal task-elaboration hypothesis that states: “by default, readers will parse a request into a sense of the final goal and create enough of a general plan to be able to proceed based on activated goal structure” (Britt et al., 2018). Thus, we would expect that a student would create a top-level goal but would not break it down into detailed subgoals at this point. Thus, the reader could begin with the following Context and Task Model: Context Model Request: Task: Write an argument of causes of climate change Constraint: Use available documents Requester: Experimenter; authority; knowledgeable Supports/Obstacles: Computer; document set Self: Know that argument has the form of a claim with support; care a little about climate change; have an opinion and some reasons but don’t have detailed facts Task Model Goal: Write an argument about causes of climate change Initial Action/Plan: Read these documents Value: Moderate

2. Goal: write an argument of causes of climate change

8a. Yes

14a. Yes

6b. No

4. Get goal

10. Change goal

5b. No

5c. Somewhat 5. Information relevant?

10.2 Get action

10.1. Goal: understand cause of climate change

5a. Yes

4.3. Action: read each document to locate stated causes

4. Get action

4.1. Goal: find causes of climate change

3a. Yes

9. Goal completed

11a. Yes

12. Stop: abandon or renegotiate request

11b. No

11. Information relevant?

11c. Somewhat

10.3. Action: read each document to infer potential causes

6. Done?

3b. No

3. Need external info?

The boxes represent Task Model elements (goals and actions) and the diamonds represent decisions. The solid boxes and diamonds indicate routine processing decisions while the dotted lines indicate non-routine processing decisions.

Figure 2.1  Possible Processing Decisions for a Student Who Is Asked to Write an Argument When the Documents Have Already Created Claims and Reasons on the Controversy to Select From.

14. Information relevant? 14b. No

14c. Somewhat

13.3. Action: read each document to infer potential reasons

13.2 Get action

13.1. Goal: understand reasons for each cause

13. Change goal

8b. No

8. Information relevant?

8c. Somewhat

7.3. Action: read each document to locate stated reasons

7.2 Get action

7.1. Goal: find reasons

7. Get goal

1. PARSE REQUEST Activate “argument schema” Activate “climate change knowledge” “Use documents” constraint

6a. Yes10.3

28  •  Britt et al.

As shown in point 1 of Figure 2.1, RESOLV assumes that the reader activates relevant knowledge about the content (in this case climate change) and the task schema (in this case an argument schema). Recall that there were several differences between the two studies such as the topic and key prompt (“write an argument” vs. “express and justify your personal opinion”) so we have revised the prompt to provide as much consistency as possible other than the overlap of document and task schema. We further assume that the student has a basic argument schema (e.g., claim and support) that is activated by both “write an argument” and “support your position”. We finally assume they interpret the instruction “Use the most relevant information from the documents provided to support your position” as a “use provided documents” constraint. This Context Model leads to a high-level goal state (“2. Goal: write an argument of causes of climate change”). The solid boxes represent Task Model elements (goals and actions) and the solid diamonds indicate the three routine processing decisions of “Do I need external information?” (Need external?), “Is current information goal-relevant?” (Relevant?), and “Is the goal satisfied?” (Done?). Non-routine processing decisions (i.e., get, change, drop goal or action) are represented by boxes and diamonds with broken lines. How do typical college readers in such a study who are not experts on the topic and not reading for a specific class achieve their goal in this scenario? First, the reader evaluates whether there is a need for external information (Diamond 3). Given that the Context Model of our typical student has the constraint of using the documents and their initial benefit–cost analysis would be high enough and the value placed on the goals high enough, the reader would likely answer this question as “YES information is needed” and would then follow Arrow 3a. Of course, if the reader does not interpret “use the provided documents” as a constraint or the benefit–cost analysis is low, the reader could decide to not read the documents (Arrow 3b). For simplicity, we only present the reader who has successfully retrieved an answer and therefore asks themselves whether they are done (Diamond 6). It is assumed that they would answer “YES” and then go to goal complete (9). This highlights the complexity of diagramming on a single figure the various “YES” and “NO” paths. At the beginning of the task, the initial Task Model is simply “to write an argument” (Rectangle 2). For those that decide to get external information (3a), there is a minor impasse because the reader has not yet defined the kind of information they need. This leads to the non-routine process of “Get goal” (Diamond 4). This process could result in the creation of a subgoal (4.1), “finding causes of climate change”. Consistent with RESOLV’s minimal task-elaboration hypothesis, we might expect students to create only a minimal subgoal of finding causes rather than creating a more complete set of subgoals (e.g., find reasons, counter-arguments, response to counter-arguments; see Kopp, 2013). The reader then needs an action for achieving that goal. In our example, since students received only documents and the instructions, they could plan an action such as to “read each document to locate stated causes” (4.3). Then the reader engages with the texts to locate and collect causes. At this point, they will ask the routine decision question “Is this information relevant?” (Diamond 5). Relevance questions can occur at the level of document selection (document as a whole) or for decisions about a piece of the content (information within a document). In this case, the student is pushed to consider all documents and contents relevant for three reasons. First, the instruction “using the documents” could be interpreted in the Context Model as meaning that all the documents are relevant and therefore the relevance question would have a very low decision threshold for including the document. Second, the Context Model

Representations and Processes  •  29

includes the information that the requester is “an authority” who is providing the documents. Therefore, they would likely not question whether the documents are relevant. Finally, the sources, although preceding the content, are likely not elaborated enough to warrant the rejection of a document. Thus, the conditions will likely push toward the “YES this document is relevant” side of the continuum at the document level. At the content level, the relevance decision should be answered as a relatively easy “YES” because the documents present explicit, clearly marked causes. At the level of whether specific content is relevant, the goal will be to read to find the causes, so the students will be reading to locate information about causes, especially those marked by the author. As shown in Table 2.1, there are several features of the Gil et al. document set that may allow the student to proceed without impasse: When the documents unambiguously state causes (current goal 4.1), the titles and headers explicitly indicate the author’s position, there is high semantic overlap between document content and the prompt, and the structure of the information makes explicit assertions about causes. In these cases, the reader should be able to successfully achieve their goal (see 5a. “YES”) because the structure of the information matches the readers’ comprehension goals. Thus, this prompt-document set pairing should not lead to an impasse and the initial Task Model should be pursued without much, if any, non-routine processing. RESOLV’s third routine question – “Done?” – is triggered as the reader begins finding elements to satisfy the subgoal (“find causes”). Thus, at Diamond 6, the reader can ask whether the subgoal is completed and whether the top-level goal is completed. As with the other routine decisions, the decision is along a YES/NO continuum that can be adjusted based on the reader’s benefit–cost analysis and decision threshold. In this case, because the documents have information to achieve the goal with the current action, the reader would be able to make the decision relatively easily and proceed without an impasse. At the level of the top goal, the “Done?” decision (“write an argument”) could be complicated by the student not reactivating the goal as “argument”, but instead confusing it with the subgoal (i.e., finding causal claims). Therefore, they could erroneously say “YES” (6a) and stop. If, however, the student remembers to make a subgoal for support, then they will follow decision “NO not done” (6b). Then since there is not another subgoal, the reader has to “Get goal” to “find reasons” and “Get action” to “read each document to locate stated reasons” (7, 7.1, 7.2, and 7.3, respectively). In the Gil et al. (2010) case, the documents are structured as arguments that specifically address the prompt. It should be easy enough to identify and select reasons and the reader should be able to achieve their subgoal of finding reasons for the claims. In fact, the presence of reasons in the documents could even help readers remember to complete the overall goal more fully (i.e., adoption of a higher performance criterion), thereby increasing the likelihood of a “NO not done” (6b) decision. At some point in reading, the reader will decide they have the relevant information (8a) and again make a decision about whether they are done with the subgoal and the top-level goal (6). Our reader, who now has both subgoals achieved, can decide they have completed their top-level goal and stop (9). This analysis of the subgoals and actions (Task Model) and processing decisions of the Gil et al. study situation illustrates why one would not necessarily expect deep processing of the documents, transformation of content, and intertext connections as compared to a summary task. In this argument condition, the reader only has to search for “answers” rather than read to understand the texts. This could involve skipping and skimming as implicitly suggested by Read&Answer, a tool for masking text until the reader clicks to unmask. In the summary condition, the reader may have had the overall goal of accumulating all causes

30  •  Britt et al.

mentioned and then dealing with or responding to the controversial perspectives. In the argument task, the reader might think all he has to do is to collect the causes and support. Thus, the summary task of connecting the causes into a coherent summary should require a different set of subgoals and in this case lead to deeper reading of more text content and transforming it. Thus, the Gil et al. (2010) findings that the “argument” task was not as beneficial as a summary task can be partially explained by this detailed analysis of initial subgoals/actions and the need to update those subgoals/actions. Documents Structure Not Aligned to Target Integrated Model One of the most interesting aspects of the Wiley and Voss document set is that none of the documents directly address the prompt, so most readers would experience an impasse and would have to update their Task Model. Thus, the key difference shown in Figure 2.1 is the need to change the goals and actions (10–10.3 and 13–13.3) because the materials are not structured to have answers to the initial goals. Again, looking at our revised prompt to equate the two studies, we illustrate the situation where the documents do not align with the structure of the Integrated Model. Thus for our revised prompt, the reader begins by trying to achieve their first subgoal of “finding the causes of climate change” by an action of “reading each document to locate stated causes” (see 4–4.3). In this case, however, they would read each document for relevant information (Diamond 5). We would expect the relative amount of skimming to increase as more documents are shown not to have “an answer”. When the documents do not include an explicit stand on the controversy (in this case “state causes”), the reader would get through all the documents without achieving the goal. They are at an impasse. At this point, they would have to answer NO (5b) because the statements from these documents cannot be simply selected to use to answer the question. Depending on their benefit–cost analysis evaluation, the student could decide to abandon the task or ask the experimenter what they are to do if no documents have the answer (12). However, if the student believes they can change subgoals and cares enough (higher benefit–cost analysis) to do so, they might decide to adopt a new subgoal (Diamond 10. Change goal). The changed goal could lead to the goal of understanding the causes of climate change” through reading the documents and making inferences (10.1–10.3). If the reader is able to transform the information presented and infer causes, then they would eventually stop reading and decide whether they are done (Diamond 6). Again, they would have to decide whether their top-level goal was achieved and, if not, try to find more reasons, reread the documents more carefully, and go quickly through the simple “find reasons” goal (7–7.3) to the “change goal” of “understanding the reasons for each cause” (13.1) by reading and inferring (13.3). However, because the documents were already skimmed and the student has realized there are no ready-made reasons, they may completely bypass the simpler “find reasons” and go directly to the “understand” goal. If successful, the student can eventually decide whether they are done (14a, 6a, 9). Otherwise they could quit or ask the experimenter (14b, 12). This mismatch between the prompt and the explicit structure of the texts and explicit content would lead to much non-routine processing (10–10.3 and 13–13.3). Rather than reading to accumulate, they would have to read to infer and transform. The student would have to deeply process the content to determine whether it can be useful in completing a subgoal. As a result, more of the content might be processed and

Representations and Processes  •  31

processed more deeply, with more elaborative inferences and restructuring. This need to update the Task Model to engage in inferencing and transformation of content could help explain why Wiley and Voss (1999) found that the argument prompt more effectively helped students learn from multiple documents than did the summary prompt. This illustration shows how RESOLV can be used to describe how different reader interpretations of the task can interact with the structure of documents within a set to explain reading behavior. Indeed, our illustration suggests that the same prompt might elicit very different behaviors with different learning outcomes based on how the prompt and document set align. If well aligned, an argument task might lead to passive, non-transformative processing, an accumulation strategy (when the materials provide ready-made arguments for an argument prompt) and perhaps not very effective in learning. When not aligned, the same task could lead to an impasse and the need to create new goals with actions for transforming the textual material. It is this transformation that leads to learning, though the uncertainty and difficulty associated with this effort would likely also lead to more task failure and abandonment. It also shows there are many predictable paths from not reading at all (3b, 6a, 9) to superficial accumulation (3a, 4 through 5a, 6b, 7 through 8a, and 9) to the reader eventually setting goals and actions for understanding and inferring (3a, 4 through 5b, 10–11a, 6b, 7 through 8b, 13–14a, 6, and 9). Thus, it is not that certain tasks will determine whether certain strategies are used or learning occurs. Rather, one must look at the whole situation, including the match between the resources (e.g., materials and tools) with the interpreted task as well as the knowledge and motivation of the reader.

DISCUSSION In this chapter, we presented a brief overview of the RESOLV model applied to multipledocument reading (Britt et al., 2018). This model extends our earlier efforts to describe the representations and processes involved in multiple-document reading. This model places new emphasis on the context and how the reader’s interpretation of their reading task arises out of what they know about themselves, prior reading situations, and the world around them. It also introduces adaptive mechanisms that enable the reader to modify not only their task strategies but the task goals themselves in order to respond to the limitations of their environment and their own cognitive capabilities. We then used this model to propose an explanation for differences in key results from two important papers on multiple-document use (Gil et al., 2010; Wiley & Voss, 1999). RESOLV focuses on a reader’s goals and their control of information gathering and processing decisions during a reading task. This emphasis is important for several reasons. First, it highlights the goal-driven nature of reading. Readers have a purpose for reading that drives their behavior and which is generally different from the purposes of the authors whose documents they read or those envisioned by the authors. Goals can originate with the reader or be derived from requests by other people. Goals have values associated with them that motivate people to adopt them and drive their completion or, sometimes, to abandon them. Understanding how readers interpret and encode requests, tasks, or prompts into task goals, and how the situation impacts this representation, is likely to prove important for interpreting research on multipledocument tasks. Second, goals influence reading behavior and the eventual representation of the content. Readers do not just read from the first to the last word of a text and represent

32  •  Britt et al.

only the situation model as intended by the author. As our analysis of the Gil et al. (2010) and Wiley and Voss (1999) studies illustrates, a reader’s purpose for reading might have a significant impact on how they read and represent document content. To examine the effect of goals, however, work is needed to understand reading behaviors in authentic, complex reading contexts. Third, there are many important elements of the situation that can influence processing decisions. We presented Table 2.1 as an initial set of features that can lead to either an easier task with routine processing or a more challenging task with nonroutine processing. If a teacher’s or experimenter’s goal is to create a situation for deep learning of all the material, then one consideration will be the interaction of the reader’s interpretation of the task and the structure of the documents. For learning, a more constructive activity may lead to deeper learning (Chi & Wylie, 2014), but it has to be interpreted as intended and other factors, such as the students’ level of motivation and the cost of the processing, must be considered. We hope that identifying such features and considerations can help teachers and experimenters more systematically understand the potential effects and interactions between goals, tasks, and texts. The interested reader can find more information about the set of hypotheses proposed by RESOLV and how they might be tested in Chapter 7 of Britt et al., (2018) and Rouet et al. (2017).

REFERENCES Albrecht, J.E., & O’Brien, E.J. (1993). Updating a Mental Model: Maintaining both local and global coherence. Journal of Experimental Psychology: Learning, Memory and Cognition, 19, 1061–1070. Braasch, J., Rouet, J.-F., Vibert, N., & Britt, M.A. (2012). Readers’ use of source information in text comprehension. Memory and Cognition, 40, 450–465. Bråten, I., Stadtler, M., & Salmerón, L. (2017). The role of sourcing in discourse comprehension. In M.F. Schober, D.N. Rapp, & M.A. Britt (Eds.), Handbook of Discourse Processes (2nd ed.). New York: Routledge. Britt, M.A., & Aglinskas, C. (2002). Improving students’ ability to use source information. Cognition and Instruction, 20(40), 485–522. Britt, M.A., & Rouet, J.-F. (2012). Learning with multiple documents: Component skills and their acquisition. In M.J. Lawson and J.R. Kirby (Eds.), Enhancing the Quality of Learning: Dispositions, Instruction, and Learning Processes (pp 276–314). Cambridge: Cambridge University Press. Britt, M.A., Perfetti, C.A., Sandak, R., & Rouet, J.-F. (1999). Content integration and source separation in learning from multiple texts. In S.R. Goldman, A.C. Graesser, & P. van den Broek (Eds.), Narrative Comprehension, Causality, and Coherence: Essays in Honor of Tom Trabasso (pp 209–233). Mahwah, NJ: Lawrence Erlbaum Associates. Britt, M.A., Rouet, J.-F., & Braasch, J.L.G. (2013). Documents as entities: Extending the Situation Model theory of comprehension. In M.A. Britt, S.R. Goldman, & J.-F. Rouet (Eds.), Reading: From Words to Multiple Texts (pp 160–179). New York: Routledge. Britt, M.A., Rouet, J.-F., & Durik, A.M. (2018). Literacy Beyond Text Comprehension. New York: Taylor & Francis. Chi, M.T.H., & Wylie, R. (2014). The ICAP Framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49, 219–243. Gil, L. Bråten, I., Vidal-Abarca, E., & Strømsø, H.I. (2010). Understanding and integrating multiple science texts: Summary tasks are sometimes better than argument tasks. Reading Psychology, 31(1), 30–68. Graesser, A.C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101, 371–395. Kintsch, W. (1998). Comprehension: A Paradigm for Cognition. Cambridge: Cambridge University Press. Kintsch, W., & Van Dijk, T.A. (1983). Strategies of Discourse Comprehension. New York: Academic Press. Kobayashi, K. (2009). The influence of topic knowledge, external strategy use, and college experience on students’ comprehension of controversial texts. Learning and Individual Differences, 19(1), 130–134.

Representations and Processes  •  33 Kopp, K. (2013). Selecting and using information from multiple documents for argumentation. Unpublished Dissertation, Northern Illinois University, DeKalb, IL. Le Bigot, L., & Rouet, J.F. (2007). The impact of presentation format, task assignment, and prior knowledge on students’ comprehension of multiple online documents. Journal of Literacy Research, 39(4), 445–470. Magliano, J.P., Trabasso, T., & Graesser, A.C. (1999). Strategic processes during comprehension. Journal of Educational Psychology, 91, 615–629. McCrudden, M.T., & Schraw, G. (2007). Relevance and goal-focusing in text processing. Educational Psychology Review, 19, 113–139. McKoon, G., & Ratcliff, R. (1998). Memory-based language processing: Psycholinguistic research in the 1990s. Annual Review of Psychology, 49(1), 25–42. McNamara, D.S., & Magliano, J. (2009). Toward a comprehensive model of comprehension. Psychology of Learning and Motivation, 51, 297–384. Meyer, B.J.F., & Freedle, R.O. (1984). Effect of discourse type on recall. American Educational Research Journal, 21, 121–143. Perfetti, C.A., Landi, N., & Oakhill, J. (2005). The acquisition of reading comprehension skill. In M.J. Snowling & C. Hulme (Eds.), The Science of Reading: A Handbook (pp 227–247). Malden, MA: Blackwell Publishing. Perfetti, C.A., Rouet, J.-F., & Britt, M.A. (1999). Towards a theory of documents representation. In H. van Oostendorp & S.R. Goldman (Eds.), The Construction of Mental Representations During Reading (pp 99–122). Mahwah, NJ: Erlbaum Associates. Richter, T., Schroeder, S., & Wöhrmann, B. (2009). You don’t have to believe everything you read: Background knowledge permits fast and efficient validation of information. Journal of Personality and Social Psychology, 96, 538–558. Rouet, J.F. (2006). The Skills of Document Use: From Text Comprehension to Web-Based Learning. Mahwah, NJ: Erlbaum. Rouet, J.-F., & Britt, M.A. (2011). Relevance processes in multiple document comprehension. In M.T. McCrudden, J.P. Magliano, & G. Schraw (Eds.), Relevance Instructions and Goal-focusing in Text Learning (pp  19–52). Greenwich, CT: Information Age Publishing. Rouet, J.-F., Britt, M.A., & Durik, A. (2017). RESOLV: Readers’ representation of reading contexts and tasks. Educational Psychologist, 52(3), 200–215. Rouet, J.-F., Favart, M., Britt, M.A., & Perfetti, C.A. (1997). Studying and using multiple documents in history: Effects of discipline expertise. Cognition and Instruction, 15(1), 85–106. Rouet, J.-F., Le Bigot, L., de Pereyra, G., & Britt, M.A. (2016). Whose story is this? Discrepancy triggers readers’ attention to source information in short narratives. Reading and Writing, 29, 1549–1570. Saux, G., Britt, M.A., Le Bigot, L., Vibert, N., Burin, D., & Rouet, J.-F. (2017). Conflicting but close: Readers’ integration of information sources as a function of their disagreement. Memory & Cognition, 45(1), 151–167. Snow, C. & the RAND Reading Study Group (2002). Reading for Understanding. Towards a R&D Program for Reading Comprehension. Santa Monica, CA: RAND. Stadtler, M., & Bromme, R. (2007). Dealing with multiple documents on the WWW: The role of metacognition in the formation of Documents Models. International Journal of Computer-Supported Collaborative Learning, 2(2), 191–210. Stadtler, M., & Bromme, R. (2014). The content–source integration model: A taxonomic description of how readers comprehend conflicting scientific information. In D.N. Rapp & J. Braasch (Eds.), Processing Inaccurate Information: Theoretical and Applied Perspectives from Cognitive Science and the Educational Sciences (pp 379–402). Cambridge, MA: MIT Press. Strømsø, H.I., Bråten, I., & Britt, M.A. (2010). Reading multiple texts about climate change: The relationship between memory for sources and text comprehension. Learning and Instruction, 20, 192–204. Wiley, J., Britt, M.A., Griffin, T.D., Steffens, B., & Project READi (2012, April). Approaching Reading for Understanding from Multiple Sources in History and Science: Initial Studies. Symposium paper presented at the 2012 AERA Annual Meeting, Vancouver, BC. Wiley, J., Goldman, S.R., Graesser, A.C., Sanchez, C.A., Ash, I.K., & Hemmerich, J.A. (2009). Source evaluation, comprehension, and learning in Internet science inquiry tasks. American Educational Research Journal, 46(4), 1060–1106. Wiley, J., & Voss, J.F. (1999). Constructing arguments from multiple sources: Tasks that promote understanding not just memory for text. Journal of Educational Psychology, 91, 301–311.

3

COLD AND WARM PERSPECTIVES ON THE COGNITIVE AFFECTIVE ENGAGEMENT MODEL OF MULTIPLE SOURCE USE Alexandra List the pennsylvania state university, usa

Patricia A. Alexander university of maryland, college park, usa

The traditional conception of reading has been one of a student immersed in a favorite book. Yet, reading in the 21st century has taken on quite a different form. Rather than reading a single text, students are often called upon to select and process multiple texts, from among the plethora of sources instantaneously available on the Internet. Beyond print books, students today are required to make sense of sources as varied as hypertexts, multimedia texts, images, videos, and text-based websites, all varying in quality and accuracy. Moreover, text processing has become increasingly conceptualized as an instrumental activity for students, guided by the tasks assigned to them rather than by the activities chosen by them. Despite the frequency with which students engage in multiple source use (MSU), research in this area has been limited in comparison to the extensive research on singletext comprehension. In addition, MSU research has primarily focused on behavioral aspects of source use and the cognitive processes involved in students’ construction of mental representations of multiple texts (Brand-Gruwel, Wopereis, & Walraven, 2009; Britt, Perfetti, Sandak, & Rouet, 1999; Perfetti, Rouet, & Britt, 1999). In contrast, affective or motivational aspects of MSU have been fairly neglected in the literature. This chapter introduces a theoretical model conceptualizing students’ MSU during complex task performance as the result of both cognitive and behavioral and motivation factors. We refer to this model as the Cognitive Affective Engagement Model of Multiple Source Use (CAEM). Our goals for this chapter are five-fold. First, we consider salient features of prominent models of MSU. Second, we discuss motivational and affective constructs

34

The Cognitive Affective Engagement Model  •  35

with direct relevance to MSU. Third, we introduce the CAEM and provide a theoretically driven rationale for its development. Fourth, we use the CAEM as an explanatory paradigm for understanding empirical findings in the literature on single-text comprehension and MSU. Fifth, we discuss implications of the CAEM for learning in the 21st century. Throughout this chapter, the term text is used to refer to information or content presented as written discourse (Ricoeur, 1991). A complementary term source is used throughout this manuscript to emphasize the authored nature of text, as generally written by a person or persons for a purpose (Alexander & Fox, 2004). As such, when referring to sources we refer to written discourse (i.e., text) while emphasizing its constructed nature or its document information (i.e., metadata about text origin, like author or publisher). While the CAEM is primarily focused on learners’ understanding and use of multiple texts, it may be extended to understand how learners also engage with non-textual or mixed-media sources of information, like images or videos. In consideration of these definitions, the CAEM is described as a model of MSU, rather than text use. In this way, the model has implications beyond students’ comprehension and integration of strictly linguistic content, encompassing the consideration and evaluation of all manner of source information.

COLD PERSPECTIVES ON MULTIPLE SOURCE USE Over the last 25 years, research on multiple-text comprehension and integration has proliferated. In that time, at least three models of MSU have been proposed. Documents Model The earliest model to address MSU was the Documents Model of Multiple Texts (DM; Britt et al., 1999; Perfetti et al., 1999). This model posits that the comprehension of multiple sources occurs through readers’ construction of two layers of cognitive representations of texts, the integrated mental model and the inter-text model. The integrated mental model is focused on content and represents students’ unified understanding of the common topic or issue discussed across texts. The inter-text model is a structural model that maps the various relations among multiple texts. In particular, the inter-text model maps the relations between information within a text and that text’s document markers (i.e., metadata, like title and author) as well as the conceptual relations among multiple texts (e.g., corroborating, complementing, or refuting one another). Comprehension occurs when students connect their integrated mental model with their inter-text model of multiple texts. These two models may be connected in four different ways (i.e., separate representation model, mush model, tag-all model, documents model), all varying in the extent to which information is integrated, important points arising across texts are emphasized, and specific pieces of information are associated with their sources of origin. For novice learners, the DM is the most adaptive representation of multiple texts. In forming the DM, students integrate information across multiple texts, while excluding unimportant or irrelevant information, and track where individual pieces of information come from, to form judgments of information quality and reconcile any discrepancies that arise.

36  •  List and Alexander

Multiple-Documents Task-Based Relevance and Content Extraction Model As a complement to the DM (Britt et al., 1999), the Multiple-Documents Task-Based Relevance and Content Extraction Model (MD-TRACE; Rouet, 2006; Rouet & Britt, 2012) was introduced to position the DM within a broader, process-oriented framework. Specifically, the MD-TRACE conceptualizes multiple document use as unfolding through a five-step process. In Step 1, individuals develop a task model or a cognitive representation of task demands and how these may be satisfied. In Step 2, individuals determine that their knowledge is insufficient to meet task demands and, thus, that MSU is necessary (i.e., information need). Step 3 includes the core behaviors associated with MSU. Specifically, in this step, students determine information relevance, process text content, and evaluate sources. These behaviors lead to students’ formation and updating of a documents model to represent their comprehension of multiple texts. Step 4 is dedicated to students’ development of a written product that they believe conforms to task demands. Finally, in Step 5, individuals verify that their written products match task demands and, if necessary, cycle back to earlier phases of the model. Information Problem Solving on the Internet Like the MD-TRACE, another model introducing a process-oriented approach to understanding MSU is the Information Problem Solving on the Internet Model (IPS-I, Brand-Gruwel et  al., 2009). The IPS-I model outlines five constituent skills that students need for online MSU. These are: (a) defining the information problem to be solved or setting goals for information use; (b) searching for and selecting information sources; (c) scanning the available information to determine relevance; and (d) processing some of the information more deeply, before (e) organizing and presenting information to generate a response. Throughout the IPS-I model, information use is supported by students’ reading skills, computer skills, skills with regard to source evaluation, and regulatory processes. Beyond the aforementioned theories, a number of other cognitivist models inform current understandings of students’ interactions with multiple texts (e.g., Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012; Lucassen & Schraagen, 2011).

COLD FACTORS IN MSU A number of conclusions about the nature of students’ MSU can be extracted from the models discussed. First, the MD-TRACE and the IPS-I recognize the importance of task goals in driving MSU. This conclusion is supported by findings that differences in task goals assigned to students result in differences in MSU and task performance (McCrudden, Magliano, & Schraw, 2010; Wiley & Voss, 1999). Second, the MD-TRACE, IPS-I, and the DM are reliant on students’ knowledge and performance of skills necessary for MSU, including those associated with determining information relevance, processing and comprehending texts, and evaluating sources. Despite these skills’ theorized importance, findings have been mixed regarding students’ abilities to perform them. Specifically, while learners have proven adept at making relevance determinations and scanning information quickly, they have been found to experience more

The Cognitive Affective Engagement Model  •  37

challenge with processing content deeply and evaluating sources (Bråten & Strømsø, 2011; Britt & Aglinskas, 2002; Goldman et al., 2012; Wolfe & Goldman, 2005). Third, all three models focus on a task product, conceptualized as either a specific answer to a query (IPS-I; Brand-Gruwel et  al., 2009) or a more elaborated written response (MD-TRACE; Rouet, 2006), as the ultimate outcome of multiple-text use. Most importantly, while all three of these models offer cognitive and procedural accounts of MSU, they do little to consider the more affective dimensions of text processing. So, collectively, we refer to these models as cold approaches to MSU.

WARM PERSPECTIVES ON MULTIPLE SOURCE USE Missing from the prior models is any meaningful conceptualization of how individuals affectively engage in the processing of multiple texts. It is understood that students respond when engaged in reading (Gambrell, 2011; Wigfield & Guthrie, 2000). These responses can be positive or negative, emotionally engaged or disengaged, sparked by internal factors, like students’ interests, motivations, or attitudes toward the topic or a task, or externally driven by features of text or task (Hidi, 2001; Schiefele, 2009; Silvia, 2005). Further, we may expect such affective reactions to arise regardless of whether one or more texts are involved in students’ processing. Motivation for Reading Although under-examined in the literature on MSU, motivation has long been associated with single-text comprehension (Guthrie & Wigfield, 1999; Wigfield & Guthrie, 2000). Guthrie, McGough, Bennett, and Rice (1996) define motivation for reading as “internalized reasons for reading that activate cognitive operations, which enable individuals to perform such acts as acquiring knowledge, enjoying aesthetic experiences, [and] performing tasks” (p. 167). In effect, Guthrie et al. (1996) refer to reading engagement as motivated cognition. When students are motivated to read, they are driven to satisfy reading goals, engage in deep-level strategic processing, activate prior knowledge to thoroughly comprehend content, and modify their initial understanding through reading. Indeed, motivation for reading has been associated with attention allocation (Anderson, 1982; Hidi, 2001), increased persistence when reading (Ainley, Hidi, & Berndorff, 2002; Guthrie et al., 2009), increased and deeper strategy use (Aarnoutse & Schellings, 2003; Pressley & Afflerbach, 1995; Schiefele, 1999), and greater selfregulation and metacognition (Paris & Winograd, 1990; Pintrich, 1999)—all factors that play a role in MSU. Interest Studies of reading engagement have investigated such diverse motivational constructs as interest, intrinsic motivation, achievement goals, task values, and self-efficacy as associated with reading comprehension (Wigfield & Cambria, 2010; Wigfield & Guthrie, 2000). No doubt, all of these factors guide students’ processing and comprehension of both single and multiple texts. Of these motivational constructs, interest may be

38  •  List and Alexander

especially pertinent to understanding MSU. Interest has been defined as the psychological state of arousal or engagement with a particular environmental stimulus (Hidi & Renninger, 2006). This arousal may take on two forms, as either an enduring predisposition toward engagement with some idea, object, or person (i.e., individual interest) or as a fleeting, externally triggered reaction to some stimulus (i.e., situational interest; Alexander, 2003; Hidi & Renninger, 2006). When it comes to MSU, we may expect students to have an abiding or established interest in some topic or domain about which they are reading or to be momentarily intrigued by some issue or task in which they are engaged. Krapp (1999) suggests that interest may be distinguished from other motivational constructs in at least two ways. First, interest, unlike other motivational factors, is not only an internal state, within the student, but rather arises through a person–object interaction. As such, interest represents a learners’ attraction to a particular topic area or activity. Second, interest is considered to be both an affective and cognitive construct (Hidi & Renninger, 2006; Kintsch, 1980; Krapp, 1999). It extends beyond positive feelings of liking or enjoyment to capture students’ cognitive appraisals of the relevance or value of a particular topic or task to them or to others. As such, interest may be an especially pertinent motivational construct to examine in understanding MSU, as it captures students’ desires to learn about a specific topic or issue using multiple texts. Understandably, therefore, interest is the motivational variable most commonly investigated in the emerging literature on motivation and MSU (Bråten et  al., 2014; Bråten & Strømsø, 2006; Grossnickle, 2014; Strømsø & Bråten, 2009). Indeed, initial work in this area has demonstrated that both individual and situational interest are associated with students’ strategy use during multiple-text task completion, as assessed through both log data and self-report (Bråten et al., 2014). Attitudes Just as interest has been described as including both affective and cognitive components, so have students’ attitudes. Ajzen and Fishbein (1977) define attitudes as individuals’ positive or negative evaluations of objects in their environment, which can be conceptualized as falling on a “bipolar evaluative or affective dimension” (p. 889). Attitudes or prior beliefs have been associated with students’ information processing in both single and multiple-text contexts (Kardash & Scholes, 1996; van Strien, BrandGruwel, & Boshuizen, 2014). Specifically, students have been found to apply different evaluative standards when judging attitude-consistent versus attitude-inconsistent information. That is to say, learners tend to more favorably evaluate information that coincides with their existing points of view, while scrutinizing evidence that seems to conflict with their perspectives (Lord, Ross, & Lepper, 1979; Maier & Richter, 2013; McCrudden & Barnes, 2016). More generally, attitudes have been found to be associated with students’ attention allocation, strategy use, and depth of processing when encountering texts (Eagly & Chaiken, 1998; Kardash & Howell, 2000). As with interest, the mechanisms whereby attitudes impact text processing have been considered to be affective and cognitive in nature. “Motivation may affect reasoning through reliance on a biased set of cognitive processes: [including] strategies for accessing, constructing, and evaluating beliefs” (Kunda, 1990, p. 480). In other

The Cognitive Affective Engagement Model  •  39

words, students’ attitudes, alongside other motivational factors, are associated with their cognitive processing of text. Collectively, we refer to motivational and affective constructs like interest and attitudes as warm factors shaping students’ MSU. While extensively explored in single-text processing (Guthrie & Wigfield, 1999; Kardash & Howell, 2000; Wigfield & Guthrie, 2000), these factors have been applied to MSU only to a limited extent. In developing the CAEM, we sought to provide a framework for conceptualizing the role of both cold and warm factors in students’ MSU and cross-textual integration and their relations.

DEVELOPMENT OF THE COGNITIVE AFFECTIVE ENGAGEMENT MODEL Initial conceptions of the CAEM emerged through a thorough review of the literature on single and multiple-text processing (List, 2014). The CAEM was further developed based on our own work examining students’ MSU across a variety of tasks (e.g., List & Alexander, 2015; List, Alexander, & Stephens, 2017; List, Grossnickle, & Alexander, 2016a, 2016b). In particular, we have asked undergraduate students to research complex and controversial topics as varied as children’s linguistic development, planetary habitability, the Arab Spring in Egypt, and overpopulation. The results of these studies were the catalyst for our reconceptualization of MSU. Limited Responses Across the studies we conducted, several intriguing patterns in participants’ written responses emerged. First, we often found a sizable population of students that constructed quite limited responses. These were responses that often-times nominally satisfied the minimum requirements posed by a particular question or task but that, nevertheless, demonstrated limited effort and source use on the part of participants. For instance, when asked: Which planetary features may promote or hinder habitability? Please explain, one student’s response was: “The factors that promote planetary habitability are the presence of oxygen, nitrogen, carbon and hydrogen, mass and water” (List & Alexander, 2015). The composition of such a terse response is, of course, entirely understandable for students asked to complete a cognitively demanding task of little relevance or interest to them as part of a research study. We gained further insight into these types of responses by asking students to rate their confidence in their answers, as well as to justify those ratings (List & Alexander, 2015). Of note in these justifications was the frequency with which participants discussed their interest and motivation, or lack thereof, as predominant factors driving their MSU and response composition. Representative of such participants, one student explained: “It was hard for me to motivate myself to read the passages to find an answer because of my disinterest in the topic,” and another who said, “The topic is extremely boring to me. It was really hard to sit and read the articles when I . . . did not care at all about the topic.” For these students, disinterest or amotivation seemed to drive their engagement with texts, despite these factors being largely absent from leading models of MSU.

40  •  List and Alexander

Quality Responses In contrast to students whose responses were characteristically terse, we observed others who composed particularly thorough and thoughtful responses to assigned tasks. Again, these were students asked to complete complex and cognitively demanding tasks in a laboratory setting. Nevertheless, in responding to the same question on planetary habitability (List & Alexander, 2015), some students produced answers like: Factors promoting or hindering planetary habitability are often related to the fulfillment of basic physiological needs. For example, planets that are able to host various life forms typically require a majority supply of water, readily available to be used as sustenance for existing life forms on the planet (The New York Times). Additionally, a varying range of temperatures, typically affected by orientation to the sun, offers the planet’s organisms a variety of living conditions and allows them to choose one that is most suited to their systems (The New York Times). One such recently discovered planet, Gliese 581g, could support diverse life forms because of its varying temperatures and abundant water supply, among other factors (The New York Times). This response is notable not only for its length, but also for its organization, inclusion of well-elaborated examples and explanations, and references to specific sources read. Students’ justifications for ratings of response confidence offer additional insights into MSU (List & Alexander, 2015). Students producing written responses of a higher quality also tended to base their response ratings on evaluations of the trustworthiness and quality of the sources they used in composing their answers and the accuracy of the information they contained—evaluation criteria rarely mentioned by other participants. One student justified her response confidence as based on corroboration and evaluation of the trustworthiness of the sources she used in composing her response: I used two different sources and read the same answers. Even though Wikipedia isn’t that reliable, I read a journal article that presented the same information. The journal article is a very reliable source, so I feel pretty confident in my answer. Another student considered author credentials and the scientific processes involved in gathering the information he used in composing his answer: “I am confident in this response because it is from a journal. This journal is probably peer-reviewed, and the information contained in the article was probably found using research methods, so it has at least some validity to it.” This focus on evaluating source and information quality was uniquely exhibited by some students and not others. In developing the CAEM, we were motivated to answer the fundamental question of which factors resulted in some students developing only brief, perfunctory responses to multiple-text tasks, while other students composed elaborated responses, based on considerations of source trustworthiness and reflecting multiple-text integration and corroboration.

The Cognitive Affective Engagement Model  •  41

Engaged Responses The most comprehensive set of data we have on undergraduates’ MSU to date comes from a study asking students to write an argument about who should hold power in Egypt, following the Arab Spring (List, 2014; List et al., 2017). As in our prior work, the resulting responses included two extremes in students’ written products, with regard to detail and quality. However, within these data, a third type of response emerged. This response group reflected extensive engagement with the information presented across multiple texts, while at the same time demonstrating limited critical analysis or source evaluation. One illustrative response was as follows: The United States should support General el Sisi and the military regime currently in power in Egypt for many reasons. General el Sisi is looking out for the Egyptian public, not a specific or particular group. In the public survey, they discuss how . . . 79% of all Egyptians want national reconciliation and with the help of Sisi, Egypt believes their country could be restored. In the blog post Sisi promised to protect peaceful protest [and is] opposed to the protest taking place by Muslim Brotherhood protestors. Sisi also promised to only use violence against those who are violent protestors . . . With the power being under Sisi there is greater control of violent attacks and protestors. This response included information from multiple sources. However, it was decidedly one-sided in its endorsement of General el Sisi, despite the sources cited actually presenting conflicting information about the benefits and limitations of El Sisi’s rule and who should hold power in Egypt. Further, this response evidenced limited integration, separately presenting information gathered from each source, in a sequential fashion, while drawing limited connections between texts. Moreover, in listing reasons for supporting General el Sisi, this respondent drew on a variety of sources without evident regard for source quality or bias. Put in DM terms, this response illustrates a mush model approach to multiple-text integration (Britt et al., 1999). In this response information from across multiple texts was nominally connected due to its support for General el Sisi, but no consideration of conflicting information presented by these same documents was evidenced. To appreciate the non-evaluative nature of the prior response, it helps to contrast it with other, more critical responses in which the students questioned the claims and evidence included in sources, through corroboration or by introducing counterpoints, or noted the varying quality of the sources themselves. As one more critical student wrote: Given the articles I have at my reach here, I know that El Sisi has suspended the Egyptian constitution and thus limited the rights of the public . . . However, Morsi’s constitution was lop-sided as it allowed him total rule in terms of decision making and it could not be appealed in a democratic manner whatsoever. Another student discussed the relative trustworthiness and merits of Twitter as an information source:

42  •  List and Alexander

Another source is Twitter. Twitter is rather unreliable, but often does post true and current events as they happen. The problem is that it is very hard to verify unless there are pictures with a date stamp, or video that has some sort of timetable proof. As these examples illustrate, key distinguishers of the quality of students’ responses were their consideration and evaluation of sources and the information they contained. These included students’ judgments of different document types, considerations of author credibility and bias, comparisons of information across sources, and analyses of information quality. Indeed, evaluative processes have repeatedly been identified as core to MSU (Braasch, Rouet, Vibert, & Britt, 2012; Britt et al., 1999; Stadtler & Bromme, 2014) and as markers of successful student engagement with multiple texts (Goldman et al., 2012; Wiley et al., 2009). As conceptualized in the DM, such responses may be considered to be reflective of documents model construction (Britt et al., 1999). These responses may be distinguished from the mush model previously exemplified through their explicit and evaluative consideration of document information and their reconciliation of conflicting information presented across texts. Based on our analysis of the various typologies of students’ responses, we developed the CAEM. The CAEM pertains to reading situations that can be characterized in four ways. At the most basic level, the CAEM conceptualizes students’ engagement with multiple textual sources. Models exist that, like the CAEM, specify the role of cognitive and motivational factors in single-text processing (e.g., Guthrie & Wigfield, 1999; Pintrich, Marx, & Boyle, 1993). However, the CAEM is specifically focused on reading situations requiring the understanding, integration, and reconciliation of information across multiple sources of information. Further, the CAEM considers students’ engagement with multiple texts in response to a task either externally assigned or internally generated by the learner (McCrudden & Schraw, 2007). While some models consider the tasks assigned to learners to be directly assumed (McCrudden & Schraw, 2007), the CAEM conceptualizes even externally assigned tasks to be reinterpreted by learners prior to task completion. Moreover, the CAEM addresses MSU in response to complex tasks, requiring elaborated responses based on information in texts. Such complex tasks require that students produce external products (e.g., written responses) for task resolution, corresponding to learning. Finally, in contrast to existing models in the field (e.g., Braasch et al., 2012; Stadtler & Bromme, 2014), the CAEM addresses MSU when multiple sources relate to one another in a myriad of ways, including agreeing with, complementing, contradicting, or elaborating upon another. In formulating this model, we were specifically interested in how students’ motivations and evaluations of information sources alter the course of their engagement with multiple texts.

MULTIPLE SOURCE PROCESSING ACCORDING TO THE CAEM The CAEM conceptualizes students’ MSU as including three distinct phases. MSU commences with (a) an initiating task; in response to this task, students adopt (b) a default stance toward multiple-text engagement; this default stance corresponds to students’ execution of a host of (c) MSU processes and behaviors. The three phases of the CAEM are depicted in Figure 3.1.

The Cognitive Affective Engagement Model  •  43

Initiating Task

Phase 1

Interpreted by students in terms of:

Results in: Phase 2

Habituated Source Evaluation Behavioral and MSU Skills Dispositions

Attitudes toward, interest in, etc. toward the Topic

Expected Cognitive Products

Assumption of a Default Stance

Evaluative

Critical Analytic

Disengaged

Affective Engagement

Affective Engagement Motivation (i.e., interest, attitudes)

Manifests as:

Phase 3

Multiple Source Use Behaviors (e.g., time on texts, text access, text processing, sourcing, cessation)

Figure 3.1  Cognitive Affective Engagement Model of Multiple Source Use.

INITIATING TASK In keeping with prior models (Brand-Gruwel et al., 2009; Rouet, 2006; Rouet & Britt, 2012), the CAEM posits the process of MSU as beginning with an initiating task. Whether externally assigned or self-generated, this initiating task serves to set the goals and establish the parameters for multiple-text engagement. According to the CAEM, any initiating task may be defined by two key characteristics: its topic or focus and the cognitive product that students may expect to result from MSU. According to the MD-TRACE, the task model that students develop to initiate MSU is a cognitive representation of task demands that unites the instructions that students receive with a host of pragmatic factors, like the variety of documents available to students and the time required for task completion (Rouet & Britt, 2012). In the CAEM, we conceptualize the cognitive product aspect of an initiating task more narrowly, as the specific learning or knowledge outcomes that students expect as a consequence of multiple-text engagement. In this way, the CAEM emphasizes the importance of students’ personal re-interpretations of task demands in guiding MSU. The cognitive

44  •  List and Alexander

products that students expect to emerge may be distinguished according to whether those expectations reflect a desire for an expedient, albeit shallow, response or for deeper examination and understanding (Alexander & DRLRL, 2012). As such, these expectations serve as initial indicators of students’ motivation for task completion and their ensuing level of engagement with multiple texts. Prior work has typically shown little regard for topic when investigating the role of different task features in guiding MSU. The CAEM is unique in identifying the topic of the task as a key factor in determining how students perceive and approach MSU. Drawing on investigations of constructs like interest and prior attitudes (Ainley et al., 2002; Hidi, 2001; Kardash & Howell, 2000), we argue that in the CAEM that the specific topic students investigate using multiple sources plays a defining motivational and cognitive role. For instance, while some topics may be interesting or important to students, others may not be. Further, while some topics may address controversial issues, like climate change or evolution, other topics may be of a more quotidian nature.

DEFAULT STANCE Once students represent an initiating task in terms of its topic and the cognitive products expected to result, they then adopt a default stance toward the task. A default stance constitutes a cognitive and motivational orientation toward a particular multiple source task. The phrase default stance is used purposefully to emphasize students’ initial starting points in approaching multiple-text tasks, with the recognition that such stances may evolve during the course of task completion. At the same time, students’ default stances or initial orientations toward MSU may be expected to have a somewhat general pattern across many multiple-text situations. Moreover, in electing to use the term default stance we emphasize that students’ orientation toward a particular task may arise somewhat automatically or heuristically, without particular metacognitive engagement on the part of the learner. The default stance is defined by two axes: students’ affective engagement with and their behavioral dispositions toward MSU and evaluation. The affective engagement dimension captures students’ motivational orientation toward the initiating task, including their initial levels of individual and situational interest and existing attitudes. The behavioral dispositions dimension reflects students’ habituated practices with regard to source use and evaluation, that may be employed across various task conditions. These axes and resulting default stance profiles are depicted in Figure 3.1. Depending on their positioning along both of these axes, students may be expected to adopt one of four default stances toward any initiating task. At one extreme, there are those students who possess well-developed habits with regards to source evaluation and a high level of affective engagement with a particular initiating task. These students are both highly engaged by and sophisticated in their MSU, accessing rich sources and critically analyzing and evaluating these sources and their contents to achieve deep understanding of the topic. We refer to those occupying this quadrant as manifesting a critical-analytic default stance. At the other extreme, there are students who exhibit neither a high degree of affective engagement with a particular task nor robust habits with regard to source evaluation. We label those manifesting this default stance as disengaged. We would expect disengaged students to be limited both in terms of the volume of information they access during MSU and the depth with which they process the resulting content.

The Cognitive Affective Engagement Model  •  45

Two additional default stances fall at the extremes of each axis. Students who are highly engaged by the topic of a particular initiating task, without evidencing any evaluative behaviors, may be said to adopt an affectively engaged default stance. These students may be distinguished by the volume of information they process during MSU, while not engaging in particularly critical or deep-level processing. Conversely, students habituated to engaging in behaviors associated with the evaluation of multiple texts, even absent particularly high levels of motivation for an initiating task, may adopt an evaluative default stance toward MSU. These students may be characterized by their engagement in behaviors associated with source evaluation. However, these evaluations may only be heuristic or perfunctory in nature. When enacted, each default stance profile adopted may result in students carrying out a distinct pattern of source use processes and behaviors. Such processes and behaviors, captured in the empirical literature on MSU, will be discussed in light of insights from the CAEM.

MSU PROCESSES AND BEHAVIORS Existing models have identified a host of behaviors associated with MSU (BrandGruwel et  al., 2009; Rouet & Britt, 2012). These include students’ text access, processing and strategy use, source evaluation, and, ultimately, cessation of engagement with multiple texts. The CAEM considers these same behaviors but is more specific in hypothesizing that, while all learners may engage in the same behaviors during MSU, the nature of these behaviors differs according to the stances students adopt. Text Access Text access is defined by the number and type of texts students visit during task completion. We may expect students adopting a disengaged stance toward MSU to be especially limited in their text access, visiting few sources. Conversely, students characterized by an affectively engaged stance may be expected to access a substantial number of texts, irrespective of trustworthiness. Relative to the two extremes of disengaged or affectively engaged learners, those adhering to an evaluative or critical-analytic stance may be expected to access a moderate number of sources. Moreover, what would distinguish these two profiles from their counterparts would be their preference for accessing sources they regard as high in trustworthiness or reliability. The empirical literature offers initial support for these hypothesized patterns of text access. For example, in a study involving a single text, Ainley et al. (2002) found students demonstrating high levels of individual and topic interest to be more persistent in text access. Persistence was operationalized as engagement or the number of text sections students were willing to process and the amount of time they devoted to text access. Further, Goldman et al. (2012) found a key differentiator of students learning a great deal or disproportionately little from a multiple-text task to be their engagement in source evaluation. Specifically, students could be distinguished according to the ratio of time they devoted to accessing reliable versus unreliable texts and the number of visits and revisits they had to these two source types. Likewise, Anmarkrud, Bråten, and Strømsø (2014) found students’ think-aloud utterances evidencing evaluative

46  •  List and Alexander

strategy use to be associated not only with their discounting of untrustworthy sources during a ranking task, but also with performance on an argumentative writing task. Processing A fundamental distinction drawn in the literature on reading comprehension is between surface- and deep-level processing (Dinsmore & Alexander, 2012; Salmerón, Naumann, García, & Fajardo, 2017; Wolfe & Goldman, 2005). While surface-level processing is superficial in nature, deep-level processing has been associated with the deployment of strategies for developing understanding, building connections, evaluating content, monitoring performance, and considering new information in light of prior knowledge. Such strategies have been shown to result in improved comprehension. Within the context of MSU, Bråten and Strømsø (2011) carry this distinction forward by separating out low-level MSU strategies focused on information accumulation from deep-level strategies addressing cross-textual elaboration. Default stances identified in the CAEM may be distinguished according to their relative engagement in surface- versus deep-level strategies during MSU. Students in the disengaged stance may be expected to primarily rely on low-level strategies, like scanning to quickly identify a response located explicitly in text (Salmerón et al., 2016). For the disengaged profile, we would expect deep-level strategy use to be minimal, as these students would be unmotivated to engage in the cognitively demanding processing necessary for high-level strategy use. Students classified as affectively engaged may adopt strategies associated with information accumulation. These students may be eager to learn as much as possible about their topic of interest, but may not have the strategic or evaluative skills in place to engage in the deep-level processing necessary for multiple-text corroboration and integration. Students adopting an evaluative default stance may possess the strategic abilities to both accumulate information and evaluate source quality and content, but may elect to use the latter set of skills only to a limited extent. Those fitting the critical-analytic default stance would be expected to manifest deep-level strategies in their MSU. This approach to text access would include students both well practiced at the skills associated with mapping and evaluating relations among texts and sufficiently motivated to implement these strategies. In conceptualizing the role of deep-level processing in MSU, the CAEM emphasizes that students’ motivation to engage is a necessary condition for such processing to occur. Evaluation Source evaluation has been associated with deep-level processing (Afflerbach & Cho, 2009; Bråten & Strømsø, 2011) and has been identified as essential for supporting multiple-text comprehension and integration (Britt et al., 1999; Perfetti et al., 1999; Stadtler & Bromme, 2014). Yet, students have been found to engage in the processes associated with text evaluation, like considering source information or citing sources, only rarely (Britt & Aglinskas, 2002; List et al., 2017). Moreover, even when evaluating texts, students have been found to rely on heuristics or superficial cues (e.g., document type) to judge text trustworthiness (Brem, Russell, & Weems, 2001; List et al., 2017), rather than engaging in deeper analysis.

The Cognitive Affective Engagement Model  •  47

The evaluation of texts may be considered to be a key feature distinguishing different CAEM default stances. Disengaged and affectively engaged students likely evaluate sources only infrequently, if at all. In contrast, students manifesting an evaluative or critical-analytic default stance are characterized by their tendency to evaluate sources to varying degrees. Specifically, we might expect those taking on an evaluative stance to routinely judge texts, perhaps by using some general “rule of thumb” for source evaluation (e.g., “Rely on journal articles and avoid blogs.”). By comparison, those adopting a critical-analytic stance would do so more deliberately or in a manner more sensitive to the contextual features of the topic or the task. For instance, students adhering to a critical-analytic stance might find that forms of social media, while typically not reliable sources of information, are potentially credible sources in the case of very contemporary and unfolding events (e.g., a mounting protest). We would also hypothesize that these critical-analytic students would demonstrate cross-textual corroboration and evaluation in the service of forming an integrated representation of multiple texts. Cessation Limited work has considered why students may stop accessing information from multiple texts. The CAEM considers the end of MSU (i.e., cessation) to be a particularly important step for two primary reasons. First, this is a critical regulatory moment during which students reflect on their information access and determine that their desired cognitive products have been sufficiently formed. Second, cessation represents a key juncture between students’ engagement in information access and response composition; a transition identified as important in other theoretical models of MSU (BrandGruwel et al., 2009; Rouet & Britt, 2012). The four CAEM profiles are all expected to differ according to the markers used to determine when the need for information is satisfied (i.e., cessation). Students exhibiting a disengaged profile may simply cease source use once a minimally satisfying response to an initiating task has been crafted. Students in the affectively engaged profile may cease text access for external, rather than internal, reasons, such as when the time allocated for the location of sources has been reached. Motivated to accumulate information, affectively engaged students might also cease source access when they achieve content saturation; that is, when little new information is able to be located. Cessation for students adopting an evaluative stance may be triggered by habits of practice or generic thresholds, such as consulting three sources from a Google search results page. Finally, critical-analytic students might decide to cease text access by considering specific criteria associated with topic or task or by judging their cognitive products to be successfully developed. For instance, when presented with a controversial topic, criticalanalytic students may cease source use only after seeking out several credible sources on both sides of an issue to make an evidence-based decision about which position to support. In a rare study examining a cessation-related construct, students’ justifications for leaving webpages, Goldman et al. (2012) found the reasoning of better and poorer learners to differ. Specifically, while poorer learners left webpages because they lacked keywords matching exactly to their search query, better learners were significantly more likely to leave webpages with the desire to seek out additional information. Such differences in reasoning point to distinctions suggested by the CAEM.

48  •  List and Alexander

CAEM-LIKE PROFILES IN PRIOR RESEARCH Prior research has provided initial evidence for the emergence of CAEM-like profiles in students’ multiple-text use. For example, Lawless and Kulikowich (1996) used cluster analysis to identify three profiles of undergraduates’ hypertext navigation. Specifically, Lawless and Kulikowich characterized students as belonging to apathetic, knowledge-seeking, and feature-exploring user profiles. Apathetic navigators were characterized as devoting limited time to information access, overall, and as being restricted in their access of both informational content and multimedia features. Knowledge-based navigators constituted a user profile characterized by their particularly strong attendance to information-based hypertexts. As a contrast, feature explorers were navigators devoting particular attention to the multimedia features in hypertext, over informational content, while also spending a substantial amount of time on hypertext access. Although imperfect analogs, these user profiles can be said to correspond to disengaged, critical-analytic, and affectively engaged CAEM profiles, respectively. Moreover, as hypothesized by the CAEM, students belonging to the apathetic/disengaged user profile performed the lowest, with feature explorers/affectively engaged students performing at or below the average, and knowledge seekers/critical-analytic users demonstrating the highest level of performance. Like Lawless and Kulikowich (1996), Killi, Laurinen, and Marttunen (2008) used cluster analysis of think-aloud and log data to classify secondary students into five profiles of source evaluation. The profiles identified included: versatile evaluators, relevance-oriented evaluators, limited evaluators, disoriented readers, and uncritical readers. These profiles were distinguished according to a number of dimensions. Limited evaluators, disoriented readers, and uncritical readers were limited in their source evaluation. As a contrast, students in the versatile evaluators profile produced a large number of evaluation-related utterances, overall, along with reporting a high number of utterances specifically related to source credibility. By comparison, relevance-oriented evaluators were particularly noteworthy for the number of relevance-related comments they produced, leading to a moderate number of evaluations overall, but a low number of credibility-related utterances proffered. Killi et al. (2008) delved deeply into students’ source evaluations, as emphasized in the CAEM, to produce student profiles corresponding to disengaged (e.g., dis­ oriented readers, uncritical readers) and critical-analytic source users (i.e., versatile evaluators). Whereas Lawless and Kulikowich (1996) emphasize the degree of learner engagement (i.e., time spent on hypertext access, types of hypertexts accessed) in defining navigation profiles, Killi et  al. (2008) emphasize the importance of source evaluation in defining profiles of students’ multiple-text use. The CAEM unifies these two approaches in a single framework and attributes emergent profiles to a host of individual difference factors and their interactions, allowing for targeted learner development and intervention.

FUTURE DIRECTIONS The CAEM is certainly not the first model to suggest that discernible profiles may be identified in students’ MSU (e.g., Alexander & DRLRL, 2012; Lawless &

The Cognitive Affective Engagement Model  •  49

Kulikowich, 1996; Reader & Payne, 2007). Nevertheless, the CAEM is comprehensive in associating individual difference factors, both cognitive and motivational, with the behaviors students manifest during MSU and their quality. There is certainly much work to be done in ascertaining the viability of this model. For one, although developed based on empirical evidence, the CAEM is only a theoretical model. As such, the proffered descriptions of students’ MSU remain highly speculative in nature. Empirical investigations are necessary to specifically evaluate CAEM profiles, as well as the factors and processes upon which they are grounded. In addition, the CAEM is limited in the variety of individual difference factors it considers, focusing on students’ interest, attitudes, and source evaluation behaviors. Unquestionably, a host of factors, including prior knowledge, need for cognition, selfregulation and metacognition, and epistemic beliefs, have been identified as important for both single and multiple-text comprehension. In the future, we would hope to expand the person-centered factors examined in the CAEM, but for now the imperative is to test the model, as articulated, under varying conditions. The latter point regarding varying conditions brings forward questions of setting, topics, and tasks. It is critical to ascertain whether the CAEM holds not only for laboratory-based studies, when topics are controversial and the catalogue of sources available pre-specified, but also for search tasks routinely undertaken in classrooms, when topics are more mundane and students are relatively free to locate sources on the Internet. Moreover, in light of the significance of motivational factors in the CAEM, it is imperative to explore patterns in MSU when individuals are free to engage in a task of their own choosing, for their own reasons. Such a contrast between other-given and self-directed MSU would be highly informative to both the CAEM and the broader literature on MSU. Indeed, the importance or value that students assign to different tasks would have important implications not only for their conception of task demands and assumption of a default stance, but also for their decisions about text access and cessation and ultimate task product. Finally, existing models of MSU, including the CAEM, deal in a limited way with issues of development, including how these models may function for learners of differing ages as well as the nature of changes in individuals’ MSU over time. Based on the extant literature, for example, there seems to be the expectation that the cognitive capacities necessary to evaluate conflicting points of view and make judgments based on evidence, independently, are more likely to emerge in late childhood or early adolescence (Hofer & Pintrich, 1997; Kuhn, 1999). However, with assistance and when working in an area for which they have sufficient background knowledge, even threeor four-year-olds can demonstrate a preference for trustworthy rather than unreliable sources (Koenig, Clément, & Harris, 2004). Additionally, research on false-beliefs has documented children’s emergent conceptions of knowledge as potentially true or false (Chandler, Hallett, & Sokol, 2002) and intervention work has found that elementary students’ evaluations of information sources can be developed (Macedo-Rouet, Braasch, Britt, & Rouet, 2013). Considering these findings, the CAEM suggests that students, even at very young ages, are capable of engaging in processes associated with MSU when certain conditions are in place (e.g., familiar topic, limited and accessible sources, given students’ age and reading level, and teacher scaffolding). While existing research has only

50  •  List and Alexander

examined MSU in elementary-school populations to a limited extent (VanSledright & Kelly, 1998; Wallace, Kupperman, Krajcik, & Soloway, 2000), we suggest that limitations in students’ MSU come less from developmental limitations, and more from the dearth of experiences that prime MSU. Future Directions for Investigation Many empirical avenues exist for investigating the CAEM. We suggest two directions for initial research stemming from this model. First, students engaged in MSU should be surveyed or interviewed to be positioned in accordance with the default stances specified in the CAEM. The relative distinctions among and prevalence of students’ default stances should be established and these should be connected with students’ manifest source use behaviors. Second, the extent to which various default stance profiles may be engendered in students should also be explored. This could be done through careful topic selection or through more direct intervention, by including specific instructions assigning students to evaluate texts. The correspondence of these assigned default stance profiles to students’ behaviors during MSU should then be investigated.

INSTRUCTIONAL IMPLICATIONS Despite the years of research that lie ahead for the CAEM, there are several instructional implications that presently warrant consideration. First, there is ample evidence that students do not often develop strategies, especially deep-processing strategies, without guidance (Paris & Winograd, 1990; Pressley & Afflerbach, 1995). That admonition applies to strategies required for locating, evaluating, and integrating texts from multiple sources as well (Bråten & Strømsø, 2011; Britt & Aglinskas, 2002). Given the growing presence of MSU in online and classroom contexts, it is advisable for teachers to explicitly instruct students in how to engage in effective online search, comprehend information, corroborate and evaluate sources, and monitor comprehension to support students in developing integrated representations of topics or issues based on multiple texts. Related to the point of explicit instruction, it also seems wise for teachers to provide students with ample opportunities to engage in MSU. In this way, students become familiar and somewhat habituated to processing multiple texts (Brem et al., 2001). Yet, simultaneously students should have the chance to process multiple texts under varying conditions. In this way, they may come to recognize how differing topics or task parameters should translate into deviations in their default approach to MSU. This recognition, of course, is more likely to occur with teacher direction and modeling. Finally, motivational and affective factors, which are a defining feature of the CAEM, are key to students’ successful engagement in MSU (Bråten et al., 2014; Grossnickle, 2014). Pedagogically, therefore, teachers need to create learning conditions that should enhance the motivational and affective dimensions of MSU for students. The literature suggests several steps that teachers can take in this regard. For one, offering students a choice of the topic to be explored or a role in defining the task to be completed could stimulate investment in MSU (Wigfield & Cambria, 2010). In addition, when students perceive the tasks to be performed as directly relevant to their personal interests and experiences, the “value” of that task should increase (Gambrell, 1996; Hidi, 2001;

The Cognitive Affective Engagement Model  •  51

Wigfield & Guthrie, 2000). Greater commitment to and persistence with the task should follow from this added value (Ainley et al., 2002).

CONCLUSION We conclude this chapter by outlining a number of specific hypotheses that ought be evaluated in establishing CAEM validity. These include examining the role of motivational factors, namely topic interest and attitudes, in manifest source use behaviors (e.g., time on texts). Likewise, the CAEM demands an examination of the role of MSU evaluation skills and habits as they manifest during task completion (e.g., accessing document information). Most of all, the interactions among these motivational and cognitive factors need to be examined as evidence during task completion and through performance. To evaluate these hypotheses, there is a need to reliably assess students’ (a) motivation for task completion, including individual interest and attitudes toward the assigned topic, and situational interest as arising through task completion, (b) skills and habits with regard to source evaluation and MSU, and (c) processing during multiple-text use. There is a corresponding need to examine varied data sources in assessing each of these constructs, including self-report (e.g., survey), behavioral (e.g., log data, eye-tracking), and cognitive (e.g., think-aloud) indicators. Finally, qualitative data may help us to understand underspecified aspects of the CAEM including students’ expectations around cognitive products resulting from task completion, assumptions of a default stance, and decisions regarding MSU cessation. In this chapter, the Cognitive Affective Engagement Model of Multiple Source Use is explicated. This model draws on earlier efforts in the MSU literature to describe the cognitive processes involved in students’ engagement with multiple texts, as well as literature on single-text processing emphasizing the importance of motivational factors. Building on the extant literature, the CAEM introduces a three-step process characterizing students’ MSU and identifies four profiles that depict students’ typical pattern of interaction with multiple texts. Through its incorporation of motivational and cognitive factors in readers’ interactions with multiple texts, and its consideration of task, externally given or personally formed, the CAEM serves as a valuable alternative to prior models. Of course, much more theoretical and empirical work is required to provide critical evidence for the viability of this new model, and to lay the groundwork for classroom-based interventions.

REFERENCES Aarnoutse, C., & Schellings, G. (2003). Learning reading strategies by triggering reading motivation. Educational Studies, 29(4), 387–409. Afflerbach, P., & Cho, B. Y. (2009). Identifying and describing constructively responsive comprehension strategies in new and traditional forms of reading. In S. E. Israel & G. G. Duffy (Eds.), Handbook of research on reading comprehension (pp. 69–90). London: Routledge. Ainley, M., Hidi, S., & Berndorff, D. (2002). Interest, learning, and the psychological processes that mediate their relationship. Journal of Educational Psychology, 94(3), 545–561. Ajzen, I., & Fishbein, M. (1977). Attitude-behavior relations: A theoretical analysis and review of empirical research. Psychological Bulletin, 84(5), 888–918. Alexander, P. A. (2003). The development of expertise: The journey from acclimation to proficiency. Educational Researcher, 32(8), 10–14.

52  •  List and Alexander Alexander, P. A., & Fox, E. (2004). Historical perspective on reading research and practice. In R. B. Ruddell & N. Unrau (Eds.), Theoretical models and processes of reading (pp. 33–68). Newark, DE: International Reading Association. Alexander, P. A., & The Disciplined Reading and Learning Research Laboratory. (2012). Reading into the future: competence for the 21st century. Educational Psychologist, 47(4), 259–280. Anderson, R. C. (1982). Allocation of attention during reading. Advances in Psychology, 8, 292–305. Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. Braasch, J. L., Rouet, J. F., Vibert, N., & Britt, M. A. (2012). Readers’ use of source information in text comprehension. Memory and Cognition, 40(3), 450–465. Brand-Gruwel, S., Wopereis, I., & Walraven, A. (2009). A descriptive model of information problem solving while using internet. Computers & Education, 53(4), 1207–1217. Bråten, I., Anmarkrud, Ø., Brandmo, C., & Strømsø, H. I. (2014). Developing and testing a model of direct and indirect relationships between individual differences, processing, and multiple-text comprehension. Learning and Instruction, 30, 9–24. Bråten, I., & Strømsø, H. I. (2006). Epistemological beliefs, interest, and gender as predictors of Internet-based learning activities. Computers in Human Behavior, 22(6), 1027–1042. Bråten, I., & Strømsø, H. I. (2011). Measuring strategic processing when students read multiple texts. Metacognition and Learning, 6(2), 111–130. Brem, S. K., Russell, J., & Weems, L. (2001). Science on the Web: Student evaluations of scientific arguments. Discourse Processes, 32(2–3), 191–213. Britt, M. A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20(4), 485–522. Britt, M. A., Perfetti, C. A., Sandak, R., & Rouet, J. F. (1999). Content integration and source separation in learning from multiple texts. In S. R. Goldman, A. C. Graesser, & P. van den Broek (Eds.), Narrative comprehension, causality, and coherence: Essays in honor of Tom Trabasso (pp. 209–233). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Chandler, M. J., Hallett, D., & Sokol, B. W. (2002). Competing claims about competing knowledge claims. In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 145–168). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Dinsmore, D. L., & Alexander, P. A. (2012). A critical discussion of deep and surface processing: What it means, how it is measured, the role of context, and model specification. Educational Psychology Review, 24(4), 499–567. Eagly, A., & Chaiken, S. (1998). Attitude structure. In D. Gilbert, S. Fiske, & G. Lindsay (Eds.), Handbook of social psychology (pp. 269–322). New York: McGraw-Hill. Gambrell, L. B. (1996). Creating classroom cultures that foster reading motivation. The Reading Teacher, 50(1), 14–25. Gambrell, L. B. (2011). Seven rules of engagement: What’s most important to know about motivation to read. The Reading Teacher, 65(3), 172–178. Goldman, S. R., Braasch, J. L., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47(4), 356–381. Grossnickle, E. M. (2014). The expression and enactment of interest and curiosity in a multiple source use task. Unpublished dissertation thesis. Guthrie, J. T., McGough, K., Bennett, L., & Rice, M. E. (1996). Concept-oriented reading instruction: An integrated curriculum to develop motivations and strategies for reading. In L. Baker, P. Afflerbach, & D. Reinking (Eds.), Developing engaged readers in school and home communities (pp.  165–190). Mahwah, NJ: Erlbaum. Guthrie, J. T., McRae, A., Coddington, C. S., Lutz Klauda, S., Wigfield, A., & Barbosa, P. (2009). Impacts of comprehensive reading instruction on diverse outcomes of low- and high-achieving readers. Journal of Learning Disabilities, 42(3), 195–214. Guthrie, J. T., & Wigfield, A. (1999). How motivation fits into a science of reading. Scientific Studies of Reading, 3(3), 199–205.

The Cognitive Affective Engagement Model  •  53 Hidi, S. (2001). Interest, reading, and learning: Theoretical and practical considerations. Educational Psychology Review, 13(3), 191–209. Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111–127. Hofer, B. K., & Pintrich, P. R. (1997). The development of epistemological theories: Beliefs about knowledge and knowing and their relation to learning. Review of Educational Research, 67(1), 88–140. Kardash, C. M., & Howell, K. L. (2000). Effects of epistemological beliefs and topic-specific beliefs on undergraduates’ cognitive and strategic processing of dual-positional text. Journal of Educational Psychology, 92(3), 524–535. Kardash, C. M., & Scholes, R. J. (1996). Effects of pre-existing beliefs, epistemological beliefs, and need for cognition on interpretation of controversial issues. Journal of Educational Psychology, 88(2), 260–271. Killi, C., Laurinen, L., & Marttunen, M. (2008). Students evaluating Internet sources: From versatile evaluators to uncritical readers. Journal of Educational Computing Research, 39(1), 75–95. Kintsch, W. (1980). Learning from texts, levels of comprehension, or: Why anyone would read a story anyway. Poetics, 9, 87–98. Koenig, M. A., Clément, F., & Harris, P. L. (2004). Trust in testimony: Children’s use of true and false statements. Psychological Science, 15(10), 694–698. Krapp, A. (1999). Interest, motivation and learning: An educational-psychological perspective. European Journal of Psychology of Education, 14(1), 23–40. Kuhn, D. (1999). A developmental model of critical thinking. Educational Researcher, 28(2), 16–46. Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480–498. Lawless, K. A., & Kulikowich, J. M. (1996). Understanding hypertext navigation through cluster analysis. Journal of Educational Computing Research, 14(4), 385–399. List, A. (2014). Modeling multiple source use: Using learner characteristics and source use behaviors to predict response quality. Unpublished dissertation thesis. List, A., & Alexander, P. A. (2015). Examining response confidence in multiple text tasks. Metacognition and Learning, 10(3), 407–436. List, A., Alexander, P. A., & Stephens, L. A. (2017). Trust but verify: Examining the association between students’ sourcing behaviors and ratings of text trustworthiness. Discourse Processes, 54(2), 83–104. List, A., Grossnickle, E. M., & Alexander, P. A. (2016a). Profiling students’ multiple source use by question type. Reading Psychology, 37(5), 753–797. List, A., Grossnickle, E. M., & Alexander, P. A. (2016b). Undergraduate students’ justifications for source selection in a digital academic context. Journal of Educational Computing Research, 54(1), 22–61. Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37(11), 2098–2109. Lucassen, T., & Schraagen, J. M. (2011). Factual accuracy and trust in information: The role of expertise. Journal of the Association for Information Science and Technology, 62(7), 1232–1242. Macedo-Rouet, M., Braasch, J. L. G., Britt, M. A., & Rouet, J.-F. (2013). Teaching fourth and fifth graders to evaluate the cognitive authority of information sources during text comprehension. Cognition and Instruction, 31, 204–226. Maier, J., & Richter, T. (2013). Text belief consistency effects in the comprehension of multiple texts with conflicting information. Cognition and Instruction, 31(2), 151–175. McCrudden, M. T., & Barnes, A. (2016). Differences in student reasoning about belief-relevant arguments: A mixed methods study. Metacognition and Learning, 11(3), 1–29. McCrudden, M. T., Magliano, J. P., & Schraw, G. (2010). Exploring how relevance instructions affect personal reading intentions, reading goals and text processing: a mixed methods study. Contemporary Educational Psychology, 35(4), 229–241. McCrudden, M. T., & Schraw, G. (2007). Relevance and goal-focusing in text processing. Educational Psychology Review, 19(2), 113–139. Paris, S. G., & Winograd, P. (1990). How metacognition can promote academic learning and instruction. Dimensions of Thinking and Cognitive Instruction, 1, 15–51. Perfetti, C. A., Rouet, J.-F., & Britt, M. A. (1999). Towards a theory of documents representation. In H. van Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 99–122). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

54  •  List and Alexander Pintrich, P. R. (1999). The role of motivation in promoting and sustaining self-regulated learning. International Journal of Educational Research, 31(6), 459–470. Pintrich, P. R., Marx, R. W., & Boyle, R. A. (1993). Beyond cold conceptual change: The role of motivational beliefs and classroom contextual factors in the process of conceptual change. Review of Educational Research, 63(2), 167–199. Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Reader, W. R., & Payne, S. J. (2007). Allocating time across multiple texts: Sampling and satisficing. Human– Computer Interaction, 22(3), 263–298. Ricoeur, P. (1991). What is a text? In P. Ricoeur, From text to action: Essays in hermeneutics (pp. 105–124). London: The Athlone Press. Rouet, J. F. (2006). The skills of document use: From text comprehension to Web-based learning. Mahwah, NJ: Erlbaum. Rouet, J. F., & Britt, M. A. (2012). Relevance processes in multiple documents comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 19–52). Charlotte, NC: Information Age Publishing. Salmerón, L., Naumann, J., García, V., & Fajardo, I. (2017). Scanning and deep processing of information in hypertext: An eye tracking and cued retrospective think‐aloud study. Journal of Computer Assisted Learning, 33(3), 222–233. Schiefele, U. (1999). Interest and learning from text. Scientific Studies of Reading, 3(3), 257–279. Schiefele, U. (2009). Situational and individual interest. In K. R. Wentzel & A. Wigfield (Eds.), Handbook of motivation at school (pp. 197–223). Mahwah, NJ: Erlbaum. Silvia, P. J. (2005). What is interesting? Exploring the appraisal structure of interest. Emotion, 5(1), 89–102. Stadtler, M., & Bromme, R. (2014). The content-source integration model: A taxonomic description of how readers comprehend conflicting scientific information. In D. N. Rapp & J. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 379–402). Cambridge, MA: MIT Press. Strømsø, H. I., & Bråten, I. (2009). Beliefs about knowledge and knowing and multiple‐text comprehension among upper secondary students. Educational Psychology, 29(4), 425–445. VanSledright, B. A., & Kelly, C. (1998). Reading American history: The influence of multiple sources on six fifth graders. The Elementary School Journal, 239–265. van Strien, J. L., Brand-Gruwel, S., & Boshuizen, H. P. (2014). Dealing with conflicting information from multiple nonlinear texts: Effects of prior attitudes. Computers in Human Behavior, 32, 101–111. Wallace, R. M., Kupperman, J., Krajcik, J., & Soloway, E. (2000). Science on the Web: Students online in a sixthgrade classroom. The Journal of the Learning Sciences, 9(1), 75–104. Wigfield, A., & Cambria, J. (2010). Students’ achievement values, goal orientations, and interest: Definitions, development, and relations to achievement outcomes. Developmental Review, 30(1), 1–35. Wigfield, A., & Guthrie, J. T. (2000). Engagement and motivation in reading. In M. L. Kamil, P. B. Mosentha, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research, 3 (pp. 403–422). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009). Source evaluation, comprehension, and learning in Internet science inquiry tasks. American Educational Research Journal, 46(4), 1060–1106. Wiley, J., & Voss, J. F. (1999). Constructing arguments from multiple sources: Tasks that promote understanding and not just memory for text. Journal of Educational Psychology, 91(2), 301–311. Wolfe, M. B., & Goldman, S. R. (2005). Relations between adolescents’ text processing and reasoning. Cognition and Instruction, 23(4), 467–502.

4

TOWARD A NEW LITERACIES PERSPECTIVE OF SYNTHESIS Multiple Source Meaning Construction Douglas K. Hartman michigan state university, usa

Michelle Schira Hagerman university of ottawa, canada

Donald J. Leu university of connecticut, usa

The purpose of this chapter is to advance theoretical thinking about meaning construction with multiple online information sources. Advancing theoretical perspective in this area is important because the potential for complex, multiple source, meaning construction, identified in the 1990s with offline texts (cf. Hartman, 1995; Rouet, Favart, Britt, & Perfetti, 1997; Wineburg, 1991), has now become intrinsic to our online and media-diverse lives. The nature of “text” is somewhat different in this rapidly developing informational context. Online, the nature of “text” is richer and more complex with multiple information sources being ubiquitous (Vodanovich, Sundaram, & Myers, 2010), immediately addressable (Loh & Kanai, 2016), and essential to understanding in an increasingly complex world (Goldman & Scardamalia, 2013; Goodman, Sands, & Coley, 2015). Moreover, online information sources are commonly multimodal (Kress, 2003) in both form and type, shifting the terrain for various lines of research in this area (Cope & Kalantzis, 2000) and requiring a reconsideration of how we think about multiple source meaning construction. Finally, recent research on adolescents’ and adults’ performance with “fake news” (Stanford History Education Group, 2016) suggests that the use of multiple sources for online meaning construction is more important than ever before since readers must often seek corroboration of online information from multiple sources

55

56  •  Hartman et al.

to determine accuracy. For all of these reasons, it is important to advance theoretical understanding of meaning construction with multiple online information sources. Despite this importance, our understanding of meaning construction with multiple online information sources is limited. Much of the available work investigating the integration of multiple sources during meaning construction has used the reading of a small set of offline documents (cf. Bråten & Strømsø, 2010a; Strømsø, Bråten, & Britt, 2010; Wolfe & Goldman, 2005) or pre-selected online texts in a restricted web environment (cf. Le Bigot & Rouet, 2007; van Strien, Brand-Gruwel, & Boshuizen, 2014; Wiley et al., 2009). These, of course, control the experimental context in important ways but they also limit potential decisions about what to read. As a result, they may not adequately represent the nature of online reading, where unlimited information sources exist and the reader, not the experimenter, selects the set of texts that are read. The Internet introduces additional challenges to coordinate and synthesize vast amounts of information presented in multiple media formats, from a nearly unlimited set of sources, and for a wide array of purposes. Only a few studies (e.g., Barzilai & Zohar, 2012; Hagerman, 2017; Kiili, Hirvonen, & Leu, 2013) have explored the synthesis of multiple online sources within the actual Internet environment where synthesis may be more common and more complex. Given the paucity of research in this increasingly important area, it seems important to advance theoretical work to provide organization for the studies that are needed for both reading and learning in online spaces. While it is clearly important to advance theory in this area, rapid and continuous changes in online information technologies generate an important conundrum for both theory and research: How can we develop adequate theory to design research for multiple source use when the nature of these technologies is ephemeral, continuously being redefined as new ones appear (Leu, Kinzer, Coiro, Castek, & Henry, 2013)? Put another way, at the same time that our research provides new insight for theory, the online world on which that insight is based happens to be quickly and continuously shifting, making the generalizability and application of research to extant theory problematic. We note that research on multiple text integration has, to date, focused largely on academically situated skills, strategies, and purposes (Anmarkrud, Bråten, & Strømsø, 2014; Braasch et  al., 2013; Cho, Woodward, Li, & Barlow, 2017; Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012). Consistent with this orientation, a documents model (Perfetti, Rouet, & Britt, 1999; Rouet, 2006) and, more recently, the MD-TRACE model (Rouet & Britt, 2011) have been used to direct research. Focusing, as these do, on academically situated contexts, both provide an orientation toward multiple text use for the purpose of constructing an argument. We recognize this line of work as foundational for our own work and yet we suggest that we need to consider expanding our conceptualization of multiple source synthesis. Rather than constrain synthesis through the sole lens of academic or argumentative purpose, developed within a generally static context, we borrow from New Literacies theory. We look to multiple perspectives and lines of work to develop an understanding that enriches and complicates the nature of multiple source meaning construction for diverse purposes, with diverse technologies, with diverse texts, in diverse contexts, and through diverse modes. We believe that online contexts present special and unique informational spaces for meaning construction. It is likely, therefore, to be more appropriate to develop theory from this somewhat unique context than simply adapting a perspective from offline contexts.

Toward a New Literacies Perspective  •  57

BORROWING FROM NEW LITERACIES THEORY In our analysis, we borrow three elements from New Literacies theory (Leu et al., 2013; Coiro, Knobel, Lankshear, & Leu, 2008). First, we use the term synthesis rather than integration. Given the genealogy of scholarship on synthesis (e.g., Fling, 1903; Holbrook, 2013; Klein, 1990; Wilson, 1998), we think it is a more dimensional and inclusive term for representing online meaning-making from multiple sources. For instance, synthesis includes a broader range of purposes (from academic, personal, and professional to religious, entertainment, and political). The term signals a more extensive set of contexts (where cognitive, social, and cultural processes are enacted). Synthesis encompasses a fuller complement of text types (e.g., alphabetic, image, artifact, acoustic, tactile, gesture, graphic, kinesthetic, numeric). The term also underscores that the construction of meaning is by more than readers (to include writers, artists, musicians, architects, impresarios, filmmakers, designers, welders, and entrepreneurs too). And synthesis includes technologies across the spectrum (from analog to digital with both low and high thresholds). In contrast, the scholarship on integration of multiple texts focuses largely on a single purpose (academic), context (educational), text type (alphabetic print), reader (of print), and technology (print on paper) (e.g., Litman, Marple, Greenleaf, Charney-Sirott, Bolz, Richardson, Hall, George, & Goldman, 2017). Second, we focus on the construct of synthesis used in work on the new literacies of online research and comprehension (Dwyer, 2016; Leu et al., 2015). This construct is used to explain the meshwork done with multiple online and offline sources during meaning construction. Synthesis is an essential aspect of multiple source information use since it defines how ideas are meshed, meaning is constructed, and learning takes place. Third, to account for the rapid changes taking place in online contexts, we borrow the dual-level approach used in New Literacies theory (Leu, Kinzer, Coiro, Castek, & Henry, 2013; Leu, Everett-Cacopardo, Zawilinski, McVerry, & O’Byrne, 2012), to establish generally common principles that are found in separate technologies, perspectives, and lines of research, each of which is rapidly changing. In brief, this dual-level approach defines, in an upper-case theory of New Literacies, patterns that are common to different, lower-case lines of research appearing in rapidly changing areas of literacy and technology. By looking for common principles that are found in separate areas of work, we seek to uncover a more stable theoretical understanding of common and generalizable patterns. Such an approach is necessary in contexts that change rapidly. In addition, this theoretical framing has emerged directly from online contexts. Work in new literacies is somewhat siloed, with cross-disciplinary work only infrequently taking place and with far too little synthesis of findings from separate lines of research. This chapter will review work across multiple disciplines and lines of research to look for a set of generally common principles. We suggest that these principles, common to much of the separate work taking place, are likely to be more stable and common across the many, continuously changing contexts that define this area. Having greater stability in an unstable context will be useful as we develop theory to help us study and understand how we read and learn with online information. We organize the heart of this review around five elements that appear to be important to the synthesis of multiple sources of information: purpose, context, texts, readers, and technologies.1 Writing about each of these elements as though they could

58  •  Hartman et al.

be extricated from the complex systems to which they contribute is purely academic—a rhetorical exercise meant to highlight the layers of complexity that we can conceptualize and attribute (mostly) to the reader, the text, the context, the purpose, the technology. This expanded conceptualization of synthesis forms a new literacies perspective which upends any sense of linearity or compartmentalization that might still exist about literacy’s nature. Because conceptions of literacy have been constrained by the medium of print itself, we’ve taken this approach in order to momentarily constrain the complexities and think deeply about categories of elements that interactively contribute to the building of a more unified understanding of multiple source use. As Hartman, Morsink, and Zheng (2010) write, “Online, it quickly becomes impossible to ignore that each individual element is already by itself plural” (p. 140).

PURPOSE The concept of purpose and its relationship to the construction of meaning have been considered by scholars for nearly a century. Gates (1930) conducted studies in the 1920s about the interaction of purpose and reading. Gray (1940), who summarized research from several previous decades, saw evidence that “the degree of comprehension is influenced materially by the purpose of reading” (p. 498). Blommers and Lindquist (1944) controlled the effects of reading purpose when studying the relationships between reading rate and reading comprehension. Anderson and Pichert (1978) found that the purpose for making sense of a text influenced recall of content from the text. And Narvaez, van den Broek, and Ruiz (1999) found that readers constructed different meanings of a text depending on their purpose. To a large extent, these views of purpose are an artifact of the material conditions which made possible a particular view of purpose. For example, the offline, print-paper conditions of reading during the 20th century formed three assumptions about purpose: •• Purpose is stable. The reason for meaning-making remains constant throughout the making of meaning. •• Purpose is singular. The reason for meaning-making is constituted by one thing and therefore can only be one thing. •• Purpose can be assigned. The reason for meaning-making can be transferred or imposed “as is” from one person or task to another. The material conditions of the last century also shaped the methodological means by which reading research and activity were imagined and conducted. For instance, offline, print-paper purposes for reading: •• •• •• ••

focused on the construction of meaning of a single text focused on the construction of meaning of a print text focused on the construction of meaning of an offline text located intent in one—and only one—“place” during meaning-making, whether the text, reader, author, task, or teacher.

Taken together, the assumptions and methodological means of scholarship during the last century shaped the conception of purpose associated with reading. But as the

Toward a New Literacies Perspective  •  59

material basis for reading has shifted from print-paper to pixel-screen and from a single text to multiple texts, the construct of purpose has begun to evolve, largely as a result of shifts in the technological substrata of digital communication. For instance, Gil, Bråten, Vidal-Abarca, and Strømsø (2010) and Bråten and Strømsø (2010b) prompted undergraduate students to read a defined set of multiple print texts for different purposes. They found that purpose shaped the understandings that readers constructed from multiple texts in powerful ways. Zhang and Duke (2008) prompted adults to read a defined set of multiple online texts for different purposes. They found that the strategies and purposes for reading were very fluid across time and space, contingent on a host of factors. And DeSchryver (2014) prompted graduate students to read multiple multimodal texts on the open web for different purposes. He found that purpose continuously shape-shifted as reading occurred across hours of scouring the content of multimodal texts to construct an understanding of an ill-structured content domain. Because the types of tasks are evolving in the school context, the purposes for multisource reading for synthesis online are evolving too. This shift can be easily seen in a scenario of how the purpose of multiple source meaning-making is constructed in a middle school classroom where teens of today have access to resources online when engaging in project or inquiry work. For example, Mr. Zammit’s grade 6 classroom in Sydney, Australia is preparing to read a print text on the significance of Australia Day for their country. The teacher’s purpose is for pairs of students to read the text so they understand the origins of the holiday more deeply. As students grapple with the text in light of the purpose that’s been assigned to them, the pairs start to negotiate the reading purpose. The negotiation inevitably leads some pairs to renegotiate the task to fit their own understanding or interests. And within a few minutes some pairs have broadened the purpose of their reading to that of understanding the formation of national days across many countries. They start reading across multiple texts, in multiple languages, of multiple modes. In one pair, a partner IMs a friend in France to get information about their national day. The other partner emails his father’s friend in Malaysia to learn about theirs. These inquiries spark their efforts to view videos and listen to podcasts that describe and celebrate their own national holiday in more detail than the article given by their teacher and leads them to learn about the national holidays of a dozen other countries they find interesting. Several other pairs of readers renegotiate their purpose in a similar direction. When students reach a bottleneck moment in their search, they refocus on a promising sub-purpose they think will loosen the bottleneck and move them back to their more primary purpose.

TEXT The idea of text and its relationship to the construction of meaning has been considered by scholars for several centuries. The earliest known consideration was by the Roman rhetorician Quintilian (95 ce), who described text as an artifact that “weaves [words] together” (in textu iungantur) (Institutio Oratorico, 95 ce, 9.4.13 3) until they have a “fine and delicate texture” (text umtenue atque rasum) (Institutio Oratorico, 95 ce, 9.4.17). His metaphor of text-as-woven-material-from-other-sources has endured for two centuries—we still think of texts today as braided with a thematic or topical thread, embroidered with important or seductive details, a place where one spins a yarn, and knitted with “a particular texture, pile, and grain” (Hartman, 1995, p. 17).

60  •  Hartman et al.

Quintilian’s use of the Latin textus (n.), “which means woven, as in a fabric—further suggests that the composition of any text is interwoven with previous resources” (Hartman, 1995, p. 296). In time, Classical and Modernist scholars described the text as a work that stands alone, un-entwined with urtexts, contexts, objects, or the pretexts of intention or culture (Barthes, 1977, 1986). Similarly, educational scholars assumed the text to be a stable, singular, and discrete object (Hartman, 1995, 2004). Postmodern scholars re-appropriated the metaphors of Quintilian and others by envisioning the text in two ways. One was to see text more broadly as any semiotic or multimodal artifact that can be “read,” whether it is a poem, icon, building, logo, gesture, design, photo, video, music, or garment (Kress, 2003; Lotman, 1977). The other was to see text as a site of synthesis, as an “intertext” which echoes the content, structures, ideas, and motifs of preceding texts; and in turn will echo its content, structure, ideas, and motifs into future texts (Derrida, 1967/76; Plottel, 1978). With the advent of digital technologies, scholars expanded their conceptions of text yet again because of the affordances made possible by the medium (Bolter, 1990, 2001; Landow, 2006; Reinking, 1998). Texts became “hypertexts,” which were materially connected to other texts via digital links. Taken together, these links among texts— located within, between, and beyond any given text—wove a layered digital tapestry that is analogous in concept to the Latin “textus” of Quintilian, but different materially. The study of text as the site of synthetic work appeared prominently in the 1980s and 1990s through a number of print-on-paper-based studies that examined “writing from sources.” These studies traced the origins of ideas, clauses, or voices that appeared in a composed text back to the source texts used by the writers (e.g., McGinley, 1992; Kamberelis & Scott, 1992; Rouet, Britt, Mason, & Perfetti, 1996; Spivey & King, 1989). Related work can be found in chapters by Bråten and Braasch and Britt, Rouet, and Durik of this Handbook. More than a decade later, the study of digital texts as sites of synthetic work followed a similar path, tracing the digital sources used by writers as they composed an essay or other product. For example, Hagerman (2014, 2017) prompted adolescent dyads to read online texts together and then write an argument essay by themselves, after which she traced in detail where information in the essay came from using screen and link capture software. Similarly, Kiili, Hervonen, and Leu (2013) studied adolescent dyads who synthesized information from multiple online sources to write an essay using a graphic representational tool to take/use notes, which was then analyzed by tracing out the web-based sources for information appearing in the essay. The ubiquity of bricolage-like texts in everyday life goes unnoticed by most. The evolution from autonomous to networked texts—and from uni-modal (e.g., print only) to multimodal texts—has been epochal. To juxtapose a news story from today’s online New York Times with the 1618 Dutch broadsheet Courante uyt Italien, Duytslandt, &c. (Dahl, 1946) to a 202 bce Chinese tipao (or dipao, a handwritten news sheet) from the Han Dynasty (Fang, 1997) is to recognize the profound conceptual and material shifts in what constitutes a text and how a text is constituted. It is commonplace for the composition of a story in today’s online New York Times to be a co-created, multimodal text that synthesizes media of many types into the text—print that includes quotes taken from other texts, a video clip from a network (that is often a fusion of video snippets from several sources), an audio clip from a phone interview, tweets posted on

Toward a New Literacies Perspective  •  61

the web, graphics from other reports, a photo gallery of images from the event, links to related materials, and correction statements about an earlier version of the Times’ text. And the Times is one of thousands of news sources and other everyday texts that are a pastiche and meshwork of digital texts that echo other texts unto near infinity.

READERS The reader and her relationship to the construction of meaning have been considered by scholars in more ways than is possible to review here (see the four Handbooks of Reading Research and six editions of Theoretical Models and Processes of Reading for starters). These complexities are important to the research that will be needed in this area, especially in relation to individual differences. Currently, we know little about how individual differences are related to online research and comprehension (Leu, Kiili, & Forzani, 2015). It appears that some relationships found during offline reading, such as the importance of prior knowledge (Kintsch, 1998, 2013; Spilich, Vesonder, Chiesi, & Voss, 1979), may have far less impact online (Leu et al., 2014; Leu, Kiili, & Forzani, 2015), perhaps because online readers have rapid access to information online that fills in missing prior knowledge. Others, such as the achievement gap associated with income inequality, appear substantially greater during online reading (Collin, Karsenti, Ndimubandi, & Saffari, 2016; Hargittai & Hsieh, 2013; Leu et al., 2014). Still others, such as gender and motivation for online reading, have yet to be studied extensively. In any case, much more work is required before we have a complete understanding of the role of individual differences during multiple source use in online contexts. For the purpose of this review, we selected the lower-case conception of the new literacies of online research and comprehension to describe the processes readers employ online as they identify a problem, locate information sources, evaluate information critically, synthesize the information, and communicate their understandings (Leu et al., 2013). It is in the context of these five strategic processes that the interminable complexities of synthesizing occur in the complex contexts that appear online. Although synthesis is named as one of the five lower-case processes, we understand it to depend on all of the other processes. Reading Strategies for Synthesis Foundational research on the strategies that online readers use to construct meaning from multiple Internet texts has shown that, like readers of printed texts, online readers flexibly engage diverse strategies to construct meaning (Afflerbach & Cho, 2009; Barzilai & Zohar, 2012; Cho, Woodward, Li, & Barlow, 2017; Coiro & Dobler, 2007; Pressley & Afflerbach, 1995). Although readers engage many of the same strategies offline, with printed texts, and online, with digital multimodal texts, (e.g., previewing, inferencing, monitoring understanding), evidence suggests that the nature and patterns of strategic activity for reading printed texts and Internet texts differ (Coiro, 2011; Leu et  al., 2013), and that differences may emerge or develop along different trajectories (Coiro, 2011). So, although Internet readers must be able to decode and construct a situation model that represents their understanding of one or more texts

62  •  Hartman et al.

(Kintsch, 1998; Perfetti, Rouet, & Britt, 1999), the strategies that readers use to construct an integrated understanding of multiple printed texts are insufficient for the online context due, in part, to the fact that online, readers must find and construct their own sets of texts to read closely after searching for and evaluating a potentially infinite set of texts for relevance and trustworthiness. As Coiro (2015) writes, the complex nature of reading environments appears to require new metacognitive regulatory strategies to flexibly and repeatedly move between rapid readingto-locate processes (that occur, for example, when skimming search engine results and navigating through levels of websites) and deeper processes of meaning construction. (p. 56) Specifically, synthesis seems to depend on an integrated set of strategies for information seeking, gathering, and evaluation. Proficient online readers and synthesizers consider their reading purpose and generate search terms that will allow them to find relevant, credible information. They use search engines iteratively, through repeated cycles, to realize and construct a set of potential texts. Good online readers make anticipatory predictions about the relevance and credibility of texts using clues from site URLs, from the snippet text on the Search Engine Results Page (SERP), and from knowledge of Internet text structures (e.g., wiki, blog, BuzzFeed list?) (Coiro & Dobler, 2007; DeSchryver, 2014). All the while, they engage a complex set of thinking routines to contextualize information and assess source credibility (Wineburg & Reisman, 2015; Wineburg & McGrew, 2016). Good online readers use “multilayered inferences across the three-dimensional space of Internet hypertext to anticipate meaning of texts that are hidden from view, or to be encountered” (Afflerbach & Cho, 2009, p. 83). And, when they have chosen an information source to read closely, better online learners tend to re-evaluate the reliability and credibility of the text, continue to maintain a focus on their task, and monitor their emergent understanding of the topic (Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012). Better online readers also combine disparate forms of information to construct meaning, including text, graphics, illustrations, and embedded video (Afflerbach & Cho, 2009, p. 84). Importantly, better online reading skills may compensate for low background knowledge of topics, which means that even if they know very little about a topic, better online readers may be better equipped to construct a unified understanding from multiple texts because they can more easily locate, critically evaluate, synthesize, and communicate information (Coiro, 2011; Lawless & Schrader, 2008). Evidence also suggests that proficient online readers can generate creative syntheses from texts, drawing conclusions and developing theories that go beyond ideas present in any of the texts they read. Central to this process is notetaking, which suggests that writing is integral to the construction of a unified understanding too (DeSchryver, 2014). The importance of notetaking for the construction of meaning within and across multiple printed texts has been well established. Kobayashi (2009) found that students who noted similarities or differences in author stance, or who drew lines to indicate relationships as they took notes while reading, scored higher on a test of intertextual relations. Similarly, Hagen, Braasch, and Bråten (2014) found that when students’ notes included combinations

Toward a New Literacies Perspective  •  63

of information from two or more texts and instances of pre-existing knowledge, their performance on tests of integrated understanding within and across texts was better. Online, DeSchryver (2015) suggests that notetaking applications such as Diigo that allow readers to tag, comment on, and mark up texts may be especially helpful for synthesis of meaning within and among digital texts. Strategy instruction may also support online synthesis. One study found that ninthgrade students who learned a set of integrated new literacies strategies and practiced them during three dyadic online inquiry sessions were able to synthesize information into written arguments from a broader set of information sources at posttest than students who did not learn and practice those skills (Hagerman, 2017). Several studies have found that precursor skills considered essential for effective synthesis such as critical evaluation (Barzilai & Zohar, 2012) can be taught to elementary and secondary-school students (Braasch, Bråten, Strømsø, Anmarkrud, & Ferguson, 2013; Macedo-Rouet, Braasch, Britt, & Rouet, 2013; Zhang & Duke, 2011). And yet, the new literacies strategies used by better online readers to synthesize meaning and construct new knowledge from multiple texts online are not being taught on the scale needed to ensure all citizens are fully literate (Coiro, 2015; Dwyer, 2016; Thurlings, Evers, & Vermeulen, 2014; Wineburg & Reisman, 2015; Wineburg & McGrew, 2016). The integrated and iterative nature of information gathering, evaluating, and synthesizing across the multiple complex contexts of the Internet means that teachers must offer students a range of complex, web-based problem-solving experiences, over time, and for diverse purposes so that all readers can acquire and learn to flexibly apply this vast repertoire of strategies for synthesis. Reader Stances as Guiding Frameworks for Synthesis In addition to strategy use, readers’ pre-existing understandings of epistemology and their perceived power in relation to texts also shape meaning construction from multiple texts (Barzilai & Zohar, 2012; Hartman, 1995; Jiménez & Meyer, 2016). In one study, sixth-grade “absolutist” students who tended to see knowledge as facts that are certain, as correct or incorrect, and located in the external world (Kuhn, Cheney, & Weinstock, 2000) used integrative strategies less frequently on open online inquiry tasks than “evaluativist” students who tended to see knowledge as judgments that are constructed by an individual, and based on criteria of argument and evidence (Barzilai & Zohar, 2012). With a set of printed texts, Hartman (1995) also found that readers’ activities could be characterized along a continuum of perceived power, ranging from a stance that was deferent to authorial authority to a stance that was resistant, or critical, of the author’s assumptions and privilege. Whether online or offline, a critical reader stance seems predictive of synthesis, perhaps because this positionality activates thinking routines, or frameworks for deconstructing and reconstructing meaning. More research is needed to test this claim. Knowledge of epistemology or the adoption of more evaluativist stances may also help readers to manage the metacognitive demands of the online context. Synthesis of meaning from multiple, multimodal texts is a complex metacognitive activity requiring both knowledge of strategies and the cueing of systems of executive control (Winne & Azevedo, 2014; Kiili, Laurinen, & Marttunen, 2009; Putman, Hathaway, Coiro, &

64  •  Hartman et al.

Quinn, 2015; Quintana, Zhang, & Krajcik, 2005). Online, the extent to which readers are able to use diverse strategies depends, in part, on their ability to manage demands that Hartman, Morsink, and Zheng (2010) have called n-adic, infinitely dynamic and plural (p. 150). New literacies researchers might think of reader stances as metacognitive frameworks that enable readers to rein in or control the complexities of the synthesis activity. Rather than characterize readers as novice or expert as framed by the discrete skills they can or cannot yet apply, future research could position online reading activity along trajectories of reader stances that may emerge, as many other complex skill sets develop, in waves of overlapping skill development (Barzilai & Zohar, 2012; Hartman & Morsink, 2015; Kwong & Varnhagen, 2005; Opfer & Siegler, 2007; Siegler, 1996; Siegler & Chen, 2008; Sharp, Sinatra, & Reynolds, 2008). As Barzilai and Zohar (2012) write, “Student views of knowledge as constructed, complex, and developing appear to be central to the ability to integrate multiple documents, offline and online, in meaningful ways” (p. 73). Critical, evaluativist stances seem especially supportive of synthesis, but it is not yet clear whether, or to what extent, the emergence or the application of these stances depends on text, purpose, context, background knowledge, or even whether teaching students how and when to adopt critical stances might allow for selection of more productive synthesis strategies in the moment. Collaborative, Dyadic Readers Several studies have found that as children and adolescents learn to read online, collaboration can support learning outcomes (Kiili et al., 2013; Leu et al., 2013, p. 1168). In dyads, evidence suggests that readers employ integrative strategies more frequently than when they read alone (Coiro, Castek, & Guzniczak, 2011). Also, dyads who spend more time gathering information to develop and evaluate arguments produce more integrative essays (Kiili, Laurinen, & Marttunen, 2009). Indeed, positioning online reading and research for synthesis as a social, collaborative process may be especially supportive of the stances, or patterns of strategy engagement that enable synthesis. Much more research is needed to explore the complexities of such a claim. In this light, binary categorizations of readers as novice/ expert or more/less sophisticated seem insufficient. Instead, we need broader conceptualizations of diverse synthesis trajectories that more fully reflect the many paths readers take over time, for a range of purposes, in diverse contexts and with diverse Internet technologies.

TECHNOLOGIES Conceptualizing Technology From the Greek words techne, meaning technical action, and logos, meaning consciousness, technologies are usually conceptualized as physical objects, knowledges, and activities (Mitcham, 1994). A full review of the philosophy and historical conceptions of technology is beyond the scope of this chapter (cf. Aunger, 2009; Mitcham, 1994) but it bears mentioning that given our theoretical focus on New Literacies we have necessarily constrained our definition of technology to digital Internet technologies. Paper and pencil are technologies (objects) that have enabled reading and writing

Toward a New Literacies Perspective  •  65

(knowledges and activities) to evolve, but through a New Literacies lens, we focus here on the ways that the products and processes of synthesis are shaped by the physical, digital, and social technologies of the Internet. We assert that the physical technologies of the Internet are the objects that give access to the network (e.g., laptop computers, tablet computers, smart phones). The digital technologies are the websites, or applications that create spaces for literacies activities on the Internet (e.g., web browsers, search engines, social networking sites, Google Apps, RSS feeds). These digital technologies are necessarily constrained by the physical technologies on which they are coded and are integral to the social and cultural practices (activities and knowledges) of the Internet. As a complex process, we assume that synthesis of multiple Internet texts depends on the social technologies, or systems of “sociotechnical use” (Cole & Derry, 2005; Kline, 1985), that these physical and digital technologies enable. Indeed, any construction of meaning is theorized to be inextricable from the technologies on which and through which the information sources, themselves, are created and understood (cf. McLuhan, 1964). Social Technologies and Synthesis Social networking sites such as Facebook, Twitter, LinkedIn, and YouTube (Boyd & Ellison, 2007; Greenhow & Gleason, 2012) permit people and communities to create, share, and participate in new literacies social practices that shape knowledge construction (Lankshear & Knobel, 2011; Leu et al., 2013, p. 1158; New London Group, 1996; Street, 2005). They are also technologies for synthesis that have been undertheorized. Gleason’s (2013) study of the #OccupyWallStreet movement, for example, showed how Twitter users shared information to mobilize democratic protest. He noted that the real-time contingencies of this technology may make Twitter, as a platform for knowledge construction, especially well suited to critical evaluation and to synthesis of understanding across multiple, divergent perspectives as an issue evolves. He writes, “In a learning space where source credibility is unstable, an ideal strategy may be to reserve judgment until a broader survey of the content is feasible” (p. 979). Because Twitter aggregates information in real time, and because information sources are often difficult to validate (many Twitter accounts, for instance, are fake, or are bots, programmed to inflate affirmations of particular ideas by particular people), processes of synthesis on Twitter seem especially dependent on a critical stance and on careful regulation of text choice. These judgments will fundamentally determine what readers believe, retweet, or contribute as unique or amplified additions to any ongoing discussion. The temporal and extraordinarily contingent nature of information on this digital technology expands our conceptions of synthesis processes, particularly because on Twitter, readers can be writers and participants who simultaneously and strategically (re)shape what is understood by others. On this point, Gleason (2013) found that Twitter facilitates multiple social practices central to democratic action that allow communities to construct understandings as issues evolve over time. As people construct understandings, they can create and participate in the ongoing evolution of the story itself. Citizen journalists can tweet commentary, videos, and images, and use hashtags to engage a particular audience. During natural disasters, for instance, Twitter is often used to curate and disseminate situational updates about evacuations, the location of the hazard, and to report

66  •  Hartman et al.

damage (Thomson et al., 2012; Vieweg, Hughes, Starbird, & Palen, 2010, p. 1085), permitting rapid, iterative synthesis of information to inform action, including the dissemination of information to networks as a situation evolves in real time. Time and again, this technology in particular has become central to the expression and practice of social values and culture over time. Hashtags such as #blacklivesmatter, #neverthelessshepersisted, and #edchat have become powerful curators for ongoing discussions of diverse ideas, insights, and in-the-moment happenings that resonate and mobilize action. Multimodal Remix and Social Curation Log in to your Facebook account and in a single session, you make meaning from texts that include multiple modes, including, but not limited to: images, emoticons, video, words, hyperlinks, color, spatial orientation, and proximity. These texts will have diverse rhetorical purposes: to persuade, to inform, to describe, to mislead, to evoke emotion, to build community, to rally support, to sell. Images of puppies and children’s birthday parties are interwoven with restaurant reviews, political commentary, pleas to support social causes, links to news items, cartoons, advertisements, and live video streams of events. In a single session, and also over time, the curated collection of texts we serendipitously encounter through this technology give access to an unprecedented set of ideas that connect us to a community, and to their thoughts—at once individual and collective. And our choice to engage with, respond to, communicate, or remix understandings of any of these texts can take on many forms. We might respond to a text by simply clicking the thumbs-up (like) button, or the angry face; we might comment in words, with an image, by creating or sharing a meme or an animated .gif that expresses a shared perspective or a contrasting one; we might repost a text with added commentary to our own friends; we might bear witness to civil protest via livestream (e.g., Women’s Marches on Washington and around the world in January 2017) comment, and then take and post a selfie from the event. Together all of these texts, and our in-the-moment, micro-synthetic responses to them, become the informational threads of our lives, woven together in a tapestry of thought and understanding that is inextricably multimodal and participatory. As this Facebook scenario suggests, however, the fluid and contingent processes and products of synthesis enabled through social media technologies are about creation, remix, and participation. Analyses of LGBT youth and their constructed lifestreams across diverse social media spaces (Wargo, 2015) show how youth curate identities with every selfie and hyperlink. In school, understandings of character motivations in a novel can emerge through the construction of a fictitious Facebook profile (e.g., for Atticus Finch) that mixes conversation with other “characters,” images, and video over weeks or months (White & Hungerford-Kresser, 2014). Social curation websites such as Pinterest (pinterest.com) and tesTeach with Blendspace (www.tes.com/ lessons) allow users to gather, organize, display, and share diverse, multimodal Internet texts on boards or galleries with a community of followers (Hall & Zarro, 2012; Pearce & Learmonth, 2013). Pearce (2012) calls this fluid, iterative, social curation of multimodal texts “clickolage” which may be a new form of synthesis processing, possible because of these technologies.

Toward a New Literacies Perspective  •  67

To date, products of synthesis have been theorized as integrated conceptualizations, presented in an argumentative essay or in an email (Leu et al., 2013). On social networking sites, synthesis processes and products are dynamic, contingent, latent, multimodal, and emergent.

CONTEXT Having just reviewed the ways that the physical, digital, and social technologies of the Internet have expanded our conceptions of synthesis processes and products, we turn to a consideration of systems-level factors that undergird the design of the Internet itself. We use the familiar lower-case/upper-case structure outlined in the Dual-level Theory of New Literacies (Leu et al., 2013) to make the point, rhetorically, that the physical, digital, and social technologies that readers use to synthesize information across multiple sources (i.e., search engines and social networking sites) are themselves the products of broader social, political, and mathematical applications. We assert that the small-c technological contexts for synthesis (i.e., the physical, digital, and social), and the small-s synthesis processes they permit, cannot be fully understood without a broader discussion of the Big-C Context of Internet algorithms designed to serve corporate, political, and ideological interests. For us, this is central to an expanded conception of synthesis from a New Literacies perspective. Big-C Context of Synthesis on the Internet Many Internet technologies, whether they are web browsers, social curation sites, social media sites, or RSS feeds, afford both the exploration of diverse, multimodal information sources and the construction of a diverse network of perspectives. Actively seeking out diverse views through these technologies could support the construction of integrated understandings from multiple information sources that, in some idealized fashion, could include a balance of viewpoints corroborated across multiple reliable sources of information. Analyses by O’Neil (2016) and Pariser (2012) have shown, however, that this vision, though possible, is far removed from what actually occurs. Invisible to most users are algorithms that use hundreds of data points to predict what they want to see, and filter information feeds accordingly. These algorithms are designed around the idea of relevance, as determined by user profiles that resemble one another, and by every piece of data that users feed into the systems themselves— where they go, which posts they like, which items they retweet, what they voluntarily share in their own posts about what they have done or purchased. And obviously, this notion of relevance means that advertisers can target users who are most likely to purchase their products or services. As Internet readers, writers, and participants use multiple strategies and thinking routines to synthesize meanings for diverse purposes while also managing the metacognitive demands of the activity, they do so in the big-C Context of the Internet that stealthily decides which texts the reader will see, and in which order (O’Neil, 2016; Pariser, 2012). Google Search, for instance, uses “over 200 variables” to inform what a reader sees on the Search Engine Results Page after any information query (Google, 2017), which means that the terroir from which any synthesized construction of meaning grows will be unique to

68  •  Hartman et al.

every user, not just because they must find and select their own set of texts, or because the texts can change in the moment, but because the texts to which each user has access via these technologies will never be the same. And yet, as search and social media technologies build models of individual user behaviors and the ways that these align or diverge from the behaviors of users who share certain key predictive tendencies, we can quickly become trapped in filter bubbles (Pariser, 2012). At once, this context for synthesis becomes personalized and homogenized because of the larger Contextual interests at play. Algorithmic technologies that personalize information serving create the impression of a vast information landscape when, in truth, every Internet reader may be gathering information on a very small information island that is used by, built by, and maintained by people who are just like them (Rainie & Anderson, 2017). When access to ideas, to information, to facts are filtered, synthesized conclusions will reflect those filters too. A necessary part of the critical positioning required for synthesis may be a recognition of how this big-C Context is deliberately designed to predict, to restrict, and to selectively disseminate ideas in ways that further economic, political, or ideological interests. Importantly, we need New Literacies synthesis research that not only takes this big-C context into consideration for its impact on readers, reading, and understandings, but that also weaves critical conceptions of synthesis processes with the study of critical, social justice-oriented digital citizenship (Kahne, Hodgin, & Eidman-Aadahl, 2016; Westheimer, 2015).

UNDERLYING PRINCIPLES FOR SYNTHESIS AS MULTIPLE SOURCE MEANING CONSTRUCTION This review of five areas, central to a consideration of synthesis in shifting online contexts, suggests that meaning construction with multiple online information sources is not simply the integration of several propositions fused into a single idea. When synthesis of multiple, multimodal Internet texts is the goal, readers are also writers, creators, and participants in social spaces with different purposes at different times. Inputs come from multiple multimodal texts and the product, or outputs, include not just the integration of ideas presented with a single textual mode, but rather the (re)mixing of textual, visual, and graphic modes so that the meaning that emerges is something new; something greater than the sum of the parts. In short, synthesis is a complex remixing of not just meaning but also roles, purpose, and technologies, often in a highly social context. Synthesis is also design. As Hull and Nelson (2005) write, multimodal composing . . . is not simply an additive art whereby images, words, and music, by virtue of being juxtaposed, increase the meaning-making potential of a text. Rather . . . through a process of braiding or orchestration, a multimodal text can create a different system of signification, one that transcends the collective contribution of its constituent parts. Multimodality can afford not just a new way to make meaning, but a new meaning. (p. 225) The complexities involved in meaning construction with multiple online information sources are profound, especially in a shifting landscape such as exists online, and they present a number of obstacles to the study of this important area. Nevertheless,

Toward a New Literacies Perspective  •  69

there appear to be several underlying principles that are likely to be somewhat stable and may be used as an initial foundation on which to develop theory. These include: 1. The Internet is this generation’s defining technology for literacy and learning. 2. Simple approaches to online synthesis may lead to inadequate representations within a theoretical space as well as faulty and inaccurate learning outcomes for individuals within a practice space. 3. Synthesis is often shaped by the purpose, context, texts, readers, and technologies as well as their interactions, and these may change, sometimes even during the process itself. 4. Readers must engage in more complex and more profound ways during online synthesis to fully benefit from the information available. 5. Given the complexity of online synthesis, and instructional tendencies to simplify, educators need to think hard about how to more adequately represent these complexities in productive ways within teaching and learning contexts. The Internet is this Generation’s Defining Technology for Literacy and Learning Much of the work reviewed here is based on perspectives suggesting that literacy is embedded in and develops out of the social practices of a culture. As a result, it should come as no surprise, then, that a central principle for much of this work is that the Internet has become this generation’s defining technology for literacy and learning. Almost 40% of the world’s population currently has access to the Internet (Broadband Commission for Digital Development, 2014). At the current rate of adoption, half of the world’s population will have access to the Internet in 2017 (Broadband Commission for Digital Development, 2014), and everyone will have access within 10 years. By the time today’s 8-year-olds graduate from high school, nearly everyone in the world will be connected online. The Internet and the online information it makes available is a cornerstone technology for learning and will be especially important to the next generation. While the increasingly ubiquitous nature of Internet access is profoundly changing the literacy practices of our emerging global culture, we must also recognize the continuing use of offline information. Online information will be with us for some time. Thus, any theory of multiple source meaning construction must take into account the separate and combined integration of both offline and online information. Simple Approaches to Online Synthesis May Lead to Inadequate Representations Within a Theoretical Space as well as Faulty and Inaccurate Learning Outcomes for Individuals Within a Practice Space Since the 14th century and the development of what has come to be known as Occam‘s razor, scientific approaches have always favored simple, elegant solutions, when all things are equal. This is especially true in theory development. Simpler theories that account for all known possibilities have fewer elements that require testing for verification. While this strategy must inform theory development with multiple source integration, we have seen the exceptional complexity that exists in this terrain.

70  •  Hartman et al.

Moreover, it must always be kept in mind that online contexts are in a continuous state of change so the nature of this context includes a certain degree of unknowable complexity. All of this suggests that adequate theories of synthesis and multiple source meaning construction must include these complexities. This also points us to the enormous challenges that we face in developing theory. It will not be easy. There is a parallel challenge for practice. We believe the complexity of this context also suggests that simple approaches to teaching online synthesis may lead to faulty and inaccurate learning outcomes. The failure to develop approaches that equip readers for complexity online can be exceptionally dangerous as we have recently seen in politics, where simple understandings are sometimes used when a richer and more nuanced understanding is required. We must begin to conceptualize instruction in synthesis in ways that are much more complex than the simplistic ways this is conceptualized now in the classroom. Concepts such as close reading, inferential reasoning, summarizing, or synthesizing information from multiple sources (cf. Common Core State Standards Initiative, 2015) must give way to more complex constructs as guides to instruction. Synthesis is Often Shaped by the Purpose, Context, Texts, Readers, and Technologies as well as Their Interactions, and These May Change, Sometimes Even During the Process Itself We have seen one of the explanations for why synthesis of multiple online sources is so challenging to appreciate; it includes multiple dimensions such as purpose, context, texts, readers, and technologies. In addition, it takes place within a space, online information, that continuously changes, adding an additional dimension to this complexity and one that is impossible to fully anticipate. This brief review has shown how multiple aspects contribute important richness and complexity to the notion of multiple source synthesis. These elements are not intended to be exclusive; there are likely to be additional elements that will need to be considered in any theoretical framing of meaning construction with multiple online information sources. Nevertheless, they do suggest the complexity of the task in an online space that is continually changing. Readers Must Engage in More Complex and More Profound Ways During Online Synthesis to Fully Benefit From the Information Available We have suggested that simple approaches to synthesis lead to troublesome outcomes. The reciprocal is that readers must engage with information in more complex and more profound ways to fully benefit from the information that is available online. In addition to considering purpose, context, text, reader, and technology, individuals will also need to evaluate carefully the reliability of the multiple online sources that they select, often by corroborating initial information with other information, holding ideas in contingent and emergent juxtaposition over time, and synthesizing the multiple sources in thoughtful, critical ways. We know from initial studies that students are exceptionally unprepared for the more complex synthesis and evaluation required in online spaces. Forzani (2016) showed that only 4% of seventh-grade students in two states (Connecticut and Maine) were able to fully evaluate the reliability of an online site about energy drinks.

Toward a New Literacies Perspective  •  71

A 2016 report from Stanford (Stanford History Education Group, 2016) showed that most students did not know when news was fake, and that 82% of middle school students could not tell the difference in an ad labeled with the words “sponsored content” and a legitimate news story. A study (Kiili et al., 2017) now being prepared in Finland by the eSeek Project at the University of Jyvaskyala with 12- and 13-year-olds shows that less than 20% of students could recognize commercial bias at a commercial website on energy drinks and few corroborated information with information at other sites. It appears that students are largely unprepared for the multiple source synthesis of online information when it comes to evaluating online information. Given the Complexities of Synthesis in Online Spaces, and Instructional Tendencies to Simplify, Educators Need to Think Hard About How to More Adequately Represent These Complexities in Productive Ways Within Learning Contexts Evidence suggests that teachers have a tendency to simplify the instruction of complex tasks (Rix, 2009). While breaking tasks down and simplifying the learning may be helpful with procedural or algorithmic tasks, our analysis of multiple source synthesis online suggests that it may not be either possible or optimal to do so. The complexity and the changing nature of purposes, contexts, and other aspects of online synthesis from multiple sources may make it impossible to teach in a manner that breaks complex tasks down into constituent elements and teach each one in an incremental layering. It may not be optimal since it would only prepare students for the simplest of learning in this area. This suggests that we need to think hard about how to more adequately scaffold developing proficiency with these complexities in productive ways within learning contexts. The problem may suggest that more of a social practice approach is needed (Chaiklin, Hedegaard, & Jensen, 1999), one that acknowledges the complexities and creates contexts where those complexities are encountered and incorporated into the learning process in authentic problem-solving tasks as part of a culture’s social practices. It may also require some meta-awareness of the challenges as well as the monitoring of one’s use of the many different elements that appear to be required. In practical terms, what might teachers do with students to develop ways of synthesizing that retain the ecological complexities of multiple text tasks? •• Simplify the role of the learner, rather than the task itself, by having the teacher use scaffolds that selectively assist the learner as needed. •• Pluralize the practice of scaffolding, so that purposes, contexts, texts, readers, and technologies all serve the purpose of assisting learning. •• Mark critical features within complex tasks by calling attention to important aspects of purposes, contexts, texts, readers, and technologies. •• Approach multiple source teaching and learning as a cooperative task, where the teacher takes greater responsibility than the learner (early on) for synthesizing multiple sources by using what the learner can do on their own, but “filling in” what the learner can learn to do with assistance.

72  •  Hartman et al.

THE JOURNEY FORWARD The future of synthesis at the present time is one where extraordinary complexities exist, yet greater learning opportunities are possible because we have access to an amazing array of information sources online. At this moment of complexity and opportunity, it seems important to advance our theoretical understanding. This chapter has sought to advance our understanding of meaning construction with multiple online information sources. It has explored a number of different areas of work through a New Literacies lens. In general, it found that synthesis from a New Literacies perspective is much more complex than the simple integration of several propositions into a single idea. If we are wise, we will use this understanding, and others, to prepare students for the additional challenges that meaning construction online requires, bringing new opportunities to the next generation.

NOTE 1

Rouet’s (2010) MD-TRACE Model overlaps somewhat with the five elements we discuss here. See Chapters 2, 3, 6, and 28 in this volume to compare how MD-TRACE models online reading in contrast to the five elements we outline.

REFERENCES Afflerbach, P., & Cho, B.-Y. (2009). Identifying and describing constructively responsive comprehension strategies in new and traditional forms of reading. In S. E. Israel & G. G. Duffy (Eds), Handbook of research on reading comprehension (pp. 69–90). New York: Routledge. Anderson, R.C., & Pichert, J.S. (1978). Recall of previously unrecallable information following shift in perspective. Journal of Verbal Learning and Verbal Behavior, 17, 1–12. Anmarkrud, Ø., Bråten, I., & Strømsø, H.I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. Aunger, R. (2009). What’s special about human technology? Cambridge Journal of Economics, 34(1), 115–123. http://doi.org/10.1093/cje/bep018. Barthes, R. (1977). From work to text. Image-music-text, 155–164. London: Fontana. Barthes, R. (1986). The rustle of language. Oxford: Blackwell. Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online sources. Cognition and Instruction, 30(1), 39–85. http://doi.org/10.1080/07370008.2011.636495. Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of designs for learning. Written Communication, 25(2), 166–195. Blommers, P., & Lindquist, E.F. (1944). Rate of comprehension of reading: Its measurement and its relation to comprehension. Journal of Educational Psychology, 35, 449–473. Bolter, J.D. (1990). Writing space: The computer, hypertext, and the history of writing. Mahwah, NJ: Lawrence Erlbaum. Bolter, J.D. (2001). Writing space: The computer, hypertext, and the remediation of print (2nd edition). Mahwah, NJ: Lawrence Erlbaum. boyd, danah m., & Ellison, N.B. (2007). Social network sites: Definition, history and scholarship. Journal of Computer-Mediated Communication, 13(1), 210–230. http://dx.doi.org/10.1111/j.1083-6101.2007.00393.x. Braasch, J.L., Bråten, I., Strømsø, H.I., Anmarkrud, Ø., & Ferguson, L.E. (2013). Promoting secondary school students’ evaluation of source features of multiple documents. Contemporary Educational Psychology, 38(3), 180–195. Bråten, I., & Strømsø, H.I. (2010a). Effects of task instruction and personal epistemology on the understanding of multiple texts about climate change. Discourse Processes, 47, 1–31. Bråten, I., & Strømsø, H.I. (2010b). When law students read multiple documents about global warming: Examining the role of topic-specific beliefs about the nature of knowledge and knowing. Instructional Science, 38, 635–657.

Toward a New Literacies Perspective  •  73 Broadband Commission for Digital Development (2014, September). The state of broadband 2014: Broadband for all—A report by the Broadband Commission. Geneva, Switzerland: ITU/UNESCO. Accessed December 8, 2017 at: www.itu.int/dms_pub/itu-s/opb/pol/S-POL-BROADBAND.10-2014-PDF-E.pdf. Chaiklin, S., Hedegaard, M., & Jensen, U.J. (1999). Activity theory and social practice. Aarhus, Denmark: Aarhus University Press. Cho, B.-Y, Woodward, L., Li, D., & Barlow, W. (2017). Examining adolescents’ strategic processing during online reading with a question-generating task. American Educational Research Journal, 54(4), pp. 644–690. DOI: 10.3102/0002831217701694. Coiro, J. (2011). Predicting reading comprehension on the Internet: Contributions of offline reading skills, online reading skills, and prior knowledge. Journal of Literacy Research: A Publication of the Literacy Research Association, 43(4), 352–392. http://doi.org/10.1177/1086296X11421979. Coiro, J. (2015). Purposeful, critical and flexible: Vital dimensions of online reading and learning. In R.J. Spiro, M. DeSchryver, M.S. Hagerman, P.M. Morsink, & P. Thompson (Eds), Reading at a crossroads? Disjunctures and continuities in current conceptions and practices (pp. 53–64). New York: Routledge. Coiro, J., Castek, J., & Guzniczak, L. (2011). Uncovering online reading comprehension processes: Two adolescents reading independently and collaboratively on the Internet. In P.J. Dunston, L.B. Gambrell, K. Headley, S.K. Fullerton, P.M. Stecker, V.R. Gillis, & C.C. Bates (Eds), 60th Yearbook of the Literacy Research Association (pp. 354–369). Oak Creek, WI: Literacy Research Association. Coiro, J., & Dobler, E. (2007). Exploring the online reading comprehension strategies used by sixth-grade skilled readers to search for and locate information on the Internet. Reading Research Quarterly, 42(2), 214–257. http://doi.org/10.1598/RRQ.42.2.2. Coiro, J., Knobel, M., Lankshear, C., & Leu, D. (2008). Handbook of research on new literacies. New York: Erlbaum. Cole, M., & Derry, J. (2005). We have met technology and it is us. In R.J. Sternberg & D.D. Preiss (Eds), Intelligence and technology: The impact of tools on the nature and development of human abilities (pp.  209–227). Mahwah, NJ: Lawrence Erlbaum Associates. Collin, S., Karsenti, T., Ndimubandi, A., & Saffari, H. (2016). Connected generation? Digital inequalities in elementary and high school students according to age and socioeconomic level. Canadian Journal of Learning and Technology, 42(5), 1–17. http://dx.doi.org/10.21432/T21K7T. Common Core State Standards Initiative (2015). Common Core State Standards Initiative: Preparing America’s students for college and career. Retrieved from www.corestandards.org/ELA-Literacy. Cope, B., & Kalantzis, M. (2000). Multiliteracies: Literacy learning and the design of social futures. Hove, UK: Psychology Press. Cunningham, A.E., & Stanovich, K.E. (1997). Early reading acquisition and its relation to reading experience and ability 10 years later. Developmental Psychology, 33(6), 934–45. Retrieved from www.ncbi.nlm.nih.gov/ pubmed/9383616. Dahl, F. (1946). Dutch corantos, 1618–1650. Koninklijke Bibliotheek: The Hague. Derrida, J. (1976). Of grammatology (G.C. Spivak, Trans.). Baltimore, MD: Johns Hopkins University Press. (Original work published 1967.) Deschryver, M. (2014). Higher order thinking in an online world: Toward a theory of web-mediated knowledge synthesis. Teachers College Record, 116, 1–44. DeSchryver, M. (2015). Web‐mediated knowledge synthesis for educators. Journal of Adolescent & Adult Literacy, 58(5), 388–396. http://10.1002/jaal.373. Dwyer, B. (2016). Developing online reading comprehension. In K. Hall, T. Cremin, B. Comber, & L. Moll (Eds), International handbook of research on children’s literacy, learning and culture (pp. 344–358). Hoboken, NJ: Wiley-Blackwell. Fang, I. (1997). A history of mass communication: Six information revolutions. New York: Taylor & Francis. Fling, F.M. (1903). Historical synthesis. American Historical Review, 9(1), 1–22. Forzani, E. (2016). Individual differences in evaluating the credibility of online information in science: Contributions of prior knowledge, gender, socioeconomic status, and offline reading ability (Doctoral Dissertation). Retrieved from http://digitalcommons.uconn.edu/dissertations/1242. Gates, A.I. (1930). Interest and ability in reading. New York: Macmillan. Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H.I. (2010). Understanding and integrating multiple science texts: Summary tasks are sometimes better than argument tasks. Reading Psychology, 31, 30–68. Gleason, B. (2013). #OccupyWallStreet: Exploring informal learning about a social movement on Twitter. American Behavioral Scientist, 57(7), 966–982. DOI: 10.1177/00002764213479372.

74  •  Hartman et al. Goldman, S.R., Braasch, J.L.G., Wiley, J., Graesser, A.C., & Brodowinska, K. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381. Goldman, S.R., & Scardamalia, M. (2013). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities. Cognition and Instruction, 31, 255–269. Goodman, M.J., Sands, A.M., & Coley, R.J. (2015). America’s skills challenge: Millennials and the future. Princeton, NJ: Educational Testing Service. Retrieved from: www.ets.org/s/research/30079/asc-millennialsand-the-future.pdf. Google (2017). Algorithms. Retrieved from www.google.ca/insidesearch/howsearchworks/algorithms.html. Gray, W.S. (1940). Summary of reading investigations (July 1, 1938 to June 30, 1939). Journal of Educational Research, 33(7), 481–523. Greenfield, P. (1984). A theory of the teacher in the learning activities of everyday life. In B. Rogoff & J. Lave (Eds), Everyday cognition (pp. 117–138). Cambridge, MA: Harvard University Press. Greenhow, C., & Gleason, B. (2012). Twitteracy: Tweeting as a new literacy practice. The Educational Forum, 76, 464–478. http://doi.org/10.1080/00131725.2012.709032. Hagen, Å.M., Braasch, J.L.G., & Bråten, I. (2014). Relationships between spontaneous note-taking, self-reported strategies and comprehension when reading multiple texts in different task conditions. Journal of Research in Reading, 37(S1), S141–S157. http://doi.org/10.1111/j.1467-9817.2012.01536.x. Hagerman, M.S. (2014). Disruptive promise: The LINKS intervention and its impact on multiple, multimodal internet text integration. (Doctoral dissertation). Retrieved from ProQuest (UMI 3615972). Hagerman, M.S. (2017). Disrupting students’ online reading and research habits: The LINKS intervention and its impact on multiple Internet text integration processes. Journal of Literacy and Technology, 18(1), 105–156. Retrieved from www.literacyandtechnology.org/uploads/1/3/6/8/136889/jlt_v18_1_hagerman.pdf. Hall, C., & Zarro, M. (2012). Social curation on the website Pinterest.com. Proceedings of the American Society for Information Science and Technology, 49, 1–9. DOI: 10.1002/meet.14504901189. Hargittai, E. (2010). Digital na(t)ives? Variation in Internet skills and uses among members of the “net generation.” Sociological Inquiry, 80(1), 92–113. http://doi.org/10.1111/j.1475-682X.2009.00317.x. Hargittai, E., & Hsieh, Y.P. (2013). Digital inequality. In W.H. Dutton (Ed.), The Oxford handbook of Internet studies (pp.  129–150). Oxford, UK: Oxford University Press. http://doi.org/0.1093/oxfor dhb/9780199589074.001.0001. Hartman, D.K. (1995). Eight readers reading: The intertextual links of proficient readers reading multiple passages. Reading Research Quarterly, 30, 520–561. Hartman, D.K. (2004). Deconstructing the reader, the text, the author, and the context: Intertextuality and reading from a “cognitive” perspective. In N. Shuart-Faris & D. Bloome (Eds), Uses of intertextuality in classroom and educational research (pp. 353–372). Greenwich, CT: Information Age. Hartman, D.K., & Morsink, P.M. (2015). Reading at a million crossroads: Massively pluralized practices and conceptions of reading. In R.J. Spiro, M. Deschryver, M.S. Hagerman, P.M. Morsink, & P. Thompson (Eds), Reading at a crossroads? Disjunctures and continuities in current conceptions and practices (pp.  74–88). New York: Routledge. Hartman, D.K., Morsink, P.M., & Zheng, J.J. (2010). From print to pixels: The evolution of cognitive conceptions of reading comprehension. In E.A. Baker (Ed.), The new literacies: Multiple perspectives and practice (pp. 131–164). New York: Guilford Press. Holbrook, J.B. (2013). What is interdisciplinary communication? Reflections on the very idea of disciplinary integration. Synthese, 190(11), 1865–1879. DOI: 10.1007/s11229-012-0179-7. Hull, G.A., & Nelson, M.E. (2005). Locating the semiotic power of multimodality. Written Communication, 22(2), 224–261. doi: 10.1177/0741088304274170. Jackson, L.A., Zhao, Y., Kolenic, A., Fitzgerald, H.E., Harold, R., & Von Eye, A. (2008). Race, gender, and information technology use: The new digital divide. CyberPsychology & Behavior, 11(4), 437–442. http:// doi.org/10.1089/cpb.2007.0157. Jiménez, L.M., & Meyer, C.K. (2016). First impressions matter: Navigating graphic novels utilizing linguistic, visual and spatial resources. Journal of Literacy Research, 48(4), 423–447. DOI: 10.1177/1086296 X16677955. Kahne, J., Hodgin, E., & Eidman-Aadahl, E. (2016). Redesigning civic education for the digital age: Participatory politics and the pursuit of democratic engagement. Theory & Research in Social Education, 44(1), 1–35. http://doi.org/10.1080/00933104.2015.1132646.

Toward a New Literacies Perspective  •  75 Kamberelis, G., & Scott, K.D. (1992). Other people’s voices: The coarticulation of texts and subjectivities. Linguistics and Education, 4, 359–403. Kiili, C., Hirvonen, S., & Leu, D. J. (2013). Synthesizing information during online reading and writing a joint argumentative essay. In 15th Biennial EARLI Conference for Research on Learning and Instruction. Munich, Germany. Kiili, C., Laurinen, L., & Marttunen, M. (2009). Skillful Internet reader is metacognitively competent. In L.T.W. Hin & R. Subramaniam (Eds), Handbook of research on new media literacy at the K-12 level: Issues and challenges (pp. 654–656). Hershey, PA: ICI Global. Kiili, C., Leu, D.J., Marttunen, M., Hautala, J., & Leppänen, P. H. T. (2017). Exploring early adolescents’ evaluation of academic and commercial online resources related to health. Reading and Writing. Early online publication. https://doi.org/10.1007/s11145-017-9797-2 Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge: Cambridge University Press. Kintsch, W. (2013). Discourse comprehension. Control of Human Behavior, Mental Processes, and Consciousness: Essays in Honor of the 60th Birthday of August Flammer, 125. Klein, J.T. (1990). Interdisciplinarity: History, theory, and practice. Detroit, MI: Wayne State University Press. Kline, S.J. (1985). What is technology? Bulletin of Science Technology & Society, 5(1), 215–218. http://doi. org/10.1177/027046768500500301. Kobayashi, K. (2009). The influence of topic knowledge, external strategy use, and college experience on students’ comprehension of controversial texts. Learning and Individual Differences, 19(1), 130–134. http:// doi.org/10.1016/j.lindif.2008.06.001. Kress, G.R. (2003). Literacy in the new media age. London: Psychology Press. Kuhn, D., Cheney, R., & Weinstock, M. (2000). The development of epistemological understanding. Cognitive Development, 15(3), 309–328. Kwong, T.E., & Varnhagen, C.K. (2005). Strategy development and learning to spell new words: Generalization of a process. Developmental Psychology, 41(1), 148–59. http://doi.org/10.1037/0012-1649.41.1.148. Landow, G.P. (2006). Hypertext 3.0: Critical theory and new media in an era of globalization (Parallax: Re-visions of culture and society) (3rd edition). Baltimore, MD: Johns Hopkins University Press. Lankshear, C., & Knobel, M. (2011). New literacies: Everyday practices and social learning (3rd ed.). Maidenhead, UK: Open University Press. Lawless, K., & Schrader, P.G. (2008). Where do we go now? Understanding research on navigation in complex digital environments. In C. Coiro, M. Knobel, C. Lankshear, & D.J. Leu (Eds), Handbook of research on new literacies (pp. 267–296). New York: Lawrence Erlbaum Associates. Le Bigot, L., & Rouet, J.F. (2007). The impact of presentation format, task assignment, and prior knowledge on students’ comprehension of multiple online documents. Journal of Literacy Research, 39, 445–470. Leu, D.J., Everett-Cacopardo, H., Zawilinski, L., McVerry, J.G., & O’Byrne, W.I. (2012). The new literacies of online reading comprehension. In C.A. Chapelle (Ed.), The encyclopedia of applied linguistics (pp. 4239–4247). Oxford, UK: Wiley-Blackwell. Leu, D.J., Forzani, E., & Kennedy, C. (2015). Income inequality and the online reading gap. The Reading Teacher, 68(6), 422–427. http://doi.org/10.1002/trtr.1328. Leu, D.J., Forzani, E., Rhoads, C., Maykel, C., Kennedy, C., & Timbrell, N. (2014). The new literacies of online research and comprehension: Rethinking the reading achievement gap. Reading Research Quarterly, 50(1), 37–59. Leu, D.J., Kiili, C., & Forzani, E. (2015). Individual differences in the new literacies of online research and comprehension. In P. Afflerbach (Ed.), Handbook of individual differences in reading: Reader, text, and context (pp. 259–272). New York: Routledge. Leu, D.J., Kinzer, C.K., Coiro, J., Castek, J., & Henry, L.A. (2013). New literacies: A dual level theory of the changing nature of literacy, instruction, and assessment. Theoretical models and processes of reading, 6, 1150–1181. Litman, C., Marple, S., Greenleaf, C., Charney-Sirott, I., Bolz, M., Richardson, L., Hall, A., George, M., & Goldman, S. (2017). Text-based argumentation with multiple sources: A descriptive study of opportunity to learn in secondary English language arts, history, and science. Journal of Learning Sciences, 26(1), 79–130. www.projectreadi.org/wp-content/uploads/2017/05/Litman_2017_Text_based_ argumentation.pdf. Loh, K.K., & Kanai, R. (2016). How has the Internet reshaped human cognition?. The Neuroscientist, 22(5), 506–520.

76  •  Hartman et al. Lotman, J. (1977). The concept of the text. In J. Lotman, The structure of the artistic text (pp. 50–55) [Translated by Ronald Vroon]. Ann Arbor, MI: University of Michigan Press. Macedo-Rouet, M., Braasch, J.L.G., Britt, M.A., & Rouet, J. (2013). Teaching fourth and fifth graders to evaluate information sources during text comprehension. Cognition and Instruction, 31(2), 204–226. http://doi.org /10.1080/07370008.2013.769995. McGinley, W. (1992). The role of reading and writing while composing from sources. Reading Research Quarterly, 27, 227–248. McLuhan, M. (1964). Understanding media: The extensions of man. New York: New American Library. Mitcham, C. (1994). Thinking through technology: The path between engineering and philosophy. Chicago, IL: University of Chicago Press. Narvaez, D., van den Broek, P., & Ruiz, A.B. (1999). The influence of reading purpose on inference generation and comprehension in reading. Journal of Educational Psychology, 91(3), 488–496. OECD (2013). OECD Skills Outlook 2013: First results from the Survey of Adult Skills. OECD Publishing. DOI: 10.1787/9789264204256-en. O’Neil, C. (2016). Weapons of math destruction: How big data increases inequality and threatens democracy. New York: Penguin Random House. Opfer, J.E., & Siegler, R.S. (2007). Representational change and children’s numerical estimation. Cognitive Psychology, 55(3), 169–195. http://doi.org/10.1016/j.cogpsych.2006.09.002. Pariser, E. (2012). The filter bubble: How the new personalized web is changing what we read and how we think. New York: Penguin Books. Pearce, N. (2012). Clickolage: Encouraging the student bricoleur through social media. Teaching Anthropology, 2(1), 14–21. Retrieved from www.teachinganthropology.org/ojs/index.php/teach_anth/article/view/28. Pearce, N., & Learmonth, S. (2013). Learning beyond the classroom: Evaluating the use of Pinterest in learning and teaching in an introductory anthropology class. Journal of Interactive Media in Education, 2013(2), p.Art. 12. doi: http://doi.org/10.5334/2013-12. Perfetti, C.A., Rouet, J.-F., & Britt, M.A. (1999). Toward a theory of documents representation. In H. van Oostendorp & S.R. Goldman (Eds), The construction of mental representations during reading (pp. 99–122). Mahwah, NJ: Lawrence Erlbaum Associates. Plottel, J.P. (1978). Introduction. In J.P. Plottel & H. Charney (Eds.), Intertextuality: New perspectives in criticism (pp. xi–xx). New York: New York Literary Forum. Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Lawrence Erlbaum Associates. Putman, S.M., Hathaway, J.I., Coiro, J., & Quinn, D.J. (2015). Putting on CAPES to search the Internet: A framework for self-regulated learning. Literacy Learning: The Middle Years, 23(3), 38–49. Rainie, L., & Anderson, J. (2017). Code dependent: Pros and cons of the algorithm age. Pew Research Center. Retrieved from www.pewinternet.org/2017/02/08/code-dependent-pros-and-cons-of-the-algorithm-age. Rix, J. (2009). A model of simplification: The ways in which teachers simplify learning materials. Educational Studies, 35(2), 95–106. Rouet, J.-F., & Britt, M.A. (2011). Relevance processes in multiple document comprehension. In M.T. McCrudden, J.P. Maglia-no, & G. Schraw (Eds), Relevance instructions and goal-focusing in text learning. Greenwich, CT: Information Age Publishing. Quintana, C., Zhang, M., & Krajcik, J. (2005). A framework for supporting metacognitive aspects of online inquiry through software-based scaffolding. Educational Psychologist, 40(4), 37–41. Quintilian. (95 ce/1875) [Translated into English by J.S. Watson]. Institutio oratoria/Institutes of oratory. London: Bell. Reinking, D. (1998). Synthesizing technological transformations of literacy in a post-typographic world. In D. Reinking, M.C. McKenna, L.D. Labbo, & R.D. Kieffer (Eds), Handbook of literacy and technology: Transformations in a post-typographic world (pp. xi–xxx). Mahwah, NJ: Erlbaum. Ricoeur, P. (1971). What is a text? Explanation and interpretation. In P. Ricoeur, Mythic-symbolic language and philosophical anthropology (pp. 135–150). The Hague: Springer. Rouet, J.-F. (2006). The skills of document use: From text comprehension to web-based learning. Mahwah, NJ: Lawrence Erlbaum Associates. Rouet, J.-F. (2010). MD-TRACE: A cognitive model of multiple document use. A paper presented at the seminar on Computational Cognitive Modeling of Web Navigation. Utrecht, Netherlands: Utrecht University. www. researchgate.net/publication/238788660_MD-TRACE_A_Cognitive_Model_of_Multiple_Document_Use.

Toward a New Literacies Perspective  •  77 Rouet, J.-F., Britt, M.A., Mason, R.A., & Perfetti, C.A. (1996). Using multiple sources of evidence to reason about history. Journal of Educational Psychology, 88(3), 478–493. Rouet, J.-F., Favart, M., Britt, M.A., & Perfetti, C.A. (1997). Studying and using multiple documents in history: Effects of discipline expertise. Cognition and Instruction, 15, 85–106. Sharp, A.C., Sinatra, G.M., & Reynolds, R.E. (2008). The development of children’s orthographic knowledge: A microgenetic perspective. Reading Research Quarterly, 43(3), 206–226. Siegler, R. (1996). Emerging minds: The process of change in children’s thinking. New York: Oxford University Press. Siegler, R.S., & Chen, Z. (2008). Differentiation and integration: Guiding principles for analyzing cognitive change. Developmental Science, 11(4), 433–448. http://doi.org/10.1111/j.1467-7687.2008.00689.x. Spilich, G.J., Vesonder, G.T., Chiesi, H.L., & Voss, J.F. (1979). Text processing of domain-related information for individuals with high and low domain knowledge. Journal of Verbal Learning and Verbal Behavior, 18(3), 275–290. Spivey, N.N., & King, J.R. (1989). Readers as writers composing from sources. Reading Research Quarterly, 24(1), 7–26. Stanford History Education Group (November 22, 2016). Evaluating information: The cornerstone of civic responsibility. https://sheg.stanford.edu/upload/V3LessonPlans/Executive%20Summary%2011.21.16.pdf. Strømsø, H.I., Bråten, I., & Britt, M.A. (2010). Reading multiple texts about climate change: The relationship between memory for sources and text comprehension. Learning and Instruction, 20, 192–204. Street, B. (2005). At last: Recent applications of new literacy studies in educational contexts. Research in the Teaching, 39(4), 417–423. The New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66(1), 60–91. Thomson, R., Ito, N., Suda, H., Lin, F., Liu, Y., Hayasaka, R., Isochi, R., & Wang, Z. (2012). Trusting Tweets: The Fukushima disaster and information source credibility on Twitter. In Proceedings of 9th Annual International ISCRAM Conference (Vancouver, Canada). Retrieved from www.academia.edu/download/ 13370142/Thomson_et_al_2012_Trusting_Tweets.pdf. Thurlings, M., Evers, A.T., & Vermeulen, M. (2014). Toward a model of explaining teachers’ innovative behavior: A literature review. Review of Educational Research, 85(3), 1–42. http://doi.org/10.3102/00 34654314557949. Turner, K.H., Jolls, T., Hagerman, M.S., O’Byrne, W.I., Hicks, T., Eisenstock, B., & Pytash, K.E. (in press). Developing digital literacies in children and teens: A call for research and policy action. Pediatrics. Van Strien, J.L., Brand-Gruwel, S., & Boshuizen, H. (2014). Dealing with conflicting information from multiple nonlinear texts: Effects of prior attitudes. Computers in Human Behavior, 32, 101–111. Vieweg, S., Hughes, A.L., Starbird, K., & Palen, L. (2010). Microblogging during two natural hazards events: What Twitter may contribute to situational awareness. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1079–1088. Retrieved from http://dl.acm.org/citation.cfm?id=1753486. Vodanovich, S., Sundaram, D., & Myers, M. (2010). Research commentary: Digital natives and ubiquitous information systems. Information Systems Research, 21(4), 711–723. Wargo, J.M. (2015). “Every selfie tells a story . . .”: LGBTQ youth lifestreams and new media narratives as connective identity texts. New Media & Society. http://doi.org/10.1177/1461444815612447. Westheimer, J. (2015). What kind of citizen? Educating our children for the common good. New York: Teachers College Press. White, J.W., & Hungerford-Kresser, H. (2014). Character journaling through social networks: Exemplifying tenets of the new literacy studies. Journal of Adolescent and Adult Literacy, 57(8), 642–654. http://doi. org/10.1002/jaal.306. Wiley, J., Goldman, S.R., Graesser, A.C., Sanchez, C.A., Ash, I.K., & Hemmerich, J.A. (2009). Source evaluation, comprehension, and learning in Internet science inquiry tasks. American Educational Research Journal, 46, 1060–1106. Wilson, E.O. (1998). Consilience: The unity of knowledge. New York: Alfred A. Knopf. Wineburg, S.S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87. Wineburg, S., & McGrew, S. (2016). Why students can’t google their way to the truth: Fact-checkers and students approach websites differently. Education Week, 36(11), 22–28. Retrieved from www.edweek.org/ew/ articles/2016/11/02/why-students-cant-google-their-way-to.html.

78  •  Hartman et al. Wineburg, S., & Reisman, A. (2015). Disciplinary literacy in history: A toolkit for digital citizenship. Journal of Adolescent and Adult Literacy, 58(8), 636–639. http://doi.org/10.1002/jaal.410. Winne, P., & Azevedo, R. (2014). Metacognition. In K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 63–87). Cambridge: Cambridge University Press. Wolfe, M.B.W., & Goldman, S.R. (2005). Relations between adolescents’ text processing and reasoning. Cognition and Instruction, 23, 467–502. Zhang, S., & Duke, N.K. (2008). Strategies for Internet reading with different reading purposes: A descriptive study of twelve good Internet readers. Journal of Literacy Research, 40(1), 128–162. Zhang, S., & Duke, N.K. (2011). The impact of instruction of the WWWDOT framework on students’ disposition and ability to evaluate web sites as sources of information. The Elementary School Journal, 112(1), 132–154.

5

A SOCIAL PSYCHOLOGICAL PERSPECTIVE ON MULTIPLE SOURCE USE Elaboration and Persuasion Duane T. Wegener, Kathleen M. Patton, and Curtis P. Haugtvedt ohio state university, usa

Every day people encounter information and persuasive messages from many sources and in many formats (e.g., newspapers, television, social media, and face-to-face communications). Consider a person who forms a favorable opinion toward a new restaurant based on a newspaper article, but later sees a negative review of the restaurant on social media. What factors influence how the new information will affect the person’s opinion? Fortunately, research in the social psychology of persuasion helps us understand the extent to which (and the processes by which) multiple sources of information are incorporated into people’s attitudes and beliefs. Persuasion has been examined in social psychology for roughly 100 years, but the systematic study of persuasion began in earnest during World War II (see Briñol & Petty, 2012). Part of the reason that persuasion receives so much attention is that behavior is guided in part by the attitudes that people hold toward the target of a behavior or toward the behavior itself (e.g., Ajzen & Fishbein, 2005; Kraus, 1995). In this literature, attitudes are defined as evaluations of targets along a valenced good/bad, favorable/ unfavorable dimension. For example, a person could think that building nuclear power plants is a good idea or could evaluate a movie unfavorably. Attempts at persuasion then refer to attempts to change those evaluations, such as an editorial focusing on the environmental dangers of nuclear waste (aimed at making people less favorable toward nuclear power) or an advertisement edited to make the movie seem funny and enjoyable (with a goal of having people develop a positive attitude toward the movie). Research has explicitly dealt with persuasion settings in which two or more messages are encountered (or the same message is encountered on more than one occasion). One example of encountering the same message multiple times would be when the same

79

80  •  Wegener et al.

advertisement is presented repeatedly within a given television program. Research on such repetitions initially showed that repetition can increase message effectiveness to a point, but additional repetitions can undermine that initial effectiveness (e.g., Cacioppo & Petty, 1979; see Pechmann & Stewart, 1989, for a review). As research developed and an emphasis was placed on processing of persuasive communications, however, it was found that early benefits of message repetition were limited to repetition of strong (compelling) arguments. When arguments were weak (specious), repetition only led the arguments to be less effective (Cacioppo & Petty, 1989). In the text comprehension literature, the term “processing” may be limited to mental activities that occur during reading per se. In the persuasion literature, however, the term “processing” is used more broadly to pertain to active evaluation of the attitude object addressed by the message. Such processing could occur on-line during reading per se, but it could also continue after reading is complete. In particular, in the Elaboration Likelihood Model (Petty & Cacioppo, 1986; Petty & Wegener, 1999), the concept of “elaboration” built on the elaboration and depth of processing concepts in cognitive psychology (e.g., Craik & Lockhart, 1972; Craik & Tulving, 1975) to emphasize that people go beyond comprehending or placing in memory the information in a persuasive message. Instead, when people elaborate, they actively evaluate the merits of the described object or issue in relation to existing knowledge. They do this by activating relevant knowledge in memory, comparing the new information with this existing knowledge, and evaluating the implications for the overall merits of the attitude object or issue (see also Wegener, Clark, & Petty, in press). This active evaluation, captured by terms like elaboration, depth of processing, or message scrutiny, should create new cognitive structures and mental representations that can help to index the amount of elaboration that has occurred. For example, persuasion researchers have sometimes measured memory for presented message arguments (with more recall associated with greater processing of the arguments), or the number of thoughts (or reactions) people have toward the attitude object (with more thoughts indexing greater elaboration). However, these measures do not correspond very closely to the active elaboration that scrutinizes the qualities of the object or issue per se. Thus, a more common method has involved experimentally manipulating the qualities of the available information about the attitude object by providing either strong (compelling) or weak (specious) arguments in the message. Then, one examines the extent to which the manipulated quality of the object influences the thoughts (reactions) or resulting evaluations of the object. Greater elaboration is shown by larger effects of argument quality (i.e., strong arguments being more persuasive than weak arguments; for additional discussion of argument pretesting and example arguments, see Petty & Cacioppo, 1986). Strong influences of argument quality on thoughts and attitudes (one index of high levels of elaboration) also often produce strong correlations between thoughts and attitudes (another potential index of elaboration). In most persuasion studies, the primary experimental activity is not to evaluate the persuasiveness of the message but, rather, to engage with some more surface feature such as assessing text readability or audio clarity. The research then examines whether features of the situation (such as the personal relevance of the topic) or facets of the message recipient (such as individual differences in enjoyment of effortful cognitive activity) can spontaneously motivate people to actively evaluate the object or issue. Because the quality of arguments is varied within the message itself and other potential prompts to

A Social Psychological Perspective  •  81

consider the attitude object (such as an attitude question) are equated across all conditions, differences in effects of argument quality on thoughts or attitudes can be attributed to differences in processing (elaboration) of the content of the message. In this chapter, when we refer to differential processing of a persuasive communication, therefore, we are generally referring to effects of message content (argument quality) on the resulting mental representations of the object (especially object evaluations). Most persuasion research focuses on impact of a single persuasive message provided by a particular source. Even in these cases, however, the existence of a pre-message attitude toward the object or issue implies that previous information (from other sources) has been encountered. Thus, we include such studies in the current review. In other cases, however, research has explicitly included more than one message. Most typically, this would be two messages taking opposing viewpoints on the object or issue, and those two messages would be attributed to two different sources (i.e., people or entities providing the message). To be clear, in this chapter, when describing the message per se, we refer to it as the message or information. When we refer to sources, we are generally referring to the person presenting the message (such as an editorial writer or a product endorser) or the entity that is responsible for the message (such as a university committee proposing a university policy). The chapter begins by discussing two primary theories that help to organize the work on persuasion. In particular, the Elaboration Likelihood Model (ELM; Petty & Cacioppo, 1986) arrays persuasion processes along a continuum of the amount of processing (elaboration) involved in the process. A key for the current context is that the ELM makes specific predictions linking the level of elaboration involved in forming or changing one’s attitude to the consequences of that attitude for resisting later attempts at persuasion. A theory that supplements the ELM approach is the Discrepancy Motives Model (DMM; Clark & Wegener, 2013). The DMM addresses how the proor counterattitudinal nature of a message can change the extent to which message recipients think carefully about information in the persuasive appeal (as well as how additional variables, such as characteristics of the source of that pro- or counterattitudinal message, might further influence how recipients deal with that message). After discussing these two theoretical approaches, we review research (a) linking multiple sources to elaboration and (b) linking elaboration to resistance to attitude change. The chapter ends with conclusions and implications for future research.

THEORETICAL BACKGROUND Elaboration Likelihood Model The first primary persuasion theory to be discussed is Petty and Cacioppo’s (1986) Elaboration Likelihood Model (ELM). Perhaps the single most influential theoretical framework for understanding attitude change (see Petty & Wegener, 1998), the ELM provides a useful framework for considering how people deal with persuasive information coming from multiple sources or from a new source after basing one’s attitude on previously encountered information. The ELM arrays processes underlying persuasion along a continuum ranging from relatively low-thought processes (i.e., low elaboration) to relatively high-thought processes (i.e., high elaboration; see Petty & Wegener, 1999). At the lower end of the elaboration

82  •  Wegener et al.

continuum (sometimes referred to as the peripheral route to persuasion), non-effortful/ superficial processing encourages the use of simple cues to the adequacy of an attitudinal position. For example, if a credible source supports a policy, recipients of that message who are unmotivated or unable to think carefully about the message might use the credibility of the source as a cue to evaluate the policy positively. In contrast, at the higher end of the elaboration continuum (when motivation and ability are high), the person’s evaluation of the message is more likely to reflect thoughtful evaluation of the merits of the policy. As noted earlier, this higher-thought elaboration consists of comparing available information to related knowledge in memory and engaging in active evaluation of the central merits of the attitude object (i.e., the primary qualities of the object that would make it a good or bad instance of that kind of object). For example, in the case of a policy, central merits might include implications of the policy for people affected by it (the extent to which those consequences are favorable or unfavorable), costs of implementing the policy, or benefits to society more generally. This high end of the elaboration continuum is sometimes referred to as the central route to persuasion (Petty & Cacioppo, 1986; for discussion see Petty & Wegener, 1999). Examples of motivational variables that place people relatively high or low on the elaboration continuum include the extent to which the topic of the message is perceived as personally relevant to the message recipient (i.e., perceived as likely to impact the person’s life; Petty, Cacioppo, & Schumann, 1983) or a sense of accountability (e.g., the need for the person to explain his or her opinion to others; Chaiken, 1980). An example ability variable would be the extent to which the person is distracted during receipt of the message (e.g., Petty, Wells, & Brock, 1976). A key point for the current discussion is that the amount of elaboration involved in evaluating a message also helps to determine the consequences of the attitudes that are formed. Specifically, higher levels of elaboration are associated with greater persistence of the attitudes over time (in the absence of any attacking information), greater resistance to change when attacked, and greater impact of the attitudes on related thinking and behavior (for a review, see Petty, Haugtvedt, & Smith, 1995). The other key aspect of the ELM is the notion of multiple roles for persuasion variables (Petty & Cacioppo, 1986; Petty & Wegener, 1998, 1999). That is, at different levels of elaboration, the same persuasion variable, such as a credible source, could influence attitudes through different mechanisms. When elaboration is low (because either motivation or ability is lacking), a credible source could be used as a simple cue to accept the message. When elaboration is high, however, source credibility would only be expected to influence persuasion to the extent that it can serve as an argument itself (as when the source is being evaluated, and credibility constitutes a central merit of the source) or can influence the evaluations of the central merits of the object. The ELM outlines a number of high-elaboration roles that a variable like source credibility might play. That is, even when the credibility of the source cannot serve as an argument itself, it could bias assessments of the central merits (Chaiken & Maheswaran, 1994) or validate thoughts the person has already had (e.g., Briñol, Petty, & Tormala, 2004). Thoughts are most likely to be biased (in a direction toward a credible source or away from a non-credible source) when the message is ambiguous (i.e., not clearly strong or weak; Chaiken & Maheswaran, 1994). Once a person has had thoughts about the message, learning that the source of that message is credible can increase confidence in whatever thoughts are in mind (Briñol et  al., 2004). Thus, the high-elaboration roles have additional limiting conditions (i.e., biases in elaboration are unlikely when

A Social Psychological Perspective  •  83

information is already quite clear, and thought validation is unlikely if the validating factor occurs before thoughts are in mind and salient). When background motivation and ability conditions do not encourage particularly high or low levels of elaboration, source characteristics can affect persuasion by moving people up or down the elaboration continuum. For example, when unsure whether processing will be worth the effort, message recipients might be more willing to elaborate when the information is coming from a credible rather than non-credible source (Tobin & Raymundo, 2009). As discussed under the rubric of the Discrepancy Motives Model (Clark & Wegener, 2013), however, the impact of source or other variables on amount of processing can often be further moderated by the extent to which the message agrees with or disagrees with the person’s existing attitude. Discrepancy Motives Model Research on discrepancies between a persuasive message and a person’s existing attitude generally use only a single message. However, the existing attitude implies that previous sources of information have been encountered. This discrepancy between the attitude the person holds and the position taken by a new source of information can have important implications for how the new information is processed and how persuasive it might ultimately be. Our discussion is based on the Discrepancy Motives Model (DMM; Clark & Wegener, 2013). This account focuses on the different motives underlying the processing of information that opposes one’s current attitude (i.e., counterattitudinal information) versus information that supports one’s current attitude (i.e., proattitudinal information). When a person receives a counterattitudinal message, its threat to the person’s current opinion (cf., Cacioppo & Petty, 1979) may motivate the person to defend his or her attitude. In that context, variables capable of enhancing or reducing the perceived threat should moderate (i.e., vary) the extent to which the message motivates the person to extensively scrutinize the message. From the DMM perspective, then, counterattitudinal messages would sometimes prompt effortful processing, but sometimes they would not. At the same time, motives related to threat and defense form only part of the picture. The processing of proattitudinal messages might often serve different goals. When a message supports one’s current view, s/he may seek out and elaborate on that information as a way to support or bolster the attitude. If so, variables that change the extent to which the message appears supportive or the extent to which people want to bolster their attitude could influence message processing. Therefore, similar to counterattitudinal messages, there should be circumstances in which proattitudinal messages receive a great deal of elaboration and others in which they do not.

RESEARCH LINKING MULTIPLE SOURCES TO ELABORATION AND ELABORATION TO RESISTANCE TO CHANGE Influences of Message Sources on Amount of Elaboration Since the emergence of dual- (Chaiken, Liberman, & Eagly, 1989) and multi-process (Petty & Cacioppo, 1986) models of persuasion, much research has focused on the

84  •  Wegener et al.

extent to which variables (such as the discrepancy between the message position and the recipients’ pre-message opinion) can increase or decrease motivation to process information. Influences on amount of processing were postulated to occur when background levels of motivation and ability are relatively moderate and not constrained to be particularly high or low (Petty & Cacioppo, 1986). These models specified conditions under which effects on amount of information processing were likely, but they did not specify the particular pattern of effects for each persuasion variable. To make such predictions, complementary theories were developed. The first part of this section will briefly discuss “first-generation” theories linking sources to elaboration of messages given by those sources. The majority of the discussion, however, will focus on recent research examining moderation of those first-generation effects (as discussed by the DMM; see also Clark & Wegener, 2013). This recent research has examined how the proattitudinal or counterattitudinal position of the message (agreement or disagreement with opinions formed based on previous sources) can change the effects of other persuasion variables on the amount of processing given to the message. First-Generation Source Influences on Elaboration The “first-generation” questions linking sources to elaboration examined conceptual main effects of a given source-related variable on the amount of elaboration undertaken. One example of such an effect would be source magnification (i.e., the multiple source effect). This occurs when the same (strong or weak) arguments have greater impact on post-message attitudes (are processed more extensively) when the arguments are presented by multiple sources – one argument per source – than when all the same arguments are presented by a single source (Harkins & Petty, 1981; Moore & Reardon, 1987). This effect is thought to stem from the perception that each source is likely to draw on independent pools of knowledge. Consistent with this idea, Harkins and Petty (1987) manipulated whether the sources were depicted as independent versus part of the same committee and, in another study, whether committee members had similar versus dissimilar perspectives. The multiple source effect on processing only occurred when the sources were described as independent or representing different perspectives (Harkins & Petty, 1987). As noted earlier, another important consideration is the extent to which a current message agrees or disagrees with the person’s current attitude (which was presumably formed based on exposure to prior information). Until recently, the prevailing “first-generation” conceptualization was that counterattitudinal (disagreeable) messages were more threatening than proattitudinal (agreeable) messages and, therefore, received greater processing (Cacioppo & Petty, 1979; Petty, Cacioppo, & Haugtvedt, 1992). Early evidence consistent with this idea came from work linking counterattitudinal messages to greater counterarguing of message arguments (i.e., more negative thoughts about the advocacy; Brock, 1967; Ditto & Lopez, 1992), enhanced time evaluating the message (Edwards & Smith, 1996), and greater argument recall compared to proattitudinal messages (Cacioppo & Petty, 1979). Worth and Mackie (1987) manipulated the quality of arguments provided by a political candidate who was either opposed to or in favor of government controls on acid rain. Argument quality had little impact on post-message attitudes when the message was relatively proattitudinal, but the quality of the arguments had significantly greater influence on post-message

A Social Psychological Perspective  •  85

attitudes (i.e., strong arguments were more persuasive than weak arguments) when the appeal was relatively counterattitudinal. Thus, a good deal of early research was consistent with the idea that prior attitudes (based on previous use of source information) would influence how thoroughly new information would be considered. Second-Generation Source Influences on Elaboration As noted earlier, the DMM approach built on the idea of counterattitudinal messages being more threatening than proattitudinal messages. If counterattitudinal messages are threatening, then variables increasing the perceived threat or the motive to defend the attitude could also affect how much a counterattitudinal message is processed. Similarly, if proattitudinal messages present an opportunity to bolster or support the attitude, then variables increasing the motive to bolster the attitude could also affect how much a proattitudinal message is processed. Recent research supports each of these contentions. As noted in the description of the ELM, influences on message processing should occur mostly when elaboration likelihood is not particularly high (i.e., message recipients highly motivated and able to process) or low (i.e., message recipients lacking in motivation or ability). Some research examining such conditions of moderate elaboration likelihood identified first-generation effects of source credibility on message processing (e.g., Tobin & Raymundo, 2009). More recently, however, second-generation research has suggested that source credibility effects depend on the extent to which the message is agreeable (proattitudinal) or disagreeable (counterattitudinal). In an initial investigation, Clark, Wegener, Habashi, and Evans (2012) measured participants’ pre-message attitude toward the taxation of junk food. Later, participants were told that the source of a forthcoming message was either highly expert (a leading scholar in the field of health and food sciences) or lacking expertise (a high school junior). The message presented either strong or weak arguments advocating junk food taxes in the U.S. When the message was relatively counterattitudinal (because recipients held unfavorable pre-message attitudes toward the proposal), argument quality influenced post-message attitudes more when the source was an expert than when the source lacked expertise. In other words, expert sources were more persuasive when they presented strong rather than weak arguments, whereas non-expert sources were equally persuasive regardless of which arguments they presented. In contrast, when the message recipient held more favorable pre-message attitudes (so the message was relatively proattitudinal), argument quality influenced persuasion more when the source lacked expertise than when the source was highly expert. Conceptually similar effects have been found when the current message source is manipulated to be effective versus ineffective at implementing his or her preferred policies (Clark & Wegener, 2009). That is, in general, source characteristics that trigger stronger expectations of validity of their message or perceptions that the source will be able to facilitate the proposed outcomes have been associated with (a) greater processing of counterattitudinal information, and (b) decreased processing of proattitudinal information. Conversely, source characteristics that lead message recipients to anticipate invalidity of the message or inability of the source to bring about the advocated policy have been linked to greater processing of proattitudinal messages but reduced processing of counterattitudinal messages. As specified by the DMM, these processing outcomes and the

86  •  Wegener et al.

mediational evidence are consistent with sources influencing motives to defend or bolster one’s pre-message views. It is also important to note that motives to defend or bolster one’s attitude would not be the only possible reason for moderation of source effects by message discrepancy. For example, one early example of proattitudinal messages receiving greater scrutiny than counterattitudinal messages was when the source represented a numeric minority (Baker & Petty, 1994). More generally, Baker and Petty found a similar pattern to the credibility studies. Majority sources enhanced processing of counterattitudinal messages, but minority sources enhanced processing of proattitudinal messages (see also Martin & Hewstone, 2008). Baker and Petty (1994) interpreted the pattern as stemming from the surprise involved in learning that one holds a minority position (both when a counterattitudinal appeal is characterized as the majority position and when a proattitudinal appeal is characterized as the minority position). Perhaps future research taking a DMM approach will link majority–minority status of the source with perceived threat of counterattitudinal messages (and the accompanying defense motivation) and perceived support by proattitudinal messages (and accompanying motivations to bolster one’s attitude). Influences of Pre-Message Attitudes on Amount of Elaboration In addition to early work linking discrepancy between message position and prior attitudes to amount of processing (Cacioppo & Petty, 1979; Worth & Mackie, 1987), the dual- and multi-process models also inspired research linking properties of premessage attitudes to message processing. Similar to the work on message sources, it later became clear that effects of these attitude properties depend on the extent to which the persuasive message agrees with or disagrees with the person’s existing attitude (i.e., the extent to which the message is proattitudinal or counterattitudinal). In the following sections, we discuss first-generation research linking the attitude properties of accessibility and ambivalence to message processing and second-generation research showing that these initial effects depend on the pro- or counterattitudinal nature of the message that is encountered (i.e., on the extent to which the current message agrees or disagrees with previously encountered information). First-Generation Research Linking Pre-Message Attitude Properties to Elaboration One key property of attitudes is the degree to which people hold mixed or conflicted opinions. For example, a person might be quite favorable toward ice cream as a dessert but also realize and acknowledge that ice cream has high levels of fat and calories that are bad for health. The simultaneous activation of positive and negative evaluations is called attitude ambivalence (see Olson & Zanna, 1993). Ambivalent attitudes are considered as structurally weak (easy to change) and as psychologically uncomfortable (motivating people to reduce the discomfort; for a review, see van Harreveld, van der Pligt, & De Liver, 2009). The prominent first-generation view linking pre-message attitude ambivalence to message processing was to suggest that ambivalence would

A Social Psychological Perspective  •  87

lead people to effortfully process attitude-relevant information as a way to resolve the conflicting evaluations (e.g., Maio, Bell, & Esses, 1996). Another prominent property of attitudes is their accessibility in memory – how quickly and easily they are retrieved from memory upon encountering the attitude object. The degree of accessibility is conceptualized as reflecting the associative strength between a person’s mental representation of the object and their evaluation of it (Fazio, 1995). First-generation research suggested that higher levels of premessage attitude accessibility were associated with greater message scrutiny (Fabrigar, Priester, Petty, & Wegener, 1998). The puzzle in these findings, however, was that ambivalence and accessibility are negatively correlated. Thus, at least when measuring each property (as in Maio et al., 1996, and one of the Fabrigar et al., 1998, studies), the message recipients who were ambivalent should also have been low in attitude accessibility (and vice versa). With a closer examination of the discrepancies between the prevailing pre-message attitudes and the message positions in the respective studies, these patterns made more sense. That is, a second-generation emphasis on message discrepancies solved the puzzle nicely. Second-Generation Research Linking Pre-Message Attitude Properties to Elaboration The DMM perspective suggested that ambivalence would motivate message recipients to bolster their existing attitude, but this bolstering motive would not be equally served by processing any type of information. Proattitudinal information should serve this motive better than counterattitudinal information. In fact, the message used by Maio et al. (1996) appeared largely consistent with recipients’ pre-message attitudes. Thus, from the DMM perspective, ambivalent attitudes (that could follow from previous exposure to mixed information) should lead people to process proattitudinal more than counterattitudinal information. In contrast, when people are unambivalent, counterattitudinal information might receive more processing than proattitudinal information for all the reasons previously discussed related to early work relating message discrepancy to elaboration (e.g., Cacioppo & Petty, 1979; Edwards & Smith, 1996). Clark, Wegener, and Fabrigar (2008b) tested this possibility in a pair of studies examining attitudes toward nuclear power and taxation of junk food. The valence and ambivalence of participants’ pre-message attitudes were measured. The message contained either strong (compelling) or weak (specious) arguments supporting the advocacy. In each study, when encountering a relatively counterattitudinal message (where defense motives would encourage processing), message processing was greater (i.e., larger effects of argument quality on post-message attitudes) when the participant’s pre-message attitude was relatively unambivalent rather than ambivalent. However, when the message was relatively proattitudinal (and motives to bolster would encourage processing), message processing was greater when the participant’s pre-message attitude was relatively ambivalent rather than unambivalent. Additional data showed that highly ambivalent participants believed that a proattitudinal message would better reduce their ambivalence than a counterattitudinal message, and such perceptions accounted for effects of message discrepancy on processing by these ambivalent message recipients.

88  •  Wegener et al.

Regarding attitude accessibility, the DMM might suggest that a counterattitudinal message would be perceived as more of a threat when message recipients have a clearer idea of what their attitudes are (when their attitudes are more accessible). In contrast, an agreeable (proattitudinal) message might seem redundant and unneeded when one already has an accessible attitude. Yet, when one lacks attitude clarity (an inaccessible attitude), the proattitudinal message might be viewed as capable of bolstering the attitude and making it stronger. To test these predictions, Clark, Wegener, and Fabrigar (2008a) manipulated participants’ attitudes toward a tuition proposal at an unnamed university and manipulated attitude accessibility by having participants express the attitude either once (low accessibility) or six times (high accessibility) over the course of a survey (see Fabrigar et al., 1998). Then message recipients encountered either strong or weak arguments in favor of the tuition plan (which was counterattitudinal for some recipients and proattitudinal for others based on the attitude manipulation). When the message was counterattitudinal (and message processing would be encouraged by defense motives), the argument quality effect on post-message attitudes was greater when pre-message attitudes were high rather than low in accessibility. When the message was proattitudinal (and message processing would be encouraged by bolstering motives), however, the pattern was opposite. The argument quality effect on post-message attitudes was greater when pre-message attitudes were low rather than high in accessibility (for a description of related research dealing with pre-message attitude certainty, see Clark & Wegener, 2013). First-generation theories had related each of the pre-message attitude properties to the overall extent of processing of attitude-related information. As a result, the early effects appeared paradoxical because accessible attitudes were likely associated with low rather than high levels of ambivalence (and yet, both high ambivalence and high accessibility seemed to increase message processing). By proposing that such effects would often reverse when comparing relatively proattitudinal versus counterattitudinal messages, the DMM helped to resolve these seeming contradictions in the literature. Thus, in addition to the ability of previous information to make later information more versus less agreeable with the person’s opinion, it is also important to consider whether previous information created an attitude that is ambivalent, highly accessible, etc. By knowing more about the properties of the pre-message attitude, one can better predict how carefully they will deal with new information on that topic. The existing research is unlikely to be exhaustive. Additional properties such as the importance of the pre-message attitude (Eaton & Visser, 2008) or the amount of attitude-relevant knowledge (Wood, Rhodes, & Biek, 1995) might combine with message discrepancy to influence message processing. Wrapped up in the processing patterns in this section are instances in which a weak second (or most recent) message is less persuasive for one pre-message attitude property than another. For example, when high levels of ambivalence reduce processing of a counterattitudinal message, one could say that the greater persuasion by weak counterattitudinal arguments constitutes a case of poor resistance to change when initial attitudes were ambivalent (cf. Armitage & Conner, 2000). An entire literature has developed over the years to examine resistance to change when an existing attitude is directly attacked, and much of this work links the level of elaboration given to initial information as a determinant of how effectively the attitude withstands such attacks

A Social Psychological Perspective  •  89

(not only when the attack is rather weak, but also when it is quite strong). In the following section of the chapter, then, we review this literature directly linking the level of elaboration given to initial information to the level of resistance of the attitude to new attacking (counterattitudinal) information. Influences of Elaboration on Resistance to Later Attempts at Persuasion Research using the ELM framework has directly linked the elaboration of initial information to attitudes that successfully resist subsequent attempts at changing those attitudes (i.e., the elaboration-resistance hypothesis; Petty et al., 1995). Studies examining such issues have typically measured or manipulated motivation or ability to process that have been shown in earlier studies to influence the amount of processing (typically by creating large versus small influences of argument quality on thoughts and attitudes). In the resistance studies, researchers exposed research participants to initial information that created equally extreme initial attitudes across levels of motivation or ability to process, thereby making the amount of change in relation to attacking information directly informative about relative resistance (for additional discussion, see Petty et al., 1995; Wegener, Petty, Smoak, & Fabrigar, 2004). In an early example, Haugtvedt and Petty (1992) measured message recipients’ need for cognition (i.e., an individual difference in motivation to process information; Cacioppo & Petty, 1982). To create equally favorable initial attitudes, they gave participants an initial strong message about the safety of a familiar food additive from an expert source. This initial message presumably created equally favorable attitudes because of low-thought use of the expertise cue by people unmotivated to think and more elaborated thinking about the strong arguments by people motivated to think. Participants next read a relatively weak attacking message questioning the food additive’s safety provided by a different expert source. As expected, those high in need for cognition were significantly more resistant to the attacking message than people low in need for cognition. Conceptually similar effects of need for cognition on resistance to change have been shown when disagreeing dyad members must come to a consensus (i.e., pre-discussion attitudes of dyad members high in need for cognition guide the resulting consensus; Shestowsky, Wegener, & Fabrigar, 1998). These examples demonstrate the durability of attitudes in the face of discordant information when those attitudes were initially formed under conditions of high elaboration (here, due to being personally motivated to think deeply about a message). Beyond individual differences in motivation, manipulations of factors that influence the initial amount of elaboration have also been shown to create differences in resistance to an attacking message. For example, Blankenship and Wegener (2008) had participants consider a message in relation to important or unimportant values. Linking the message to important values led to higher levels of elaboration (i.e., greater persuasion by strong than weak arguments manipulated within-subject), and individual-level measures of this amount of elaboration mediated the link between value importance and resistance to change when the person’s initial opinion was later attacked. Manipulations of personal relevance of an attitude topic have also created attitudes that are more resistant to change when later attacked by a written message (see Petty, Haugtvedt, Heesacker, & Cacioppo, 1995, described in Petty, Haugtvedt, &

90  •  Wegener et al.

Smith, 1995) or by a face-to-face communication (see Johnson, Fabrigar, Wegener, & Rosen, 2002, discussed in Wegener et al., 2004). One potential shortcoming of motivational manipulations of elaboration is that the motivation may carry through the receipt of the attacking message, thereby creating different levels of processing of the attacking message as well as the initial message. Manipulations of ability to process when initial attitudes or judgments are formed, however, should be less likely to influence how attacking information is processed if the distraction is removed prior to receipt of the attacking information. Wegener, Clark, and Petty (2006) conducted such a study in the area of stereotyping and prejudice. Research participants encountered ambiguous intelligence test performance by a child high or low in socioeconomic status (SES) under conditions of high or low cognitive load. Despite equal stereotype-consistent judgments of the low-SES child as having lower intelligence, lower cognitive load during the test information led to greater resistance to a new communication from a person who had observed the child in different circumstances. In the area of numerical anchoring, Blankenship, Wegener, Petty, Detweiler-Bedell, and Macy (2008) created equal amounts of anchoring across levels of cognitive load (that in an earlier study had successfully influenced the extent to which background knowledge was brought to bear on the anchored judgment). After being told that a portion of prior participants made considerably different judgments from their own, research participants’ anchored judgments were less affected by the consensus information when the anchored judgments had been formed under low rather than high levels of cognitive load. Another context that should leave processing of attacking information equal across conditions involves cosmetic versus substantive variation in initial information. That is, Haugtvedt, Schumann, Schneier, and Warren (1994) varied whether the same argument was presented three times by three different endorsers (cosmetic variation) or the same endorser presented three different arguments (substantive variation). An initial study showed that cosmetic variation drew attention to endorser characteristics (relatively low elaboration of product attributes) whereas substantive variation drew attention to characteristics of the product (relatively high elaboration of product attributes). The two types of message repetition led to equivalently extreme initial attitudes and level of attitude confidence, and the attitudes persisted across a one-week delay to an equal degree. After that one-week delay, however, participants received information unfavorable toward the product. Substantive variation in the initial information led to greater resistance to change in face of the attacking information compared with cosmetic variation in the initial information. Another way to address the notion that some resistance outcomes reflect greater processing of weak attacking messages by highly motivated message recipients is to use equally strong messages for both the initial and attacking message. Haugtvedt and Wegener (1994) did exactly that by providing two equally strong messages in either a pro-con or con-pro order (so across conditions, the attacking messages would have exactly the same level strength as the initial information). Haugtvedt and Wegener manipulated motivation to process by making the topic of the message personally relevant (or irrelevant). Historically, there had been evidence that the first message encountered is advantaged in shaping the resultant attitude (i.e., a primacy effect; Lund, 1925; Knower, 1936), suggesting overall resistance to the second message. However, outcomes have been mixed, with some studies showing no order effects or even

A Social Psychological Perspective  •  91

an advantage for the second message (i.e., recency effects; e.g., Hovland & Mandell, 1957; Lana, 1961, 1963). Taking an ELM perspective, Haugtvedt and Wegener (1994) hypothesized and found that primacy effects (strong resistance) are more likely when the message topic is personally relevant, and recency effects (weak resistance) are more likely when the message topic is low in relevance. Moreover, high-motivation primacy effects were related to active counterarguing of the second (attacking) message, whereas low-motivation recency effects were related to memory for the second (most recently encountered) message. The Haugtvedt and Wegener (1994) pattern occurred when the messages were separated and attributed to different sources (i.e., chunked). However, a different pattern may occur when the messages run together in a single stream of information. That is, Petty, Tormala, Hawkins, and Wegener (2001) conceptually replicated the Haugtvedt and Wegener (1994) pattern when each message was separate and attributed to a different source (and motivation to think was measured using the need for cognition scale). When the information was unchunked, however, high motivation to think led to greater recency effects than when motivation to think was low. This shift in effects may suggest that the opportunity and signal to consolidate one’s opinion when the messages were separated constituted an important part of the primacy effects when motivation to think was high. When information is unchunked and no signal occurs to stop and consolidate, those high in motivation to think may attend to all information and only consolidate at the end (when the most recently encountered information is salient).

CONCLUSIONS AND IMPLICATIONS FOR FUTURE RESEARCH When it comes to persuasive messages, people do not simply learn new information and place it in memory. When motivated and able, they elaborate on the information, scrutinize its merits, compare it with other related information in memory, and evaluate the extent to which it fits or conflicts with what they already know. This elaboration has consequences for the extent to which later information is successful or unsuccessful in changing the person’s attitudes. In addition, existing opinions can influence the extent to which new agreeable (proattitudinal) or disagreeable (counterattitudinal) information receives processing. Such influences of previous evaluations and knowledge may add to or influence the types of “sourcing” that have been recently studied in comprehension of texts that take inconsistent or opposing positions (e.g., Braasch, Rouet, Vibert, & Britt, 2012). In fact, one potentially interesting direction for future research might examine whether the same kinds of discrepancy-induced source comprehension occur when people are first encountering novel information as opposed to when the discrepant information appears only after forming a clear evaluation of the issue (cf. Petty et al., 2001). As highlighted earlier, persuasion research approaches message processing a bit differently than in the text comprehension literature. Although use of post-message measures has proven useful (and has allowed such processing to proceed uninterrupted), it may be beneficial to integrate methods used in the text comprehension literature to better capture particular features of on-line processing of written persuasive messages. Some within-message techniques have been used, such as having message recipients stop and indicate agreement or disagreement with message arguments while reading (e.g., Carter

92  •  Wegener et al.

& Simpson, 1973). However, the concern with such procedures is that the inclusion of the option to stop during on-line processing may in itself change the processing or its effects on the attitude. Subtler methods may help further refine our understanding of persuasion processes. For example, in the chunking research described earlier (Petty et al., 2001), the observed effects indicate potential differences between situations in which the attitude is consolidated prior to receipt of the opposing information and situations in which the only opportunity to consolidate comes after all information has been presented. Including measures of on-line processing in the future might shed further light on attempts at consolidation of attitudes and the processes underlying these differences. This is but one example of research warranted in a multi-message domain, as most psychological work on persuasion uses single messages. There is still much to learn about potential roles for serially encountered messages, as well as processes involved when multiple messages are available and the reader is able to choose which to read (i.e., selective exposure; for a review, see Hart et  al., 2009). For example, in singlemessage research, recent studies have shown that learning about the credibility of the source only after information is encountered can lead to validation effects (where the quality of the source can either increase or decrease confidence in one’s thoughts about the object of the message; Briñol et al., 2004). For some of these effects, confirmation or disconfirmation of one’s existing thoughts about the source information appears to be a key component (Clark, Wegener, Sawicki, Petty, & Briñol, 2013). If so, the same type of validation effect should also be possible when the confirmation or disconfirmation comes from a separate message rather than from characteristics of the initial source. These and other new roles for sequentially encountered messages await future research. Importantly, the ELM and DMM approaches provide guidance on what some of those yet-to-be-studied roles are likely to be.

REFERENCES Ajzen, I., & Fishbein, M. (2005). The influence of attitudes on behavior. In D. Albarracín, B. T. Johnson, & M. P. Zanna (Eds.), The handbook of attitudes and attitude change (pp. 173–221). Mahwah, NJ: Erlbaum. Armitage, C. J., & Conner, M. (2000). Attitudinal ambivalence: A test of three key hypotheses. Personality and Social Psychology Bulletin, 26(11), 1421–1432. Baker, S. M., & Petty, R. E. (1994). Majority and minority influence: Source-position imbalance as a determinant of message scrutiny. Journal of Personality and Social Psychology, 67, 5–19. Blankenship, K. L., & Wegener, D. T. (2008). Opening the mind to close it: Considering a message in light of important values increases message processing and later resistance to change. Journal of Personality and Social Psychology, 94, 196–213. Blankenship, K. L., Wegener, D. T., Petty, R. E., Detweiler-Bedell, B., & Macy, C. L. (2008). Elaboration and consequences of anchored estimates: An attitudinal perspective on numerical anchoring. Journal of Experimental Social Psychology, 44, 1465–1476. Braasch, J. L. G., Rouet, J.-F., Vibert, N., & Britt, M. A. (2012). Readers’ use of source information in text comprehension. Memory & Cognition, 40, 450–465. Briñol, P., & Petty, R. E. (2012). The history of attitudes and persuasion research. In A. Kruglanski & W. Stroebe (Eds.), Handbook of the history of social psychology (pp. 285–320). New York: Psychology Press. Briñol, P., Petty, R. E., & Tormala, Z. L. (2004). Self-validation of cognitive responses to advertisements. Journal of Consumer Research, 30, 559–573. Brock, T. C. (1967). Communication discrepancy and intent to persuade as determinants of counterargument production. Journal of Experimental Social Psychology, 3, 296–309.

A Social Psychological Perspective  •  93 Cacioppo, J. T., & Petty, R. E. (1979). Effects of message repetition and position on cognitive responses, recall, and persuasion. Journal of Personality and Social Psychology, 37, 97–109. Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 51, 1032–1043. Cacioppo, J. T., & Petty, R. E. (1989). Effects of message repetition on argument processing, recall, and persuasion. Basic and Applied Social Psychology, 10(1), 3–12. Carter, R. F., & Simpson, R. (1973). Application of signaled stopping technique to communication research. In P. Clarke (Ed.), New models for mass communication research (pp. 15–44). Beverly Hills, CA: Sage. Chaiken, S. (1980). Heuristic versus systematic information processing and the use of source versus message cues in persuasion. Journal of Personality and Social Psychology, 39(5), 752–766. Chaiken, S., Liberman, A., & Eagly, A. H. (1989). Heuristic and systematic information processing within and beyond the persuasion context. In J. S. Uleman & J. A. Bargh (Eds.), Unintended thought (pp. 212–252). New York: Guilford. Chaiken, S., & Maheswaran, D. (1994). Heuristic processing can bias systematic processing: Effects of source credibility, argument ambiguity, and task importance on attitude judgment. Journal of Personality and Social Psychology, 66, 460–473. Clark, J. K., & Wegener, D. T. (2009). Source entitativity and the elaboration of persuasive messages: The roles of perceived efficacy and message discrepancy. Journal of Personality and Social Psychology, 97, 42–57. Clark, J. K., & Wegener, D. T. (2013). Message position, information processing, and persuasion: The Discrepancy Motives Model. In P. Devine & A. Plant (Eds.), Advances in experimental social psychology (Vol. 47, pp. 189–232). Burlington, VA: Academic Press. Clark, J. K., Wegener, D. T., & Fabrigar, L. R. (2008a). Attitude accessibility and message processing: The moderating role of message position. Journal of Experimental Social Psychology, 44, 354–361. Clark, J. K., Wegener, D. T., & Fabrigar, L. R. (2008b). Attitudinal ambivalence and message-based persuasion: Motivated processing of proattitudinal information and avoidance of counterattitudinal information. Personality and Social Psychology Bulletin, 34, 565–577. Clark, J. K., Wegener, D. T., Habashi, M. M., & Evans, A. T. (2012). Source expertise and persuasion: The effects of perceived opposition or support on message scrutiny. Personality and Social Psychology Bulletin, 38, 90–100. Clark, J. K., Wegener, D. T., Sawicki, V., Petty, R. E., & Briñol, P. (2013). Evaluating the message or the messenger? Implications for self-validation in persuasion. Personality and Social Psychology Bulletin, 39, 1571–1584. Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11, 671–684. Craik, F. I. M., & Tulving, E. (1975). Depth of processing and the retention of words in episodic memory. Journal of Experimental Psychology: General, 104, 268–294. Ditto, P. H., & Lopez, D. F. (1992). Motivated skepticism: Use of differential decision criteria for preferred and nonpreferred conclusions. Journal of Personality and Social Psychology, 63, 568–584. Eaton, A. A., & Visser, P. S. (2008). Attitude importance: Understanding the causes and consequences of passionately held views. Social and Personality Psychology Compass, 2, 1719–1736. Edwards, K., & Smith, E. E. (1996). A disconfirmation bias in the evaluation of arguments. Journal of Personality and Social Psychology, 71, 5–24. Fabrigar, L. R., Priester, J. R., Petty, R. E., & Wegener, D. T. (1998). The impact of attitude accessibility on elaboration of persuasive messages. Personality and Social Psychology Bulletin, 24, 339–352. Fazio, R. H. (1995). Attitudes as object-evaluation associations: Determinants, consequences, and correlates of attitude accessibility. In R. E. Petty & J. A. Krosnick (Eds.), Attitude strength: Antecedents and consequences (pp. 247–282). Hillsdale, NJ: Erlbaum. Harkins, S. G., & Petty, R. E. (1981). The effects of source magnification of cognitive effort on attitudes: An information processing view. Journal of Personality and Social Psychology, 40, 401–413. Harkins, S. G., & Petty, R. E. (1987). Information utility and the multiple source effect in persuasion. Journal of Personality and Social Psychology, 52, 260–268. Hart, W., Albarracìn, D., Eagly, A. H., Brechan, I., Lindberg, M. J., & Merrill, L. (2009). Feeling validated versus being correct: A meta-analysis of selective exposure to information. Psychological Bulletin, 135, 555–588. Haugtvedt, C. P., & Petty, R. E. (1992). Personality and persuasion: Need for cognition moderates the persistence and resistance of attitude changes. Journal of Personality and Social Psychology, 63, 308–319.

94  •  Wegener et al. Haugtvedt, C. P., Schumann, D. W., Schneier, W. L., & Warren, W. L. (1994). Advertising repetition and variation strategies: Implications for understanding attitude strength. Journal of Consumer Research, 21(1), 176–189. Haugtvedt, C. P., & Wegener, D. T. (1994). Message order effects in persuasion: An attitude strength perspective. Journal of Consumer Research, 21, 205–218. Hovland, C. I., & Mandell, W. (1957). Is there a ‘Law of Primacy’ in persuasion? In C. I. Hovland (Ed.), The order of presentation in persuasion (pp. 1–22). New Haven, CT: Yale University Press. Knower, F. H. (1936). Experimental studies of changes in attitude: A study of the effect of printed argument on changes in attitude. Journal of Abnormal and Social Psychology, 17, 315–347. Kraus, S. J. (1995). Attitudes and the prediction of behavior: A meta-analysis of the empirical literature. Personality and Social Psychology Bulletin, 21, 58–75. Lana, R. E. (1961). Familiarity and the order of presentation of persuasive communications. Journal of Abnormal and Social Psychology, 62, 573–577. Lana, R. E. (1963). Interest, media, and order effects in persuasive communications. Journal of Psychology, 56, 9–13. Lund, F. (1925). The psychology of belief: The law of primacy in persuasion. Journal of Abnormal and Social Psychology, 20, 183–191. Maio, G. R., Bell, D. W., & Esses, V. M. (1996). Ambivalence and persuasion: The processing of messages about immigrant groups. Journal of Experimental Social Psychology, 32, 513–536. Martin, R., & Hewstone, M. (2008). Majority versus minority influence, message processing, and attitude change: The source-context-elaboration model. In M. P. Zanna (Ed.), Advances in experimental social psychology, 40, 237–326. New York: Academic Press. Moore, D. J., & Reardon, R. (1987). Source magnification: The role of multiple sources in the processing of advertising appeals. Journal of Marketing Research, 24, 412–417. Olson, J. M., & Zanna, M. P. (1993). Attitudes and attitude change. Annual Review of Psychology, 44, 117–154. Pechmann, C., & Stewart, D. W. (1989). Advertising repetition: A critical review of wearin and wearout. Current Issues and Research in Advertising, 12, 285–330. Petty, R. E., & Cacioppo, J. T. (1986). Communication and persuasion: Central and peripheral routes to attitude change. New York: Springer/Verlag. Petty, R. E., Cacioppo, J. T., & Haugtvedt, C. P. (1992). Involvement and persuasion: An appreciative look at the Sherifs’ contribution to the study of self-relevance and attitude change. In D. Granberg & G. Sarup (Eds.), Social judgment and intergroup relations: Essays in honor of Muzifer Sherif (pp. 147–175). New York: Springer-Verlag. Petty, R. E., Cacioppo, J. T., & Schumann, D. (1983). Central and peripheral routes to advertising effectiveness: The moderating role of involvement. Journal of Consumer Research, 10, 135–146. Petty, R. E., Haugtvedt, C. P., & Smith, S. M. (1995). Elaboration as a determinant of attitude strength: Creating attitudes that are persistent, resistant, and predictive of behavior. In R. E. Petty & J. A. Krosnick (Eds.), Attitude strength: Antecedents and consequences (pp. 93–130). Mahwah, NJ: Erlbaum. Petty, R. E., & Wegener, D. T. (1998). Attitude change: Multiple roles for persuasion variables. In D. Gilbert, S. Fiske, & G. Lindzey (Eds.), The handbook of social psychology (4th ed., Vol. 1, pp. 323–390). New York: McGraw-Hill. Petty, R. E., & Wegener, D. T. (1999). The Elaboration Likelihood Model: Current status and controversies. In S. Chaiken & Y. Trope (Eds.), Dual process theories in social psychology (pp. 41–72). New York: Guilford Press. Petty, R. E., Wells, G. L., & Brock, T. C. (1976). Distraction can enhance or reduce yielding to propaganda: Thought disruption versus effort justification. Journal of Personality and Social Psychology, 34, 874–884. Petty, R. E., Tormala, Z., Hawkins, C., & Wegener, D. T. (2001). Motivation to think and order effects in persuasion: The moderating role of chunking. Personality and Social Psychology Bulletin, 27, 332–344. Shestowsky, D., Wegener, D. T., & Fabrigar, L. R. (1998). Need for cognition and interpersonal influence: Individual differences in impact on dyadic decisions. Journal of Personality and Social Psychology, 74, 1317–1328. Tobin, S. J., & Raymundo, M. M. (2009). Persuasion by causal arguments: The motivating role of perceived causal expertise. Social Cognition, 27, 105–127.

A Social Psychological Perspective  •  95 van Harreveld, F., van der Pligt, J., & de Liver, Y. (2009). The agony of ambivalence and ways to resolve it: Introducing the MAID model. Personality and Social Psychology Review, 13, 45–61. Wegener, D. T., Clark, J. K., & Petty, R. E. (2006). Not all stereotyping is created equal: Differential consequences of thoughtful versus non-thoughtful stereotyping. Journal of Personality and Social Psychology, 90, 42–59. Wegener, D. T., Clark, J. K., & Petty, R. E. (in press). Cognitive and metacognitive processes in attitude formation and change. In D. Albarracín & B. T. Johnson (Eds.), The handbook of attitudes (2nd ed.). New York: Psychology Press. Wegener, D. T., Petty, R. E., Smoak, N. D., & Fabrigar, L. R. (2004). Multiple routes to resisting attitude change. In E. S. Knowles & J. A. Linn (Eds.), Resistance and persuasion (pp. 13–38). Mahwah, NJ: Erlbaum. Wood, W., Rhodes, N., & Biek, M. (1995). Working knowledge and attitude strength: An information processing analysis. In R. E. Petty & J. A. Krosnick (Eds.), Attitude strength: Antecedents and consequences (pp. 283–313). Mahwah, NJ: Erlbaum. Worth, L. T., & Mackie, D. M. (1987). Cognitive mediation of positive affect in persuasion. Social Cognition, 5, 76–94.

Section II

Individual Differences, Cognitive Mechanisms, and Contextual Factors in Multiple Source Use

6

INDIVIDUAL DIFFERENCES IN MULTIPLE DOCUMENT COMPREHENSION Sarit Barzilai university of haifa, israel

Helge I. Strømsø university of oslo, norway

Practices of reading, interpreting, evaluating, and integrating multiple documents have formerly been largely the purview of specialist and expert communities (Goldman, 2015). The widespread use of the Internet, with its multifarious sources, has turned these practices into a basic prerequisite for participation in modern knowledge societies (Alexander & DRLRL, 2012; Britt, Richter, & Rouet, 2014), creating new challenges as people with diverse personal, academic, and socio-cultural backgrounds need to develop the complex skills necessary for comprehending multiple documents (MDs). Hence, the aim of this chapter is to review current research on individual differences in MD comprehension and to discuss the implications of these differences for educational research and practice. In this chapter, we use the term document to refer to an artifact that conveys information to readers. However, the present review focuses primarily on textual (printed and digital) documents rather than multimedia documents such as pictures and videos, as the latter deserve attention that exceeds the scope of this chapter. Documents are defined herein as including a source component and a content component (Rouet, 2006). The term source is used to refer to the origin of the text and the circumstances of its production such as author identity, publication venue, time of publication, and more (Bromme, Stadtler, & Scharrer, this volume). We use the term MD comprehension to refer broadly to meaning-making with MDs, including task interpretation, document selection, sourcing, content comprehension and analysis, corroboration, integration, and production (Goldman, Lawless, & Manning, 2013; Rouet & Britt, 2011). A very large body of research has been devoted to examining the role of individual differences in reading and reading comprehension (Afflerbach, 2016). Clearly, any

99

100  •  Barzilai and Strømsø

factor that impacts reading of single documents could also impact the reading of MDs. However, in this chapter, our focus is on individual differences that particularly matter to comprehension of MDs. Our identification of these differences is grounded in theoretical models of MD comprehension and extant empirical findings. To begin, we briefly discuss the ways in which MD comprehension extends single document comprehension. In what follows, we review empirical research on cognitive, metacognitive, motivational-affective, and socio-cultural differences in MD comprehension. We then discuss how these individual differences interact with each other and with task contexts, reflect on their development, and suggest educational implications and future research directions.

THEORETICAL BACKGROUND Models of MD comprehension describe how readers construct coherent mental representations based on documents that present diverse accounts regarding a particular issue (Bråten, Britt, Strømsø, & Rouet, 2011; Britt & Rouet, 2012). These mental representations serve as a basis for creating products that address task requirements, such as written explanations and arguments (Rouet & Britt, 2011). The departure point of MD comprehension models is that the multiplicity of documents requires representing not only the contents of the documents but also the documents themselves as authored entities (Britt, Rouet, & Braasch, 2013; Perfetti, Rouet, & Britt, 1999). Knowing who wrote and published the documents can help readers resolve inconsistencies and conflicts in the information they read (Stadtler & Bromme, 2014). A central model of MD comprehension is the Documents Model Framework (DMF) by Britt, Rouet, and their colleagues (Bråten & Braasch, this volume; Britt & Rouet, 2012, this volume; Perfetti et  al., 1999; Rouet, 2006). In a nutshell, this framework posits that to construct a coherent representation of MDs, good readers construct a representation of the contents of the documents (integrated mental model), including identification of inconsistencies, as well as representations of the sources of these documents and how these sources are related to each other (intertext model). A documents model is formed when readers connect the integrated mental model and the intertext model by forming links between sources and their contents, that is, when readers track who said what and use this information to interpret and evaluate document content. The Multiple-Document Task-based Relevance Assessment and Content Extraction (MD-TRACE) Model describes how good readers go about the task of constructing a documents model (Rouet & Britt, 2011). In brief, this model describes five main processes that unfold in an interactive and iterative manner: (1) constructing and updating a model of the task and its goals; (2) assessing information needs; (3) accessing, assessing, processing, and integrating information within and across documents in order to construct and update a documents model; (4) creating and updating a task product; and (5) assessing whether the product meets the goals of the task. More recently, Britt, Rouet, and colleagues proposed the RESOLV model, according to which readers initially form a model of the reading context that informs readers’ interpretation of the task (Britt, Rouet, & Durik, this volume; Rouet et al., 2017). The aforementioned processes are shaped both by external resources, such as task specifications and the documents themselves, and by readers’ internal

Individual Differences  •  101

resources, such as their prior domain knowledge, strategy knowledge, reading skills, and self-regulation skills (Rouet & Britt, 2011; Rouet et al., 2017). Thus, individual differences may play an important role in how processing of MDs unfolds. This assumption is supported by empirical studies that have documented substantial differences in MD comprehension between experts and novices (e.g., von der Mühlen, Richter, Schmid, Schmidt, & Berthold, 2015; Wineburg, 1991) and among novices (e.g., Barzilai, Tzadok, & Eshet-Alkalai, 2015; Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012). These differences in MD comprehension may be explained by multiple interacting factors. In the following sections, we address key individual differences that have been found to impact MD comprehension. These are grouped in four clusters: cognitive, metacognitive, motivational-affective, and socio-cultural differences.

COGNITIVE DIFFERENCES In this section we address several cognitive differences that have been found to play a role in MD comprehension: reading fluency, working memory, prior knowledge, prior beliefs, and strategic processing. Reading Fluency A relationship between word recognition and text comprehension has been confirmed in numerous studies (Catts, Herrera, Nielsen, & Bridges, 2015). It is, however, assumed that word recognition becomes automatic over school grades, and that increased reading fluency allows more cognitive resources to be allocated to comprehension (Kuhn & Stahl, 2003). Few studies have examined the role of word recognition in MD comprehension, probably because most of those studies include students at educational levels presupposing fluent reading skills. Indeed, in two studies no relationship was found among upper-secondary and university students (Braasch, Bråten, Strømsø, & Anmarkrud, 2014; Strømsø, Bråten, & Samuelstuen, 2008). However, word recognition strongly predicted MD comprehension among tenth graders (Bråten, Ferguson, Anmarkrud, & Strømsø, 2013a). Thus, the complexity of reading MDs may require higher levels of reading fluency than simpler reading tasks. Working Memory An important component in text comprehension is readers’ identification of semantic connections between different pieces of information in the text, and also between that information and readers’ prior knowledge. Thus, various pieces of information need to be stored in long-term memory, retrieved, and connected. These processes are assumed to take place in readers’ working memory. Readers’ capacity to handle such processes has been demonstrated to correlate with text comprehension. There is general agreement that the correlation between working memory and text comprehension increases when readers are challenged by complex textual tasks and materials, such as ambiguous information that requires consideration of alternative explanations or texts that require integration of distant pieces of information (e.g., Just & Carpenter, 1992; Unsworth & Engle, 2007).

102  •  Barzilai and Strømsø

Features of texts that may increase working memory demands may be even more prominent in MDs. Boundaries and distance between related information units may represent a challenge, as well as possible dissimilarity or ambiguity in terminology across texts, and a greater amount of distracting or redundant information. So far, only a couple of studies have found that working memory is related to MD comprehension. Banas and Sanchez (2012) reported that working memory capacity was positively related to undergraduates’ construction of global cross-textual connections across a set of documents. Likewise, Braasch et al. (2014) found that upper-secondary students’ working memory capacity predicted their intertextual comprehension of a set of MDs. Thus, these studies indicate that working memory capacity warrants more attention in studies on MD comprehension. Prior Knowledge Contemporary models of text comprehension all include prior knowledge as a critical factor in readers’ construction of meaning from text (McNamara & Magliano, 2009). Several studies have also documented relationships between prior knowledge and specific aspects of MD comprehension (e.g., Bråten, Anmarkrud, Brandmo, & Strømsø, 2014; Bråten & Strømsø, 2010b; Gil, Bråten, Vidal-Abarca, & Strømsø, 2010; Le Bigot & Rouet, 2007). Whether prior knowledge, in the sense of relevant topic knowledge, is of particular importance to MD comprehension is less certain. It is, however, plausible that the potential lack of coherence in a set of MDs will demand more from readers, in terms of prior knowledge, than single documents intentionally authored to be coherent. Earlier studies on interaction between prior knowledge and text coherence indicate this to be the case in single texts (McNamara, Kintsch, Songer, & Kintsch, 1996), whereas later studies have demonstrated that prior topic knowledge predicts readers’ ability to make inferences across MDs (e.g., Bråten et  al., 2014; Strømsø & Bråten, 2009; Strømsø et al., 2008). Disciplinary expertise implies a competence broader than topic knowledge. In this context, such expertise could include knowledge about contextual factors that are helpful in constructing an intertext model, such as characteristics of authors, publication channels, and text genres. Disciplinary expertise has indeed been demonstrated to inform and enhance readers’ MD comprehension (e.g., Rouet, Favart, Britt, & Perfetti, 1997; von der Mühlen et al., 2015; Wineburg, 1991, 1998). Prior Beliefs and Attitudes Although prior knowledge is vital to readers’ understanding of new information, it is also important to note that incorrect prior knowledge and commonsense beliefs may hinder acquisition of new knowledge (Kendeou & O’Brien, 2016; Sinatra, Kienhues, & Hofer, 2014). Prior beliefs and attitudes have been found to interfere with MD comprehension in several studies, particularly when students read contradicting MDs (e.g., Kobayashi, 2010, 2014; Richter & Maier, 2017; van Strien, Brand-Gruwel, & Boshuizen, 2014; van Strien, Kammerer, Brand-Gruwel, & Boshuizen, 2016). For example, in two studies, van Strien et al. had students read a set of websites presenting conflicting information on controversial topics. Participants holding strong prior

Individual Differences  •  103

attitudes toward the topics wrote more belief-consistent essays after reading and also included more information not presented in the reading materials than other participants. However, although readers’ understanding of the content may be biased by prior beliefs, these beliefs do not necessarily impair their memory for content. This was demonstrated by Maier and Richter (2013) who found that readers’ understanding of a controversy was biased toward belief-consistent texts, whereas their memory for texts’ content was better for belief-inconsistent texts. Hence, belief-inconsistent information seems to be attended to and remembered, but might not be integrated into a coherent representation of the texts. Strategic Processing The task of reading MDs will often be more complex than single-text reading and hence demands increased strategic effort. The nature of strategic reading will depend on the reader, the reading task, and the reading material. However, when a reader’s goal is to achieve a coherent understanding of a set of texts, results from several thinkaloud studies indicate that better readers employ linking strategies such as comparing, contrasting, interrelating, and corroborating (Anmarkrud, Bråten, & Strømsø, 2014; Cho, 2014; Goldman et al., 2012; Strømsø, Bråten, & Samuelstuen, 2003; Wolfe & Goldman, 2005). For example, Wolfe and Goldman (2005) found that sixth graders’ self-explanation inferences within and across two historical documents predicted their written explanations of the event described in the two texts. Similarly, Anmarkrud et al. (2014) found a positive relationship between undergraduates’ use of linking strategies and argumentative reasoning in essays. Results from such thinkaloud studies have been confirmed in studies on note-taking while reading multiple texts (Hagen, Braasch, & Bråten, 2014; Kobayashi, 2009), and through self-report inventories (Bråten et al., 2014; Bråten & Strømsø, 2011). Individual differences have also been found in sourcing processes. In a now classic study, Wineburg (1991) found differences in the ways in which novices and experts used information about documents, such as authorship and genre, to interpret and evaluate documents’ content. More recently, Barzilai et al. (2015) identified differences in sourcing patterns among university students who read conflicting information sources. They found a positive relation between sourcing while reading and subsequent argumentation. In sum, individual differences in strategic processing, particularly sourcing, intertextual linking, elaboration, and self-explanation, appear to play an important role in successful MD comprehension.

METACOGNITIVE DIFFERENCES Metacognition is often defined as cognition about cognition (Flavell, 1979). Two main branches of metacognition that have been studied in the context of MD comprehension are metacognitive skills, i.e., regulation of cognition, and metacognitive knowledge, i.e., knowledge of cognition. Key areas of metacognitive knowledge that have been found to contribute to MD comprehension are readers’ epistemic beliefs or understandings about the nature of knowledge and knowing, and their knowledge of epistemic criteria and strategies.

104  •  Barzilai and Strømsø

Epistemic Beliefs and Understandings About the Nature of Knowledge and Knowing Because MDs may offer different accounts of the same phenomenon or event, dealing with such documents involves assumptions about why differences between accounts occur and if and how such differences can be reconciled. Multiple studies involving school and university students have found that beliefs that knowledge is tentative and complex are related to better comprehension of MDs concerning complex controversial topics (e.g., Barzilai & Zohar, 2012; Bråten & Strømsø, 2010a; Rukavina & Daneman, 1996). Awareness that problems have multiple aspects, that there can be more than one possible explanation and solution, and that knowledge is uncertain and evolving may lead readers to pay greater attention to multiple viewpoints on the issue and to define the task as one that requires integration of information from diverse sources (Barzilai & Eshet-Alkalai, 2015; Bråten et al., 2011). In contrast, beliefs that the problem has a single correct answer might lead readers to view the task as one that requires identifying the correct answer or the correct document and hence to make less of an effort to work toward understanding differences between texts and forming connections among them. Beliefs about the justification of knowledge also come into play in MD comprehension. Specifically, beliefs that knowledge should be justified through reason, inquiry, and corroboration by multiple sources have been found to positively predict MD comprehension (e.g., Bråten et al., 2014; Bråten, Ferguson, Strømsø, & Anmarkrud, 2013b; Kammerer, Bråten, Gerjets, & Strømsø, 2013). Such justification beliefs might also predict trust in particular types of sources, such as research-based sources, and reliance on author and content evaluation criteria (Strømsø, Bråten, & Britt, 2011). In contrast, beliefs that knowledge is personally justified by readers’ personal views have been found to be negative predictors of MD comprehension (Barzilai & Eshet-Alkalai, 2015; Bråten et al., 2013b). Relatedly, beliefs that the source of knowledge is personal have been found to be less efficacious for MD comprehension than beliefs that knowledge is transmitted by expert authorities (Bråten, Strømsø, & Samuelstuen, 2008; Strømsø et al., 2008). Overreliance on personal opinions and understandings as the source of knowledge and justification may detract from attention to external sources of knowledge and consideration of authors’ viewpoints and arguments (Barzilai et al., 2015; Bråten et al., 2011). A key issue emerging from the body of research on the relation between readers’ epistemic beliefs and understandings and MD comprehension is that this relation depends on the nature of the documents and the task (Barzilai & Zohar, 2012; Bråten & Strømsø, 2010a; Strømsø et  al., 2011). Sophisticated epistemic beliefs are characterized by adaptivity to task contexts and allow for context-sensitive judgments (Bromme, Pieschl, & Stahl, 2010; Elby & Hammer, 2001). In MD comprehension studies, participants typically need to make sense of inconsistent accounts regarding complex controversial topics. In such tasks, views of knowledge as complex, evolving, and justified by multiple internal and external sources may help readers better apprehend the complexity of the task and respond adaptively to its demands (Barzilai & Eshet-Alkalai, 2015; Bromme et al., 2010; Strømsø et al., 2011). Knowledge of Epistemic Criteria and Standards Epistemic criteria and standards enable evaluating the quality and reliability of one’s own or others’ knowledge representations (Chinn, Rinehart, & Buckland, 2014).

Individual Differences  •  105

Studies have documented differences among students in knowledge of criteria for evaluating websites (Barzilai & Zohar, 2012; Walraven, Brand-Gruwel, & Boshuizen, 2009). For example, in Walraven et al.’s (2009) study, only one half of the ninth-grade focus groups that were interviewed mentioned author identity as a criterion for evaluating websites. These studies also suggest that metacognitive knowledge of evaluation criteria is related to applied use of these criteria, and yet that the application of criteria is sensitive to task and text features (Barzilai & Zohar, 2012; Walraven et al., 2009). The value of attending to students’ knowledge of evaluation criteria, and addressing gaps in this knowledge, is demonstrated by the positive outcomes of interventions in which students are taught about the nature and importance of these criteria (e.g., MacedoRouet, Braasch, Britt, & Rouet, 2013; Mason, Junyent, & Tornatora, 2014; Stadtler, Scharrer, Macedo-Rouet, Rouet, & Bromme, 2016; Wiley et al., 2009). Likewise, readers’ knowledge of integration criteria could inform their evaluation of integrative task products (Bråten et al., 2011). However, more research is needed to examine readers’ integration criteria. Knowledge of Epistemic Strategies An early study by Englert et al. (1988) found that high-achieving students had greater metacognitive knowledge about strategies for integrating ideas from MDs, such as using common categories to organize information across documents, compared to students diagnosed with learning disabilities, and that knowledge of these strategies was positively correlated with writing performance. Similarly, in two studies, Barzilai and her colleagues documented positive relations between students’ knowledge about epistemic strategies for integrating information from MDs and their integration performance (Barzilai & Ka’adan, 2016; Barzilai & Zohar, 2012). A positive association has also been found between knowledge of website trustworthiness evaluation strategies and evaluation performance (Barzilai & Zohar, 2012). Indeed, teaching strategies for evaluating and integrating MDs has been found to result in better MD comprehension (e.g., Barzilai & Ka’adan, 2016; Braasch, Bråten, Strømsø, Anmarkrud, & Ferguson, 2013; De La Paz & Felton, 2010; Maier & Richter, 2014). All in all, these findings indicate that MD comprehension benefits from metacognitive knowledge of the cognitive processes and strategies entailed in MD comprehension (Maier & Richter, 2014). Metacognitive Skills Meaning-making from MDs involves multiple cognitive processes such as tracking and evaluating sources, forming connections between texts, monitoring comprehension, deciding what to read next, what to reread, and how much weight to give various sources and claims. Hence, successful MD comprehension benefits from metacognitive skills for effectively regulating these cognitive processes, such as planning, predicting, monitoring, and evaluation (Alexander & DRLRL, 2012; Brand-Gruwel, Wopereis, & Walraven, 2009; Cho, 2014; Coiro & Dobler, 2007). Think-aloud studies have found that individual differences in metacognitive skills are evident during MD comprehension. For example, Goldman et al. (2012) found that better learners engaged in more comprehension monitoring processes than poorer learners during an inquiry task, particularly when reading reliable websites compared to unreliable

106  •  Barzilai and Strømsø

ones. The authors observed that when self-explanation and comprehension monitoring were used in concert, they informed evaluation and navigation decisions and led to better management of comprehension efforts (Goldman et al., 2012).

MOTIVATIONAL AND AFFECTIVE DIFFERENCES Comprehension of content across MDs, frequently representing different perspectives, may be perceived by some readers as a challenging and effortful task. Consequently, in order to learn and use such skills and strategies, readers need to be adequately motivated. Research on reading motivation emphasizes the importance of readers’ beliefs, values, and goals to both reading development and comprehension (Alexander & DRLRL, 2012; Guthrie, McRae, & Klauda, 2007; Wigfield, Gladstone, & Turci, 2016). Yet so far, only a limited number of studies have examined the relations between MD comprehension and variables such as interest, self-beliefs, need for cognition, and affect. Interest Several studies have documented the impact of readers’ interest on MD comprehension (Bråten et al., 2014; Grossnickle, 2014; Strømsø & Bråten, 2009). Strømsø and Bråten (2009) measured upper-secondary students’ topic interest concerning climate change and had them read a set of seven partly contradictory texts on that topic. Topic interest predicted students’ scores only on the most demanding comprehension task, in which students were required to compare and integrate across texts, whereas memory for content or single-text comprehension were not related to interest. These results confirm the assumption that interest is more strongly related to readers’ comprehension of more demanding tasks than to simpler comprehension or memory tasks (Schiefele, 1999). In a later study, it was demonstrated that the effect of students’ domain interest on MD comprehension was mediated by the effort students invested in the reading task (Bråten et al., 2014). In that study, students’ situational interest during reading had an effect on their use of deep-level strategies but not on MD comprehension. Thus, students’ more stable interest in a topic or domain seemed to be more strongly related to MD comprehension than their situational interest. Self-Beliefs Students’ beliefs about their ability to accomplish different tasks have also been found to shape text comprehension (Wigfield et al., 2016). This was confirmed in a study in which tenth graders read MDs representing different perspectives on sun exposure and health (Bråten et al., 2013a). Participants’ self-efficacy beliefs regarding their capability to understand what they read about science positively predicted MD comprehension. Additionally, students’ implicit beliefs about the nature of intelligence have been demonstrated to predict MD comprehension, with beliefs in intelligence as malleable positively predicting students’ intertextual inferences (Braasch et al., 2014). Students’ self-beliefs are partly the result of prior learning experiences and feedback from significant others (Bandura, 1997). Thus, one would expect positive feedback to potentially affect both reading behaviors and outcomes. Indeed, Maier and Richter (2014) found that positive feedback on undergraduates’ reading skills increased

Individual Differences  •  107

motivation to use demanding reading strategies while reading conflicting texts and reduced their tendency to favor belief-consistent claims. Need for Cognition Students’ need for cognition (NFC)—that is, their self-reported reaction to demands for effortful cognitive processing—might impact MD comprehension as well. In one study, Winter and Krämer (2012) found that greater NFC led to a stronger preference for selecting two-sided over one-sided articles. Additionally, Bråten et al. (2014) documented a direct effect of NFC on participants’ use of adaptive reading strategies and an indirect effect on MD comprehension. Affect Recently, several studies demonstrated how emotions, such as curiosity, anxiety, confusion, surprise, and enjoyment, come into play in reading and learning from MDs (Grossnickle, 2014; Muis et al., 2015; Trevors, Muis, Pekrun, Sinatra, & Muijselaar, 2016). Such emotions have been labeled epistemic emotions when they concern cognitive activities and knowledge construction. Grossnickle (2014) found that undergraduates enjoyed using multiple digital sources, but did not find a relationship between students’ curiosity and time on task. However, Muis et al. (2015) and Trevors et al. (2016) showed that positive emotions predicted learning from MDs. In summary, readers’ topic interest, self-beliefs, and emotions may guide both processing of MDs and their comprehension. List and Alexander (2017) proposed the Cognitive Affective Engagement Model (CAEM), which integrates the cognitive and motivational-affective aspects of engagement with multiple texts. However, more research is needed to examine how motivational and emotional differences impact MD comprehension.

SOCIO-CULTURAL DIFFERENCES Many individual differences reflect the interaction of primary cognitive characteristics and broader socio-cultural contexts, such as families, classrooms, school, cultural communities, ethnicity, or gender (Loughlin & Alexander, 2016). Several studies have addressed the impact of socio-cultural identities, socio-economic status (SES), and gender on MD comprehension. Socio-Cultural Identities Social and cultural identities and commitments can shape the prior ideas and beliefs that students bring to the reading of texts, and consequently, their understanding and evaluation of these texts. For example, VanSledright (2016) described how AfricanAmerican and European-American students may bring different beliefs about American history and values to the interpretation of a set of historical documents. Socio-cultural differences may also impact the ways in which readers evaluate sources. For example, students from minority groups may distrust textbooks’ dominant narratives and prefer participant accounts or oral histories (Goldberg & Ron, 2014; VanSledright, 2016).

108  •  Barzilai and Strømsø

Students’ socio-cultural identities could also lead them to disparage the reliability of historical sources that present their group in unfavorable ways (Goldberg, 2013). In a similar vein, socio-cultural identities may shape ideas and beliefs about science and, consequently, the reading and evaluation of scientific documents (Sinatra et al., 2014). However, socio-cultural commitments are not only obstacles to reasoning about sources; sometimes, they may spur deeper engagement in sourcing, corroboration, and argumentation from sources (Goldberg, 2013; Goldberg, Schwarz, & Porat, 2011). Epistemic beliefs and understandings can reflect socio-cultural values, norms, and assumptions (Hofer, 2008; Tabak & Weinstock, 2008). Hence, readers’ sociocultural backgrounds might impact MD comprehension also via their influence on their epistemic assumptions. Gottlieb and Wineburg (2012) examined how religious and non-religious historians read sets of religiously charged documents. They found that the religious historians drew on multiple frameworks of epistemological assumptions and practices, related both to their professional and their religious identities, and coordinated them while interpreting the documents. More recently, Strømsø and his colleagues (2016) suggested that the role of epistemic beliefs in MD comprehension may depend on readers’ socio-cultural backgrounds. Specifically, they found that the effects of some justification beliefs were apparent only among ethnic minority students (Strømsø, Bråten, Anmarkrud, & Ferguson, 2016). Socio-Economic Status In the PISA 2012 survey, SES accounted for 12% of the variance in digital literacy tasks that involved navigating, evaluating, and integrating multiple texts (OECD, 2015b). Most of this variance was due to the indirect effect of SES on digital literacy through print literacy, yet a small direct effect of SES on digital literacy was also observed, which may be attributed to differences in navigation and evaluation skills. Similarly, Leu et  al. (2012) found that students in a low-SES school district had lower online reading and comprehension scores than students in a high-SES school district, even when controlling for offline reading ability. These differences may be due in part to more limited access to computers and the Internet among the economically disadvantaged (Hargittai & Hsieh, 2013; Leu et al., 2012; OECD, 2015b). However, even when economically disadvantaged users have Internet access, they have been found to be less likely to engage in activities such as searching for information or reading online news, compared to economically advantaged users (Hargittai & Hsieh, 2013; OECD, 2015a). Thus, low SES may be associated with fewer opportunities for developing skills for reading and understanding multiple digital documents. This emerging digital divide could limit opportunities for participating in and benefiting from connected knowledge societies and hence merits further attention. Gender According to national and international assessments, girls typically outperform boys in reading (e.g., OECD, 2016). Advantages in reading may be expected to translate to advantages in MD comprehension. However, studies that have directly examined the relation between gender and MD comprehension report mixed results, with females sometimes scoring higher (Bråten & Strømsø, 2006; Forzani, 2016), lower (Barzilai

Individual Differences  •  109

et al., 2015; Bråten & Strømsø, 2010a), or no different (Strømsø, Bråten, & Britt, 2010) than males. Beyond reading ability, gender differences in topic-specific prior knowledge and interest or in online experiences and attitudes might potentially be reflected in MD comprehension. For example, women have been found to score lower than men on self-reported Internet skills which, in turn, predict engagement in online activities (e.g., Hargittai & Hsieh, 2013). Among middle-school students, girls have expressed more positive attitudes than boys toward academic digital reading, yet this pattern was reversed with regards to recreational digital reading (McKenna, Conradi, Lawrence, Jang, & Meyer, 2012). However, whether and how gender differences impact MD comprehension is still unclear.

INDIVIDUAL DIFFERENCES INTERACT WITH EACH OTHER AND WITH THE TASK CONTEXT Interactions between different variables have been demonstrated in several recent MD comprehension studies. Grossnickle’s (2014) study of students’ use of digital MDs revealed an interaction between interest and prior knowledge, indicating a positive relationship between time on task and prior knowledge for highly interested students and a negative relationship for students with low interest. Thus, high prior knowledge combined with low interest might evoke a feeling of sufficient knowledge. Path analytic approaches also indicate that individual difference variables may indirectly influence MD comprehension. For example, Bråten et al. (2014) and Muis et al. (2015) documented that several individual difference variables indirectly informed MD comprehension through their influence on students’ strategic approach and effort. Other studies have revealed interactions between the reading task and individual differences, such as prior knowledge, epistemic beliefs, and topic beliefs. Several studies have compared the outcomes of reading MDs in order to write either an argumentative or a summary essay. Gil et al. (2010) found that high prior knowledge students demonstrated better MD comprehension than low prior knowledge students in the argumentative, but not in the summary, condition. However, Bråten and Strømsø (2010b) showed that only students believing knowledge about climate change to be tentative and evolving, rather than certain, profited from the argumentation task. In a later study, Maier and Richter (2016) reported an interaction of topic beliefs and task such that students in the argumentation condition evaluated belief-inconsistent content more thoroughly than those in the summary condition, and thus constructed a more balanced mental representation of the documents. Maier and Richter (2013) found that the order in which texts were read also interacted with prior beliefs: When belief-consistent and inconsistent texts were presented alternatingly, instead of block-by-block, students’ understanding of the content was less biased toward their prior beliefs. In summary, preliminary studies indicate the combined effects on MD comprehension of prior knowledge, beliefs, motivation, and affect, and suggest the contingency of these effects on task contexts.

DEVELOPMENT OF INDIVIDUAL DIFFERENCES Reading skills, disciplinary knowledge, metacognition, and epistemic thinking develop over age and education (e.g., Alexander, 2005; Kuhn, Cheney, & Weinstock, 2000;

110  •  Barzilai and Strømsø

Veenman, 2016). The paucity of cross-sectional and longitudinal studies of MD comprehension makes it currently difficult to clarify how the development of individual differences relates to growth in MD comprehension. Nonetheless, some studies suggest that developmental differences might impact MD comprehension. Readers’ developing disciplinary knowledge may shape MD comprehension. Keil and Kominsky (2013) reported that middle-school students found it more difficult to identify the disciplinary relatedness of search results, compared to high school students and adults, suggesting that lower disciplinary content knowledge in middle school may underlie differences in search competence. The development of strategic processing capabilities appears to be another important driver of growth in MD comprehension. Salmerón et al. (2016) found that primary school students were less likely to combine author expertise and evidence quality cues when they evaluated conflicting forum messages, compared to university students. The younger students tended to prefer expert authors regardless of the quality of their evidence. Developmental differences in MD comprehension strategies were also documented by Brand-Gruwel et al. (2009), who found that, during online information problem solving, secondary school students searched, scanned, and processed information in a more iterative and rapid manner, compared to adult students. The secondary school students also devoted less time to organizing and presenting information and engaged in less regulation of their problem solving. Beyond age and education level, students’ interactions with MDs may contribute to growth in their strategic capabilities. Strømsø et al. (2003) examined changes in the ways in which law students read multiple texts over the course of a semester. They found that, as the semester progressed, students focused less on the current documents and more on external documents that were related to the current documents. That is, they gradually came to perceive the task less as reviewing single documents and more as integrating MDs. All in all, these studies point to the evolving nature of MD comprehension.

EDUCATIONAL IMPLICATIONS Given the multitude of texts surrounding students in modern societies, educational institutions and systems should address students’ capacities for critically dealing with the overflow of information. Reading MDs typically demands more of readers than single-text reading, hence individual differences in memory, reading skills, prior knowledge, prior beliefs, metacognition, and motivation may decrease readers’ capacity to deal with these complex reading materials. One promising approach to addressing these differences is by attending to task, text, and context features. Modifying texts to improve readability and reduce length can help struggling readers engage with MDs (Reisman, 2012a). Interventions among fourth and fifth graders indicate that young students can profit when materials, tasks, and instruction are adapted to their levels of reading skills and prior knowledge (Macedo-Rouet et al., 2013; VanSledright, 2002) and are aimed at stimulating interest and motivation (Guthrie et al., 2007). Inclusion of instructional and motivational supports in tasks involving MDs representing different perspectives can potentially reduce biased processing of documents (e.g., Goldberg et al., 2011; Maier & Richter, 2014). Finally, providing scaffolds that support coordination and regulation of document-based inquiry may help compensate for low metacognitive skillfulness (e.g., Zhang & Quintana, 2012).

Individual Differences  •  111

Another approach to addressing individual differences is to actively reduce gaps among readers. Following Wineburg (1991), several interventions have aimed to teach students epistemic strategies and criteria identified by experts in the discipline of history (e.g., Britt & Aglinskas, 2002; Nokes, Dole, & Hacker, 2007; Reisman, 2012b). Explicit strategy instruction, whether in the form of cognitive modeling or guided practice, seems to be important in the instructional design of programs demonstrating improved MD comprehension (e.g., Braasch et al., 2013; Mason et al., 2014; Reisman, 2012b; Wiley et al., 2009). The above studies show that MD interventions have much in common with more general reading comprehension programs (Duke & Pearson, 2008), although the nature of strategies and materials may differ. Furthermore, the development of students’ epistemic understandings is often emphasized more in MD interventions: It remains, however, to be seen to what extent addressing these understandings can promote MD comprehension.

FUTURE DIRECTIONS The studies reviewed in this chapter demonstrate the critical importance of individual differences in MD comprehension. However, in most of the areas surveyed, the understanding of the nature and the impact of these individual differences is still in its nascent phases. Additionally, there are individual differences that have yet to be explored. For example, individual differences in cognitive load during learning with MDs have so far not been studied thoroughly, despite the well-documented impact of cognitive load on learning with multimedia representations (Mayer, 2014). Thus, much more work is needed to promote better understanding of individual differences in MD comprehension. Beyond this call, the following specific issues could benefit from future research. First, there is a need to expand theories of MD comprehension to better address individual differences in MD comprehension. For example, the MD-TRACE model generally assumes that individual differences may impact processing of MDs (Rouet & Britt, 2011). However, it is unclear how these differences play out in the phases of MD comprehension; for example, how they relate to task definition, document processing, and creating and evaluating task products. Second, and relatedly, there is a need to better understand how individual differences in MD comprehension interact with tasks, texts, and contexts. Current studies suggest that such interactions are critical for understanding the impact of individual difference variables. However, there is need for more research examining the conditions under which individual differences affect MD comprehension. An important issue is studying how tasks, texts, and contexts can be designed to mitigate potentially detrimental effects of individual differences such as low prior knowledge, maladaptive epistemic beliefs, and entrenched topic beliefs. Third, we have little understanding of how the development of individual differences and the development of MD comprehension are related. However, current studies suggest that not only may the development of individual difference variables enhance MD comprehension, but also that engagement in MD comprehension may lead, under certain conditions, to changes in individual difference variables such as strategic processing capabilities (e.g., Strømsø et al., 2003) or epistemic understandings (e.g., Barzilai & Ka’adan, 2016). Hence, more research is needed to understand

112  •  Barzilai and Strømsø

the developmental links between the diverse individual differences reviewed in this chapter and MD comprehension.

REFERENCES Afflerbach, P. (Ed.). (2016). Handbook of individual differences in reading: Reader, text, and context. New York, NY: Routledge. Alexander, P. A. (2005). The path to competence: A lifespan developmental perspective on reading. Journal of Literacy Research, 37(4), 413–436. Alexander, P. A., & DRLRL. (2012). Reading into the future: Competence for the 21st century. Educational Psychologist, 47(4), 259–280. Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. Banas, S., & Sanchez, C. A. (2012). Working memory capacity and learning underlying conceptual relationships across multiple documents. Applied Cognitive Psychology, 26(4), 594–600. Bandura, A. (1997). Self-efficacy: The exercise of control. New York, NY: W. H. Freeman. Barzilai, S., & Eshet-Alkalai, Y. (2015). The role of epistemic perspectives in comprehension of multiple author viewpoints. Learning and Instruction, 36, 86–103. Barzilai, S., & Ka’adan, I. (2016). Learning to integrate divergent information sources: The interplay of epistemic cognition and epistemic metacognition. Metacognition and Learning. Advance online publication. Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online sources. Cognition and Instruction, 30(1), 39–85. Barzilai, S., Tzadok, E., & Eshet-Alkalai, Y. (2015). Sourcing while reading divergent expert accounts: Pathways from views of knowing to written argumentation. Instructional Science, 43(6), 737–766. Braasch, J. L. G., Bråten, I., Strømsø, H. I., & Anmarkrud, Ø. (2014). Incremental theories of intelligence predict multiple document comprehension. Learning and Individual Differences, 31, 11–20. Braasch, J. L. G., Bråten, I., Strømsø, H. I., Anmarkrud, Ø., & Ferguson, L. E. (2013). Promoting secondary school students’ evaluation of source features of multiple documents. Contemporary Educational Psychology, 38(3), 180–195. Brand-Gruwel, S., Wopereis, I., & Walraven, A. (2009). A descriptive model of information problem solving while using internet. Computers & Education, 53(4), 1207–1217. Bråten, I., & Strømsø, H. I. (2006). Effects of personal epistemology on the understanding of multiple texts. Reading Psychology, 27(5), 457–484. Bråten, I., & Strømsø, H. I. (2010a). Effects of task instruction and personal epistemology on the understanding of multiple texts about climate change. Discourse Processes, 47(1), 1–31. Bråten, I., & Strømsø, H. I. (2010b). When law students read multiple documents about global warming: Examining the role of topic-specific beliefs about the nature of knowledge and knowing. Instructional Science, 38(6), 635–657. Bråten, I., & Strømsø, H. I. (2011). Measuring strategic processing when students read multiple texts. Metacognition and Learning, 6(2), 111–130. Bråten, I., Strømsø, H. I., & Samuelstuen, M. S. (2008). Are sophisticated students always better? The role of topic-specific personal epistemology in the understanding of multiple expository texts. Contemporary Educational Psychology, 33(4), 814–840. Bråten, I., Anmarkrud, Ø., Brandmo, C., & Strømsø, H. I. (2014). Developing and testing a model of direct and indirect relationships between individual differences, processing, and multiple-text comprehension. Learning and Instruction, 30, 9–24. Bråten, I., Britt, M. A., Strømsø, H. I., & Rouet, J.-F. (2011). The role of epistemic beliefs in the comprehension of multiple expository texts: Toward an integrated model. Educational Psychologist, 46(1), 48–70. Bråten, I., Ferguson, L., Anmarkrud, Ø., & Strømsø, H. (2013a). Prediction of learning and comprehension when adolescents read multiple texts: The roles of word-level processing, strategic approach, and reading motivation. Reading and Writing, 26(3), 321–348. Bråten, I., Ferguson, L. E., Strømsø, H. I., & Anmarkrud, Ø. (2013b). Justification beliefs and multiple-documents comprehension. European Journal of Psychology of Education, 28(3), 879–902.

Individual Differences  •  113 Britt, M. A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20(4), 485–522. Britt, M. A., & Rouet, J.-F. (2012). Learning with multiple documents: Component skills and their acquisition. In J. R. Kirby & M. J. Lawson (Eds.), Enhancing the quality of learning: Dispositions, instruction, and learning processes (pp. 276–314). New York, NY: Cambridge University Press. Britt, M. A., Richter, T., & Rouet, J.-F. (2014). Scientific literacy: The role of goal-directed reading and evaluation in understanding scientific information. Educational Psychologist, 49(2), 104–122. Britt, M. A., Rouet, J.-F., & Braasch, J. L. G. (2013). Documents as entities: Extending the situation model theory of comprehension. In M. A. Britt, S. R. Goldman & J.-F. Rouet (Eds.), Reading: From words to multiple texts (pp. 160–179). New York, NY: Routledge. Bromme, R., Pieschl, S., & Stahl, E. (2010). Epistemological beliefs are standards for adaptive learning: A functional theory about epistemological beliefs and metacognition. Metacognition and Learning, 5(1), 7–26. Catts, H. W., Herrera, S., Nielsen, D. C., & Bridges, M. S. (2015). Early prediction of reading comprehension within the simple view framework. Reading and Writing, 28(9), 1407–1425. Chinn, C. A., Rinehart, R. W., & Buckland, L. A. (2014). Epistemic cognition and evaluating information: Applying the air model of epistemic cognition. In D. Rapp & J. Braasch (Eds.), Processing inaccurate information (pp. 425–454). Cambridge, MA: MIT Press. Cho, B.-Y. (2014). Competent adolescent readers’ use of internet reading strategies: A think-aloud study. Cognition and Instruction, 32(3), 253–289. Coiro, J., & Dobler, E. (2007). Exploring the online reading comprehension strategies used by sixth-grade skilled readers to search for and locate information on the internet. Reading Research Quarterly, 42(2), 214–257. De La Paz, S., & Felton, M. K. (2010). Reading and writing from multiple source documents in history: Effects of strategy instruction with low to average high school writers. Contemporary Educational Psychology, 35(3), 174–192. Duke, N. K., & Pearson, P. D. (2008). Effective practices for developing reading comprehension. The Journal of Education, 189(1/2), 107–122. Elby, A., & Hammer, D. (2001). On the substance of a sophisticated epistemology. Science Education, 85(5), 554–567. Englert, C. S., Raphael, T. E., Fear, K. L., & Anderson, L. M. (1988). Students’ metacognitive knowledge about how to write informational texts. Learning Disability Quarterly, 11(1), 18–46. Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34(10), 906–911. Forzani, E. (2016). Individual differences in evaluating the credibility of online information in science: Contributions of prior knowledge, gender, socioeconomic status, and offline reading ability. Unpublished doctoral dissertation, University of Connecticut, Storrs, CT. Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H. I. (2010). Summary versus argument tasks when working with multiple documents: Which is better for whom? Contemporary Educational Psychology, 35(3), 157–173. Goldberg, T. (2013). “It’s in my veins”: Identity and disciplinary practice in students’ discussions of a historical issue. Theory & Research in Social Education, 41(1), 33–64. Goldberg, T., & Ron, Y. (2014). “Look, each side says something different”: The impact of competing history teaching approaches on Jewish and Arab adolescents’ discussions of the Jewish–Arab conflict. Journal of Peace Education, 11(1), 1–29. Goldberg, T., Schwarz, B. B., & Porat, D. (2011). “Could they do it differently?”: Narrative and argumentative changes in students’ writing following discussion of “hot” historical issues. Cognition and Instruction, 29(2), 185–217. Goldman, S. R. (2015). Reading and the web: Broadening the need for complex comprehension In R. J. Spiro, M. DeSchryver, P. Morsink, M. S. Hagerman, & P. Thompson (Eds.), Reading at a crossroads? Disjunctures and continuities in current conceptions and practices (pp. 89–103). New York, NY: Routledge. Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47(4), 356–381. Goldman, S. R., Lawless, K., & Manning, F. (2013). Research and development of multiple source comprehension assessment. In M. A. Britt, S. R. Goldman, & J.-F. Rouet (Eds.), Reading: From words to multiple texts (pp. 160–179). New York, NY: Routledge.

114  •  Barzilai and Strømsø Gottlieb, E., & Wineburg, S. (2012). Between veritas and communitas: Epistemic switching in the reading of academic and sacred history. Journal of the Learning Sciences, 21(1), 84–129. Grossnickle, E. M. (2014). The expression and enactment of interest and curiosity in a multiple source use task. Unpublished doctoral dissertation, University of Maryland, College Park, MD. Guthrie, J. T., McRae, A., & Klauda, S. L. (2007). Contributions of concept-oriented reading instruction to knowledge about interventions for motivations in reading. Educational Psychologist, 42(4), 237–250. Hagen, Å. M., Braasch, J. L. G., & Bråten, I. (2014). Relationships between spontaneous note-taking, selfreported strategies and comprehension when reading multiple texts in different task conditions. Journal of Research in Reading, 37(1), 141–157. Hargittai, E., & Hsieh, Y. P. (2013). Digital inequality. In W. H. Dutton (Ed.), Oxford handbook of internet studies (pp. 129–150). Oxford, UK: Oxford University Press. Hofer, B. K. (2008). Personal epistemology and culture. In M. S. Khine (Ed.), Knowing, knowledge and beliefs: Epistemological studies across diverse cultures (pp. 3–22). New York, NY: Springer Science + Business Media. Just, M. A., & Carpenter, P. A. (1992). A capacity theory of comprehension: Individual differences in working memory. Psychological Review, 99(1), 122–149. Kammerer, Y., Bråten, I., Gerjets, P., & Strømsø, H. I. (2013). The role of internet-specific epistemic beliefs in laypersons’ source evaluations and decisions during web search on a medical issue. Computers in Human Behavior, 29(3), 1193–1203. Keil, F. C., & Kominsky, J. F. (2013). Missing links in middle school: Developing use of disciplinary relatedness in evaluating internet search results. PLOS ONE, 8(6), e67777. Kendeou, P., & O’Brien, E. J. (2016). Prior knowledge: Acquisition and revision. In P. Afflerbach (Ed.), Handbook of individual differences in reading: Reader, text, and context (pp.  151–163). New York, NY: Routledge. Kobayashi, K. (2009). Comprehension of relations among controversial texts: Effects of external strategy use. Instructional Science, 37(4), 311–324. Kobayashi, K. (2010). Strategic use of multiple texts for the evaluation of arguments. Reading Psychology, 31(2), 121–149. Kobayashi, K. (2014). Students’ consideration of source information during the reading of multiple texts and its effect on intertextual conflict resolution. Instructional Science, 42(2), 183–205. Kuhn, D., Cheney, R., & Weinstock, M. (2000). The development of epistemological understanding. Cognitive Development, 15(3), 309–328. Kuhn, M. R., & Stahl, S. A. (2003). Fluency: A review of developmental and remedial practices. Journal of Educational Psychology, 95(1), 3–21. Le Bigot, L., & Rouet, J.-F. (2007). The impact of presentation format, task assignment, and prior knowledge on students’ comprehension of multiple online documents. Journal of Literacy Research, 39(4), 445–470. Leu, D. J., Coiro, J., Kulikowich, J. M., Nell, S., Everett-Cacopardo, H., McVerry, G., . . . Burlingame, C. (2012). An initial study of online reading and internet research ability in rich and poor school districts: Do the rich get richer and the poor get poorer? Paper presented at the 2012 Annual Meeting of the American Educational Research Association, Vancouver, BC, Canada. List, A., & Alexander, P. A. (2017). Cognitive affective engagement model of multiple source use. Educational Psychologist. Advance online publication. Loughlin, S. M., & Alexander, P. A. (2016). Individual differences relations and interrelations: Reconciling issues of definition, dynamism, and development. In P. Afflerbach (Ed.), Handbook of individual differences in reading: Reader, text, and context (pp. 377–393). New York, NY: Routledge. Macedo-Rouet, M., Braasch, J. L. G., Britt, M. A., & Rouet, J.-F. (2013). Teaching fourth and fifth graders to evaluate information sources during text comprehension. Cognition and Instruction, 31(2), 204–226. Maier, J., & Richter, T. (2013). Text belief consistency effects in the comprehension of multiple texts with conflicting information. Cognition and Instruction, 31(2), 151–175. Maier, J., & Richter, T. (2014). Fostering multiple text comprehension: How metacognitive strategies and motivation moderate the text-belief consistency effect. Metacognition and Learning, 9(1), 51–74. Maier, J., & Richter, T. (2016). Effects of text-belief consistency and reading task on the strategic validation of multiple texts. European Journal of Psychology of Education, 31(4), 479–497. Mason, L., Junyent, A. A., & Tornatora, M. C. (2014). Epistemic evaluation and comprehension of web-source information on controversial science-related topics: Effects of a short-term instructional intervention. Computers & Education, 76, 143–157.

Individual Differences  •  115 Mayer, R. E. (2014). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), Cambridge handbook of multimedia learning, 2nd ed. (pp. 43–71). New York, NY: Cambridge University Press. McKenna, M. C., Conradi, K., Lawrence, C., Jang, B. G., & Meyer, J. P. (2012). Reading attitudes of middle school students: Results of a survey. Reading Research Quarterly, 47(3), 283–306. McNamara, D. S., & Magliano, J. (2009). Toward a comprehensive model of comprehension. Psychology of Learning and Motivation, 51, 297–384. McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14(1), 1–43. Muis, K. R., Pekrun, R., Sinatra, G. M., Azevedo, R., Trevors, G., Meier, E., & Heddy, B. C. (2015). The curious case of climate change: Testing a theoretical model of epistemic beliefs, epistemic emotions, and complex learning. Learning and Instruction, 39, 168–183. Nokes, J. D., Dole, J. A., & Hacker, D. J. (2007). Teaching high school students to use heuristics while reading historical texts. Journal of Educational Psychology, 99(3), 492–504. OECD. (2015a). Are there differences in how advantaged and disadvantaged students use the internet? PISA in focus. Paris, France: OECD. OECD. (2015b). Students, computers and learning: Making the connection. Paris, France: OECD. OECD. (2016). Pisa 2015 results (volume I): Excellence and equity in education. Paris, France: OECD. Perfetti, C. A., Rouet, J.-F., & Britt, M. A. (1999). Toward a theory of documents representation. In H. van Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 99–122). Mahwah, NJ: Lawrence Erlbaum Associates. Reisman, A. (2012a). The “document-based lesson”: Bringing disciplinary inquiry into high school history classrooms with adolescent struggling readers. Journal of Curriculum Studies, 44(2), 233–264. Reisman, A. (2012b). Reading like a historian: A document-based history curriculum intervention in urban high schools. Cognition and Instruction, 30(1), 86–112. Richter, T., & Maier, J. (2017). Comprehension of multiple documents with conflicting information: A two-step model of validation. Educational Psychologist. Advance online publication. Richter, T., & Schmid, S. (2010). Epistemological beliefs and epistemic strategies in self-regulated learning. Metacognition and Learning, 5(1), 47–65. Rouet, J.-F. (2006). The skills of document use: From text comprehension to web-based learning. Mahwah, NJ: Lawrence Erlbaum Associates. Rouet, J.-F., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. In M. T. McCrudden, J. P. Magliano & G. Schraw (Eds.), Text relevance and learning from text (pp. 19–52). Charlotte, NC: Information Age Publishing. Rouet, J.-F., Britt, M. A., & Durik, A. M. (2017). Resolv: Readers’ representation of reading contexts and tasks. Educational Psychologist. Advance online publication. Rouet, J.-F., Favart, M., Britt, M. A., & Perfetti, C. A. (1997). Studying and using multiple documents in history: Effects of discipline expertise. Cognition and Instruction, 15(1), 85–106. Rukavina, I., & Daneman, M. (1996). Integration and its effect on acquiring knowledge about competing scientific theories for text. Journal of Educational Psychology, 88(2), 272–287. Salmerón, L., Macedo-Rouet, M., & Rouet, J.-F. (2016). Multiple viewpoints increase students’ attention to source features in social question and answer forum messages. Journal of the Association for Information Science and Technology, 67(10), 2404–2419. Schiefele, U. (1999). Interest and learning from text. Scientific Studies of Reading, 3(3), 257–279. Sinatra, G. M., Kienhues, D., & Hofer, B. K. (2014). Addressing challenges to public understanding of science: Epistemic cognition, motivated reasoning, and conceptual change. Educational Psychologist, 49(2), 123–138. Stadtler, M., & Bromme, R. (2014). The content–source integration model: A taxonomic description of how readers comprehend conflicting scientific information. In D. N. Rapp & J. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 379–402). Cambridge, MA: MIT Press. Stadtler, M., Scharrer, L., Macedo-Rouet, M., Rouet, J.-F., & Bromme, R. (2016). Improving vocational students’ consideration of source information when deciding about science controversies. Reading and Writing, 29(4), 705–729. Strømsø, H. I., & Bråten, I. (2009). Beliefs about knowledge and knowing and multiple-text comprehension among upper secondary students. Educational Psychology, 29(4), 425–445.

116  •  Barzilai and Strømsø Strømsø, H. I., Bråten, I., & Britt, M. A. (2010). Reading multiple texts about climate change: The relationship between memory for sources and text comprehension. Learning and Instruction, 20(3), 192–204. Strømsø, H. I., Bråten, I., & Britt, M. A. (2011). Do students’ beliefs about knowledge and knowing predict their judgement of texts’ trustworthiness? Educational Psychology, 31(2), 177–206. Strømsø, H. I., Bråten, I., & Samuelstuen, M. S. (2003). Students’ strategic use of multiple sources during expository text reading: A longitudinal think-aloud study. Cognition and Instruction, 21(2), 113–147. Strømsø, H. I., Bråten, I., & Samuelstuen, M. S. (2008). Dimensions of topic-specific epistemological beliefs as predictors of multiple text understanding. Learning and Instruction, 18(6), 513–527. Strømsø, H. I., Bråten, I., Anmarkrud, Ø., & Ferguson, L. E. (2016). Beliefs about justification for knowing when ethnic majority and ethnic minority students read multiple conflicting documents. Educational Psychology, 36(4), 638–657. Tabak, I., & Weinstock, M. (2008). A sociocultural exploration of epistemological beliefs. In M. S. Khine (Ed.), Knowing, knowledge and beliefs: Epistemological studies across diverse cultures (pp. 177–195). New York, NY: Springer. Trevors, G. J., Muis, K. R., Pekrun, R., Sinatra, G. M., & Muijselaar, M. M. L. (2016). Exploring the relations between epistemic beliefs, emotions, and learning from texts. Contemporary Educational Psychology. Advance online publication. Unsworth, N., & Engle, R. W. (2007). The nature of individual differences in working memory capacity: Active maintenance in primary memory and controlled search from secondary memory. Psychological Review, 114(1), 104–132. van Strien, J. L. H., Brand-Gruwel, S., & Boshuizen, H. P. A. (2014). Dealing with conflicting information from multiple nonlinear texts: Effects of prior attitudes. Computers in Human Behavior, 32, 101–111. van Strien, J. L. H., Kammerer, Y., Brand-Gruwel, S., & Boshuizen, H. P. A. (2016). How attitude strength biases information processing and evaluation on the web. Computers in Human Behavior, 60, 245–252. VanSledright, B. A. (2002). Fifth graders investigating history in the classroom: Results from a researcherpractitioner design experiment. The Elementary School Journal, 103(2), 131–160. VanSledright, B. A. (2016). Individual differences in reading history. In P. Afflerbach (Ed.), Handbook of individual differences in reading: Reader, text, and context (pp. 245–258). New York, NY: Routledge. Veenman, M. V. J. (2016). Metacognition. In P. Afflerbach (Ed.), Handbook of individual differences in reading: Reader, text, and context (pp. 26–40). New York, NY: Routledge. von der Mühlen, S., Richter, T., Schmid, S., Schmidt, E., & Berthold, K. (2015). The use of source-related strategies in evaluating multiple psychology texts: A student–scientist comparison. Reading and Writing, 1–22. Walraven, A., Brand-Gruwel, S., & Boshuizen, H. P. A. (2009). How students evaluate information and sources when searching the world wide web for information. Computers & Education, 52(1), 234–246. Wigfield, A., Gladstone, J. R., & Turci, L. (2016). Beyond cognition: Reading motivation and reading comprehension. Child Development Perspectives, 10(3), 190–195. Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009). Source evaluation, comprehension, and learning in internet science inquiry tasks. American Educational Research Journal, 46(4), 1060–1106. Wineburg, S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83(1), 73–87. Wineburg, S. (1998). Reading Abraham Lincoln: An expert/expert study in the interpretation of historical texts. Cognitive Science, 22(3), 319–346. Winter, S., & Krämer, N. C. (2012). Selecting science information in web 2.0: How source cues, message sidedness, and need for cognition influence users’ exposure to blog posts. Journal of Computer-Mediated Communication, 18(1), 80–96. Wolfe, M. B. W., & Goldman, S. R. (2005). Relations between adolescents’ text processing and reasoning. Cognition and Instruction, 23(4), 467–502. Zhang, M., & Quintana, C. (2012). Scaffolding strategies for supporting middle school students’ online inquiry processes. Computers & Education, 58(1), 181–196.

7

POTENTIAL PROCESSING CHALLENGES OF INTERNET USE AMONG READERS WITH DYSLEXIA Øistein Anmarkrud, Eva Wennås Brante, and Anette Andresen university of oslo, norway

PURPOSE OF THE CHAPTER The main purpose of this chapter is to address the potential processing challenges of the Internet for readers with dyslexia, as the Internet has become the defining technology for literacy and learning in today’s society (Leu, Kinzer, Coiro, Castek, & Henry, 2013). Close to 100% of adolescents, at least in the Western world, have access to and use the Internet on a daily basis (Center for the Digital Future, 2016), and it is widely used in school-related activities such as homework (IIL, 2016). Today’s readers, including those with dyslexia, have access to a wide variety of information sources through the Internet, such as texts, video, animations, audio, pictures, and interactive tables. Hence, the ability to benefit from multiple information sources via the Internet is an important component of literacy in the digital age (Goldman et al., 2010; Stadtler & Bromme, 2013). Having a vast amount of information easily available, just a mouse click or finger swipe away, has certainly provided new affordances for learning, but it also gives rise to new concerns. For example, when readers acquire new knowledge from the web, they need to identify relevant information from credible sources and integrate this information into a coherent mental representation across sources, types of media, and often opposing perspectives before they finally integrate this mental representation with their prior knowledge (e.g., Kamil & Chou, 2009; Rouet & Britt, 2014). This task can be challenging for any reader, regardless of reading proficiency; however, the task might be particularly challenging for readers with dyslexia, who already struggle with more traditional forms of reading. In this chapter, we will review theory and research to examine the potential demands the Internet place on the reading skills and associated processes for readers with dyslexia.

117

118  •  Anmarkrud et al.

OVERVIEW OF THE CHAPTER We will start by defining the term source and describing single source use and the associated reading processes. Then, we describe dyslexia and the challenges that readers with dyslexia typically encounter when reading single sources. In the second section, we first describe multiple source use and the associated reading processes that extend beyond the reading processes associated with single source use, with a particular emphasis on reading via the Internet. Then, we describe the particular challenges readers with dyslexia may encounter during multiple source use on the Internet. In the third section, we will review the limited amount of studies on multiple source use in an Internet context that involves readers with dyslexia. Finally, we will identify some future directions for research. The dictionary definition of a source is the origin of a message or information (Chandler & Munday, n.d.). Although this definition seems to be uncontroversial, Goldman and Scardamalia (2013) were “struck by the polysemy” (p. 257) and the multiple referents of the term in the scientific literature on multiple source use. In the research literature, a source in some instances refers to the information resource itself (e.g., an article published in the online edition of a newspaper), whereas in other instances, the term refers to the author of that information resource (Goldman & Scardamalia, 2013). In this chapter, we will use the first notion of the term; that is, the term source will refer to information resources where individuals and organizations create and publish content on the Internet independently of the media format in which the content is presented. In the context of online reading, the term Internet source is used, and it will include everything ranging from a video a teenager publishes on YouTube, to a blog written by a journalist, to a peer-reviewed research article found on the web page of a “top-notch” scientific journal. With more than one billion websites with unique hostnames (www.internetlivestats.com), Internet sources vary considerably in their credibility, rhetoric aim, media format, and content.

SINGLE SOURCE READING AND DYSLEXIA Comprehending a single text, whether it is a paper-based text or a blog entry, involves the construction of mental representations on different levels. According to Kintsch’s (1998) highly influential construction-integration model of single text comprehension, in addition to the surface code (i.e., a verbatim representation of the text), the reader constructs two additional levels of mental representation for a single text: the textbase and the situation model. The textbase represents the text-internal or semantic meaning of the text which is constructed when the reader decodes the graphic symbols (letters), recognizes the words, and understands the meaning of the sentences, either through explicit ideas in the text or through inferences. The situation model goes deeper and represents an interpretation of the situation described in the text, based on an integration of the text-internal meaning (the textbase) and relevant prior knowledge. Hence, the textbase is sufficient to reproduce the content of a text on a superficial level, but it is not considered sufficient for a deep understanding (Kintsch & Rawson, 2007). For that to occur, readers need to integrate the information in the text with their background knowledge; that is, construct a coherent interpretation of

Potential Processing Challenges  •  119

the situation described in the text. Comprehension of a text rests on several factors, and is beyond the scope of this chapter. Hence, we will focus only briefly on decoding and prior knowledge; we refer the readers to Israel (2017) for an updated and extensive presentation of reading comprehension and its underlying processes and competencies. Fluent and accurate decoding can be conceived as the “bottleneck” of reading comprehension. Thus, if readers have poor decoding skills and struggle to identify the words in a text, their reading comprehension is negatively affected. The relationship between decoding skills and reading comprehension is also related to processing capacity such that both word recognition and comprehension processes compete for working memory capacity (e.g., Stanovich, 1986). The relationship between decoding and reading comprehension is particularly evident early in reading development, but word recognition is also a strong predictor of reading comprehension among adolescent (e.g., Samuelstuen & Bråten, 2005) and adult readers (e.g., Cunningham, Stanovich, & Wilson, 1990). Notably, however, word recognition skills are a necessary, but not sufficient factor in the development of reading comprehension (Stanovich, 2000). There is solid evidence from research on single text comprehension that prior knowledge is linked to readers’ understanding of text. With the exception of decoding skills, no other factor exerts more influence on reading comprehension than a reader’s prior knowledge (Alexander & Jetton, 2000). The term prior knowledge has many referents in the literature on reading comprehension. On the most basic level, letter knowledge is a form of linguistic knowledge underpinning decoding skills, and it is consequently important for reading comprehension. Vocabulary is another form of knowledge that is widely examined in relation to reading comprehension. Several studies indicate that vocabulary can have a direct (e.g., Cromley & Azevedo, 2007; McKeown, Beck, Omanson, & Perfetti, 1983) as well as an indirect influence on reading comprehension through the inferences the reader is able to make (e.g., Calvo, 2004; Cromley & Azevedo, 2007). Prior knowledge has also been examined in relation to reading comprehension through the term subject-matter knowledge, and domain and topic knowledge are often distinguished from one another (e.g., Alexander, 2005). In the context of reading, domain knowledge refers to the breadth of the readers’ knowledge within a certain area (e.g., history or sport), whereas topic knowledge refers to the depth of knowledge in a particular topic (e.g., the Russian Revolution or Spanish soccer teams) within that domain. Subject-matter knowledge affects reading comprehension in several ways. It can give the reader contextual cues that can support word recognition (e.g., Martin & Duke, 2011), it facilitates the use of comprehension strategies (e.g., Cromley & Azevedo, 2007; Vidal-Abarca, Martínez, & Gilabert, 2000), and it helps the reader draw inferences when encountering coherence gaps in a text (e.g., McNamara, Kintsch, Songer, & Kintsch, 1996). Thus, to be able to comprehend a text, the reader must, within the limits of working memory, be able to recognize the words in the text with sufficient accuracy and fluency and have adequate language comprehension, domain-specific knowledge, and world knowledge (Vellutino, Fletcher, Snowling, & Scanlon, 2004). Children struggle to read for various reasons, including learning disabilities such as dyslexia. Dyslexia is a specific learning disability that is characterized by difficulties with

120  •  Anmarkrud et al.

accurate and/or fluent word recognition, poor decoding skills, and spelling difficulties (Lyon, Shaywitz, & Shaywitz, 2003). Dyslexia has a neurobiological origin (e.g., Shaywitz & Shaywitz, 2008), and the associated reading and writing difficulties are typically a result of a deficit in the phonological component of language (e.g., Harm & Seidenberg, 1999; Rasmus et al., 2003). Phonological skills are critical for learning to read, since basic decoding skills rest on the mappings between speech sounds in oral language (phonemes) and the symbols representing the written version of these sounds (graphemes) (Seidenberg & McClelland, 1989). Hence, due to these phonological difficulties, one of the earliest signs of dyslexia is problems related to learning letter sounds and names when formal reading instruction starts (Hulme & Snowling, 2009). Phoneme awareness, which is the ability to identify and manipulate phonemic units in spoken words, is another phonological skill widely examined in relation to dyslexia in both longitudinal and concurrent studies. In a meta-analysis of this body of research, Melby-Lervåg, Lyster, and Hulme (2012) found a very large deficit (pooled effect size estimated d = −1.37) in phoneme awareness when comparing children with dyslexia with typically developing children of the same age. Importantly, both phoneme awareness and letter knowledge are causes of difficulties in learning to read, not consequences of reading difficulties (Hulme & Snowling, 2014). Many models of word reading distinguish between a phonological pathway to word reading and a more direct lexical pathway (also referred to as semantic or orthographic) (e.g., Rack, Snowling, & Olson, 1992; Seidenberg, 2006). In the phonological pathway, the reader “translates” the written form of the word into its corresponding spoken form. The prototypical example of this pathway is beginning readers’ blending of sounds into a word, which is a slow and capacity-demanding form of word reading. In the lexical pathway, a written word leads to a direct activation of a word’s meaning in a reader’s lexicon, and it is generally assumed to be a prerequisite for rapid and accurate reading (e.g., Martens & de Jong, 2008). This lexical pathway does, however, require that words have been encountered before and that the reader already has established the phonological route. Because of their phonological problems, readers with dyslexia struggle to master the phonological route, with the consequence that the establishment of the lexical pathway may be delayed or hampered (Hulme & Snowling, 2009). Hence, readers with dyslexia will often to a larger degree, and for a longer time, base their word reading on the slow and capacity-demanding phonological route; this uses a lot of their cognitive capacity to decode words and has consequences for reading comprehension and text-based learning. Whereas poor skills in word recognition, decoding, and spelling are the primary characteristics of dyslexia, these primary characteristics may lead to secondary consequences, such as problems in reading comprehension and reduced reading experience, which in turn can further hamper learning and vocabulary development (Bishop & Snowling, 2004; Lyon et al., 2003). Apart from the phonological deficit, research indicates that deficits in executive functions such as working memory (e.g., Gathercole, Alloway, Willis, & Adams, 2006; Melby-Lervåg, Lyster, & Hulme, 2012), shifting/cognitive flexibility (e.g., Helland & Asbjørnsen, 2000; Horowitz-Kraus, 2012), and attentional inhibition (e.g., Brosan et  al., 2002; Facoetti et al., 2006) are associated with dyslexia. Given that between 3% and 6% of the population has dyslexia (Hulme & Snowling, 2009), readers with dyslexia are present in most classrooms, and understanding how their difficulties affect reading and learning is an important topic that researchers have

Potential Processing Challenges  •  121

investigated for decades (Melby-Lervåg, Lyster, & Hulme, 2012; Vellutino et al., 2004). Hence, there is an extensive body of research literature on dyslexia and the particular challenges dyslexic readers experience in reading and text-based learning. As previously described, the majority of research on dyslexia has focused on the phonological difficulties that create severe problems for readers in mapping the letter sequences of printed words onto corresponding speech sounds, how these difficulties can hamper the development of proficient reading and subsequent comprehension, and how these difficulties can be addressed by educational interventions (Snowling & Hulme, 2011). This research has been conducted almost exclusively in contexts where dyslexic readers read traditional, often single, texts on paper (Henry et al., 2012).

MULTIPLE SOURCE USE IN A DIGITAL CONTEXT AND POTENTIAL CHALLENGES FOR READERS WITH DYSLEXIA Up to the early 2000s, research examining reading comprehension and text-based learning typically involved a reader encountering a single text, usually on paper. With the development of new and user-friendly information technologies such as the Internet, the conception of what constitutes a typical reading situation has changed considerably, and this change has had implications for research on reading (Fox & Alexander, 2017) and what it means to be a skilled reader (Cho & Afflerbach, 2017; Leu, Kiili, & Forzani, 2016; Salmerón, Strømsø, Kammerer, Stadtler, & van den Broek, in press). Whereas conventional printed textbooks typically include text and various forms of illustrations, Internet sites regularly combine various representations, such as text, illustrations, animations, sounds, and video files. This combination of different representations pushes the boundaries of comprehension. Readers of Internet sites not only have to comprehend the text on the site but also have to integrate the textual information with disparate information from other representations (Kamil & Chou, 2009). Several cognitive models have been developed to explain multimedia learning, with Mayer’s (2009, 2014) cognitive theory of multimedia learning being the most influential. The basic assumption in this model is that multimedia learning rests on a cognitive system with multiple memory stores, and a working memory system of limited capacity is a central processing component. Finally, the model posits that good multimedia learning requires the integration of information from various representations and that comprehension and learning from multimedia can be hampered by the constraints of the human cognitive system, particularly working memory. A significant body of research indicates the benefits of using multiple representations in academic learning (e.g., Butcher, 2014; Frisch & Schulz, 2013), but there are also findings indicating that multiple representations can have a detrimental effect on learning (e.g., Berends & van Lieshout, 2009; Mayer et  al., 2001). For example, Mayer et al. (2001) found that inserting interesting but not particularly task-relevant videos into learning material led to weaker performance on transfer tests than when no such videos were inserted. Similarly, Mayer and Jackson (2005) found that including additional, relevant, narrated animation in a multimedia presentation to explain mathematical formulae for understanding wave movements led to less learning compared with a multimedia environment without these extra animations. These findings

122  •  Anmarkrud et al.

suggest that the inferences the reader would have to make to fill the coherence gaps between the different representations may increase the extraneous cognitive load of the task. Moreover, research on learning from multiple representations has examined the effect of redundant information on multimedia learning. Several studies (e.g., Gerjets et al., 2009; Moreno & Mayer, 2002) show that when one representation (e.g., an animation) includes redundant information also found in another representation (e.g., text), the result can impede learning. The proposed explanation for this finding is that when the same information is included in two (or more) representations, the learner has to process two (or more) distinct sources of information without gaining an increased understanding of the information; additionally, this extra processing increases the extraneous cognitive load (i.e., capacity devoted to processing that does not contribute to learning) of the task. One individual difference variable found to moderate the effect of design principles such as the coherence principle and the redundancy principle is the learner’s prior knowledge. For example, several studies have shown that high-knowledge readers are affected much less by the low-coherence structure of hypertext (e.g., Amadieu et  al., 2009; Calisir & Gurel, 2003) than lowknowledge readers. Several arguments have been presented for why hypermedia environments, such as the Internet, could be beneficial for student learning. One potential benefit is that hypermedia gives learners control over several aspects of their own learning, such as defining the order in which they access different information units, control over which content to obtain, and control over the presentation format (e.g., text, audio, or video) (Scheiter & Gerjets, 2007). Although there may be several advantages of using hypermedia/Internet in a learning context, there is also a substantial body of research that demonstrates the challenges associated with hypermedia use in learning (Scheiter & Gerjets, 2007). In an extensive review, Scheiter and Gerjets (2007) argued that the same features of hypermedia that can promote learning may also be detrimental for learning. For example, the abundant amount of information and representations found across Internet sites has to be integrated by learners. Previous work on the integration of verbal and visual information in the reading of traditional texts indicates that readers can have difficulties with this integration process (Chan & Unsworth, 2011), and there are indications that meaning making in online reading environments poses even greater processing demands on readers (Chan & Unsworth, 2011; Hartman et al., 2010; Kamil & Chou, 2009). Hence, the magnitude of information across Internet sites and various representations, while representing opportunities for good learning, may also result in cognitive overload and interfere with learning (Scheiter & Gerjets, 2007). Imagine a 10th-grade science student with dyslexia using the Internet to learn about the potential harmful effects of sunbathing. How would her difficulties affect the competencies found to be important for reading and learning from the Internet? Salmerón et  al. (in press) emphasize three distinctive and crucial competencies required in Internet reading. First, the Internet, with its hypertext structure and constantly growing body of information, has to be navigated to identify relevant sources of information. An efficient Internet query requires both spelling skills (e.g., Barsky & Bar-Ilan, 2012) and prior knowledge (e.g., Chu & Law, 2008; Laberge & Scialfa, 2005). Navigation in a hypertext structure, such as the Internet, also draws heavily on working memory capacity (e.g., DeStefano & LeFevre, 2007; Rosman, Mayer, & Krampen, 2016). Hence,

Potential Processing Challenges  •  123

spelling difficulties (one of the primary characteristics of dyslexia), deficits in prior knowledge (e.g., vocabulary and subject-matter knowledge), and deficits in working memory, which are often observed in dyslexia, can influence dyslexic students’ navigation skills negatively. The second competency is that the sources of information identified through navigation on the Internet have to be evaluated with regard to task relevance and credibility Determining the relevance of a source found on the Internet involves judgments of the perceived value of information from that source in relation to the readers’ goal or purpose for reading (e.g., Anmarkrud, McCrudden, & Bråten, 2013). Making such judgments involves strategic behavior, where the reader is required to evaluate the content of any particular source in relation to his or her task model. A task model is a mental representation of the expected outcome of reading (e.g., “I want to understand whether sun exposure can have negative effects on health”) and can vary considerably in complexity (Rouet, Britt, & Durik, 2017). Our fictive student finds a web page that provides information about the increased risk of melanoma related to sun exposure, information that would be highly relevant for her task. But can she trust the information? The strategic evaluation of the credibility of sources, often referred to as sourcing (e.g., Braasch & Bråten, 2017), can include examining the authors’ credentials, the publication venue, the date of publication, and so forth, and it has been found to be related to multiple source comprehension (e.g., Anmarkrud, Bråten, & Strømsø, 2014; Cho, 2014). The web page our fictive student has identified comes from a cancer research institute, and it seems to be written by oncologists; therefore, it is judged as trustworthy. The third competency emphasized by Salmerón et al. (in press) is integration. The web page identified by our student contains text, an embedded video interview with an oncologist, pictures, and illustrations. Hence, the student has to process information from the various representations, select information that is relevant for the task s/he is currently working on, construct and organize knowledge structures based on the information from the various representations, and finally integrate these knowledge structures with each other, with information from other web sites (e.g., that sun exposure can have positive health effects due to the increased production of vitamin D) and with prior knowledge from long-term memory. Both judgments of task relevance and credibility, as well as the integration of information across web pages and representations, are strategic behaviors (Cho & Afflerbach, 2017). Comprehension strategies may be defined as intentional attempts to control and modify meaning construction during reading (cf., Afflerbach, Pearson, & Paris, 2008). Contrary to learning from a single source, when learning from multiple sources (e.g., the Internet), readers themselves are the “authors” of the integrated and coherent mental representation of the issue they read about. When there is a high amount of semantic overlap between sources, automatic, bottom-up resonance processes may drive integration (Myers & O’Brien, 1998); otherwise, top-down, attentional strategic activity seems required to integrate information from multiple sources (Kurby, Britt, & Magliano, 2005). Consequently, when attentional strategic behavior seems to be a requirement for deep learning from multiple sources, such as the Internet, the learner has to devote working memory capacity to execute and monitor his/her strategy use. Hence, such learning can potentially challenge the limits of working memory capacity, especially among dyslexic readers, who have to devote a large amount of WM capacity to more basic reading processes. This is particularly true since Internet sources are not

124  •  Anmarkrud et al.

necessarily designed according to multimedia principles (e.g., Mayer, 2014) to reduce the load on working memory.

EMPIRICAL RESEARCH ON DYSLEXIA AND INTERNET READING In this part of the chapter, we will review empirical research that examines dyslexic readers’ Internet reading and use of multiple sources, which is generally limited. There are some studies on dyslexia, web accessibility, and interface design (e.g., Borg, Lantz, & Gulliksen, 2015; McCarthy & Swierenga, 2010), other studies on how technology, software, and web-based resources can contribute to struggling readers’ literacy development (e.g., Harrison, 2012), and some studies on how multimedia tools can enhance dyslexic students’ learning (e.g., Feeney, 2003). However, this chapter focuses on the challenges and affordances dyslexic readers experience when using the Internet in their daily learning activities, not how specially designed web pages, multimedia tools, ICTs, fonts, synthetic speech, or interfaces can enhance learning. The presentation of empirical research will be structured according to the skills identified as important for efficient comprehension in digital reading (Salmerón et al., in press): navigation skills, skills in evaluating information found online, and the ability to integrate information across multiple sources (e.g., different representations and web pages). Dyslexic Readers’ Navigation in a Hypertext Structure Three major challenges that readers with dyslexia face when navigating a hypertext structure to identify information include: misspelling in search queries, working memory capacity related to navigation orientation during Internet reading, and difficulties in fast and flexible shifting of attention. Misspelling in Search Queries Berget and Sandnes (2015) examined how spelling skills affected information searches in a sample of 40 university students, 20 of whom were diagnosed with dyslexia. The participants were given several search tasks (e.g., “Find a play written by William Shakespeare” or “Find a document about women in Algeria”) to solve using a search engine from a university library. All instructions were presented orally to prevent misinterpretations of tasks due to reading problems and to prevent the participants from seeing the spelling of the search terms. The results showed that dyslexic participants misspelled queries more than the controls and consequently obtained fewer relevant hits from the search engine. Additionally, many dyslexic participants applied strategies to compensate for their spelling problems. For instance, dyslexic participants used external search engines that provide suggestions on the correct spelling in cases of misspelled search terms to check their spelling during the task to a greater extent than controls. In another study, Al-Wabil, Zaphiris, and Wilson (2007) examined the search and navigation behavior of 10 adults with dyslexia. In the study, the participants were presented different web sites and were interviewed about how they would interact with the sites. Spelling was a main theme in many of the interviews, particularly for search

Potential Processing Challenges  •  125

options embedded in web pages that are meant to direct visitors to more detailed information within the web site. Many of the participants stated that they generally avoided the use of search options on the websites since these often do not accept misspellings. Again, a common strategy was to leave the website and use external search engines that suggest correct spelling in cases of misspellings. Thus, misspelling may not only be a challenge to search for relevant information but also affect the ability to search for further information within a website when a relevant site has been identified through an initial search. Profound challenges related to search iterations were also reported by MacFarlane et al. (2010), who examined the search and navigation behavior of 10 students (five students with dyslexia and five students without dyslexia). The results showed that students with dyslexia performed fewer search iterations, each of their searches took a longer amount of time, and they reviewed fewer of the documents found through the search compared with the control group. However, when dyslexic readers learn to master the use of search engines, they can be an efficient and helpful learning tool. Castek et  al. (2011) followed four young struggling readers during online reading and found that they had learned to use search engines effectively. They carefully read search engine results rather than clicking and scrolling through the hit list. Because of their reading problems, their careful reading of the hit search results was a slow and effortful procedure, but the outcome seemed to be deliberate and informed their decisions concerning what to skip and what to read more carefully. The deliberate and careful processing of search results also activated background knowledge and helped the readers gradually gain more information about the issue they were researching. The ability to search for and identify relevant information and, in that sense, have agency in constructing their own text were also described as supporting sustained reading, engagement, and reading motivation. Working Memory Capacity A second challenge in dyslexic readers’ search and navigation behavior seems to be working memory capacity in relation to memory for navigation paths. Al-Wabil et al. (2007) interviewed their participants about how they kept track of their search and navigation paths when searching for information on the web. Several of the participants described it as challenging to shift their attention between the screen and the keyboard. Instead of using bookmarks and the inbuilt search history, a common strategy was to press the “home button” and start all over again or use the “back button” repeatedly to get back to their starting point. Hence, keeping track of their navigation path simultaneously while reading the content of the pages and using the keyboard was described as challenging among the participants, often resulting in an inefficient strategy of hitting the “home button” and starting all over again. Shifting The third mayor challenge is related to the executive function shifting. Dyslexia has been found to be associated with slower shifts in attention (e.g., Moura, Simões, & Pereira, 2015), meaning that readers with dyslexia have difficulty shifting their attention from one item to another, especially when they face a stimulus in rapid sequences

126  •  Anmarkrud et al.

of information. Berget, Mulvey and Sandnes (2015) conducted an eye-tracking study to examine whether visual content (icons) in search interfaces were beneficial to readers with dyslexia. The sample included 42 college students, 21 of whom had been diagnosed with dyslexia. One of the hypotheses in the study was that the inclusion of icons in the search interface would help dyslexics locate the target more easily than when the search tasks included only words. The results were consistent with this hypothesis, but only when the distance between the two representations (icons and written words) prevented them from being in central vision simultaneously. When the two representations were in central vision at the same time, the search time increased, and the fixations were longer. This finding indicated that the shift in attention between the two representations was particularly challenging for the dyslexic participants and imposed an increased mental load. Hence, when Internet pages include multiple representations (e.g., text, pictures, and video both in content and in commercials), the transition of attention between the representations can be a challenge for dyslexic readers. Dyslexic Readers’ Evaluation of Information Found on the Internet Research on offline and paper-based reading has documented that struggling readers use reading strategies less than more proficient readers (e.g., Furnes & Norman, 2015), which has also been found in studies explicitly examining strategies for critical evaluation of texts during reading (Cantrell, Almasi, Rintamaa, Pennington, & Buchman, 2014). To our knowledge, only two studies have examined struggling readers’ critical evaluation during Internet reading (Castek et al, 2011; Henry et al., 2012). Castek et al. (2011) reported the results of a comparative case study examining four struggling readers’ Internet reading. Only one of four participants in the study checked the accuracy of information on a website; more specifically, this participant proposed that one could compare information found on a web site with information from another “site or book” (p. 100). Henry et al. (2012) presented the results from a case study examining the implementation of an empowerment model for struggling readers who use the Internet as a context for reading, writing, and communication. The development of strategies for critically evaluating information for relevance, accuracy, and reliability is one important component of the program. Although strategies for critically evaluation of information found on the Internet were in focus during the intervention, no information is provided regarding whether the struggling readers participating in the intervention were able to learn such strategies. Dyslexic Readers’ Integration of Information Across Multiple Sources It is well documented in the literature on multimedia learning that the combination of text and other representations can enhance learning more than text-only scenarios can (Butcher, 2014), but research has also found evidence that the combination of text and pictures can hamper the reading comprehension of individuals with dyslexia (e.g., Beacham & Alty, 2006; Harber, 1983; Olander, Brante, & Nyström, 2017). In a recent study, Andresen, Anmarkrud, and Bråten (2017) compared dyslexic and non-dyslexic students’ integration of information across different representations and web pages. Participants in the study were 44 10th-graders, 22 of whom had been diagnosed with dyslexia after undergoing an assessment with a comprehensive battery of standardized cognitive, linguistic, and reading tests. A diagnosis of severe difficulties was based on

Potential Processing Challenges  •  127

single-word decoding and phonological processing. The non-dyslexic group consisted of students all scoring above the 20th percentile on a national standardized reading test. The participants were given access to a mock Internet site containing three web pages about the controversial issue of sun exposure and health. One page presented neutral information about different types of UV radiation and how UV radiation is measured, one page presented information on the potential risks of sun exposure, particularly the increased risk of skin cancer, and one page presented information about the benefits of sun exposure, particularly the increased production of vitamin D that is found to protect against prostate cancer and colorectal cancer. All web pages contained a title and a lead paragraph explaining the overall content of the web page, followed by a video, a short piece of text, and a picture. There were no differences between the two groups with respect to their ability to gain pieces of factual and conceptual knowledge from the web pages and the different representations (texts, videos, and pictures). However, the students with dyslexia had difficulty integrating information across different types of media and across web pages compared with the non-dyslexic group, resulting in a less coherent and sophisticated mental representation of the issue in question. Specifically, when compared with students without dyslexia, students with dyslexia failed to include textual information when trying to integrate information across representations and web pages. In other words, the results indicated that students with dyslexia were able to extract bits of factual knowledge from the texts on the web pages, but they did not seem able to use this information when trying to make an integrated mental representation of the issue across representations and the web pages. A study by MacCullagh, Bosanquet, and Badcock (2017) also highlights the challenges student with dyslexia have integrating information across representations. Interviews were conducted with 13 university students with dyslexia to investigate how they compensate for their reading difficulties. Several participants reported that recorded lectures that they viewed online were challenging to follow when the lectures combined text boxes, recordings of the lecturer, and animations. The participants described that they often had to go through the online lectures several times to be able to benefit from them (e.g., just listen to the lecturer the first time, pay attention to the text boxes and animations the second time, and take notes the third time).

CONCLUSION AND FUTURE RESEARCH In this chapter, we have addressed some challenges readers with dyslexia can experience when using the Internet as a source for information. The very limited amount of research on this particular group of readers when learning from multiple sources on Internet indicates that the well-documented difficulties dyslexic readers experience when reading single texts can be even more profound in a multiple source context on the Internet due to the presumably increased load the integration of information within and across representations and web pages places on working memory. However, reading and learning is more than cognitive processing. Leu et  al. (2014) suggest that a new technology (with a new literacy) always be taught to the weakest reader first. This serves two purposes: it positions the weak reader as an expert, and it gives the struggling reader a head start. Castek et  al. (2011) state that when struggling readers are confronted with the normal layout of textbooks, they tend to shut down and lose motivation to attempt further reading. The Internet, with short texts and different types of media, gives struggling readers the ability to construct their own texts and hence control

128  •  Anmarkrud et al.

their own information path; this could result in higher engagement in reading (ibid.). If they do not understand what they are currently reading, they can search the web to find something they can understand. In MacCullagh et al.’s (2017) interview study with dyslexic students, nearly half of the participants reported that they searched for videos online to replace or supplement assigned readings. Hence, they strategically used the multimedia characteristics of the Internet to compensate for their reading difficulties, something also found in case studies of younger readers (Castek et al., 2011). In the case study by Henry et al. (2012), collaborative learning based on Internet reading led to more inclusion and increased participation among struggling readers, and they found situations where poor readers could be experts due to their mastery of search engines, chat software, and graphic organizers, to name a few examples. When struggling readers learn to benefit from the often inbuilt possibilities of changing fonts, font size, synthetic speech, spelling suggestions in search engines, recording of their own voice when reading, and the opportunity to find the definition of words, the Internet can be an important and compensatory learning tool for struggling readers (Castek et al., 2011; Harrison, 2012; Henry et al., 2012). Hence, these promising findings of the Internet as a motivating learning tool compensating for dyslexic readers’ word decoding difficulties should be followed up with more research to identify ways to help readers with dyslexia develop their understanding of multiple sources on the Internet. At the same time, our review of the literature on struggling readers’ Internet reading in this chapter revealed three main findings. First, despite the prevalence of reading difficulties in our society, this group of readers is almost invisible in the literature on Internet reading, multiple source use, and multimedia learning. As a research community, we know a lot about struggling readers and about reading and learning in digital contexts, but we know very little about struggling readers and how their difficulties affect reading and learning in digital environments. Second, the little research that is available largely consists of case studies, interview studies, or studies with a very small number of participants. Although these studies provide important and valuable insight into the affordances and challenges struggling readers experience when working with multiple sources on the Internet, there is a dire need for cognitive research examining how the particular processing demands of struggling readers affect their reading in digital environments and experimental research aiming to identify efficient interventions that promote struggling readers’ digital literacy, such as single-case designs or even experiments with larger numbers of participants. Lastly, the small amount of available research has largely examined the search behavior and search strategies of struggling readers; high-quality research on the evaluation and integration of information found online is virtually non-existent. At the same time, there is emerging evidence that the use of the Internet can engage struggling readers and to some degree compensate for their reading difficulties, which makes the need for an increased research effort even more important.

REFERENCES Afflerbach, P., Pearson, P. D., & Paris, S. G. (2008). Clarifying differences between reading skills and reading strategies. The Reading Teacher, 61, 364–373. Alexander, P. A. (2005). The path to competence: A lifespan developmental perspective on reading. Journal of Literacy Research, 37, 413–436.

Potential Processing Challenges  •  129 Alexander, P. A., & Jetton, T. (2000). Learning from text: A multidimensional and developmental perspective. In M. Kamil, P. Mosenthal, P. Pearson, & R. Barr (Eds.), Handbook of reading research, vol. 3 (pp. 285–310). Mahwah, NJ: Lawrence Erlbaum Associates. Al-Wabil, A., Zaphiris, P., & Wilson, S. (2007). Web navigation for individuals with dyslexia: An explanatory study. In C. Stephandis (Ed.) Proceedings from the International Conference on Universal Access in HumanComputer Interaction (pp. 593–602). Berlin: Springer. Amadieu, F., Tricot, A., & Mariné, C. (2009). Prior knowledge in learning from a non-linear electronic document: Disorientation and coherence of the reading sequences. Computers in Human Behavior, 25, 381–388. Andresen, A., Anmarkrud, Ø., & Bråten, I. (2017). Investigating multiple source use among students with and without dyslexia. Paper presented at the Annual meeting of the American Educational Research Association, San Antonio, Texas. Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. Anmarkrud, Ø., McCrudden, M. T., & Bråten, I. (2013). Task-oriented reading of multiple documents: Online comprehension processes and offline products. Instructional Science, 41, 873–894. Barsky, E., & Bar-Ilan, J. (2012). The impact of task phrasing on the choice of search keywords and on the search process and success. Journal of the Association for Information Science and Technology, 63, 1987–2005. Beacham, N. A., & Alty, J. L. (2006). An investigation into the effects that digital media can have on the learning outcomes of individuals who have dyslexia. Computers & Education, 47, 74–93. Berends, I. E., & van Lieshout, E. C. D. M. (2009). The effect of illustrations in arithmetic problem-solving: Effects of increased cognitive load. Learning and Instruction, 19, 345–353. Berget, G., Mulvey, F., & Sandnes, F. E. (2015). Is visual content in textual search interfaces beneficial to dyslexic users? International Journal of Human–Computer Studies, 92–93, 17–29. Berget, G., & Sandnes, F. E. (2015). Searching databases without query-building aids: Implications for dyslexic users. Information Research, 20, paper 689. Bishop, D. V. M., & Snowling, M. J. (2004). Developmental dyslexia and specific language impairment: Same or different? Psychological Bulletin, 130, 858–886. Borg, J. N., Lantz, A., & Gulliksen, J. (2015). Accessibility to electronic communication for people with cognitive disabilities: A systematic search and review of empirical evidence. Universal Access in the Information Society, 14, 547–562. Brosan, M., Demetre, J., Hamill, S., Robson, K., Shepherd, H., & Cody, G. (2002). Executive functioning in adults and children with developmental dyslexia. Neuropsychologia, 40, 2144–2155. Braasch, J. L. G., & Bråten, I. (2017). The discrepancy-induced source comprehension (D-ISC) model: Basic assumptions and preliminary evidence. Educational Psychologist. DOI: 10.1080/00461520.2017.1323219. Butcher, K. R. (2014). The multimedia principle. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning, 2nd edition (pp. 174–205). New York: Cambridge University Press. Calisir, F., & Gurel, Z. (2003). Influence of text structure and prior knowledge of the learner on reading comprehension, browsing, and perceived control. Computers in Human Behavior, 19, 135–145. Calvo, M. G. (2004). Relative contribution of vocabulary knowledge and working memory span to elaborate inferences in reading. Learning and Individual Differences, 15, 53–65. Cantrell, S., Almasi, J., Rintamaa, M., Pennington, J., & Buchman, D. (2014). The impact of supplemental instruction on low achieving adolescents’ reading engagement. The Journal of Educational Research, 107, 36–58. Castek, J., Zawilinski, L., McVerry, J. G., O’Byrne, W. I., & Leu, D. J. (2011). The new literacies of online reading comprehension: New opportunities and challenges for students with learning difficulties. In C. WyattSmith, J. Elkins, & S. Gunn (Eds.), Multiple perspectives on difficulties in learning literacy and numeracy (pp. 91–110). New York: Springer. Center for the Digital Future (2016). The world Internet project: International report, 6th edition. Los Angles, CA: University of Southern California. Chan, E., & Unsworth, L. (2011). Image-language interaction in online reading environments: Challenges for students’ reading comprehension. Australian Educational Researcher, 38, 181–202. Chandler, D., & Munday, R. (n.d.) Source. In A Dictionary of Media and Communication. Oxford University Press. Retrieved 29 June 2017 from www.oxfordreference.com/view/10.1093/acref/9780191800986.001. 0001/acref-9780191800986-e-2580.

130  •  Anmarkrud et al. Cho, B.-Y. (2014). Competent adolescent readers’ use of Internet reading strategies: A think-aloud study. Cognition and Instruction, 32, 253–289. Cho, B.-Y., & Afflerbach, P. (2017). An evolving perspective of constructively responsive reading comprehension strategies in multilayered digital text environments. In S. E. Israel (Ed.), Handbook of research on reading comprehension, 2nd edition (pp. 109–134). New York: The Guilford Press. Chu, S.K.-W., & Law, N. (2008). The development of information search expertise of research students. Journal of Librarianship and Information Science, 40, 165–177. Cromley, J. G., & Azevedo, R. (2007). Testing and refining the direct and inferential mediation model of reading comprehension. Journal of Educational Psychology, 99, 311–325. Cunningham, A. E., Stanovich, K. E., & Wilson, M. R. (1990). Cognitive variation in adult college students differing in reading ability. In T. H. Carr & B. A. Levy (Eds.), Reading and its development: Component skill approaches (pp. 129–159). San Diego, CA: Academic Press. DeStefano, D., & LeFevre, J.-A. (2007). Cognitive load in hypertext reading: A review. Computers in Human Behavior, 23, 1616–1641. Facoetti, A., Zorzi, M., Cestnick, L., Lorusso, M. L., Molteni, M., Paganoni, P., . . . Mascetti, G. G. (2006). The relationship between visuo-spatial attention and nonword reading in developmental dyslexia. Cognitive Neuropsychology, 23, 841–855. Feeney, A. E. (2003). Using interactive multimedia for dyslexic students in geography education. Journal of Geography, 102, 21–28. Fox, E., & Alexander, P. A. (2017). Text and comprehension. In S. E. Israel (Ed.), Handbook of research on reading comprehension, 2nd edition (pp. 335–352). New York: The Guilford Press. Frisch, A.-L., & Schulz, P. J. (2013). The impact of presentation style on the retention of online health information: A randomized-controlled experiment. Health Communication, 28, 286–293. Furnes, B., & Norman, E. (2015). Metacognition and reading: Comparing three forms of metacognition in normally developing readers and readers with dyslexia. Dyslexia, 21, 273–284. Gathercole, S. E., Alloway, T. P., Willis, C., & Adams, A.-M. (2006). Working memory in children with reading disabilities. Journal of Experimental Psychology, 93, 265–281. Gerjets, P., Scheiter, K., Opfermann, M., Hesse, F. W., & Eysink, T. H. S. (2009). Learning with hypermedia: The influence of representational formats and different levels of learner control on performance and learning behavior. Computers in Human Behavior, 25, 360–370. Goldman, S. R., & Scardamalia, M. (2013). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities. Cognition and Instruction, 31, 255–269. Goldman, S. R., Lawless, K. A., Gomez, K. W., Braasch, J., MacLeod, S. M., & Manning, F. H. (2010). Literacy in the digital world. Comprehending and learning from multiple sources. In M. G. McKeown & L. Kucan (Eds.), Bringing reading research to life (pp. 257–284). New York: The Guilford Press. Harber, J. R. (1983). The effects of illustrations on the reading performance of learning disabled and normal children. Learning Disability Quarterly, 6, 55–60. Harm, M. W., & Seidenberg, M. S. (1999). Phonology, reading acquisition, and dyslexia: Insights from connectionist models. Psychological Review, 106, 491–528. Harrison, C. (2012). Literacy, technology and the Internet: What are the challenges and opportunities for learners with reading difficulties, and how do we support them in meeting those challenges and grasping those opportunities? In C. Wyatt-Smith, J. Elkins, & S. Gunn (Eds.), Multiple perspectives on difficulties in learning literacy and numeracy (pp. 111–132). New York: Springer. Hartman, D. K., Morsink, P. M., & Zheng, J. (2010). From print to pixels: The evolution of cognitive conceptions of reading comprehension. In E. A. Baker (Ed.), The new literacies: Multiple perspectives on research and practice (pp. 131–164). New York: Guilford Press. Helland, T., & Asbjørnsen, A. (2000). Executive functions in dyslexia. Child Neuropsychology, 6, 37–48. Henry, L. A., Castek, J., O’Byrne, W. I., & Zawilinski, L. (2012). Using peer collaboration to support online reading, writing, and communication: An empowerment model for struggling readers. Reading and Writing Quarterly, 28, 279–306. Horowitz-Kraus, T. (2012). Pinpointing the deficit in executive functions in adolescents with dyslexia performing the Wisconsin Card Sorting Test: An ERP study. Journal of Learning Disabilities, 47, 208–223. Hulme, C., & Snowling, M. J. (2009). Developmental disorders of language learning and cognition. Chichester: Wiley-Blackwell.

Potential Processing Challenges  •  131 Hulme, C., & Snowling, M.J. (2014). The interface between spoken and written language: Developmental disorders. Philosophical Transactions of The Royal Society B369. 20120395. http://dx.doi.org/10.1098/ rstb.2012.0395. IIL (2016). Svenskarna och internet 2016: Undersøkning om svenskarnas internetvanor [The Swedes and the Internet 2016: A survey of the Swedes’ use of the Internet]. Stockholm: Internetstiftelsen i Sverige. Israel, S. E. (Ed.) (2017). Handbook of research on reading comprehension, 2nd edition. New York: The Guilford Press. Kamil, M. L., & Chou, H. K. (2009). Comprehension and computer technology: Past results, current knowledge, and future promises. In S. E. Israel & G. G. Duffy (Eds.), Handbook of research on reading comprehension (pp. 289–304). New York: Routledge. Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge: Cambridge University Press. Kintsch, W., & Rawson, K. (2007). Comprehension. In M. J. Snowling & C. Hulme (Eds.), The science of reading: A handbook (pp. 209–226). Malden, MA: Blackwell Publishing. Kurby, C. A., Britt, M. A., & Magliano, J. P. (2005). The role of top-down and bottom-up processes in betweentext integration. Reading Psychology, 26, 335–362. Laberge, J. C., & Scialfa, C. T. (2005). Predictors of web navigation performance in a life span sample of adults. Human Factors, 47, 289–302. Leu, D. J., Kiili, C., & Forzani, E. (2016). Individual differences in the new literacies of online research and comprehension. In P. Afflerbach (Ed.), Handbook of individual differences in reading (pp. 259–272). New York: Routledge. Leu, D. J., Kinzer, C. K., Coiro, J., Castek, J., & Henry, L. A. (2013). New literacies: A dual-level theory of the changing nature of literacy, instruction, and assessment. In D. E. Alvermann, N. J. Unrau, & R. B. Ruddell (Eds.), Theoretical models and processes of reading, 6th edition (pp. 1150–1181). Newark, DE: International Reading Association. Leu, D. J., Zawilinski, L., Forzani, E., & Timbrell, N. (2014). Best practices in new literacies and the new literacies of online research and comprehension. In L. M. Morrow & L. B. Gambrell (Eds.), Best practices in literacy instruction, 5th edition (pp. 343–364). New York: Guilford Press. Lyon, G. R., Shaywitz, S. E., & Shaywitz, B. A. (2003). A definition of dyslexia. Annals of Dyslexia, 53, 1–14. MacCullagh, L., Bosanquet, A., & Badcock, N. (2017). University students with dyslexia: A qualitative exploratory study of learning practices, challenges and strategies. Dyslexia, 23, 3–23. MacFarlane, A., Al-Wabil, A., Marshall, A., Albrair, C. R., Jones, S. A., & Zaphiris, P. (2010). The effect of dyslexia on information retrieval: A pilot study. Journal of Documentation, 66, 307–326. Martens, V. E. G., & de Jong, P. F. (2008). Effects of repeated reading on the length effect in word and pseudoword reading. Journal of Research in Reading, 31, 40–54. Martin, N. M., & Duke, N. K. (2011). Interventions to enhance informational text comprehension. In A. McGill-Franzen & R. L. Allington (Eds.), Handbook of reading disability research (pp. 345–361). New York: Routledge. Mayer, R. E. (2009). Multimedia learning, 2nd edition. New York: Cambridge University Press. Mayer, R. E. (2014). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 43–71). Cambridge: Cambridge University Press. Mayer, R. E., Heiser, J., & Lonn, S. (2001). Cognitive constraints on multimedia learning: When presenting more material results in less understanding. Journal of Educational Psychology, 93, 187–198. Mayer, R. E., & Jackson, J. (2005). The case for coherence in scientific explanations: Quantitative details can hurt qualitative understanding. Journal of Experimental Psychology: Applied, 11, 13–18. McCarthy, J. E., & Swierenga, S. J. (2010). What we know about dyslexia and Web Accessibility: A research review. Universal Access in the Information Society, 9, 147. McKeown, M. G., Beck, I. L., Omanson, R. C., & Perfetti, C. A. (1983). The effects of long-term vocabulary instruction on reading comprehension: A replication. Journal of Reading Behavior, 15, 3–18. McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14, 1–43. Melby-Lervåg, M., Lyster, S. A. H., & Hulme, C. (2012). Phonological skills and their role in learning to read: A meta-analytic review. Psychological Bulletin, 138, 322–352. Moreno, R., & Mayer, R. E. (2002). Verbal redundancy in multimedia learning: When reading helps listening. Journal of Educational Psychology, 94, 156–163.

132  •  Anmarkrud et al. Moura, O., Simões, M. R., & Pereira, M. (2015). Executive functions in children with developmental dyslexia. The Clinical Neuropsychologist, 28, 20–41. Myers, J. L., & O’Brien, E. J. (1998). Accessing the discourse during reading. Discourse Processes, 26, 131–157. Olander, M. H., Brante, E. W., & Nyström, M. (2017). The effect of illustration on improving text comprehension in dyslexic adults. Dyslexia, 23, 42–65. Rack, J. P., Snowling, M. T., & Olson, R. K. (1992). The nonword reading deficit in developmental dyslexia: A review. Reading Research Quarterly, 27, 28–53. Rasmus, F., Rosen, S., Dakin, S. C., Day, B. L., Castellote, J. M., White, S., & Frith, U. (2003). Theories of developmental dyslexia: Insights from a multiple case study of dyslexic adults. Brain, 126, 841–865. Rosman, T., Mayer, A.-K., & Krampen, G. (2016). A longitudinal study of information-seeking knowledge in psychology undergraduates: Exploring the role of information literacy instruction and working memory capacity. Computers & Education, 96, 94–108. Rouet, J. F., & Britt, M. A. (2014). Multimedia learning from multiple documents. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning, 2nd edition (pp. 813–841). New York: Cambridge University Press. Rouet, J. F., Britt, M. A., & Durik, A. M. (2017). RESOLV: Readers’ representation of reading contexts and tasks. Educational Psychologist. DOI: 10.1080/00461520.2017.1329015. Salmerón, L., Strømsø, H. I., Kammerer, Y., Stadtler, M., & van den Broek, P. (in press). Comprehension processes in digital reading. In P. van den Broek (Ed.), Learning to read in a digital world. Amsterdam: John Benjamins. Samuelstuen, M. S., & Bråten, I. (2005). Decoding, knowledge, and strategies in comprehension of expository text. Scandinavian Journal of Psychology, 46, 107–117. Scheiter, K., & Gerjets, P. (2007). Learner control in hypermedia environments. Educational Psychology Review, 19, 285–307. Seidenberg, M. S. (2006). Connectionist models of reading. In G. Gaskell (Ed.), The Oxford handbook of psycholinguistics (pp. 235–250). Oxford: Oxford University Press. Seidenberg, M. S., & McClelland, J. A. (1989). A distributed, developmental model of word recognition. Psychological Review, 96, 523–568. Shaywitz, S. E., & Shaywitz, B. A. (2008). Paying attention to reading: The neurobiology of reading and dyslexia. Development and Psychopathology, 20, 1329–1349. Snowling, M. J., & Hulme, C. (2011). Evidence-based interventions for reading and language difficulties: Creating a virtuous circle. British Journal of Educational Psychology, 82, 1–23. Stadtler, M., & Bromme, R. (2013). Multiple document comprehension: An approach to public understanding of science. Cognition and Instruction, 31, 122–129. Stanovich, K. E. (1986). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 21, 360–407. Stanovich, K. E. (2000). Progress in understanding reading: Scientific foundations and new frontiers. New York: Guilford. Vellutino, F. R., Fletcher, J. M., Snowling, M. J., & Scanlon, D. M. (2004). Specific reading disability (dyslexia): What have we learned in the past four decades? Journal of Child Psychology and Psychiatry, 45, 2–40. Vidal-Abarca, E., Martínez, G., & Gilabert, R. (2000). Two procedures to improve instructional text: Effects on memory and learning. Journal of Educational Psychology, 92, 107–116.

8

STRATEGIC PROCESSING IN ACCESSING, COMPREHENDING, AND USING MULTIPLE SOURCES ONLINE Byeong-Young Cho university of pittsburgh, usa

Peter Afflerbach university of maryland, usa

Hyeju Han university of pittsburgh, usa

GOAL FOR THE CHAPTER Our chapter draws on a broad range of research to describe readers’ strategic processing with multiple sources. Specifically, we focus on the reading of multiple sources that takes place in the intertextual space for meaning making presented by online text environments. We begin the chapter by offering a definition of reading and describing the nature of strategic processing with multiple online sources. Then, we build a research-based account of the interactive operation of the cognitive and metacognitive strategies that multisource readers use in the course of meaning making and knowledge building. We conclude the chapter with suggestions for future research, informed by our theoretical exploration and research synthesis.

(RE)CONCEPTUALIZING READING WITH MULTIPLE ONLINE SOURCES Our view of reading is grounded in the large corpus of research on reading comprehension (Cho & Afflerbach, 2017; Britt, Goldman, & Rouet, 2013; McNamara & Magliano, 2009; Pressley & Afflerbach, 1995; van Dijk & Kintsch, 1983); we view reading as the act of constructing meaning through accessing, comprehending, and using texts. In this view, 133

134  •  Cho et al.

“texts” can represent multiple sources containing information—information that is understood and used by readers. This notion of source (and text) is determined, in part, by a reader’s goals and purposes. As well, source is situational, compared with other conceptions of source in text and discourse studies. For instance, source has been conceived as the author to which text ideas are attributed (Macedo-Rouet, Braasch, Britt, & Rouet, 2013), more broadly as the type, date, and genre of publication (Braasch, Bråten, Strømsø, Anmarkrud, & Ferguson, 2013), or both (Afflerbach & VanSledright, 2001). These conceptions suggest essential source features, which readers must consider to judge characteristics of sources (e.g., how reliable, believable, and trustworthy a source is), but they offer little to describe the situational and pragmatic meanings of source (e.g., when and where a source is used, who it is used by, for whom it is used, and in what manner). In this chapter, we contemplate source as text in use. This means that a text becomes a source when readers perceive potential uses of it and in fact use the text and its information for certain purposes. This can include connecting an understanding of one text to the interpretation of another text, or deriving evidence from different texts to build a claim. As we will describe, individuals often use multiple sources when they read online. For example, the online reading of a single text often leads to subsequent realizations and opportunities related to other texts. Readers may not immediately identify these “other” texts as relevant; rather, readers may recognize their interconnectedness later in the course of reading—which can enhance (and make more complex) meaning construction (Cho & Afflerbach, 2015). In such cases, a particular text may be situated with other texts in a reader’s mind, and these texts collectively constitute the textual environment in which the reader identifies useful sources, constructs meaning from the sources, and uses them to achieve reading goals. Indeed, some have proposed that the Internet is fundamentally intertextual (Landow, 1992), such that online readers often engage in intertextual meaning making (Goldman & Scardamalia, 2013). Thus, strategic readers in an online setting can be characterized as intertextual readers, who keep in mind their responsibilities in identifying, comprehending, and using multiple sources selectively and coherently. We view reading as a process of constructing meaning in which a reader performs a wide array of actions for accessing, comprehending, and using multiple sources. Indeed, the importance of these actions is undergirded by research in the last two decades, which affords several conceptualizations of reading multiple sources. These include intertext model building (Perfetti, Rouet, & Britt, 1999), document use (Rouet, 2006), multiple-documents literacy (Anmarkrud, Bråten, & Strømsø, 2014), sourcing and corroborating (Wineburg, 1991), and learning from multiple sources (Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012). These models and frameworks complement and sometimes compete with each other. Yet they share a common ground, and they influence our definition of multiple source reading: Multiple source reading requires more than the comprehension of one text and its content; it is construed more completely as the course in which readers analyze meaningful linkages between and across the texts identified as important sources, determine the role and contribution of each source to an understanding of what’s being read, and discern credible sources and their information value as they construct meaning. Our definition honors readers’ constructive and critical processes, broadening the idea of reading to more fully account for a successful, engaged processing of multiple sources. Accordingly, this definition requires a specification of the knowledge

Strategic Processing  •  135

and strategies that readers use for constructing meaning from the sources they access, comprehend, and use to achieve their goals, which are planned and developed before and while reading. Although word-level processing, content learning, and comprehension monitoring within one text are still crucial skills, they may not be sufficient for successful multiple source reading. A premise of our definition is that multisource readers, in many situations, are motivated to think outside the given text. These readers are mindful of the possibility of accessing the texts that are potentially useful to them and become sources for their learning (Cho, 2014), and actively seek cohesiveness in their transitions between multiple sources (Salmerón & Garcia, 2011). Focused readers invest their time and effort in analyzing the trustworthiness of sources and weaving the ideas from each source into a coherently represented understanding (Bråten, Strømsø, & Salmerón, 2011). At the same time, successful readers must selfassess their work in an ongoing manner to best deal with the challenges of reading in a multisource environment (Winters, Greene, & Costich, 2008). That said, we do not make an arbitrary and binary distinction between “old” and “new” forms of reading. Although we acknowledge that some aspects of online environments differ from paper-based environments, such as greater and faster access to information and sources through numerous links and pathways, we are concerned with claims that there is a new species of cognitive processing for online reading that is distinct from conventional, print-based reading. Rather, whether online or print-based, reading often involves constructing meaning from multiple sources and sustained control of attention. Successful multisource readers regulate their reading by self-assessing the goals, problems, tasks, and textual environments independently of whether the content is communicated digitally or on paper.

STRATEGIC PROCESSING OF MULTIPLE SOURCES ONLINE This definition of reading informs our ideas about strategic processing of multiple sources in nature. What is a strategy? What are the characteristics of strategic readers? What is non-strategic processing? These are all important theoretical and empirical questions. Here, we briefly contemplate the nature of strategic processing with regard to reading for constructing meaning through access to, and integration of, multiple sources online. To begin with, we point out a difference between “strategy” and the generic term “process.” As Pressley and his colleagues described, “a strategy is composed of cognitive operations over and above the processes that are natural consequences of carrying out the task . . . (strategies) achieve cognitive purposes . . . and (are) potentially conscious and controllable activities” (Pressley, Forrest-Pressley, Elliott-Faust, & Miller, 1985, p. 4). In this account, processes are components of larger thinking operations, regardless of their levels of sophistication, organization, and intentionality, each of which is determined by the nature of the cognitive task performance, and readers’ level of development. In other words, processes are neither necessarily interconnected during reading nor coherently planned in the beginning of reading, and they may not necessarily work in a goal-oriented manner. In contrast, strategies are the deliberate applications of one’s processes, which are more likely to accompany reader intention and consciousness as the underlying forces (or motivations) for the operation of thinking. Strategies, although planned and implemented in an organized way, may not always guarantee learning success because they are only the means to the end of reading. Nevertheless, the strategy use is what good readers must go through to achieve

136  •  Cho et al.

their reading goals (Pressley & Afflerbach, 1995), particularly provided the challenging task and situation of negotiating multiple sources online (Cho & Afflerbach, 2017). Consistent with the account of strategies, Afflerbach, Pearson, and Paris (2008) drew distinctions between “strategy” and “skill” in reading, and further described strategies related to the reader effort, intentionality, and goal orientation that determine the fluency and effectiveness of strategic processing. They explained that strategies are skills under conscious control, akin to two sides of the same coin. Within this conception, the same activity can be either a more deliberate strategy or a more automatic skill. Acts of reading performed in a goal-directed manner are therefore composed of both strategies and skills, which operate as moment-to-moment processes depending upon reader knowledge and ability (e.g., domain knowledge, reading proficiency, metacognitive knowledge), reading goals (e.g., to summarize, to understand, to share, to use the texts read), and types and purposes of texts (e.g., text structures, forms of information representation, author intentions such as explanation or persuasion). As well as the intentionality of reader actions, we also consider our evolving understanding of reading and the involved thinking to be important for our conception of reading strategies (and skills). Our observations of skilled readers conducting an online inquiry task (Cho, 2014; Cho & Afflerbach, 2015; Cho, Woodward, Li, & Barlow, 2017) describe reading strategies (and skills) as the cognitive activities of agentive sensemakers that are situated with reader goals, to be pursued within the textual environment. This contemplation reflects the recognition of the particularities of text environments, as our definition of reading underscores that meaning is constructed by readers accessing, comprehending, and using multiple sources. In this sense, strategic processing occurs as readers select, coordinate, and apply diverse goal-oriented thoughts and actions in response to the challenging environments of multiple source use. A key aspect of strategy use, then, is that readers must be responsive to their text environments. Responsivity, featured in comprehensive research syntheses of reading strategies (Cho & Afflerbach, 2017; Pressley & Afflerbach, 1995), explains how readers interact with available (or yet-to-be-identified) sources while keeping evolving goals and reading priorities in their minds. Therefore, responsivity is the hallmark of strategic readers who opportunistically determine their selection and coordination of reading strategies and skills. Strategic processing plays a central role in successfully accessing, comprehending, and using a variety of different sources in online environments. This has been consistently demonstrated in multiple strands of research, including reading (Cho, 2014; Hartman, 1995), cognitive psychology (Bråten & Strømsø, 2003; Britt & Rouet, 2012; Wiley et al., 2009), disciplinary practices (Leinhardt & Young, 1996; Wineburg, 1991), multimedia learning (Hill & Hannafin, 1997; McNamara & Shapiro, 2005), and information sciences (Agosto, 2002; Kuiper, Volman, & Terwel, 2005). Recent research on strategic reading evokes a reframing of the core reading strategies to serve diverse functions and purposes in response to the prominent features of online text environments, such as nonlinear, multimodal, and multilayered spaces of information (Cho & Afflerbach, 2017; Leu, Kinzer, Coiro, & Cammack, 2004). Readers’ understanding of texts develops from processing and retaining information to the orchestration of choosing, evaluating, and integrating multiple sources in a coherent manner. Further, it involves the application of newly developed knowledge and perspectives toward achieving their reading goals.

Strategic Processing  •  137

Research has documented that strategic readers are active in locating information, cross-textual linking, source evaluation, and self-monitoring. Table 8.1 samples empirical studies and their versions of strategies for multiple source reading in a digital context. Table 8.1  Multisource Reading Strategies in Digital Contexts. Strategic processing

Example strategies (Relevant studies)

Constructiveintegrative processing

Exploring goal-relevant information spaces by taking novel approaches to information searching: Generating keywords, refining search terms informed by prior knowledge (e.g., topic knowledge, domain knowledge, inquiry knowledge, system knowledge) and previous information searches, examining search entries to figure out what information is available or missing online, and changing and specifying targeted sources (Cho, 2014; Rogers & Swan, 2004). Selecting multiple hyperlinks connected to different web sources and building a coherent mental model of reading paths to obtain useful sources: Going back and forth between pages, moving to a different part of the online space by selectively choosing hyperlinks, reserving or rejecting chosen sources, and revisiting search results pages to reassess useful sources (Bilal & Kirby, 2002; Hölscher & Strube, 2000; Salmerón, Kintsch, & Canãs, 2006; Salmerón & García, 2011; Tu, Shih, & Tsai, 2008). Specifying the nature of tasks (e.g., time constraints, available resources, guiding questions, outcome products) and the domain of knowledge (e.g., particular themes and topics, subareas of science disciplines, diverse dimensions of life experience) to identify important ideas across sources (Brand-Gruwel, Wopereis, & Vermetten, 2005): Assigning the role of each source in meaning making and task completion according to its contribution to a global understanding of the document set (Barzilai, Tzadok, & Eshet-Alkalai, 2015). Elaborating content understanding (e.g., accuracy, consistency, and cohesiveness) by using the text ideas gained from different sources: Information to information, source to source, and source to prior knowledge (Anmarkrud, McCrudden, Bråten, & Strømsø, 2013; List, Grossnickle, & Alexander, 2016; Sánchez & García-Rodicio, 2013). Questioning and challenging self-initiated sources and their information to infer about, compare and contrast, and examine multiple facts, arguments, and perspectives that may often conflict with each other (Brand-Gruwel, Wopereis, & Vermetten, 2005; Cho, Woodward, & Li, 2017; Kiili, Laurinen, Marttunen, & Leu, 2012). Constructing a meta-representation of the selected sources and their information while reconciling inconsistencies across the sources in relation to task goals, demands, and situations (Braasch, Rouet, Vibert, & Britt, 2012; Stadtler, Scharrer, Brummernhenrich, & Bromme, 2013). Recognizing and attending to the explicit markers of source reliability and trustworthiness: Author credentials (e.g., the degree of author expertise indicated by areas of work, professions, affiliations, positions) and source features (e.g., when, where, how, and by whom a source was created, used, and promoted) (Kammerer, Bråten, Gerjets, & Strømsø, 2013; Stadtler & Bromme, 2007). Comparing the analyses of author identification and source information done for different information sources to make accurate prediction of the reliability of a particular source (Cho, 2014; Strømsø, Bråten, Britt, & Ferguson, 2013; Wiley et al., 2009).

Searching for and locating information

Identifying and learning important ideas featuring across sources

Building new knowledge and perspectives

Criticalanalytical processing

Sourcing

(Continued)

Table 8.1  (Continued) Strategic processing Textual analysis

Source judgment

Metacognitivereflective processing

Controlling the construction of the reading path

Controlling the construction of meaning

Monitoring the self

Example strategies (Relevant studies) Implementing textual inquiry strategies (e.g., parsing out what the author presents in text into argumentative components, analyzing the development of ideas and claims) to assess a self-chosen source’s content validity, information trustworthiness, hidden assumptions and biases, and text-implicit purposes in accordance with the entire process of reading to construct meaning, build knowledge and perspectives, and achieve personal and social goals (Anmarkrud, Bråten, & Strømsø, 2014; Strømsø & Bråten, 2014). Valuing, and also challenging, the suitability of information organization, writing styles, word choices, voices and tones, rhetorical devices, design features, and other methods of information representation (Braasch, Bråten, Strømsø, Anmarkrud, & Ferguson, 2013). Collecting evidence from the results of textual analysis against the textual hypothesis having emerged from anticipatory sourcing: Expecting whether a certain source or content will be adequate or inadequate (Johnson, Azevedo, & D’Mello, 2011). Determining whether current and previous textual choices are productive by using the comprehensive mental model of text usefulness (accuracy, validity, reliability, etc.) and rating the importance of each source in the broader context of multisource text environments (Barzilai & Zohar, 2012; Cho, Woodward, & Li, 2017). Planning and managing reading paths to meet both original and emerging (sub)goals for reading by taking into account textual and contextual factors (e.g., task features and environments, projected amount of needed time and effort, the nature of topics and knowledge domains) (Azevedo, Moos, Greene, Winters, & Cromley, 2008; Greene & Azevedo, 2007; Stadtler & Bromme, 2007; Zhang & Duke, 2008). Controlling the scope and speed of cyclical processes in information access including searching, predicting, selecting, analyzing, sensemaking, and evaluating (Coiro & Dobler, 2007; Greene & Azevedo, 2007). Monitoring the progress of meaning construction (time allocation, attention allocation, refocusing, etc.) and facilitating the reading progress toward the goal attainment (note-taking, pausing, rereading, revisiting, summarizing, questioning, etc.) (Greene & Azevedo, 2007; Greene et al, 2015; Goldman et al., 2012). Regulating sensemaking strategies by recognizing the boundary of information space in which reading takes place: Imposing the understanding of its affordances and constraints in strategic decisionmaking (Cho, 2014; Johnson, Azevedo, & D’Mello, 2011). Rehearsing and reflecting on the strategic actions in relation to the purpose of reading: Evaluating one’s reading performance and self-questioning about the current direction and status of reading in an ongoing process (Cho, Woodward, Li, & Barlow, 2017; Kiili, Laurinen, & Marttunen, 2009). Perceiving and identifying the self as a digital multisource text reader with respect to prior knowledge, cognitive capacity, reading experience, orientation toward knowledge and knowing, and agency as an active meaning-maker in complex and untested online text environments (Azevedo, Moos, Greene, Winters, & Cromley, 2008; Greene & Azevedo, 2007).

Strategic Processing  •  139

As we will illustrate in detail, the table presents strategies in relation to constructiveintegrative, critical-analytical, and metacognitive-reflective processing. We note that strategic readers carefully orchestrate these cognitive and metacognitive activities, which often leads to successful reading. Notably, the use of a particular strategy may inform the selection and application of subsequent strategies, forming a larger processing chain (Cho, Woodward, Li, & Barlow, 2017). The operation of such strategic decision-making is monitored in relation to readers’ reading goals, a developing understanding of task features, available sources, and the environments (Britt & Rouet, 2012). In what follows, we further examine the interactions of these strategies in the situation of multisource reading online.

HOW STRATEGIC PROCESSING COMPLEMENTS ACCESSING, COMPREHENDING, AND USING MULTIPLE ONLINE SOURCES Multisource reading strategies may be categorized in relation to each strategy’s complexity and sophistication, the depth of the underlying thinking, and the ways in which strategies interact. As illustrated in Figure 8.1, we propose three layers of strategic processing across which readers’ cognitive and metacognitive strategies are recursively organized and mutually informed. They include constructive-integrative processing (e.g., searching for and locating information, identifying and learning important ideas from the sources, building new knowledge and perspectives), critical-analytical processing (e.g., sourcing, textual analysis, source judgment and evaluation), and metacognitive-reflective processing (e.g., regulating sensemaking processes, controlling path construction, assessing the self). Constructive-integrative processing directly contributes to the reader’s sensemaking of sources and their contents. Construction of a robust and reliable understanding is enhanced by the reader’s engagement in criticalanalytical processing to choose, filter, and judge the information value and the source authority. The reader’s metacognitive-reflective processing guides these two layers of strategic processing toward achieving the goals for reading.

Figure 8.1  Strategic processing of multiple online sources.

140  •  Cho et al.

As shown in Figure 8.1, these three layers of strategic processing have a mutual, formative influence on each other. For example, readers’ evaluation of an author’s expertise based in a particular area of knowledge and problem (i.e., critical-analytical processing) may help them identify and learn from sources that are both important and credible (i.e., constructive-integrative processing). These strategic processes are controlled under the function of readers’ monitoring and adjustment, as well as their capacity to take alternative approaches to reading according to their goals, prior knowledge and experience, and attitudes toward the particular problem and topic (i.e., metacognitive-reflective processing). Hence, readers’ critical-analytical strategies mediate their constructive-integrative processing, allowing for better identification and use of sources, as they accurately self-assess their own strategy use in a timely manner. Remember that the construction of meaning (and knowledge building) may rest on a shallow foundation if the reader is satisfied only with locating information in a piecemeal fashion, followed by cursory information processing. Consider a student reading a short text found on the Internet. The student’s goal is to determine the cause of the Johnstown Flood of 1889. This disastrous historical event occurred after the failure of the South Fork Dam on the Little Conemaugh River, which had been severely damaged by several days of heavy rainfall. A result of the dam’s failure was the loss of 2,209 citizens of Johnstown, Pennsylvania. A surface understanding of what caused the Johnstown Flood can be accomplished through the relatively simple acts of locating a source on the Internet, and comprehending a few paragraphs. However, constructing meaning through multisource reading goes further beyond easy-to-implement, surface-level processing. Strategic multisource readers create the opportunities to come to know about the cause of the Johnstown Flood, purposefully identifying multiple accounts of the event (e.g., timelines, a terrain map of Conemaugh Valley and Johnstown, a drawing of the embankment dam design, documentary narratives of poor dam maintenance, eyewitness accounts of the flood), and draw evidence from them to support a claim of whether the main cause is due to “acts of God,” human beings who neglected to maintain the dam properly, or interactions of both. Further, these readers make an effort to make sense of and examine competing perspectives as to the question of “What and who was to blame?” in order to establish their stance toward the interpretation of the context, conditions, and causal chains interwoven with the history. Such readers are conceived as multisource readers, able to recognize potential uses of different texts as important sources for learning, seek to find the sources, and integrate them for the sensemaking of the historical event. As such, robust meaning construction and knowledge building require readers to coordinate reading strategies to explore and understand multiple information sources, and to revise (modify, update, or even disrupt and rebuild) their prior understandings as they construct new knowledge. In the following sections, we describe how each of the three classes of strategic processing—cognitive-integrative, critical-analytical, and metacognitivereflective—operate individually and interactively during multisource, online reading. Constructive-Integrative Processing Strategic multisource reading is constructive and integrative in nature. This view is supported by Kintsch’s perspective on mental representations of written text (1998) and also by research on comprehending multiple documents (Perfetti et al., 1999; Rouet & Britt, 2010). Kintsch offers a compelling account of the process of

Strategic Processing  •  141

text comprehension. Text comprehension begins with reader actions to decode and understand explicit representations of written language at both micro and macrolevels in the text (i.e., textbase model). Strategic comprehension then moves along toward constructing a coherent understanding of text ideas, which is situational to the reader’s knowledge and goals (i.e., situation model). Perfetti, Rouet, Britt, and their colleagues proposed another level of mental model that represents textual relations—whether texts complement, conflict with, or refute each other— as the central component of a situational understanding across the set of texts (i.e., intertext model). Building on these foundations, we propose the series of readers’ strategic processes for source identification and path construction as crucial part of such complex mental model-building processes (Cho & Afflerbach, 2017). That is, our conception of constructive-integrative processing as the central layer of strategic multisource reading includes building coherent mental models that integrate not only text ideas (situation model) and textual relations (intertext model), but also the paths on which the sources are accessed, identified, and used (reading path model). Strategic multisource readers must be conscious about the construction of reading paths. A reading path reflects when, where, and how a reader chooses and sequences sources while navigating within, between and across the textual spaces online. Strategic multisource readers construct coherent reading paths that are unique to their learning goals. Therefore, building a model of path construction in an ongoing manner must count as a constructive and integrative process, as it requires a construction of the mental model that reflects the multiple links and routes of information access and reader attention as well as an integration of knowledge and goals to elaborate and refine the model. Constructive-integrative processing begins with information location. Locating information is a prerequisite for meaning making because it contributes to readers’ building of their own text environment, with which they interact to learn important ideas and construct meaning (Naumann, 2015). Yet information location (e.g., finding out where targeted information is stored) must develop toward information identification (e.g., a preliminary determination of whether the located information is what is being sought to meet the goals for reading). Information identification can be achieved when readers are able to predict whether their link selections would lead to useful sources by examining multiple facets of information authority and quality (Braasch et al., 2009). They may trace back in the course of reading to inspect what they have found and determine what they need more, based upon the focus of reading. These strategies help readers make decisions about what and how to access, examine source features as to their importance and value, and determine a reading path that is relevant to and reliable for their meaning-making purposes and reading goals (Salmerón, Canãs, Kintsch, & Fajardo, 2005). Constructive-integrative strategies are then used to examine and build specific intertextual connections across the sources, as readers gradually move toward knowledge building (DeSchryver, 2015; Rouet & Britt, 2010). Specifically, constructing meaning from multiple sources online is influenced by readers’ commitment to building cross-textual linkages. Multisource readers in an online environment need to be cognizant that they need to identify cross-textual linkages to create cohesive meanings from the sources in accordance with their goals. The bottom line is that intertextuality is built in the mind of the reader, who is active in the use of linking strategies between

142  •  Cho et al.

and across diverse sources, and also in drawing evidence from such cross-textual linkages to support their development of knowledge and perspectives. Critical-Analytical Processing Another layer of strategic multisource processing is critical-analytical processing— how readers evaluate information and sources to facilitate their meaning making. The idea of critical-analytical processing is informed by the research literature that examined readers’ reasoning not only with information per se but about the sources of information. We first consider sourcing strategies. In his expert–novice study, Wineburg (1991) identified sourcing as one of historians’ hallmark reading strategies. Sourcing is “the act of looking first to the sources of the document before reading the body of the text” (p. 77) and provides “anticipatory framework for the subsequent encoding of text” (p. 79). This is typified in situations where readers predict when and where a web source is created and posted by whom prior to clicking on the hyperlink and reading the body of the text (Braasch et al., 2009; Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012). For strategic online readers, sourcing is managed as momentto-moment processes as they face a continual task to deal with the links and texts that are often time recognizably discrepant and conflicting (Braasch & Bråten, 2017). Research on accomplished readers further indicates that anticipatory sourcing must be followed upon by textual analysis and confirmatory source judgment (Pressley & Afflerbach, 1995). In an Internet setting, for example, as they select a particular link and the connected text, strategic readers shift their reading focus from the tentative judgment of the possible value of the link and text to an indepth analysis of the main ideas and supporting details represented in the text for its validity and plausibility (e.g., analyzing claim–evidence relations, logical flows of ideas, fact-based evidentiary reasoning). The results of such deep-level analysis represent a merging of an initial source judgment and the more complex judgment of the source authority and the overall information value. This can result from the detection and investigation of the often hidden clues such as subtexts, concealed purposes, intended audiences, deceptive language, and manipulated facts. In sum, critical-analytical processing elicits a recursive use of sourcing, textual analysis, and source judgment. Critical-analytical processing is demanding. For example, a majority of adolescent readers seem to be ill-prepared to learn in uncontrolled online information environments. These readers rarely evaluate the digital sources that they use, or fail to distinguish between fake and trustworthy sources (Stanford History Education Group, 2016). This is in contrast to their proficiency in basic information-managing skills (Organisation for Economic Co-operation and Development, 2011). One cause of such processing impairments may stem from the nature of online text environments. For example, the Internet can baffle readers when there is little or no evidence to use in making a source judgment. That is, clues used to infer an author’s expertise and related source authority may be minimal, difficult to locate, purposely obfuscating, or absent in online settings. With such texts and sources, readers may opt for an ease of information access and comprehension over the seeking of knowledge founded upon profundity, provenance, authorship, and ownership of information (Hofer, 2004; McCrudden, Stenseth, Bråten, & Strømsø, 2016).

Strategic Processing  •  143

Such non-critical reading habits fundamentally stem from readers’ underdeveloped strategies and skills for critical-analytical processing, although they are also influenced by the undefined space of information online. Novice readers are often focused on surface markers, such as web addresses or dates of copyrights that are evident and quickly noticeable (Coiro, Coscarelli, Maykel, & Forzani, 2015; Tabatabai & Shore, 2005). In contrast, in order to find, analyze, and judge the multisource online information, accomplished readers consider the author (e.g. expertise, background, known agenda), text content (e.g., ideas and claims), and rhetoric (e.g., logic and reasoning). Should they fail to find sufficient evidence for the judging of the source features, accomplished readers actively conduct follow-up searches online and further determine whether their initial hypotheses about author and text are tenable. Therefore, a strategic cycle of sourcing, textual analysis, and source judgment contributes to the reader’s informed decisions about their actions to meet emerging information needs and developing reading goals. All readers of multiple sources online may face an interesting paradox (Lankes, 2008; Metzger & Flanagin, 2013). Online readers may have the privilege of so-called information self-sufficiency, which describes information users (in this case readers) who do not necessarily need assistance and advice from traditional intermediaries to find what to read (librarians, teachers, clerks, etc.), because of the abundance of information in texts and the ease of information accessibility. However, as Lankes (2008) points out, this information self-sufficiency presents readers with a challenge in that what is available online may or may not be credible. The variability in vetting, ranging from comprehensive to none, complicates matters for readers and adds to the paradox. Readers must be responsible for critically evaluating the nature of sources, the availability of sources, and the location of sources. However, these tasks can be arduous. Of course, all vetting is conducted in relation to a reader’s specific set of criteria and preferences. Even in the “old days” of single, traditional text reading, there were plentiful unreliable sources in need of reader scrutiny. It is more the issue of ease of online access to a surfeit of text sources on a continuum of text legitimacy. Nevertheless, given the complicatedness of online text environments, careful judgment of sources and finding of evidence to support the judging process remain critical to online multisource readers, who are conscious about the paradox and the situation of ongoing information credibility assessment (Cho, Woodward, & Li, 2017). Metacognitive-Reflective Processing Metacognitive-reflective processing is the third necessary component of strategic multisource reading. Roles and functions of this processing are informed by theoretical accounts of strategic reading; they include metacognition (Veenman, 2016), comprehension monitoring (Kinnunen & Vauras, 2010), and self-regulation (Tonks & Taboada, 2011) as central features of strategic reading. Notably, the recent body of work on self-regulatory processes in hypermedia environments reveals the importance of these metacognitive-reflective processes that impact students’ learning with the complex system as well as the challenges the students face in the processing of nonlinear representation of information (Azevedo, Moos, Johnson, & Chauncey, 2010). In multisource environments, strategic readers must distribute their attention and effort according to the demands of reading, because navigating the surfeit of online

144  •  Cho et al.

sources threatens cognitive overload (Cho, 2014). Failure to allocate attention and effort can frustrate readers unable to orient themselves; readers in such complex spaces and problematic situations can be lost in a maze of information (e.g., What webpage to access? What to look at? What to click next?). Their reading can become ineffective and acritical due to their lacking attention to the goal of reading (e.g., What was I supposed to find and read about? What was my reading plan?). Disorientation problems may disrupt learning that would otherwise be constructive and critical. Strategic readers intentionally reflect on their thinking processes during reading, which helps them to detect processing problems and difficulties and be ready to address the problems with alternative strategic actions. Metacognitive-reflective readers focus on monitoring the construction of reading paths (e.g., controlling link selection, regulating text selection, deciding whether to stop reading or go further) as well as the construction of meaning (e.g., checking the current status of their understanding, identifying their information needs). Monitoring of these constructive processes informs the actions for identifying texts because readers know which information has been found and what additional work is needed in relation to reader goals. Monitoring also helps readers comprehend information from across multiple sources because they have a better sense of reading progress. Thus, readers can allot their attention between processing information within each text and processing information across texts. In addition, monitoring the self is crucial, particularly to readers applying what they know and believe about truth, facts, or opinions when judging the reliability and trustworthiness of information (Cho et  al., 2017; Hofer, 2004). More importantly, self-monitoring may improve learning and hence yield better reading outcomes (Dunlosky, Hertzog, Kennedy, & Thiede, 2005) because it guides self-regulation by which readers can maintain their focus on productive learning (Azevedo & Hadwin, 2005). Taken together, as described in the above accounts, research suggests that readers’ intentional and responsive strategy use plays a crucial role in reading multiple sources online. Sophisticated readers use diverse strategies, monitoring the function of each, and deciding upon alternative choices of strategies if progress is halted. These readers’ cognitive activities are performed interactively, and the influences of strategy uses flow in multiple directions between each layer of strategic processing—that is, constructive-integrative, critical-analytical, and metacognitive-reflective processing. A combination of these strategic processes are necessary and essential to success in reading, working altogether to serve diverse functions and purposes with different levels of reader attention and thinking complexities at various moments of reading with multiple sources. Thus, successful readers understand that strategic processes may influence each other’s operation, and their thoughts and actions become increasingly sophisticated toward higher-level processing.

IMPLICATIONS FOR THEORY, RESEARCH, AND PRACTICE In this chapter, we identified three important categories of strategic processing used in reading multiple sources online: constructive-integrative, critical-analytical, and metacognitive-reflective. We situated these acts of meaning making in relation to readers’ goals, affording the opportunity to advocate for attention to the strategies

Strategic Processing  •  145

typically associated with constructing meaning from text, and also those strategies that help readers determine how their ongoing, online work compares with their reading goals. Further, we described the manner in which talented readers of multiple sources online chain together their strategies for maximum performance. We also noted that there are considerable similarities between the “newer” online reading strategies, and those used in “traditional” (print) reading. In our estimation, true differences in traditional and online reading center around the number of texts available, the ease of access to these texts, the quality of the texts, and means to identify the quality of texts. Each of these differences helps underline the importance of strategies that include critiquing and evaluating, and monitoring comprehension. In any act of reading, strategies feature in the achievement of a reader’s goals. Going forward, we advocate for research that continues to investigate the contextspecific and nuanced nature of strategy use with online, multiple sources. One potentially fruitful avenue of research will describe how readers’ individual differences interact with and influence the use and success of strategies. For example, readers’ prior knowledge—related to text topic, a domain of knowledge, online reading experiences, and multiple sources—has a significant impact on the process and product of reading. This prior knowledge and the sources of that knowledge are factors that merit ongoing research (Kendoeou & O’Brien, 2016). Metacognition is another crucial factor in successful, strategic processing (Veenman, 2016). We believe that online multiple source reading presents an ideal environment in which to study the complementary roles of reading strategies and metacognitive strategies. Teaching reading strategies is a challenging but attainable task for teachers who are knowledgeable about the nature and process of strategic reading, when and where particular strategies can be used and coordinated, and how such strategic reading can be assessed. Many of the strategies described in this chapter fit well with the strategies required to succeed at curriculum and assessment initiatives, including the Common Core State Standards (National Governors Association Center for Best Practices & Council of Chief State School Officers, 2010). We would be remiss to not emphasize the importance of metacognitive strategies that are necessary for coordinating, regulating, and working through multisource text reading situations. Metacognitive strategies are sometimes missing in reading curricula, and they must be featured, front and center, in any approaches to reading instruction what would ably prepare students for online multisource reading. Finally, we note that how reading “matters” (or not) in the development of an individual’s beliefs and values is a fruitful avenue of inquiry. For example, readers’ orientations, attitudes, and beliefs in relation to knowledge (what counts as knowledge) and knowing (how one comes to know) may guide their critical processing of sources in certain directions, as a substantial body of research has supported important relationships between student learning, cognitive processing, and personal epistemology (Sandoval, Greene, & Bråten, 2016). Personal beliefs about knowing and learning may change toward developing more sophisticated habits of thinking when learners are allowed to participate in diverse situations of reading to hypothesize about, test, and reflect on how their beliefs may or may not work in a productive process of meaning making with different textual environments. We are not so naïve to believe that a text that is well chosen, amply vetted, and strategically read will single-handedly influence

146  •  Cho et al.

a particular reader’s mind. However, research in this area can help us identify when, where, and how these opportunities arise.

CONCLUSION The growth of the Internet brings an unprecedented number of new and hybrid sources of online information. These sources populate a vast, uncontrolled space of information, and often they are not vetted for accuracy or reliability. Accordingly, successful readers of multiple sources online must be able to decide what to read, how to access and integrate information, and when to use the sources of information. To construct knowledge in this challenging environment, strategic readers use and manage their acts of reading responsively while navigating the complex space of multiple sources online to achieve their goals. These readers use three general classes of strategies that we describe as constructive-integrative, critical-analytical, and metacognitive-reflective. It is not only the type of strategy, but the chaining together of these strategies in relation to the ever-changing reading landscape that marks successful online readers reading multiple texts. Reader responsivity to text sources and text content, tied to readers’ goals and the monitoring of progress toward goals, drives strategy choice. A burgeoning research base informs models of multisource reading, and in this chapter we offered our preliminary account of strategic processing for successful multisource reading. We hope that our account can help continuing inquiry into the teaching and learning of readers’ deliberate and critical strategy use to identify and learn important and reliable sources for building new knowledge and perspectives, as readers are situated in a multisource digital-text environment. As we consult the research and theory that focus specifically on online reading, it is important to remember that there are considerable, related precedents in the research on “traditional” print text. A single, traditional (print) text can be more challenging than multiple online texts. Unsubstantiated claims, hyperbole, inaccurate information, conflicting points of view, and subtexts exist in all forms of text, and all have received scrutiny from successful strategic readers. Research and theory that honor both “traditional” and “new” forms of text (and reading) will continue to identify unique and shared aspects of strategic reading across the forms. Teaching strategies for online reading of multiple sources should be informed by the successes of teaching “traditional” reading strategies. Inferencing, summarizing, predicting, analyzing, and evaluating are strategies that serve readers of both multiple online texts, and single “traditional” texts. Instructional methods that focus on explaining, modeling, and thinking aloud about strategy use should help teachers help students develop into successful online readers.

REFERENCES Afflerbach, P., Pearson, P. D., & Paris, S. G. (2008). Clarifying differences between reading skills and reading strategies. The Reading Teacher, 61(5), 364–373. Afflerbach, P., & VanSledright, B. (2001). Hath? Doth? What!: The challenges middle school students face when reading innovative history text. Journal of Adolescent and Adult Literacy, 44, 696–707. Agosto, D. E. (2002). A model of young people’s decision-making in using the Web. Library & Information Science Research, 24, 311–341.

Strategic Processing  •  147 Anmarkrud, O., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. Anmarkrud, Ø., McCrudden, M. T., Bråten, I., & Strømsø, H. I. (2013). Task-oriented reading of multiple documents: online comprehension processes and offline products. Instructional Science, 41(5), 873–894. Azevedo, R., & Hadwin, A. F. (2005). Scaffolding self-regulated learning and metacognition: Implications for the design of computer-based scaffolds. Instructional Science, 33(5–6), 367–379. Azevedo, R., Moos, D. C., Greene, J. A., Winters, F. I., & Cromley, J. G. (2008). Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia?. Educational Technology Research and Development, 56(1), 45–72. Azevedo, R., Moos, D. C., Johnson, A. M., & Chauncey, A. D. (2010). Measuring cognitive and metacognitive regulatory processes during hypermedia learning: Issues and challenges. Educational Psychologist, 45(4), 210–223. Barzilai, S., Tzadok, E., & Eshet-Alkalai, Y. (2015). Sourcing while reading divergent expert accounts: Pathways from views of knowing to written argumentation. Instructional Science, 43(6), 737–766. Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online sources. Cognition and Instruction, 30(1), 39–85. Bilal, D., & Kirby, J. (2002). Differences and similarities in information seeking: Children and adults as Web users. Information Processing & Management, 38(5), 649–670. Braasch, J. L., & Bråten, I. (2017). The Discrepancy-Induced Source Comprehension (D-ISC) model: Basic assumptions and preliminary evidence. Educational Psychologist, 1–15. DOI: 10.1080/00461520.2017.1323219. Braasch, J. L., Bråten, I., Strømsø, H. I., Anmarkrud, Ø., & Ferguson, L. E. (2013). Promoting secondary school students’ evaluation of source features of multiple documents. Contemporary Educational Psychology, 38(3), 180–195. Braasch, J. L. G., Lawless, K. A., Goldman, S. R., Manning, F. H., Gomez, K. W., & MacLeod, S. M. (2009). Evaluating search results: An empirical analysis of middle school students’ use of source attributes to select useful sources. Journal of Educational Computing Research, 41(1), 63–82. Braasch, J. L., Rouet, J. F., Vibert, N., & Britt, M. A. (2012). Readers’ use of source information in text comprehension. Memory & Cognition, 40(3), 450–465. Brand-Gruwel, S., Wopereis, I., & Vermetten, Y. (2005). Information problem solving by experts and novices: Analysis of a complex cognitive skill. Computers in Human Behavior, 21, 487–508. Bråten, I., & Strømsø, H. I. (2003). A longitudinal think-aloud study of spontaneous strategic processing during the reading of multiple expository texts. Reading and Writing: An Interdisciplinary Journal, 16, 195–218. Bråten, I., & Strømsø, H. I., & Salmerón, L. (2011). Trust and mistrust when students read multiple information sources about climate change. Learning and Instruction, 21, 180–192. Britt, M. A., Goldman, S. R., & Rouet, J. F. (Eds.) (2013). Reading: From words to multiple texts. New York: Routledge. Britt, M. A., & Rouet, J.-F. (2012). Learning with multiple documents: Component skills and their acquisition. In J. R. Kirby & M. J. Lawson (Eds.), Enhancing the quality of learning: Dispositions, instruction, and learning processes (pp. 276–314). Cambridge: Cambridge University Press. Cho, B.-Y. (2014). Competent adolescent readers’ use of Internet reading strategies: A think-aloud study. Cognition and Instruction, 32(3), 253–289. Cho, B.-Y., & Afflerbach, P. (2015). Reading on the Internet: Realizing and constructing potential texts. Journal of Adolescent and Adult Literacy, 58, 504–517. Cho, B.-Y., & Afflerbach, P. (2017). An evolving perspective of constructively responsive reading comprehension strategies in multilayered digital text environments. In S. Israel (Ed.), Handbook of research on reading comprehension (2nd ed., pp. 109–134). New York, NY: Guilford. Cho, B.-Y., Woodward, L., & Li, D. (2017). Epistemic processing when adolescents read online: A verbal protocol analysis of more and less successful online readers. Advance Online Publication. Reading Research Quarterly. DOI: 10.1002/rrq.190. Cho, B.-Y., Woodward, L., Li, D., & Barlow, W. (2017). Examining adolescents’ strategic processing during online reading with a question-generating task. Advance Online Publication. American Educational Research Journal, 54(4), 691–724. DOI: 10.3102/0002831217701694. Coiro, J., Coscarelli, C., Maykel, C., & Forzani, E. (2015). Investigating criteria that seventh graders use to evaluate the quality of online information. Journal of Adolescent and Adult Literacy, 59(3), 287–297.

148  •  Cho et al. Coiro, J., & Dobler, E. (2007). Exploring the online reading comprehension strategies used by sixth-grade skilled readers to search for and locate information on the Internet. Reading Research Quarterly, 42(2), 214–257. DeSchryver, M. (2015). Higher order thinking in an online world: Toward a theory of web-mediated knowledge synthesis. Teachers College Record, 116(12), 1–44. Dunlosky, J., Hertzog, C., Kennedy, M., & Thiede, K. (2005). The self-monitoring approach for effective learning. Cognitive Technology, 10, 4–11. Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47(4), 356–381. Goldman, S. R., & Scardamalia, M. (2013). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities. Cognition and Instruction, 31(2), 255–269. Greene, J. A., & Azevedo, R. (2007). Adolescents’ use of self-regulatory processes and their relation to qualitative mental model shifts while using hypermedia. Journal of Educational Computing Research, 36(2), 125–148. Greene, J. A., Bolick, C. M., Jackson, W. P., Caprino, A. M., Oswald, C., & McVea, M. (2015). Domain-specificity of self-regulated learning processing in science and history. Contemporary Educational Psychology, 42, 111–128. Hartman, D. K. (1995). Eight readers reading: The intertextual links of proficient readers reading multiple passages. Reading Research Quarterly, 30(3), 520–561. Hill, J. R., & Hannafin, M. J. (1997). Cognitive Strategies and Learning from the World Wide Web. Educational Technology Research and Development, 45(4), 37. Retrieved from http://search.ebscohost.com/login.aspx? direct=true&db=eric&AN=EJ558449&site=ehost-live. Hofer, B. K. (2004). Epistemological understanding as a metacognitive process: Thinking aloud during online searching. Educational Psychologist, 39(1), 43–55. Hölscher, C., & Strube, G. (2000). Web search behavior of Internet experts and newbies. Computer Networks, 33(1), 337–346. Johnson, A. M., Azevedo, R., & D’Mello, S. K. (2011). The temporal and dynamic nature of self-regulatory processes during independent and externally assisted hypermedia learning. Cognition and Instruction, 29(4), 471–504. Kammerer, Y., Bråten, I., Gerjets, P., & Strømsø, H. I. (2013). The role of Internet-specific epistemic beliefs in laypersons’ source evaluations and decisions during Web search on a medical issue. Computers in Human Behavior, 29(3), 1193–1203. Kendoeou, P., & O’Brien, E. J. (2016). Prior knowledge: Acquisition and revision. In P. Afflerbach (Ed.), Handbook of individual differences in reading: Reader, text, and context (pp.  151–163). New York, NY: Routledge. Kiili, C., Laurinen, L., & Marttunen, M. (2009). Skillful Internet reader is metacognitively competent. In L.T.W. Hin & R. Subramaniam (Eds.), Handbook of research on new media literacy at the K–12 level: Issues and challenges (pp. 654–668). Hershey, PA: IGI Global. Kiili, C., Laurinen, L., Marttunen, M., & Leu, D. J. (2012). Working on understanding during collaborative online reading. Journal of Literacy Research, 44(4), 448–483. Kinnunen, R. & Vauras, M. (2010). Tracking on-line metacognition: Monitoring and regulating comprehension in reading. In A. Efklides & P. Misailidi (Eds.), Trends and prospects in metacognition research (pp. 209–229). New York, NY: Springer. Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge: Cambridge University Press. Kuiper, E., Volman, M., & Terwel, J. (2005). The Web as an information resource in K-12 education: Strategies for supporting students in searching and processing information. Review of Educational Research, 75(3), 285–328. Landow, G. P. (1992). Hypertext: The convergence of contemporary critical theory and technology. Baltimore, MD: The Johns Hopkins University Press. Lankes, R. D. (2008). Trusting the Internet: New approaches to credibility tools. In M. J. Metzger & A. J. Flanagin (Eds.), Digital media, youth, and credibility (pp. 101–122). Cambridge, MA: MIT Press. Leinhardt, G., & Young, K. M. (1996). Two texts, three readers: Distance and expertise in reading history. Cognition and Instruction, 14(4), 441–486. Leu, D. J., Kinzer, C. K., Coiro, J. L., & Cammack, D. W. (2004). Toward a theory of new literacies emerging from the Internet and other information and communication technologies. In R. B. Ruddell & N. J. Unrau (Eds.), Theoretical models and processes of reading (5th ed., pp.  1570–1613). Newark, DE: International Reading Association.

Strategic Processing  •  149 List, A., Grossnickle, E. M., & Alexander, P. A. (2016). Profiling students’ multiple source use by question type. Reading Psychology, 37(5), 753–797. Macedo-Rouet, M., Braasch, J. L., Britt, M. A., & Rouet, J. F. (2013). Teaching fourth and fifth graders to evaluate information sources during text comprehension. Cognition and Instruction, 31(2), 204–226. McCrudden, M. T., Stenseth, T., Bråten, I., & Strømsø, H. I. (2016). The effects of topic familiarity, author expertise, and content relevance on Norwegian students’ document selection: A mixed-methods study. Journal of Educational Psychology, 108(2), 147–162. McNamara, D. S., & Magliano, J. P. (2009). Towards a comprehensive model of comprehension. In B. Ross (Ed.), The psychology of learning and motivation (pp. 297–384). New York, NY: Academic Press. McNamara, D. S., & Shapiro, A. M. (2005). Multimedia and hypermedia solutions for promoting metacognitive engagement, coherence, and learning. Journal of Educational Computing Research, 33(1), 1–29. Metzger, M. J., & Flanagin, A. J. (2013). Credibility and trust of information in online environments: The use of cognitive heuristics. Journal of Pragmatics, 59, 210–220. National Governors Association Center for Best Practices & Council of Chief State School Officers. (2010). Common Core State Standards for English language arts and literacy in history/social studies, science, and technical subjects. Washington, DC: NGA. Naumann, J. (2015). A model of online reading engagement: Linking engagement, navigation, and performance in digital reading. Computers in Human Behavior, 53, 263–277. Organisation for Economic Co-operation and Development. (2011). PISA 2009 results: Students online: Digital technologies and performance (Volume VI). Retrieved, October 11, 2016, from http://dx.doi. org/10.1787/9789264112995-en. Perfetti, C. A., Rouet, J.-F., & Britt, M. A. (1999). Toward a theory of documents representation. In H. v. Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 99–122). Mahwah, NJ: Lawrence Erlbaum Associates. Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Lawrence Erlbaum Associates. Pressley, M., Forrest-Pressley, D. L, Elliott-Faust, D., & Miller, R. (1985). Children’s use of cognitive strategies: How to teach strategies and what to do if they can’t be taught. In M. Pressley & J. C. Brainerd (Eds.), Cognitive learning and memory in children: Progress in cognitive development research (pp.  1–47). New York, NY: Springer. Rogers, D., & Swan, K. (2004). Self-regulated learning and Internet searching. Teachers College Record, 106(9), 1804–1824. Rouet, J.-F. (2006). The skills of document use: From text comprehension to web-based learning. New York, NY: Routledge. Rouet, J.-F., & Britt, M. A. (2010). Relevance processes in multiple document comprehension. In J. P. Magliano & G. Schraw (Eds.), Relevance instructions and goal-focusing in text learning (pp. 19–52). Greenwich, CT: Information Age Publishing. Salmerón, L., Canãs, J. J., Kintsch, W., & Fajardo, I. (2005). Reading strategies and hypertext comprehension. Discourse Processes, 40(3), 171–191. Salmerón, L., & García, V. (2011). Reading skills and children’s navigation strategies in hypertext. Computers in Human Behavior, 27, 1143–1151. Salmerón, L., Kintsch, W., & Canãs, J. J. (2006). Reading strategies and prior knowledge in learning from hypertext. Memory & Cognition, 34(5), 1157–1171. Sánchez, E., & García-Rodicio, H. (2013). Using online measures to determine how learners process instructional explanations. Learning and Instruction, 26, 1–11. Sandoval, W. A., Greene, J. A., & Bråten, I. (2016). Understanding and promoting thinking about knowledge: Origins, issues, and future directions of research on epistemic cognition. Review of Research in Education, 40, 457–496. Stadtler, M., & Bromme, R. (2007). Dealing with multiple documents on the WWW: The role of metacognition in the formation of documents models. International Journal of Computer-Supported Collaborative Learning, 2, 191–210. Stadtler, M., Scharrer, L., Brummernhenrich, B., & Bromme, R. (2013). Dealing with uncertainty: Readers’ memory for and use of conflicting information from science texts as function of presentation format and source expertise. Cognition and Instruction, 31(2), 130–150. Stanford History Education Group. (2016). Evaluating information: The cornerstone of civic online reasoning. Retrieved December 20, 2016, from https://sheg.stanford.edu/upload/V3LessonPlans/Executive%20 Summary%2011.21.16.pdf.

150  •  Cho et al. Strømsø, H. I., & Bråten, I. (2014). Students’ sourcing while reading and writing from multiple web documents. Nordic Journal of Digital Literacy, 9(2), 92–111. Strømsø, H. I., Bråten, I., Britt, M. A., & Ferguson, L. E. (2013). Spontaneous sourcing among students reading multiple documents. Cognition and Instruction, 31(2), 176–203. Tabatabai, D., & Shore, B. M. (2005). How experts and novices search the Web. Library & Information Science Research, 27, 222–248. Tonks, S. M., & Taboada, A. (2011). Developing self-regulated readers through instruction for reading engagement. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance (pp. 173–186). New York, NY: Routledge. Tu, Y. W., Shih, M., & Tsai, C. C. (2008). Eighth graders’ web searching strategies and outcomes: The role of task types, web experiences and epistemological beliefs. Computers & Education, 51(3), 1142–1153. van Dijk, T. A., & Kintsch, W. (1983). Strategies in discourse comprehension. New York, NY: Academic Press. Veenman, M. V. J. (2016). Metacognition. In P. Afflerbach (Ed.), Handbook of individual differences in reading: Reader, text, and context (pp. 26–40). New York, NY: Routledge. Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. A., & Hemmerich, J. A. (2009). Source evaluation, comprehension, and learning in Internet science inquiry tasks. American Educational Research Journal, 46(4), 1060–1106. Wineburg, S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87. Winters, F. I., Greene, J. A., & Costich, G. M. (2008). Self-regulation of learning with computer-based learning environments: A critical analysis. Educational Psychology Review, 20, 429–444. Zhang, S., & Duke, N. K. (2008). Strategies for Internet reading with different reading purposes: A descriptive study of twelve good Internet readers. Journal of Literacy Research, 40(1), 128–162.

9

THE ROLE OF VALIDATION IN MULTIPLE-DOCUMENT COMPREHENSION Tobias Richter and Johanna Maier1 university of wuerzburg, germany

INTRODUCTION AND PURPOSE Readers using the World Wide Web as a source for informal learning are often confronted with documents that provide partial and one-sided information supporting only one position in a controversy, or argue for divergent positions and viewpoints, or provide alternative and contradictory evidence for the same circumstance. This is especially true with topics of high social or individual relevance that are debated controversially in public (e.g., controversial political or socio-scientific issues). How do readers comprehend multiple documents with conflicting information? How do they achieve a coherent and consistent representation of controversially debated issues? Ideally, readers would form a documents model that adequately represents the content of each text that they read, in addition to the semantic and argumentative relationships between texts (see Britt, Rouet, & Durik, this volume; Perfetti et  al., 1999; Rouet & Britt, 2011). As such, they would also integrate information from the various texts into a coherent mental model of the controversial issue and weigh this information according to the perceived trustworthiness of the sources. However, ample research shows that readers’ actual processing of multiple documents and the resulting mental representations seldom come close to this ideal model of multipledocument comprehension. Unless readers are explicitly trained in sourcing strategies, readers often fail to spontaneously pay attention to characteristics of the source (von der Mühlen, Richter, Schmid, Schmidt, & Berthold, 2016). As a consequence, they fail to consider this information in comprehending multiple documents about controversial topics (e.g., disputed historical topics, Wineburg, 1991). In many cases, readers also fail to reflect on and incorporate the information from divergent perspectives and argumentative stances (Rouet, 2006), resulting in the construction of a one-sided representation of controversial issues (Britt, Perfetti, Sandak, & Rouet, 1999). 151

152  •  Richter and Maier

It is now a commonplace assumption in text comprehension research that readers usually opt for an adequate representation of the linguistic message content that suits their given purpose, which minimizes cognitive effort during comprehension (Ferreira, Bailey, & Ferraro, 2002). It seems that multiple-text comprehension is no exception to the rule. In this chapter, we advance the idea that readers rely on a general comprehension mechanism (interchangeably) called validation (Singer, 2013) or epistemic monitoring (Isberner & Richter, 2014a) to achieve a coherent, albeit onesided representation of multiple documents containing controversial information. Validation means that readers use their knowledge and beliefs plus the linguistic context to monitor the validity (i.e., the truth, plausibility, or consistency) of text information (Singer, 2013). When readers possess strong and accessible beliefs about a controversial issue, this mechanism continually generates implicit assessments of plausibility that indicate the degree of fit between a given piece of information and readers’ beliefs (for a similar definition of plausibility, see Connell & Keane, 2006, p. 98). We propose that these implicit plausibility assessments regulate comprehension and encoding of controversial issues in such a way that belief-consistent information has a processing advantage over belief-inconsistent information in comprehension and memory (text-belief consistency effect, e.g., Eagly & Chaiken, 1993; KnoblochWesterwick, & Meng, 2011; Maier & Richter, 2013a; Wiley, 2005). The remainder of this chapter is organized as follows: We will first discuss the concept of validation during comprehension that has been proposed as a general and routine comprehension mechanism. We will then sketch a simple two-step model of the cognitive processing of conflicting information in multiple documents (Richter, 2011; Richter & Maier, 2017) and discuss studies on multiple-text comprehension through the lens of this model. The chapter ends with a discussion of the implications of validation for research on multiple-text comprehension and educational practice.

COMPREHENSION AND VALIDATION OF TEXT INFORMATION Successful text comprehension involves not only the construction of a propositional text base that presents readers’ memory for text as a network of propositions but also a situation model of the text content (Johnson-Laird, 1983; van Dijk & Kintsch, 1983). The situation model is often conceived as a referential representation of the state of affairs described in a text that integrates text information with prior knowledge (van Dijk & Kintsch, 1983; Zwaan & Radvansky, 1998). Knowledge activation during situation model construction is largely a passive, memory-based process. That is, knowledge is passively triggered by concepts and propositions in the text (e.g., Albrecht & O’Brien, 1993; O’Brien & Myers, 1999), and it becomes reactivated if it sufficiently resonates as a result of a signal from currently read information (memory-based text processing; O’Brien & Myers, 1999; O’Brien, Rizella, Albrecht, & Halleran, 1998). Text comprehension research traditionally focused on the interplay of knowledge activation and integration during comprehension (e.g., Kintsch, 1988). However, several researchers have proposed the validation of text information as a third type of cognitive process routinely involved in comprehension (Cook & O’Brien, 2015;

The Role of Validation  •  153

Richter, 2015; Singer, 2006, 2013). The basic idea is that information from previously read text that is activated passively through memory-based processes is used not only to interpret and augment text information during comprehension, but also to assess its validity. A growing body of literature from language and text comprehension research supports the assumption of a passive validation process during comprehension. Results from different experimental paradigms suggest that readers evaluate the consistency of text information with prior knowledge and beliefs non-strategically; that is, even without an evaluative reading goal and also early in comprehension. For example, reading time experiments based on the inconsistency paradigm have indicated that readers tacitly check whether text information is consistent with the linguistic context (including pertinent world knowledge) during reading. A number of reading time experiments based on the inconsistency paradigm also support the assumption of routine validation processes (Albrecht & O’Brien, 1993; Kendeou, Smith, & O’Brien, 2013; Myers, O’Brien, Albrecht, & Mason, 1994; O’Brien et al., 1998). In several experiments based on this paradigm, participants read stories that include sentences about a protagonist’s actions (e.g., Mary ordered a cheeseburger) that were consistent or inconsistent with character traits (e.g., Mary is a vegetarian or eats fast food) introduced earlier in the story. Such inconsistencies are routinely detected under conditions that cause the relevant information to be (re-)activated by memory-based processes. With similar textual manipulations, researchers have shown that readers are also sensitive to spatial, causal, temporal, logical, and other kinds of situational inconsistencies (Albrecht & Myers, 1995; Lea, Mulligan, & Walton, 2005; O’Brien & Albrecht, 1992; Rinck, Hähnel, & Becker, 2001; Singer, 1993; Singer, 2006; Singer, Halldorson, Lear, & Andrusiak, 1992). Refining the results from reading time studies, eye-tracking experiments have shown that inconsistencies with readers’ world knowledge affect early fixation measures (such as first fixation durations) when the described situations touch upon readers’ typical experiences (Matsuki, Chow, Hare, Elman, Scheepers, & McRae, 2011; Staub, Rayner, Pollatsek, Hyönä, & Majewski, 2007). These findings support the idea that validation takes effect early in comprehension when readers possess strong and available knowledge and beliefs (for event-related potential (ERP) studies supporting the same conclusion, see Ferretti, Singer, & Patterson, 2008; Hagoort, Hald, Bastiaansen, & Petersson, 2004). A different approach that sheds light on the involuntary character of validation processes is the epistemic Stroop paradigm (Richter, Schroeder, & Wöhrmann, 2009). This paradigm affords opportunities to examine whether readers monitor violations of world knowledge in linguistic messages even when this activity hampers performance in their actual task. In an epistemic Stroop experiment, single words are presented at a fixed rate (e.g., 300 ms). The words successively form sentences. At some point, participants are prompted to make binary judgments unrelated to the sentence content. For example, participants judge the spelling of words (Richter et al., 2009), the font color (Isberner & Richter, 2013), or they respond to probe words (TRUE or FALSE) with different keys (Isberner & Richter, 2014b). These experiments have repeatedly shown that responses slow down when participants are required to give a positive response (e.g., YES or CORRECT) or respond to the TRUE probe after

154  •  Richter and Maier

invalid sentences compared to the same responses to valid sentences. This epistemic Stroop effect was obtained for outright false sentences (e.g., Computers have emotions, Richter et al., 2009; Isberner & Richter, 2014b) but also for sentences that were merely implausible in the context of a preceding sentence (e.g., Frank has a broken leg . . . He calls the plumber, Isberner & Richter, 2013). These results have advanced the findings from the reading time studies by showing that invalid sentences induce a negative response tendency. Readers not only experience comprehension problems when encountering implausible and inconsistent information but also implicitly judge the validity of the presented information. Two aspects of validation require further elaboration because they are important in understanding the role of validation in the comprehension of controversial information. First, the distinction between (inter-subjectively shared) knowledge and (subjectively held) beliefs is sometimes important in psychology. Interestingly, this distinction is less important for validation, which can be based on knowledge and beliefs alike, provided that they are easily accessible and passively activated during comprehension. An experiment by Voss, Fincher-Kiefer, Wiley, and Silfies (1993, Experiment 1) is a case in point. These authors found that readers holding strong beliefs (indicated by strong agreement or disagreement) make evaluative judgments regarding sentences as fast as judgments of meaningfulness. Thus, validation keeps pace with comprehension when readers hold strong beliefs (for similar results on the immediacy of belief-based validation responses, see Wyer & Radvansky, 1999). A second important point to keep in mind is that validation provides a shallow assessment of text information consistency with co-activated information at a given point during reading. The processing precludes a complete and thorough assessment of the epistemic status or the internal consistency of information. These statements are consistent with findings from metacomprehension research showing that comprehension monitoring during reading is often poor (for an overview see Baker, 1989). Most studies on metacomprehension used an error-detection paradigm to examine readers’ evaluation of comprehension as an indicator of their sensitivity to inconsistencies and contradictions (e.g., Baker, 1985; Winograd & Johnston, 1982). For example, Baker (1985) found that without specific instructions college students detected 68% of nonsense words but only 22% of prior knowledge violations and 12% of internal inconsistencies embedded in a text. At first glance, these and other similar results (e.g., Baker & Beall, 2009; Otero & Kintsch, 1992) seem to be at odds with the assumption that text comprehension entails the routine validation of text information and the supporting experimental evidence. However, one plausible interpretation that reconciles the seemingly discordant results is that participants in the metacomprehension studies might not have detected the inconsistencies because the inconsistent information was not co-activated when the relevant text information was processed. Moreover, the explicit (and often retrospective) judgments required in the error-detection paradigm are not identical to the implicit (and immediate) validity assessments that are assumed to be generated by passive validation processes. Grabe and colleagues illustrated this point well (Grabe, Antes, Kahn, & Kristjanson, 1991). In their study, participants reported less than half of the embedded internal and external errors. But Grabe et al. also observed readers’ eye

The Role of Validation  •  155

movements. Contrary to the fact that readers failed to verbally report the errors in the text, their reading behavior changed in response to world knowledge violations and internally inconsistent information. Readers not only spent more time fixating on critical sentences than non-critical sentences but also re-fixated on critical sentences longer.

A TWO-STEP MODEL OF VALIDATION IN MULTIPLE TEXTS When readers process multiple texts on (socio-)scientific or political issues that are relevant to them, they are often inclined to endorse one argumentative position in the controversy more than others. For example, a person searching on the Internet for potential health risks of the electromagnetic radiation emitted by cell phones, the causes of climate change, the risks and benefits of childhood vaccinations, or the value of educational reform is likely to have prior beliefs about the issue, despite having little knowledge about the topic. In this section, we outline a two-step model of validation in multiple-text comprehension (see also Richter, 2011). In the first obligatory step of epistemic monitoring, readers validate text information based on their beliefs, which leads to better comprehension and memory for belief-consistent information. In the second optional and goal-dependent step, readers may attempt to resolve the inconsistency revealed by epistemic monitoring by elaborative processing. As a result, readers are able to construct a more balanced representation of controversial information. The two-step model of validation is summarized in Figure 9.1 and will be described in the following sections.

Step 1: Non-strategic validation

Conditions

(Strong) prior beliefs

Processes

Epistemic monitoring of textbelief consistency

Step 2: Strategic elaboration of inconsistencies - Working memory resources - Background knowledge - Advanced epistemological beliefs - Epistemic reading goals - Metacognitive strategies only if

Detection of beliefinconsistent information

Elaboration of belief-inconsistent information

default

Outcomes

Belief-biased mental model of controversy

Balanced mental model of controversy

Figure 9.1  The Two-Step Model of Validation in Multiple-Text Comprehension (Richter & Maier, 2017).

156  •  Richter and Maier

STEP 1: BELIEF-BASED EPISTEMIC MONITORING OF CONFLICTING INFORMATION According to theory and research on validation, the consistency of text information with readers’ beliefs is checked—passively and involuntarily—when the beliefs are activated through concepts and propositions in the text. Thus, independent of their reading goal, readers implicitly judge the plausibility of the content when processing multiple texts on controversial issues. What is the psychological value of the plausibility judgments generated by epistemic monitoring? One function might be that the perceived plausibility serves as a heuristic that helps readers to regulate their cognitive resources during reading. Being cognitive misers, readers process information more deeply when they encounter plausible information, but tend to spend less cognitive effort on information that they find less plausible. In normal reading situations, characterized by a receptive reading goal, limited cognitive resources, and possibly time to invest in comprehending multiple texts, this assumption implies a text-belief consistency bias in multiple documents comprehension (Maier & Richter, 2013a) and more generally a comprehension and memory advantage for belief-consistent information and arguments. In essence, readers construct a stronger mental model for texts that align with their beliefs. This memory advantage may be regarded as an instance of a more general confirmation bias (Nickerson, 1998). However, going beyond previous work that focused mostly on the belief-stabilizing function of the confirmation bias, the two-step model makes the prediction of a textbelief consistency effect in comprehension and memory.

STEP 2: ELABORATIVE PROCESSING OF CONFLICTING INFORMATION Although less intensive processing of belief-inconsistent information is the default, readers sometimes actively try to resolve inconsistencies that have arisen between text information and their knowledge and beliefs. For example, they might try to think about or search the text for alternative reasons that may support the implausible information. Unlike the first step of epistemic monitoring, these elaborative activities are under the strategic control of the reader. Only readers following an epistemic reading goal (Richter, 2003); that is, a reading goal that includes a justified and defensible point of view on a controversial issue will engage in elaborative processing of conflicting information. This condition might occur, for example, when readers want to defend their view in front of others (fear of invalidity, Kruglanski & Webster, 1996), when they want to actively discount alternative views (Edwards & Smith, 1996), or when they are epistemically curious (Richter & Schmid, 2010). Epistemological beliefs may play a role in these instances at the metacognitive level. A mature epistemological position (e.g., commitment within relativism, Perry, 1970, or reflective judgment, King & Strohm Kitchener, 1994) promotes epistemic reading goals and the engagement in the elaborative processing of belief-inconsistent information (Richter, 2011; Richter & Schmid, 2010). Engaging in the elaborative processing of belief-inconsistent information will often strengthen the comprehension of especially belief-inconsistent information. For example, Wiley and Voss (1999) found that students wrote more coherent essays

The Role of Validation  •  157

with stronger causal links and scored better in comprehension tasks when they had received the instruction to write an argumentative essay (i.e., an epistemic reading goal) compared to the instruction to write a summary or a narrative text. Of course, such beneficial effects of elaborative processing are not guaranteed. The literature on dealing with inconsistencies suggests that readers sometimes settle with suboptimal ways to resolve inconsistencies, i.e. by creating causal relationships between pieces of information that are actually inconsistent with one another (Blanc, Kendeou, van den Broek, & Brouillet, 2008) or by distorting inconsistent information to make it consistent (cf. the findings by Hakala & O’Brien, 1995, Exp. 2, for locally inconsistent information). Overall, however, conditions that enhance elaborative processing of inconsistencies should also improve comprehension of the inconsistent information.

CONSEQUENCES OF VALIDATION FOR COMPREHENSION OUTCOMES The two-step model of validation posits that readers use their prior knowledge and beliefs to validate incoming text information. The plausibility of new information— that is, its consistency with readers’ prior knowledge, beliefs, and the current situation model—is used as a heuristic for information selection and for the allocation of cognitive resources during reading. As a consequence, the model proposes that readers will evaluate information judged as consistent during epistemic monitoring as more plausible, will process it more deeply, and will achieve a stronger mental model for information that is consistent with their knowledge, beliefs, or current mental representation of the discourse. In this section, we will first review empirical results investigating knowledge revision or situation model updating when readers are confronted with information challenging or discrediting their current mental representation or their prior knowledge and beliefs. Although most of this research was conducted with single texts, the reviewed experiments resemble multiple documents comprehension in relevant ways. In particular, the texts and tasks used in these experiments require readers to build and eventually update their mental representation of the discourse based on partially conflicting information. Afterwards, we discuss research on multiple-document comprehension that has investigated the extent that readers’ prior beliefs lead to a biased processing of conflicting information and a one-sided mental model of controversial issues (i.e., the text-belief consistency effect). The important point to keep in mind is that although the studies reviewed focus on different types of inconsistencies—between knowledge and new text information, beliefs and new text information, or previously read texts and new text information—all of them can be explained by the same passive validation mechanism. This mechanism is based on the information that is activated at a given moment during comprehension, regardless of the type of information.

CONTINUED INFLUENCE OF MISINFORMATION, FALSE KNOWLEDGE, AND BELIEFS Research on the continued influence of misinformation after explicit corrections aligns well with the two-step model of validation. Many studies have shown that readers continue to rely on previously learned information even when this information is

158  •  Richter and Maier

discredited or corrected by subsequent information and even when readers notice and remember the correction (Ecker, Lewandowsky, & Tang, 2010; Ecker, Lewandowsky, Swire, & Chang, 2011; Fein, McCloskey, & Tomlinson, 1997; Johnson & Seifert, 1994, 1998; Rich & Zaragoza, 2016; van Oostendorp, 1996; van Oostendorp & Bonebakker, 1999; Wilkes & Leatherbarrow, 1988; Wilkes & Reynolds, 1999; for a review see Ecker, Swire, & Lewandowsky, 2014). For example, Wilkes and Leatherbarrow (1988) examined whether readers’ inference generation was based on previously encoded information that was later discredited. Readers read a series of messages about a warehouse fire in which one cause of the fire (e.g., a closet contained volatile material) was later corrected (e.g., closet was empty). In comprehension questions posed after reading, more than 90% of participants still used the old invalid information for inferences. Johnson and Seifert (1994) used a similar series of messages about a warehouse fire and found that the misinformation continued to have an influence regardless of the time that had elapsed between the misinformation and the correcting information. In particular, readers continued to use the old invalidated information and to refer to false information when the correction was immediate (in the next message) or delayed (after five messages). In addition, a continuous influence of misinformation was found when readers received general warnings prior to reading (Ecker et  al., 2010), which suggests a rather robust reliance on misinformation. An important result in both studies (Johnson & Seifert, 1994; Wilkes & Leatherbarrow, 1988) is that over 90% of participants recalled the correction when directly asked. In sum, readers have a tendency to hold fast to acquired beliefs (such as the cause of a warehouse fire) even when they are presented with discrediting information. According to the two-step model of validation, the validation of the discrediting information and its rejection may be one of the mechanisms underlying the continued influence of misinformation. Several studies identified conditions that reduced the continued influence of the misinformation effect. For example, Johnson and Seifert (1994) found that the effect only occurred when the possible cause of the fire (e.g., volatile material) was later corrected in a causal context (e.g., volatile material in closet vs. in a nearby store). In contrast, when readers were offered an alternative explanation, the amount of inferences that was based on new rather than old invalidated information increased (van Oostendorp & Bonebakker, 1999). Similarly, a specific warning that made readers suspicious was able to reduce the impact of misinformation (Ecker et al., 2010). According to the two-step model presented in this chapter, both conditions, the presentation of an alternative explanation and a critical mindset, should foster elaborative processing, thus creating conditions in which readers are open to integrate conflicting information in their situation model of the text content. A continued influence has also been found for beliefs that were explicitly corrected or discredited (Anderson, Lepper, & Ross, 1980; Ross, Lepper, & Hubbard, 1975) and misconceptions rooted more deeply in the learning history of individuals (Alvermann, Smith, & Readence, 1985; Chinn & Brewer, 1993; di Sessa, 1993; Kendeou & van den Broek, 2005; Limon & Carretero, 1997; Mason, 2001; Vosniadou, 1994; for an overview see Murphy & Mason, 2006). Participants in a study from Anderson et al. (1980) read case studies that suggested a relationship between risk-taking behavior and the professional ability of firefighters. Participants were then told that the initial information was false (debriefing condition), or they received no information after reading the

The Role of Validation  •  159

case study (no-debriefing condition). Participants in both conditions used their (false) beliefs when making judgments about the relationship between the trait and the behavioral outcome of the firefighters when generalizing on new cases and test items. The false beliefs were particularly difficult to correct when participants had provided an explicit explanation for the relationship between the trait and behavior (Anderson et al., 1980; for an experiment suggesting similar conclusions, see Rich & Zaragoza, 2016). In sum, information integrated into readers’ situation model, as well as prior knowledge and beliefs reactivated during comprehension, are able to influence whether new information is integrated into the evolving mental model. The influence of misinformation, false knowledge, and incorrect beliefs may be partly explained by memory processes, for example the passive re-activation of discredited but nevertheless salient and easily accessible concepts (Ecker et al., 2014; Kendeou & O’Brien, 2014; see Richter & Singer, in press, for an overview). Nevertheless, validation during comprehension, in particular the implicit plausibility judgments generated by this process, are likely to contribute to these effects (see also Lombardi, Nussbaum, & Sinatra, 2016). Schroeder, Richter, and Hoever (2008) directly investigated the effects of perceived plausibility on the integration of information into readers’ situation model of the text content. In their study, participants read single texts in their area of study (psychology) that contained plausible as well as implausible (faulty) information and then provided recognition and plausibility judgments on the same set of test items (paraphrases of text information and inferences that could be derived from the text). Multinomial models analysis revealed a close bi-directional relationship between perceived plausibility of information and comprehension. Specifically, information perceived as plausible was more often integrated into readers’ situation model compared to information judged as implausible. However, after information had become part of the situation model, it was also perceived as more plausible—regardless of its objective plausibility. This plausibility effect occurred without readers following a specific reading goal. The results from the reviewed studies suggest that after information has become part of a reader’s situation model of the text content, it is used for monitoring the validity of incoming information. When readers process partly conflicting multiple documents, the same mechanisms are expected to take effect unless readers are motivated and able to engage in elaborative processing of implausible or inconsistent information. Hence, readers are expected to concentrate their cognitive resources on information that they find plausible (i.e., consistent with prior beliefs) and to construct only a sufficient rather than the best possible mental representation. In the next section, we review research that supports the idea that similar mechanisms apply to the comprehension of multiple documents.

BELIEF CONSISTENCY EFFECTS IN MULTIPLEDOCUMENT COMPREHENSION Experiments on multiple-document comprehension have revealed that readers provide biased essays or concluding paragraphs after reading belief-consistent and beliefinconsistent information (Anmarkrud, Bråten, & Strømsø, 2014; Kardash & Scholes, 1996; van Strien, Brand-Gruwel, & Boshuizen, 2014; van Strien, Kammerer, BrandGruwel, & Boshuizen; 2016), have a better recognition for belief-consistent arguments

160  •  Richter and Maier

or texts in recall or recognition tasks (Maier & Richter, 2013a, 2014; Wiley, 2005), and evaluate belief-consistent compared to belief-inconsistent information better in their evaluations of arguments (Kobayashi, 2010, 2014; McCrudden & Barnes, 2016; McCrudden & Sparks, 2014). We view all of these findings as instances of text-belief consistency effects, which indicate a general preference for belief-consistent information in the comprehension and evaluation of information from multiple documents. For example, in a study by van Strien et al. (2014), students read 13 documents (one neutral, six pro, and six contra) on the link between violent videogames and aggression. The authors found that participants were more likely to write essays that were consistent with their prior beliefs than essays that were at odds with their beliefs. Using a similar method, van Strien et al. (2016) identified belief strength as a moderator of the text-belief consistency effect in essay-writing tasks. Participants with strong prior beliefs used far more arguments from belief-consistent web pages compared to belief-inconsistent web pages, whereas the effect was not found for participants with weak prior beliefs. An experiment by Kobayashi (2014) is an example of text-belief consistency effects in the evaluation of information from multiple documents. In this study, undergraduates read two texts that took contrary stances on the question of whether a relationship between blood type and personality exists. Results revealed that participants holding prior beliefs in favor of this assumption evaluated the pro text as more acceptable than the con text. In addition, the way in which participants tried to resolve the debate also strongly depended on their beliefs. Participants who endorsed the view that a link between blood type and personality exists were more likely to resolve the conflict in favor of such a link. Maier and Richter (2013a) measured readers’ recognition of belief-consistent and belief-inconsistent texts. In their study, participants read four texts on either climate change or vaccinations and then provided recognition judgments for different test items (paraphrases, inferences, and distracters). Readers had a stronger situation model (measured by participants’ responses to inference items, corrected for response bias) for texts with an argumentative position that was in line with their beliefs compared to texts with an argumentative position that opposed their beliefs. Moreover, this study identified the mode of text presentation as a moderator for the text-belief consistency effect in recognition tasks (for similar findings see Wiley, 2005). Specifically, the textbelief consistency effect for situation model strength occurred only when two texts taking the same argumentative side of the controversy were presented in a blocked manner, whereas the effect was not found when texts with different stances were presented interleaved. This study also showed that the text-belief consistency effect in the blocked presentation mode became stronger when participants spent less time reading belief-inconsistent texts. Hence, as suggested by the two-step model of validation, superficial processing supported a comprehension advantage for belief-consistent texts. Research in multiple documents comprehension suggests some additional conditions that might be able to foster elaborative epistemic processing and to reduce the influence of beliefs. For example, several studies found that reader characteristics such as epistemological beliefs (Mason & Boscolo, 2004), belief strength (Kardash & Scholes, 1996; McCrudden & Barnes, 2016), prior knowledge, and level of education (Wiley, 2005) can moderate text-belief consistency effects. Moreover, rationale and evidence instructions (McCrudden & Sparks, 2014) as well as argument instructions (Maier & Richter, 2016a), which may all be regarded as epistemic reading goals, have been found

The Role of Validation  •  161

to decrease text-belief consistency effects. Similarly, a study by McCrudden and Sparks (2014) suggests that a belief-reflection goal (i.e., being open to belief-inconsistent information and critically evaluating arguments and evidence from both sides) decreases text-belief consistency effects. These findings are in line with the two-step model of validation, which suggests that text-belief consistency effects result from cognitive processes inherent in comprehension (i.e., routine validation) but can be countered by elaborative processes that depend on epistemic reading goals. The finding that text-belief consistency effects are accompanied by a relatively lower level of strategic elaborative processing of belief-inconsistent information further supports this proposition. In a study by Maier and Richter (2016a), readers either received a summary task that required receptive processing or an argument task that was expected to induce an epistemic reading goal. With these goals in mind, participants read one belief-consistent and one belief-inconsistent text about health risks caused by cell phone use. Consistent with the two-step model of validation, participants focused their cognitive resources on belief-consistent information when they worked on the summary task. In contrast, when participants were given the argument task, no differences were found between reading times of both texts, and participants used more strategic, elaborative validation strategies when reading the belief-inconsistent text. In sum, the research on the comprehension of multiple documents shows that readers’ beliefs strongly influence the evaluation and comprehension of belief-relevant information. Readers holding prior beliefs about a given controversial issue evaluate belief-consistent arguments as more plausible, process belief-consistent information more deeply, and achieve a stronger mental model for documents that are consistent with their prior beliefs. These findings are fully in line with the two-step model of validation. A study by Maier and Richter (2013b) suggests that the perceived plausibility of information might contribute to these text-belief consistency effects, as proposed by the two-step model of validation. In these studies, participants judged the plausibility of paraphrased text sentences and inference sentences after reading a set of multiple texts on social science issues (e.g., interpretation of PISA results) and performed recognition judgments on the same set of items. For both topics, a strong relationship emerged between perceived plausibility and recognition judgments. Information (paraphrases or inferences) judged as plausible was more likely to be judged as coming from the text than information judged as implausible.

IMPLICATIONS FOR THEORY, RESEARCH, AND PRACTICE Understanding multiple documents often involves comprehending controversial and conflicting information. Based on research on validation in comprehension and the general finding of text-belief consistency effects in multiple-text comprehension, the two-step model of validation in multiple-text comprehension suggests that readers’ prior knowledge and beliefs serve as a kind of epistemic gatekeeper (Schroeder et al., 2008). For multiple documents with conflicting information, the two-step model of validation suggests that readers make more cognitive resources available for the comprehension of information that they perceive as plausible. Moreover, the two-step model proposes that such biased processing is caused by passive and routine validation processes (epistemic monitoring) that are part of basic comprehension.

162  •  Richter and Maier

The concept of validation raises questions that should be clarified for future theoretical and empirical work. For example, it is still largely unknown how the basic processes advocated in the two-step model of validation are related to other component processes of comprehension. One first attempt to clarify the time course of knowledge activation, integration, and validation is proposed in the RI-Val model of comprehension (O’Brien & Cook, 2016). Similar to the two-step model of validation, the RI-Val model proposes that activation, integration, and validation are passive continuous processes that run to completion. Moreover, the RI-Val model specifies that the onsets of the three processes are asynchronous with activation starting first and validation occurring last. The RI-Val model allows for predicting the extent that these passive processes (i.e., spillover effects) influence comprehension when reading one sentence. The next step will be to investigate these effects on comprehension when reading an entire document or even multiple documents. Recent experiments by Beker, Jolles, Lorch, and van den Broek (2016) provide a first step in this direction. They extended the inconsistency paradigm to multiple texts and demonstrated that information encountered in a previous text slows down the reading of inconsistent sentences in a second text. This may be considered as evidence for passive activation of information across multiple texts; at the same time it provides evidence for validation of information encountered in the second text on the basis of the activated information from the first text. More theoretical work and empirical research are needed to explain when readers find that their mental representation resulting from passive processes are insufficient and start to engage in more strategic validation. One factor that influences whether readers rely on regular validation or engage in strategic validation might be the standards of coherence readers adopt during reading (van den Broek, Beker, & Oudega, 2015). In general, a higher standard of coherence should be associated with more elaborative processing; for example, investing more cognitive effort to search, retrieve, and validate relevant information strategically. In line with this assumption, readers’ sensitivity to implausible information seems to depend on the goals (e.g., Rapp, Hinze, Kohlhepp, & Ryskin, 2014) and the mindset of readers (with possibly reduced validation when readers read narrative fiction, Appel & Richter, 2007). The two-step model of validation can also be used as the basis for developing training of multiple documents comprehension. One promising avenue for such training is to strengthen metacognitive strategies that promote strategic elaborative processing of belief-inconsistent information and, importantly, promoting knowledge about passive validation processes and their consequences (Maier & Richter, 2014). Maier and Richter (2016b) found that a metacognitive training that made readers aware of potential biases resulting from routine validation, and to foster elaborative processing of belief-inconsistent information, reduced the belief consistency effect in comprehending multiple documents. The control group in this experiment received the PQ4R training (Thomas & Robinson, 1972), and this training was not effective in eliminating the text-belief consistency effect. Hence, training classical cognitive and metacognitive strategies (which is the focus of the PQ4R training) is insufficient. Instead, attention to passive and routine validation processes is needed for successful multiple-document comprehension.

The Role of Validation  •  163

The two-step model of validation also suggests some general principles that educational practitioners might use for designing instruction with multiple documents. First, the two-step model proposes that reading instruction requiring readers to take a neutral perspective is an ineffective approach to overcoming potential biases in the comprehension of controversial documents. A more promising strategy is to first become more aware of the non-strategic biases that can be caused by validation. Then, metacognitive knowledge about non-strategic and strategic validation should be integrated into teaching, including training for specific metacognitive strategies directed at such processes. Other means such as epistemic reading goals (e.g., argument tasks or belief-reflection tasks) can enhance critical thinking and minimize biased processing as a result of non-strategic validation when processing multiple documents.

CONCLUSION Validation during comprehension enables readers to establish and monitor local and global coherence, and it can protect readers from processing false and implausible information (see Richter, 2015). However, when readers comprehend multiple documents with conflicting information, non-strategic validation processes can lead to processing and comprehending difficulties, to the persistence of misconceptions and false beliefs, and to a rather one-sided mental representation of the discourse such as text-belief consistency effects. Acknowledging the concept of validation in research, theory, and practice can broaden and enrich the understanding of multiple-document comprehension, and can assist researchers and practitioners in understanding the challenges readers are confronted with during multiple-document comprehension.

NOTE 1

The authors’ research described in this chapter was supported by the German Research Foundation (DFG, grant RI 1100/5) and the German Federal Ministry of Education and Research (grant 01PK15009B).

REFERENCES Albrecht, J. E., & Myers, J. L. (1995). Role of context in accessing distant information during reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(6), 1459–1468. Albrecht, J. E., & O’Brien, E. J. (1993). Updating a mental model: Maintaining both local and global coherence. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 106l–1070. Alvermann, D. E., Smith, L. C., & Readence, J. E. (1985). Prior knowledge activation and the comprehension of compatible and incompatible text. Reading Research Quarterly, 20, 420–436. Anderson, C. A., Lepper, M. R., & Ross, L. R. (1980). Perseverance of social theories: The role of explanation in the persistence of discredited information. Journal of Personality and Social Psychology, 39, 1037–1049. Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. Appel, M., & Richter, T. (2007). Persuasive effects of fictional narratives increase over time. Media Psychology, 10, 113–134. Baker, L. (1985). Differences in the standards used by college students to evaluate their comprehension of expository prose. Reading Research Quarterly, 20, 297–313. Baker, L. (1989). Metacognition, comprehension monitoring and the adult reader. Educational Psychology Review, 1, 3–38.

164  •  Richter and Maier Baker, L., & Beall, L. C. (2009). Metacognitive processes and reading comprehension. In S. E. Israel & G. G. Duffy (Eds.), Handbook of research on reading comprehension (pp. 373–388). New York: Routledge. Beker, K., Jolles, D., Lorch, R. F., & van den Broek, P. (2016). Learning from texts: Activation of information from previous texts during reading. Reading and Writing, 29, 1161–1178. Blanc, N., Kendeou, P., van den Broek, P., & Brouillet, D. (2008). Updating situation models during reading of news reports: Evidence from empirical data and simulations. Discourse Processes, 45, 103–121. Britt, M. A., & Larson, A. (2003). Construction of argument representations during on-line reading. Journal of Memory and Language, 48, 749–810. Britt, M. A., Perfetti, C. A., Sandak, R., & Rouet, J. F. (1999). Content integration and source separation in learning from multiple texts. In S. R. Goldman, A. C. Graesser, & P. van den Broek (Eds.), Narrative comprehension, causality, and coherence: Essays in honor of Tom Trabasso (pp. 209–233). Mahwah, NJ: Erlbaum. Chinn, C. A., & Brewer, W. F. (1993). The role of anomalous data in knowledge acquisition: A theoretical framework and implications for science education. Review of Educational Research, 63, 1–49. Connell, L., & Keane, M. T. (2006). A model of plausibility. Cognitive Science, 30, 95–120. Cook, A. E., & O’Brien, E. J. (2015). Passive activation and instantiation of inferences during reading. In E. J. O’Brien, A. E. Cook, & R. F. Lorch (Eds.), Inferences during reading (pp.  19–41). Cambridge, UK: Cambridge University Press. di Sessa, A. (1993). Towards an epistemology of physics. Cognition and Instruction, 10, 105–225. Eagly, A. H., & Chaiken, S. (1993). The psychology of attitudes. Fort Worth, TX: Harcourt Brace Jovanovich. Ecker, U. K. H., Lewandowsky, S., Swire, B., & Chang, D. (2011). Correcting false information in memory: Manipulating the strength of misinformation encoding and its retraction. Psychonomic Bulletin & Review, 18, 570–578. Ecker, U. K. H., Lewandowsky, S., & Tang, D. T. W. (2010). Explicit warnings reduce but do not eliminate the continued influence of misinformation. Memory & Cognition, 38, 1087–1100. Ecker, U. K. H., Swire, B., & Lewandowsky, S. (2014). Correcting misinformation: A challenge for education and cognitive science. In D. N. Rapp & J. L. G. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 13–18). Cambridge, MA: MIT Press. Edwards, K., & Smith, E. E. (1996). A disconfirmation bias in the evaluation of arguments. Journal of Personality and Social Psychology, 71, 5–24. Fein, S., McCloskey, A. L., & Tomlinson, T. M. (1997). Can the jury disregard that information? The use of suspicion to reduce the prejudicial effects of pretrial publicity and inadmissible testimony. Personality and Social Psychology Bulletin, 23, 1215–1226. Ferreira, F., Bailey, K. G., & Ferraro, V. (2002). Good-enough representations in language comprehension. Current Directions in Psychological Science, 11, 11–15. Ferretti, T. R., Singer, M., & Patterson, C. (2008). Electrophysiological evidence for the time course of verifying text ideas. Cognition, 108, 881–888. Grabe, M., Antes, J., Kahn, H., & Kristjanson, A. (1991). Adult and adolescent readers’ comprehension monitoring performance: An investigation of monitoring accuracy and related eye movements. Contemporary Educational Psychology, 16, 45–60. Hagoort, P., Hald, L., Bastiaansen, M., & Petersson, K. M. (2004). Integration of word meaning and world knowledge in language comprehension. Science, 304, 438–441. Hakala, C. M., & O’Brien, E. J. (1995). Strategies for resolving coherence breaks in reading. Discourse Processes, 20, 167–185. Isberner, M.-B. & Richter, T. (2013). Can readers ignore implausibility? Evidence for nonstrategic monitoring of event-based plausibility in language comprehension. Acta Psychologica, 142, 15–22. Isberner, M.-B., & Richter, T. (2014a). Comprehension and validation: Separable stages of information processing? A case for epistemic monitoring in language comprehension. In D. N. Rapp & J. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 245–276). Boston, MA: MIT Press. Isberner, M.-B. & Richter, T. (2014b). Does validation during language comprehension depend on an evaluative mindset? Discourse Processes, 51, 7–25. Johnson, H. M., & Seifert, C. M. (1994). Sources of the continued influence effect: When discredited information in memory affects later inferences. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20, 1420–1436.

The Role of Validation  •  165 Johnson, H. M., & Seifert, C. M. (1998). Updating accounts following a correction of misinformation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 1483–1494. Johnson-Laird, P. N. (1983). Mental models. Cambridge, UK: Cambridge University Press. Kardash, C. M., & Scholes, R. J. (1996). Effects of preexisting beliefs, epistemological beliefs, and need for cognition on interpretation of controversial issues. Journal of Educational Psychology, 88, 260–271. Kendeou, P., & O’Brien, E. J. (2014). The Knowledge Revision Components (KReC) framework: Processes and mechanisms. In D. N. Rapp & J. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 353–377). Cambridge, MA: MIT Press. Kendeou, P., Smith, E. R., & O’Brien, E. J. (2013). Updating during reading comprehension: Why causality matters. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 854–865. Kendeou, P., & van den Broek, P. (2005). The role of readers’ misconceptions on text comprehension. Journal of Educational Psychology, 97, 235–245. King, P. M., & Strohm Kitchener, K. (1994). Developing reflective judgment. San Francisco, CA: Jossey-Bass. Kintsch, W. (1988). The role of knowledge in discourse comprehension: A Construction-Integration model. Psychological Review, 95, 163–182. Knobloch-Westerwick, S., & Meng, J. (2011). Reinforcement of the political self through selective exposure to political messages. Journal of Communication, 61, 349–368. Kobayashi, K. (2010). Strategic use of multiple texts for the evaluation of arguments. Reading Psychology, 31, 121–149. Kobayashi, K. (2014). Students’ consideration of source information during the reading of multiple texts and its effect on intertextual conflict resolution. Instructional Science, 42, 183–205. Kruglanski, A. W., & Webster, D. M. (1996). Motivated closing of the mind: “Seizing” and “freezing”. Psychological Review, 103, 263–283. Larson, M., Britt, M. A., & Larson, A. (2004). Disfluencies in comprehending argumentative texts. Reading Psychology, 25, 205–224. Lea, R. B., Mulligan, E. J., & Walton, J. L. (2005). Accessing distant premise information: How memory feeds reasoning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31, 387–395. Limon, M., & Carretero, M. (1997). Conceptual change and anomalous data: A case study in the domain of natural sciences. European Journal of Psychology of Education, 12, 213–230. Lombardi, D., Nussbaum, E. M., & Sinatra, G. M. (2016). Plausibility judgments in conceptual change and epistemic cognition. Educational Psychologist, 51, 35–56. Maier, J., & Richter, T. (2013a). Text-belief consistency effects in the comprehension of multiple texts with conflicting information. Cognition and Instruction, 31, 151–175. Maier, J., & Richter, T. (2013b). How nonexperts understand conflicting information on social science issues: The role of perceived plausibility and reading goals. Journal of Media Psychology, 25, 14–26. Maier, J., & Richter, T. (2014). Fostering multiple text comprehension: How metacognitive strategies and motivation moderate the text-belief consistency effect. Metacognition & Learning, 9, 51–74. Maier, J., & Richter, T. (2016a). Effects of text-belief consistency and reading task on the strategic validation of multiple texts. European Journal of the Psychology of Education, 31, 479–497. Maier, J., & Richter, T. (2016b). Training metacognitive strategies on multiple text: When elaboration is not enough. Manuscript in preparation. Mason, L. (2001). Responses to anomalous data on controversial topic and theory change. Learning and Instruction, 11, 453–483. Mason, L., & Boscolo, P. (2004). Role of epistemological understanding and interest in interpreting a controversy and in topic-specific belief change. Contemporary Educational Psychology, 29, 103–128. Matsuki, K., Chow, T., Hare, M., Elman, J. L., Scheepers, C., & McRae, K. (2011). Event-based plausibility immediately influences on-line sentence comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37, 913–934. McCrudden, M. T., & Barnes, A. (2016). Differences in student reasoning about belief-relevant arguments: A mixed methods study. Metacognition and Learning, 11, 275–303. McCrudden, M. T., & Sparks, P. C. (2014). Exploring the effect of task instructions on topic beliefs and topic belief justifications: A mixed methods study. Contemporary Educational Psychology, 39, 1–11. Murphy, P. K., & Mason, L. (2006). Changing knowledge and beliefs. In P. A. Alexander & P. H. Winne (Eds.), Handbook of educational psychology (pp. 305–324). Mahwah, NJ: Erlbaum.

166  •  Richter and Maier Myers, J. L., O’Brien, E. J., Albrecht, J. E., & Mason, R. A. (1994). Maintaining global coherence during reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20, 876–885. Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2, 175–220. O’Brien, & Albrecht, J. E. (1992). Comprehension strategies in the development of a mental model. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 777–784. O’Brien, E. J., & Cook, A. E. (2016). Coherence threshold and the continuity of processing: The RI-Val model of comprehension. Discourse Processes, 53, 326–338. O’Brien, E. J., & Myers, J. L. (1999). Text comprehension: A view from the bottom up. In S. R. Goldman, A. C. Graesser, & P. van den Broek (Eds.), Narrative comprehension, causality, and coherence: Essays in honor of Tom Trabasso (pp. 35–54). Mahwah, NJ: Erlbaum. O’Brien, E., Rizella, M. L., Albrecht, J. E., & Halleran, J. G. (1998). Updating a situation model: A memory-based text processing view. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 1200–1210. Otero, J., & Kintsch, W. (1992). Failures to detect contradictions in text: What readers believe versus what they read. Psychological Science, 3, 229–235. Perfetti, C. A., Rouet, J.-F., & Britt, M. A. (1999). Toward a theory of documents representation. In H. van Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 99–122). Mahwah, NJ: Erlbaum. Perry, W. G. (1970). Forms of intellectual and ethical development in the college years. New York: Holt, Rinehart and Winston. Rapp, D. N., Hinze, S. R., Kohlhepp, K., & Ryskin, R. A. (2014). Reducing reliance on inaccurate information. Memory and Cognition, 42, 11–26. Rich, P. R., & Zaragoza, M. S. (2016). The continued influence of implied and explicitly stated misinformation in news reports. Journal of Experimental Psychology: Learning, Memory, and Cognition, 42, 62–74. Richter, T. (2003). Epistemologische Einschätzungen beim Textverstehen [Epistemic validation in text comprehension]. Lengerich, Germany: Pabst. Richter, T. (2011). Cognitive flexibility and epistemic validation in learning from multiple texts. In J. Elen, E. Stahl, R. Bromme, & G. Clarebout (Eds.), Links between beliefs and cognitive flexibility (pp. 125–140). Berlin: Springer. Richter, T. (2015). Validation and comprehension of text information: Two sides of the same coin. Discourse Processes, 52, 337–352. Richter, T., & Maier, J. (2017). Comprehension of multiple documents with conflicting information: The role of epistemic validation. Educational Psychologist. doi: 10.1080/00461520.2017.1322968. Richter, T., & Schmid, S. (2010). Epistemological beliefs and epistemic strategies in self-regulated learning. Metacognition and Learning, 5, 47–65. Richter, T., Schroeder, S., & Wöhrmann, B. (2009). You don’t have to believe everything you read: Background knowledge permits fast and efficient validation of information. Journal of Personality and Social Psychology, 96, 538–598. Richter, T., & Singer, M. (in press). Discourse updating: Acquiring and revising knowledge through discourse. In D. Rapp, A. Britt, & M. Schober (Eds.), Handbook of discourse processes (2nd ed.). New York: Taylor & Francis. Rinck, M., Hähnel, A., & Becker, G. (2001). Using temporal information to construct, update, and retrieve situation models of narratives. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27, 67–80. Ross, L., Lepper, M. R., & Hubbard, M. (1975). Perseverance in self-perception and social perception: Biased attributional processes in the debriefing paradigm. Journal of Personality and Social Psychology, 32, 880–892. Rouet, J.-F. (2006). The skills of document use: From text comprehension to web-based learning. Mahwah, NJ: Erlbaum. Rouet, J. F., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. In G. Schraw, M. McCrudden & J. P. Magliano (Eds.), Text relevance and learning from text (pp. 19–52). Charlotte, NC: Information Age Publishing. Schroeder, S., Richter, T., & Hoever, I. (2008). Getting a picture that is both accurate and stable: Situation models and epistemic validation. Journal of Memory and Language, 59, 237–255. Singer, M. (1993). Causal bridging inferences: Validating consistent and inconsistent sequences. Canadian Journal of Experimental Psychology, 47, 340–359.

The Role of Validation  •  167 Singer, M. (2006). Verification of text ideas during reading. Journal of Memory and Language, 54, 574–591. Singer, M. (2013). Validation in reading comprehension. Current Directions in Psychological Science, 22, 361–366. Singer, M., Halldorson, M., Lear, J. C., & Andrusiak, P. (1992). Validation of causal bridging inferences. Journal of Memory and Language, 31, 507–524. Staub, A., Rayner, K., Pollatsek, A., Hyönä, J., & Majewski, H. (2007). The time course of plausibility effects on eye movements in reading: Evidence from noun-noun compounds. Journal of Experimental Psychology: Learning, Memory, and Cognition, 33, 1162–1169. Thomas, E. L., & Robinson, H. A. (1972). Improving reading in every class: A sourcebook for teachers. Boston, MA: Houghton Mifflin. Toulmin, S. E. (1958). The uses of argument. Cambridge, MA: Cambridge University Press. van den Broek, P., Beker, K., & Oudega, M. (2015). Inference generation in text comprehension: Automatic and strategic processes in the construction of a mental representation. In E. J. O’Brien, A. E. Cook, & R. F. Lorch (Eds.), Inferences during reading (pp. 94–121). Cambridge, UK: Cambridge University Press. van Dijk, T.A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic Press. van Oostendorp, H. (1996). Updating situation models derived from newspaper articles. Medienpsychologie, 8, 21–33. van Oostendorp, H., & Bonebakker, C. (1999). Difficulties in updating mental representations during reading news reports. In H. van Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 319–340). Mahwah, NJ: Erlbaum. van Strien, J. L. H., Brand-Gruwel, S., & Boshuizen, H. P. A. (2014). Dealing with conflicting information from multiple nonlinear texts: Effects of prior attitudes. Computers in Human Behavior, 32, 101–111. van Strien, J. L. H., Kammerer, Y., Brand-Gruwel, S., & Boshuizen, H. P. A. (2016). How attitude strength biases information processing and evaluation on the web. Computers in Human Behavior, 60, 245–252. von der Mühlen, S., Richter, T., Schmid, S., Schmidt, L. M., & Berthold, K. (2016). Use of source-related strategies in evaluating multiple psychology texts: A student-scientist comparison. Reading and Writing, 8, 1677–1698. Vosniadou, S. (1994). Capturing and modeling the process of conceptual change. Learning and Instruction, 4, 45–69. Voss, J. F., Fincher-Kiefer, R., Wiley, J., & Silfies, L. N. (1993). On the processing of arguments. Argumentation, 7, 165–181. Wiley, J. (2005). A fair and balanced look at the news: What affects memory for controversial arguments? Journal of Memory and Language, 53, 95–109. Wiley, J., & Voss, J. F. (1999). Constructing arguments from multiple sources: Tasks that promote understanding and not just memory for text. Journal of Educational Psychology, 91, 301–311. Wilkes, A. L., & Leatherbarrow, M. (1988). Editing episodic memory following the identification of error. The Quarterly Journal of Experimental Psychology Section A: Human Experimental Psychology, 40, 361–387. Wilkes, A. L., & Reynolds, D. J. (1999). On certain limitations accompanying readers’ interpretations of corrections in episodic text. The Quarterly Journal of Experimental Psychology, 52, 165–183. Wineburg, S. S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87. Winograd, P., & Johnston, P. (1982). Comprehension monitoring and the error detection paradigm. Journal of Reading Behaviour, 14, 61–76. Wolfe, C. R., Britt, M. A., & Butler, J. A. (2009). Argumentation schema and the my-side bias in written argumentation. Written Communication, 26, 183–209. Wyer, R. S. Jr., & Radvansky, G. A. (1999). The comprehension and validation of social information. Psychological Review, 106, 89–118. Zwaan, R. A., & Radvansky, G. A. (1998). Situation models in language comprehension and memory. Psychological Bulletin, 123, 162–185.

10

TEXT RELEVANCE AND MULTIPLE-SOURCE USE Matthew T. McCrudden victoria university of wellington, new zealand

We access text to help us answer questions, solve problems, and to improve our knowledge and understanding (Graesser & Lehman, 2011). Thus, accessing and reading text is a goal-directed task (Graesser, Singer, & Trabasso, 1994; McCrudden & Schraw, 2007). We may read for an assigned task, such as when a student reads to prepare for a class test or to gather information for an essay (What do starfish eat?). Or, we may read for a self-generated task, such as when we seek information about an impending decision (e.g., how to address a health-related issue or what type of mobile phone is best suited to our needs) or to become informed about a socio-scientific topic (e.g., climate change). What all of these situations have in common is that they involve reading to reduce or eliminate a knowledge gap between what we know currently and what we want to know. When there is a gap between what we know and want to know, some information helps us address this gap better than other information; that is, some information is more relevant than other information for filling this gap. When the answers to our questions are complex, we commonly need to access information from multiple sources to generate answers to our questions. This process can be demanding because we need to identify, evaluate, and integrate relevant information from different sources that may contain overlapping, unique, or even conflicting messages that may be distributed across more than one document. For the purposes of this chapter, information source (or ‘source’ for short) will refer to the person or entity to whom information is attributed (MacedoRouet, Braasch, Britt, & Rouet, 2013), such as the author of a document or an author referenced within a document. In these situations, not only do we need to identify and evaluate the extent to which the informational content is relevant, we need to evaluate the credibility of the source. The purpose of this chapter is to discuss the construct of text relevance in the context of multiple-source use. It is important to acknowledge that most of the research on text relevance over the past 40 years has been conducted with single-source

168

Text Relevance and Multiple-Source Use  •  169

narrative or expository text and limited focus has been placed on the source. This is largely because researchers in the area of reading comprehension have been primarily concerned with understanding students’ comprehension of text and identifying ways to improve comprehension, particularly for struggling readers in schools. In such circumstances, source information may be unavailable, or available yet peripheral to the focus of the reading task. More recently, however, the research focus has expanded to include not only students’ comprehension of text, but also their evaluation and integration of information across sources (e.g., Goldman & Scardamalia, 2013). This is particularly important given the increased access to and reliance on information from the Internet, which can vary greatly with respect to its relevance and reliability (Goldman, Braasch, Wiley, & Brodowinska, 2012). The shift in emphasis from comprehension of single texts from one source to the evaluation and integration of multiple texts from several sources has led to a greater emphasis on the role that source credibility plays in readers’ selection, processing, and use of text information. The remainder of this chapter consists of five main sections. The first section defines text relevance and describes the goal-focusing model. The second section distinguishes text relevance from text importance. The third section discusses text relevance and source credibility in the context of multiple-source use. The fourth section discusses text relevance and source credibility in relation to usefulness. The last section offers key questions and directions for future research. Where appropriate, several studies are described in detail to illustrate important empirical findings that relate to ideas presented in the chapter.

TEXT RELEVANCE AND GOAL-FOCUSING As indicated above, reading is a goal-directed task. Importantly, our reading goals can affect reading processes that support our understanding of text. Specifically, goals affect moment-by-moment reading processes that readers use to comprehend and interpret text information. For instance, readers generally direct more attention toward goal-relevant information and use more effortful reading strategies when reading this information (Kaakinen & Hyönä, 2011; Kaakinen, Hyönä, & Keenan, 2002; Linderholm, Kwon, & Wang, 2011; Magliano et  al., 1999; McCrudden & Schraw, 2007; Wiley et al., 2009). These reading processes form the basis of a reader’s mental model for a reading experience. A coherent mental model consists of a mental network of interrelated propositions that mirrors content that is stated explicitly and the reader’s inferences that are used to determine how the explicit content is interconnected and related to background knowledge (Kintsch, 1998; Magliano, McCrudden, Rouet, & Sabatini, in press; Zwaan & Radvansky, 1998). The mental model that develops during reading tends to reflect information that is deemed more relevant to readers’ goals (relative to information that is less relevant) (Kaakinen & Hyönä, 2011; McCrudden & Schraw, 2007; van den Broek & Helder, in press). Thus, readers’ mental models reflect their determinations of text relevance, the perceived instrumental value of text information for meeting a goal for reading, and their standards of relevance, the criteria readers use to determine the relevance of text information in relation to their goals (Lehman & Schraw, 2002; McCrudden,

170  •  Matthew T. McCrudden

Magliano, & Schraw, 2011). Information that more effectively helps readers meet their goals is more relevant, whereas information that less effectively helps readers meet their goals is comparatively less relevant. Thus, readers’ goals inform their determinations of text relevance and their standards of relevance. In the context of an assigned reading task, pre-reading task instructions initially can guide determinations of text relevance before a reading experience. The goalfocusing model (McCrudden, Magliano, & Schraw, 2011; McCrudden & Schraw, 2007) describes how readers’ goals affect moment-by-moment processes that form the basis of a reader’s mental model for a reading experience (see Figure 10.1). It is a heuristic model that describes reading for assigned and self-chosen reading tasks (for a more in-depth discussion of the construction of a reader’s mental representations and processes used to create these representations, refer to Britt, Rouet, & Durik, this volume and van den Broek & Helder, 2017). It is important to note that these events occur dynamically as the reading experience unfolds such that the relations between these events can be bi-directional and recursive, as indicated by the two-way arrows in Figure 10.1. For instance, task instructions may initially inform readers’ goals, but readers may modify and refine these standards as they build a mental representation of the textual information. Task instructions, such as those provided by a teacher, orient a reader toward a reading task in three interrelated ways. First, they signal why students should read, or the purpose for reading, such as reading to write an essay or to prepare for an exam (Alexander & Jetton, 1996; Jetton & Alexander, 1997; Ramsay & Sperling, 2011). Second, task instructions signal what students should read. Task instructions affect text relevance and standards of relevance, such that some information more effectively helps readers meet their goals for reading (Lehman & Schraw, 2002; McCrudden, Magliano, & Schraw, 2011). Third, task instructions signal how students should read, or strategies that could be useful for the task (Cerdán & Vidal-Abarca, 2008; Gil, Bråten, Vidal-Abarca, & Strømsø, 2010; Linderholm et  al., 2011; Lorch, Lorch, & Klusewitz, 1993; Magliano, Trabasso, & Graesser, 1999; van den Broek, Lorch, Linderholm, & Gustafson, 2001). For instance, students use different reading strategies when they read for different purposes (e.g., van den Broek et al., 2001). Thus, task instructions potentially signal why students should read, what they should read, and how they should read.

Task instructions Goals for reading Reader intentions

Figure 10.1  Goal-Focusing Model.

Moment-bymoment processing

Learning and memory

Text Relevance and Multiple-Source Use  •  171

Importantly, readers’ personal intentions also affect determinations of relevance in both assigned and self-chosen reading tasks. Readers’ personal intentions, informed by their knowledge, beliefs, values, expectations, and experiences, can affect their decisions about why, what, and how to read. For instance, Bohn-Gettler and McCrudden (in press) asked participants to read a dual-position text about a controversial topic (i.e., whether intelligent design should be taught in science classrooms). Participants were either asked to focus on arguments ‘for’ or arguments ‘against’ while they did a think-aloud reading protocol. Their topic beliefs were also measured. Participants used confirmation strategies when they read belief-consistent segments and disconfirmation strategies when they read belief-inconsistent segments, independently of whether the information was relevant to their task instructions. So, readers who read the same text and are given the same task instructions may develop different goals and standards of relevance. Thus, goals could reflect readers’ personal intentions, provided task instructions, or both depending on the context. Further, readers’ determinations of relevance can be updated during a reading experience (Rapp & Gerrig, 2002; Rapp & McCrudden, in press; van den Broek & Helder, in press). For instance, task instructions may initially inform readers’ determinations of relevance, but readers may modify and refine these standards as they build a mental representation of the textual information. When reading to determine whether a dolphin is a mammal or a fish, a young reader who has just read that fish use gills to breathe may seek to determine how dolphins breathe, a goal that s/he generated during the course of reading, which in turn can affect subsequent processing. A mixed-methods study by McCrudden, Magliano, and Schraw (2010) illustrates how task instructions and personal intentions can affect determinations of relevance for an assigned reading task. In the quantitative phase of the study, undergraduates read a text passage about several remote countries, including Pitcairn and Honduras. The passage described the same features of each country including geography, climate, government, economy, transportation, and language. Participants were randomly assigned to one of three conditions and were given different instructions before they read. Students in the Pitcairn condition were asked to imagine they would be living in Pitcairn for several years and to focus on information about Pitcairn. Students in the Honduras condition were asked to imagine they would be living in Honduras for several years and to focus on information about Honduras. Students in the control condition were asked to read for understanding. Reading time was recorded as a measure of moment-by-moment processing; after reading, the students did a free recall test to measure memory for text information. The reading time data showed that participants spent more time reading information that was relevant to their pre-reading instructions (see Figure 10.2). For instance, students in the Pitcairn condition spent more time reading information about Pitcairn than information about Honduras. The same pattern was true for the participants who were asked to focus on Honduras, except they spent more time on information about Honduras. Students in the control group spent the same amount of time reading both types of information. These findings showed that the pre-reading instructions affected students’ moment-by-moment processing, such that they allocated more attention to information that was signaled by the prereading task instructions.

READING TIME PER WORD IN MILLISECONDS

172  •  Matthew T. McCrudden

0.400

0.350

0.300

0.250

0.200

0.150 Pitcairn group

Honduras group

Pitcairn sentences

Control group

Honduras sentences

Figure 10.2  The Reading Time Results from McCrudden et al. (2010). Reading time is reported in milliseconds per word for the most relevant sentences in the text. Higher scores are indicative of slower reading times. Error bars are standard errors of the mean.

On the recall test, students remembered more of the information that was signaled by the task instructions (see Figure 10.3). For instance, students in the Pitcairn condition remembered more information about Pitcairn than about Honduras. Similarly,

PROPORTION RECALLED

0.6 0.5 0.4 0.3 0.2 0.1 6E-16 Pitcairn group

Honduras group

Control group

-0.1 Pitcairn sentences

Honduras sentences

Figure 10.3  The Free Recall Results from McCrudden et al. (2010). Proportion of recall of the most relevant ideas in the text passage. There were 22 highly relevant ideas for Pitcairn and 26 for Honduras. Error bars are standard errors of the mean.

Text Relevance and Multiple-Source Use  •  173

students in the Honduras condition remembered more information about Honduras than about Pitcairn. Collectively, participants spent more time reading task-relevant information, and they remembered this information better than task-irrelevant information. However, in the follow-up qualitative phase, interviews indicated that participants implemented task instructions differently, such that some students focused exclusively on task-relevant information, whereas others focused on both task-relevant and task-irrelevant information, which indicated that personal intentions also affected determinations of relevance. For instance, one student who focused almost exclusively on task-relevant information said, “I focused more on the sections that dealt with Honduras and quickly went over the rest of it because it wasn’t as relevant or didn’t seem as relevant” (p. 236). Conversely, one student who focused on both task-relevant and task-irrelevant information said, “I focused on the comparison of the other countries to Pitcairn. I took Pitcairn and I read all this other information and then I sat there and I compared and contrasted the other three countries to Pitcairn” (p. 236). The reading time and recall data corroborated the interview data. For instance, students in the experimental conditions who spent more time reading irrelevant sentences also recalled more of this information than students who spent less time reading irrelevant sentences. The interview data showed that students who spent more time reading task-irrelevant sentences said that they evaluated some task-irrelevant information, whereas students who spent less time reading taskirrelevant sentences said they processed task-irrelevant information minimally. Thus, readers who are given the same task instructions and read the same text may process the text differently, and remember different elements of the text. These differences suggest that both task instructions and personal intentions affected determinations of relevance before and during the reading process.

TEXT RELEVANCE DIFFERS FROM TEXT IMPORTANCE The relevance and importance of text information are not the same (McCrudden & Schraw, 2007; Schraw et al., 1993). Information is relevant to the extent that it is instrumental in helping a reader meet a reading goal, whereas information is important to the extent that it is necessary for helping a reader comprehend or establish the coherence of a particular text. A particular text segment can be important to understanding a section of text, for instance, but a reader may judge it to be irrelevant and potentially allocate less attention to that information. For example, suppose you are seated on an airplane awaiting your departure. The flight crew asks you to read about the safety features of the plane. An excerpt in the document provides the following information about what to do if the plane has to make an emergency landing on water: Life jackets are found under your seat. In the event of the plane having to land on water, put your head through the hole and pull the jacket over your head. Click the waist band clip and tighten your belt. To inflate your jacket, pull the tabs at the end of the chords. If more inflation is needed, blow through the tube on the jacket. Do not fully inflate your jacket until you are leaving the plane. If you are flying over the Pacific Ocean from Los Angeles to Hawaii, you may view this information as highly relevant because you know the plane’s path will be almost entirely over water and it will enable you to locate and use your life jacket. However, if you are

174  •  Matthew T. McCrudden

PROPORTION OF SEGMENTS OF RECALLED

flying over desert from Los Angeles to Las Vegas, you may view this information as having little or no relevance because you know the plane’s path will be entirely over land, specifically over a desert, and you have no expectation that the plane will land on water. Although the relevance of the information changes based on the situation, the segment about how to inflate the jacket, for example, remains a very important segment in the text because it explains how to inflate the jacket. This information is necessary; if this segment of text was missing, the procedure for inflating the life jacket would not be clear. Thus, the relevance of information in relation to a reading goal may be different from the importance of the information in relation to the coherence of the text (McCrudden & Schraw, 2007; Schraw, Wade, & Kardash, 1993) or what a teacher considers to be important (Jetton & Alexander, 1997). A study by Schraw et al. (1993; Experiment 2) illustrates the difference between relevance and importance with respect to memory for text information. Undergraduates read a short narrative text about two boys who left school early and then went to the house of one of the boys, an expanded version of the text used by Pichert and Anderson (1977). Participants were randomly assigned to one of two groups and were given different instructions before they read. Students in the burglar group were asked to read the story the story from the perspective of a person thinking about burglarizing the house. Students in homebuyer group were asked to read the story from the perspective of a person thinking about buying the house. After they read, all students did a five-minute short distracter task (i.e., vocabulary test) and then were asked to recall as much of the text as possible. When information was relevant to an assigned perspective, it was recalled equally well independently of whether it was of high, medium, or low importance to understanding the text (see Figure 10.4). For example, students in the burglar group recalled

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Burglar Burglar Burglar Homebuyer Homebuyer Homebuyer segments: segments: segments: segments: segments: segments: High Med Low High Med Low importance importance importance importance importance importance Burglar group

Homebuyer group

Figure 10.4  The Results from Schraw et al. (1993). The results are presented in terms of the proportion of segments recalled out of five for each category. Error bars are standard errors of the mean.

Text Relevance and Multiple-Source Use  •  175

burglar information equally well independently of its importance. However, when information was not relevant to an assigned perspective, recall for the information was primarily based on its importance (see Figure 10.4). For example, students in the burglar group recalled more homebuyer information when it was of high importance than when it was of low importance. Thus, subjective determinations of relevance may differ from more objective measures of importance.

TEXT RELEVANCE AND SOURCE CREDIBILITY As indicated earlier in the chapter, research on text relevance has generally involved the use of pre-reading instructions (e.g., pre-reading questions or pre-reading perspectives) to investigate reading processes and memory from single-author texts. However, the focus on multiple-source use has led to a greater emphasis on the role that source credibility plays in readers’ selection, processing, and use of text information (Britt & Aglinskas, 2002; Bråten, Strømsø, & Britt, 2009; Wiley et al., 2009). Further, in the context of multiple-source use, readers may evaluate the perceived relevance of a document based on features such as the title or source (e.g., author, publisher). When a relevant document is accessed, information within that document may also be more or less relevant to the reader’s goals. Source credibility refers to the perceived expertise and trustworthiness of the author (Pornpitakpan, 2004). Author expertise is the reader’s perception of the author’s ability to make correct assertions. Author trustworthiness is the reader’s perception that the author believes the information that s/he communicates is valid. In the context of multiple-source use, the credibility of a source can affect readers’ standards of relevance and judgments of text relevance. That is, information may be topically relevant, but perceptions of source credibility may also determine whether readers select, process, or use the information (Anmarkrud, Bråten, & Strømsø, 2014). A mixed-methods study by McCrudden, Stenseth, Bråten, and Strømsø (2016) illustrates the effect of source credibility on students’ selection of relevant documents for an academic task. In the quantitative phase of the study, undergraduates were asked to select web pages to prepare for a class presentation on a more familiar topic (i.e., climate change), and later did the same task for a less familiar topic (i.e., nuclear power). The web pages included source information (e.g., credentials, affiliation) and a summary of the information. Further, documents varied with respect to author expertise (higher or lower expertise) and text relevance (more or less relevant). For both topics, students selected a greater number of documents from higherexpertise authors (than from lower-expertise authors) and a greater number of more-relevant documents (than less-relevant documents) (see Figure 10.5). However, the effect of author expertise was more pronounced for the less familiar topic when students selected more-relevant documents. Specifically, when students selected more-relevant documents, they relied more heavily on higher-expertise authors versus lower-expertise authors for the less familiar topic than when they selected more-relevant documents for the more familiar topic. This suggests that source credibility affects students’ selection of highly relevant content, such that they rely more heavily on higher-expertise authors when a topic is less familiar, even when content is more relevant.

PROPORTION OF DOCUMENTS SELECTED

176  •  Matthew T. McCrudden

1.20 1.00 0.80 0.60 0.40 0.20 0.00 More relevant, Higher expertise More familiar topic

More relevant, Lower expertise Less familiar topic

Figure 10.5  The Results from McCrudden et al. (2016). The results are presented in terms of the proportion of more-relevant documents selected (out of five) for the more and less familiar topics from authors with higher or lower expertise. Error bars are standard errors of the mean.

In the follow-up qualitative phase of the study, interviews provided insights into why students valued author expertise to a greater extent for the less familiar topic. Three main themes were identified. First, students believed that higher-expertise authors (i.e., scientists who study the topic) had a greater ability to make correct scientific assertions than lower-expertise authors (i.e., journalists). Second, when information was more familiar and considered to be general knowledge, students saw less need to rely on author expertise when selecting information to use. Third, when information was less familiar and not considered to be general knowledge, students saw more need to rely on author expertise. In sum, source credibility affects document selection, and its impact is much greater when students are less familiar with a topic. Thus, the credibility of a source can ultimately affect the extent to which a reader deems text information to be task-relevant (Goldman et al., 2012; Mason, Ariasi, & Boldrin, 2011; Mason, Junyent, & Tornatora, 2014; Rouet & Britt, 2011; Van Boekel, Lassonde, O’Brien, & Kendeou, 2017). Specifically, readers who are less familiar with a topic may focus on the credibility of the source when determining the task-relevance of information (Bromme & Thomm, 2016; Lucassen, Muilwijk, Noordzij, & Schraagen, 2013; Lucassen & Schraagen, 2013; Stadtler & Bromme, 2014). In a subsequent study, Bråten, McCrudden, Lund, Brante, and Strømsø (in press) measured students’ selection, processing, and use of multiple documents. Secondary students were asked to select documents (i.e., web pages) to write a letter to the editor either on a more familiar topic (i.e., climate change) or a less familiar topic (i.e., nuclear power). The documents varied with respect to author expertise (higher or lower expertise) and text relevance (more or less relevant). Students received a list of the documents, which included source information (e.g., credentials, affiliation) and a summary of the information contained in the document. Once students had selected

Text Relevance and Multiple-Source Use  •  177

documents for the online task, they were able to read the expanded texts and write their letters to the editor. Reading time for the expanded texts was recorded as a measure of processing. Information units from the various documents that were included in the letter to the editor were scored as a measure of document use. With respect to document selection, students selected a greater number of morerelevant documents than less-relevant documents, independently of topic familiarity. Similarly, students selected a greater number of documents from higher-expertise authors than from lower-expertise authors. However, the effect of author expertise was more pronounced for the less familiar topic. Specifically, when students selected more-relevant documents for the less familiar topic, they relied more heavily on higher-expertise authors. These findings replicated McCrudden et  al. (2016), who found that source credibility affects students’ selection of highly relevant content, such that they rely more on higher-expertise sources for a less familiar topic. With respect to reading time, students spent a greater amount of time on morerelevant documents than on less-relevant documents, independently of topic familiarity. They also spent more time reading documents from higher-expertise authors than lower-expertise authors. However, the effect of author expertise was more pronounced for the less familiar topic. Specifically, when students read more-relevant documents, they generally spent more time on documents from authors with higher expertise. The use of information units provided a measure of the extent to which students included information from each document type in their letters to the editor. Students used more information units from more-relevant documents than from lessrelevant documents, independently of topic. They also included more information units from documents from higher-expertise authors than from lower-expertise authors. Again, the effect of author expertise was more pronounced for the less familiar topic. Specifically, students generally included more ideas units from more-relevant documents from higher-expertise authors than from lower-expertise authors. In summary, both relevance and author expertise affected document selection, processing, and use. Students selected, processed, and used documents that contained more-relevant information to a greater extent than documents that contained less-relevant information. Similarly, students selected, processed, and used documents that came from higher-expertise authors more than from lower-expertise authors. However, author expertise was more influential for the less familiar topic when students selected, processed, and used more-relevant documents. Thus, these data suggested that students’ use of relevant information is affected by the source of that information. In other words, when the topic was less familiar it not only mattered what was said, but also who said it.

TEXT RELEVANCE, SOURCE CREDIBILITY, AND USEFULNESS Historically, text relevance has been investigated with single-source texts, such as narrative texts (e.g., Pichert & Anderson, 1977; Schraw et al., 1993) or descriptive expository texts (e.g., Kaakinen et al., 2002; McCrudden et al., 2010). In a typical experiment, a researcher would randomly assign participants to different pre-reading task instruction conditions, then measure moment-by-moment processing of and memory for text segments that are more or less relevant to the assigned task instructions. For instance, in the classic burglar/homebuyer studies inspired by Pichert and Anderson (1977),

178  •  Matthew T. McCrudden

some segments of the text are more relevant to a burglar perspective, whereas other text segments are more relevant to a homebuyer perspective. A researcher would then randomly assign some participants to read from a burglar’s perspective and other participants to read from a homebuyer’s perspective, measure processing of segments that are relevant to each perspective, and score recall protocols for the frequency of idea units recall from each perspective. Thus, text relevance has been predominantly investigated in the context of single-source texts in which source information is unavailable or peripheral to the reading task. In the context of multiple-source reading, factors other than text relevance, such as the source information (e.g., author, publisher), may affect whether text information is selected, processed, and used. For instance, in McCrudden et al. (2016), when students selected more-relevant documents for a less familiar topic (compared to a more familiar topic), they relied more heavily on higher-expertise authors (than lowerexpertise authors). In another example, Van Boekel et al. (2017) asked participants to read refutation texts in which a misconception statement (e.g., humans only use 10% of their brain) was refuted by either a higher- or lower-credibility source (e.g., professor vs. a participant in a study) and to pay particular attention to the source. When the misconception was referenced later in the passage, participants spent more time reading the information than when the refutation was attributed to a lower-credibility source. In other words, readers were less likely to engage in knowledge revision (i.e., update a misconception) when the source lacked credibility. Lastly, in Bråten et al. (in press) when students used idea units from more-relevant documents for a less familiar topic compared to a more familiar topic, they relied more heavily on higher-expertise authors. Thus, the extent to which relevant content is selected, processed, and used can be affected by the source of the information. Therefore, it is important for researchers to be clear about distinctions between text relevance and source credibility and how each of these constructs may individually and jointly affect the usefulness of information (i.e., the extent to which information contributes to the fulfillment a goal), particularly for unfamiliar topics. For instance, highly relevant information from a lower-credibility source may be less useful to a reader than highly relevant information from a more credible source (Bråten et al., in press). As such, in the context of multiple-source use, text relevance and source credibility can provide independent contributions to the usefulness of text segments in service of readers’ goals.

CONCLUSION AND FUTURE DIRECTIONS This chapter provided an overview of readers’ determinations of text relevance in single-source and multiple-source contexts. Readers in the modern era face a number of challenges when they need to identify, evaluate, and integrate information from multiple sources that may contain overlapping, unique, and conflicting messages. Developing technology gives us unprecedented access to information. However, the increase in accessibility to information means that we need to more selective in determining the relevance of information to any particular task. For instance, not only must we evaluate the relevance of information, we increasingly need to evaluate the credibility of the source to determine the extent to which the information is useful.

Text Relevance and Multiple-Source Use  •  179

Researchers are obviously well on their way to investigating factors that affect multiple-source use. Given the added dimensions of multiple-source use compared to comprehension of single-source text, there are several key questions that warrant further investigation. First, it is important to understand the interplay between determinations of text relevance (which pertain to the content of a message) and determinations of source credibility (which pertain to characteristics of the source). It may be fruitful to conceptualize usefulness as function of text relevance and source credibility (see Figure 10.6). In Figure 10.6, text relevance lies along the x-axis and ranges from more to less, and source credibility lies along the y-axis and ranges from higher to lower. Such a framework may enable researchers to investigate when and why readers use information from documents that roughly fall within each of these quadrants. For instance, McCrudden et al. (2016) found that students selected more documents from the top-left quadrant when they were asked to gather information for a class presentation on a more familiar topic, whereas they selected fewer documents from this quadrant when they gathered information for a less familiar topic. Determinations of source credibility are informed by source expertise and source trustworthiness, both of which can affect perceptions of usefulness for more-relevant information. For instance, suppose a student is given several sources that contain various perspectives on eating meat, and is asked to write an argument to address the following question: Should people eat meat? The student reads the following comment from a physician: There are many health benefits, including lower cholesterol level and lower blood pressure, associated with replacing meat with other food sources, such as beans and nuts. However, a vegetarian diet is not a very efficient way for people to get enough protein, iron, and vitamin B12. Meat contains all of these nutrients. Later, the student reads the following from a lobbyist for the beef industry: ‘Reducing meat consumption could damage an important domestic industry’. The student could view statements from both sources to be relevant to the task; however, the student

More text relevance, Lower source credibility

More text relevance, Higher source credibility

Less text relevance, Lower source credibility

Less text relevance, Higher source credibility

Figure 10.6  Quadrants of Usefulness in Multiple-Source Reading as Function of Text Relevance and Source Credibility.

180  •  Matthew T. McCrudden

may view the statement from the physician as more relevant because the source has higher expertise and is perceived as more trustworthy than the lobbyist, who has a vested interest in promoting meat consumption. As this example illustrates, future research could investigate different dimensions of source credibility in relation to text relevance how they may affect the perceived usefulness of information. Second, future research could investigate the updating of goals and determinations of relevance as a reading experience unfolds. In the previous example, the student who was reading to answer the question, ‘Should people eat meat?’ may have initially focused on the health benefits associated with not eating meat, but later included a focus on the impact on industry (e.g., if people replace meat with other food sources, then eating other food sources could benefit domestic industry in other ways). Reading goals affect moment-by-moment processing and the reader’s mental model. As goals are updated or changed in some way, processing should be affected, as should the reader’s mental model. Thus, future research could investigate how goals and determinations of relevance are established before a reading experience, and whether and how they change during the reading experience, as well as changes in processing and the reader’s mental representation. Third, researchers could investigate whether the relevance and usefulness of text segments differ between single-source and multiple-source reading contexts. It may be the case that relevance and usefulness are psychologically indistinguishable when information comes from a single source. However, this may not necessarily be the case if the information is attributed to different sources, particularly if the sources vary with respect to credibility. Previous research has shown that more-relevant information from authors with higher expertise is more useful when the topic is unfamiliar (Bråten et al., in press; McCrudden et al., 2016). Based on these findings, it is reasonable to assume that usefulness may change based on the number of sources. Further, is this information represented differently in the reader’s mental model based on the number and credibility of the sources? For instance, do readers create different mental models for each source, do they create mental models based on an event, or are both the source and the event incorporated into the mental model (de Pereyra, Britt, Braasch, & Rouet, 2014)? And, does the mental model of the reading experience change over time, such that source attributions become lost or blurred and does information originally attributed to less credible sources gain prominence in the mental model? This is an area ripe for research. Lastly, one reader characteristic that may affect determinations of relevance is topic beliefs. Research in social, political, and cognitive psychology has shown readers’ beliefs guide processing and source evaluation. Readers commonly evaluate information differently based on whether the information is consistent with their beliefs rather than on the quality of the information (Klaczynski, 2000). Specifically, they commonly apply higher evaluation standards (e.g., criteria used to justify one’s evaluations) to beliefinconsistent information, and as a result, evaluate belief-consistent information more favorably than belief-inconsistent information, independently of the quality of the information (McCrudden & Barnes, 2016). Thus, a key research question is: When students select and evaluate belief-related information across multiple documents, to what extent does the belief-consistency of the information affect their process of the information? Further, how is this information represented in memory (Maier & Richter, 2016)?

Text Relevance and Multiple-Source Use  •  181

In conclusion, unprecedented access to information from multiple sources amplifies the need to be able to identify relevant information and to critically evaluate sources. Hopefully, this chapter has provided some beneficial ideas to consider as we try to understand relevance processing in the context of multiple-source use.

REFERENCES Alexander, P. A., & Jetton, T. L. (1996). The role of importance and interest in the processing of text. Educational Psychology Review, 8(1), 89–122. Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. Bohn-Gettler, K., & McCrudden, M. T. (in press). The effects of task relevance instructions and topic beliefs on reading processes and memory. Discourse Processes. doi: 10.1080/0163853X.2017.1292824. Bråten, I., Britt, M. A., Strømsø, H. I., & Rouet, J. F. (2011). The role of epistemic beliefs in the comprehension of multiple expository texts: Toward an integrated model. Educational Psychologist, 46, 48–70. Bråten, I., McCrudden, M. T., Lund, E. S., Brante, E. W., & Strømsø, H. I. (in press). Task-oriented learning with multiple documents: Effects of topic familiarity, author expertise, and content relevance on document selection, processing, and use. Reading Research Quarterly. Bråten, I., & Strømsø, H. I. (2012). Knowledge acquisition: Constructing meaning from multiple information sources. In N. S. Seel (Ed.), Encyclopedia of the sciences of learning (Part 11, pp. 1677–1680). Heidelberg: Springer. Bråten, I., Strømsø, H. I., & Britt, M. A. (2009). Trust matters: Examining the role of source evaluation in students’ construction of meaning within and across multiple texts. Reading Research Quarterly, 44, 6–28. Britt, M. A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20, 485–522. Britt, M. A., Rouet, J. F., & Braasch, J. L. G. (2013). Documents as entities: Extending the situation model theory of comprehension. In M. A. Britt, S. R. Goldman, & J. F. Rouet (Eds.), Reading: From words to multiple texts (pp. 160–179). New York: Routledge. Bromme, R., & Thomm, E. (2016) Knowing who knows: Laypersons’ capabilities to judge experts’ pertinence for science topics. Cognitive Science, 40, 241–252. Cerdán, R., & Vidal-Abarca, E. (2008). The effects of tasks on integrating information from multiple documents. Journal of Educational Psychology, 100, 209–222. de Pereyra, G., Britt, M. A., Braasch, J. L., & Rouet, J. F. (2014). Readers’ memory for information sources in simple news stories: Effects of text and task features. Journal of Cognitive Psychology, 26, 187–204. Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H. I. (2010). Summary versus argument tasks when working with multiple documents: Which is better for whom? Contemporary Educational Psychology, 35, 157–173. Goldman, S. R., Braasch, J. L. G., Wiley, J., & Brodowinska, K. (2012). Comprehending and learning from internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381. Goldman, S. R., & Scardamalia, M., (2013). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities. Cognition and Instruction, 31, 255–269. Graesser, A. C., & Lehman, B. A. (2011). Questions drive comprehension of text and multimedia. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 53–74). Greenwich, CT: Information Age Publishing. Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological Review 101, 371–395. Jetton, T. L., & Alexander, P. A. (1997). Instructional importance: What teachers value and what students learn. Reading Research Quarterly, 32, 290–308. Kaakinen, J. K., & Hyönä, J. (2011). Online processing of and memory for perspective relevant and irrelevant text information. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 223–242). Greenwich, CT: Information Age Publishing. Kaakinen, J. K., Hyönä, J., & Keenan, J. M. (2002). Perspective effects on online text processing. Discourse Processes, 33(2), 159–173.

182  •  Matthew T. McCrudden Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, UK: Cambridge University Press. Klaczynski, P. A. (2000). Motivated scientific reasoning biases, epistemological beliefs, and theory polarization: A two-process approach to adolescent cognition. Child Development, 71, 1347–1366. Lehman, S., & Schraw, G. (2002). Effects of coherence and relevance on shallow and deep text processing. Journal of Educational Psychology, 94, 738–750. Linderholm, T., Kwon, H., & Wang, X. (2011). The effects of reading purpose on advanced readers. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 197–222). Greenwich, CT: Information Age Publishing. Lorch, R. F., Jr., Lorch, E. P., & Klusewitz, M. A. (1993). College students’ conditional knowledge about reading. Journal of Educational Psychology, 85(2), 239–252. Lucassen, T., & Schraagen, J. M. (2013). The influence of source cues and topic familiarity on credibility evaluation. Computers in Human Behavior, 29, 1387–1392. Lucassen, T., Muilwijk, R., Noordzij, M. L., & Schraagen, J. M. (2013). Topic familiarity and information skills in online credibility evaluation. Journal of the American Society for Information Science and Technology, 64, 254–264. Macedo-Rouet, M., Braasch, J. L. G., Britt, A. M., & Rouet, J.-F. (2013). Teaching fourth and fifth graders to evaluate information sources during text comprehension. Cognition and Instruction, 31(2), 204–226. Maier, J., & Richter, T. (2016). Effects of text-belief consistency and reading task on the strategic validation of multiple texts. European Journal of Psychology in Education, 31, 479–497. Magliano, J., McCrudden, M. T., Rouet, J. F., & Sabatini, J. (in press). The modern reader: Should changes to how we read affect research and theory? In M. F. Schober, D. N. Rapp, & M. A. Britt (Eds.), Handbook of discourse processes (2nd ed.). London: Taylor & Francis. Magliano, J. P., Trabasso, T., & Graesser, A. C. (1999). Strategic processes during comprehension. Journal of Educational Psychology, 91, 615–629. Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action: spontaneous reflections about knowledge and knowing during online information searching and their influences on learning. Learning and Instruction, 21, 137–151. Mason, L., Junyent, A. A., & Tornatora, M. C. (2014). Epistemic evaluation and comprehension of web-source information on controversial science-related topics: Effects of a short-term instructional intervention. Computers & Education, 76, 143–157. McCrudden, M. T., & Barnes, A. (2016). Differences in student reasoning about belief-relevant arguments: A mixed methods study. Metacognition & Learning, 11(3), 275–303. McCrudden, M. T., Magliano, J., & Schraw, G. (2010). Exploring how relevance instructions affect personal reading intentions, reading goals, and text processing: A mixed methods study. Contemporary Educational Psychology, 35(4), 229–241. McCrudden, M. T., Magliano, J. P., & Schraw, G. (2011). Relevance in text comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp.  1–17). Greenwich, CT: Information Age Publishing. McCrudden, M. T., & Schraw, G. (2007). Relevance and goal-focusing in text processing. Educational Psychology Review, 19(2), 113–139. McCrudden, M. T., Stenseth, T., Bråten, I., & Strømsø, H. I. (2016). The effects of topic familiarity, author expertise, and content relevance on Norwegian students’ document selection: A mixed methods study. Journal of Educational Psychology, 108(2), 147–162. Pichert, J. W., & Anderson, R. C. (1977). Taking different perspectives on a story. Journal of Educational Psychology, 69, 309–315. Pornpitakpan, C. (2004). The persuasiveness of source credibility: A critical review of five decades’ evidence. Journal of Applied Social Psychology, 34, 243–281. Ramsay, C. M., & Sperling, R. A. (2011). The relevance of purpose: Why purpose situates relevance instructions. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 243–265). Greenwich, CT: Information Age Publishing. Rapp, D. N., & Gerrig, R. J. (2002). Readers’ reality-driven and plot-driven analyses in narrative comprehension. Memory & Cognition, 30, 779–788. Rapp, D. N., & McCrudden, M. T. (in press). Relevance before, during, and after discourse experiences. In J. Strassheim & H. Nasu (Eds.), Relevance and irrelevance: Theories, factors, and challenges. Berlin: De Gruyter.

Text Relevance and Multiple-Source Use  •  183 Rouet, J. F., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 19–52). Greenwich, CT: Information Age Publishing. Schraw, G., Wade, S. E., & Kardash, C. A. (1993). Interactive effects of text-based and task-based importance on learning from text. Journal of Educational Psychology, 85, 652–661. Stadtler, M., & Bromme, R. (2014). The content-source integration model: A taxonomic description of how readers comprehend conflicting scientific information. In D. N. Rapp & J. L. G. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 379–402). Cambridge, MA: The MIT Press. Van Boekel, M., Lassonde, K. A., O’Brien, E. J., & Kendeou, P. (2017). Source credibility and the processing of refutation texts. Memory and Cognition, 45, 168–181. van den Broek, P., & Helder, A. (2017). Cognitive processes in discourse comprehension: Passive processes, reader-initiated processes, and evolving mental representations. Discourse Processes, 54(5–6), pp. 360–372. van den Broek, P., Lorch, R. F., Linderholm, T., & Gustafson, M. (2001). The effects of readers’ goals on inference generation and memory for texts. Journal of Memory & Cognition, 29(8), 1081–1087. Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerick, J. (2009). Source evaluation, comprehension, and learning in internet science inquiry tasks. American Educational Research Association Journal, 46, 1060–1106. Zwaan, R. A., & Radvansky, G. A. (1998). Situation models in language comprehension and memory. Psychological Bulletin, 123(2), 162–185.

11

THE ROLE OF CONFLICT IN MULTIPLE SOURCE USE Ivar Bråten university of oslo, norway

Jason L. G. Braasch university of memphis, usa

INTRODUCTION The purpose of this chapter is to review theory and research on the role of conflict in multiple source use, discuss implications for educational research and practice, and suggest future directions. Focusing on textual discourse involving multiple conflicting sources, we define sources as information about individuals and organizations that create and publish textual content, including information about when, where, and for what purpose the content is created and published (cf., Britt, Rouet, & Braasch, 2013; Goldman & Scardamalia, 2013). Accordingly, we define the process of sourcing as attending to, representing, evaluating, and using available or accessible information about the sources of textual content, for example about the author or publisher (Bråten, Stadtler, & Salmerón, 2018). Finally, conflicts between multiple sources refer to situations where two or more sources present opposing or discrepant views on the same situation or issue (Braasch, Rouet, Vibert, & Britt, 2012; Stadtler & Bromme, 2014). While conflicts between sources may exist because sources try to persuade people toward their positions (Petty & Briñol, 2012), or even try to intentionally convey misinformation (Lewandowsky & Oberauer, 2016), multiple sources also may present conflicting explanations, arguments, and conclusions about an issue because real controversies exist between sources with informative purposes. Take, for example, the topic of sun exposure and health, which can be considered a controversial scientific issue with personal as well as public health implications (Moan, Baturaite, Juzeniene, & Porojnicu, 2012). Laypersons living in northerly regions of the world may want to understand this issue as a basis for deciding whether they should use sunbeds during the winter. These individuals may come across warnings against such behavior in

184

The Role of Conflict  •  185

newspaper articles authored by medical professionals who highlight the causal relation between ultraviolet radiation and skin cancer. At the same time, however, they may have read on a separate website produced by an indoor tanning organization that scientists have now proven that exposure to all forms of ultraviolet light—both indoors and out—actually protects against cancer because it stimulates the production of vitamin D. Moreover, the latter view may have been corroborated by a popular science article that they came across in a research magazine, where a recognized scientist explains how vitamin D protects against all forms of cancer and recommends people to use a sunbed weekly during the winter. Of course, the described scenario is but one realistic example of conflict in multiple source use that can have important consequences for individuals’ meaning-making processes and comprehension. Similar controversial socio-scientific issues are ubiquitous and conflicting views on them are now just a mouse-click away (Bråten & Braasch, 2017; Bromme & Goldman, 2014). Thus, in the 21st-century literacy context, where individuals have unprecedented access to vast amounts of sources that present conflicting views on almost any issue (Magliano, McCrudden, Rouet, & Sabatini, 2018), understanding literacy competencies involves understanding how individuals deal with conflict in multiple source use. Not surprisingly, dealing with conflict in multiple source use is a great challenge to many individuals regardless of age and educational level, which may sometimes lead to confusion rather than clarity regarding the issue at hand (Rouet, 2006). At the same time, however, working with multiple conflicting sources may have beneficial effects on individuals’ processing and comprehension. That is, compared to working with multiple consistent sources or a single source that presents only one view on an issue, multiple conflicting sources may allow individuals to construct a deeper, more flexible understanding of an issue that also takes information about the sources themselves into consideration (Britt & Aglinskas, 2002; van den Broek & Kendeou, 2015; Ferguson, Bråten, Strømsø, & Anmarkrud, 2013; Jacobson & Spiro, 1995; Perfetti, Britt, & Georgi, 1995). Importantly, however, the extent to which learners are able to reap the potential benefits of working with multiple conflicting sources may vary substantially with individual and contextual factors (Braasch & Bråten, 2017; Bråten, Braasch, & Salmerón, in press; Bråten, Gil, & Strømsø, 2011). The remainder of this chapter is divided into three main sections. In the first, we provide a theoretical background by discussing relevant frameworks for understanding the role of conflict in multiple source use, in particular the Discrepancy-Induced Source Comprehension (D-ISC) model of Braasch et al. (2012) and the Plausibility-Induced Source Focusing assumption of de Pereyra, Britt, Braasch, and Rouet (2014a). In the second, we review important empirical work framed by these theoretical assumptions, as well as empirical work addressing the role of individual and contextual factors when dealing with multiple conflicting sources. In a final section, we draw conclusions based on this body of theoretical and empirical work, discuss implications for educational research and practice, and suggest some directions for future research in this area.

THEORETICAL BACKGROUND The D-ISC model of Braasch et  al. (2012) specifies one particular aspect of the Documents Model (DM) framework, which was originally developed by Perfetti and

186  •  Bråten and Braasch

colleagues (Britt, Perfetti, Sandak, & Rouet, 1999; Perfetti, Rouet, & Britt, 1999). The DM has been updated in more recent work (Britt & Rouet, 2012; Britt et  al., 2013; Rouet, 2006). In brief, the DM explains that multiple-text comprehension involves constructing a coherent mental representation that integrates contents across texts dealing with the same situation or issue, while also taking note of the sources of the different pieces of content information and understanding the relationships among those sources. In this way, content integration as well as source–content links and source–source links are considered central aspects of multiple-text comprehension (see Britt, Rouet, and Durik, this volume). Of note is the fact that the DM is relevant to situations where information from multiple sources is consistent, componential (i.e., information across sources is part of a larger whole not specified by any single source), or conflicting. In comparison, the D-ISC model can be described as a micro-model stemming from the DM that is particularly concerned with contexts where multiple sources present conflicting information about the same situation or issue (e.g., that sun exposure is harmful vs. that sun exposure is healthy). In such contexts, it may be essential to pay attention to the sources of the different views (i.e., who says what) and relationships between the sources (e.g., that author A contradicts author B) because this can help individuals understand the conflict, reconcile the different perspectives, and build an integrated mental model of the situation or issue despite the existing discrepancy (Britt et  al., 2013; Strømsø, Bråten, & Britt, 2010). Accordingly, the main idea underlying the D-ISC model is that individuals’ attention to sources, in particular to “who says what” (i.e., source–content links), will increase when different sources provide discrepant accounts of a situation or issue (Braasch et al., 2012). Specifically, Braasch et al. (2012) proposed that when different sources make conflicting claims about a controversial situation or issue, one mechanism for resolving the resulting break in situational coherence (Graesser, Singer, & Trabasso, 1994) and constructing an integrated mental representation may be to link discrepant content information to the respective sources. Models of text comprehension are quite consistent in highlighting the need to construct a coherent mental representation during reading (for a review, see McNamara & Magliano, 2009). As explained by van den Broek and colleagues (van den Broek, Bohn-Gettler, Kendeou, Carlson, & White, 2011; van den Broek, Risden, & Husebye-Hartmann, 1995), an important component of this construction process concerns readers’ standards of coherence, which refer to the criteria or benchmarks for coherent understanding that readers adopt, and against which they assess the coherence constructed during reading. To the extent that readers experience that their current standards of coherence are not being met during reading (i.e., detect a break in situational coherence; Graesser et al., 1994), which may be particularly relevant to contexts where multiple sources provide conflicting views on the same situation or issue (van den Broek & Kendeou, 2015), they may select and engage effortful strategic processes in order to achieve a satisfactory level of coherence in their mental representations (van den Broek et  al., 2011; van den Broek & Kendeou, 2015). According to the D-ISC model, readers in such instances are likely to invest strategic efforts in constructing mental representations that also include information about the sources of the conflicting views as organizational elements (Braasch & Bråten, 2017; Braasch et al., 2012).

The Role of Conflict  •  187

Recently, Saux et al. (2017) suggested that conflicting views presented by different sources may not only prompt readers to include source-content links in their mental representations, as emphasized within the original D-ISC model (Braasch et  al., 2012), but also strengthen associations between the sources providing the conflicting claims (i.e., source–source links). That is, individuals may also try to resolve or regulate perceived discrepancies to meet their standards of coherence by establishing links between sources (i.e., who agrees/disagrees with whom), which may potentially occur during reading (i.e., online) or after reading (i.e., offline) through further elaboration triggered or required by particular tasks (Saux et al., 2017). Of note is the fact that this attempt to extend the scope of the D-ISC model is consistent with the broader theoretical proposal represented by the DM (e.g., Rouet, 2006). A further extension of the D-ISC model is represented by the Plausibility-Induced Source Focusing assumption, which was recently proposed by de Pereyra, Britt, Braasch, and Rouet (2014a). These authors assumed that when there are discrepancies between individuals’ prior knowledge and textual information, textual information will be considered less plausible. Moreover, when individuals try to make sense of information that they deem less plausible, they may seek support from information about the sources, which results in greater attention to and memory for source feature information (de Pereyra et al., 2014a). According to Bråten, Salmerón, and Strømsø (2016), the Plausibility-Induced Source Focusing assumption is also relevant to situations where individuals consider text information less plausible because it is discrepant with their prior beliefs or attitudes about the issue in question (rather than their prior knowledge). Compared to knowledge, beliefs seem to be more experiential (i.e., rooted in episodic memory), to have stronger affective and evaluative components, and to be more resistant to change (Andiliou, Ramsay, Murphy, & Fast, 2012; Eichenbaum & Bodkin, 2000; Kane, Sandretto, & Heath, 2002; Nespor, 1987; Pajares, 1992). Therefore, it seems less likely that individuals simply revise their beliefs when encountering discrepant textual information (as may occur with prior world knowledge) and hence perceive the information as plausible (Bråten, Salmerón, & Strømsø, 2016). A notable difference between the D-ISC model and the Plausibility-Induced Source Focusing assumption is that the former concerns discrepancies between textual claims coming from different external sources whereas the latter concerns discrepancies between textual claims coming from one or more external sources and individuals’ prior mental representations. This means that the sources of the conflict are external to the individual in the former case whereas one of the sources of the conflict is internal to the individual in the latter (i.e., self as a source). However, in both cases, the discrepancies may make it difficult to construct a coherent mental representation of the issue, with an increased attention to source information being one potential mechanism for restoring coherence. In both cases, readers also may try to restore coherence by integrating or reconciling discrepant views or by choosing one particular view. That is, in some instances, source information may help individuals understand the conflict and reconcile the different views, while in others, source information may help them take (or retain) a particular stance on the issue (Bråten, Salmerón, & Strømsø, 2016). Given that beliefs can be conceived of as more difficult to alter than knowledge (see above), it can also be assumed that

188  •  Bråten and Braasch

individuals are more likely to attend to sources that present claims in conflict with their prior beliefs in order to reject or counterargue them than in order to use them in the service of integrating or reconciling prior beliefs and discrepant textual claims (Bohn-Gettler & McCrudden, in press; Bråten, Salmerón, & Strømsø, 2016). Of note is the fact that the Plausibility-Induced Source Focusing assumption of de Pereyra et  al. (2014a) also may be linked to the broader Plausibility Judgments in Conceptual Change model of Lombardi, Nussbaum, and Sinatra (2016). In this model, plausibility is defined as a “judgment of potential truthfulness when evaluating explanations” (Lombardi et al., 2016, p. 35). Presumably, when dealing with controversial issues, individuals will use their prior mental representations in validating the potential truthfulness of explanations they encounter in texts, with explanations tending to be perceived as potentially less truthful when opposing individuals’ prior mental representations (Maier & Richter, 2014). While plausibility judgments may be automatically made during reading (Isberner & Richter, 2013; Richter, Schroeder, & Wöhrmann, 2009), individuals may also strategically control their plausibility judgments (Lombardi et  al., 2016). That is, when automatic processing cannot satisfy individuals’ situated standards of coherence (van den Broek et  al., 2011), they may strategically (re)consider textual information that is in conflict with their prior knowledge or beliefs. Moreover, when they find it difficult to (re)consider plausibility on the basis of the content alone, individuals may strategically seek support from source information in weighing the merits of textual claims against their prior mental representations. Efforts to resolve breaks in coherence may therefore result in increased attention to source information during reading and better source memory after reading (Bråten, Salmerón, & Strømsø, 2016).

EMPIRICAL WORK Since Braasch et  al. (2012) introduced the D-ISC model, there has been a growing interest among literacy researchers to study situations where individuals have to deal with multiple sources that present conflicting views on a situation or issue. In this section, we review empirical evidence for the D-ISC model and its extension, the Plausibility-Induced Source Focusing assumption (de Pereyra et al., 2014a). Some of this evidence comes from studies where individuals read conflicting information from multiple sources in one and the same text (i.e., single-text studies); other evidence comes from studies where conflicting information from multiple sources are distributed across different texts (i.e., multiple-text studies). Research Framed by the D-ISC In their initial study, Braasch et al. (2012) had French undergraduates read brief news reports in which two sources presented claims about the same situation that were either conflicting or consistent. For example, in the conflicting version of one report, an art critic claimed that the public cheered a new show at the Paris opera, whereas a lighting technician claimed that half the public went back home before the intermission. In the consistent version of this report, the claims of the art critic and the lighting technician were consistent (i.e., the art critic and the lighting technician claimed

The Role of Conflict  •  189

that the public booed the show). In two experiments, Braasch et al. (2012) collected online (eye movement) and offline (source memory) data indicating that reading conflicting claims from different sources promoted deeper processing of and better memory for the sources of the claims (i.e., who said what), as compared to the reading of consistent claims. The claims and their sources were embedded in very brief (i.e., two-sentence) single texts in the Braasch et al. (2012) study. Similar materials and procedures have been used in follow-up work (de Pereyra, Belkadi, Marbach, & Rouet, 2014b; Rouet, Le Bigot, de Pereyra, & Britt, 2016; Saux et  al., 2017). De Pereyra et  al. (2014b), who compared the performance of French seventh- and ninth-graders with that of undergraduates, found that even seventh- and ninth-graders improved their recall of who said what when reading conflicting compared to consistent information. The effect of conflict on sourcing increased with age and educational level, however. Recently, Rouet et al. (2016), in a study with American and French undergraduates, provided additional support for the D-ISC model. In that study, those who read conflicting news reports were more likely to refer to sources when summarizing the reports (Experiments 1 and 2), as well as to recall the sources of the claims (Experiment 3). Rouet et al. (2016; Experiment 3) also showed that the D-ISC model was supported when the conflicting claims and the two embedded sources who provided them were included in somewhat longer (i.e., four-sentence) texts. Finally, in two experiments with Argentinian undergraduates, Saux et al. (2017) found that those who read conflicting claims by two different embedded sources in 6–7-sentence news reports were also more likely to construct source–source links than those who read consistent claims. Comparing participants’ performance on an online source recognition task and an offline source recall task, these authors suggested that such source–source links were constructed after rather than during reading. The D-ISC model also has received empirical support in reading contexts where conflicting claims about the same issue are presented by different sources distributed across multiple texts (Barzilai & Eshet-Alkalai, 2015; Ferguson et al., 2013; Kammerer & Gerjets, 2014; Kammerer, Kalbfell, & Gerjets, 2016; Salmerón, Macedo-Rouet, & Rouet, 2016; Stang Lund, Bråten, Brante, & Strømsø, 2017; Stadtler, Scharrer, Skodzik, & Bromme, 2014; Strømsø & Bråten, 2014; Strømsø, Bråten, Britt, & Ferguson, 2013). Typically, these multiple-text studies have also used longer, more authentic texts than the single-text studies referred to above. In two exploratory studies where Norwegian undergraduates thought aloud while reading multiple texts on the controversial socio-scientific issue of whether cell phone radiation poses any health risk, Strømsø and colleagues (Strømsø & Bråten, 2014; Strømsø et  al., 2013) observed that participants increased their attention to sources embedded within the texts when those texts presented conflicting views on the issue. Building on this exploratory work, Kammerer and Gerjets (2014) conducted an experimental study where German undergraduates read about a controversial health-related issue. Two conditions were designed: one in which there were conflicts between claims presented on an institutional web page and several other, partly commercial, web pages, and one in which consistent claims about the issue were presented across the web pages. Eye movement data showed that in the conflicting claims condition, more participants paid attention to the source of the institutional web

190  •  Bråten and Braasch

page during reading. Further, in this condition, participants included more source citations in their written summaries. In a follow-up study, Kammerer et al. (2016) had German undergraduates read two web pages on the same health-related issue. In that study, eye movement and think-aloud data indicated that participants who read conflicting information on the web pages paid more attention to source information and made more evaluative judgments about the sources than did participants who read consistent information. Moreover, after reading, those who previously encountered conflicting information included more references to sources in recommendations that they generated on the issue and discriminated better between more and less trustworthy sources. Thus far, few studies have investigated the effects of conflicting textual information when K-12 students read more than one text. In one such study, Salmerón et al. (2016; Experiment 2) found that Spanish students at primary (5th–6th grade), secondary (7th–8th grade), and undergraduate levels who read conflicting recommendations from two different sources in a social question-and-answer forum took source information (i.e., expertise) into consideration when selecting which recommendation to follow and explaining their selections regardless of educational level. In contrast, students at all three educational levels tended to ignore source information when different sources did not provide conflicting recommendations (Salmerón et al., 2016; Experiment 1). Also, a recent study of Norwegian secondary school students reading multiple texts containing conflicting claims from different sources found that the better students remembered that the texts contained conflicting claims on the issue, the more likely they were to include source–content links in their mental representations of the texts (Stang Lund et al., 2017). Of note is the fact that these findings with K-12 students have been obtained with very different textual materials: while participants in the Salmerón et al. (2016) study read brief recommendations dealing with daily life topics, participants in the Stang Lund et al. (2017) study read longer expository texts on the issue of sun exposure and health. Finally, Ferguson et al. (2013) examined the role of conflict in multiple source use among Norwegian 10th-graders reading about a controversial socio-scientific issue in five different texts. Students who read conflicting information about the issue reported on a questionnaire that they increased their reliance on multiple sources to corroborate knowledge claims, as compared to controls who read consistent information about the same issue. Importantly, those who read conflicting information about the issue in multiple texts also outperformed controls with respect to deep intertextual (i.e., integrated) understanding of the issue, as shown in their responses to short-essay questions. Research Addressing the Plausibility-Induced Source Focusing Assumption In the first test of the Plausibility-Induced Source Focusing assumption, de Pereyra et  al. (2014a) conducted two experiments. French undergraduate and graduate students read three-sentence news reports that contained implausible or plausible information in light of their prior world knowledge. This information was presented by sources embedded within the news reports. For example, in one report, astronauts presented the implausible information that a space station was equipped

The Role of Conflict  •  191

with a bowling alley and a Jacuzzi. In the plausible version of this report, astronauts presented the information that a space station was equipped with treadmills and a home gym. Contrary to their expectations, de Pereyra et al. (2014a) did not find any effect of the plausibility manipulation on participants’ memory for sources in these experiments. One possible explanation for this lack of effect is that participants did not need any support from source information to make sense of the implausible claims. That is, the conflict between these claims and common world knowledge likely was so simple and obvious that participants could reject them right away based on the content information alone. In a follow-up study, Bråten, Salmerón, and Strømsø (2016) presented Norwegian undergraduates with more complex text materials, assuming that such materials would make it harder to evaluate a claim and resolve the conflict based on the sole content of the text (i.e., without taking information about the source into consideration). Moreover, this follow-up study presented readers with textual information that was in conflict with their prior beliefs about the targeted issue, rather than with their prior world knowledge. In brief, when participants read a single text discussing cell phone use and potential health risks, memory for the source increased when the text’s main conclusion contradicted the belief that cell phone use involves serious health risks (i.e., the text concluded that cell phone use does not have any health risks). However, when the text’s main conclusion contradicted the belief that cell phone use does not have any health risks (i.e., the text concluded that cell phone use involves serious health risks), no such increase in memory for source feature information was observed. Thus, while this study partly confirmed the Plausibility-Induced Source Focusing assumption, it also suggests that conflicts between textual claims and prior topic beliefs may have different functional value in terms of promoting attention to source feature information depending on the exact nature of the conflict and how it is perceived by readers (Bråten, Salmerón, & Strømsø, 2016). Related evidence for the Plausibility-Induced Source Focusing assumption can also be found in a multiple-text study by Maier and Richter (2013). In that study, German undergraduates read two texts that presented information in conflict with their prior beliefs about the topic of global warming or the topic of vaccination, as well as two texts that presented information consistent with their prior beliefs about either of these topics. Each participant thus read four texts on either global warming or vaccination (i.e., topic was a between-subjects factor), two conflicting and two consistent with their prior beliefs about the assigned topic. As a measure of source memory, Maier and Richter (2013) assessed readers’ ability to assign paraphrased sentences back to their respective information sources, specifically their ability to match each sentence with the corresponding text title. The findings showed that across topics, readers displayed better source memory for texts presenting information in conflict with their prior beliefs than for texts presenting information consistent with their prior beliefs. In addition, comprehension at the level of the propositional text base (Kintsch, 1988) was found to be better for the former than for the latter type of texts. Finally, an intriguing think-aloud study by Gottlieb and Wineburg (2012) presented findings relevant to the Plausibility-Induced Source Focusing assumption. In that study, religious (Jewish and Christian) and non-religious (atheist and agnostic) individuals read multiple texts about the Biblical Exodus. When religious individuals

192  •  Bråten and Braasch

read texts that cast doubt on this event, which is at the heart of Judeo-Christian belief, and when non-religious individuals read texts emphasizing the role of the Divine in human history, they evaluated knowledge claims and their sources differently than when they read texts that did not challenge their prior beliefs. That is, for these individuals, conflicts between textual claims and prior beliefs seemed to provide a critical lens employed during meaning-making, with consequences for how content as well as sources were attended to and evaluated. Individual and Contextual Factors A range of individual factors may influence how learners deal with conflict in multiple source use. Although a comprehensive review of such factors is beyond the scope of this chapter, we will briefly highlight the potential roles that epistemic beliefs, knowledge, metacognition, and self-efficacy may play in a multiple-text context. There is a growing body of research indicating that individuals’ epistemic beliefs— that is, their beliefs about knowledge and knowing (Hofer & Bendixen, 2012)—may influence their sourcing as well as their comprehension performance when encountering conflicts in multiple texts (for reviews, see Bråten, Britt, Strømsø, & Rouet, 2011; Bråten, Strømsø, & Ferguson, 2016). This is well illustrated by Barzilai and EshetAlkalai (2015), who found that presenting conflicting information across multiple texts only promoted sourcing among individuals believing in uncertain knowledge and the need to justify knowledge claims through critical thinking and evidence. In turn, sourcing activities predicted the integration of information from multiple texts in written arguments. Moreover, as discussed by Braasch and Bråten (2017), prior domain knowledge seems important for noticing conflicts between sources because it facilitates co-activation of conflicting information in working memory during reading. In support of this view, Trevors, Feyzi-Behnagh, Azevedo, and Bouchet (2016; Study 2), for example, presented think-aloud and retrospective interview data from American undergraduates indicating that prior knowledge played an important role in detecting conflicts and resolving them. Thus, participants that did not notice conflicts in the textual materials typically referred to their limited prior knowledge as a reason for not being able to do so. However, epistemic beliefs and prior domain knowledge may not only independently, but also interactively, influence how individuals deal with conflicting information in multiple texts. Accordingly, in a cluster-analytic study with Norwegian 10th-graders, Ferguson and Bråten (2013) showed that students who had weak beliefs in personal justification of knowledge claims and strong beliefs in justification by multiple sources, along with high prior knowledge, were particularly successful when trying to construct deep, integrated understanding from multiple conflicting texts on the issue of sun exposure and health. Individuals’ metacognition, in particular their comprehension monitoring in order to evaluate emerging mental representations for coherence during reading, is also likely to influence the extent to which they notice existing conflicts between multiple sources (cf., Baker & Zimlin, 1989; Braasch & Bråten, 2017). Presumably, such monitoring will vary as a function of the standard of coherence applied by a particular

The Role of Conflict  •  193

reader, with some readers typically setting high standards of coherence, others typically low (van den Broek & Kendeou, 2015). Moreover, as shown by Trevors et al. (2016; Study 1), readers’ epistemic beliefs may relate to their metacognitive processing when encountering conflicting information. Thus, these authors found that students believing science knowledge to be complex and tentative were also more likely to display adaptive metacognitive judgments and self-regulatory processes to cope with conflicting information encountered in science texts. Finally, Trevors et al. (2016; Study 2) found that students’ perceived self-efficacy for evaluating science knowledge claims was related to their ability to detect conflicts in science texts. In the same vein, Andreassen and Bråten (2013) reported that individuals’ perceived source evaluation self-efficacy was related to the likelihood that they attended to and used source information when reading about controversial issues. In brief, then, lack of confidence in one’s ability to critically evaluate claims and sources also makes it less likely that individuals notice conflicts between multiple sources during reading. A range of contextual factors also may influence how individuals cope with conflicts among multiple sources. Without providing an exhaustive review of research on such factors in this chapter, we will briefly highlight the potential impact of reading materials and reading tasks in the following, as well as some potential interaction effects involving contextual factors. Although expected effects on sourcing have been observed when conflicts between sources exist within single texts, of note is the fact that this research has used very brief single texts where such conflicts have been made very salient (e.g., Braasch et al., 2012). Other research using longer texts indicates that individuals are less likely to notice conflicts within single texts than between multiple texts (Stadtler, Scharrer, Brummernhenrich, & Bromme, 2013). Thus, Stadtler and colleagues (2013) found that memory for conflicts was poorer when German undergraduates read conflicting information on a controversial medical issue in a single text than when they read it in four different texts. In a follow-up study, Stadtler et al. (2014) had German undergraduates read nine texts from different sources on a controversial medical issue. Students read the texts in conditions that either did or did not signal the existence of conflicting arguments across texts through rhetorical means. For example, such signaling could involve starting a text with the following phrase: “Contrary to what some health professionals argue . . .”. Participants who read texts where conflicts were explicitly signaled had better memory for conflicts after reading and also wrote essays on the issue that reported more conflicts in a balanced way and included more references to sources. Further research on the role of reading materials has noted that when individuals read conflicting texts on controversial issues, attention to, representation, evaluation, and use of source information may vary with text type (Bråten, Braasch, Strømsø, & Ferguson, 2015; List, Alexander, & Stephens, 2017; Rouet, Britt, Mason, & Perfetti, 1996; Wineburg, 1991). Interestingly, the type of source also may influence the detection of conflicting information (Keck, Kammerer, & Starauschek, 2015). Thus, Keck et al. (2015) found that German secondary school students had better memory for textual conflicts after reading texts from unambiguously authoritative or non-authoritative sources than after reading texts from sources whose authoritativeness was ambiguous. How individuals deal with conflicting information from multiple sources also may be influenced by the reading task (Bråten et  al., 2011). In particular, the effects of

194  •  Bråten and Braasch

“general purpose instructions” (McCrudden & Schraw, 2007) to construct arguments have been compared with the effects of other general purpose instructions, especially with instructions to summarize information. In general, this research suggests that argument tasks can lead to more elaborative processing and deeper understanding than summary tasks when individuals work with multiple conflicting texts (e.g., Le Bigot & Rouet, 2007; Naumann, Wechsung, & Krems, 2009: Stadtler et al., 2014; Wiley et  al., 2009; Wiley & Voss, 1999). However, there is also evidence that the reading task may interact with the reading materials, as well as with several individual factors. As an example of the former, Stadtler et al. (2014) found that participants given an argument task included more source references in their post-reading essays than did participants given other task instructions, but this effect depended on the explicit signaling of conflicts between the texts that they read (see above). As examples of the latter, Bråten and colleagues (Bråten et  al., 2011; Bråten & Strømsø, 2010; Gil, Bråten, Vidal-Abarca, & Strømsø, 2010b) reported on three experiments showing that readers’ ability to construct integrated understanding from multiple conflicting texts in argument task conditions depended on their epistemic beliefs, specifically on the belief that knowledge about the issue was tentative and evolving rather than absolute and unchanging. Further, Gil, Bråten, Vidal-Abarca, and Strømsø (2010a) demonstrated that facilitative effects of argument tasks on the understanding of conflicting information presented by multiple sources may depend on readers’ prior knowledge about the issue, with readers low in prior knowledge seemingly more hindered than helped by argument task instructions. The take-home message of this section is that a range of individual factors, cognitive as well as motivational, is likely to influence learners’ conflict detection, sourcing, and comprehension when encountering conflicting information in multiple texts. Likewise, contextual factors, such as the reading materials and the reading tasks, have been found to influence how learners process and comprehend multiple conflicting texts (i.e., conflict detection, sourcing, and comprehension performance). Finally, there is some evidence to suggest that individual and contextual factors may interact to influence how learners deal with conflicts across multiple texts.

CONCLUSIONS, IMPLICATIONS, AND FUTURE DIRECTIONS The theoretical and empirical work discussed in this chapter highlights that as challenging as conflict in multiple source use may be, it may also have adaptive consequences in terms of individuals’ processing and comprehension performance. In particular, consistent with the D-ISC model (Braasch et al., 2012), when individuals encounter conflicting textual claims from different external sources, they may become more vigilant regarding those sources and also take them more into consideration when constructing and communicating their understanding of controversial situations and issues. This is an important insight because a growing body of empirical work, correlational (Anmarkrud, Bråten, & Strømsø, 2014; Barzilai & Eshet-Alkalai, 2015; Barzilai, Tzadok, & Eshet-Alkalai, 2015; Bråten, Strømsø, & Britt, 2009; Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012; List et  al., 2017; Strømsø et  al., 2010; Wiley et  al., 2009) as well as experimental (Barzilai & Ka’adan, 2017; Braasch, Bråten, Strømsø, Anmarkrud, & Ferguson, 2013; Mason,

The Role of Conflict  •  195

Junyent, & Tornatora, 2014; Wiley et al., 2009), also indicates that increased sourcing may promote comprehension when individuals work with multiple conflicting texts. Albeit preliminary, there is also some evidence suggesting that conflicts between external sources and self as a source may impact on readers’ sourcing and meaningmaking, as consistent with the Plausibility-Induced Source Focusing assumption (de Pereyra et al., 2014a). Much more empirical work is needed to substantiate this latter assumption, however. As indicated by the preceding review, the effects of conflict in multiple source use on individuals’ processing and comprehension may be moderated by individual and contextual factors, as well as their interaction. Further research on such factors is needed to construct a refined, empirically based D-ISC model that allows for more precise predictions regarding which individuals in which contexts may be best equipped to meet, and even profit from, the challenge represented by multiple conflicting sources. For example, little is still known about how the type and strength of individuals’ standards of coherence influence how they cope with conflicts in multiple source use, how they are able to maintain their standards of coherence when working with multiple conflicting sources, and which are the optimal types and levels of standards of coherence given particular reading materials, reading tasks, and contexts. Of note is also the fact that other types of conflicts than those hitherto studied are likely to impact on individuals’ meaning-making during multiple-text reading. For example, while researchers have primarily focused their attention on conflicts between sources embedded within the texts (i.e., cited sources; e.g., Braasch et al., 2012; Strømsø et al., 2013) and between sources of the texts themselves (i.e., main sources; Kammerer et al., 2016; Salmerón et al., 2016), conflicts between different layers of sources (i.e., between embedded and main sources) have received little attention (see, however, Foy, LoCasto, Briner, & Dyar, 2017; Experiment 3). Moreover, it is still an open question whether conflicts between embedded sources are more difficult to detect than conflicts between main sources, with one possibility being that embedded sources are generally less salient to readers than main sources (Bråten, Strømsø, & Andreassen, 2016). Of course, this may also vary according to whether the embedded sources are cited within one and the same text or whether they are cited in different texts. However, there are also other conflicts that may be relevant to how individuals deal with textual materials on controversial issues. One such conflict concerns the distinct dimensions of content relevance and source credibility, with individuals often encountering texts where content relevance is high but source credibility low and vice versa (Bråten, McCrudden, Stang Lund, Brante, & Strømsø, in press; McCrudden, Stenseth, Bråten, & Strømsø, 2016). Research on how such conflicts may impact individuals’ selection, processing, and use of information from multiple sources has just begun, however (see McCrudden, this volume). Moreover, as suggested by Salmerón, Berry, Rouet, and Macedo-Rouet (2016), the violation of readers’ expectations based on the source’s competence may induce sourcing, as when a highly competent author provides a useless recommendation in a social question-and-answer forum. Finally, as suggested in recent work by Muis and colleagues (Muis et al., 2015; Trevors, Muis, Pekrun, Sinatra, & Muijselaar, 2017), individuals may experience conflicts between

196  •  Bråten and Braasch

their epistemic beliefs and the epistemic nature of tasks and contents during multiple source use. Such experiences may occur, for example, when individuals who believe in certain and simple knowledge are tasked to learn from diverse texts where multiple sources present conflicting views on the same issue. In turn, such experiences may give rise to less adaptive emotions (e.g., confusion, frustration, or boredom) that mediate how individuals strategically learn from multiple conflicting texts. Thus, despite the remarkable progress made in this area of research during the last decade, much remains to be known. This concerns not only individual and contextual factors, but also how different types of conflicts may come into play and have an impact when individuals deal with multiple conflicting sources. Importantly, further research on these issues may have practical as well as theoretical implications. Of course, presenting students with multiple conflicting sources will not automatically turn them into competent sourcers and comprehenders. If that were the case, educational practitioners and policy-makers around the world would, indeed, have reason to celebrate, as just accessing the Internet to learn about almost any topic would give students the kind of practice required to develop such critical literacy skills. For most students, scaffolded practice is highly needed to deal effectively and efficiently with such demanding literacy contexts, however. Accordingly, a number of promising interventions to improve students’ sourcing and integration when dealing with multiple conflicting documents have been launched in recent years (for reviews, see Brante & Strømsø, in press; Bråten & Braasch, 2017; Bråten, Stadtler, & Salmerón, 2018). For example, Braasch et al. (2013), building on a contrasting-cases approach, had secondary school students compare and contrast the ways two hypothetical students dealt with multiple conflicting sources, decide which were the most appropriate strategies, and explain why. The results indicated that the contrasting-cases intervention promoted students’ source evaluation skills. More recently, Barzilai and Ka’adan (2017) provided secondary school students with strategic scaffolds in the form of organizers supporting their sourcing and integration performance when working with multiple conflicting sources, with this intervention improving students’ ability to integrate information across such sources. Both the Braasch et al. (2013) and the Barzilai and Ka’adan (2017) studies included dyadic and whole-class discussions as part of the interventions. Although such studies have given important insights into how students’ sourcing and integrated understanding may be improved through scaffolded practice with multiple conflicting sources, they may not have paid sufficient attention to the fact that dealing with multiple conflicting sources requires a variety of personality, cognitive, and motivational resources that are unevenly distributed among students (Bråten, Anmarkrud, Brandmo, & Strømsø, 2014; Barzilai & Strømsø, this volume). This suggests that further efforts should be made to ensure that not only the more resourceful students are able to fully benefit from the scaffolding provided, consistent with the Matthew Effect or “richget-richer” phenomenon described by Stanovich (1986). For example, some students may need to work with only one textbook-like source for a longer period of time to build a sufficient knowledge base on an issue before gradually moving to multiple conflicting sources, while others may need extended scaffolded practice with multiple conflicting sources, also supported by more competent peers through collaborative projects. In any case, much further research is needed on how instruction to cope with multiple conflicting sources can be adapted to different individuals, including struggling readers

The Role of Conflict  •  197

(Anmarkrud, Brante, & Andresen, this volume). Finally, various types of conflicts in multiple source use may have to be targeted through instruction, and the extent to which any effects obtained in schooled contexts may (or may not) transfer to students’ use of multiple conflicting sources out of school needs to be studied.

REFERENCES Andiliou, A., Ramsay, C.M., Murphy, P.K., & Fast, J. (2012). Weighing opposing positions: Examining the effects of intratextual persuasive messages on students’ knowledge and beliefs. Contemporary Educational Psychology, 37, 113–127. Andreassen, R., & Bråten, I. (2013). Teachers’ source evaluation self-efficacy predicts their use of relevant source features when evaluating the trustworthiness of web sources on special education. British Journal of Educational Technology, 44, 821–836. Anmarkrud, Ø., Bråten, I., & Strømsø, H.I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. Baker, L., & Zimlin, L. (1989). Instructional effects on children’s use of two levels of standards for evaluating their comprehension. Journal of Educational Psychology, 81, 340–346. Barzilai, S., & Eshet-Alkalai, Y. (2015). The role of epistemic perspectives in comprehension of multiple author viewpoints. Learning and Instruction, 36, 86–103. Barzilai, S., & Ka’adan, I. (2017). Learning to integrate divergent information sources: The interplay of epistemic cognition and epistemic metacognition. Metacognition and Learning, 12, 193–232. Barzilai, S., Tzadok, E., & Eshet-Alkalai, Y. (2015). Sourcing while reading divergent expert accounts: Pathways from views of knowing to written argumentation. Instructional Science, 43, 737–766. Bohn-Gettler, C.M., & McCrudden, M.T. (in press). Effects of text relevance instructions and topic beliefs on reading processes and memory. Discourse Processes. Braasch, J.L.G., & Bråten, I. (2017). The Discrepancy-Induced Source Comprehension (D-ISC)model: Basic assumptions and preliminary evidence. Educational Psychologist, 52, 167–181. Braasch, J.L.G., Bråten, I., Strømsø, H.I., Anmarkrud, Ø., & Ferguson, L.E. (2013). Promoting secondary school students’ evaluation of source features of multiple documents. Contemporary Educational Psychology, 38, 180–195. Braasch, J.L.G., Rouet, J.F., Vibert, N., & Britt, M.A. (2012). Readers’ use of source information in text comprehension. Memory & Cognition, 40, 450–465. Brante, E.W., & Strømsø, H.I. (in press). Sourcing in text comprehension: A review of interventions targeting sourcing skills. Educational Psychology Review. Bråten, I., Anmarkrud, Ø., Brandmo, C., & Strømsø, H.I. (2014). Developing and testing a model of direct and indirect relationships between individual differences, processing, and multiple-text comprehension. Learning and Instruction, 30, 9–24. Bråten, I., & Braasch, J.L.G. (2017). Key issues in research on students’ critical reading and learning in the 21st century information society. In C. Ng & B. Bartlett (Eds.), Improving reading and reading engagement in the 21st century: International research and innovations (pp. 77–98). Singapore: Springer. Bråten, I., Braasch, J.L.G., & Salmerón, L. (in press). Reading multiple and non-traditional texts: New oppor­ tunities and new challenges. In E.B. Moje, P. Afflerbach, P. Enciso, & N.K. Lesaux (Eds.), Handbook of reading research (Vol. V). New York: Routledge. Bråten, I., Braasch, J.L.G., Strømsø, H.I., & Ferguson, L.E. (2015). Establishing trustworthiness when students read multiple documents containing conflicting scientific evidence. Reading Psychology, 36, 315–349. Bråten, I., Britt, M.A., Strømsø, H.I., & Rouet, J.F. (2011). The role of epistemic beliefs in the comprehension of multiple expository texts: Toward an integrated model. Educational Psychologist, 46, 48–70. Bråten, I., Gil, L., & Strømsø, H.I. (2011). The role of different task instructions and reader characteristics when learning from multiple expository texts. In M.T. McCrudden, J.P. Magliano, & G. Schraw (Eds.), Text rel­ evance and learning from text (pp. 95–122). Greenwich, CT: Information Age. Bråten, I., McCrudden, M.T., Stang Lund, E., Brante, E.W., & Strømsø, H.I. (in press). Task-oriented learning with multiple documents: Effects of topic familiarity, author expertise, and content relevance on document selection, processing, and use. Reading Research Quarterly.

198  •  Bråten and Braasch Bråten, I., Salmerón, L., & Strømsø, H.I. (2016). Who said that? Investigating the Plausibility-Induced Source Focusing Assumption with Norwegian undergraduates. Contemporary Educational Psychology, 46, 253–262. Bråten, I., Stadtler, M., & Salmerón, L. (2018). The role of sourcing in discourse comprehension. In M.F. Schober, D.N. Rapp, M.A. Britt (Eds.), Handbook of discourse processes (2nd ed., pp.141–166). New York: Routledge. Bråten, I., & Strømsø, H.I. (2010). Effects of task instruction and personal epistemology on the understanding of multiple texts about climate change. Discourse Processes, 47, 1–31. Bråten, I., Strømsø, H.I., & Andreassen, R. (2016). Sourcing in professional education: Do text factors make any difference? Reading and Writing, 29, 1599–1628. Bråten, I., Strømsø, H.I., & Britt, M.A. (2009). Trust matters: Examining the role of source evaluation in stu­ dents’ construction of meaning within and across multiple texts. Reading Research Quarterly, 44, 6–28. Bråten, I., Strømsø, H.I., & Ferguson, L.E. (2016). The role of epistemic beliefs in the comprehension of single and multiple texts. In P. Afflerbach (Ed.), Handbook of individual differences in reading: Text and context(pp. 67–79). New York: Routledge. Britt, M.A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20, 485–522. Britt, M.A., Perfetti, C.A., Sandak, R., & Rouet, J.F. (1999). Content integration and source separation in learning from multiple texts. In S.R. Goldman, A.C. Graesser, & P. van den Broek (Eds.), Narrative, comprehension, causality, and coherence: Essays in honor of Tom Trabasso (pp. 209–233). Mahwah, NJ: Erlbaum. Britt, M.A., & Rouet, J.F. (2012). Learning with multiple documents: Component skills and their acquisition. In J.R. Kirby & M.J. Lawson (Eds.), Enhancing the quality of learning: Dispositions, instruction, and learning processes (pp. 276–314). New York: Cambridge University Press. Britt, M.A., Rouet, J.F., & Braasch, J.L.G. (2013). Documents as entities: Extending the situation model theory of comprehension. In M.A. Britt, S.R. Goldman, & J.F. Rouet (Eds.), Reading: From words to multiple texts (pp. 160–179). New York: Routledge. Bromme, R., & Goldman, S.R. (2014). The public’s bounded understanding of science. Educational Psychologist, 49, 59–69. de Pereyra, G., Belkadi, S., Marbach, L., & Rouet, J.F. (2014b, August). Do teenage readers’ use source information when faced with discrepant information? Paper presented at the annual meeting of the Society for Text and Discourse, Chicago, USA. de Pereyra, G., Britt, M.A., Braasch, J.L.G., & Rouet, J.F. (2014a). Readers’ memory for information sources in simple news stories: Effects of text and task features. Journal of Cognitive Psychology, 26, 187–204. Eichenbaum, H., & Bodkin, J.A. (2000). Belief and knowledge as distinct forms of memory. In D.L. Schachter & E. Scarry (Eds.), Memory, brain, and belief (pp. 176–207). Cambridge, MA: Harvard University Press. Ferguson, L.E., & Bråten, I. (2013). Student profiles of knowledge and epistemic beliefs: Changes and relations to multiple-text comprehension. Learning and Instruction, 25, 49–61. Ferguson, L.E., Bråten, I., Strømsø, H.I., & Anmarkrud, Ø. (2013). Epistemic beliefs and comprehension in the context of reading multiple documents: Examining the role of conflict. International Journal of Educational Research, 62, 100–114. Foy, J.E., LoCasto, P.C., Briner, S.W., & Dyar, S. (2017). “Would a madman have been so wise as this?” The effects of source credibility and message credibility on validation. Memory & Cognition, 45, 281–295. Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H.I. (2010a). Summary versus argument tasks when working with multiple documents: Which is better for whom? Contemporary Educational Psychology, 35, 157–173. Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H.I. (2010b). Understanding and integrating multiple science texts: Summary tasks are sometimes better than argument tasks. Reading Psychology, 31, 30–68. Goldman, S.R., Braasch, J.L.G., Wiley, J., Graesser, A.C., & Brodowinska. K.M. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381. Goldman, S.R., & Scardamalia, M. (2013). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities. Cognition and Instruction, 31, 255–269. Gottlieb, E., & Wineburg, S. (2012). Between veritas and communitas: Epistemic switching in the reading of academic and sacred history. The Journal of the Learning Sciences, 21, 84–129. Graesser, A.C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101, 371–395.

The Role of Conflict  •  199 Hofer, B.K., & Bendixen, L.D. (2012). Personal epistemology: Theory, research, and future directions. In K.R. Harris, S. Graham, & T. Urdan (Eds.), APA educational psychology handbook: Vol. 1. Theories, constructs, and critical issues (pp. 227–256). Washington, DC: American Psychological Association. Isberner, M.B., & Richter, T. (2013). Can readers ignore implausibility? Evidence for nonstrategic monitoring of event-based plausibility in language comprehension. Acta Psychologica, 142, 15–22. Jacobson, M.J., & Spiro, R.J. (1995). Hypertext learning environments, cognitive flexibility, and the transfer of complex knowledge: An empirical investigation. Journal of Educational Computing Research, 12, 301–333. Kammerer, Y., & Gerjets, P. (2014). Quellenbewertungen und Quellenverweise beim Lesen und Zusammenfassen wissenschaftsbezogener Informationen aus multiplen Webseiten [Source evaluations and source references when reading and summarizing science-related information from multiple web pages]. Unterrichtswissenschaft, 42, 7–23. Kammerer, Y., Kalbfell, E., & Gerjets, P. (2016). Is this information source commercially biased? How contradictions between web pages stimulate the consideration of source information. Discourse Processes, 53, 430–456. Kane, R., Sandretto, S., & Heath, C. (2002). Telling half the story: A critical review of research on the teaching beliefs and practices of university academics. Review of Educational Research, 72, 177–228. Keck, D., Kammerer, Y., & Starauschek, E. (2015). Reading science texts online: Does source information influence the identification of contradictions within texts? Computers & Education, 82, 442–449. Kintsch, W. (1988). The role of knowledge in discourse comprehension: A construction-integration model. Psychological Review, 95, 163–182. Le Bigot, L., & Rouet, J.F. (2007). The impact of presentation format, task assignment, and prior knowledge on students’ comprehension of multiple online documents. Journal of Literacy Research, 39, 445–470. Lewandowsky, S., & Oberauer, K. (2016). Motivated rejection of science. Current Directions in Psychological Science, 25, 217–222. List, A., Alexander, P.A., & Stephens, L.A. (2017). Trust but verify: Examining the association between students’ sourcing behaviors and ratings of text trustworthiness. Discourse Processes, 54, 83–104. Lombardi, D., Nussbaum, E.M., & Sinatra, G.M. (2016). Plausibility judgments in conceptual change and epistemic cognition. Educational Psychologist, 51, 35–56. Magliano, J.P., McCrudden, M.T., Rouet, J.F., & Sabatini, J. (2018). The modern reader: Should changes to how we read affect research and theory? In M. Schober, D. Rapp & M.A. Britt (Eds.), Handbook of discourse processes (2nd ed., pp. 343–361). New York: Routledge. Maier, J., & Richter, T. (2013). Text-belief consistency effects in the comprehension of multiple texts with conflicting information. Cognition and Instruction, 31, 151–175. Maier, J., & Richter, T. (2014). Fostering multiple text comprehension: How metacognitive strategies and motivation moderate the text-belief consistency effect. Metacognition and Learning, 9, 51–74. Mason, L., Junyent, A.A., & Tornatora, M.C. (2014). Epistemic evaluation and comprehension of web-source information on controversial science-related topics: Effects of a short-term instructional intervention. Computers & Education, 76, 143–157. McCrudden, M.T., & Schraw, G. (2007). Relevance and goal-focusing in text processing. Educational Psychology Review, 19, 113–139. McCrudden, M.T., Stenseth, T., Bråten, I., & Strømsø, H.I. (2016). The effects of author expertise and content relevance on document selection: A mixed methods study. Journal of Educational Psychology, 108, 147–162. McNamara, D.S., & Magliano, J. (2009). Toward a comprehensive model of comprehension. Psychology of Learning and Motivation, 51, 297–384. Moan, J., Baturaite, Z., Juzeniene, A., & Porojnicu, A.C. (2012). Vitamin D, sun, sunbeds and health. Public Health Nutrition, 15, 711–715. Muis, K.R., Pekrun, R., Sinatra, G.M., Azevedo, R., Trevors, G., Meier, E., & Heddy, B.C. (2015). The curious case of climate change: Testing a theoretical model of epistemic beliefs, epistemic emotions, and complex learning. Learning and Instruction, 39, 168–183. Naumann, A.B., Wechsung, I., & Krems, J.F. (2009). How to support learning from multiple hypertext sources. Behavior Research Methods, 41, 639–646. Nespor, J. (1987). The role of beliefs in the practice of teaching. Journal of Curriculum Studies, 19, 317–328. Pajares, M.F. (1992). Teachers’ beliefs and educational research: Cleaning up a messy construct. Review of Educational Research, 62, 307–332. Perfetti, C.A., Britt, M.A., & Georgi, M.C. (1995). Text-based learning and reasoning: Studies in history. Hillsdale, NJ: Erlbaum.

200  •  Bråten and Braasch Perfetti, C.A., Rouet, J.F., & Britt, M.A. (1999). Towards a theory of documents representation. In H. van Oostendorp & S.R. Goldman (Eds.), The construction of mental representations during reading (pp. 99–122). Mahwah, NJ: Erlbaum. Petty, R.E., & Briñol, P. (2012). The elaboration likelihood model. In P.A.M. Van Lange, A. Kruglanski, & E.T. Higgins (Eds.), Handbook of theories of social psychology (Vol. 1, pp. 224–245). London: Sage. Richter, T., Schroeder, S., & Wöhrmann, B. (2009). You don’t have to believe everything you read: Background knowledge permits fast and efficient validation of information. Journal of Personality and Social Psychology, 96, 538–558. Rouet, J.F. (2006). The skills of document use. Mahwah, NJ: Erlbaum. Rouet, J.F., Britt, M.A., Mason, R.A., & Perfetti, C.A. (1996). Using multiple sources of evidence to reason about history. Journal of Educational Psychology, 88, 478–493. Rouet, J.F., Le Bigot, L., de Pereyra, G., & Britt, M.A. (2016). Whose story is this? Discrepancy triggers readers’ attention to source information in short narratives. Reading and Writing, 29, 1549–1570. Salmerón, L., Berry, B., Rouet, J.F., & Macedo-Rouet, M. (2016, November). Violations of competency-based expectations increases sourcing in SQA forums. Paper presented at the annual Workshop on Multiple Document Literacy, University of Paris 8, France. Salmerón, L., Macedo-Rouet, M., & Rouet, J.F. (2016). Multiple viewpoints increase students’ attention to source features in social question and answer forum messages. Journal of the Association for Information Science and Technology, 67, 2404–2419. Saux, G., Britt, A., Le Bigot, L., & Vibert, N., Burin, D., & Rouet, J.F. (2017). Conflicting but close: Readers’ integration of information sources as a function of their disagreement. Memory and Cognition, 45, 151–167. Stadtler, M., & Bromme, R. (2014). The content-source integration model: A taxonomic description of how readers comprehend conflicting scientific information. In D.N. Rapp & J.L.G. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 379–402). Cambridge, MA: The MIT Press. Stadtler, M., Scharrer, L., Brummernhenrich, B., & Bromme, R. (2013). Dealing with uncertainty: Readers’ memory for and use of conflicting information from science texts as a function of presentation format and source expertise. Cognition and Instruction, 31, 130–150. Stadtler, M., Scharrer, L., Skodzik, T., & Bromme, R. (2014). Comprehending multiple documents on scientific controversies: Effects of reading goals and signaling rhetorical relationships. Discourse Processes, 51, 93–116. Stang Lund, E., Bråten, I., Brante, E.W., & Strømsø, H.I. (2017). Memory for textual conflicts predicts sourcing when adolescents read multiple expository texts. Reading Psychology, 38, 417–437. Stanovich, K.E. (1986). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 21, 360–407. Strømsø, H.I., & Bråten, I. (2014). Students’ sourcing while reading and writing from multiple web documents. Nordic Journal of Digital Literacy, 9, 92–111. Strømsø, H.I., Bråten, I., & Britt, M.A. (2010). Reading multiple texts about climate change: The relationship between memory for sources and text comprehension. Learning and Instruction, 20, 192–204. Strømsø, H.I., Bråten, I., Britt, M.A., & Ferguson, L.E. (2013). Spontaneous sourcing among students reading multiple documents. Cognition and Instruction, 31, 176–203. Trevors, G., Feyzi-Behnagh, R., Azevedo, R., & Bouchet, F. (2016). Self-regulated learning processes vary as a function of epistemic beliefs and contexts: Mixed method evidence from eye tracking and concurrent and retrospective reports. Learning and Instruction, 42, 31–46. Trevors, G.J., Muis, K.R., Pekrun, R., Sinatra, G.M., & Muijselaar, M.M.L. (2017). Exploring the relations between epistemic beliefs, emotions, and learning from texts. Contemporary Educational Psychology, 48, 116–132. van den Broek, P., Bohn-Gettler, C.M., Kendeou, P., Carlson, S., & White, M.J. (2011). When a reader meets a text: The role of standard of coherence in reading comprehension. In M.T. McCrudden, J.P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 123–139). Charlotte, NC: Information Age. van den Broek, P., & Kendeou, P. (2015). Building coherence in web-based and other non-traditional reading environments: Cognitive opportunities and challenges. In R.J. Spiro, M. DeSchryver, M.S. Hagerman, P.M. Morsink, & P. Thompson (Eds.), Reading at a crossroads? Disjunctures and continuities in current conceptions and practices (pp. 104–114). New York: Routledge.

The Role of Conflict  •  201 van den Broek, P., Risden, K., & Husebye-Hartmann, E. (1995). The role of readers’ standards for coherence in the generation of inference during reading. In R.F. Lorch & E.J. O’Brian (Eds.), Sources of coherence during reading (pp. 353–373). Hillsdale, NJ: Erlbaum. Wiley, J., Goldman, S.R., Graesser, A.C., Sanchez, C.A., Ash, I.K., & Hemmerick, J. (2009). Source evaluation, comprehension, and learning in internet science inquiry tasks. American Educational Research Association Journal, 46, 1060–1106. Wiley, J., & Voss, J.F. (1999). Constructing arguments from multiple sources: Tasks that promote understanding and not just memory for text. Journal of Educational Psychology, 91, 301–311. Wineburg, S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87.

Section III

Multiple Source Use in Specific Content Areas

12

MULTIPLE SOURCE USE IN HISTORY Emily Fox university of maryland, usa

Liliana Maggioni the catholic university of america, usa

Every historian would agree, I think, that history is a kind of research or inquiry. (Collingwood, 1946/2014, p. 9) To ask questions implies the desire to answer them. (National Council for the Social Studies [NCSS], 2013, p. 85)

Historians ask and seek answers to questions about the past. The evidence-based accounts of the past that are the products of historians’ disciplinary work are built upon the use of multiple sources (Seixas, 1999). In these argumentative, interpretive accounts, historians can describe, explain, and pass judgment (explicitly or more implicitly) on actors and their actions (Megill, 2007), as well as acknowledging and responding to the work of prior historians (Monte-Sano, 2016). Historical inquiry, whether conducted by historians or by students in the classroom, requires particularly challenging types of reading and reasoning (e.g., Leinhardt & Young, 1996; Nokes, 2011), interwoven around the disciplinary task of constructing an integrated, substantiated account from such remnants of the past as can be marshaled (e.g., Marrou, 1959/66; Monte-Sano, 2016). In this chapter, we review theory and research on the role of multiple source use in history; on the basis of that review, we further consider what implications can be derived for educational research and practice. Multiple source use in history is taken here to mean the work of finding, selecting, understanding, interpreting, cross-referencing, situating, evaluating, and synthesizing a specific set of historical sources. This work supports the pursuit of a specific historical inquiry aimed at the development of a specific historical account to address a specific 205

206  •  Fox and Maggioni

question about the past. Historical sources are not limited to the typical written documents; rather, they include written, visual, oral, or material remnants of human and natural activities that can be analyzed to provide evidence supporting claims about the past (NCSS, 2013, p. 105). In addition to such primary sources of evidence, historical inquiry also makes use of secondary sources, which are defined as analyses or accounts related to the topic of the historical inquiry, typically based in some way upon primary sources and typically produced after the time period of interest (NCSS, 2013, p. 104). It is important to note that although the term “source” is reserved here for the historical materials being analyzed, source can also be used alternatively to indicate the producer of these materials, such as the author of a document (see, e.g., Wineburg, 1991a, p. 77). In this chapter we follow the suggestion by Goldman and Scardamalia (2013) to use “sourcing” to refer to identification and consideration of information about the maker or origin of an historical source, while “source” refers to the material itself as a potential object of analysis. In the education-related literature, the aspect of multiplicity and the need to apply source-related historical thinking heuristics (e.g., the canonical trio of sourcing, contextualization, and corroboration) are typically introduced in the context of responding to specific classroom or assessment tasks. This type of task is exemplified by the Document-Based Question (DBQ) included in Advanced Placement (AP) history exams taken by American high school students in advanced history courses (Wineburg, 1998). Multiple sources and source-related heuristics also come into view when the goal of fostering student ability to think historically by mimicking expert disciplinary thinking is emphasized. However, there is a stronger discipline-based claim to be made about the unique role of multiple source use in history: the work of historical inquiry—that is, accounting for the past through analysis of its traces—demands the use of multiple sources and entails a particular type of stance toward encountering a source. In other words, it is the very nature of historical knowledge and the characteristic path of knowing which makes such knowledge possible that call for the use of multiple sources. This claim is not epistemologically neutral. Specifically, it rejects a conceptualization of historical knowledge as isomorphic to the past and of historical knowing as appropriation of a set of information that is preserved, ready-made, in appropriate repositories or sources, albeit the “right” ones. At the same time, it also recognizes that historical knowledge rests on the evidence preserved in the archive (or, more generally, on the available remnants of the past), thus refuting the view that the past is what the historian makes it to be. In positive terms, historical knowledge is conceptualized as the result of the enquiring and attentive work of the historian (whether professional or amateur), in which the past (as conveyed by its remnants) and the present (as brought to bear by the historian) are engaged in a dialogical relation (Marrou, 1959/66). The type of question driving historical inquiry and the constructive, interpretive, and argumentative nature of historical accounts are among the most evident contributions of the historian (and thus of the present) to such dialogue. In fact, as Lowenthal (2000) observed, we cannot look at the past but through the benefit of hindsight and with the benefit of our own eyes. At the same time, the sources that are at the historian’s disposal have been contributed by the past. The very nature of the dialogical relation between past and present implies that reliance on a single, definitive source (supposing such a source could be found) cannot possibly suffice for such work. In fact, the historian seeking to investigate

Multiple Source Use in History  •  207

the past is immediately faced with the otherness, distance, and strangeness of the remnants of the past and of the experience of the human beings that these remnants convey (Marrou, 1959/66; Wineburg, 1999). Such difference from the present needs to be acknowledged, or, as Lowenthal (2000) suggestively put it, “historical enlightenment requires being receptive to astonishment”. To be appreciated and tentatively understood, each source needs to be read from within the context in which it was generated, which entails that its reading always implies other sources, since time travel is not an option. The need to construct a context within which historical sources can be interpreted without being distorted by presentist lenses is of paramount importance, and is an essential step in fostering the type of historical empathy necessary for historical understanding. Yet, it further complexifies the use of historical sources, because a single source requires additional support in order to provide this needed context (e.g., Leinhardt & Young, 1996). Furthermore, many of the primary (and presumably all of the secondary) sources used in historical inquiry have human voices, making the dialogical aspect of historical knowledge more than a mere metaphor. As such, sources are therefore essentially imbued with intentionality, bias, and perspectival limitations (e.g., Wineburg, 1991b). Those that do not have voices of their own (such as artifacts and natural materials) cannot be interpreted standing alone (e.g., Anderson, Frappier, Neswald, & Trim, 2013; Ashby, 2011). Once again, reliance on multiple sources becomes necessary to appreciate, understand, and evaluate the contribution of each voice. Having briefly established these epistemic foundations, the first question to address in reviewing the literature on multiple source use in history is what multiple source use in history involves. Our overview spells out more explicitly the role of multiple source use in historical inquiry as based on the disciplinary nature of history and the work of historians, as well as acknowledging the generic cognitive demands of multiple source use per se. This establishment of background is followed by discussion of the empirical research on multiple source use by historians—what do we know about what historians can do as far as multiple source use is concerned? Given that foundation in relevant theory and research, the next step is to consider translation of this to K-12 students—what should students be expected to do, as far as multiple source use? With those expectations having been outlined, a major goal of the chapter can be addressed, which is to consider what we know about what students can actually do as far as multiple source use. Therefore, we next review research on students and multiple source use in history: the processes they use, the outcomes they aim at or achieve, what they do on their own, and what they can do in the context of relevant instruction or interventions. Having completed this review, the overarching question of the chapter can be addressed: what is known about the use of multiple sources in history? In particular, what is known about this in the context of education in the classroom, and what supports these conclusions? Based on this review and the discussion of the expectations for students’ use of multiple sources in history, consideration of possible educational implications and directions for future research conclude the chapter.

OVERVIEW OF MULTIPLE SOURCE USE IN HISTORY Historical inquiry begins with developing a question about the past, which for the historian usually also means having a sense of what type of information would be

208  •  Fox and Maggioni

needed to answer it, and whether and where there are relevant and sufficient sources available to do this (Marrou, 1959/66; Voss & Wiley, 2006). Not all questions about the past are equally interesting or significant; understanding what counts as a good question requires having a grounding in knowledge about the relevant time period or topic, as well as the type of meta-knowledge about sources and their availability just mentioned. It also means knowing that the question has not yet been addressed, or if it has, that it remains open for further consideration or interrogation of the previous attempts to address it. Given the substantial challenges involved in producing an historical account, the task of historical inquiry through use of multiple sources is not one to be undertaken lightly. The first step of developing an investigable question matters a great deal. Because historical inquiry involves investigation of the past, historians confront a number of challenges related to obtaining, interpreting, and evaluating the primary and secondary sources they use. Remnants of the past are just that, remnants; historical sources are therefore essentially fragmentary and limited in nature (Marrou, 1959/66). In terms of facilitating historical understanding and providing evidence regarding claims about the past, a bygone time, these remnants taken by themselves are unfamiliar and out of context (Marrou, 1959/66; Wineburg, 1999). Where these remnants or secondary accounts are presenting a human voice or the work of human hands, they are essentially shaped by the perspectives, purposes, and available knowledge base of their authors or creators (Wineburg, 1991b). The historian takes all of these aspects of historical sources into account when evaluating their relevance, accuracy, and trustworthiness as providing evidence about claims being made about the past. It is important to note that these evaluations fall upon a continuum of degrees of relevance, accuracy, and trustworthiness; it is not a binary categorization. Furthermore, such evaluations are always made in relation to the specific historical inquiry. Thus, the resulting characterizations are not intrinsic features of the specific historical source, but a function of its potential contribution to the overall inquiry and in dialogue with the rest of the available sources. Beyond these history-specific aspects related to obtaining, evaluating, and interpreting multiple sources in history, there are also certain generic and certain discipline-specific aspects to the work historians engage in when using multiple sources in generating historical accounts. Generic aspects include the cognitive and affective demands associated with managing information from multiple sources (addressed more fully in Chapters 2 and 3 of this volume). These types of demands have been spelled out in earlier models related to multiple document use and focusing on a reader encountering multiple texts (e.g., Perfetti, Rouet, & Britt, 1999; Rouet & Britt, 2011). Over and above the reader’s work in comprehending each individual text, a multiple text situation requires a multitude of additional processes and strategies for tracking, evaluating, interrelating, and integrating the information understood from each text, in combination with the reader’s understanding of the task demands and the reader’s prior knowledge (Rouet & Britt, 2011). Assuming that the reader’s work of comprehension involves construction of mental representations (e.g., Kintsch, 1998), a multiple text reading situation adds to the complexity of this work. The reader must simultaneously develop a representation of the intertextual relations of the documents and a representation of the situations they describe, while also typically working to develop an integrated representation of what is being

Multiple Source Use in History  •  209

stated concerning the overarching topic across all of the texts (Perfetti et al., 1999). Working with multiple texts is therefore demanding for the reader, even in the relatively rare case that the task and the nature of the sources mean that it is more a matter of assembling a coherent story out of consistent and complementary documents, as in writing a school report. These generic demands of reading multiple documents carry over to multiple source use in history as well. Historians can be said to read sources, even when these sources are not written or verbal in nature (e.g., VanSledright, 2016), insofar as they are creating mental representations of the meaning of the source as a kind of text from or about the past. However, along with those generic demands there are history-specific demands associated with the nature of the task involved—that is, creating an historical account, and the object of inquiry—that is, the past. Among these is the limited, often contradictory or conflicting nature of the evidence provided by the available sources, which makes integration and interrelation of this information an additionally challenging task. Another history-specific demand is the necessarily speculative and inferential type of reasoning involved in building a story from these pieces of evidence (e.g., Voss & Wiley, 2006). Another is the importance of identifying gaps or limitations in what can be concluded based upon the sources at hand; these limitations themselves can become part of the historian’s account (e.g., Leinhardt & Young, 1996). Another is meeting the disciplinary criteria for presenting and supporting claims; this requires a high level of tracking of the source(s) behind any given claim and corroboration of information across sources. Another is the need for extensive prior knowledge related to the historical territory concerned (e.g., Wineburg, 1998). Finally, in using multiple sources, historians aim to produce some type of extended, substantiated account of the past that has persuasive argumentative force and is intended for others to read (Marrou, 1959/66). Their work is not just a matter of building a personally satisfying coherent internal understanding of what is known about a particular issue or topic. All of these history-specific demands involved with using multiple sources bring to the fore the epistemic challenge that characterizes the discipline. In fact, the dialogue between the present and the past we described in the prior section requires openness in front of what is irreducibly “other”, acceptance of the intrinsic limitation of what can be known about the past, and willingness to submit one’s insights to the test of available evidence. Affectively, history requires a sort of very delicate balancing act: on one hand, the very questions that drive its inquiry are worth pursuing inasmuch as they matter to the human experience of the historian. On the other hand, though, the historian needs to make space for the past to speak on its own terms. Disciplinary practices, such as contextualization, corroboration, sourcing, and, more generally, respect for the entire body of historical evidence, whether compatible or not with the personal inclinations or initial insights of the historian, are powerful aids in this endeavor, but are no substitute for the deeply personal affective and cognitive development required by such an epistemic stance.

RESEARCH ON MULTIPLE SOURCE USE IN HISTORY: HISTORIANS There does not appear to be any empirical research published in peer-reviewed journals that investigates or observes historians engaging in multiple source use while

210  •  Fox and Maggioni

carrying out their own historical inquiries. Instead, historians have been studied when addressing specific researcher-provided questions and reading and analyzing sources provided by the researcher. In addition, historians have not typically been asked to produce a coherent written account addressing the historical question presumed to be guiding their analysis, interpretation, and synthesis of the provided sources. As a consequence, there is no solid research basis for our understandings of how historians develop questions, find and select sources, and work with a set of sources to produce historical accounts addressing their own questions about the past. The empirical research on multiple source use by historians has focused on what historians do given a question and a pre-selected set of documents, which mimics a student’s task in confronting a Document-Based Question (DBQ) on an AP test rather than exemplifying the historian’s disciplinary work (Wineburg, 1998). In particular, there is a critical gap in terms of our understandings of the beginning phases of historical inquiry, the development of questions, and search for and identification of relevant sources (Voss & Wiley, 2006). Another gap concerns our understandings of how historians do the extended, complex work of interpreting, analyzing, evaluating, and synthesizing multiple sources in the service of producing the particular form of argumentative, evidence-based, and discipline-specific type of writing that is an historical account (Voss & Wiley, 2006). Despite these gaps, the small set of studies that have investigated historians’ reading of researcher-provided multiple sources to address an historical question (Wineburg, 1991a, 1991b, 1998) have been profoundly influential. In particular, Wineburg’s (1991a) identification of three key heuristics (used by at least four of the eight historians he observed thinking aloud when reading sources related to the Battle of Lexington) brought sourcing, corroboration, and contextualization into the educational lexicon and also, remarkably, into history classrooms (e.g., Montgomery County Public Schools, 2014). In two studies involving the use of sources related to the Battle of Lexington (Wineburg, 1991a, 1991b), eight historians, four of them specialists in American history, were asked to read two diary entries, an autobiography excerpt, a formal deposition, a newspaper report, and a protest letter, all written near the time of the event, along with excerpts from a textbook and an historical novel. Their instructions were to try to understand what happened. They thought aloud while reading, and read each text aloud; documents were read twice, first in their entirety, and then as presented sentence-by-sentence on index cards. Information about the origin of the source was provided at the end of the document. Additional tasks involved rating the accuracy of the depiction of the event in three different paintings, and ranking the documents in terms of comparative trustworthiness of the evidence provided for understanding what happened. In this multiple text reading (but not writing) situation, historians engaged in three key processes that support historical reasoning: (a) corroboration, the act of comparing documents with one another; (b) sourcing, the act of looking first to the source of the document before reading the body of the text; and (c) contextualization, the act of situating a document in a concrete temporal and spatial context. (Wineburg, 1991a, p. 77; note that here Wineburg used source to refer to the provided information about the document’s origin)

Multiple Source Use in History  •  211

Despite their awareness that this task was not a typical thing for them to do, the historians showed what they can do as historians, even if not the extent of what they actually do in the service of historical inquiry. Wineburg also emphasized that historians’ engagement in the use of these heuristics came naturally and as part of an integrated belief-system about how one interacts with historical sources (1991a, 1991b); the heuristics were not simply available tools to be picked up, used, and then put away again. This suggests that if we want to make students’ performance more like that of historians as far as the reading involved in history DBQ-style tasks, it is not enough simply to train them to do sourcing, contextualization, and corroboration. Wineburg’s other study of multiple source use by historians (1998) compared two historians’ readings (as observed from think-alouds) of a set of primary source documents concerning Lincoln’s views on race; one of the historians specialized in that time period, and the other did not. In this way, the historian’s historical expertise could be separated from the contribution of extensive background knowledge, insofar as the non-specialist possessed the former but not the latter. One of the key points that emerged was that the non-specialist’s eventual development of an interpretation of the set of documents rested upon his effortful creation of a context within which the documents could make sense together for him. For the specialist, such a context already existed in the form of his extensive background knowledge of both the time period and the relevant historical scholarship. Contextualization therefore seems to be critical, yet likely to be difficult for students, who lack both background knowledge and the understandings and motivations that would drive them to work to construct a context that supports integration across multiple, conflicting sources. The work done on multiple source use by historians has been structured as expertise research. Wineburg’s studies involved novice–expert (Wineburg, 1991a, 1991b) and expert–expert (Wineburg, 1998) comparisons, although only the conclusions regarding experts are reviewed in this section. In addition, a further novice–expert comparison study (Rouet, Favart, Britt, & Perfetti, 1997) compared the reading of multiple historical sources by eight graduate students in history (disciplinary specialists) and 11 graduate students in psychology (disciplinary novices). Here the intent was to distinguish between generic literacy skills, presumably possessed by both groups of advanced students, and those pertaining particularly to reading multiple historical sources. In this timed studying situation, sources were presented via computer in a hypertext format, and participants were required to produce an opinion essay from memory after reading the documents. The sources included conflicting historian essays and first-person accounts, along with official documents and a textbook-style account. In addition, participants were given background information about the topic (the history of the Panama Canal) prior to reading the documents. As is typical in this type of reading task, the background information was not treated as a source to be analyzed, but as an external form of reliable background knowledge intended to make up for likely deficits in that regard. Findings regarding the specialists included that they applied multiple criteria for usefulness of different types of documents, which varied depending upon the document type. Although first-person accounts were seen as likely to be untrustworthy, specialists also saw them as useful for forming an interpretation of the controversy being analyzed. In their opinion essays, specialists did engage in historically oriented contextualization; however, no evidence emerged that

212  •  Fox and Maggioni

sourcing, corroboration, or attending to missing information were particular features of multiple source use by history specialists at this level when given this type of task.

TRANSLATING THIS TO STUDENTS So far, the focus has been on historians and their practice of historical inquiry. Given the role of multiple source use in history as outlined above, and given what the research has shown about how historians read multiple documents under certain rather constrained conditions, what do we want students to be able to do as far as learning to use multiple sources in history? A comprehensive and developmentally articulated answer to this question has been provided in the C3 Framework for Social Studies State Standards recently published in the U.S. by the NCSS (2013). This framework was developed as a resource to guide states in revising their social studies standards and curricula, as part of the current national educational reform effort associated with the development and publication of the Common Core State Standards for English Language Arts/Literacy (National Governors Association Center for Best Practices and Council of Chief State School Officers, 2010a) and Mathematics (National Governors Association Center for Best Practices and Council of Chief State School Officers, 2010b). The C3 Framework takes as its foundation that learning social studies content should be framed around inquiry. Within this framework, which covers history as well as other types of social studies learning, four dimensions making up an arc of inquiry are identified: developing questions and planning inquiries, applying disciplinary concepts and tools, evaluating sources and using evidence, and communicating conclusions and taking informed action (NCSS, 2013, p. 17). For history, this means, first, that students are expected to become increasingly capable of directing their own inquiry (which will be based on multiple source use) by generating their own compelling and supporting questions related to the past. Here, compelling questions are defined as questions that address disciplinary problems and issues, that require use of disciplinary concepts and tools, and that result in the construction of arguments and interpretations (NCSS, 2013, p. 97). Supporting questions concern “descriptions, definitions, and processes about which there is general agreement” (NCSS, 2013, p. 105) within the discipline, and serve an informationgathering, explanatory function within the larger inquiry associated with a given compelling question. The C3 Framework goes into considerable detail for the second dimension of the inquiry arc, use of disciplinary concepts and tools, identifying what this entails for each of the social studies areas addressed (NCSS, 2013). For history, the ambitious program outlined for students’ progressive stages of learning with regard to the use of disciplinary concepts and tools covers multiple indicators. These indicators concern students’ development of understandings of and ability to work independently with: change and continuity over time and across areas of life, evaluation of context, appreciation of multiple perspectives, use of multiple historical sources as evidence, and reasoning related to causation and argumentation (see NCSS, 2013, Tables 20–3, pp. 46–49). The third dimension of the C3 Framework inquiry arc zeroes in specifically on source use (NCSS, 2013). In particular, students are expected to learn how to gather and evaluate sources, and how to use the evidence from sources to develop and support claims. The goal is that:

Multiple Source Use in History  •  213

Students should use various technologies and skills to find [and evaluate] information . . . Through the rigorous analysis of sources and application of information from those sources, students should make the evidence-based claims that will form the basis for their conclusions. (NCSS, 2013, p. 53) Part of the developmental trajectory outlined here includes the shift from looking at a single source (by the end of grade 2), to finding and evaluating multiple sources that can represent a wide range of perspectives (by the end of grade 12). Starting in grade 3, students are expected to use multiple sources to identify evidence addressing their compelling questions, while by the end of grade 12, they are expected to corroborate and integrate the information across multiple, divergent sources. The final dimension of the C3 inquiry arc addresses what is to be done once an answer to a compelling question has been developed, namely the communication and critique of conclusions and, where appropriate, taking informed action through civic engagement (NCSS, 2013). The expectation is that students will learn to create formal, well-structured, substantiated arguments presenting and justifying their conclusions regarding their inquiry. They should be able to present their arguments and conclusions in multiple formats and modalities in order to communicate them within and outside of the disciplinary and educational context. As part of the process of developing and communicating conclusions, students should also become increasingly adept at critiquing the arguments and explanations offered by themselves and others. Finally, the role of social studies learning as preparation for civic life entails that students develop understandings of the types of problems and possible solutions that fall within the realm of the social studies disciplines, as well as becoming practiced in making decisions and taking action related to such problems. This condensed overview of the C3 Framework and its centering of social studies learning around the arc of inquiry makes it evident that multiple source use is the bedrock upon which this learning will rest. In this framework, it is expected that students will begin working with identification and analysis of historical sources at the very earliest stages of their schooling, and will work with multiple historical sources beginning in grade 3. They will take increasing ownership of the inquiry process, the development of their own compelling questions, and the gathering and selection of sources. They will begin their thinking about historical inquiry (in grades K-2) with thinking about differences in perspective and different accounts of the same historical event. And they will conduct their inquiry with the expectation that the outcome will be a public statement of their conclusions, which they must be prepared to defend and able to adapt for different types of audiences and presentation format. These are high expectations, but they are certainly consistent with the view of the nature of history and the role of multiple source use in history outlined above. However, the suggested developmental pathways and breakdown of the progression of learning from grades K through 12 that will bring students to the desired level of proficiency for each of the indicators may be somewhat optimistic. The approach taken to history learning in the C3 Framework (NCSS, 2013) does address a number of identified potential barriers to students’ ability to think historically (Nokes, 2011), including deficits in background knowledge and the use of it inappropriately, students’ limited and non-adaptive approaches to knowledge about the world and about the past, and their

214  •  Fox and Maggioni

lack of understanding of the discipline of history and the work of historians. However, it remains to be seen whether instruction that follows the developmental progression outlined in the C3 Framework will be successful in overcoming the possible barrier presented by the level of cognitive demand of this type of thinking, reading, and writing. In addition, although historical sources are acknowledged to include all types of remnants of the past, the C3 Framework does not address how students (and teachers) are to learn to analyze and interpret such varied sources as works of art (e.g., Kingkaysone, 2014), music (e.g., Pellegrino, 2013), artifacts (e.g., Anderson et al., 2013), films (e.g., Stoddard, 2012), photographs (e.g., Callahan, 2015), or collective memory (Wineburg, Mosborg, Porat, & Duncan, 2007), or how they can successfully navigate the challenges presented by the types of information and potential sources (varying widely in their transparency, trustworthiness, and accuracy) that are available online (e.g., Wineburg & Reisman, 2015). Building on the research experience of academic faculty and on the teaching experience of expert teachers, a few publications have been made available, whether in print or online, that offer examples and guidance for accomplishing this type of work in the classroom (e.g., Lesh, 2011; Mandell & Malone, 2007; Wineburg, Martin, & Monte-Sano, 2013). Even with such support, the C3 Framework, if adopted by the states, will certainly place high demand on teacher preparation and professional development, particularly for teachers at the elementary and middle school levels, who are not typically specialists in history or other social studies disciplines. Teachers’ and pre-service teachers’ understandings of multiple source use in history (e.g., Yeager & Davis, 1996) and of how to incorporate multiple source use in history (e.g., Friedman, 2006; Salinas, Bellows, & Liaw, 2011; Stoddard, 2010; van Hover, Hicks, & Dack, 2016) have not been observed to be consistently such as can support the type of instruction outlined here. Given these rigorous expectations regarding what K-12 students should be able to do in terms of multiple source use in history, what does the existing body of research show about their capabilities when they are asked to do such work, either on their own or in the context of instruction or an intervention?

RESEARCH ON MULTIPLE SOURCE USE IN HISTORY: K-12 STUDENTS Elementary School Students There are few studies of multiple source use in history by elementary school students. They have all been small-scale qualitative studies involving think-alouds by a small number of students (Afflerbach & VanSledright, 2001; Fillpot, 2012; VanSledright & Kelly, 1998). In these studies, the students were generally given primary or secondary source materials that had been created, edited, or adapted, primarily to improve their readability; these sources were predominantly written documents of some type. In the studies of multiple source use involving 5th-grade participants (Afflerbach & VanSledright, 2001; VanSledright & Kelly, 1998), it appeared that students did not make connections across the provided sources, which were parallel textbook passages including embedded primary sources or other types of enrichment materials. Even when students had been exposed to inquiry-style tasks in history and were familiar with the use of diverse types of history texts (including biographies and historical fiction), they

Multiple Source Use in History  •  215

did not interact particularly effectively with the multiple sources they were asked to read (VanSledright & Kelly, 1998). They did not engage in sourcing or consider reliability (VanSledright & Kelly, 1998). Where the primary source material (a diary excerpt) and alternative text (a poem) had not been adapted for readability, less able readers also had difficulty with these somewhat unfamiliar embedded texts (Afflerbach & VanSledright, 2001). The larger conclusion from these studies appears to be that upper elementary students need more than just exposure to inquiry-type tasks, primary sources, or alternative texts in order to begin to engage effectively in multiple source use in history. The two 3rd-grade students in the study by Fillpot (2012) had been exposed to a fullblown history curriculum, Bringing History Home, that was intended to promote the development of historical thinking skills. Fillpot found that one 3rd grader appeared to be quite willing and able to use key historical heuristics, including empathy and analogical reasoning in particular. He did this when reading (or having read to him) an extremely challenging set of 17 sources (adapted for readability) concerning a complex and unfamiliar historical topic (the Dawes Severalty Act of 1887). The researcher, who was also the creator of the Bringing History Home curriculum, suggested that this student’s use of these heuristics was scaffolded by his exposure to this curriculum, although she also voiced the alternative possibility that this particular student had a domain-specific talent for historical thinking. Overall, the two 3rd-grade students were able to make only limited headway with this difficult multiple source use task on an unfamiliar topic. However, it is not clear whether this was due to shortcomings in their prior instruction, or simply due to the high difficulty level of the task. Middle School Students The studies of multiple source use by middle school students included one qualitative, think-aloud study (Wooden, 2008), and two larger-scale experimental studies (Wissinger & De La Paz, 2016; Wolfe & Goldman, 2005). One of these experimental studies involved a manipulation of the type of text provided (Wolfe & Goldman, 2005), and the other tested an extended instructional intervention (Wissinger & De La Paz, 2016). The qualitative study was similar to those done with elementary students, in which a few students thought aloud while reading a set of sources, often adapted for readability. In this case, the two 6th-grade participants read adapted excerpts from primary source documents about Lincoln’s views on race (Wooden, 2008). The findings were also similar to those from the studies with elementary students. Even though these students had prior instructional exposure and experiences with inquirystyle tasks and the reading of diverse types of historical texts, they did not engage in sourcing, consider reliability, or appropriately contextualize the provided documents (Wooden, 2008). In the one-time manipulation study (Wolfe & Goldman, 2005), 6th graders thought aloud while reading two directly contradictory and highly cross-referential researchercreated “historian” explanations for the fall of Rome. They had studied this topic earlier in the school year (noteworthy as the only non-U.S. history topic addressed in the entire body of empirical research reviewed). In this reading situation intended to support cross-text connections, students were generally capable of making such connections between the two documents, as well as to a map and timeline viewed before reading. Further, students who devoted more effort to making such connections and

216  •  Fox and Maggioni

explaining the connections to themselves also showed stronger performance on a postreading reasoning task asking for their own explanation of the fall of Rome. The authors suggested that the types of cross-referencing of provided sources that students were observed to engage in during this scaffolded multiple source reading situation could be a valuable precursor of more developed historical corroboration skills or propensities. In the longer-term instructional intervention (Wissinger & De La Paz, 2016), 6th and 7th-grade students received instruction about either history-specific argument schemes and critical questions related to those schemes, or more traditional, generic reading strategies such as considering the author’s purpose and identifying main ideas and supporting details. Students worked with prepared sets of source documents on four historical controversies related to American history; these sets each included two conflicting primary source documents (adapted for readability), as well as a secondary source that gave background information. The students’ final product from their investigation of each of these controversies was an argumentative opinion essay addressing the question that guided the investigation (Wissinger & De La Paz, 2016, p. 47). Students who received instruction about argument schemes and critical questions learned more historical content, and their written arguments used stronger evidence and better rebuttals. However, the intervention did not appear to foster improvement in students’ ability to acknowledge the role of perspective or their use of contextualization, when historical thinking scores for the post-test essay were compared across control and comparison groups. High School Students Studies of multiple source use by high school students are more numerous, but are strictly limited to the context of the study of U.S. history, predominantly by 11th graders. A small set of studies investigated what students can do on their own when given a task involving the use of multiple sources (Britt & Aglinskas, 2002, Exp. 1; Monte-Sano, 2010; Wineburg, 1991a, 1991b). The remainder either involved a one-time manipulation of text, topic, or task (Halvorsen, Harris, Martinez, & Frasier, 2016; Stahl, Hynd, Britton, McNish, & Bosquet, 1996; Stahl, Hynd, Montgomery, & McClain, 1997) or compared different types of instruction or instructional supports (Britt & Aglinskas, 2002, Exp. 3; De La Paz & Felton, 2010; Monte-Sano, 2008; Nokes, Dole, & Hacker, 2007; Reisman, 2012). In the studies investigating how students approached the task of reading a set of historical primary and secondary sources, it appeared that 11th graders (Britt & Aglinskas, 2002, Exp. 1; Monte-Sano, 2010) and 12th graders who had taken 11th-grade U.S. history (Wineburg, 1991a, 1991b) did not consistently use sourcing, contextualization, or corroboration. In addition, students (some of whom had received instruction regarding use of evidence and historical thinking) did not always use evidence or interpret documents appropriately, and did not show strong awareness of the criteria for presenting a historical argument as distinct from the type of information-assembly appropriate to a report (Monte-Sano, 2010). Similarly, the eight student participants in Wineburg’s (1991a, 1991b) investigations tended to treat the task as one of information-gathering, and texts as either biased or authoritative. Studies of one-time manipulations related to multiple source use tasks in history investigated the effectiveness of including refutational text to address 9th graders’

Multiple Source Use in History  •  217

misconceptions about Columbus (Stahl et al., 1997), the relative influence of a culturally relevant task for Latino/a 11th-grade U.S. history students (Halvorsen et al., 2016), and the influence of task type (reading to form an opinion or to describe) for students in 10th-grade AP U.S. history (Stahl et al., 1996). The broad conclusion from these three studies is that brief manipulations such as these do not have the desired impact. It was found that students’ attitudes were easier to shift than their misconceptions, and that refutational texts were effective at changing attitudes but did not work as well with misconceptions (Stahl et al., 1997). Cultural relevance did not make a difference in students’ performance on the post-test writing task, but students did express somewhat higher interest in the more culturally relevant task, possibly because of its relative unfamiliarity (Halvorsen et al., 2016). Students did not change how they read the sources depending on whether they were asked to form an opinion or develop a description, but tended to view the task of forming an opinion as freeing them from the obligation of sticking to the information provided in the texts (Stahl et al., 1997). Longer-term instructional interventions have had better success with improving high school students’ performance when reading multiple historical sources to produce a written response to address a given question. A relatively brief intervention (one class period) using a computer program intended to scaffold 11th-grade students’ notetaking and integration of multiple source documents did appear to have some impact (Britt & Aglinskas, 2002, Exp. 3). After having worked with multiple primary and secondary documents presented through the computer program, students did more sourcing, contextualization, and corroboration, and provided more document-based information in their written essays, compared to students who did unscaffolded reading of the same information provided as a textbook-style account. The duration of longer-term interventions or periods of instruction, all with 11thgrade students, has ranged from a matter of a few weeks (De La Paz & Felton, 2010; Nokes et al., 2007) to 6 or 7 months (Monte-Sano, 2008; Reisman, 2012). The types of instruction provided included attention to historical thinking strategies intended to support management of conflicting primary or secondary accounts (De La Paz & Felton, 2010); use of multiple texts and instruction on sourcing, contextualization, and corroboration (Nokes et al., 2007); inquiry-based instruction that involved collaborative work to make sense of historical sources (Monte-Sano, 2010); and implementation of the Reading Like A Historian curriculum (Reisman, 2012). This curriculum (Reisman, 2012) includes collaborative investigation of central historical questions using multiple documents and strategy instruction related to sourcing, contextualization, corroboration, and close reading. It appears from these studies that corroboration and contextualization were more challenging for students (Nokes et  al., 2007; Reisman, 2012). However, sourcing (De La Paz & Felton, 2010; Nokes et al., 2007; Reisman, 2012) and close reading (Reisman, 2012) were significantly improved by these interventions, as was the quality of students’ argumentative writing and use of evidence (De La Paz & Felton, 2010; Monte-Sano, 2008).

CONCLUSIONS In the research on K-12 students’ multiple source use in history, there are a number of noteworthy commonalities: students were always given a pre-assembled set of multiple sources (which were typically only or predominantly documents) and a question;

218  •  Fox and Maggioni

the topics were drawn from U.S. history (with a single exception); where a think-aloud was not involved, there was also a time limit for students’ reading and production of some type of (typically overtly argumentative) written outcome. In other words, investigations of students’ use of multiple sources in history have consistently been modeled upon a DBQ-style assessment situation such as occurs in the AP U.S. History exam. This suggests that the overarching aim has been to improve performance in this particularly constrained multiple source use situation. This aim can also mean working toward helping students to think more like historians, but such thinking is not being pursued for its own merits or grounded within its own frame of reference (that is, historical inquiry as a way to build historical understandings and the epistemic development that such work implies). The researchers are certainly not presenting themselves as having this aim, but the body of research, taken as a whole, provides rather compelling evidence of a consistent orientation toward such assessment-like performance as the goal in multiple text use in history. When the reviewed research is viewed against the C3 Framework’s expectations for what students should be able to do as multiple source users in history, there are evident gaps in what is known about how instruction can support students in meeting these expectations. This makes it challenging to forward recommendations as far as educational practice; the assumptions behind the vision of students’ historical learning in the C3 Framework are wonderful, but the reality of what will be involved in bringing this into K-12 classrooms is quite daunting. There will be a need for research on teacher preparation and professional development, as well as better-grounded support for the suggested developmental pathways articulated for each of the indicators associated with each of the dimensions of the inquiry arc. A commitment to having students engage in doing their own historical inquiry brings in a whole host of issues, some of which have been addressed in the science literature, where support for guided inquiry has long been a focus of instructional research (e.g., de Jong, 2017). Although there seems to be much work ahead, the price does not seem too large when weighed against the potential outcome: students who are able to ask and answer questions for themselves “about how the world came to be how it is, about how power, inclusion and exclusion operate in the world, and about how an understanding of the past might influence the present and the future” (Levstik, 2017, p. 115).

REFERENCES Afflerbach, P., & VanSledright, B. (2001). Hath! Doth! What? Middle graders reading innovative history text. Journal of Adolescent & Adult Literacy, 44, 696–707. Anderson, K., Frappier, M., Neswald, E., & Trim, H. (2013). Reading instruments: Objects, texts and museums. Science & Education, 22, 1167–1189. Ashby, R. (2011). Understanding historical evidence: Teaching and learning challenges. In I. Davies (Ed.), Debates in history teaching (pp. 135–147). New York: Routledge. Britt, M. A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20, 485–522. Callahan, C. (2015). Creating or capturing reality? Historical photographs of the Progressive Era. The Social Studies, 106, 57–71. Collingwood, R. G. (1946/2014). The idea of history. Mansfield Centre, CT: Martino Publishing. De La Paz, S., & Felton, M. K. (2010). Reading and writing from multiple source documents in history: Effects of strategy instruction with low to average high school writers. Contemporary Educational Psychology, 35, 174–192.

Multiple Source Use in History  •  219 de Jong, T. (2017). Instruction based on computer simulations and virtual laboratories. In R. E. Mayer & P. A. Alexander (Eds.), Handbook of research on learning and instruction (pp.  502–521). New York: Routledge. Fillpot, E. (2012). Historical thinking in the third grade. The Social Studies, 103, 206–217. Friedman, A. M. (2006). World history teachers’ use of digital primary sources: The effect of training. Theory and Research in Social Education, 34, 124–141. Goldman, S. R., & Scardamalia, M. (2013). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities. Cognition and Instruction, 31, 255–269. Halvorsen, A., Harris, L. M., Martinez, G. A., & Frasier, A. S. (2016). Does students’ heritage matter in their performance on and perceptions of historical reasoning tasks? Journal of Curriculum Studies, 48, 457–478. Kingkaysone, J. (2014). Reading art: Multiliteracies and history education. The Educational Forum, 78, 409–420. Kintsch, W. (1998). Comprehension: A paradigm for cognition. New York: Cambridge University Press. Leinhardt, G., & Young, K. M. (1996). Two texts, three readers: Distance and expertise in reading history. Cognition and Instruction, 14, 441–486. Lesh, B. A. (2011). “Why won’t you just tell us the answer?”: Teaching historical thinking in grades 7–12. Portland, ME: Stenhouse Publishers. Levstik, L. S. (2017). Learning history. In R. E. Mayer & P. A. Alexander (Eds.), Handbook of research on learning and instruction (pp. 115–130). New York: Routledge. Lowenthal, D. (2000). Dilemmas and delights of learning history. In P. N. Stearns, P. Seixas, & S. Wineburg (Eds.), Knowing, teaching and learning history: National and international perspectives (pp. 63–82). New York: New York University Press. Mandell, N., & Malone, B. (2007). Thinking like a historian: Rethinking history instruction. Madison, WI: Wisconsin Historical Society Press. Marrou, H. (1959/66). The meaning of history (4th ed., R. J. Olsen, trans.). Baltimore, MD: Helicon Press. Megill, A. (2007). Historical knowledge, historical error. Chicago, IL: The University of Chicago Press. Monte-Sano, C. (2008). Qualities of historical writing instruction: A comparative case study of two teachers’ practices. American Educational Research Journal, 45, 1045–1079. Monte-Sano, C. (2010). Disciplinary literacy in history: An exploration of the historical nature of adolescents’ writing. Journal of the Learning Sciences, 19, 539–568. Monte-Sano, C. (2016). Argumentation in history classrooms: A key path to understanding the discipline and preparing citizens. Theory into Practice, 55, 311–319. Montgomery County Public Schools. (2014). United States history: Semester A. Retrieved from: www.montgomery schoolsmd.org/uploadedFiles/curriculum/socialstudies/high/grade9/US-A%20Overview%201-pager(2).pdf. National Council for the Social Studies (NCSS). (2013). The college, career, and civic life (C3) framework for social studies state standards: Guidance for enhancing the rigor of K-12 civics, economics, geography, and history. Silver Spring, MD: NCSS. National Governors Association Center for Best Practices and Council of Chief State School Officers. (2010a). Common core state standards for English language arts and literacy in history/social studies, science, and technical subjects. Washington, DC: NGA/CCSSO. National Governors Association Center for Best Practices and Council of Chief State School Officers. (2010b). Common core state standards for mathematics. Washington, DC: NGACCSSO. Nokes, J. D. (2011). Recognizing and addressing the barriers to adolescents’ “reading like historians”. The History Teacher, 44, 379–404. Nokes, J. D., Dole, J. A., & Hacker, D. J. (2007). Teaching high school students to use heuristics while reading historical texts. Journal of Educational Psychology, 99, 492–504. Pellegrino, A. M. (2013). Employing music in the history classroom: Four models. The Social Studies, 104, 217–226. Perfetti, C. A., Rouet, J., & Britt, M. A. (1999). Toward a theory of documents representation. In H. van Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 99–122). Mahwah, NJ: Lawrence Erlbaum Associates. Reisman, A. (2012). Reading like a historian: A document-based history curriculum intervention in urban high schools. Cognition and Instruction, 30, 86–112. Rouet, J., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. In M. T. McCrudden, J. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 19–52). Charlotte, NC: Information Age Publishing.

220  •  Fox and Maggioni Rouet, J., Favart, M., Britt, M. A., & Perfetti, C. A. (1997). Studying and using multiple documents in history: Effects of discipline expertise. Cognition and Instruction, 15, 85–106. Salinas, C., Bellows, M. E., & Liaw, H. L. (2011). Preservice social studies teachers’ historical thinking and digitized primary sources: What they use and why. Contemporary Issues in Technology and Teacher Education, 11, 184–204. Seixas, P. (1999). Beyond “content” and “pedagogy”: In search of a way to talk about history education. Journal of Curriculum Studies, 31, 317–337. Stahl, S. A., Hynd, C., Montgomery, T., & McClain, V. (1997). In fourteen hundred and ninety-two, Columbus sailed the ocean blue: The effects of multiple document readings on student attitudes and misconceptions (Reading Research Report No. 82). Washington, DC: Office of Educational Research and Improvement. Stahl, S. A., Hynd, C. R., Britton, B. K., McNish, M. M., & Bosquet, D. (1996). What happens when students read multiple source documents in history? Reading Research Quarterly, 31, 430–456. Stoddard, J. D. (2010). The roles of epistemology and ideology in teachers’ pedagogy with historical ‘media.’ Teachers and Teaching, 16, 153–171. Stoddard, J. D. (2012). Film as a “thoughtful” medium for teaching history. Learning, Media and Technology, 37, 271–288. van Hover, S., Hicks, D., & Dack, H. (2016). From source to evidence? Teachers’ use of historical sources in their classrooms. The Social Studies, 107, 209–217. VanSledright, B. (2012). Learning with texts in history: Protocols for reading and practical strategies. In T. L. Jetton & C. Shanahan (Eds.), Adolescent literacy in the academic disciplines (pp. 199–226). New York: The Guilford Press. VanSledright, B. (2016). Individual differences in reading history. In P. Afflerbach (Ed.), Handbook of individual differences in reading (pp. 245–258). New York: Routledge. VanSledright, B. A., & Kelly, C. (1998). Reading American history: The influence of multiple sources on six fifth graders. The Elementary School Journal, 98, 239–265. Voss, J. F., & Wiley, J. (2006). Expertise in history. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 569–584). New York: Cambridge University Press. Wineburg, S. (1991a). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87. Wineburg, S. (1991b). On the reading of historical texts: Notes on the breach between school and academy. American Educational Research Journal, 28, 495–519. Wineburg, S., (1998). Reading Abraham Lincoln: An expert/expert study in the interpretation of historical texts. Cognitive Science, 33, 319–346. Wineburg, S. (1999). Historical thinking and other unnatural acts. Phi Delta Kappan, 80, 488–499. Wineburg, S., Martin, D., & Monte-Sano, C. (2013). Reading like a historian: Teaching literacy in middle & high school history classrooms. New York: Teachers College Press. Wineburg, S., Mosborg, S., Porat, D., & Duncan, A. (2007). Common belief and the cultural curriculum: An intergenerational study of historical consciousness. American Educational Research Journal, 44, 40–76. Wineburg, S., & Reisman, A. (2015). Disciplinary literacy in history: A toolkit for digital citizenship. Journal of Adolescent & Adult Literacy, 58, 636–639. Wissinger, D. R., & De La Paz, S. (2016). Effects of critical discussions on middle school students’ written historical arguments. Journal of Educational Psychology, 108, 43–59. Wolfe, M. B. W., & Goldman, S. R. (2005). Relations between adolescents’ text processing and reasoning. Cognition and Instruction, 23, 467–502. Wooden, J. A. (2008). “I had always heard Lincoln was a good person, but . . .”: A study of sixth graders’ reading of Lincoln’s views on Black–White relations. The Social Studies, 99, 23–31. Yeager, E. A., & Davis, O. L., Jr. (1996). Classroom teachers’ thinking about historical texts: An exploratory study. Theory and Research in Social Education, 24, 146–166.

13

FUNCTIONAL SCIENTIFIC LITERACY Disciplinary Literacy Meets Multiple Source Use Iris Tabak ben-gurion university of the negev, israel

Lab coats and microscopes, rather than keyboards and word processors, and experimentation and discovery, rather than composition and rhetoric, are what people tend to associate with scientists and scientific work (Van Gorp, Rommes, & Emons, 2014). Yet, the production of scientific knowledge occurs as much through the selection and arrangement of prose as through the choice of variables and their manipulation (Latour, 1981). Literacy practices are inherent to scientific activity (Bazerman, 1988; Neuwirth & Contijoch, 2003; Yarden, Norris, & Phillips, 2015). This is true for lay as well as professional engagement with science, and science communication plays a key role in public engagement with science (Bromme & Goldman, 2014; National Science Board, 2016). In particular, laypeople turn increasingly to the Internet to access scientific information (Bromme & Goldman, 2014; Brossard & Scheufele, 2013; National Science Board, 2016). Online information presents the public with a wealth of information that varies in terms of relevance, quality, scope, and consistency of information (Bromme & Goldman, 2014; Grant et al., 2015). Thus, to understand scientific topics people need to comprehend and integrate information from multiple sources. On the one hand, people gravitate toward this process with the aim of having more agency in making informed decisions on science-related issues, but on the other hand, they often end up feeling frustrated and ill prepared for this task (Ladwig, Dalrymple, Brossard, Scheufele, & Corley, 2012; Ward, Henderson, Coveney, & Meyer, 2012). These feelings of frustration and sense of ineptness are a consequence of the dual challenge that laypeople face: the need to be able to proficiently read and comprehend multiple, concurrent, possibly conflicting, texts, as well as the need to effectively understand and evaluate scientific information. Individuals consulting online scientific information resources need to understand, through their reading, what the reported scientific work aims to achieve, how it tries to accomplish this, and what criteria can be 221

222  •  Iris Tabak

used to evaluate how well these goals have been achieved (Chinn, Rinehart, & Greene, 2016; Goldman et al., 2016). They need to apply this process to multiple individual information resources, discount irrelevant or unreliable information, and coordinate and resolve discrepancies among different information resources. Evaluating scientific information resources requires knowledge of the criteria that are relevant to science (Goldman et  al., 2016). For example, scientific information presents cause-and-effect relationships, and evaluating causal claims calls for assessing whether there is evidence that this relationship occurs beyond chance, or that the effect is not caused by other causes (Sutherland, Spiegelhalter, & Burgman, 2013). Such evaluation criteria are not intuitive, and even undergraduate science majors may not use such science-specific criteria extensively (Yang, Chen, & Tsai, 2013), though science majors are more likely to use such criteria than non-science majors (Lin, 2014). To better understand how science-specific evaluation criteria can infuse processes involved in comprehending multiple science information resources that are available online, I review a general multiple source comprehension framework (Goldman et al., 2016) against the backdrop of the reading processes of scientists. I examine scientists’ reading strategies, and the ways in which they can inform laypeople’s comprehension, evaluation, and use of multiple scientific information resources. It is difficult to draw a direct parallel between scientists’ and laypeople’s reading, because scientists read academic scientific articles with the goal of producing new knowledge, while laypeople read science news or informational texts in order to satisfy their curiosity or to guide decisions. Therefore, I will also discuss points of incongruence between these two reading contexts, as well as whether and how they might be reconciled. I will use the term layperson to refer to any individual who is not a practicing natural scientist, or studying toward or holding an advanced degree in a natural science. Following Goldman and Scardamalia (2013), source will refer to the origin of the information, such as the website on which it is posted, source information will refer to metatextual attributes such as the author or date of the information, and information resource will refer to the information itself, used synonymously with text or document. I will refer to three main information resources: (1) articles published in academic natural science journals that I will refer to as scientific articles; (2) science news articles published online in journalistic venues, such as science sections of general news media (e.g., the science section of the New York Times online edition) or science news media such as the Scientific American; and (3) scientific informational (expository) texts published on governmental, non-governmental, public, or private sector websites (e.g., Center for Disease Control website, Merck Manual website, or the SaneVax Inc. website). Although blogs and social media are gaining momentum as sources for public engagement with science, they represent a different genre that is not as well documented as journalism or expository writing. Moreover, when polled, most people list journalistic reports as their main source of science information (National Science Board, 2016; Schäfer, 2017). Therefore, I focus on online science news and scientific informational texts, using the acronym OSNIT to refer to both types of sources simultaneously.

HOW SCIENTISTS READ SCIENTIFIC ARTICLES In this section, I present a review of research that examined how scientists read scientific articles by having the scientists verbalize their thought processes while reading

Functional Scientific Literacy  •  223

(think alouds). Unfortunately, this review is limited in scope, because there are few such studies. However, the characterization of scientists’ reading processes is based on more than the handful of reading sessions studied, because the research that I review also drew on interview data in which the scientists described their reading processes beyond the think-aloud setting. In these studies, scientists did not read articles comprehensively or linearly (Bazerman, 1985; Charney, 1993; Shanahan, Shanahan, & Misischia, 2011), nor did they necessarily read one article at a time. Rather, they moved back and forth between articles. Scientists read from an evaluative1 stance, stating that being critical and wary was an explicit goal (Bazerman, 1985; Charney, 1993; Geisler, 1994/2013; Shanahan et al., 2011). Scientists browsed the title and name of the author and skimmed the abstract, then decided whether to continue reading the article (Bazerman, 1985; Neuwirth & Contijoch, 2003; Shanahan et al., 2011). They decided to read the article if they were familiar with the author, or decided not to read the article if the title and abstract pointed to a focus outside their current purpose. For example, a physicist changed his mind about reading an article on “remote sensing,” because on second glance it indicated that the wavelength region with which it dealt meant that the problems of measurement involved were different from those with which he was concerned (Bazerman, 1985). This previewing was also used to help maintain a critical stance by gaining an idea of the gist of the article before going through it, rather than figuring out the gist while going through the paper, which was mentioned as something that could distract from raising critical questions of the text (Charney, 1993). After scanning the title, author, and abstract, the scientists engaged in a cursory reading, moving between sections of the article. They often read the Results section before the Methods section. Following this non-linear, cursory reading, scientists went through the article a second time, reading more closely, but still not comprehensively (Bazerman, 1985; Charney, 1993). This second, closer reading included a number of practices, such as working through a derivation for themselves, verifying equations, rereading paragraphs, paraphrasing, and making inferences and predictions. In their non-linear reading of the articles, the scientists also moved back and forth between visuals and text (Shanahan et al., 2011), turning first to the visuals and only then to the text (Berkenkotter & Huckin, 1995). In subsequent closer readings, the scientists sometimes interleaved reading of additional articles with the reading of the current article (Bazerman, 1985; Charney, 1993). They turned to other articles that were cited in the current article (Shanahan et al., 2011) to compare or deepen their understanding of particular points, or to help them understand unfamiliar terms or methods. Throughout their reading scientists tried to evaluate and not just understand the article, judging the validity of the methods, the quality of the evidence, and the extent to which the claims aligned with other research (Charney, 1993; Shanahan et  al., 2011). Scientists expressed evaluation goals explicitly. Evaluation goals often drove the movement between articles, or the movement between sections of an article, such as when considering results in light of the methods. A longitudinal study that examined a science undergraduate (BSc) student’s reading processes from her first through the final year of study (Haas, 1994), illustrates how she developed practices akin to those identified in the above think-aloud studies.

224  •  Iris Tabak

In particular, she moved from linear reading of scientific articles to selective reading, paying special attention to tables and their legends in her final year of study. She became aware of the authors’ voices, their purpose and interests, and she developed an inclination toward evaluation, noting that not all claims were equally supported. Speaking of her future self in graduate school, she exclaimed that she would “just look at articles and tear them apart, say what’s wrong with them” (Haas, 1994, p. 68). This student is unique, because she worked as an assistant in a research laboratory for the last two years of her studies. Nonetheless, the case study is indicative of the values and practices that scientific communities cultivate.

A FRAMEWORK FOR MULTIPLE SOURCE USE The think-aloud studies of scientists reading scientific articles revealed practices, such as non-linear reading, that were not obvious from the content and structure of the scientific articles themselves. At the same time, the findings that these studies reported were of overt reading practices, such as moving back and forth between sections of the article, or relating results displayed in a table to the textual description of the methods that were used. Yet, reading also involves tacit processing through which readers construct models—that is, mental representations—of the content that they read. In this section, I provide an overview of a framework that describes such processing when reading and comprehending multiple sources. Building on prior work concerning both single and multiple source comprehension, Goldman et  al. (2016) developed a framework that describes the mental representations that are involved in comprehending multiple information resources. This framework includes four components: (1) the surface model, (2) the integrated model, (3) the intertext model, and the (4) task model. Below I describe how readers construct these models in the process of multiple source comprehension. At a basic level, readers need to form a representation of the meaning of the explicit content expressed in the words of the text. This meaning is represented in the surface model. The knowledge that can be gleaned from a text is not captured entirely in the words of the text, so further knowledge construction is needed. This additional layer of knowledge, the integrated model, arises from inferences that readers make based on the text. For example, if the text states that “owls sleep during the day,” a reader might infer that owls hunt for food at night even though information about owls hunting at night may not appear in the text. Readers make use of their knowledge of the world in order to draw such inferences. Familiarity with the structure of the text and the style of writing can help readers monitor which inferences are more or less valid. In the case of multiple source use, such inferences can be made by combining ideas across texts from different information resources. Such cross-information-resource inferences can help readers benefit from the wealth of knowledge that multiple rather than single information resources can provide. However, readers need to keep track of which content came from which information resource, as well as who created the information, when, and for what purpose, because this will drive or constrain the inferences that they can make. This metatextual information about the sources is represented in the intertext model, along with information about relationships between sources, such as whether sources contradict or support each other.

Functional Scientific Literacy  •  225

Readers approach texts for different purposes, for example to answer questions about natural phenomena, such as whether water goes down the drain in different directions in different parts of the world, or to help them to decide whether Bluetooth headsets are dangerous. These different purposes influence the depth with which readers will process the text, and whether they will attend to some aspects of the text more than others (Britt, Richter, & Rouet, 2014; Maier & Richter, 2013). The task model is a representation of these purposes and knowledge of how best to achieve these purposes. The task model serves an executive function in enacting, coordinating, and monitoring the construction of the other models. These models are interdependent, and iteratively constructed. Refining one model can lead to changes in one of the other models. For example, recognizing that claims that are supported by a larger body of evidence are a more reliable basis for a decision (knowledge in the task model) can drive a reader to examine whether evidence presented in different sources points to the same conclusions (an act that elaborates the integrated model). If the reader finds that a number of sources provide evidence that points to the same conclusion, but one source provides evidence that points to a different conclusion, then the reader might decide that the discrepant information resource is less reliable, and update their intertext model by “tagging” the discrepant information resource as less reliable in the intertext model. Subsequently, based on knowledge in the task model, the reader may tend to ignore this discrepant, less reliable, source. Readers who move back and forth between sources and engage in more iterative refinement of these models reach higher levels of comprehension (Anmarkrud, Bråten, & Strømsø, 2014; Britt et al., 2014; Cho & Afflerbach, 2015; Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012; Tabak, 2015).

SCIENTIFIC CONSIDERATIONS IN MULTIPLE SOURCE USE IN SCIENCE While texts are written to present a flowing, linear, representation, reading and deep comprehension may require violating this structure and moving back and forth between sections of the text (or between multiple sources), as well as constructing mental representations (the integrated model) that augment the literal information in the text. This was reflected in the scientists’ reading practices that included considering authors; previewing; and dynamic non-linear reading that coordinated between results and methods, results and conclusions, and text and visuals. Geisler (1994/ 2013, p. 26) described this as a tension between scientists-as-writers and scientistsas-readers: writers who want to drop all reference to the local context in order to extract statements about objects out there and readers who go out of their way to reconstruct the local context of these statements in order to decide how to interpret and use them. In other words, the scientists’ task model seems to include evaluation as a prominent goal (Mercier & Heintz, 2014), and the alignment between methods and findings (what Geisler refers to as “the local context of the statements”) functions as a central means for achieving this purpose.

226  •  Iris Tabak

This is an idealized depiction. In day-to-day practice, scientists can be less critical when considering their own versus others’ work, or when launching a new investigation versus refereeing for a journal (Geisler, 1994/2013). Similarly, when considering whether and how scientists’ reading practices can inform laypeople’s comprehension of OSNIT, it is important to note that both scientists and laypeople may be inclined to seek information that supports their intuitive or preconceived ideas (Nickerson, 1998; Shtulman & Harrington, 2016). Scientists and laypeople respond to unexpected information in a number of different ways (Brewer & Chinn, 1994). These responses can include ignoring pertinent information, or trying to make sense of the unexpected findings (Dunbar, 1997), which may lead to revising how one thinks about an issue. Although an evaluative stance can mitigate biases, no one can entirely circumvent human tendency to bias (Kenski, 2017). Scientists’ and laypeople’s vulnerability to bias brings to light the distinction between comprehension and adoption of ideas in a text. Readers may comprehend a text well, but reject its claims (Fox, 2009). The ability to comprehend and evaluate a text is a product of interactions between qualities of texts, such as their structure and content, and qualities of readers, such as prior knowledge, skill sets, values, and dispositions (Fox, 2009). Challenges Imposed by Scientific Texts that Laypeople Use The structure of scientific articles, as well as scientists’ knowledge of scientific principles and of how to produce and evaluate scientific claims, mediate scientists’ reading processes. Scientific articles follow a conventional structure composed of the sections: Introduction, Methods, Results, and Discussion (Bazerman, 1988; O’Neill, 2001). This canonical structure facilitates scientists’ non-linear reading of these articles (Shanahan et al., 2011). Scientific articles include details about the methods that were employed and about how the reported findings contribute to mechanistic and causal explanations. These details align with the scientific community’s preference for explanations that provide better causal accounts of the data (Koslowski, 2013), where “better” has much to do with methodological rigor (Mercier & Heintz, 2014). The literature review that is reported in the scientific article also facilitates judging whether an account is “better” than other scientific accounts, because it allows scientists to compare the explanation proffered in the scientific article with prior or competing explanations. In contrast, OSNIT are not as well tailored to applying the types of reading and evaluation strategies that the scientists employed. Science news is characterized by the inverted pyramid (IP) structure where the key conclusions are presented first, followed by main details, and concluding with background information (Boykoff & Rajan, 2007). Neither science news nor informational texts conform to a canonical structure as prominently or as consistently as scientific articles (Brossard & Scheufele, 2013), which makes it more difficult to move back and forth between sections, and to focus on particular targets, such as methods. OSNIT are far less detailed than scientific articles (Holtzman et al., 2005), and often omit precisely the information that scientists consider salient to evaluation, such as how a reported scientific investigation was conducted (Zimmerman, Bisanz, Bisanz,

Functional Scientific Literacy  •  227

Klein, & Klein, 2001). Informational texts can be especially difficult to evaluate, because they appear as a set of unequivocal assertions, with no agentive author behind what is stated in the text (Fox, 2009; Paige, 2014; Smith & Robertson, 2016). Scientific texts include specialized language and concepts that are unfamiliar to laypeople (Graesser, León, & Otero, 2002), and in some cases, words that are used in everyday contexts have alternative meanings in scientific contexts. Thus, the basic level of processing, constructing the surface model, can be difficult, even for proficient readers. Similarly, scientific texts may pose particular challenges for laypeople in constructing and refining the integrated, intertext, and task models, as well as in the iterative refinement of these models. OSNIT present only partial information concerning relationships, processes, or causal mechanisms. In constructing the integrated model, readers need to mentally elaborate and supplement the written information with their own prior knowledge in order to construct fuller (mental) models of the relationships, processes, or causal mechanisms that are depicted in the text. Laypeople tend to lack the knowledge required for this elaboration and supplementation (Graesser et al., 2002). Yet, these fuller causal models may be essential for reconciling information between different scientific information resources. As a result, limited prior knowledge can impede critique, evaluation, and integration of information. For example, upon reading a statement that high protein consumption can put stress on the kidneys, laypeople may easily form a declarative representation that high protein intake is related to kidney stress. However, they may not elaborate the representation in their integrated model with finer details about how the digestion process converts consumed food into other materials, which the body uses or discards. A more elaborate and scientifically informed mental model that supplements the information explicitly depicted in the text would also include information on how converting protein results in different outcome materials than converting carbohydrates. These “outcome materials” may affect the kidneys in different ways, which can be relevant to evaluating additional information resources that argue for or against high protein versus high carbohydrate diets. An integrated model that does not contain this more detailed information provides little basis with which to compare competing accounts. Difficulties in connecting information from scientific information resources to prior knowledge, and in constructing elaborate integrated models based on these connections, can also lead to flawed conclusions. Laypeople may be more influenced by ancillary features, such as the order in which information is encountered, rather than the substance of the information, such as the strength of the evidence or the cohesion of an explanation (Blanc, Kendeou, van den Broek, & Brouillet, 2008; Van Oostendorp, Otero, Leon, & Graesser, 2002). Thus, readers may adopt less supported claims, because of the order in which they appear. Another challenge of scientific information resources is that they include visuals (e.g., tables, graphs, figures) that convey information not provided in the text (Falk & Yarden, 2009; Graesser et  al., 2002). Therefore, readers need to be familiar with a range of representations and how to interpret them (Ainsworth, 2008), as well as able to integrate information from texts and visuals (Schnotz, 2002). This is very difficult for laypeople (Ainsworth, 2008; Cromley, Snyder-Hogan, & Luciw-Dubas, 2010; Schnotz et al., 2011; Shah & Hoeffner, 2002).

228  •  Iris Tabak

Adopting an Evaluative Stance in Multiple Scientific Source Use In this section, I discuss whether and how the reading practices that scientists have been found to employ can infuse laypeople’s multiple scientific source use. I consider these practices in light of the framework for multiple source use, and the challenges imposed by OSNIT. Considering Authors The scientists, whose reading was reviewed above, attended to information about the scientific articles’ authors, and used this information to decide whether the article was relevant, or to set up expectations about rigor and trustworthiness. In some cases, familiarity with authors and their rigor influenced how carefully scientists attended to methodological details. This process of noting who created an information resource or when it was written, and using this information to regulate reading and judge trustworthiness, is referred to as sourcing (e.g., Stadtler, Bromme, Scharrer, this volume). Sourcing can facilitate constructing the intertext model by providing information for mentally labeling and distinguishing between documents, and it can facilitate constructing the integrated model because it can direct attention to more trustworthy sources (Bråten & Strømsø, 2011; Bråten, Strømsø, & Britt, 2009; Wiley et al., 2009). Information resources can also contain embedded source information (Strømsø, Bråten, Britt, & Ferguson, 2013), such as when authors mention that they are quoting another author, in which case the embedded source information is the name of the quoted author. Attending to embedded source information can help readers discern whether quoted information is more, less, or as credible as the citing source (Strømsø et  al., 2013). Science news articles may not systematically attribute specific statements to a cited source, even if the science news article includes embedded source information. For example, the article may note that recent findings from the lab of Dr. X suggest that pollutants from contaminated fish may be stored in fat cells in the body. The article may go on to state that this means that the pollutants are retained in the body over time, and that people might experience their harmful effects even some time after the actual exposure to the pollutant. However, the text of the article may not explicitly mention whether this second statement was quoted from Dr. X’s study or was an interpretation offered by the journalist. Readers might give different weight to statements made by the scientist conducting the research and to statements made by the journalist, but the ambiguity over the source of the statement prevents readers from differentially weighing information. Laypeople do not engage much in sourcing spontaneously (Barzilai & Zohar, 2012; Mason, Ariasi, & Boldrin, 2011; Wiley et al., 2009). However, following interventions that instructed and prompted participants to engage in sourcing, participants were more likely to do so, and correctly identified more reliable sources than those who did not receive such guidance (Mason, Junyent, & Tornatora, 2014; Wiley et al., 2009). Laypeople are also unlikely to have the prior knowledge needed to engage in the type of sourcing that the scientists displayed. The scientists drew on familiarity with a relatively small community of people who conduct similar research. Laypeople will not be able to judge the credibility of the journalists or reported scientists in this way.

Functional Scientific Literacy  •  229

Working with multiple sources may compensate for limited prior knowledge and enable readers to make relative judgments among sources, especially if there are large differences in the authors’ credentials, such as a medical doctor versus a consumer advocate in the context of health information (Stadtler, Scharrer, Brummernhenrich, & Bromme, 2013). In reading OSNIT, it may be the case that sourcing is more central to keeping track of multiple sources and less relevant to evaluation. Yet, sourcing is effective in eliminating strikingly non-credible sources (e.g., when there are obvious conflicts of interest). Laypeople can assess author credibility based on the level of detail provided in the text and the extent to which the authors signify their own versus the quoted scientists’ perspective. Previewing Another strategy that the scientists used was previewing. They skimmed the abstract, the main graphs and figures, and the results section of the article. This provided them with an initial mental sketch of the claims and evidence of the article. They stated that this helped them maintain an evaluative stance, because they were not distracted in subsequent cycles of reading by the need to figure out the main message of the article. In some ways, a science news report performs the previewing for the reader through the inverted pyramid structure, placing the main findings upfront (Boykoff & Rajan, 2007). However, this can be misleading, and detract from an evaluative stance. Headlines may be composed by the editor, not the reporter, with a stronger emphasis on grabbing attention than on accuracy, and reporters may also be influenced by attention grabbing goals when composing the opening segment of their report (Boykoff & Rajan, 2007). The more detailed segment toward the end of the news report may offer a more tempered account than the opening segment. Given laypeople’s propensity to be influenced by the order in which claims appear (Van Oostendorp et al., 2002), skimming a news report and attending to the more detailed tail end of the report may help in constructing more dynamic models and in maintaining an evaluative stance. Dynamic Non-Linear Reading A characteristic of the scientists’ reading was dynamic and non-linear reading, which was directed at the dual goals of comprehension and evaluation. In the multiple source use framework, such a process can be key to developing the integrated model, and it has characterized deeper comprehension and evaluation of laypeople, especially when their movement within and between texts was purposeful (Anmarkrud et al., 2014; Bråten & Strømsø, 2011; Goldman et al., 2012; Mason et al., 2011). The scientists had particular aims in moving around the text that have not come up explicitly in studies of laypeople. These aims can be realized as a set of coordination moves, in particular: results-methodsconclusions coordination, and text-visual coordination (Falk & Yarden, 2009). Results-Methods-Conclusions Coordination Scientists’ prior knowledge includes representations of different types of scientific explanations and associated methods (Chinn et  al., 2016; Ohlsson, 2002; Tabak &

230  •  Iris Tabak

Reiser, 2008). These representations are used to evaluate whether the parts of a scientific article hang together well, and to what extent the article is reliable and convincing. The scientists whose reading was studied made comments such as “are the effects that should be there, there, and the effects that shouldn’t be there, not there”; they also raised concerns over whether the techniques used to measure were adequate (Bazerman, 1988). These questions drove the back-and-forth reading between results, methods, and conclusions, with one serving to qualify the other. How something was measured will influence the meaning of the results (Falk & Yarden, 2009), as well as the degree of generalization. Author considerations may guide the selection and depth of reading of an article, but these content evaluations determine whether scientists will sanction the findings and conclusions and incorporate them into their own research or professional worldview (Bazerman, 1988). In the case of laypeople’s evaluation of science news, similar results-methodsconclusions coordination and evaluation can enable laypeople to decide whether the conclusions of reported studies are convincing enough to guide their decisions. For example, a study where a smoking-cessation drug was tested as part of an overall intervention that included counseling as well as diet and lifestyle changes might leave open questions as to the role that the drug itself played. If the drug has disturbing side effects, this might dissuade people from choosing the drug. Results-methodsconclusions coordination can also help to distinguish between the reporter’s and the scientist’s voice. If the reporter’s framing seems stronger than the strength of the reported findings, people can recognize this framing as positional writing, which can be subtle and difficult to discern (Manuel, 2002). If they recognize positional writing, readers might modify their integrated model by moderating the tenor of their representation of the report. Results-methods-conclusions coordination can facilitate comparing between sources. A straightforward comparison between different studies (reported in different science news sources) is not always possible. Baselines, measurement criteria, and variable operationalizations need to be considered to determine whether and how to align the studies in comparison. A smoking-cessation drug that appears to yield better outcomes may actually be inferior to another drug, which was tested under stricter definitions of “cessation” (Asher, Nasser, Ganaim, & Tabak, 2010). This type of coordination offers a powerful interpretation tool, but it is contingent on reader knowledge and on sufficient information in the science news report. There is little to suggest that most citizens currently possess the necessary knowledge for these types of sophisticated coordination moves. It might be helpful, though, to distinguish between knowledge of scientific content, such as particular facts and principles, and knowledge of reading practices, scientific information resource genres, and criteria for evaluating scientific knowledge. When laypeople seek scientific information, it will often relate to different topics, requiring a vast scope of relevant knowledge that even scientists do not possess beyond their own fields of specialization. Hence, most likely, content knowledge will always be limited (Bromme & Goldman, 2014). However, knowledge of reading practices and criteria for evaluating scientific knowledge are more limited in scope, and the same criteria apply across many different topics. Therefore, these aspects seem open to educational intervention. In fact, according to the National (U.S.) Science Foundation’s 2016 Science and Engineering

Functional Scientific Literacy  •  231

indicators, although the public’s scientific content knowledge has remained fairly stable over the last decade, the public’s knowledge of how science generates and assesses knowledge has grown (National Science Board, 2016). OSNIT omit much of the information that is needed to make the types of coordination moves described above (Zimmerman et al., 2001), regardless of the fairly consistent accuracy of science news (Holtzman et al., 2005), or the highly variable accuracy of informational texts (Zhang, Sun, & Xie, 2015). OSNIT can omit qualifications and hedges depriving lay audiences of the discourse markers that signal the degrees of certainty that are associated with different claims (Geisler, 1994/2013). For example (Geisler, 1994/2013, pp. 13–14), a text intended for lay audiences contained the statement: “The bees masticate and consume flesh at the feeding site after coating it with an enzyme that breaks it down.” The original text, from a scientific article, signaled that this was not directly observed, but rather inferred based on observing wetness at the feeding site: “appear to hydrolyze it with a secretion produced by either mandibular or salivary glands, which gives the feeding site a wet appearance.” Despite these rhetorical limitations, it is possible to employ more science-specific considerations in evaluating OSNIT, and these appear to offer an advantage in decision making (Lin, 2014). Text-Visual Coordination The scientists’ non-linear reading and results-methods-conclusions coordination included a process of coordinating between text and visual representations such as graphs, tables, and figures. As noted earlier, scientific texts include visuals (e.g., tables, graphs, figures) that convey information that is not duplicated in the text (Falk & Yarden, 2009; Graesser et al., 2002). Most of this visual information depicts research results, but it can also depict the apparatus that was used in the research, as well as additional methods-related information. The scientists tended to look over the visual information before the textual information. Interestingly, there is some evidence that when integrating visual and textual information, processing the visual information first can facilitate comprehension (Verdi & Kulhavy, 2002). Interpreting visual information, and graphs in particular, is challenging, requiring knowledge of the qualities of the representations and of the ways in which they depict disciplinary ideas (Ainsworth, 2008; Schnotz, 2002). It also requires knowledge of how visual features can be manipulated to connote an interpretation that does not align with the data it represents, such as skewing scaling to make differences appear significantly large (Jarman, McClune, Pyle, & Braband, 2012). Integration between text and visuals is even more complex, because the processes involved in visual and textual comprehension are not identical, so it is necessary to coordinate between modes of processing as well as between interpreted content (Schnotz, 2002; Schnotz et al., 2011). It is not surprising that learners tend to ignore visuals and focus on the text (Ainsworth, 2008; Cromley et  al., 2010). However, when learners do attend to visuals they make more inferences and are better able to answer questions about the information than when attending solely to text (Cromley et al., 2010). Multiple source use adds more complexity to these already challenging processes. Readers need to coordinate text and visuals within a source, and coordinate this with

232  •  Iris Tabak

text-visual coordination of other sources. A graph may depict values of measurements, while the text may tell you how data was measured, so the text and the graph need to be coordinated in order to fully understand the results and their limitations (Falk & Yarden, 2009). In scientific articles, different facets of information, such as the methods or the results, are located in labeled conventional sections of the document. OSNIT do not share these conventions, and readers must rely on skimming to locate the relevant information, which may not even be included (see discussion above on the level of detail that is typically included in OSNIT). This coordinated understanding within information resources needs to be used to reason across information resources. However, different information resources might show graphs or other visuals that depict different units, operationalizations (e.g., defining smoking cessation as no smoking for 3 months or for 6 months), or scales, making such cross-resource integration very difficult and dependent on an understanding of the scientific implications of these differences (Asher et al., 2010).

IMPLICATIONS AND FUTURE DIRECTIONS Multiple source use is a mainstay of everyday life in a networked society (Goldman & Scardamalia, 2013). The construction of the mental models that are involved in comprehending multiple sources is, or should be, shaped by discipline-specific strategies (Brand-Gruwel, Wopereis, & Vermetten, 2005; Goldman et al., 2016; Tabak, 2015). In this chapter, I explored the ways in which scientists’ reading practices might inform laypeople’s reading of OSNIT. This analysis is a first step toward articulating an idealized model of the ways in which science-specific considerations can infuse a general model of multiple source comprehension. Work toward articulating this model requires specifying a set of science-specific reasoning strategies used to construct the surface, intertext, integrated, and task models that comprise multiple source comprehension. Additional research is needed to understand the characteristics of other online genres such as blogs (Barzilai & Eshet-Alkalai, 2015) and social media, whether people recognize these genre differences, and whether and how knowledge of genre characteristics influences comprehension within and across information resources of different genres. Another area that is central to comprehension in science and merits greater attention in the study of multiple source use is text-visual integration. There are open questions concerning such a model’s pragmatic value. Many people do not have the knowledge needed to engage in the types of reading strategies described in this chapter, and many science information resources do not include the requisite level of detail. However, this is a moving target, as both science education and science communication are changing in response to growing recognition of the public’s science information needs (Tabak, 2015). The science education community recognizes the interrelationship between literacy and science (National Research Council, 2012; Pearson, Moje, & Greenleaf, 2010). Less clear is how to design the science curricula. Developing curricula requires making choices among alternatives, and prioritizing certain topics or types of activities over others. The skill sets that are relevant to preparing learners for careers in science and for cultivating functional scientific literacy (Feinstein et al., 2013; Ryder, 2001)

Functional Scientific Literacy  •  233

only partially overlap. Which of these two goals should receive higher priority? What should the relative emphasis be on reading, writing, and first-hand investigations (e.g., experimentation)? The analysis in this chapter suggests that functional scientific literacy (Ryder, 2001) can be enhanced through greater contact with the practice of scientists. Science learners can benefit from a range of experiences. Relevant experiences include those that reflect the knowledge production practices of scientists (Duschl & Hamilton, 2011), the literacy practices of scientists (e.g., Yarden et al., 2015), and the use of science communication genres for everyday goals (e.g., Greenleaf, Brown, Goldman, & Ko, 2013; Polman & Hope, 2014). Each of these experiences contributes a facet of knowledge and skills pertinent to science-specific multiple source use. Experience with first-hand investigations equips learners with insight into the interrelationship between methods and results, and the ways in which this interrelationship can qualify conclusions. Experience with the literacy practices of scientists provides greater ability to contend with scientific articles, which are increasingly linked to popular science communications. The hope is that learners who share such experiences will have greater agency in the use of scientific information resources in their daily lives.

AUTHOR’S NOTE I am grateful to my colleagues Michael Weinstock and Sarit Barzilai, as well as colleagues in the Learning in a Networked Society (LINKS ICORE) center, for our many productive discussions.

NOTE 1

Evaluative is used in this chapter to indicate an orientation toward critique and evaluation. It is not suggestive of the evaluativist epistemological level (Kuhn, Cheney, & Weinstock, 2000), though people who hold evaluativist epistemological positions are probably evaluative.

REFERENCES Ainsworth, S. (2008). The educational value of multiple-representations when learning complex scientific concepts. In J. K. Gilbert, M. Reiner, & M. Nakhleh (Eds.), Visualization: Theory and practice in science education (pp. 191–208). Dordrecht: Springer. Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. doi:http://dx.doi.org/10.1016/j.lindif.2013.01.007. Asher, I., Nasser, S., Ganaim, L., & Tabak, I. (2010). Putting the pieces together: The challenge and value of synthesizing disparate graphs in inquiry-based science learning. In K. Gomez, L. Lyons, & J. Radinsky (Eds.), Proceedings of the 9th international conference of the learning sciences: Volume 2 (pp. 340–341). Chicago, IL: International Society of the Learning Sciences. Barzilai, S., & Eshet-Alkalai, Y. (2015). The role of epistemic perspectives in comprehension of multiple author viewpoints. Learning and Instruction, 36, 86–103. doi:http://dx.doi.org/10.1016/j.learninstruc. 2014.12.003. Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online sources. Cognition and Instruction, 30, 39–85. doi:10.1080/07370008.2011.636495. Bazerman, C. (1985). Physicists reading physics. Written Communication, 2, 3–23. doi:10.1177/07410883850 02001001.

234  •  Iris Tabak Bazerman, C. (1988). Shaping written knowledge: The genre and activity of the experimental article in science. Madison, WI: University of Wisconsin Press. Berkenkotter, C., & Huckin, T. (1995). News value in scientific journal articles. In C. Berkenkotter & T. Huckin (Eds.), Genre knowledge in disciplinary communication: Cognition/culture/power (pp. 27–44). Mahwah, NJ: Erlbaum. Blanc, N., Kendeou, P., van den Broek, P., & Brouillet, D. (2008). Updating situation models during reading of news reports: Evidence from empirical data and simulations. Discourse Processes, 45, 103–121. doi:10.1080/01638530701792784. Boykoff, M. T., & Rajan, S. R. (2007). Signals and noise. EMBO Reports, 8, 207–211. Brand-Gruwel, S., Wopereis, I., & Vermetten, Y. (2005). Information problem solving by experts and novices: Analysis of a complex cognitive skill. Computers in Human Behavior, 21, 487–508. doi:http://dx.doi. org/10.1016/j.chb.2004.10.005. Bråten, I., & Strømsø, H. I. (2011). Measuring strategic processing when students read multiple texts. Metacognition and Learning, 6, 111–130. doi:10.1007/s11409-011-9075-7. Bråten, I., Strømsø, H. I., & Britt, M. A. (2009). Trust matters: Examining the role of source evaluation in students’ construction of meaning within and across multiple texts. Reading Research Quarterly, 44, 6–28. doi:10.1598/RRQ.44.1.1. Brewer, W. F., & Chinn, C. A. (1994). Scientists’ responses to anomalous data: Evidence from psychology, history, and philosophy of science. In D. L. Hull, M. Forbes, & R. M. Burian (Eds.), Psa 1994: Proceedings of the 1994 biennial meeting of the philosophy of science association. Volume one: Contributed papers (pp. 304–313). East Lansing, MI: Philosophy of Science Association. Britt, M. A., Richter, T., & Rouet, J.-F. (2014). Scientific literacy: The role of goal-directed reading and evaluation in understanding scientific information. Educational Psychologist, 49, 104–122. doi:10.1080/0046152 0.2014.916217. Bromme, R., & Goldman, S. R. (2014). The public’s bounded understanding of science. Educational Psychologist, 49, 59–69. doi:10.1080/00461520.2014.921572. Brossard, D., & Scheufele, D. A. (2013). Science, new media, and the public. Science, 339, 40–41. doi:10.1126/ science.1232329. Charney, D. (1993). A study in rhetorical reading: How evolutionists read “the spandrels of San Marco”. In J. Selzer (Ed.), Understanding scientific prose (pp. 203–231). Madison, WI: University of Wisconsin Press. Chinn, C. A., Rinehart, R. W., & Greene (2016). Epistemic cognition and philosophy: Developing a new framework for epistemic cognition. In J. A. Greene, W. A. Sandoval, & I. Bråten (Eds.), Handbook of epistemic cognition (pp. 460–478). New York, NY: Routledge. Cho, B. Y., & Afflerbach, P. (2015). Reading on the internet. Journal of Adolescent & Adult Literacy, 58, 504–517. doi:10.1002/jaal.387. Cromley, J. G., Snyder-Hogan, L. E., & Luciw-Dubas, U. A. (2010). Cognitive activities in complex science text and diagrams. Contemporary Educational Psychology, 35, 59–74. doi:http://dx.doi.org/10.1016/j. cedpsych.2009.10.002. Dunbar, K. (1997). How scientists think: On-line creativity and conceptual change in science. In T. B. Ward, S. M. Smith, & J. Vaid (Eds.), Conceptual structures and processes: Emergence, discovery, and change. Washington D.C: American Psychological Association Press. Duschl, R. A., & Hamilton, R. (2011). Learning science. In R. E. Mayer & P. A. Alexander (Eds.), Handbook of research on learning and instruction (pp. 78–107). New York, NY: Routledge. Falk, H., & Yarden, A. (2009). “Here the scientists explain what I said”: Coordination practices elicited during the enactment of the Results and Discussion sections of adapted primary literature. Research in Science Education, 39, 349–383. doi:10.1007/s11165-008-9114-9. Feinstein, N. W., Allen, S., & Jenkins, E. (2013). Outside the pipeline: Reimagining science education for nonscientists. Science, 340, 314–317. doi:10.1126/science.1230855. Fox, E. (2009). The role of reader characteristics in processing and learning from informational text. Review of Educational Research, 79, 197–261. doi:10.3102/0034654308324654. Geisler, C. (1994/2013). Academic literacy and the nature of expertise: Reading, writing, and knowing in academic philosophy. Published online. Routledge. Retrieved from https://books.google.co.il/books?id=HUD8AQA AQBAJ&lpg=PR4&hl=iw&pg=PR1#v=onepage&q&f=false (reprinted from Mahwah, NJ: Erlbaum, 1994). Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381. doi:10.1002/rrq.027.

Functional Scientific Literacy  •  235 Goldman, S. R., Britt, M. A., Brown, W., Cribb, G., George, M., Greenleaf, C., . . . Project READI (2016). Disciplinary literacies and learning to read for understanding: A conceptual framework for disciplinary literacy. Educational Psychologist, 51, 219–246. doi:10.1080/00461520.2016.1168741. Goldman, S. R., & Scardamalia, M. (2013). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities. Cognition and Instruction, 31, 255–269. doi:10.1080/10824669.2013.773217. Graesser, A. C., León, J. A., & Otero, J. (2002). Introduction to the psychology of science text comprehension. In J. Otero, J. A. León, & A. C. Graesser (Eds.), The psychology of science text comprehension (pp. 1–15). Mahwah, NJ: Erlbaum. Grant, L., Hausman, B. L., Cashion, M., Lucchesi, N., Patel, K., & Roberts, J. (2015). Vaccination persuasion online: A qualitative study of two provaccine and two vaccine-skeptical websites. Journal of Medical Internet Research, 17, e133. doi:10.2196/jmir.4153. Greenleaf, C., Brown, W., Goldman, S. R., & Ko, M.-L. (2013). READI for science: Promoting scientific literacy practices through text-based investigations for middle and high school science teachers and students. Paper presented at the Workshop on Literacy for Science. Washington, DC: National Research Council. Haas, C. (1994). Learning to read biology: One student’s rhetorical development in college. Written Communication, 11, 43–84. doi:10.1177/0741088394011001004. Holtzman, N. A., Bernhardt, B. A., Mountcastle-Shah, E., Rodgers, J. E., Tambor, E., & Geller, G. (2005). The quality of media reports on discoveries related to human genetic diseases. Public Health Genomics, 8, 133–144. Jarman, R., McClune, B., Pyle, E., & Braband, G. (2012). The critical reading of the images associated with science-related news reports: Establishing a knowledge, skills, and attitudes framework. International Journal of Science Education, Part B, 2, 103–129. doi:10.1080/21548455.2011.559961. Kenski, K. (2017). Overcoming confirmation and blind spot biases when communicating science. In K. H. Jamieson, D. Kahan, & D. Scheufele (Eds.), The Oxford handbook of the science of science communication (pp. 369–376). New York, NY: Oxford University Press. Koslowski, B. (2013). Scientific reasoning: Explanation, confirmation bias, and scientific practice. In G. J. Feist & M. E. Gorman (Eds.), Handbook of the psychology of science (pp. 151–192). New York, NY: Springer. Kuhn, D., Cheney, R., & Weinstock, M. (2000). The development of epistemological understanding. Cognitive Development, 15, 309–328. doi:http://dx.doi.org/10.1016/S0885-2014(00)00030-7. Ladwig, P., Dalrymple, K. E., Brossard, D., Scheufele, D. A., & Corley, E. A. (2012). Perceived familiarity or factual knowledge? Comparing operationalizations of scientific understanding. Science and Public Policy, 39, 761–774. doi:10.1093/scipol/scs048. Latour, B. (1981). Is it possible to reconstruct the research process?: Sociology of a brain peptide. In K. D. Knorr, R. Krohn, & R. Whitley (Eds.), The social process of scientific investigation (pp.  53–73). Dordrecht, The Netherlands: Springer. Lin, S.-S. (2014). Science and non-science undergraduate students’ critical thinking and argumentation performance in reading a science news report. International Journal of Science and Mathematics Education, 12, 1023–1046. doi:10.1007/s10763-013-9451-7. Maier, J., & Richter, T. (2013). How nonexperts understand conflicting information on social science issues. Journal of Media Psychology, 25, 14–26. doi:10.1027/1864-1105/a000078. Manuel, K. (2002). How first-year college students read popular science: An experiment in teaching media literacy skills. SIMILE: Studies In Media & Information Literacy Education, 2, 1–12. doi:10.3138/sim.2.2.002. Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action: Spontaneous reflections about knowledge and knowing during online information searching and their influence on learning. Learning and Instruction, 21, 137–151. doi:http://dx.doi.org/10.1016/j.learninstruc.2010.01.001. Mason, L., Junyent, A. A., & Tornatora, M. C. (2014). Epistemic evaluation and comprehension of web-source information on controversial science-related topics: Effects of a short-term instructional intervention. Computers & Education, 76, 143–157. doi:http://dx.doi.org/10.1016/j.compedu.2014.03.016. Mercier, H., & Heintz, C. (2014). Scientists’ argumentative reasoning. Topoi, 33, 513–524. National Research Council. (2012). A framework for k-12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press. National Science Board. (2016). Science and engineering indicators 2016 (NSB-2016-1). Arlington, VA: National Science Foundation. Neuwirth, C. M., & Contijoch, A. M. (2003). Scientists’ reading and writing processes: Implications for electronic journal design. Science Editor, 26, 185.

236  •  Iris Tabak Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2, 175–220. doi:10.1037/1089-2680.2.2.175. Ohlsson, S. (2002). Understanding qualitative explanations. In J. Otero, J. A. León, & A. C. Graesser (Eds.), The psychology of science text comprehension (pp. 91–123). Mahwah, NJ: Erlbaum. O’Neill, D. K. (2001). Knowing when you’ve brought them in: Scientific genre knowledge and communities of practice. Journal of the Learning Sciences, 10, 223–264. doi:10.1207/S15327809JLS1003_1. Paige, S. R. (2014). Physical activity informational websites: Accuracy, language ease, and fear appeal (Master of Public Health Thesis). Purdue University, West Lafayette, IN. Retrieved from ProQuest Dissertations & Theses A&I database. (1565104). Pearson, P. D., Moje, E., & Greenleaf, C. (2010). Literacy and science: Each in the service of the other. Science, 328, 459–463. doi:10.1126/science.1182595. Polman, J. L., & Hope, J. M. G. (2014). Science news stories as boundary objects affecting engagement with science. Journal of Research in Science Teaching, 51, 315–341. doi:10.1002/tea.21144. Ryder, J. (2001). Identifying science understanding for functional scientific literacy. Studies in Science Education, 36, 1–44. doi:10.1080/03057260108560166. Schäfer, M. S. (2017). How changing media structures are affecting science news coverage. In K. H. Jamieson, D. Kahan, & D. Scheufele (Eds.), The Oxford handbook of the science of science communication (pp. 51–60). New York, NY: Oxford University Press. Schnotz, W. (2002). Commentary: Towards an integrated view of learning from text and visual displays. Educational Psychology Review, 14, 101–120. doi:10.1023/a:1013136727916. Schnotz, W., Ullrich, M., Hochpochler, U., Horz, H., McElvany, N., Schroeder, S., & Baumert, J. (2011). What makes text-picture-integration difficult? A structural and procedural analysis of textbook requirements. Ricerche di Psicologia, 1, 103–135. Shah, P., & Hoeffner, J. (2002). Review of graph comprehension research: Implications for instruction. Educational Psychology Review, 14, 47–69. doi:10.1023/a:1013180410169. Shanahan, C., Shanahan, T., & Misischia, C. (2011). Analysis of expert readers in three disciplines: History, mathematics, and chemistry. Journal of Literacy Research, 43, 393–429. doi:10.1177/1086296x11424071. Shtulman, A., & Harrington, K. (2016). Tensions between science and intuition across the lifespan. Topics in Cognitive Science, 8, 118–137. doi:10.1111/tops.12174. Smith, J. M., & Robertson, M. K. (2016). Going beyond text features in informational text: It’s more than just a table of contents and an index. In E. Martinez & J. Pilgrim (Eds.), Literacy summit yearbook volume 2: October 2016 (pp. 32–37). San Antonio, TX: Specialized Literacy Professionals and Texas Association for Literacy Education. Stadtler, M., Scharrer, L., Brummernhenrich, B., & Bromme, R. (2013). Dealing with uncertainty: Readers’ memory for and use of conflicting information from science texts as function of presentation format and source expertise. Cognition and Instruction, 31, 130–150. doi:10.1080/07370008.2013.769996. Strømsø, H. I., Bråten, I., Britt, M. A., & Ferguson, L. E. (2013). Spontaneous sourcing among students reading multiple documents. Cognition and Instruction, 31, 176–203. doi:10.1080/07370008.2013.769994. Sutherland, W. J., Spiegelhalter, D., & Burgman, M. A. (2013). Twenty tips for interpreting scientific claims. Nature, 503, 335–337. Tabak, I. (2015). Functional scientific literacy: Seeing the science within the words and across the web. In L. Corno & E. M. Anderman (Eds.), Handbook of educational psychology: 3rd edition (pp.  269–280). London, UK: Routledge. Tabak, I., & Reiser, B. J. (2008). Software-realized inquiry support for cultivating a disciplinary stance. Pragmatics & Cognition, 16, 307–355. Van Gorp, B., Rommes, E., & Emons, P. (2014). From the wizard to the doubter: Prototypes of scientists and engineers in fiction and non-fiction media aimed at Dutch children and teenagers. Public Understanding of Science, 23, 646–659. doi:10.1177/0963662512468566. Van Oostendorp, H., Otero, J., Leon, J., & Graesser, A. (2002). Updating mental representations during reading scientific text. In J. Otero, J. A. León, & A. C. Graesser (Eds.), The psychology of science text comprehension (pp. 309–329). Mahwah, NJ: Erlbaum. Verdi, M. P., & Kulhavy, R. W. (2002). Learning with maps and texts: An overview. Educational Psychology Review, 14, 27–46. doi:10.1023/a:1013128426099. Ward, P. R., Henderson, J., Coveney, J., & Meyer, S. (2012). How do south Australian consumers negotiate and respond to information in the media about food and nutrition?: The importance of risk, trust and uncertainty. Journal of Sociology, 48, 23–41. doi:10.1177/1440783311407947.

Functional Scientific Literacy  •  237 Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009). Source evaluation, comprehension, and learning in internet science inquiry tasks. American Educational Research Journal, 46, 1060–1106. doi:10.3102/0002831209333183. Yang, F.-Y., Chen, Y.-H., & Tsai, M.-J. (2013). How university students evaluate online information about a socio-scientific issue and the relationship with their epistemic beliefs. Journal of Educational Technology & Society, 16, 385–399. Yarden, A., Norris, S. P., & Phillips, L. M. (2015). Adapted primary literature: The use of authentic scientific texts in secondary schools. Dordrecht, The Netherlands: Springer. Zhang, Y., Sun, Y., & Xie, B. (2015). Quality of health information for consumers on the web: A systematic review of indicators, criteria, tools, and evaluation results. Journal of the Association for Information Science & Technology, 66, 2071–2084. doi:10.1002/asi.23311. Zimmerman, C., Bisanz, G. L., Bisanz, J., Klein, J. S., & Klein, P. (2001). Science at the supermarket: A comparison of what appears in the popular press, experts’ advice to readers, and what students want to know. Public Understanding of Science, 10, 37–58. doi:10.1088/0963-6625/10/1/303.

14

THE ROLE OF SOURCING IN MATHEMATICS Keith Weber rutgers university, usa

HOW DO WE KNOW WHICH MATHEMATICAL STATEMENTS ARE TRUE? Some scholars will find it surprising that there is a chapter on mathematics in a handbook on multiple source use. The standard view of mathematics is that mathematical results are objective, infallible, and incorrigible. Mathematical results ostensibly have these attributes because they are established by deductive justifications or proofs. These deductive justifications are based on cold, impersonal logic that would be convincing to any knowledgeable mathematician. Why, then, would a mathematician ever need to consider the source of a justification when deciding how much evidentiary weight that the justification should have? The mathematician can just check the logic of the justification herself. Indeed, as Shanahan, Shanahan, and Misischia (2011) documented, some mathematicians claim that they actively ignore the source of a justification when evaluating it. The author of a justification should be irrelevant to the logical validity of the justification, which is the sole source of reliability for mathematical claims. This chapter has three sections. In the first section, I explain why mathematicians need to engage in sourcing. Although the standard viewpoint of mathematics that I described above has an important kernel of truth, it is an oversimplification of how mathematicians actually practice their craft. I explain why mathematicians often do not rely on their own reasoning to check the correctness of a justification and they sometimes consider an argument’s source when assessing its persuasiveness. In the second section, I describe four common sources that mathematicians consider when estimating the likelihood that a justification is valid. These sources include who wrote the justification, whether the justification was published, where it was published, and the importance of the result. In the third section, I discuss the broader implications of mathematicians’ use of sources. In particular, I identify similarities

238

The Role of Sourcing in Mathematics  •  239

and differences between the ways in which mathematicians and scholars from other disciplines use sources to evaluate disciplinary claims. I also discuss implications for the teaching of mathematics. Terminology Used in this Chapter I begin by defining central terms that I will use throughout this chapter. A mathematical statement is an assertion about mathematical objects that permits a truth-value. For instance, “The sum of two even numbers is an even number” is a true mathematical statement. If mathematicians are unsure if a mathematical statement is true or false, they refer to that statement as a conjecture. A proof of a mathematical statement is a particular type of justification in favor of a mathematical statement. A proof is a chain of statements where each statement either is known to be true or is a logically necessary consequence of previous statements. A proof concludes with the mathematical statement being proven. Proofs play a central role in mathematical practice. With rare exceptions, the mathematical community accepts a mathematical statement as true exactly when a mathematician produces an acceptable proof of that statement. A mathematical statement affirmed by a proof is called a theorem. Proving is thus the process by which conjectures become theorems. Much of the professional work of pure mathematicians involves formulating interesting conjectures, proving these conjectures, and then publishing these proofs in mathematical journals. Most papers that are published in pure mathematics journals are largely composed of proofs of conjectures (e.g., Kuteeva & McGrath, 2015). I say a mathematician’s confidence in a mathematical statement is the subjective probability that she would assign to that statement being true and a mathematician’s confidence in a proof is the subjective probability that she would assign to a proof being correct and not containing a mistake. Finally, I define the source of a theorem and proof as information about the publishing process that is independent of the mathematical content of the theorem and the proof. In particular, in this chapter, sources include who wrote a mathematical paper, which journal chose to publish the paper, and when the paper was published. I say that a mathematician is engaging in sourcing if she uses information pertaining to the source of a statement or proof to increase her confidence in that statement or proof. Key aims of this chapter include establishing that mathematicians engage in sourcing and describing how the source of a theorem and proof affects mathematicians’ evaluation of the theorem and proof. Purported Virtues of Mathematical Knowledge There is a widespread belief that due to mathematicians’ emphasis on proof, mathematical knowledge has special virtues that distinguish it from other types of scientific knowledge (Berry, in press; for a critical discussion, see Buldt, Löwe, & Müller, 2008). In this section, I describe two purported virtues of mathematical knowledge—reliability and autonomy—and discuss how each virtue reduces the need for mathematicians to engage in sourcing when evaluating mathematical statements or proofs.

240  •  Keith Weber

Many scholars claim that proven mathematical statements are reliable or always true (Berry, in press). This is because a proof shows how a theorem is a logically necessary consequence of things that are known to be true. In other words, a proof does not just transform a conjecture into an accepted theorem but also into an indubitable fact (e.g., Harel & Sowder, 1998). In other disciplines, all knowledge claims are regarded as intrinsically tentative. Scholars often have to balance conflicting evidence in deciding what to believe, which requires these scholars to consider the reliability of the evidence that they are given. These judgments about reliability often involve considering the source of that evidence (e.g., Wineburg, 1991). Mathematics is different in that a proof of a theorem purportedly guarantees the theorem is true. Hence there is no need to consider alternative evidence in favor of or against a mathematical statement after a proof of that statement is presented because a proof is definitive. Mathematical knowledge is also said to have the virtue of autonomy, meaning that any capable mathematician can convince herself that a published theorem is true by checking that its associated proof is correct. Thus, a mathematician never needs to rely on the testimony of another mathematician to decide what to believe (Berry, in press). Or more generally, a mathematician never needs to engage in sourcing because all that is needed to obtain certainty is contained in the published proof. Information such as who wrote the proof or where the proof was published is irrelevant in deciding whether the proof is correct. Indeed, some scholars go further and claim that it is irresponsible for a mathematician to use a theorem without verifying the proof of the theorem herself (e.g., Berry, in press; Biss, 2004). In this sense, mathematical knowledge differs from scientific knowledge that is based on the collection of empirical data. While an empirical scientific paper typically describes how data was collected and analyzed, the reader cannot directly witness the process of data collection. Hence, believing a scientific paper requires trusting the testimony of those who collected the data (e.g., Chinn, Buckland, & Samarapungavan, 2011). In contemporary science, the accurate collection of data often requires substantial expertise and the risk of an error in the data collection process is not negligible. Furthermore, cost and technology often prohibit individual readers from replicating empirical studies. For these reasons, some scientists engage in sourcing as a heuristic to estimate how much trust to put in scientific papers; they consider factors such as the reputation of the research team for doing reliable work and the institution where the research was conducted (Shanahan et al., 2011). However, this is purportedly not necessary in mathematics. In summary, many scholars believe that the reliability and autonomy of mathematical knowledge entail that mathematicians do not need to consider the source of an argument. The truth of mathematical theorems relies solely on the mathematics and logic in proofs, both of which are available for verification by any qualified mathematician. Some scholars go further and claim that mathematicians should not and do not engage in sourcing. A More Nuanced Perspective of Mathematical Knowledge I believe that the previous sub-section is an oversimplification of mathematical practice. My position is that (i) mathematical knowledge is not completely reliable because mathematicians are fallible and some of their proofs contain errors; (ii) mathematical knowledge is not autonomous because mathematicians often

The Role of Sourcing in Mathematics  •  241

lack the background knowledge to comprehend proofs from outside their areas of expertise; and (iii) mathematicians frequently lack the time or motivation to check a published proof of a theorem and act on the assumption that published results are correct. As a consequence of these three points, mathematicians frequently need to engage in sourcing and they routinely do so. I elaborate on each of these three points below. Regarding (i): for the sake of argument, I will suppose that a correct proof of a theorem guarantees that a theorem is correct.1 I nonetheless maintain that mathematicians will sometimes retain doubt in statements for which there is a purported proof (Paseau, 2011; Weber & Mejia-Ramos, 2015). Paseau (2011) made this point simply as follows: That we are in possession of a proof of p does not imply we should be certain of p . . . The proof may be long and hard to follow, so that any flesh-and-blood mathematician should assign a non-zero probability to its being invalid. The longer and more complex the proof, the less secure its conclusions. (p. 143) That mathematicians cannot be certain that a proof is correct is not merely a philosophical concern. Some scholars have claimed that incorrect proofs are common in the mathematics literature (e.g., De Millo, Lipton, & Perlis, 1979; Krantz, 2011). Consequently, there are many cases in which a proof is not definitive. Regarding (ii): Like many areas of study, mathematics is a fragmented discipline. Following proofs in various sub-disciplines often requires a great deal of background knowledge that most mathematicians outside of that sub-discipline lack (e.g., Auslander, 2008; Thurston, 1994). Thus, the typical mathematician cannot independently verify many published proofs. Therefore not all of the mathematical knowledge a mathematician needs is autonomous. Hence, using others’ results sometimes requires accepting these results as correct on the testimony of the author and the other mathematicians who sanctioned the proof. Regarding (iii): Checking a proof is a time-consuming process that can take months or even years. Hence even in cases where a mathematician could theoretically scrutinize the proofs of every result she wishes to use, it would be practically difficult to do so while still having the time to pursue her own research agenda (Auslander, 2008). Central Questions and Caveats In this chapter, I address the following two questions: If mathematical knowledge is not completely reliable and autonomous, how do mathematicians decide which results to trust? How do mathematicians increase their confidence that a proof of a theorem does not contain an error? The main argument in this chapter is one way that mathematicians increase their confidence in proofs and statements is to engage in sourcing.2 Before proceeding, I highlight two important limitations to research on mathematicians’ sourcing. First, as Shanahan et  al. (2011) and Weber and Mejia-Ramos (2013) have emphasized, empirical and sociological studies on how mathematicians read mathematics and choose which results to trust are sparse. Buldt et  al. (2008)

242  •  Keith Weber

hypothesized that the prevalence of the naïve view of mathematics described above has dissuaded many scholars from doing this type of research on mathematicians’ practices. If mathematical practice is predominantly based on the dispassionate application of impersonal logic, it makes little sense to study the sociology of mathematics or how mathematicians consider sources of theorems and proofs. Second, mathematical practice is heterogeneous. Mathematicians vary substantially in how they practice their craft (e.g., Weber, Inglis, & Mejia-Ramos, 2014). Thus, the reflections of individual mathematicians often do not generalize to the larger population of mathematicians. This is especially problematic, as large-scale studies on mathematicians’ practice are uncommon. Consequently our understanding of mathematical practice is based predominantly on the reflections of individual mathematicians (e.g., Auslander, 2008; Thurston, 1994) and small-scale qualitative studies of individual mathematicians (e.g. Shanahan et al., 2011), which can have limited generalizability. In this chapter, I will describe the ways in which different source information can increase mathematicians’ confidence in a mathematical statement or a mathematical proof. Specifically, I will claim that mathematicians have greater confidence in proofs that are published in an academic journal, especially if these proofs are published in prestigious journals, and that mathematicians have greater confidence in proofs that were written by an authoritative source with a reputation for producing reliable work. I will discuss how each of these claims is supported by empirical studies. Next I will suggest that mathematicians use the reputation of the author in deciding which papers to read or review and that “classical results” in the mathematical canon are accepted as unimpeachable. These last two claims are largely based on mathematicians’ and philosophers’ reflections. I believe these claims are accurate, but caution the reader that they are not based on systematic empirical research. Hence, I encourage the reader to view these claims as hypotheses that should be tested with subsequent empirical studies.

SOURCING IN MATHEMATICAL PRACTICE Publication Status Was the Result Published in an Academic Journal? In contemporary mathematical practice, mathematicians have two primary means of disseminating their written mathematical contributions. The first is to self-publish a paper in a publicly accessible repository.3 The second is to publish the paper in an academic journal that requires that the paper undergo peer review. Mathematicians will often disseminate their written papers in both venues. However, some papers that are published in public repositories do not appear in mathematical journals. In some cases, mathematicians do so because the papers have been submitted to a journal and are under review but mathematicians want to share their results or “claim credit” before the review process is complete. In other cases, mathematicians feel that their results were interesting enough to share with their colleagues but not important enough to warrant journal space. Mathematicians have greater confidence in papers that underwent a peer-review process and were published in journals. Auslander (2008) presented what I consider to be standard mathematical practice in this regard:

The Role of Sourcing in Mathematics  •  243

Usually, certification of a result is a consequence of its appearance in a refereed journal. In this case, it is generally accepted that the “burden of proof” (the pun is inevitable) has shifted and the result is presumed correct, unless there is compelling reason to believe otherwise. (p. 65) Several empirical studies support the claim that mathematicians trust the published literature. Mejia-Ramos and Weber (2014) surveyed 118 research active mathematicians on how they read proofs published in the literature. The mathematicians were asked to rate their agreement to the following statement on a five-point Likert scale on whether it was common for them to have confidence that a proof is correct because it is published in an academic journal. Most participants (72%) agreed and few participants (12%) disagreed (Mejia-Ramos & Weber, 2014). Heinze (2010) found similar results in a survey with 40 mathematicians. On average, the mathematicians claimed they frequently accepted a proof if it was in a peer-reviewed journal. Across two separate studies, my colleague and I interviewed 17 mathematicians about how they read published proofs (Weber, 2008; Weber & Mejia-Ramos, 2011). Many participants commented that they usually did not bother to check published proofs for correctness, but simply assumed that they were. For instance, consider the following transcript below: Interviewer:

One of the things that you didn’t say was you would read [a published proof] to be sure the theorem was true. Is that because it was too obvious to say or is that not why you read the proof? Mathematician:  Well, it depends. If it’s something in the published literature then . . . I’ve certainly encountered mistakes in the published literature, but it’s not high on my mind. So, in other words, I’m open to the possibility that there’s a mistake in the proof, but it’s not [pause] I: But you act on the assumption that it’s probably correct? M: Yeah, that’s right. That’s right. (Weber & Mejia-Ramos, 2011, p. 334) I see two reasons why mathematicians may place more stock in a published result. First, a published paper ordinarily survived a review process in which at least one knowledgeable referee endorsed the proofs in the paper as correct (c.f., Krantz, 2011; but see Nathanson, 2008, and Frans & Kosolosky, 2014, who question whether mathematicians’ faith in the review process is warranted). Second, and perhaps more importantly, results published in journals are more widely read than proofs published in other outlets. This public exposure provides the mathematical community opportunity to locate a mistake in a proof, should one exist (Berry, in press). The confidence that mathematicians place in published results is of paramount importance to mathematical practice. Many mathematicians feel free to apply published theorems without reading their proofs on the grounds that other mathematicians verified the proofs (Auslander, 2008; Geist, Löwe, & Van Kerkhove, 2010). Where Was the Result Published? Mathematicians have greater confidence in results that were published in higher quality journals. Mejia-Ramos and Weber (2014) asked 118 mathematicians if the

244  •  Keith Weber

quality of a journal increased their confidence that a proof was correct. Most (67%) agreed; few (17%) disagreed. I propose at least two reasons for why this might be the case that parallel those in the sub-section above. First, more prestigious journals presumably employ better referees who might be better able to identify errors in the papers that they are reviewing. For instance, Krantz (2011) suggested that if a paper is published in a top-ranked journal, “one may infer that a solid referee with strict standards agreed” that the proofs in the paper were correct (p. 215). Second, proofs published in prestigious outlets are likely to be read by a wider mathematical audience, meaning any errors would be more likely to be identified. The Author’s Reputation The Author’s Reputation Influences How Mathematicians Read and Evaluate a Proof Mathematicians are more likely to believe that a mathematical statement is true when it was proven by a famous mathematician, particularly a famous mathematician with a reputation for doing reliable work. Inglis and Mejia-Ramos (2009) asked 190 mathematicians to read an argument and rate how persuasive they found it on a scale of 1 through 100. Half of the participants were randomly assigned to the “Named Group” and were told that J.E. Littlewood, an eminent mathematician, wrote the argument. The other half of the participants were assigned to the “Anonymous Group” and were not told who wrote the argument. The Named Group found the argument to be significantly more persuasive than the Anonymous Group and judged the argument to be 17 points higher in terms of persuasiveness. These results are consistent with the results from Mejia-Ramos and Weber’s (2014) survey. We asked 118 mathematicians if it was common for them to be very confident that a published result was correct because it was written by an authoritative source that they trusted; 83% agreed and 7% disagreed. Mathematicians consider the reputation of the author not only when reading a published proof. Some mathematicians also do so when refereeing a paper and deciding whether a paper warrants publication. Papers submitted to mathematical journals typically undergo a single-blind review in which the referee knows who wrote the paper, but not vice versa (Frans & Kosolosky, 2014). I contend that this process is in place because the identity of the author of a paper provides useful information to the referee. For instance, consider Müller-Hill (2010), who interviewed mathematicians about how they refereed proofs and reported one mathematician saying: Let’s say a famous mathematician comes up with a paper and I have to referee it. Then I am preoccupied with the fact that he is a famous mathematician and so that it will probably be correct. And then you say, “Yes, this really seems plausible, but I’m not really sure if it is true” and then you end up with the question, “Is this because I don’t have enough knowledge?” (Müller-Hill, 2010, as cited and translated by Geist et al., 2010, p. 162) In our interview studies with mathematicians, no participant said that when refereeing, she would recommend accepting a proof based on who wrote the paper when

The Role of Sourcing in Mathematics  •  245

she was unsure of its correctness. However, some of our participants reported reading a paper differently depending on who wrote it. In particular, they indicated that they would be quicker to dismiss a proof as incorrect if was written by someone that they did not trust (Weber, 2008). In Mejia-Ramos and Weber (2014), we surveyed 54 mathematicians who had experience refereeing papers. We asked the participants if they would be confident that a proof in a paper that they were refereeing was correct because it came from an author that they trusted. Their response was equivocal: 39% agreed; 41% disagreed.4 We can compare this to an earlier result that we reported. Most mathematicians said they would consider the reliability of the author in deciding whether a published proof was correct, but less than half said they would do so when refereeing a paper that was submitted for publication. Hence I cannot claim that most mathematicians consider who wrote a paper when refereeing, but I observe that a substantial minority said they would do so. The Author’s Reputation Influences if Mathematicians Read and Evaluate a Proof Mathematics is often viewed as a discipline satisfying an egalitarian ideal in which anyone can receive credit for solving a problem or proving a theorem, provided that her work is correct. For instance, consider the comment of Ed Dubinsky (1997), a mathematician, responding to a critique that mathematics was “oppressive.” There are no doubt some examples of suppression that have occurred in mathematics, but I think that this term is probably less applicable to mathematics than just about any other field of human endeavor. I know of no other field that openly lists challenges to its practitioners that, however “narrow” one might find them, make the promise that anyone who solves the problems will be fully recognized, no matter what other attributes they have. I think that the record of mathematics in keeping these promises is not too bad. (p. 89) Dubinksy’s passage contains an important kernel of truth as mathematicians hold the value that the validity of a proof is independent of its author (although, as I previously illustrated, the identity of the author may influence how a mathematician decides if a proof is valid). However, the claim that anyone who solves a problem will receive their due recognition is not completely accurate. If an amateur mathematician with no reputation claimed to prove an important conjecture and sent her proof to a highly prestigious journal, she probably would not have her paper reviewed. In his advice to amateur mathematicians, the mathematician Henry Cohn wrote: If you appear out of nowhere claiming to have solved a famous open problem, nobody will pay any attention. In principle you might be right, but many people claim to have done this and virtually all of them are wrong. If you want anyone to take your work seriously, you need to develop a track record that separates you from the cranks . . . It’s ridiculous to argue that all proposed solutions to famous problems must either be accepted as true or be refuted to the satisfaction of the author. The mathematical community couldn’t function under such a constraint.5

246  •  Keith Weber

Scott Aronson, a computer scientist and mathematician, made a similar point. He said that “to avoid spending his life reading the work of crackpots,” he had to assess the plausibility that the proofs in an unpublished paper were correct, which includes considering the author of the paper: In deciding whether to spend time on a paper, obviously the identity of a paper plays some role. If Razborov [an esteemed mathematician] says he proved a superlinear circuit bound for SAT [an important mathematical problem in computer science], the claim on our attention is different than if Roofus McLoofus said the same thing.6 In some unusual cases, even prominent mathematicians have difficulty getting the mathematical community to review their work. This can occur if a mathematician has a reputation for submitting proofs that contain mistakes and the new proof that she wishes to have reviewed is lengthy. Most individual mathematicians will wager that it is not worth investing the time to scrutinize a 300-page proof if they do not think that the proof is likely to be correct. For high-profile instances of such occurrences, see Krantz (2011, chapter 10.2) and Sabbagh (2004). Increased Trust in Landmark Results Scholars studying the sociology of mathematics differ as to whether the published mathematics literature is reliable. Some researchers take a skeptical view of the published literature in mathematics. In an influential paper, De Millo et al. (1979) questioned the claim that proven statements are guaranteed to be true. They write that amongst the hundreds of thousands of proven theorems that are published annually, “a number of these [theorems] are subsequently contradicted or otherwise disallowed, others are thrown into doubt, and most are ignored. Only a tiny fraction come to be understood or believed by any sizable number of mathematicians” (p. 272). Other scholars view the literature as very highly reliable to the point that one can rationally bestow near certainty in a proven mathematical result that has been published. For instance, Berry (in press) claimed that, “natural scientists and others can take mathematical results ‘off the shelf’ without worrying about their veracity. They can regard the results they find in mathematics journals and textbooks as true simpliciter” (italics in original). Those advancing this position often support their beliefs by noting that many mathematical statements proven in antiquity are still accepted as true today, a phenomenon that is rare in other scientific disciplines. For instance, Berry (in press) references Euclid’s 2300-year-old proof that there are infinitely many prime numbers: It is unreasonable to claim that there might still be errors in every single discourse purporting to present what we surely all believe is a genuine proof of Euclid showing that there is an infinity of primes. Too many thinkers have internalised the proof and come up with their own novel presentations . . . [Denying the reliability of this proof] gives expression only to a rather extreme form of skepticism. I believe the inconsistency above can be resolved by recognizing that published mathematical theorems and proofs do not all undergo the same sociological process.

The Role of Sourcing in Mathematics  •  247

The large majority of published theorems are checked by a single referee (who might not read the proof that closely) and then read by at most a handful of specialists who work in that area. Therefore De Millo et al. (1979) and others (e.g., Nathanson, 2008) are right to question whether the published mathematics literature in its entirety is reliable. However, a relatively small percentage of published results, such as Euclid’s proof on the infinitude of primes, are closely scrutinized by a large number of mathematicians. When this occurs, Berry (in press) is right to say those who doubt these results are being overly skeptical. There are at least two types of important theorems that received an increased level of scrutiny. I refer to the first as “landmark results,” proofs of long-standing and famous conjectures. Mathematicians who succeed in proving such important results receive increased status in their fields, and often prizes and monetary rewards as well. Naturally, other mathematicians will read this mathematician’s paper carefully, both to verify that the mathematician actually succeeded in proving the result and to learn from the way this mathematician approached the problem. Consequently, if several years have passed since a landmark result has been proven, many mathematicians will be highly confident that the proof is correct. They may assume that since many mathematicians have inspected the proof carefully and were unable to find an error, an error is unlikely to exist. This is consistent with Heinze’s (2010) survey with 40 mathematicians, in which his participants claimed to frequently accept a published proof if it was in print for a long time and no contradiction was found. For an interesting case study, consider Jackson’s (2006) analysis of the sociology of Grigori Perelman’s proof of the Poincare Conjecture, a famous mathematical conjecture. Perelman posted three papers that culminated with a proof of the Poincare Conjecture in 2002 in a public repository but Perelman refused to submit any of these papers for publication in an academic journal. Jackson (2006) concluded: While it seems that Perelman’s papers were never refereed in the traditional sense, they have been subjected to extraordinary scrutiny in the three and a half years since their posting on the Web. The simple passage of time without anyone finding a serious problem in his work has, at least for many nonexperts, led to a conviction that it must be correct. (p. 899) I refer to the second exceptional type of theorem and proof that engenders unusually high levels of confidence in mathematicians as “classical results.” By classical results, I am referring to theorems and proofs that are so important and widely known that they have appeared in undergraduate or graduate mathematics textbooks for several decades. Since generations of mathematical students have studied these results and no one has found a mistake in these proofs, most mathematicians accept that mistakes in these proofs simply do not exist and believe that classical results are indubitable. In this sense, results that appear in several textbooks over a long period of time may be deemed as more trustworthy than those that appear in journals. For both landmark results and classical results, the higher level of confidence that mathematicians place in these theorems and proofs is not due to the content of the proofs, but instead due to the fact that the importance of the result and the passage of time implied that a large number of mathematicians inspected the result carefully.

248  •  Keith Weber

The collective testimony of the mathematical community is sufficient evidence for many mathematicians to be highly confident that the proof of an important result does not contain a mistake.

DISCUSSION Summary of Main Results and Limitations In mathematics, conjectures become theorems when they are proven and proofs are based on logic arguments that are objective and impersonal. Nonetheless, mathematicians sometimes engage in sourcing when deciding which proofs to accept. That is, mathematicians’ confidence in a theorem and its proof sometimes is dependent on the source of a proof and involve the consideration of factors that are independent of the mathematical content of the proof. Based on empirical studies of mathematicians’ practice, I have claimed that mathematicians are more confident that a proof is correct if it has been published, particularly in a top-ranked journal, and if it was written by an authoritative source, particularly an authority with a reputation for producing reliable work. I have also used mathematicians’ writings to hypothesize that a mathematician’s reputation can influence how carefully other mathematicians check her work and that mathematicians have more confidence in important results because they have been thoroughly scrutinized by the mathematical community. Not all scholars agree with the claims in this chapter. For instance, the mathematician Daniel Biss (2004) wrote that “no honest mathematician uses a result simply because it has been published” (p. 1218). Shanahan et al. (2011) documented how a pair of mathematicians claimed that when they read mathematical papers, they would make “an active effort not to use source as an interpretive consideration” (p. 406) because the only thing that mattered was the ideas in the text. I believe that both Biss (2004) and Shanahan et al. (2011) have made valuable contributions by describing how some mathematicians practice their craft. However, their contributions are consistent with the following observation of Geist et al. (2010), which I believe to be accurate: We know a substantial number of mathematicians who want to understand all proofs that form part of their papers and who will reprove even classical statements7 to be completely sure of their own results; but we also know that many mathematicians are not as meticulous and accept results in the published literature as black boxes in their own research. Many mathematicians tend to trust the experts. (pp. 158–159) This passage is consistent with the observations of Biss (2004) and Shanahan et al. (2011), but also with the interview and survey studies cited in this chapter (MejiaRamos & Weber, 2014; Weber & Mejia-Ramos, 2011). In my opinion, the extant literature on how mathematicians use sources is limited. While research studies provide strong evidence for the claims that mathematicians sometimes engage in sourcing to increase their confidence in a theorem and its proof, more work is needed in how this is done and the extent to which it is used. As Alcock et al. (2016) argued, investigating authentic mathematical practice poses substantial methodological challenges; triangulation using multiple experimental techniques

The Role of Sourcing in Mathematics  •  249

is needed to generate reliable claims about mathematical practice. I suggest that we need more fine-grained studies on how mathematicians engage in sourcing. One way to do this is to continue doing task-based interviews (Shanahan et al., 2011; Weber, 2008) or open-ended interviews (Müller-Hill, 2010; Weber & Mejia-Ramos, 2011) where mathematicians are asked to discuss their engagement in sourcing. While I would encourage researchers to conduct studies of this type, I would also advocate for naturalistic studies in which mathematicians are actually debating on which results should be used, trusted, and published. Observations of contentious seminars might be one venue for exploring these issues. These types of qualitative studies can provide important insights into how mathematicians do, and do not, engage in sourcing when reading mathematics and deciding which results to trust. However, given the heterogeneity of mathematical practice, we need larger studies to measure the extent that the hypotheses generated from small-scale qualitative studies generalize. To my knowledge, the only empirical studies on mathematicians’ sourcing with more than ten participants was the experimental study of Inglis and Mejia-Ramos (2009) and the surveys of Heinze (2010) and Mejia-Ramos and Weber (2014); these studies were certainly not robust enough to address all of the issues discussed in this chapter. Comparing Sourcing in Mathematics and Other Disciplines The Role of Sourcing in Mathematics with Respect to Interpretation and Bias Historians (and presumably other social scientists) find it difficult to distinguish between fact and interpretation. Consequently, when historians read texts, they constantly consider sources, including the motivations of the author and the context in which a paper is written, as an aid to interpret the text that they are reading. In particular, historians are aware that texts by some authors or within some genres tend to advance certain narratives and so these historians read these narratives with a critical eye (e.g., Shanahan et al., 2011; Wineburg, 1991). Shanahan et al. (2011) conjectured that mathematicians, like other scientists, do not use source as a tool for interpretation and do not concern themselves with the biases of the authors. I presume that mathematicians do not view mathematical statements as value-laden and perspectival. Instead, mathematicians regard the meaning of a mathematical statement as shared by the mathematical community. An Analogy Between Technical Results in a Proof and Empirical Data in the Sciences Empirical scientific papers rely on the collection of data. In evaluating how persuasive a paper is, a scientist must consider the quality of the data that was collected, which necessarily involves considering the reliability of the testimony of the authors (e.g., Chinn et al., 2011). Shanahan et al. (2011) analyzed how a pair of chemists engaged in sourcing in the reading of papers in their fields. They found that these chemists put greater confidence in papers written by known authorities in the field or that were published in premier journals. These chemists also were more skeptical of, and less likely to read, papers that were written by authors from institutions that they deemed questionable, such as those in developing countries.

250  •  Keith Weber

I contend that mathematicians behave similarly to Shanahan et al.’s (2011) chemists. Mathematicians often lack the background, time, or motivation to carefully scrutinize the technical details of a proof that they are reading. When this occurs, many mathematicians engage in sourcing to determine how likely the proof is to be correct. As I have documented in this chapter, this includes considering the reputation of the author of the article as well as the journal in which the article appeared. Some mathematicians will also consider the reputation of the author in deciding what to read. Differences Between Mathematicians’ Sourcing and Scientists’ Sourcing Aside from Shanahan et al.’s (2011) study, I am not aware of studies that compare the sourcing of mathematicians to other disciplines. Shanahan et al. noted that the chemists in their study gave less credence to papers that were several decades old. As one chemist noted, “For my kind of science, the information loses its relevance, because the principles are changing . . . The second paper is very driven by theory. A paper 20 years old would be totally irrelevant” (p. 412). In contrast, the mathematicians in Shanahan et al.’s study said time was irrelevant: proven results were still true and unsolved problems were still relevant. Indeed, in this chapter, I argued that for “classical theorems” in mathematics, older results were given more credibility because more mathematicians have had an opportunity to see if the proofs of these results contained a mistake. I suspect that the difference between science and mathematics arises due to the role that theory plays in each discipline. In mathematics, a theory does not require the virtue of verisimilitude to be viable. As a result, it is rare for a mathematical theory to be overturned, thereby making results within that theory obsolete. For instance, mathematicians do not debate whether Euclidean (planar) geometry or spherical geometry is “the real geometry”; the discovery of non-Euclidean (non-planar) geometries did not invalidate the results of Euclidean geometry. Rather, most mathematicians are happy to let Euclidean geometry and non-Euclidean geometries coexist. The claim that the angles of a triangle add up to 180 degrees in Euclidean geometry is considered to be an important valid theorem. That the angles of triangles do not add up to 180 degrees in spherical geometry is also considered an important valid theorem. This stands in contrast with other scientific disciplines, in which results based on discarded theories are often seen as obsolete. Further speculation on this issue is beyond the scope of this chapter (and beyond my expertise). As a direction for future research, I suggest looking at how the differing values of mathematicians and scientists influence the ways in which they engage in sourcing. For instance, mathematicians privilege a priori ways of knowing (c.f., Dawkins & Weber, 2017). Further, although there is no guarantee that a proven statement is true (the proof might contain a mistake), most mathematicians believe that a proof offers a conditional guarantee that a theorem is true (Fallis, 2002). If the proof does not contain an error and is couched in a sound conceptual system, the resulting theorem will be true. Whether and how these factors influence mathematicians’ sourcing would be a promising direction for future research. Implications for the Teaching of Mathematics I conclude this chapter by briefly considering implications for mathematics education. A more comprehensive discussion of these points can be found in Weber et al. (2014).

The Role of Sourcing in Mathematics  •  251

Many mathematics educators desire that students in mathematics classrooms gain conviction in mathematical statements in a manner akin to mathematicians (Dawkins & Weber, 2017). Further, to many mathematics educators, this means that students should not accept a mathematical statement as true if it is not supported by a proof. These students should not appeal to authoritative sources like teachers or textbooks in deciding what to believe (e.g., Harel & Sowder, 1998). I contend that if a student believes a mathematical statement is true because an authoritative source said it was so, that student is neither behaving irrationally nor behaving in a manner inconsistent with mathematical practice. Earlier, I noted that mathematicians do not always have the ability to determine if a proof is correct, especially if the proof uses ideas from a sub-discipline in which they lack expertise (Auslander, 2008). Consequently some mathematicians will accept some proofs as correct and some theorems as true if they were published in a respectable outlet or written by a reliable course. Students of mathematics are in a similar position. Most students lack the ability to determine if a proof is correct (e.g., Inglis & Alcock, 2012; Selden & Selden, 2003; Weber, 2010). Hence, it makes sense that they would engage in sourcing to decide what to believe. Results that appear in textbooks or are sanctioned by their teachers are probably true. There are, however, important differences in mathematicians’ professional practice within the mathematical classroom. First, the knowledge of the professional mathematical community is distributed throughout the community. It would be infeasible for every mathematician to understand every important mathematical result. It is enough that a significant subset of mathematicians grasps these results. Mathematics classrooms are different in that we have an ideal that all students should try to understand all material covered in the class. We would not absolve one student from comprehending why a mathematical statement was true because several of her classmates understood this. Second, important goals of mathematics education include helping students develop the disposition to justify claims with deductive reasoning (Harel & Sowder, 1998) and increasing students’ mathematical autonomy (Yackel & Cobb, 1996). Presumably most members of the professional mathematical community have already developed these dispositions. For these reasons, I agree with mathematics educators that students should not simply accept results as true, but should try to produce and understand proofs for why the mathematical statements that they learn are true. In short, it is not problematic if students engage sourcing to increase their confidence in mathematical claims, or even in the proofs that they generate. It is rational for students to trust what teachers and textbooks say and it is wise for students to ask teachers if their explanations are mathematically correct. However, it is problematic if students only consider the source of a claim or a proof in deciding what to believe.

NOTES 1 2

Whether this is the case is a deep philosophical issue beyond the scope of the chapter. However, I believe that the viewpoint that a proven statement is definitely true if the proof is correct is held by most mathematicians. There are other factors that mathematicians consider that are not independent of the content of the theorem and proof. For instance, mathematicians will have greater confidence in a proven result if the mathematical statement is well supported by empirical evidence, the statement is consistent with a large body of known results, or there are multiple independent proofs of the statement. These issues are interesting, but beyond the scope of this chapter.

252  •  Keith Weber 3 4 5 6 7

arXiv.org is a repository that is widely used for this purpose. This is consistent with Heinze’s (2010) survey with 40 mathematicians in which the mathematicians said they sometimes trusted results that were put forth by a well-known and respected colleague, but were more likely to consider other sources, such as publication status, in deciding what to believe. From Cohn’s “Advice for amateur mathematicians on writing and publishing papers.” Downloaded from: http://math.mit.edu/~cohn/Thoughts/advice.html. Last downloaded January 1, 2017. From Scott Aronson’s “Ten Signs a Claimed Mathematical Breakthrough is Wrong.” Downloaded from: www.scottaaronson.com/blog/?p=304. Last downloaded: January 1, 2017. Note here that this passage suggests that many mathematicians would consider it unusual and perhaps even unreasonable to seriously question classical results, supporting speculations that I made earlier in this chapter.

REFERENCES Alcock, L., Ansari, D., Batchelor, S., Bisson, M.-J., De Smedt, B., Gilmore, C., . . . Weber, K. (2016). Challenges in mathematical cognition: A collaboratively-derived research agenda. Journal of Numerical Cognition, 2, 20–41. Auslander, J. (2008). On the roles of proof in mathematics. In B. Gold & R. A. Simons (Eds.), Proofs and other dilemmas: Mathematics and philosophy (pp. 61–77). Washington, DC: Mathematical Association of America. Berry, D. (in press). Proof and the virtues of shared enquiry. Philosophia Mathematica. Biss, D. K. (2004). The elephant in the Internet. Notices of the American Mathematical Society, 51, 1217–1219. Buldt, B., Löwe, B., & Müller, T. (2008). Towards a new epistemology of mathematics. Erkenntnis, 68, 309–329. Chinn, C. A., Buckland, L., & Samarapungavan, A. (2011). Expanding the dimensions of epistemic cognition: Arguments from philosophy and psychology. Educational Psychologist, 46, 141–167. Dawkins, P., & Weber, K. (2017). Values and norms of proof for mathematicians and students. Educational Studies in Mathematics, 95, 123–142. De Millo, R. A., Lipton, R. J., & Perlis, A. J. (1979). Social processes and proofs of theorems and programs. Communications of the ACM, 22, 271–280. Dubinsky, E. (1997). A reaction to ‘a critique of the selection of mathematical objects as central metaphor for advanced mathematical thinking’ by Confrey and Costa. International Journal of Computers for Mathematical Learning, 2, 67–91. Fallis, D. (2002). What do mathematicians want? Probabilistic proofs and the epistemic goals of mathematicians. Logique et Analyse, 45, 373–388. Frans, J., & Kosolosky, L. (2014). Mathematical proofs in practice: Revisiting the reliability of published mathematical proofs. Theoria, 29, 345–360. Geist, C., Löwe, B., & Van Kerkhove, B. (2010). Peer review and testimony in mathematics. In B. Löwe & T. Müller (Eds.), Philosophy of mathematics: Sociological aspects and mathematical practice (pp. 155–178). London: College Publications. Harel, G., & Sowder, L. (1998). Students’ proof schemes: Results from exploratory studies. In A. H. Schoenfeld, J. Kaput, & E. Dubinsky (Eds.), Issues in mathematics education: Vol. 7. Research in Collegiate Mathematics Education, III (pp. 234–283). Providence, RI: American Mathematical Society. Heinze, A. (2010). Mathematicians’ individual criteria for accepting theorems as proofs: An empirical approach. In G. Hanna, H. N. Jahnke, & H. Pulte (Eds.), Explanation and proof in mathematics: Philosophical and educational perspectives (pp. 101–111). New York: Springer. Inglis, M., & Alcock, L. (2012). Expert and novice approaches to reading mathematical proofs. Journal for Research in Mathematics Education, 43, 358–390. Inglis, M., & Mejia-Ramos, J. P. (2009). The effect of authority on the persuasiveness of mathematical arguments. Cognition and Instruction, 27, 25–50. Jackson, A. (2006). Conjectures no more? Consensus forming on the proof of the Poincare and geometrization conjectures. Notices of the American Mathematical Society, 53, 897–901. Krantz, S. G. (2011). The proof is in the pudding: The changing nature of mathematical proof. New York: Springer. Kuteeva, M., & McGrath, L. (2015). The theoretical research article as a reflection of disciplinary practices: The case of pure mathematics. Applied Linguistics, 36, 215–235.

The Role of Sourcing in Mathematics  •  253 Mejia-Ramos, J. P., & Weber, K. (2014). How and why mathematicians read proofs: Further evidence from a survey study. Educational Studies in Mathematics, 85, 161–173. Müller-Hill, E. (2010). Die epistemische Rolle formalisierbarer mathematischer Beweise. Formalisierbarkeitsbasierte Konzeptionen mathematischen Wissens und mathematischen Rechtfertigung innerhalb einer sozioempirisch informierten Erkenntnistheorie der Mathematik. Unpublished doctoral dissertation. Rheinische FriedrichWilhelms-Universitat Bonn. Nathanson, M. (2008). Desperately seeking mathematical truth. Notices of the American Mathematical Society, 55, 773. Paseau, A. (2011). Mathematical instrumentalism, Gödel’s theorem, and inductive evidence. Studies in History and Philosophy of Science Part A, 42, 140–149. Sabbagh, K. (2004). The strange case of Louis de Brange. London Review of Books, 26, 13–14. Selden, A., & Selden, J. (2003). Validation of proofs considered as texts: Can undergraduates tell whether an argument proves a theorem? Journal for Research in Mathematics Education, 34, 4–36. Shanahan, C., Shanahan, T., & Misischia, C. (2011). Analysis of expert readers in three disciplines: History, mathematics, and chemistry. Journal of Literacy Research, 43, 393–429. Thurston, W. P. (1994). On proof and progress in mathematics. Bulletin of the American Mathematical Society, 30, 161–177. Weber, K. (2008). How mathematicians determine if an argument is a valid proof. Journal for Research in Mathematics Education, 39, 431–459. Weber, K. (2010). Mathematics majors’ perceptions of conviction, validity, and proof. Mathematical Thinking and Learning, 12, 306–336. Weber, K., Inglis, M., & Mejia-Ramos, J. P. (2014). How mathematicians obtain conviction: Implications for mathematics instruction and research on epistemic cognition. Educational Psychologist, 49, 36–58. Weber, K., & Mejia-Ramos, J. P. (2011). How and why mathematicians read proofs: An exploratory study. Educational Studies in Mathematics, 76, 329–344. Weber, K., & Mejia-Ramos, J. P. (2013). The influence of sources in the reading of mathematical text: A reply to Shanahan, Shanahan, and Misischia. Journal of Literacy Research, 45, 87–96. Weber, K., & Mejia-Ramos, J. P. (2015). The contextual nature of conviction in mathematics. For the Learning of Mathematics, 35, 9–14. Wineburg, S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87. Yackel, E., & Cobb, P. (1996). Sociomathematical norms, argumentation, and autonomy in mathematics. Journal for Research in Mathematics Education, 27, 458–477.

15

MULTIPLE SOURCE USE WHEN READING AND WRITING IN LITERATURE AND LANGUAGE ARTS IN CLASSROOM CONTEXTS David Bloome the ohio state university, usa

Minjeong Kim university of massachusetts, lowell, usa

Huili Hong towson university, usa

John Brady the ohio state university, usa

INTRODUCTION The use of multiple sources when reading and writing in literature and language arts in classroom contexts is ubiquitous. Among other instructional activities, teachers often organize a set of literary texts around a theme or as part of genre study; they supplement a target literary text with other texts such as information about the author or the historical period in which the literary text was written; and increasingly teachers are engaging students in reading literature on-line incorporating a broad range of annotations and links as well as new literary formats such as interactive fiction (Beach, 2012; Beach, Appleman, Fecho, & Simon, 2016; Turner et al., 2017). Although the use of multiple sources is ubiquitous in educational practice, compared to academic fields such as the teaching of history, it is our view that the use of the terms (and concepts of) multiple source and multiple text use in scholarship on the teaching of literature and the language arts is under-theorized. This under-theorizing may be the result of differences between the academic field of literature and language arts education

254

Multiple Source Use When Reading  •  255

versus other academic fields. In history, for example, an academic task is likely to involve the construction of a situation model of an historical event and that situation model can be warranted by the assessment of the reliability and trustworthiness of information from multiple sources (Britt & Aglinskas, 2002). However, in literature and language arts education the task is likely to involve orienting to a different definition of knowledge, a different way of reasoning, and a different set of learning goals (Goldman et al., 2016). These goals may include generating insight about the nature of the human condition including oneself, interpreting and responding to an artistic work, and pondering philosophical and existential questions about the world in which we live (Goldman et al., 2016). Thus, in literature and language arts education, the warranting of the insights, questions, and interpretations may involve a substantively different process than the reading of historical texts (Lee & Goldman, 2015; Goldman & Snow, 2015). We further note that the terms “multiple sources” and “multiple texts” are rarely used in scholarship on the teaching of literature and the language arts in preschool through secondary school. A search of the journal Research in the Teaching of English found only six articles using those terms in the years 2014 to 2017. More frequent has been the use of the term “intertextuality,” as it is a tradition of literary scholars to investigate the connections among literary texts (e.g., Allen, 2000; Bakhtin, 1981, 1984; Barthes, 1987; Genette, 1997; Kristeva, 1980, 1986; Worton & Still, 1990). The purpose of this chapter is to advance the theorizing of multiple source use when reading and writing in literature and language arts in classroom contexts. To do so, we offer a heuristic distinction between two theoretical frameworks. The first, grounded in cognitive science and educational psychology, focuses on the cognitive and linguistic processes involved in how a student might use multiple sources in literary study and in crafting a literary argument. Attention focuses on how multiple sources may be integrated, evaluated, and used to enhance the comprehension and interpretation of a literary text or a set of literary texts, and on how multiple literary and non-literary texts might be used to craft a literary argument (e.g., Chinn, Anderson, & Waggoner, 2001; Murphy, Wilkinson, Soter, Hennessey, & Alexander, 2009). The second theoretical framework, grounded in the fields of literary scholarship (e.g., Bakhtin, 1981; Volosinov, 1929/73), the anthropological study of literacy (e.g., Street, 1984, 1995), interactional ethnography (e.g., Castanheira, Crawford, Dixon, & Green, 2001), phenomenological studies of classroom literacy practices (e.g., Kamberelis & de la Luna, 2004), social semiotics (e.g., Lemke, 1992, 1995), and microethnographic discourse analysis (e.g., Bloome, Carter, Christian, Otto, & Shuart-Faris, 2005), focuses on the social, cultural, language, and interactional processes involved in the teaching and learning of literature and language arts. From this framework, the reading of literature and the writing of arguments are conceptualized as essentially a socially constructed process of people interacting with each other and multiple texts. We label these two frameworks a ‘Reader/Writer-Texts’1 framework and a ‘SocialInteractive-Texts’ framework. Following Street (1984, 1995), the labels reflect different models of literacy. The Reader/Writer-Texts framework reflects what Street (1995) has called an autonomous model; the Social-Interactive-Texts framework reflects what Street (1995) has called an ideological model. An autonomous model conceptualizes the use of written language as primarily involving a set of decontextualized cognitive and linguistic processes. From this framework, when the “social” is considered it refers to either the relationship of readers and authors or to social factors that mediate the relationship of the Reader/Writer and the text. An ideological model conceptualizes the use of written language as primarily involving social and cultural practices embedded

256  •  Bloome et al.

in socially situated local and macro social contexts (cf., Baker, 1993; Bourdieu, 1977). Co-present interlocutors interactionally, socially, and publically construct meaning within the social contexts of the immediate social situation, the institutional situation, and the macro-sociological situation. We begin this chapter by discussing the theorizing of multiple source use from a Reader/Writer-Texts framework and then from a Social-Interactive-Texts framework. We end with some final comments in which we suggest that a synthesis of the two frameworks is inappropriate. Rather, both scholars and educators need to craft a dialectical relationship between the two frameworks and struggle with the tensions generated by that dialectical relationship.

A READER/WRITER-TEXTS FRAMEWORK OF MULTIPLE SOURCE USE IN READING AND WRITING IN LITERATURE AND LANGUAGE ARTS EDUCATION Researching multiple source use in literature and language arts education from a Reader/Writer-Texts framework focuses on (1) the types and features of the multiple sources students use to enrich the processes of interpreting and composing literary and non-literary texts; (2) the cognitive and linguistic processes employed in the use of multiple sources; (3) the issues involved in the use of multiple sources across genres and text types (e.g. narrative, informational, and argumentative texts); and (4) how the assigned instructional task influences what students subsequently do and the cognitive processes they employ (Goldman et al., 2016; Lee & Goldman, 2015; Smith & Hillocks, 1988). While there is similarity with the questions asked about multiple source use in other subject areas, the inquiries need to be contextualized by the nature of literature and language arts education (Goldman et al., 2016). For example, consider multiple source use in Ms. Cook’s ninth-grade classroom2 when she assigned her students to write an argumentative essay about Ford Maddox Brown’s painting Work.3 An excerpt from the essay Kane4 wrote is below.

Text 1 – Kane’s Essay According to the Communist Manifesto, the working class “has but established new classes, new conditions of oppression, new forms of struggle in place of the old ones” (Marx 1). Marx and many others believed that the working class just created new problems, while Brown believed that the new working class actually fixed them. [C] Considering the problems evident in the past class systems, [A] Ford Maddox Brown painted work [B] to show that the classes can exist in peace [D] through nature, interactions between classes, and the overall change attitude of the upper class. Brown uses nature to show how he agrees with the way things are. Through the blue sky and green leaves, Brown is showing balance, order, and peace. Just like how “When Nippers’ was on, Turkey’s was off, and vice versa” (Melville 4). Although the Lawyer would prefer them both working, he accepts it and think that it is a “good natural arrangement” (Melville 5). Such is the case in this painting. Brown

Multiple Source Use When Reading  •  257

also shows that the classes have a long way to go, shown by the dirt road. The dirt shows begin in a past time, because the roads are still dirt and not pavement. The path itself shows they still have a long way to go, but they are slowly starting to communicate between classes. (from Newell et al., 2015, p. 129)

In Kane’s essay, multiple sources are explicitly referenced: Marx’s Communist Manifesto, Brown’s painting, and Melville’s “Bartleby the Scrivener.” The teacher provided those texts as well as lectures (with handouts) she gave on those texts and on constructing a claim and an argument. In the way in which the teacher organized her instruction, she also made available to Kane conversations with peers from which Kane could extract ideas to use in his essay. One of the key cognitive strategies involved in multiple source use is evaluating the sources one is using to construct a representation or an argument. Perfetti, Rouet, and Britt (1999) argued that constructing a representation based on multiple sources and texts differs from constructing a representation from a single source or text. With a single text, the text-base can be used to construct a situation model. However, with multiple texts a reader might not be able to integrate the text-bases from each text as there may be differences and disagreements. Thus, the process of constructing a representation requires attention to the relationships among the multiple sources and texts (what Perfetti et  al., 1999, labeled an intertext model) as well as to the broader situation (what Perfetti et al., 1999, labeled a situation model) and the task model (that is, how the task structures how the reader engages the text and constructs a meaning representation) that drives the reading activity (Goldman et  al., 2016). The three models taken together constitute what has been called a documents model (Britt, Rouet, & Braasch, 2013; Perfetti et al., 1999). Inherent to a documents model is that students need to evaluate the validity and trustworthiness of sources of information by considering the author, the content, the spatial-temporal distance from the event, and the documents’ relation to other documents (Britt et al., 2013). In conceptualizing the documents model, Britt et al. (2013) argued that the social relationship between the reader and the author of the text also needs to be considered. That social relationship may be important in readers’ assessment of the nature and validity of the information and knowledge provided in the texts. According to Slattery (1991), students may take a dogmatic approach to preselecting their stance on an issue and looking through their sources for affirming information to support their stance; a noncommittal approach in which students summarize information from the texts but do not critique the texts or shape a unique understanding based on the multiple perspectives; or an analytic approach, in which they analyze different perspectives and stances on an issue across texts, considering authorial bias and the ways in which the articles could be related to one another in crafting their own position. Goldman and Bloome (2005), in a study of two secondary language arts classrooms in which the teachers explicitly focused on constructing intertextuality, argued that language arts classrooms “provide opportunities for

258  •  Bloome et al.

students to construct and integrate in ways that lead to mental representations that reflect appropriately constrained interweaving of text and knowledge” (p. 180). Goldman (2004) argued that there are variations in how students construct and integrate meanings from multiple sources depending on the quality of the text-base and the student’s use of prior knowledge resulting in differing representations that Goldman labeled fragmentary, assimilated, coherent encapsulated, and coherent integrated representations (Goldman, 2004; Goldman et al., 2013). A fragmentary representation involves poor encoding of the text and the construction of unconnected, fragmented bits. An assimilated representation involves a poor text-base and the use of lots of prior knowledge such that a reader is incorporating bits of new information into what they already know. Nothing new is learned. A coherent encapsulated representation involves the reader well understanding the text but that understanding exists mostly in isolation from everything else the reader knows. The coherent integrated representation involves the encoding of a good-quality text-base and prior knowledge is connected appropriately, resulting in a coherent, well-integrated representation. Learning has occurred and is connected to what the reader already knows in a meaningful way. By examining Kane’s essay to explore cognitive processes he employed in constructing his argument and using post hoc interviews conducted with him and his teacher, we infer that Kane began his essay by establishing a tension between The Communist Manifesto and Brown’s Work. Kane’s use of “while” in the second sentence (“Marx and many others believed that the working class just created new problems, while Brown believed . . .”) contrasts Marx’s and Brown’s views. He positions Brown’s work in contrast to the ideals of Marx, while acknowledging the different media in which they are conveyed. To do so, Kane had to first construct a meaning and understanding for each work to at least some degree and assign to each a thematic interpretation; he could then juxtapose them, employing what Goldman (2004) labeled a coherent, encapsulated model of bringing together multiple sources. Kane continues his argument in the next paragraph by first drawing points of agreement between Melville’s work and Brown’s before conceding that Brown, in disagreement with Marx, recognizes that the classes have to continue working toward a better relationship. This suggests that Kane recognizes the intertextual relationship of the texts and the validity of their respective claims and is able to analyze the texts in a way that synthesizes the points of view in order to argue a claim through the use of evidence and warrant. In other words, Kane is able to recognize the problem of class struggle proposed by Marx’s philosophy and the counterargument proposed by Brown’s first-person account of the daily life of the worker in tandem with the reported experience of the worker in Melville’s story in order to argue a claim (of course, these are Kane’s interpretations of the meaning of Marx’s, Brown’s, and Melville’s artistic works and not necessarily the authors’ intentions or what others might interpret from these artistic works). What Kane has done in the second paragraph reflects what Goldman (2004) called coherent, integrated representation. Kane’s use of “Just like how” connects and prefaces an integration of the knowledge and concepts in Brown’s painting and Melville’s story; an integration Kane signals having accomplished by returning to Brown’s painting after paraphrasing and quoting from Melville’s story, writing “Such is the case in the painting.” From an interview with the teacher (see Olsen, 2013), Kane successfully accomplished the assigned task. The explicit task was to use Brown’s painting, Marx’s

Multiple Source Use When Reading  •  259

Communist Manifesto, and Melville’s “Bartleby the Scrivener” to construct an argument with evidence about “how you have begun to think about the text(s), and how your insights lend a key understanding to analysis or interpretation of the text.” Beyond the explicit task, there was an implied task to use the heuristic the teacher had earlier provided for constructing a claim and to make use of the feedback she had provided students on their first drafts. Kane made visible his adherence to the heuristic by inserting [A] for Author, [B] for aBstract concept examined, [C] for Commentary on B, and [D] for the rhetorical, literary Device used to develop B. (For a detailed analysis of Kane’s argumentative writing, see Newell, Bloome, & Hirvela, 2015; Olsen, 2013.)

A SOCIAL-INTERACTIVE-TEXTS FRAMEWORK OF MULTIPLE SOURCE USE IN READING AND WRITING IN LITERATURE AND LANGUAGE ARTS EDUCATION A Social-Interactive-Texts framework builds on the literary scholarship of Bakhtin (1981, 1984, 1986) and his colleagues (e.g., Medvedev, 1985; Volosinov, 1929/73) and how their theories of heteroglossia, genre, intertextuality, chronotope, dialogue, and socially and historically situated meaning in speech and in literature have been taken up by literary theorists (e.g., Barthes, 1987; Kristeva, 1980), discourse analysts (e.g., Bazerman, 2004; Bloome & Egan-Robertson, 1993; Erickson, 2004; Matusov, 2009), and educational researchers concerned with the teaching and learning of literature and the language arts (e.g., Beach, Appleman, & Dorsey, 1990; Lee, 2004; Maybin, 2001; Newell et al., 2015; Nystrand & Gamoran, 1997). A Social-Interactive-Texts framework is grounded in an ideological model of literacy (Street, 1984, 1995). An ideological model focuses attention on the situated and culturally driven practices readers and writers employ in their use of multiple sources in social events such as classroom lessons. From this framework, every event, utterance, use of text, and interaction is essentially social, reflecting and refracting a previous utterance, event, interaction, and inherently bound up in a social interaction from which it is impossible to extract the individual or individual utterance, text, or act (Gergen, 1999, 2001; Volosinov, 1929/73). From this framework, it is a non-sequitur to ask, “Were multiple sources used or only a single source or text?” as one can only ask, “What multiple sources were used? How? By and with whom? When? Where? Why? For what purposes?” Every text is viewed as an “intertext” as every text is viewed as inherently heteroglossic, intertextual, and dialogic (Hanks, 2000; Orr, 2003; Volosinov, 1929/73). What is at issue is how it is an “intertext” (what other texts and sources are being reflected and refracted). In other words, at issue is how, when, where, and who socially constructs what series of intertexts; and how did the series of intertexts evolve over time and space (Bloome & Egan-Robertson, 1993)? Consider the writing of a story in a preschool classroom. Twice a week, Ms. Moore’s preschool students engaged in what they called, “The Storytelling Project” (for details see Bloome, Champion, Katz, Morton, & Muldrow, 2000). The teacher read a story to the students, followed by student storytelling, followed by having the students write or draw a story to be later shared with the class. Below is one page from Tanya’s five-page written/drawn story titled, “The Bad Wolf.”

260  •  Bloome et al.

Figure 15.1  Text 2: Page One of Tanya’s “Bad Wolf” Story.

The scribble marks to the left and top are Tanya’s writing (a transcription is at the bottom of the page); the rest is a picture of the wolf. From a Social-Interactive-Texts framework, the unit of analysis is the series of social interactions, social events, and social contexts of the story writing, inclusive of the people therein (Bloome et al., 2005; Hong, 2015). Questions are asked about what the students are socially constructing during the composing of their stories: What social relationships? What identities? What shared definitions of story and narrative? What intertextualities are being socially constructed (Bloome & Egan-Robertson, 1993) with what meanings and social consequences? How is the social event of the students’ composing connected to which other social events? Questions are also asked about the evolving meaningfulness of the event. From a Social-Interactive-Texts framework the meaningfulness of the event is not only in the fashioning of a storytelling text (the entextualization; Bauman & Briggs, 1990; Silverstein, 1998) but also in the lived social events of composing and telling (what Blommaert, 2016, p. 15, called the “total semiotic fact”). Tanya is interacting with other students and the teacher at her table while writing. Some of the students draw on her story and some are writing other bad wolf stories that they are also sharing while they write. As the teacher circulates through the classroom she asks the students what the story “says” and she writes it on the picture; but what the students say often differs from what they said as they were writing. In this classroom, the girls often draw rainbows on each other’s stories. They did so without reference to the extant story, although when the teacher asked what the story says, the students would often incorporate the rainbow into the story in a way that was distinct from the rest of the story (e.g., “There was a rainbow.”). The meaningfulness

Multiple Source Use When Reading  •  261

of these rainbow practices appears to be the use of writing to establish a gendered identity, signal membership in a social group, and maintain a friendship network. The meaningfulness of the written text itself appeared only to be pertinent when the teacher intervened and asked what the story says. Thus, to ask about multiple source use in an emic manner requires recognition that the teacher and students locate and define meaning differently at different times and thus what constituted appropriateness and effectiveness of multiple source use needs to be multiply and temporally framed. The sources that the students used to compose their stories included broad cultural ideologies for storytelling. Champion (1998) noted that the structure of stories valued in school may differ from the structure of stories in some African-American communities. She argued that some stories in African-American communities are organized around a shared set of moral principles that do not need to be explicitly stated, whereas stories grounded in European-American cultures are often structured around a topic and coherently linked actions (see also Michaels, 1986). For the children in this classroom, who came from a predominantly African-American community, constructing a narrative appropriate to the classroom setting was a cross-cultural endeavor as they needed to learn how to background the narrative structures they had acquired from their family and community life and foreground school-appropriate narrative structures. Their interactions with teachers who re-voiced student narratives and asked questions that scaffolded movement toward school-oriented narrative structures, and mentor narrative texts such as “The Three Little Pigs” and “Little Red Riding Hood,” were sources that the students employed to build school-appropriate narratives. From a language socialization perspective (cf., Ochs & Schieffelin, 2011), the teacher is using language to socialize the students to the cultural practices and ideologies of the school and the dominant culture for structuring narratives and using language. It is in this sense that cultural and institutional ideologies become a resource for students; and knowing which ideology and which related sources to draw on for story making in which social contexts is part of the communicative competence that students acquire over time. This is no less so for students whose home culture is aligned with the dominant societal and school culture. They, too, are using institutional and broad, cultural ideologies as sources for composing and interpreting narratives; the lack of a contrastive context may make it less visible and seemingly natural but it is nonetheless a social and cultural process of language socialization (cf., Freebody, Luke, & Gilbert, 1991).

FINAL COMMENTS We conceptualize the use of multiple sources when reading and writing in literature and language arts in classroom contexts as located heuristically in two frameworks. The first is a Reader/Writer-Texts framework grounded in cognitive science and focused on the cognitive and linguistic processes and strategies an individual might employ when engaging multiple sources and literary texts or in composing a narrative or an argument that involves multiple sources. The second is a Social-Interactive-Texts framework grounded in the fields of literary theory, discourse analysis, the anthropological study of literacy, and interactional ethnography, focused on the situated social construction of meaningfulness as people (teachers and students) interact with each other, with multiple sources, within multiple social contexts and cultural ideologies. While these two frameworks are not antagonistic to each other, neither are they complementary.

Table 15.1  Differences Across a Reader/Writer-Texts Framework and a Social-Interactive-Texts Framework Reader/Writer-Texts Framework Constructing and composing an interpretation of a literary text or set of literary texts in ways approaching that of an “expert” in the field. Constructing an essayist argument regarding a literary or non-literary text or set of texts in ways approaching that of an “expert” in the field. Gaining individual insight about the human condition and oneself. Acquiring “expertise” in literary analysis. Acquiring skills of reading, writing, and narrative production (especially at earlier grades). Unit of The individual reader/writer, the multiple Analysis sources they use, and the text produced (such a text may be an interpretation of a set of texts read or it may be a composition). Social factors may mediate how the individual reader/writer uses multiple sources in the production of a text. Key What is the assigned task? Questions What sources are made available? How do Asked students access them? What is the nature (the attributes and content) of the multiple sources? How do students evaluate the validity and trustworthiness of the sources made available and used? What cognitive and linguistic processes and strategies do students employ in bringing together multiple sources? How does instruction improve students’ use of cognitive and linguistic processes and strategies? Nature of An interaction of an individual with one or more written texts involving a Reading universalist set of cognitive and linguistic and processes and strategies, mediated by Writing social factors. Movement A reader/writer uses a set of cognitive and linguistic processes and strategies during Through Time learning events to produce a spoken or written text (linear movement through time) in response to a literary text or literary theme. Over time, the individual gains increasing expertise in the appropriate cognitive and linguistic processes for using multiple sources (linear movement through time). Goals of Multiple Source Use in Literature and Language Arts Education

Social-Interactive-Texts Framework Gaining collective insight about the human condition. Engaging appropriately in the social events of interpreting literary texts and composing arguments using multiple sources. Acquiring the social practices needed for appropriate participation in the social events noted above (and in so doing, becoming socialized to the cultural community of the literature and language arts classroom and of the field of literature study). Establishing social relationships with others (both those in the social events noted above and others). Social events in which literary texts and essayist arguments are used in a non-trivial way within which multiple people within a social group (such as a classroom) interact with each other and with multiple sources. The social contexts of the social events noted above. What are the situated social practices employed in the social events noted above? How do those social practices define and construct knowledge? Social relationships? Social identities? Relationships among sources and texts? How do social practices vary across situations and institutions, and how do they evolve over time? What is the relationship of the situated social events to other social events and to broader cultural ideologies? Reading and writing are situated social practices involving the non-trivial use of written language embedded in social contexts. Movement through time is not given, but socially constructed within each social event involving the use of multiple sources when reading and writing in literature and language arts education. Within and across such events is an implied chronotope, an ideology about movement through space and time.

Multiple Source Use When Reading  •  263

Synthesizing these two frameworks is inappropriate because of paradigmatic differences in underlying assumptions about the goals of multiple source use when reading and writing in literature and language arts education, units of analysis, the questions that can be asked, the nature of reading and writing, and underlying assumptions about movement through time. We summarize these differences in Table 15.1. Regarding educational goals across the two frameworks, there is similarity at a surface level. From both frameworks, there is an emphasis on students gaining acumen in the use of multiple sources in reading and writing in literary analysis and the production of literary arguments. Substantive differences, however, focus on whether the educational goal is oriented to the individual’s acquisition of cognitive strategies and processes similar to that of an expert in the field or whether the educational goal is oriented to the social construction and enactment of the social event of multiple source use in the literature and language arts classroom, and the acquisition of situated social practices by participants in those events in ways reflective of the cultural ideologies of the institutional culture (the school) and the community of literature study (e.g., Gutiérrez, 2008). As discussed earlier, the unit of analysis differs across the two frameworks. From a Reader/Writer-Texts framework the unit of analysis is the interaction between the individual and the multiple texts to be used, and factors that might mediate that relationship. From a Social-Interactive-Texts framework the unit of analysis is the social event of multiple text use in a literature and language arts classroom and its relationship with other social events and social contexts. Differences in the unit of analysis reflect differences in how multiple text use is defined when reading and writing in literature and language arts education, and in the questions that can be asked. The difference in the questions also reflects differences in how reading and writing are defined. From a Reader/Writer-Texts framework, reading and writing are defined as a set of cognitive processes and strategies employed in the interaction between the individual and the text, whereas from a Social-Interactive-Texts framework, reading and writing are defined as situated social practices involving interaction among people employing multiple texts. The two frameworks also differ in how movement through time is conceptualized. In a Reader/Writer-Texts framework, students produce a representation based on the use of multiple texts. It takes the formula of “doing X leads to representation Y.” Students move through instructional events and their interactions with multiple texts to arrive at a representation of an interpretation of a target literary text or at a representation of a theme. Over time, students’ use of cognitive processes and strategies and literary heuristics moves closer to that of an expert in the field. By contrast, the conception of time from a Social-Interactive-Texts framework is socially constructed by how people engage each other in their use of multiple texts when reading and writing literature. For Tanya, when she was involved in composing her story, her presenttense participation in the social event itself was foregrounded (interacting with her friends at her table as well as with the multiple sources available). The differences in the two frameworks are not just a theoretical issue for researchers and scholars, but also an issue for educators as they consider how to construct curriculum and how to engage their students in the use of multiple sources in literature and language arts education. While a synthesis of the two frameworks may not be appropriate at a substantive level, movement across the two frameworks provides a broader set of questions and insights about multiple source use when reading and writing in literature and language arts in classroom contexts.

264  •  Bloome et al.

NOTES 1

2 3

4

We use the phrases “multiple sources” and “multiple texts” interchangeably. In literature and language arts classrooms the multiple sources students use are primarily written or spoken texts. Other sources, such as pictures, experiences, videos, memories, etc., are often textualized before or during use (e.g., during discussion of a literary text students often share their experiences before using them in crafting an interpretation or an argument). The example from Ms. Cook’s classroom is taken from Newell, Bloome, and Hirvela (2015) and was derived from Olsen (2013). The use of student artifacts in this manuscript is intended to illustrate theoretical issues in students’ uses of multiple sources in literacy and language arts education. The two examples used in this chapter are not presented for comparative purposes, but rather to provide different affordances for exploring theoretical issues. In our own empirical studies, we approach texts and other data within a dialectical framework in which we work back-and-forth between a set of theoretical constructs, a logic-of-inquiry, and the data (for a detailed explanation see Bloome et al., 2005). All names are pseudonyms.

REFERENCES Allen, G. (2000). Intertextuality. New York: Routledge. Baker, C. (1993). Literacy practices and social relations in classroom reading events. In C. Baker & A. Luke (Eds.), The critical sociology of reading pedagogy (pp. 141–160). Amsterdam: John Benjamins. Bakhtin, M. (1981). The dialogical imagination: Four essays of M. M. Bakhtin (C. Emerson & M. Holquist Trans.). Austin, TX: University of Texas Press. Bakhtin, M. (1984). Problems of Dostoevsky’s poetics [trans. C. Emerson]. Minneapolis, MN: University of Minnesota Press. Bakhtin, M. (1986). Speech genres and other late essays. Austin, TX: University of Texas Press. Barthes, R. (1987). The rustle of language. New York: Macmillan. Bauman, R., & Briggs, C. L. (1990). Poetics and performance as critical perspectives on language and social life. Annual Review of Anthropology, 19, 59–88. Bazerman C. (2004). Intertextualities: Volosinov, Bakhtin, literary theory, and literacy studies. In A. Ball & S. Freedman (Eds.), Bakhtinian perspectives on language, literacy, and learning (pp. 53–65). Cambridge, UK: Cambridge University Press. Beach, R. (2012). Uses of digital tools and literacies in the English language arts classroom. Research in the Schools, 19(1), 45–59. Beach, R., Appleman, D., & Dorsey, S. (1990). Adolescents’ uses of intertextual links to understand literature. In R. Beach & S. Hynds (Eds.), Developing discourse practices in adolescence and adulthood (pp. 224–245). Norwood, NJ: Ablex. Beach, R., Appleman, D., Fecho, B., & Simon, R. (2016). Teaching literature to adolescents. New York: Routledge. Blommaert, J. (2016). Meaning as a non-linear effect: The birth of cool. AILA Review, 28, 7–27. Bloome, D., Carter, S., Christian, B., Otto, S., & Shuart-Faris, N. (2005). Discourse analysis and the study of classroom language and literacy events: A microethnographic approach. Mahwah, NJ: Erlbaum. Bloome, D., Champion, T., Katz, L., Morton, M., & Muldrow, R. (2000). Spoken and written narrative development: African-American preschoolers as storytellers and storymakers. In J. Harris, M. Kamhi, & K. Pollock (Eds.), Literacy in African-American communities (pp. 45–76). Mahwah, NJ: Erlbaum. Bloome, D., & Egan-Robertson, A. (1993). The social construction of intertextuality and classroom reading and writing. Reading Research Quarterly, 28(4), 303–333. Bourdieu, P. (1977). Outline of a theory of practice. Cambridge, UK: Cambridge University Press. Britt, M. A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20(4), 485–522. Britt, M. A., Rouet, J. F., & Braasch, J. L. G. (2013). Documents as entities: Extending the situation model theory of comprehension. In M. A. Britt, S. R. Goldman, & J. F. Rouet (Eds.), Reading: From words to multiple texts (pp. 160–179). New York: Routledge. Castanheira, M. L., Crawford, T., Dixon, C. N., & Green, J. L. (2001). Interactional ethnography: An approach to studying the social construction of literate practices. Linguistics and Education, 11(4), 353–400.

Multiple Source Use When Reading  •  265 Champion, T. (1998). ‘Tell me somethin’ good’: A description of narrative structures among African-American children. Linguistics and Education, 9(3), 251–286. Chinn, C. A., Anderson, R. C., & Waggoner, M. A. (2001). Patterns of discourse in two kinds of literature discussion. Reading Research Quarterly, 36(4), 378–411. Erickson, F. (2004). Talk and social theory: Ecologies of speaking and listening in everyday life. Cambridge, UK: Polity Press. Freebody, P., Luke, A., & Gilbert, P. (1991). Reading positions and practices in the classroom. Curriculum Inquiry, 21(4), 435–457. Genette, G. (1997). Paratexts: Thresholds of interpretation. New York: Cambridge University Press. Gergen, K. J. (1999). An invitation to social construction. Thousand Oaks, CA: Sage. Gergen, K. J. (2001). Social construction in context. Thousand Oaks, CA: Sage. Goldman, S. R. (2004). Cognitive aspects of constructing meaning through and across multiple texts. In N. Shuart-Faris & D. Bloome (Eds.), Uses of intertextuality in classroom and educational research (pp. 317–352). Charlotte, NC: Information Age. Goldman, S. R., & Bloome, D. (2005). Learning to construct and integrate. In A. F. Haley (Ed.), Experimental cognitive psychology and its applications: Festschrift in Honor of Lyle Bourne, Walter Kintsch, and Thomas Landauer (pp. 169–182). Washington, DC: American Psychological Association. Goldman, S. R., Britt, M. A., Brown, W., Cribb, G., George, M., Greenleaf, C., Lee, C., Shanahan, C., & Project READI. (2016). Disciplinary literacies and learning to read for understanding: A conceptual framework for disciplinary literacy. Educational Psychologist, 51(2), 219–246. Goldman, S. R., Ozuru, Y., Braasch, J. L. G., Manning, F., Lawless, K., Gomez, K., & Slanovits, M. J. (2013). Literacies for learning: A multiple source comprehension illustration. In N. L. Stein & S. Raudenbush (Eds.), Developmental cognitive science goes to school (pp. 30–44). New York: Routledge. Goldman, S. R., & Snow, C. E. (2015). Adolescent literacy: Development and instruction. In A. Pollatsek & R. Treiman (Eds.), The Oxford handbook of reading (pp. 463–478). Oxford, UK: Oxford University Press. Gutiérrez, K. D. (2008). Developing a sociocritical literacy in the third space. Reading Research Quarterly, 43(2), 148–164. Hanks, W. F. (2000). Intertexts: Writings on language, utterance, and context. New York: Rowman & Littlefield Publishers. Hong, H. (2015). Exploring young children’s writer identity through the lens of dialogism. International Journal of Early Childhood, 47(2), 301–316. Kamberelis, G., & de la Luna, L. (2004). Children’s writing: How textual forms, contextual forces, and textual politics co-emerge. In C. Bazerman & P. Prior (Eds.), What writing does and how it does it (pp. 239–277). New York: Routledge. Kristeva, J. (1980). Desire in language: A semiotic approach to literature and art. New York: Columbia University Press. Kristeva, J. (1986). Word, dialogue, and novel. In T. Moi (Ed.), The Kristeva reader (pp. 34–61). Oxford, UK: Basil Blackwell. Lee, C., & Goldman, S. R. (2015). Assessing literary reasoning: Text and task complexities. Theory into Practice, 54(3), 213–227. Lee, C. D. (2004). Double voiced discourse: African American vernacular English as resource in cultural modeling classrooms. In A. Ball & S. Freedman (Eds.), Bakhtinian perspectives on language, literacy, and learning (pp. 129–149). New York: Cambridge University Press. Lemke, J. (1992). Intertextuality and educational research. Linguistics and Education, 4, 3–4, 257–267. Lemke, J. L. (1995). Intertextuality and text semantics. In M. Gregory & P. Fries (Eds.), Discourse in society: Functional perspectives (pp. 85–114). Norwood, NJ: Ablex. Matusov, E. (2009). Journey into dialogic pedagogy. Huntington, NY: Nova Science Publishers. Maybin, J. (2001). Language, struggle and voice: The Bakhtin/Volosinov writings. In M. Wetherell, S. Taylor, & S. Yates (Eds.), Discourse theory and practice: A reader (pp. 64–71). Thousand Oaks, CA: Sage. Medvedev, P. N. (1985). The formal method in literary scholarship: A critical introduction to sociological poetics (Albert J. Wehrle, trans.). Cambridge, MA: Harvard University Press. Michaels, S. (1986). Narrative presentations: An oral preparation for literacy with first graders. In J. CookGumperz (Ed.), The social construction of literacy (pp. 94–l16). Cambridge, UK: Cambridge University Press. Murphy, P. K., Wilkinson, I. A., Soter, A. O., Hennessey, M. N., & Alexander, J. F. (2009). Examining the effects of classroom discussion on students’ comprehension of text: A meta-analysis. Journal of Educational Psychology, 101(3), 740–764.

266  •  Bloome et al. Newell, G., Bloome, D., & Hirvela, A. (2015). Teaching and learning argumentative writing in high school English language arts classrooms. New York: Routledge. Nystrand, M., & Gamoran, A. (1997). Opening dialogue: Understanding the dynamics of language and learning in the English classroom. New York: Teachers College Press. Ochs, E., & Schieffelin, B. (2011). The theory of language socialization. In A. Duranti, E. Ochs, & B. Schieffelin (Eds.), Handbook of language socialization (pp. 1–21). New York: Blackwell. Olsen, A. W. (2013). A longitudinal examination of interactional, social, and relational processes within the teaching and learning of argumentation and argumentative writing (Doctoral dissertation). Columbus, OH: The Ohio State University. Orr, M. (2003). Intertextuality: Debates and contexts. Malden, MA: Polity. Perfetti, C. A., Rouet, J. F., & Britt, M. A. (1999). Toward a theory of documents representation. In H. van Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 88–108). Mahwah, NJ: Erlbaum. Silverstein, M. (1998). Contemporary transformations of local linguistic communities. Annual Review of Anthropology, 27(1), 401–426. Slattery, P. (1991). The argumentative, multi-source paper. Journal of Teaching Writing, 10(2), 181–200. Smith, M., & Hillocks, G. (1988). Sensible sequencing: Developing knowledge about literature text by text. English Journal, 77, 44–49. Street, B. (1984). Literacy in theory and practice. New York: Cambridge University Press. Street, B. V. (1995). Social literacies: Critical approaches to literacy in development, ethnography, and education. London: Longman. Turner, K. H., Hicks, T., Gere, A. R., Homan, E. C., Parsons, C., Spooner, R. A., Uzogara, C., & Kajder, S. (2017). Connected reading: Teaching adolescent readers in a digital world. English Journal, 106, 75–78. Volosinov, V. (1929/73). Marxism and the philosophy of language (L. Matejka & I. Titunik, trans.). New York: Seminar Press. Worton, M., & Still, J. (Eds.). (1990). Intertextuality: Theories and practices. Manchester, UK: Manchester University Press.

Section IV

Multiple Source Use Beyond the Classroom

16

THE PROVENANCE OF CERTAINTY Multiple Source Use and the Public Engagement with Science Rainer Bromme university of münster, germany

Marc Stadtler ruhr-university bochum, germany

Lisa Scharrer ruhr-university bochum, germany

INTRODUCTION Linda, a young mother, notices her 2-year-old daughter’s interest in the family tablet computer. Wondering what science has to say about the upsides and downsides of young children interacting with digital devices, she fetches her tablet, which, in fact, she has to take away from her daughter. Linda enters the search term “effects tablet use small children,” resulting in 40,300,000 hits. She notices that only a few documents are authored by scientists or experts. Further, when she sees that three of her first 10 hits are recommendations on purchasing a tablet for her daughter, she tries to take a closer look at the sources of the websites. However, gathering more information about the authors’ expertise and intentions would require additional time, and through her desperate attempts to reach for the tablet, Linda’s daughter does not leave her mother any time for further investigations. This story is fictitious, but the scenario is real: Many issues pertaining to our daily lives require the use of scientific knowledge, and we search for it on the Internet (Bromme & Goldman, 2014; Stadtler & Bromme, 2014; Tabak, 2015). The number of search results can be overwhelming, and the user cannot control the sequence of these results, nor are the results sorted by any measure of reliability or validity. Nonetheless, the user still has to determine whether various claims are relevant and reliable. One possible way to do this is to assess the provenance of the target information. Provenance 269

270  •  Bromme et al.

means the genesis of information; that is, the history of its creation and existence before it was brought to the user’s attention. A user can do this via sourcing, which refers to evaluating the provenance of target information (e.g., see Bråten & Braasch, this volume). For the purposes of this chapter, we refer to knowledge about the provenance of information as provenance information (e.g., the name of an author). In this chapter, we review the processing of provenance information (sourcing) in the context of the public engagement with science, which we divide into four sections. First, clarify several conceptual and terminological ideas that provide a foundation for a theory of processing provenance information. Second, we explain the benefits and the challenges non-experts face when finding and using scientific information. We argue that processing provenance information is necessary for citizens to benefit from the wealth of scientific insights in modern societies. Third, we provide an overview of empirical findings on—primarily—students’ sourcing behavior. Lastly, we discuss implications for theory, research, and practice.

CONCEPTUAL BUILDING BLOCKS FOR A THEORY OF PROVENANCE PROCESSING According to the document model framework (Britt & Rouet, 2012; Perfetti, Rouet, & Britt, 1999), an ideal representation of multiple documents consists both of an integrated mental representation of contents (the situations model), and a representation of the documents from which contents are taken (the intertext model). According to this framework, a full documents model results from the integration between the situations model and the intertext model. The documents model enables readers to understand and evaluate knowledge claims in light of the conditions under which they were produced. These conditions entail first of all the source, but they could also entail further circumstances of the genesis of these knowledge claims. Information Resources and Sources Goldman and Scardamalia (2013) have suggested that distinguishing between information resource and source would add clarity to research on multiple document comprehension. Information resource refers to the entity that conveys the information (e.g., a newspaper), whereas source refers to the agent (e.g., a journalist) who produces the information (e.g., an article in the newspaper). Furthermore, they characterize source information as metatextual information such as the author or date of origin of a document. We agree that it is important to distinguish between the agent who produces information and the entity that conveys the information. However, it is also important to highlight the fact that the original source of information may differ from a subsequent source of that information. For instance, embedded sources are commonly used in a newspaper or magazine article about a science topic that is produced for laypersons. In most cases, the author of the document has not directly produced the scientific knowledge. Rather, the author has reviewed original research and reported on it. Thus, information about an embedded source is not always the same as information about the source who has produced a document. Hence, information about an embedded source is not metatextual with regard to the main document, which entails the embedded source.

The Provenance of Certainty  •  271

Provenance To summarise the gist of what is meant by sourcing and source information, we suggest using the notion of the provenance of the information. Literally, this term refers to the place of origin of a work of art. But the concept is not just used in this narrow sense; it also refers to the full history of the item: Who painted a picture and when; who has owned it since then; and how, when, and why it was placed at its current location. Such information is important for uncovering art forgery, but it also matters when interpreting a work of art. We hold the view that this concept nicely characterizes the core of what source information is about and why it matters: for establishing the meaning as well as the plausibility of the knowledge claims (assertions) provided by an information resource. Similar to art, neither the meaning nor the plausibility of information can be established independently from a reader’s goals and the (historical) context of reading a document. In most everyday situations, and particularly with regard to nonfictional content, readers process documents with the goal of finding an answer to a question but not a complete understanding of the document per se. This implies that processing source information is usually confined to understanding and evaluating those claims within a document that are central to a reader’s goals. Whereas metatextual information pertains to a whole document, provenance pertains to conclusions about the meaning and plausibility of a knowledge claim within a document, and not necessarily conclusions about the whole document. Further, source information can sometimes only be inferred from the conversational setting and it would be not well characterized as metatextual information. For example, in Hendriks, Kienhues, and Bromme (2016) laypersons evaluated a critical statement about a scientific knowledge claim differently when the same statement was made as a self-critique versus a critical remark about the work of another scientist. Thus, people make inferences about the pragmatic intentions of the interlocutors based on who said what and not just on the content of what is said. Such information clearly matters for sourcing. In sum, the genesis (a process often involving several agents) of a given knowledge claim might matter when a reader has to understand and evaluate the claim.

WHY SOURCING IS CRUCIAL FOR CITIZENS’ ENGAGEMENT WITH SCIENCE Activities for improving the public engagement with science and technology provide opportunities for citizens to learn about and from science—for example, via science museums, popular science journals, television formats, or YouTube. However, people also encounter scientific information that has not been explicitly designed for the purpose of informing the public about science, for example when their doctor recommends a vaccination because it has been tested scientifically or when they watch a television discussion on climate change in which participants refer to the underlying scientific evidence. When the young mother in our introductory scenario looked for scientific evidence, her search included results that had not been designed for the purpose of informing the public about science. In this vein, public engagement with science refers to the conditions, processes, and results of citizens’ communicative, cognitive, and emotional encounters with science.

272  •  Bromme et al.

CHALLENGES FOR CITIZENS’ ENGAGEMENTS WITH SCIENCE A citizen’s engagement with science is constrained by some inherent features of science and scientific knowledge, and by the ways this knowledge is accessible today—especially on the Internet. These challenges, as they relate to sourcing, will be described below. Science Can Provide Truth, but this Truth Remains Tentative Science is an endeavor to produce truth, and people are interested in science because they are searching for true answers to their questions. Linda wants to know what science has to say about toddlers’ tablet use because she wants to do the right thing. However, just because scientific knowledge is tentative does not mean that all scientific claims are equally valid. Some claims are justified more strongly than others on the basis of evidence. For example, the evidence for the claim that there is a human impact on climate change is strong—measured by scientifically accepted criteria for evaluating such a claim. However, evidence supporting the claim that carbon emissions could be stored underground on a large scale is much weaker and more tentative. Scientists with Pertinent Expertise Are Only One Voice in a Chorus of Voices With regard to many issues of public interest, science-based knowledge has to compete with other kinds and qualities of knowledge such as folk theories, pseudoscience, esoteric belief systems, or unfounded conspiracy theories. Some of these competitors are not easy to discern from science-based proponents, and pseudoscience claims to adhere to the rules of science while actually ignoring them (Goldacre, 2008). Indeed, some of these competitors deny the potency and pertinence of science as a provider of knowledge (Munro, 2010; Nauroth, Gollwitzer, Bender, & Rothmund, 2015). Many science-related issues of public interest are intertwined with political and ethical matters; they are socio-scientific issues (Kolstø et  al., 2006). Public debates about child education, sustainable energy, and the use of genetically modified food are not just debates about the underlying science. Even when conflicts about how to deal with scientific insights are in the foreground of public debates, these conflicts are often hard to separate from the scientific issues. Science Information Is Easily Accessible on the Internet, but Control of Quality and Relevance Is Difficult Our introductory example showed how easy it is to find knowledge claims about toddlers’ tablet use. However, information is of varying quality, and search algorithms are opaque and unpredictable. Nonetheless, in many cases the Internet provides enough cues about the provenance of a certain knowledge claim. However, critical reading competencies are required to identify and to interpret such cues for judging the relative truth value of a claim (Stadtler, 2017).

The Provenance of Certainty  •  273

SHOULD I BELIEVE THIS? TWO TYPES OF STRATEGIES FOR JUDGING THE QUALITY OF SCIENTIFIC KNOWLEDGE CLAIMS Linda wants to know what to do with her child’s use of tablets and therefore when coming across contradicting claims she has to judge which of these claims she finds more plausible. This is a subjective judgment about the quality of the information with regard to her goal. Therefore, it is an empirical question as to how Linda processes the content as well as provenance information for making such judgments. It is helpful to start with an analytical distinction between two types of strategies: Logically, a citizen who processes science information could try to make a firsthand plausibility judgment by trying to evaluate the competing knowledge claims directly. People engage in such evaluations against the background of their aims and their belief and knowledge systems. Examples of firsthand strategies include weighing an assertion against one’s personal experiences or making a judgment about the evidence provided to support an assertion. A secondhand strategy could be to ask “Whom to trust?” In this case, the immediate question of truth is deferred to someone else—for example, an expert. However, this does not eliminate the challenge of making judgments and of deciding between competing accounts. Only the object of judgments is different because in this case, the trustworthiness of a source has to be established. This analytical distinction is especially relevant when it comes to citizens’ understanding of science. In many fields of scientific expertise, science understanding cannot be built on everyday experience and everyday thinking. A deep understanding of science which would allow for firsthand evaluations of a scientific knowledge claim requires scientific expertise. Acquiring scientific expertise (like any other expertise) requires long and deliberate practice. This does not exclude the possibility of citizens acquiring such expertise even outside of formal educational institutions, for example in citizen science contexts. Nonetheless, the difference between everyday experience and scientific knowledge is an inherent limitation for making firsthand evaluations (Suldovsky, 2016). In this sense, we refer to citizens’ understanding of science as a bounded understanding (Bromme & Goldman, 2014). The boundedness of citizens’ understanding of science matters especially when non-experts encounter conflicting scientific knowledge claims. Often it is possible (given that the citizen has some scientific literacy) to understand a certain claim (e.g., by grasping its gist by means of a good metaphor), but it requires more sophisticated knowledge to evaluate it against a competing, but also understandable, claim. Because of these limitations, the provenance of a knowledge claim is of high importance in the context of the public engagement with science information. Provenance information allows recipients to judge the plausibility of a knowledge claim by helping them to make inferences about whom to trust. In a similar vein, Chinn and Rinehart (2016) recently argued that source information enables recipients to judge the reliability of the processes by which a source has generated their knowledge claims. The terminological shift from judging about truth to judging about plausibility is important here. Within scientific discourse, researchers do not refer to the plausibility or credibility of a theory or a result, but rather to its validity or strength. They do so because as researchers they would principally be capable of checking the strength of evidence. Practically, this is a rather gradual difference because researchers are also bound to trust their colleagues even within their own fields of expertise.

274  •  Bromme et al.

However, for explaining the concept of citizens’ bounded understanding of science, this categorical distinction between plausibility and validity judgments is helpful. Similarly, Lombardi, Nussbaum, and Sinatra (2015) refer to plausibility judgments as judgments of the “potential truthfulness” of a scientific explanation. What Makes a Source Trustworthy? Three features of a source are typically used to evaluate its trustworthiness: expertise, benevolence, and integrity (Hendriks, Kienhues, & Bromme, 2015). First, expertise refers to the amount of knowledge and skill someone possesses. To be trustworthy, the person must also possess the relevant expertise. Hence, the dimension of expertise also encompasses the aspect of pertinence (Bromme & Thomm, 2015). Second, scientists are viewed as trustworthy if they adhere to reliable processes of knowledge production and follow the rules of their discipline. These factors constitute the scientist’s perceived integrity. Third, scientists are considered benevolent if the knowledge and actions they provide are in accordance with the aims of the trustor, or (more generally) if it is for the good of society. In three studies, Hendriks et al. (2015) found that laypeople judged the trustworthiness of scientists by assessing the scientists’ expertise, integrity, and benevolence. Expertise positively affects laypeople’s judgment on the trustworthiness of scientific sources (e.g., Bromme, Stadtler, Scharrer, & Thomm, 2015; Bromme & Thomm, 2015; Kammerer, Amann, & Gerjets, 2015; Lombardi, Seyranian, & Sinatra, 2014; Stadtler, Scharrer, Macedo-Rouet, Rouet, & Bromme, 2016). For example, Bromme, Stadtler et al. (2015) found that undergraduate students who read conflicting online documents about a medical topic perceived sources described as medical experts to be more credible than sources described as laypersons. In addition, Stadtler et al. (2016) showed that vocational students who had been introduced to the division of cognitive labor in society tended to agree more with pertinent experts than with less pertinent experts when deciding about social science controversies. The source’s benevolence and integrity are inferred from assumed pragmatic intentions as well as vested interests (e.g., Bromme, Stadtler et al., 2015; Iding, Crosby, Auernheimer, & Klemm, 2009; Kammerer, Kalbfell, & Gerjets, 2016; Stadtler, Thomm, Babiel, Hentschke, & Bromme, 2013).

THE FUNCTIONS OF PROVENANCE INFORMATION FOR THE PUBLIC ENGAGEMENT WITH SCIENCE: AN EMPIRICAL OVERVIEW Understanding and evaluating scientific information is a complex process whose outcome can manifest on different levels such as readers’ memory for text information, their interpretation of processed content, and their explicit judgment of the plausibility of textual knowledge claims. Considering the provenance of information can affect readers’ performance on all these levels. For experimental as well as practical reasons, nearly all research on sourcing in the context of scientific information has presented readers with only a relatively small, predefined fraction of provenance information—such as the publication date, the author’s affiliation, or the author’s level of expertise—rather than the full amount of information about a document’s origin (e.g., Barzilai, Tzadok, & Eshet-Alkalai, 2015; Bråten, Strømsø, & Andreassen, 2016; Bromme, Stadtler et al., 2015; Kammerer & Gerjets, 2014).

The Provenance of Certainty  •  275

This artificially restricts the choice among pieces of provenance information that readers might find useful. There are also some exceptions to this practice: Anmarkrud, Bråten, and Strømsø (2014) asked undergraduates to read authentic text documents about a controversial scientific issue from diverse publication outlets. To the extent that they had prior knowledge about these outlets, and/or by discerning the text contents for relevant clues, readers could potentially draw on a range of provenance information. Although this approach offers high ecological validity, the lack of experimental control makes it difficult to determine how far readers consider which type of provenance information for their trustworthiness evaluations. Clearly, a balance is required between strongly controlled studies examining readers’ use of particular types of provenance information and the need to account for other influences and maximize ecological validity by providing a broad range of provenance information. To gain an overview of the state of knowledge on laypeople’s use of provenance information when processing scientific information, we shall review empirical studies that have focused on how far this kind of information contributes to laypeople’s evaluation of plausibility and understanding. Specifically, we shall look at how provenance information affects laypeople’s plausibility evaluation and understanding at different levels, and how primarily content-based factors may determine or codetermine both outcomes. Hence, the focus of our review is on the question whether and under which conditions readers of scientific information consider provenance information. The studies reviewed here investigate laypeople who engage with scientific information, even though not all studies identify themselves as research on the public engagement with science.

THE ROLE OF SOURCE INFORMATION IN LAYPEOPLE’S EVALUATION OF PLAUSIBILITY A large body of studies from diverse fields including reading comprehension, persuasion, education, health information seeking, and information technologies has focused on laypeople’s consideration of provenance information when making plausibility evaluations of scientific information. As mentioned above, evaluation can manifest on different levels, and accordingly, researchers differ in their focus on these levels and their operationalization of plausibility evaluation—although many studies investigate multiple levels simultaneously. One level is laypeople’s explicit judgment of the plausibility of knowledge claims advocated by a source and their justification of such judgments. Research has shown that under some conditions laypeople consider source information when deciding about the plausibility of scientific knowledge claims from single or multiple documents and they tend to agree more with claims from more credible than less credible sources (e.g., Bromme, Scharrer, Stadtler, Hömberg, & Torspecken, 2015; Hahn, Harris, & Corner, 2009; Kammerer et al., 2015; Kobayashi, 2014; Lombardi, Seyranian, & Sinatra, 2014; Regan, Shan, McConnon, Marcu, Raats, Wall, & Barnett, 2014; Stadtler & Bromme, 2007, 2008; Stadtler, Paul, Globoschütz, & Bromme, 2015; Stadtler et  al., 2016; Thomm & Bromme, 2012). For example, Bromme, Scharrer et al. (2015) asked undergraduates to read a set of conflicting documents about a medical topic. Some of these documents contained elements typical of scientific discourse (e.g., citations), whereas others did not. Readers rated the documents with discourse-typical elements as more scientific, thus inferring provenance

276  •  Bromme et al.

information from the document contents, and they expressed more agreement with the claims contained in these documents. The extent to which laypeople consider provenance information depends on specific enabling conditions (Braasch, Bråten, Strømsø, Anmarkrud, & Ferguson, 2013; Kammerer et al., 2016; Mason, Junyent, & Tornatora, 2014). For example, in an intervention study by Braasch et al. (2013), secondary school students had to rank six documents about El Niño taken from different genres in terms of their usefulness. Students in the intervention group (who received a short classroom-based training that comprised contrasting and evaluating different sets of strategies for approaching multiple documents) discriminated more clearly between more and less useful documents and justified their document rankings by referring to relevant source features such as author credentials. Finally, evaluation of plausibility can manifest more indirectly in the effort that laypeople invest in processing scientific information. Laypeople have been found to select documents by trustworthy sources more often and spend more time reading these documents compared with documents from less trustworthy sources (e.g., Kammerer et  al., 2015; Kammerer & Gerjets, 2012; McCrudden, Stenseth, Bråten, & Strømsø, 2016; Winter & Krämer, 2012, 2014). Provenance information usually has to compete with other kinds of information in shaping laypeople’s evaluation. Research indicates that laypeople consider features related to the quality of the content such as the strength of arguments and claim-supporting evidence (Brem, Russell, & Weems, 2001; Hahn et al., 2009; Kiili, Laurinen, & Marttunen, 2008; Regan et al., 2014), the level of precision and detail of the content (Brem et al., 2001), the “message sidedness” (Winter & Krämer, 2012), the perceived tentativeness of the reported scientific findings (Flemming, Feinkohl, Cress, & Kimmerle, 2015), the extent to which information is consistent with information from other documents (Anmarkrud et al., 2014; Rowley, Johnson, & Sbaffi, 2015), the credibility of the content and coherence with their own prior knowledge (Brem et al., 2001; Rowley et al., 2015), and their prior attitude toward the issue (Regan et al., 2014; van Strien, Brand-Gruwel, & Boshuizen, 2014). People may read science-based statements that conflict with their beliefs and then shift their attention to the sources of the statements. This could also work in the other direction. The appreciation of the source may be modified in the light of how their message is evaluated (Landrum, Lull, Akin, Hasell, & Jamieson, 2017). In addition, laypeople consider surface features of the text document such as its position on a search engine results page (Eysenbach & Köhler, 2002; Kammerer & Gerjets, 2014), text length (Barzilai & Zohar, 2012; Lucassen, Muilwijk, Noordzij, & Schraagen, 2013), style and design (Barzilai & Zohar, 2012; Kiili et al., 2008; Lucassen et al., 2013; Rowley et al., 2015), the inclusion of images (Lucassen et al., 2013), and the relevance and usefulness of the information for achieving their reading goal (Anmarkrud et  al., 2014; Barzilai & Zohar, 2012; McCrudden et  al., 2016; Rowley et al., 2015; Salmerón, Kammerer, & Garcia-Carrion, 2013). Frequently, these kinds of information receive more weight in laypeople’s evaluations than information provenance (e.g., Barzilai & Zohar, 2012; Bråten, Braasch, Strømsø, & Ferguson, 2015; Bråten et al., 2016; Bråten, Strømsø, & Salmerón, 2011; Brem et al., 2001; Kobayashi, 2014). In fact, empirical findings show that laypeople often do not consider provenance information spontaneously when evaluating plausibility, or they even consider it in an inappropriate way (Barzilai & Zohar, 2012; Eysenbach & Köhler, 2002; Stadtler et al., 2016; Walraven, Brand-Gruwel, & Boshuizen, 2009).

The Provenance of Certainty  •  277

As mentioned previously, whether and how far laypeople base their evaluations on provenance appears to depend on a variety of factors including characteristics of the text and context, whether or not a reader has received a specific intervention, and a variety of reader characteristics including their judgment of their own understanding and evaluation capabilities. Among text and context characteristics, research has shown that laypeople pay more attention to provenance information in their evaluative judgments and information selection under the following conditions: (a) when encountering conflicting rather than nonconflicting scientific text documents; (b) when the salience of source information is high; and (c) when the display format of web search results does not allow an easy position ranking (Braasch & Bråten, 2017; Kammerer & Gerjets, 2012; Kammerer & Gerjets, 2014; Kammerer et al., 2016; Salmerón, Gomez, & Fajardo, 2016). Further studies have shown the use of provenance information to be affected by readers’ familiarity with the topic (McCrudden et al., 2016), by their epistemic cognition and Internet-specific epistemic beliefs (Bråten, Ferguson, Strømsø, & Anmarkrud, 2014; Kammerer & Gerjets, 2012; Kammerer, Bråten, Gerjets, & Strømsø, 2013; Kammerer et al., 2015), or by intellectual disability (Salmerón et al., 2016). For example, a study by McCrudden et al. (2016) showed that secondary students who selected web documents for an academic task considered author expertise to be more important when their familiarity with the scientific topic was low than when their topic familiarity was high. The authors argued that in the event of high topic familiarity, students relied more on firsthand evaluation whereas in the event of low topic familiarity, they resorted more extensively to secondhand evaluation. Regarding the observed effect of topic familiarity, empirical evidence suggests that laypeople’s judgments of their own understanding and evaluation capabilities have a crucial influence on their consideration of provenance information. Factors that enhance laypeople’s impression of having obtained a fairly complete understanding of the topic that will enable them to make content-based judgments may increase their reliance on firsthand evaluation and conversely decrease their inclination to take provenance information into account. One such factor is the ease of information processing. A series of studies has shown that easily understanding scientific contents induces laypeople to rely more on their own judgment of scientific claims (Scharrer, Britt, Stadtler, & Bromme, 2013; Scharrer, Bromme, Britt, & Stadtler, 2012; Scharrer, Rupieper, Stadtler, & Bromme, 2016; Scharrer, Stadtler, & Bromme, 2014). In addition, laypeople’s attempt to explain causal phenomena affects their metacognitive awareness of gaps in their own topic knowledge and can reduce overly optimistic assessments of their depth of understanding (Bromme, Thomm, & Ratermann, 2016; Mills & Keil, 2004; Rozenblit & Keil, 2002). In the same way they evaluate external sources for their ability to advocate reliable knowledge claims (Chinn & Rinehart, 2016; Chinn, Rinehart, & Buckland, 2014), laypeople need to critically evaluate their own ability to recognize such claims.

THE ROLE OF SOURCE INFORMATION IN LAYPEOPLE’S UNDERSTANDING OF SCIENTIFIC INFORMATION Laypeople’s engagement with scientific information involves not only evaluation of plausibility but also forming an understanding of the presented subject matter.

278  •  Bromme et al.

Provenance information can help laypeople make sense of scientific information and build a differentiated mental representation by helping them to anticipate and interpret the content and to draw relationships between content from different sources (Britt & Rouet, 2012; Perfetti et  al., 1999; cf. Wineburg, 1991). Although we distinguish between plausibility evaluation and understanding as two outcomes in the public engagement with science, there is a logical interplay between the two: The results of plausibility evaluation may affect understanding and vice versa; for example, when information deemed implausible is not processed deeply and thus not included in the mental representation or when the interpretation of information affects its perceived plausibility. As a result, any distinction between empirical findings on the role of sourcing in understanding and plausibility evaluation must remain tentative. Analogous to plausibility evaluations, provenance information may affect laypeople’s understanding of scientific information on different levels. Understanding may manifest on the level of laypeople’s memory for content information, which is often assessed with sentence- and inference-verification tasks. Empirical research suggests that source information affects laypeople’s content memory, and specifically, that sourcing improves memory performance (e.g., Bråten, Strømsø, & Britt, 2009; Keck, Kammerer, & Starauschek, 2015; Mason et al., 2014; Strømsø, Bråten, & Britt, 2010). For example, Bråten et al. (2009) asked undergraduate students to read a set of partly conflicting documents about climate change and found that students’ sourcing performance positively predicted their document comprehension as assessed by sentence- and inference-verification tasks. The authors speculated that students who engaged in sourcing processed the documents they believed to be trustworthy in more depth, and/or built more elaborate document models that allowed for an accurate representation of conflicting views (as described by Perfetti et al., 1999). Understanding may furthermore manifest in laypeople’s interpretation of processed content. Studies show that provenance information may help laypeople understand an author’s viewpoint or make sense of textual inconsistencies (Barzilai & Eshet-Alkalai, 2015; Stadtler, Scharrer, & Bromme, 2013; Thomm & Bromme, 2016; Thomm et al., 2015). For instance, Thomm and Bromme (2016) showed that undergraduates who read two conflicting texts from a university researcher and a researcher in industry attributed the conflict more to the authors’ personal motivation than readers encountering a conflict between two university researchers. Finally, understanding may manifest in laypeople’s communication of learned contents. A number of studies have assessed such communication by asking readers of scientific information to write essays about the target topic. Their findings suggest an impact of provenance information on essay quality (e.g., Barzilai et al., 2015; Braasch et  al., 2013; Mason et  al., 2014; Stadtler, Scharrer, Brummernhenrich, & Bromme, 2013; Wiley, Goldman, Graesser, Sanchez, Ash, & Hemmerich, 2009). Apart from provenance information, characteristics of the text, context, and reader affect laypeople’s understanding of scientific information, and the influence of these factors may complement or compete against effects of provenance information. The influence of provenance information on laypeople’s understanding appears to vary with numerous factors such as textual consistency (Steffens, Britt, Braasch, Strømsø, & Bråten, 2014), readers’ epistemic perspective and epistemic cognition (Barzilai et al.,

The Provenance of Certainty  •  279

2015; Bråten et al., 2014), and readers’ declarative knowledge about criteria for evaluating scientific online information (Mason et al., 2014; Wiley et al., 2009).

FUTURE DIRECTIONS With regard to theory building about sourcing, we argue that the notion of provenance as defined here is well suited to reflect the broad range of information that readers use to understand science claims and to judge their plausibility. The review of empirical findings shows that readers rely on a variety of different cues. They consider metatextual information (e.g., information about the credentials of the author) and information that can be found only in the text (e.g., genre-related style characteristics) to infer that the text has a scientific provenance. Further, readers make use of pragmatic cues (who said what because of which assumed intentions?; Hendriks et al., 2016). Such cues are a good example for our main argument: Any information about the genesis of a certain claim could be used for establishing understanding and truth. Therefore, we have suggested the notion of provenance in order to theoretically capture the functional role of all kinds of information about the circumstances under which a certain piece of knowledge has been produced and communicated. As stated above, the balance among the studies reviewed in this chapter is skewed toward research investigating the impact of selected pieces of provenance information. There is a need for studies that provide readers with a wider, naturalistic range of provenance information. This will not only inform us about whether and when this information is considered, but also extend our knowledge about the types of provenance information that facilitate understanding and evaluation. Science communication is becoming increasingly digital as well as interactive; blogs, tweets, and net-based discussion forums enable citizens to actively participate in communication about scientific issues. This widening of the circle of potential communicators makes adequate inferences about the authors’ pragmatic intentions and capabilities even more important. Readers cannot ignore such author characteristics, and therefore the processing of provenance information, especially of pragmatic aspects, will be an important topic for future research. This development will open new venues for research and especially for studies with a naturalistic range of provenance information. Internet-based written interactions produce a stream of data which are already available in a digital format and could be analyzed more directly than, for example, a video recording of citizens’ discussing with scientists. Of course, the sheer amount as well as the special kind of such data will ask for new analytical methods, for example for the automated analysis of social media responses to contributions from scientists (Jensen, 2015). These developments also ask for a refined conceptual understanding of what is conceived as source and as sourcing, which is why we have suggested the concept of provenance. The studies reviewed in this chapter suggest that people make use of all kinds of cues about the provenance of a certain piece of knowledge but this is not necessarily done in a skilled way. Instead, targeted interventions are necessary. Such measures include raising laypeople’s awareness of the division of cognitive labor in society, the lack of editorial control on the Internet, and the resulting need to evaluate sources (Stadtler et al., 2015, 2016); discussing the need for and practicing the consideration of source

280  •  Bromme et al.

information in evaluative judgments (Macedo-Rouet, Braasch, Britt, & Rouet, 2013), prompting laypeople to evaluate the credibility of sources (Stadtler & Bromme, 2007, 2008), and providing declarative knowledge of important criteria for the evaluation of scientific information including source information (Mason et al., 2014; Wiley et al., 2009), instructing students about the specific features of genres of science communication (Yarden, Norris, & Phillips, 2015), or providing examples of how to use such criteria in practice (Braasch et al., 2013).

AUTHORS’ NOTE Many thanks to Jonathan Harrow for English language checking, as well as to Sarit Barzilai, to the reviewers and to the editors for helpful comments on earlier versions of this chapter.

REFERENCES Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. Barzilai, S., & Eshet-Alkalai, Y. (2015). The role of epistemic perspectives in comprehension of multiple author viewpoints. Learning and Instruction, 36, 86–103. doi:10.1016/learninstruc.2014.12.003. Barzilai, S., Tzadok, E., & Eshet-Alkalai, Y. (2015). Sourcing while reading divergent expert accounts: Pathways from views of knowing to written argumentation. Instructional Science, 43, 737–766. doi:10.1007/ s11251-015-9359-4. Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online sources. Cognition and Instruction, 30, 39–85. doi:10.1080/07370008.2011.636495. Braasch, J. L. G., & Bråten, I. (2017). The Discrepancy-Induced Source Comprehension (D-ISC) Model: Basic assumptions and preliminary evidence. Educational Psychologist, 52, 167–181. https://doi.org/10.1080/00 461520.2017.1323219. Braasch, J. L. G., Bråten, I., Strømsø, H., Anmarkrud, Ø., & Ferguson, L. E. (2013). Promoting secondary school students’ evaluation of source features of multiple documents. Contemporary Educational Psychology, 38, 180–195. doi:10.1016/j.bbr.2011.03.031. Bråten, I., Braasch, J. L. G., Strømsø, H. I., & Ferguson, L. E. (2015). Establishing trustworthiness when students read multiple documents containing conflicting scientific evidence. Reading Psychology, 36, 315–349. doi: 10.1080/02702711.2013.864362. Bråten, I., Ferguson, L. E., Strømsø, H. I., & Anmarkrud, O. (2014). Students working with multiple conflicting documents on a scientific issue: Relations between epistemic cognition while reading and sourcing and argumentation in essays. British Journal of Educational Psychology, 84, 58–85. doi:10.1111/bjep.12005. Bråten, I., Strømsø, H. I., & Andreassen, R. (2016). Sourcing in professional education: Do text factors make any difference? Reading and Writing, 29, 1599–1628. doi:10.1007/s11145-015-9611-y. Bråten, I., Strømsø, H. I., & Britt, M. A. (2009). Trust matters: Examining the role of source evaluation in students’ construction of meaning within and across multiple texts. Reading Research Quarterly, 44, 6–28. doi:10.1598/RRQ.41.1.1. Bråten, I., Strømsø, H. I., & Salmerón, L. (2011). Trust and mistrust when students read multiple information sources about climate change. Learning and Instruction, 21, 180–192. Brem, S. K., Russell, J., & Weems, L. (2001). Science on the web: Student evaluations of scientific arguments. Discourse Processes, 32, 191–213. doi:10.1080/0163853X.2001.9651598. Britt, M. A., & Rouet, J. F. (2012). Learning with multiple documents: Component skills and their acquisition. In J. R. Kirby & M. J. Lawson (Eds.), Enhancing the quality of learning: Dispositions, instruction, and learning processes (pp. 276–314). New York, NY: Cambridge University Press. Bromme, R., & Goldman, S. (2014). The public’s bounded understanding of science. Educational Psychologist, 49, 59–69. doi:10.1080/00461520.2014.921572.

The Provenance of Certainty  •  281 Bromme, R., & Thomm, E. (2015). Knowing who knows: Laypersons’ capabilities to judge experts’ pertinence for science topics. Cognitive Science, 1–12. doi:10.1111/cogs.12252. Bromme, R., Thomm, E., & Ratermann, K. (2016). Who knows? Explaining impacts on the assessment of our own knowledge and of the knowledge of experts. Zeitschrift für Pädagogische Psychologie, 30, 97–108. Bromme, R., Scharrer, L., Stadtler, M., Hömberg, J., & Torspecken, R. (2015). Is it believable when it’s scientific? How scientific discourse style influences laypeople’s resolution of conflicts. Journal of Research in Science Teaching, 52, 36–57. doi:10.1002/tea.21172. Bromme, R., Stadtler, M., Scharrer, L., & Thomm, E. (2015). A scientist through and through? How the source’s commitment to science affects readers’ evaluation of source and content in the domain of medicine. Paper presented at the annual meeting of the Society for Text and Discourse, Minneapolis, USA. Chinn, C. A., & Rinehart, R. W. (2016). Commentary: Advances in research on sourcing—source credibility and reliable processes for producing knowledge claims. Reading & Writing, 29, 1701. doi:10.1007/ s11145-016-9675-3. Chinn, C. A., Rinehart, R. W., & Buckland, L. A. (2014). Epistemic cognition and evaluating information: Applying the AIR model of epistemic cognition. In D. Rapp and J. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 425–453). Cambridge, MA: MIT Press. Eysenbach, G., & Köhler, C. (2002). How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. British Medical Journal, 324, 573–577. doi:10.1136/bmj.324.7337.573. Flemming, D., Feinkohl, I., Cress, U., & Kimmerle, J. (2015). Individual uncertainty and the uncertainty of science: The impact of perceived conflict and general self-efficacy on the perception of tentativeness and credibility of scientific information. Frontiers in Psychology, 6. doi:10.3389/Fpsyg.2015.01859. Goldacre, B. (2008). Bad science. New York, NY: Faber and Faber. Goldman, S., & Scardamalia, M. (2013). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities, Cognition and Instruction, 31 255–269. doi:10.1080/10824669.2013.773217. Hahn, U., Harris, A. J. L., & Corner, A. (2009). Argument content and argument source: An exploration. Informal Logic, 29, 337–367. Hendriks, F., Kienhues, D., & Bromme, R. (2015). Measuring laypeople’s trust in experts in a digital age: The Muenster Epistemic Trustworthiness Inventory (METI). PLoS ONE 10: e0139309. doi:10.1371/journal. pone.0139309. Hendriks, F., Kienhues, D., & Bromme, R. (2016). Disclose your flaws! Admission positively affects the perceived trustworthiness of an expert science blogger. Studies in Communication Sciences, 16, 124–131. doi:10.1016/j.scoms.2016.10.003. Iding, M. K., Crosby, M. E., Auernheimer, B., & Klemm, B. E. (2009). Web site credibility: Why do people believe what they believe? Instructional Science, 37, 43–63. doi:10.1007/s11251-008-9080-7. Jensen, E. (2015). Evaluating impact and quality of experience in the 21st century: Using technology to narrow the gap between science communication research and practice. Journal of Science Communication, 14, 1–9. Kammerer, Y., Amann, D. G., & Gerjets, P. (2015). When adults without university education search the Internet for health information: The roles of Internet-specific epistemic beliefs and a source evaluation intervention. Computers in Human Behavior, 48, 297–309. doi:10.1016/j.chb.2015.01.045. Kammerer, Y., Bråten, I., Gerjets, P., & Strømsø, H. I. (2013). The role of Internet-specific epistemic beliefs in laypersons’ source evaluations and decisions during Web search on a medical issue. Computers in Human Behavior, 29, 1193–1203. doi:10.1016/j.chb.2012.10.012. Kammerer, Y., & Gerjets, P. (2012). Effects of search interface and Internet-specific epistemic beliefs on source evaluations during Web search for medical information: An eye-tracking study. Behaviour & Information Technology, 31, 83–97. doi:10.1080/0144929X.2011.599040. Kammerer, Y., & Gerjets, P. (2014). The role of search result position and source trustworthiness in the selec­ tion of web search results when using a list or a grid interface. International Journal of Human–Computer Interaction, 30, 177–191. doi:10.1080/10447318.2013.846790. Kammerer, Y., Kalbfell, E., & Gerjets, P. (2016). Is this information source commercially biased? How con­ tradictions between web pages stimulate the consideration of source information. Discourse Processes, 53, 430–456. doi:10.1080/0163853X.2016.1169968.

282  •  Bromme et al. Keck, D., Kammerer, Y., & Starauschek, E. (2015). Reading science texts online: Does source information influence the identification of contradictions within texts? Computers & Education, 82, 442–449. doi:org/10.1016/j.compedu.2014.12.005. Kiili, C., Laurinen, L., & Marttunen, M. (2008). Students evaluating Internet sources: From versatile evaluators to uncritical readers. Journal of Educational Computing Research, 39, 75–95. doi:10.2190/Ec.39.1.E. Kobayashi, K. (2014). Students’ consideration of source information during the reading of multiple texts and its effect on intertextual conflict resolution. Instructional Science, 42, 183–205. doi:10.1007/s11251-013-9276-3. Kolstø, S. D., Bungum, B., Arnesen, E., Isnes, A., Kristensen, T., Mathiassen, K., Mestad, Quale, A., Tonning, A., & Ulvik, M. (2006). Science students’ critical examination of scientific information related to socioscientific issues. Science Education 90, 632–655. doi:10.1002/sce.20133. Landrum, A. R., Lull, R. B., Akin, H., Hasell, A., & Jamieson, K. H. (2017). Processing the papal encyclical through perceptual filters: Pope Francis, identity-protective cognition, and climate change concern. Cognition (online first). doi:10.1016/j.cognition.2017.05.015. Lombardi, D., Nussbaum, E. M., & Sinatra, G. M. (2015). Plausibility judgments in conceptual change and epistemic cognition. Educational Psychologist, 51, 35–56. doi:10.1080/00461520.2015.1113134. Lombardi, D., Seyranian, V., & Sinatra, G. M. (2014). Source effects and plausibility judgments when reading about climate change. Discourse Processes, 51, 75–92. doi:10.1080/0163853X.2013.855049. Lucassen, T., Muilwijk, R., Noordzij, M. L., & Schraagen, J. M. (2013). Topic familiarity and information skills in online credibility evaluation. Journal of the American Society for Information Science and Technology, 64, 254–264. doi:10.1002/asi.22743. Mason, L., Junyent, A. A., & Tornatora, M. C. (2014). Epistemic evaluation and comprehension of web-source information on controversial science-related topics: Effects of a short-term instructional intervention. Computers & Education, 76, 143–157. doi:10.1016/j.compedu.2014.03.016. McCrudden, M. T., Stenseth, T., Bråten, I., & Strømsø, H. I. (2016). The effects of topic familiarity, author expertise, and content relevance on Norwegian students’ document selection: A mixed methods study. Journal of Educational Psychology, 108, 147–162. doi:10.1037/edu0000057. Mills, C. M., & Keil, F. C. (2004). Knowing the limits of one’s understanding: The development of an awareness of an illusion of explanatory depth. Journal of Experimental Child Psychology, 87, 1–32. Munro, G. D. (2010). The scientific impotence excuse: Discounting belief-threatening scientific abstracts. Journal of Applied Social Psychology, 40, 579–600. Nauroth, P., Gollwitzer, M., Bender, J., & Rothmund, T. (2015). Social identity threat motivates sciencediscrediting online comments. PLoS ONE, 10: e0117476. doi:10.1371/journal.pone.0117476. Perfetti, C. A., Rouet, J.-F., & Britt, M. A. (1999). Toward a theory of documents representation. In H. Van Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 99–122). Mahwah, NJ: Erlbaum. Regan, Á., Shan, L., McConnon, Á., Marcu, A., Raats, M., Wall, P., & Barnett, J. (2014). Strategies for dismissing dietary risks: insights from user–generated comments online. Health, Risk & Society, 16, 308–322. doi:10. 1080/13698575.2014.919993. Rowley, J., Johnson, F., & Sbaffi, L. (2015). Students’ trust judgements in online health information seeking. Health Informatics Journal, 21, 316–327. Rozenblit, L., & Keil, F. (2002). The misunderstood limits of folk science: An illusion of explanatory depth. Cognitive Science, 26, 521–562. Salmerón, L., Gomez, M., & Fajardo, I. (2016). How students with intellectual disabilities evaluate recommendations from Internet forums. Reading and Writing, 29, 1653–1675. doi:10.1007/s11145-016-9621-4. Salmerón, L., Kammerer, Y., & Garcia-Carrion, P. (2013). Searching the Web for conflicting topics: Page and user factors. Computers in Human Behavior, 29, 2161–2171. doi:10.1016/j.chb.2013.04.034. Scharrer, L., Britt, M. A., Stadtler, M., & Bromme, R. (2013). Easy to understand but difficult to decide: Information comprehensibility and controversiality affect laypeople’s science-based decisions. Discourse Processes, 50, 361–387. Scharrer, L., Bromme, R., Britt, M. A., & Stadtler, M. (2012). The seduction of easiness: How science depictions influence laypeople’s reliance on their own evaluation of scientific information. Learning and Instruction, 22, 231–243. Scharrer, L., Rupieper, Y., Stadtler, M., & Bromme, R. (2016). When science becomes too easy: Science popularization inclines laypeople to underrate their dependence on experts. Public Understanding of Science, 26(8), 1003–1018.

The Provenance of Certainty  •  283 Scharrer, L., Stadtler, M., & Bromme, R. (2014). You’d better ask an expert: Mitigating the comprehensibility effect on laypeople’s decisions about science-based knowledge claims. Applied Cognitive Psychology, 28, 465–471. Stadtler, M. (2017). The art of reading in a knowledge society. Commentary on the special issue on models of multiple text comprehension. Educational Psychologist, 52, 225–231. Stadtler, M., & Bromme, R. (2007). Dealing with multiple documents on the WWW: The role of metacognition in the formation of documents models. International Journal of Computer-Supported Collaborative Learning, 2, 191–210. doi:10.1007/s11412-007-9015-3. Stadtler, M., & Bromme, R. (2008). Effects of the metacognitive computer-tool met.a.ware on the web search laypersons. Computers in Human Behavior, 24, 716–737. doi:10.1016/j.chb.2007.01.023. Stadtler, M., & Bromme, R. (2014). The content–source integration model: A taxonomic description of how readers comprehend conflicting scientific information. In D. N. Rapp & J. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 379–402). Cambridge, MA: MIT Press Stadtler, M., Paul, J., Globoschütz, S., & Bromme, R. (2015). Watch out!: An instruction raising students’ epistemic vigilance augments their sourcing activities. In D. C. Noelle, R. Dale, A. S. Warlaumont, J. Yoshimi, T. Matlock, C. D. Jennings, & P. P. Maglio (Eds.), Proceedings of the 37th annual conference of the Cognitive Science Society (pp. 2278–2283). Austin, TX: Cognitive Science Society. Stadtler, M., Scharrer, L., & Bromme, R. (2013). How do readers explain the occurrence of conflicts in science texts? Effects of presentation format and source expertise. In M. Knauff, N. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th annual conference of the Cognitive Science Society (pp. 3448–3453). Austin, TX: Cognitive Science Society. Stadtler, M., Scharrer, L., Brummernhenrich, B., & Bromme, R. (2013). Dealing with uncertainty: Readers’ memory for and use of conflicting information from science texts as function of presentation format and source expertise. Cognition and Instruction, 31, 130–150. Stadtler, M., Scharrer, L., Macedo-Rouet, M., Rouet, J.-F., & Bromme, R. (2016). Improving vocational students’ consideration of source information when deciding about science controversies. Reading and Writing, 29, 705–729. doi:10.1007/s11145-016-9623-2. Stadtler, M., Thomm, E., Babiel, S., Hentschke, J., & Bromme, R. (2013). Ignorant albeit competent: Examining students’ sourcing competencies and spontaneous use of source information while reading conflicting scientific texts. Paper presented at the workshop on multiple document literacy, Muenster, Germany. Steffens, B., Britt, M. A., Braasch, J. L., Strømsø, H., & Bråten, I. (2014). Memory for scientific arguments and their sources: Claim–evidence consistency matters. Discourse Processes, 51, 117–142. doi:10.1080/01638 53X.2013.855868. Strømsø, H. I., Bråten, I., & Britt, M. A. (2010). Reading multiple texts about climate change: The relationship between memory for sources and text comprehension. Learning and Instruction, 20, 192–204. doi:10.1016/j. learninstruc.2009.02.001. Suldovsky, B. (2016). In science communication, why does the idea of the public deficit always return? Exploring key influences. Public Understanding of Science, 25, 415–426. doi:10.1177/0963662516629750. Tabak, J. (2015). Functional scientific literacy: Seeing the science within the words and across the web. In L. Corno & E. M. Anderman (Eds.), Handbook of educational psychology (3rd ed., pp. 269–280). London: Routledge. Thomm, E., & Bromme, R. (2012). “It should at least seem scientific!”: Textual features of “scientificness” and their impact on lay assessments of online information. Science Education, 96, 197–2011. Thomm, E., & Bromme, R. (2016). How source information shapes lay interpretations of science conflicts: interplay between sourcing, conflict explanation, source evaluation, and claim evaluation. Reading and Writing, 29, 1629–1652. doi:10.1007/s11145-016-9638-8. Thomm, E., Hentschke, J., & Bromme, R. (2015). The Explaining Conflicting Scientific Claims (ECSC)questionnaire: Measuring laypersons’ explanations for conflicts in science. Learning and Individual Differences, 37, 139–152. Van Strien, J. L. H., Brand-Gruwel, S., & Boshuizen, H. P. A. (2014). Dealing with conflicting information from multiple nonlinear texts: Effects of prior attitudes. Computers in Human Behavior, 32, 101–111. doi:10.1016/j.chb.2013.11.021. Walraven, A., Brand-Gruwel, S., & Boshuizen, H. P. A. (2009). How students evaluate information and sources when searching the World Wide Web for information. Computers & Education, 52, 234–246. doi:10.1016/j. compedu.2008.08.003.

284  •  Bromme et al. Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009). Source evaluation, comprehension, and learning in Internet science inquiry tasks. American Educational Research Journal, 46, 1060–1106. doi:10.3102/0002831209333183. Wineburg, S. S. (1991). Historical problem-solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87. doi:10.1037//0022-0663.83.1.73. Winter, S., & Krämer, N. C. (2012). Selecting science information in web 2.0: How source cues, message sidedness, and need for cognition influence users’ exposure to blog posts. Journal of Computer-Mediated Communication, 18, 80–96. doi:10.1111/j.1083-6101.2012.01596.x. Winter, S., & Krämer, N. C. (2014). A question of credibility: Effects of source cues and recommendations on information selection on news sites and blogs. Communications: The European Journal of Communication Research, 39, 435–456. doi:10.1515/commun-2014-0020. Yarden, A., Norris, S. P., & Phillips, L. (2015). Adapted primary literature: The use of authentic scientific texts in secondary schools. Dordrecht: Springer.

17

NON-ACADEMIC MULTIPLE SOURCE USE ON THE INTERNET Ladislao Salmerón university of valencia, spain

Yvonne Kammerer leibniz-institut für wissensmedien, tuebingen, germany

Pablo Delgado university of valencia, spain

INTRODUCTION The Internet is a network that enables individuals worldwide to comfortably and instantly retrieve an unprecedented amount of information on almost any topic. Individuals commonly use the Internet for formal research activities in educational contexts (e.g., Salmerón, Strømsø, Kammerer, Stadtler, & van den Broek, in press); however, they also use it to make informed decisions and to seek knowledge and advice about personal, medical, environmental, political, and financial issues (Estabrook, Witt, & Rainie, 2007; Smith, 2009). In this chapter, we focus on the use of multiple online sources to seek information in such informal or non-academic contexts. Interacting with multiple sources on the Internet presents different challenges than interacting with traditional print documents (Bråten, Stadtler, & Salmerón, in press). First, a greater number of information sources are more readily accessible. Accordingly, readers themselves are responsible for locating and selecting a manageable subset of potentially useful information sources for further exploration (Alexander, 2012). Further, a document on the Internet often consists of dense network of linked documents; virtually any interaction with multiple sources on the Internet involves some degree of navigation across linked documents. Second, documents on the Internet do not necessarily pass through an editorial filter, as is the case with traditional printed texts. Documents on the Internet seldom

285

286  •  Salmerón et al.

have explicit editorial review policies or undergo quality control. On the contrary, access to documents is normally granted by search engines that impose their own criteria for displaying the results to users, in ways that do not always distinguish between expert and non-expert accounts of information. As such, online information sources can often vary with respect to the credibility of the source and the reliability of the information provided. Further, readers are provided limited information for evaluating the credibility of a source (Rieh, 2002; Wirth, Böcking, Karnowski, & von Pape, 2007). Thus, readers themselves are responsible for “gatekeeping”; that is, evaluating the quality of information found online (Britt & Aglinskas, 2002), and there is often limited information available for making such evaluations. Third, interacting with documents on the Internet in non-academic contexts most often involves some degree of social interaction, in ways not present in traditional print reading. Users can share and access documents on their social networks, circumventing the potential gatekeeping of search engines. Given that people tend to share information on their social networks that is similar to their own views (Barberá, Jost, Nagler, Tucker, & Bonneau, 2015), users may be selectively exposed to a rather narrow set of views on relevant topics, which is ironic given the existence of innumerable information sources on the Internet. And, given that users tend to uncritically trust information shared by their social mates, they may end up making decisions based on personal accounts rather than on expert information (Betsch, Renkewitz, & Haase, 2013). Also, they can comment (or read comments) on existing documents, creating a sort of dialogue with the author and other readers. Such interactions can yield different meanings than those originally intended by the author (Anderson, Brossard, Scheufele, Xenos, & Ladwig, 2014). The remainder of this chapter is organized into three main sections. In the first, we discuss models relevant to understanding non-academic use of multiple sources on the Internet. Then, we review scientific literature on how people locate, access, and evaluate information on the Internet. Finally, we summarize the outcome of our review and suggest areas for future research.

THEORETICAL BACKGROUND Several theories on multiple source use on the Internet have been developed in different disciplines, including educational psychology, communication science, and computer science. We first discuss information problem-solving models; models that help explain how we interact with multiple sources on the Internet. Then, we describe models that address two main facets of the interaction with multiple sources: (1) locating and navigating web pages, and (2) evaluating information on the Internet. Information Problem-Solving Models The process of information seeking on the Internet has been conceptualized as a problem-solving process driven by an information problem (e.g., Brand-Gruwel et al., 2009; Wilson, 1999; Wopereis, Brand-Gruwel, & Vermetten, 2008). An information problem is a problem that cannot be answered from memory; it requires a search for external information (cf. Walraven, Brand-Gruwel, & Boshuizen, 2009). Information problems

Non-Academic Multiple Source Use  •  287

can be simple, like finding a specific fact (i.e., fact-finding tasks), or complex, like seeking information to support an argument or making an informed decision about conflicting topics (Kellar, Watters, & Shepherd, 2007). Whereas the former usually involves one correct answer, the latter is often characterized by fragile and conflicting evidence. Locating a specific fact often requires identifying a single source that contains the correct answer. However, when solving a complex information problem, individuals need to collect, compare, and integrate information from multiple sources (Kellar et al., 2007). A range of models describe the information seeking processes (via search engines) as the unfolding of an iterative sequence of steps (e.g., Brand-Gruwel, Wopereis, & Vermetten, 2005; Brand-Gruwel et  al., 2009; Gerjets, Kammerer, & Werner, 2011; Rouet et al., 2011). Overall, this sequence consists of five main steps: (1) identifying and defining an information problem and generating respective search terms from information represented in memory; (2) locating information sources (i.e., web pages) by evaluating and selecting links from search engine result pages (SERPs); (3) scanning and briefly evaluating the information presented in web pages; (4) thoroughly processing and extracting content from web pages identified as useful; and (5) comparing, integrating, and synthesizing information from several sources to solve the information problem. During such information seeking processes, Rieh and Hilligoss (2008) distinguish three evaluation phases: (1) predictive judgments about the usefulness of available documents (before accessing an information source), (2) evaluative judgments about the information source accessed (whether the predictive judgment is met), and (3) verification, re-evaluating the information after having accessed multiple information sources (e.g., after having encountered discrepancies between sources). Navigation Models In contrast to these descriptive models, several computational cognitive models have been developed (e.g., SNIF-ACT 2.0 by Fu & Pirolli, 2007; CoLiDeS+ by Juvina & Van Oostendorp, 2008; MESA by Miller & Remington, 2004) that explain and predict navigation behavior on the Internet based on the Information Foraging Theory (Pirolli, 2007). According to the Information Foraging Theory, the selection of hyperlinks (e.g., search results) is determined by the strength of a so-called “information scent.” Information scent reflects the perceived semantic similarity between information contained in link descriptions (i.e., proximal cues) and the reader’s search goal (i.e., the desired information), based on spreading activation mechanisms in the reader’s semantic memory network (cf. Anderson, 1983; Juvina & Van Oostendorp, 2008). A strong information scent indicates a high likelihood that the respective information source contains the desired information and thus increases the likelihood that the link will be selected. Pirolli (2007) illustrates the concept of information scent as follows: Imagine a user whose goal is to find information about “medical treatments for cancer” encounters a search result link whose description includes the terms “cell, patient, dose, beam” (these are the available proximal cues). Each proximal cue sends spreading activation in proportion to its strength of association to the goal, based on past experience. For instance, the terms “medical” and “patient” are likely to have a high strength of association in semantic memory, because the two terms co-occur frequently. The accumulated strengths of association between each cue and the goal determine the information scent.

288  •  Salmerón et al.

Furthermore, Information Foraging Theory assumes that people try to optimize the utility of information gained in relation to the cost of interaction (e.g., search and reading time, cognitive effort). In line with the concept of bounded rationality (cf. Simon, 1955), this leads to a satisficing strategy that implies that Internet readers do not evaluate the information scent of all links available, but sequentially evaluate them only until one is encountered that is “good enough.” This suggests that the position of a link on a page (e.g., on a SERP) can also affect the likelihood it will be selected. According to Information Foraging Theory, after accessing a web page, readers constantly assess the page’s potential usefulness and the costs of reading it. When the expected information gains of a web page drop below the expected costs, readers will leave the page. In sum, with regard to link-selection behavior, Information Foraging Theory (Pirolli, 2007) postulates that both the position of a particular search result in a SERP and the information scent of a particular search result determine the likelihood it will be selected. In support of these assumptions, the SNIF-ACT 2.0 (Scent-based Navigation and Information Foraging in the cognitive architecture ACT-R) computational model, which is derived from Information Foraging Theory and is based on information scent and link position, accurately predicted actual user data in a set of web search tasks (Fu & Pirolli, 2007). Moreover, SNIF-ACT 2.0 made better predictions than two alternative computational models that were based solely on information scent (SNIF-ACT 1.0) or link position (Position model). Information Evaluation Several normative or descriptive models have been proposed to account for the processes of evaluating information in non-academic settings on the Internet. Two examples are the “checklist” model that identifies five criteria according to which web information should be assessed (i.e., accuracy, authority, objectivity, currency, and coverage; cf. Tate, 2010) and the process model of three evaluation phases by Rieh and Hilligoss (2008) mentioned above (for reviews also see Metzger, 2007, and Choi & Stvilia, 2015). Such models aim to explain how readers assess whether web information is relevant and can be trusted. Both relevance and trustworthiness help readers to determine if a particular web page contains useful information to understand a particular issue (e.g., effects of nanotechnology on health) or to support them in making a particular decision (e.g., whether to receive a vaccination). Most of these models emphasize the use of heuristic processes (cf. also the satisficing strategy mentioned in the previous section), as conceived in persuasion research such as the Elaboration Likelihood model (Petty & Cacioppo, 1986). Heuristics are quick and effortless procedures that originate after a few successful experiences with a particular procedure (e.g. “If the webpage is listed on top of a list or results, it must be credible”). Such heuristics are particularly useful on the Internet to filter out information, given the vast amount of information available and the limited amount of time that people spend on web pages. For example, Yeykelis, Cummings, and Reeves (2014) tracked a group of adults while freely interacting with computers at home. On average, people switched tasks (e.g. screens or applications) every 19 seconds, and episodes longer than 2 minutes on a single screen accounted for only 13% of the total episodes. This leaves limited room for a more systematic evaluation of the information processed in non-academic Internet interactions.

Non-Academic Multiple Source Use  •  289

While interacting with the Internet, people are confronted with several layers of information which is potentially relevant for the evaluation of information: the message, the author, the website hosting the information, the medium, the features of the web page, and the actions they are allowed as users (Sundar, 2008). Such layers or cues may trigger different heuristics. Sundar (2008) organizes existing heuristics according to four different categories in the MAIN model: modality, agency, interactivity, and navigability. Modality refers to the different modalities in which information on the Internet is provided (e.g., textual, visual, audiovisual). Each of these may trigger different heuristics. Heuristics that pertain to modality include the realism heuristic (realistic modes such as audiovisual may be linked to positive evaluation), the coolness heuristic (new gadgets may trigger a positive evaluation), but also the old-media heuristic (a website resembling a newspaper in its layout may lead to positive evaluation), and the distraction heuristic (sensory overload may lead to negative evaluations). Agency refers to the source of information, which on the Internet can be a mixture of authors, the website hosting the information (e.g., newspaper, magazine), and the service distributing it (e.g., Google News, Facebook). Heuristics that pertain to agency include the machine heuristic (information perceived to be provided by machines or algorithms may be evaluated more positively than information that comes from a person), bandwagon heuristic (support of the information from other people triggers positive evaluations) and authority heuristic (an expert source leads to positive evaluations). Interactivity refers to both the level of activity and the level of interaction supported by media. Reading on the Internet usually demands a high level of activity, as users must continually use the mouse (or fingers in tablets) to scroll in a page, to access additional material, or to close unrequested windows. The interactivity heuristic may trigger positive or negative evaluations, depending on whether the reader is looking for excitement or for passive consumption of information. Interaction means that users can specify their information needs while processing the information on an ongoing basis. For example, Internet media may support users’ interaction by allowing open comments on the news reports, or by stimulating social media discussions. Thus, the interaction heuristic may trigger the feeling that the system is adapting to the specifications of the reader, which may prompt positive evaluations. Finally, navigability refers to the media design features that suggest transportation from one location to another. Heuristics that pertain to navigability are the usability heuristic (well-organized and easy-to-navigate web pages are linked to positive evaluations) and the browsing heuristic (a high number of hyperlinks options may give readers a positive sense of the lack of bias, verifiability, and corroboration). Interactive Accounts The previously reviewed models focus on navigation and evaluation as separate entities, but in many cases both processes act in coordination. When people log on into the Internet for their own purposes (i.e., no academic tutor has provided strict goals to fulfill), they are rather free to choose what to access and what to ignore. This is particularly the case when people search for personally relevant topics, such as health and political issues. For example, when readers avoid web pages that have headlines that oppose their prior beliefs, or when they prefer to access a blog post with a personal account rather than a scientific report, navigation is being guided by evaluation processes.

290  •  Salmerón et al.

In turn, such biased selection could reinforce readers’ prior beliefs, increasing the likelihood of biased information evaluation in the future. In the following, we review two models that aim to explain two phenomena that are characteristic of non-academic uses of Internet, based on classical theories of social and media psychology: biased selection of pages and preference for personal accounts. Biased selection of web pages is particularly relevant when laypersons read about socio-scientific controversies. Researchers tend to explain this phenomenon with cognitive dissonance theory (Festinger, 1957), which explains people’s reactions to situations that involve conflicting attitudes or behaviors (e.g., when a reader looks at a web page heading that presents an idea opposed to her/his prior beliefs). According to the theory, in such situations people experience an emergent need to equilibrate beliefs and actions. The theory proposes three ways to resolve such dissonance: (a) by changing the actions (e.g., abandon a web page that caused the dissonance). As a consequence of this defensive mechanism, people may be selectively exposed to web pages that favor their prior views, which in turn may reinforce their beliefs (i.e. confirmation bias); (b) by changing the prior attitude (e.g., by accepting partially or totally attitude-inconsistent information); or (c) by changing the perception of the action (e.g. by reconceptualizing the attitudeinconsistent information, such as by denying the source’s neutrality on the issue). The preference for personal accounts over scientific reports has been documented and theorized in media psychology, through the lens of the Exemplification theory (Zillmann, 1999). According to this theory, people tend to give more attention to concrete, vivid events (e.g., personal accounts), at the expense of more abstract, less vivid information (e.g., scientific reports). From this perspective, laypersons’ preferences for examples come from the fact that exemplification, i.e., accounts of an event from different people, is a major starting point to get information about the world from our childhood. In most cases, children’s beliefs about topics are formed solely on the basis of others’ observations and experiences. For adult laypersons, which may lack direct experience or background knowledge on a particular issue, exemplification can also be a privileged source of information. As we shall see in the next section, evidence to support these models comes from studies focusing on different tasks related to the interaction of multiple sources on the Internet that involve reading to locate information on web pages, reading to form an opinion, and reading to make a decision.

EMPIRICAL FINDINGS In this section we describe the empirical evidence related to people interact with multiple sources on the Internet in non-academic settings. We first describe studies focusing on information access, and then we discuss evidence about source evaluation. Lastly, we comment on studies that have focused on the intersection between information access and evaluation. Information Access Studies with search engines that have examined link-selection behavior using eyetracking and log file analyses have shown that the ranking position of a search results

Non-Academic Multiple Source Use  •  291

serves as a strong cue for selection. Individuals pay most attention to the search results at the top of the first SERP and predominantly select these links (e.g., Cutrell & Guan, 2007; Gerjets, Kammerer, & Werner, 2011). Further, several experiments (Kammerer & Gerjets, 2014; Keane et al., 2008; Pan et al., 2007; Salmerón, Kammerer, & GarcíaCarrión, 2013) have shown that this is the case even when the order of the search results presented in a SERP is systematically reversed; that is, when the top search results are the least relevant or least trustworthy ones in the SERP. In such cases, participants still paid most attention to the search results on top of the SERP and accessed less relevant or less trustworthy web pages than when the search results were presented in an ideal order. However, it should also be noted that participants did not exclusively rely on the link position. When the order of the search results was experimentally reversed, they visually inspected more search results before making their selections and did not exclusively select the top results, but spent an equal amount of time on web pages presented toward the top and the bottom of the SERP (Kammerer & Gerjets, 2014; Salmerón et al., 2013). Further, when asked which of the web pages they would bookmark for later use in their coursework, they frequently selected relevant search results located further down in the SERP (Salmerón et al., 2013). Analyses of verbal protocols (e.g., Gerjets, Kammerer, & Werner, 2011; Goldman et  al., 2012; Walraven, Brand-Gruwel, & Boshuizen 2009) also indicate that during web search individuals frequently evaluate the semantic relevance of the search results (i.e., whether they address the search topic at hand). Further, when search results on a SERP were presented in random order, university students spent more time on relevant and reliable pages than on irrelevant and unreliable pages (García-Rodicio, 2015). Importantly, increased time spent on relevant rather than irrelevant sources (GarcíaRodicio, 2015) or on reliable rather than unreliable sources, respectively (Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012; Wiley et al., 2009), resulted in better learning outcomes on the topic at hand. From a developmental perspective, by the time students are in upper secondary school, they have already achieved the adult level of identification of relevant web pages from SERPs (Keil & Kominsky, 2013; Rouet et al., 2011). Younger students (e.g., grades 5 to 7), however, seem to rely more on superficial cues such as highlighted keywords than on the underlying semantic information contained in the search result descriptions (Keil & Kominsky, 2013; Rouet et al., 2011). A comparison of search result evaluation by domain novices and domain experts (analyzing eye-tracking and log file data as well as verbal protocols) revealed that domain experts scanned more search results than novices before the first website was accessed from the SERP (Brand-Gruwel, Kammerer, Van Meeuwen, & Van Gog, in press). On a task to select and prioritize the five best websites, experts spent/needed less time on a website to decide that they wanted to use it for further study and also selected more reliable websites. Domain novices expressed less specific and more superficial utterances when judging search results and websites, whereas domain experts expressed more specific utterances regarding the reliability of information sources. Rieh (2002) also found that expert searchers are concerned about information quality and credibility to a substantial extent when making decisions about which search results to select from the SERPs. They frequently addressed and evaluated source characteristics, such as source reputation, type of source, and URL domain type. Thus, prior domain knowledge seems beneficial to the evaluation of search results.

292  •  Salmerón et al.

In summary, evidence provides ample support for the SNIF-ACT 2.0 model (Fu & Pirolli, 2007), indicating that information scent and position of hyperlinks are important cues that guide people’s navigation when locating information on web pages. Users’ prior knowledge facilitates the use of information scent cues by supporting the inferential process required to match the users’ goals and the expected information contained on a web page. Source Evaluation Because of the overwhelming amount of information sources, which often include conflicting information from different web sources (e.g., hospital web page vs. commercial page), individuals are sometimes confused by the information they find online (Goldberg & Sliwa, 2011). In addition, on the Internet, scientific reports stand hand in hand with personal narratives of people who have experienced a particular problem. Heuristic models (Sundar, 2008) propose that to cope with this complexity, readers may quickly focus on several dimensions of sources to judge the quality of information, without deeply interpreting the information. Among such characteristics, two prominent features of sources are the web page that provides the information and the author of the information. Webpage Evaluation One way to quickly evaluate information on the Internet is to inspect the web page. This is usually done by identifying its status (e.g., institutional page vs. open forum), or by inferring the quality of information based on how usable is the page (authority and usability heuristics, Sundar, 2008). While in principle laypersons may use agency heuristics to quickly identify trustworthy web pages, for example by looking at the source logos (Sundar, 2008), evidence suggests that people also use other less reliable heuristics, such as web page design, to evaluate their trustworthiness. For example, Johnson, Sbaffi, and Rowley (2016) asked undergraduates to list the factors they use to evaluate the trustworthiness of health information web pages. Together with other criteria, they reported that usability and professional style were relevant factors for evaluating web pages. Similarly, Metzger, Flanagin, Markov, Grossman, and Bulger (2015) presented adolescents two professional-looking web pages that conveyed purposefully false information. Approximately half of the participants trusted the information on those pages. Trust to unreliable web pages was higher between students that reported using heuristics to evaluate web information (Sundar, 2008). In addition, evidence suggests that people often ignore source information, and that they can be influenced by false information from untrustworthy web pages. Ivanitskaya et  al. (2006) showed that when undergraduates viewed questionable websites on nutritional supplements, only 50% of the students were able to correctly identify the website with the most trustworthy features. Less than half of the students identified the purpose of the least trustworthy website, which was to sell products and services. Further, 39% of the students believed in the effectiveness of the nutritional supplements even when the information was provided by an untrustworthy website. Similarly, a study by Kammerer, Amann, and Gerjets (2015) showed that, without training on how to evaluate search results and web pages, “non-academic” adults who

Non-Academic Multiple Source Use  •  293

searched for information regarding the effectiveness of a nutritional supplement spent an equal amount of time on scholarly websites and commercial websites. Subsequently, 40% of the participants believed the nutritional supplement was effective, even though the scholarly pages clearly stated that it was not effective. The authors also tested the effects of a short intervention aimed at (a) raising awareness of the importance of source evaluation on the Internet, and (b) increasing users’ knowledge of types of web pages (e.g., institutional vs. commercial web pages) and authors’ expertise. Such interventions increased adults’ exposure time and acceptance of information from trustworthy pages. Evaluation of Author Characteristics The evaluation of author characteristics on the Internet is also a challenging issue, as identities and professional background can be easily faked on the Internet. For example, in web forums any user can self-proclaim that she/he is competent on the topic under discussion. Evidence suggests that users are rather uncritical when analyzing authors’ characteristics. In an interview study, Jeon and Rieh (2014) found that the mere participation in online discussions in forums was considered by participants to be a sign of expertise. Users may be more strategic in situations in which authors of comments have clear potential vested interests. For example, people trust users who are deemed experts by peer ratings rather than by themselves (i.e., self-proclaimed experts) when reading online reviews of products (Willemsem et al., 2012). Characteristics of authors’ comments can also influence readers. A major aspect researched is authors’ perceived expertise (i.e., authority heuristic, Sundar, 2008): adult readers are more susceptible to messages from self-reported experts than from lay users or those who comment under a pseudonym on topics such as daily life issues (Salmerón, Macedo-Rouet, & Rouet, 2016), scientific controversies (Winter & Krämer, 2012), or news about a public scandal (Van Sikorski, 2016). Personal Narratives One source of information on the Internet is laypersons that share their own experiences on different issues, such as their personal views on socio-scientific or political issues or their satisfaction with a product bought online. The potential influence of personal narratives, as predicted by the exemplification theory (Zillmann, 1999), can become problematic when evidence from personal accounts opposes scientific views on a controversy, as is the case for instance with the anti-vaccination groups. Studies focusing on this issue present participants with conflicting information (e.g., benefits or drawbacks of vaccines). In these studies, information conveyed as personal narratives opposes the view expressed by scientific accounts. Evidence suggests that users are more influenced by personal experiences than by scientific data when they are asked to form an opinion or to make a decision about socio-scientific controversies such as vaccinations (Betsch, Ulshöfer, Renkewitz, & Betsch, 2011; Peter, Rossmann, & Keyling, 2014), climate change (Hinnant, Subramanian, & Young, 2016), or home birth (Witteman et al., 2016). It is also relevant for health-related decisions, such as weight-loss methods (Knobloch-Westerwick, & Sarge, 2015) and intention to quit smoking (Kim, Bigman, Leader, Lerman, & Cappella, 2012). Participants in a study

294  •  Salmerón et al.

by Knobloch-Westerwick and Sarge (2015) read several websites that reviewed different weight-loss methods. Online articles based on personal narratives increased participants’ intentions to engage in weight-loss behavior to a greater extent than articles supported by empirical data. The influence of personal narratives on users’ intention to buy products reveals a more complex pattern. Users access multiple sources of information to decide what and where to buy. In such contexts, users may be more aware of the fact that sources have a strong incentive not to be neutral, but rather to try to maximize the positive aspects of the products they want to sell. Potential consumers’ intentions after reading online reviews in web forums reveal that negative reviews on the products are more influential than positive ones (Lee & Koo, 2012; Pan & Chiou, 2011), probably because they are perceived as more trustworthy and less biased (Teng et al., 2017). The influence of personal narratives goes beyond the credibility of the sources. Haase, Betsch, and Renkewitz (2015) found that users’ perception of the risk of vaccination increased as a function of the number of anti-vaccination narratives read, regardless of the credibility of the web page hosting the message (anti-vaccination website or a neutral health forum). The influence of personal accounts on the Internet can also take the form of indirect social information, such as the number of “likes” given to a comment or a post (i.e., the bandwagon heuristic, Sundar, 2008). Emerging evidence, however, suggests that indirect social information has no effect on the formation of an opinion from news (Lee & Jang, 2010), or on the intentions to vaccinate in the future (Peter et al., 2014). In summary, the individual narrative itself, and not the fact that a claim is supported by individuals, influences users’ intentions more than a scientific report, at least when it comes to topics that pose a real potential threat. Interactions Between Information Access and Source Evaluation When using the Internet to gather information or form an opinion, people access a variety of websites, such as newspapers, blogs, and social media. In addition, information sources on the Internet can be enriched by the comments by other users, which may criticize or reinforce the original text. This provides the opportunity for people to access a range of web pages with different and complementary views on topics in order to construct a rich representation of an issue. However, there is also the risk that people are selectively exposed to information sources that provide support for a single view, either because Internet providers present a selective range of information, or because people actively engage in selective exposure. System-Induced Selective Exposure System-induced selective exposure on the Internet occurs when the Internet provider (e.g., search engine, web page), predominantly presents information from conflicting topics that support a particular view. One of the major threats to selective exposure comes from search engines, as most people rely on them to access information. Allam, Schulz, and Nakamoto (2014) tested the effects of presenting web pages in a SERP conveying different views on vaccinations. The authors varied the ratio of pro/againstvaccination web pages in a regular SERP containing ten search results. As compared

Non-Academic Multiple Source Use  •  295

to a SERP with only pro-vaccination web pages, SERPs that included four, six, or ten against-vaccination web pages increased participants’ fear of vaccination side effects, and reduced the acknowledgment of vaccination benefits. In a series of lab and field studies, Epstein and Robertson (2015) analyzed this effect in political campaigns. They tested the effects of position of web pages describing the candidate’s views in the SERP on participants’ vote intention. Across the different studies, people reported they were more likely to vote for the candidates ranked higher on the SERPs. A different system-induced selective exposure arises when a web page allows for the inclusion of users’ comments, which may affect the interpretations and perceptions of authored content provided by online newspapers, magazines, blogs, or web forums. Comments can vary in several features such as tone (civil vs uncivil), attitude, or affective state toward the issue they comment on, and can influence the way the issue described on the web page is perceived. Indeed, people can dedicate even more time to reading the comments section than to paragraphs of the main text that are comparable in length (Steinfeld, Samuel-Azran, & Lev-On, 2016). Biased comments, supporting a particular view on the conflicting topic discussed in a news report, can shape readers’ opinions toward such a view. For example, Lee, Kim, and Cho (2016) found that participants who read a news article about a crime in a particular area and were exposed to prejudiced comments about the criminality of that area better recalled the name of the location and estimated that it had a higher overall crime rate. A similar pattern of effects have been reported for a range of topics, including social (van Sikorski, 2016), health (Witteman, Fagerlin, Exe, Trottier, & Zikmund-Fisher, 2016), and sociopolitical issues (Houston, Hansen, & Nisbett, 2011; Miller, Xu, & Barnett, 2016). Cognitive Dissonance theory (Festinger, 1957) predicts that the effects of systeminduced selective exposure may interact with prior attitudes. Readers with strong prior opinions on an issue may activate defensive mechanisms to deny attitude-inconsistent information, especially if the way the information is provided is perceived as a threat to their prior attitude, as is the case with uncivil comments on the Internet. Anderson et al. (2014) tested this assumption by focusing on language tone and introduced the “nasty effect,” which refers to the influence of uncivil (rude or aggressive) comments on readers’ interpretation of information. They asked a representative sample of U.S. adults to read a news article on a socio-scientific controversy from a balanced point of view, followed by both civil and uncivil comments. Results indicate that exposure to uncivility bolsters readers’ previous attitudes, which can be interpreted as a defensive mechanism. In the same vein, Borah (2014) found that uncivil comments led to more narrow-mindedness and more attitude certainty for undergraduates who read about two socio-political controversies. Moreover, uncivility can lead people to judge a balanced blog post as biased (Anderson, Yeo, Brossard, Scheufele, & Xenos, 2016), or boost the perception of political polarization in society and undermine the readers’ expectations in the usefulness of public debates (Hwang, Kim, & Huh, 2014). While the effects of comments on people’s evaluation of web news have been extensively documented, recent works have questioned the prominence of such influence, as readers may perceive comments as biased (Steinfeld et  al., 2016). Because commenting on news on the Internet is a relatively new concept, we may expect that users’ perception of comments, and their corresponding influence, may vary with increasing exposure to such scenarios.

296  •  Salmerón et al.

Purposeful Selective Exposure Purposeful selective exposure on the Internet occurs when people only access web pages that convey a particular viewpoint (emotional or ideological) and avoid visiting web pages that present other viewpoints. Knobloch‐Westerwick, Johnson, and Westerwick (2015) studied the effects of selective exposure in a sample of non-student adults. Participants navigated through a series of web pages presented in a SERP, on four different political topics (health coverage, minimum wage, gun control, and abortion legalization). They were instructed “to have a brief time span to look at the related information.” Their attitudes toward the issues were analyzed before and after reading the web pages. Results showed that participants spent 65% more time on attitudeconsistent web pages approximately than on attitude-discrepant pages. Moreover, regarding the effects of selective exposure, time spent on attitude-consistent web pages strengthened preexisting attitudes, while time spent on attitude-inconsistent pages diminished prior attitudes. Furthermore, van Strien, Kammerer, Boshuizen, and Brand-Gruwel (2016) found that individuals with strong attitudes scrutinized web page logos of attitude-inconsistent websites shorter and judged the credibility of attitude-inconsistent websites lower than participants with weaker attitudes when asked to read multiple web pages about a controversial socio-scientific issue. They also included more attitude-consistent information in a subsequent essay task than participants with weaker prior attitudes. These results indicate that individuals with strong attitudes are biased toward their preexisting attitudes both in their information evaluation and in their conclusions drawn after reading. Such confirmation bias has been also reported in relation to reading online news sites and blogs about sociopolitical (e.g., Knobloch-Westerwick & Meng, 2009, 2011; Westerwick, Kleinman, & Knobloch-Westerwick, 2013), daily health (Knobloch‐Westerwick, Johnson, & Westerwick, 2013), and socio-scientific controversies (e.g., Jang, 2013; Schwind & Buder, 2012; Schwind, Buder, Cress, & Hesse, 2012). Interestingly, research has also identified ways to overcome the negative effects of confirmation bias, so that readers can resolve the cognitive dissonance by partially acknowledging the other side of the issue. This can be done, for example, by introducing a fictitious online web-recommendation system (Schwind et al., 2012), improving people’s web skills (Feufel & Stahl, 2012), or inducing a cooperative (vs. competitive) mind-set among users (Schwind & Buder, 2012).

CONCLUSIONS In our review of the scientific literature about the non-academic use of multiple sources on the Internet, three major interrelated tasks emerged: reading multiple sources to locate and access information, reading to develop an informed opinion about a particular issue, and reading to make an informed decision. We identified three key phenomena affecting readers’ comprehension, attitude, and/or behavioral intentions: position of a link in a SERP, selective exposure to web pages with a particular view, and personal narratives. As we review next, those findings come from different disciplines, which may benefit from a shared theoretical background and point to future research directions.

Non-Academic Multiple Source Use  •  297

The study of non-academic uses of multiple sources on the Internet in the last decade has been addressed by scholars of different disciplines, including communication studies, educational psychology, and computer science. Each community has addressed questions particularly relevant to their field, from their theoretical and methodological background. As is clear in our review, such diversity has allowed the field to cover a vast range of topics—from reading newspapers to understanding the current situation in a neighborhood, to reading Facebook posts, to deciding whether to get vaccinated—all focusing on different technical, individual and social influences. Such diversity also poses challenges, such as the proliferation of theoretical models that only cover a particular aspect of the interaction with multiple sources on the Internet. While it may be unrealistic to think that a single model could ever capture all issues reviewed in this chapter (i.e., navigation, informative reading and decision making), models can be used to guide the measurement of the interrelationships between the three major competences of reading on the Internet (Salmerón et al., in press): navigation, integration, and evaluation of information. For example, models focusing on navigation, such as SNIF-ACT 2.0 (Fu & Pirolli, 2007), could be expanded to explain situations in which the critical evaluation of information may prevent a cautious reader from accessing a particular web page with high information scent (for initial information on this topic, see, e.g., Gerjets & Kammerer, 2010). The Internet has introduced new possibilities that allow many people unprecedented access to information, from experts’ accounts to personal opinions, which help them to find facts and information, to develop informed opinions, and to seek advice to make decisions relevant to their lives. This holds great promise particularly for individuals for whom the Internet could reduce or eliminate many barriers that limit their access to knowledge (e.g. Chadwick, Wesson, & Fullwood, 2013). Unfortunately, most research available on the field has been conducted with undergraduate students, which makes it difficult to generalize results and conclusions to the overall population (Peterson, 2001). Future research should make an effort to expand its scope by studying the nonacademic multiple sources used by the general population, including adults without a university degree (e.g., Kammerer et al., 2015), older people (e.g., Feufel, & Stahl, 2012), teenagers (e.g., Keil & Kominsky, 2013; Rouet et al., 2011; Salmerón et al., 2016), and people with mental disabilities (e.g., Salmerón, Gómez, & Fajardo, 2016). At the same time, most research on decision making on the Internet has restricted its focus to medical decisions and commercial behavior. Extending research to other decision-making topics, such as daily life topics, may help to identify important but neglected influences. Across the chapter we have described several practical applications to support readers’ use of multiple documents on non-academic contexts on the Internet. System design (e.g., system recommendation, Schwind et  al., 2012) and brief instructions (e.g., training to identify different web providers, Kammerer et al., 2015) could help users to detect unwarranted information, to make better and more balanced medical decisions, or to form unbiased opinions of controversial issues. One could argue that as new generations are becoming more exposed and fluent on their academic interactions on the Internet, some of these applications may not be necessary in the future. However, it remains an open question to what extent web navigation and evaluation skills from academic contexts may transfer to non-academic use of multiple documents on the Internet as those described in the present chapter.

298  •  Salmerón et al.

To conclude, understanding how people use multiple sources on the Internet for non-academic purposes is a great challenge that requires diligent effort from researchers of traditionally distant (not necessarily collaborating) disciplines. Our review aims to bring together the current knowledge of the field and to stimulate an exchange between the different disciplines working on it.

REFERENCES Alexander, P.A. (2012). Reading into the future: Competence for the 21st century. Educational Psychologist, 47, 259–280. Allam, A., Schulz, P.J., & Nakamoto, K. (2014). The impact of search engine selection and sorting criteria on vaccination beliefs and attitudes: Two experiments manipulating Google output. Journal of Medical Internet Research, 16(4): e100. Anderson, A.A., Brossard, D., Scheufele, D.A., Xenos, M.A., & Ladwig, P. (2014). The nasty effect: Online incivility and risk perceptions of emerging technologies. Journal Computer-Mediated Communication, 19, 373–387. Anderson, A.A., Yeo, S.K., Brossard, D., Scheufele, D.A., & Xenos, S.A. (2016). Toxic talk: How online incivility can undermine perceptions of media. International Journal of Public Opinion Research, edw022. Anderson, J.R. (1983). A spreading activation theory of memory. Journal of Verbal Learning and Verbal Behaviour, 22, 261–295. Barberá, P., Jost, J.T., Nagler, J., Tucker, J.A., & Bonneau, R. (2015). Tweeting from left to right: Is online political communication more than an echo chamber? Psychological Science, 26(10), 1531–1542. Bates, B.R., Romina, S., Ahmed, R., & Hopson, D. (2006). The effect of source credibility on consumers’ perceptions of the quality of health information on the Internet. Medical Informatics and the Internet in Medicine, 31, 45–52. Betsch, C., Renkewitz, F., & Haase, N. (2013). Effect of narrative reports about vaccine adverse events and bias-awareness disclaimers on vaccine decisions: a simulation of an online patient social network. Medical Decision Making, 33(1), 14–25. Betsch, C., Ulshöfer, C., Renkewitz, F., & Betsch, T. (2011). The influence of narrative vs. statistical information on perceiving vaccination risks. Medical Decision Making, 31(5), 742–753. Borah, P. (2014). Does it matter where you read the news story? Interaction of incivility and news frames in the political blogosphere. Communication Research, 41(6), 809–827. Brand-Gruwel, S., Kammerer, Y., van Meeuwen, L., & van Gog, T. (in press). Source evaluation of domain experts and novices during Web search. Journal of Computer Assisted Learning. Brand-Gruwel, S., Wopereis, I., & Vermetten, Y. (2005). Information problem solving by experts and novices: analysis of a complex cognitive skill. Computers in Human Behaviour, 21, 487–508. Brand-Gruwel, S., Wopereis, I., & Walraven, A. (2009). A descriptive model of information problem solving while using internet. Computers & Education, 53, 1207–1217. Bråten, I., Stadtler, M., & Salmerón, L. (in press). The role of sourcing in discourse comprehension. In M. Schober, D.N. Rapp, & M.A. Britt (Eds.), Handbook of discourse processes. New York, NY: Taylor & Francis. Britt, M.A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20, 485–522. Carpenter, D.M, Geryk L.L., Chen, A.T., Nagler, R.H., Dieckmann, N.F., & Han, P.K (2015). Conflicting health information: A critical research need. Health Expectations, 19, 1173–1182. Chadwick, D., Wesson, C., & Fullwood, C. (2013). Internet access by people with intellectual disabilities: Inequalities and opportunities. Future Internet, 5, 376–397. Choi, W., & Stvilia, B. (2015). Web credibility assessment: Conceptualization, operationalization, variability, and models. Journal of the Association for Information Science and Technology, 66(12), 2399–2414. Cutrell, E., & Guan, Z. (2007). What are you looking for? An eye-tracking study of information usage in Web search. In M.B. Rosson & D. Gilmore (Eds.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07) (pp. 407–416). New York, NY: ACM Press. Epstein, R., & Robertson, R.E. (2015). The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of the National Academy of Sciences, 112: E4512–E4521.

Non-Academic Multiple Source Use  •  299 Estabrook, L., Witt, E., & Rainie, L. (2007). Information searches that solve problems. PEW Internet & American Life Project. Retrieved from www.pewinternet.org/~/media//Files/Reports/2007/Pew_UI_LibrariesReport. pdf.pdf. Festinger, L. (1957). A theory of cognitive dissonance. Evanston, IL: Row-Peterson. Feufel, M.A., & Stahl, S.F. (2012). What do web-use skill differences imply for online health information searches? Journal of Medical Internet Research, 14(3), e87. Fu, W.T., & Pirolli, P. (2007). SNIF-ACT: A cognitive model of user navigation on the World Wide Web. Human–Computer Interaction, 22, 355–412. García-Rodicio, H. (2015). Students’ evaluation strategies in a Web research task: Are they sensitive to relevance and reliability? Journal of Computing in Higher Education, 27, 134–157. Gerjets, P., & Kammerer, Y. (2010). Topical relevance and information quality in cognitive models of Web search behavior: Introducing epistemic scent into information foraging theory. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (pp. 2278–2283). Austin, TX: Cognitive Science Society. Gerjets, P., Kammerer, Y., & Werner, B. (2011). Measuring spontaneous and instructed evaluation processes during web search: Integrating concurrent thinking-aloud protocols and eye-tracking data. Learning and Instruction, 21, 220–231. Goldberg, J.P., & Sliwa, S.A. (2011). Communicating actionable nutrition messages: Challenges and opportunities. Proceedings of the Nutrition Society, 70, 26–37. Goldman, S.R., Braasch, J.L.G., Wiley, J., Graesser, A.C., & Brodowinska, K. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381. Haase, N., Betsch, C., & Renkewitz, F. (2015). Source credibility and the biasing effect of narrative information on the perception of vaccination risks. Journal of Health Communication, 20(8), 920–929. Hastall, M.R., & Knobloch-Westerwick, S. (2012). Severity, efficacy, and evidence type as determinants of health message exposure. Health Communication, 28(4), 378–388. Hinnant, A., Subramanian, R., & Young, R. (2016). User comments on climate stories: impacts of anecdotal vs. scientific evidence. Climatic Change, 138(3–4), 411–424. Houston, J.B., Hansen, G.J., & Nisbett, G.S. (2011). Influence of user comments on perceptions of media bias and third-person effect in online news. Electronic News, 5(2), 79–92. Hu, Y., & Sundar, S. (2010). Effects of online health sources on credibility and behavioral intentions. Communication Research, 37, 105–132. Hwang, H., Kim, Y., & Huh, C.U. (2014). Seeing is believing: Effects of uncivil online debate on political polarization and expectations of deliberation. Journal of Broadcasting & Electronic Media, 58(4), 621–633. Ivanitskaya, L., O’Boyle, I., & Casey, A.M. (2006). Health information literacy and competencies of information age students: Results from the interactive online research readiness self-assessment (RRSA). Journal of Medical Internet Research, 8(2). Jang, S.M. (2013). Seeking congruency or incongruency online? Examining selective exposure to four controversial science issues. Science Communication, 36(2), 143–176. Jeon, G.Y., & Rieh, S.Y. (2014). Answers from the crowd: How credible are strangers in social Q&A?. iConference 2014 Proceedings. Retrieved from http://hdl.handle.net/2142/47266. Johnson, F., Sbaffi, L., & Rowley, J. (2016). Students’ approaches to the evaluation of digital information: Insights from their trust judgments. British Journal of Educational Technology, 47(6), 1243–1258. Juvina, I., & Van Oostendorp, H. (2008). Modelling semantic and structural knowledge in Web navigation. Discourse Processes, 45, 346–364. Kammerer, Y., Amann, D., & Gerjets, P. (2015). When adults without university education search the Internet for health information: The roles of Internet-specific epistemic beliefs and a source evaluation intervention. Computers in Human Behavior, 48, 297–309. Kammerer, Y., & Gerjets, P. (2014). The role of search result position and source trustworthiness in the selection of web search results when using a list or a grid interface. International Journal of Human-Computer Interaction, 30, 177–191. Keane, M.T., O’Brien, M., & Smyth, B. (2008). Are people biased in their use of search engines? Communications of the ACM, 51, 49–52. Keil, F.C., & Kominsky, J.F. (2013). Missing links in middle school: Developing use of disciplinary relatedness in evaluating Internet search results. PLOS One, 8.

300  •  Salmerón et al. Kellar, M., Watters, C., & Shepherd, M. (2007). A field study characterizing Web-based information-seeking tasks. Journal of the American Society for Information Science and Technology, 58, 999–1018. Kim, H.S., Bigman, C.A., Leader, A.E., Lerman, C., & Cappella, J.N. (2012). Narrative health communication and behavior change: The influence of exemplars in the news on intention to quit smoking. Journal of Communication, 62(3), 473–492. Knobloch-Westerwick, S., & Meng, J. (2009). Looking the other way selective exposure to attitude-consistent and counterattitudinal political information. Communication Research, 36(3), 426–448. Knobloch‐Westerwick, S., & Meng, J. (2011). Reinforcement of the political self through selective exposure to political messages. Journal of Communication, 61(2), 349–368. Knobloch-Westerwick, S., & Sarge, M.A. (2015). Impacts of exemplification and efficacy as characteristics of an online weight-loss message on selective exposure and subsequent weight-loss behavior. Communication Research, 42(4), 547–568. Knobloch-Westerwick, S., Johnson, B.K., Silver, N.A., & Westerwick, A. (2015). Science exemplars in the eye of the beholder: How exposure to online science information affects attitudes. Science Communication, 37(5), 575–601. Knobloch‐Westerwick, S., Johnson, B.K., & Westerwick, A. (2013). To your health: Self‐regulation of health behavior through selective exposure to online health messages. Journal of Communication, 63(5), 807–829. Knobloch‐Westerwick, S., Johnson, B. K., & Westerwick, A. (2015). Confirmation bias in online searches: Impacts of selective exposure before an election on political attitude strength and shifts. Journal of Computer‐Mediated Communication, 20(2), 171–187. Lee, E.J., & Jang, Y.J. (2010). What do others’ reactions to news on internet portal sites tell us? Effects of presentation format and readers’ need for cognition on reality perception. Communication Research, 37(6), 825–846. Lee, K.T., & Koo, D.M. (2012). Effects of attribute and valence of e-WOM on message adoption: Moderating roles of subjective knowledge and regulatory focus. Computers in Human Behavior, 28(5), 1974–1984. Lee, E.J., Kim, H.S., & Cho, J. (2016). How user comments affect news processing and reality perception: Activation and refutation of regional prejudice. Communication Monographs, 1–19. Lewandowski, D. (2011). The influence of commercial intent of search results on their perceived relevance. In Proceedings of the 2011 iConference (iConference ’11) (pp. 452–458). New York, NY: ACM Press. Metzger, M.J. (2007). Making sense of credibility on the Web: Models for evaluating online information and recommendations for future research. Journal of the American Society for Information Science and Technology, 58, 2078–2091. Metzger, M.J., Flanagin, A.J., Markov, A., Grossman, R., & Bulger, M. (2015). Believing the unbelievable: Understanding young people’s information literacy beliefs and practices in the United States. Journal of Children and Media, 9, 325–348. Miller, B.M., Xu, Q., & Barnett, B. (2016). Commenter anonymity affects reader perceptions. Newspaper Research Journal, 37(2), 138–152. Miller, C.S., & Remington, R.W. (2004). Modeling information navigation: Implications for information architecture. Human-Computer Interaction, 19, 225–271. Modave, F., Shokar, N.K., Peñaranda, E., & Nguyen, N. (2014). Analysis of the accuracy of weight loss information search engine results on the Internet. American Journal of Public Health, 104, 1971–1978. Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007). In Google we trust: Users’ decisions on rank, position, and relevance. Journal of Computer-Mediated Communication, 12, 801–823. Pan, L.Y., & Chiou, J.S. (2011). How much can you trust online information? Cues for perceived trustworthiness of consumer-generated online information. Journal of Interactive Marketing, 25(2), 67–74. Peter, C., Rossmann, C., & Keyling, T. (2014). Roles of direct and indirect social information in conveying health messages through social network sites. Journal of Media Psychology, 26, 19–28. Peterson, R.A. (2001). On the use of college students in social science research: Insights from a second-order meta-analysis. Journal of Consumer Research, 28(3), 450–461. Petty, R.E., & Cacioppo, J.T. (1986). The elaboration likelihood model of persuasion. In Communication and persuasion (pp. 1–24). New York, NY: Springer. Pirolli, P. (2007). Information foraging theory: Adaptive interaction with information. New York, NY: Oxford University Press. Purcell, K., Brenner, J., & Rainie, L. (2012), “Search engine use 2012,” Pew Internet & American Life Project. Retrieved from www.pewinternet.org.

Non-Academic Multiple Source Use  •  301 Rieh, S.Y. (2002). Judgment of information quality and cognitive authority in the Web. Journal of the American Society for Information Science and Technology, 53, 145–161. Rieh, S.Y., & Hilligoss, B. (2008). College students’ credibility judgments in the information-seeking process. In M.J. Metzger & A.J. Flanagin (Eds.), Digital media, youth, and credibility: The John D. and Catherine T. MacArthur foundation series on digital media and learning (pp. 49–72). Cambridge, MA: The MIT Press. Rouet, J.-F., Ros, C., Goumi, A., Macedo-Rouet, M., & Dinet, J. (2011). The influence of surface and deep cues on primary and secondary school students’ assessment of relevance in Web menus. Learning and Instruction, 21, 205–219. Salmerón, L., Gómez, M., & Fajardo, I. (2016). How students with intellectual disabilities evaluate recommendations from Internet forums. Reading and Writing, 29, 1653–1675. Salmerón, L., Macedo-Rouet, M., & Rouet, J.-F. (2016). Multiple viewpoints increase students’ attention to source features in social question and answer forum messages. Journal of the Association for Information Science and Technology, 67(10), 2404–2419. Salmerón, L., Kammerer, Y., & García-Carrión, P. (2013). Searching the Web for conflicting topics: Page and user factors. Computers in Human Behavior, 29, 2161–2171. Salmerón, L., Strømsø, H.I., Kammerer, Y., Stadtler, M., & van den Broek, P. (in press). Comprehension processes in digital reading. In Paul van den Broek (Ed.), Learning to read in a digital world. Amsterdam: John Benjamins. Schwind, C., & Buder, J. (2012). Reducing confirmation bias and evaluation bias: When are preferenceinconsistent recommendations effective – and when not?. Computers in Human Behavior, 28(6), 2280–2290. Schwind, C., Buder, J., Cress, U., & Hesse, F.W. (2012). Preference-inconsistent recommendations: An effective approach for reducing confirmation bias and stimulating divergent thinking?. Computers & Education, 58(2), 787–796. Simon, H.A. (1955). A behavioral model of rational choice. Quarterly Journal of Economics, 69, 99–118. Smith, A. (2009). The internet’s role in campaign 2008. Pew Internet & American Life Project. Washington, DC. Retrieved from www.pewinternet.org/Reports/2009/6--The-Internets-Role-in-Campaign-2008.aspx. Stadtler, M., & Bromme, R. (2007). Dealing with multiple documents on the WWW: The role of metacognition in the formation of documents models. International Journal of Computer-Supported Collaborative Learning, 2(2), 191–210. Steinfeld, N., Samuel-Azran, T., & Lev-On, A. (2016). User comments and public opinion: Findings from an eye-tracking experiment. Computers in Human Behavior, 61, 63–72. Sundar, S.S. (2008). The MAIN model: A heuristic approach to understanding technology effects on credibility. In M.J. Metzger & A.J. Flanagin (Eds.), Digital media, youth, and credibility (pp. 73–100). Cambridge, MA: MIT Press. Tate, M.A. (2010). Web wisdom: How to evaluate and create information quality on the Web (2nd ed.). Boca Raton, FL: CRC Press. Teng, S., Khong, K.W., Chong, A.Y.L., & Lin, B. (2017). Persuasive electronic word-of-mouth messages in social media. Journal of Computer Information Systems, 57(1), 76–88. van Sikorski, C. (2016). The effects of reader comments on the perception of personalized scandals: Exploring the roles of comment valence and commenters’ social status. International Journal of Communication, 10, 4480–4501. van Strien, J.L.H., Kammerer, Y., Boshuizen, H.P.A., & Brand-Gruwel, S. (2016). How attitude strength biases information processing and evaluation on the web. Computers in Human Behavior, 60, 245–252. Walraven, A., Brand-Gruwel, S., & Boshuizen, H.P. (2009). How students evaluate information and sources when searching the World Wide Web for information. Computers & Education, 52, 234–246. Westerwick, A., Kleinman, S.B., & Knobloch‐Westerwick, S. (2013). Turn a blind eye if you care: Impacts of attitude consistency, importance, and credibility on seeking of political information and implications for attitudes. Journal of Communication, 63(3), 432–453. Wiley, J., Goldman, S., Graesser, A., Sanchez, C., Ash, I., & Hemmerich, J. (2009). Source evaluation, comprehension, and learning in internet science inquiry tasks. American Educational Research Journal, 46, 1060–1106. Willemsen, L.M., Neijens, P.C., & Bronner, F. (2012). The ironic effect of source identification on the perceived credibility of online product reviewers. Journal of Computer‐Mediated Communication, 18(1), 16–31. Wilson, T.D. (1999). Models in information behaviour research. Journal of Documentation, 55, 249–270.

302  •  Salmerón et al. Winter, S., & Krämer, N.C. (2012). Selecting science information in Web 2.0: How source cues, message sidedness, and need for cognition influence users’ exposure to blog posts. Journal of Computer‐Mediated Communication, 18(1), 80–96. Wirth, W., Böcking, T., Karnowski, V., & von Pape, T. (2007). Heuristic and systematic use of search engines. Journal of Computer-Mediated Communication, 12, 778–800. Witteman, H.O., Fagerlin, A., Exe, N., Trottier, M.E., & Zikmund-Fisher, B.J. (2016). One-sided social media comments influenced opinions and intentions about home birth: An experimental study. Health Affairs, 35(4), 726–733. Wopereis, I., Brand-Gruwel, S., & Vermetten, Y. (2008). The effect of embedded instruction on solving information problems. Computers in Human Behavior, 24, 738–752. Yeykelis, L., Cummings, J.J., & Reeves, B. (2014). Multitasking on a single device: Arousal and the frequency, anticipation, and prediction of switching between media content on a computer. Journal of Communication, 64(1), 167–192. Zillmann, D. (1999). Exemplification theory: Judging the whole by some of its parts. Media Psychology, 1(1), 69–94.

18

UPDATING OF CHARACTER INFORMATION WHEN READING MULTIPLE TEXTS FOR PLEASURE Amalia M. Donovan and David N. Rapp northwestern university, usa

Reading fiction involves becoming familiar with story characters, developing concerns about their struggles, and cheering them on to victory. The empathies that emerge from experiences with heroes and villains, plots and subplots, and suspense and resolution provide substantial enjoyment. Our routine investments with story characters and settings are evidenced by the popularity of book series such as Harry Potter, The Chronicles of Narnia, The Hunger Games, A Game of Thrones, The Dark Tower, Dune, and Outlander. The authors of these series detail the exploits of recurring characters that, over time, become champions and scoundrels, with readers establishing expectations they will behave in ways consistent with previous behaviors and traits. Characters like Sherlock Holmes, James Bond, and Frodo the hobbit perhaps prove popular because audiences can count on them to continue to make amazing deductions, escape impossible deathtraps, and complete perilous quests as they have done in previous stories. We read their ongoing tales to see how they will deal with new hurdles, given we know what they are capable of and the resources they can rely upon to overcome problems. These sorts of multi-text experiences, which we identify as experiences with sets of distinct yet interrelated written materials, engage a variety of cognitive processes and strategies in the service of narrative comprehension. We might begin encoding the actions, thoughts, and physical descriptions of a character into memory, establishing a mental model for what that character might do in future events. As we read new episodes in which that character acts consistently, our model can be strengthened, reflecting and offering stronger support for generating inferences about future character behaviors. Consider, as an example, Peter Parker, the alter ego of Spider-Man, a wisecracking do-gooder whose everyday worries about his elderly Aunt May, including the need to hide his alter ego from her, complicate his swashbuckling desires.

303

304  •  Donovan and Rapp

Our model of Peter Parker gives us insight into how he might behave, allowing for predictions for what will happen next in his multi-story adventures. However, what we know about characters might need to change. This can occur in stories that involve plot twists, with characters completely altering how they act and think, which requires updating our models of characters to reflect new states of affairs. Let’s again consider Peter Parker’s elderly Aunt May: It becomes a familiar routine to read about her obvious concern for Peter’s well-being (such as making sure he wears a hat when it is cold outside), and her vehement dislike for that awful Spider-Man she reads about in the newspaper. But in at least one storyline, despite her seeming obliviousness, Aunt May reveals she has actually been aware that Peter and Spidey were one and the same the whole time! Our representation of Aunt May as the clueless caregiver needs revising to reflect that she was actually cleverer than Peter much of the time. Readers’ experiences with stories involve encoding and retrieving models of characters that, depending on plot and across story situations, might remain consistent or change over time.

THE CURRENT CHAPTER To date, work on readers’ interactions with multi-text materials has tended to focus on non-fictional texts, specifically in the areas of STEM (science, technology, engineering, and math) and history (e.g., Wineburg, 1991; Wiley & Voss, 1999; Britt & Rouet, 2012; Wiley et al., 2009). But as our introduction exemplifies, readers routinely engage in multi-text experiences with popular fictional characters. This chapter reviews the processes that underlie our experiences with story characters that regularly appear in fiction, as well as the products that emerge as a function of those processes. A core process we will focus on in the chapter involves updating, which we define as constructing, adding to, or revising existing representations of characters encoded and stored in memory. We will describe experimental research that has helped identify the ways in which readers use the products of updating processes to generate predictions and expectations for future narrative events. And critically, we will be concerned with situations in which readers learn about characters across multiple narrative episodes rather than in single events. Empirical investigations of story characters and updating, while often examining multiple episodes, have not focused on multiple-text presentations to the degree that topics in other chapters in this volume have considered (e.g., reading different volumes of a series with recurring characters, or with delays between reading episodes and volumes, or from different media forms). But findings examining multi-episode presentations during a single reading experience offer useful insight into how readers construct models of story characters. These findings provide a firm grounding for developing accounts of the ways in which readers update character information when reading multiple texts. In the interest of building such accounts, we begin by reviewing previous work on readers’ tracking of story characters in fiction. This work connects to models and frameworks of text and narrative comprehension, providing background for considering the processes and products involved in readers’ use of character models. We follow with a focus on updating, including discussion of when and how it occurs. This allows us to begin contemplating more complex protagonists and antagonists, as are routinely encountered in popular novel series, and that can emerge through multiple

Updating of Character Information  •  305

experiences provided across narrative episodes. The goal of the current chapter is thus to situate contemporary findings on readers’ experiences with characters in fiction, and consider how those findings can inform models of comprehension across texts.

MENTAL REPRESENTATIONS OF NARRATIVES Most accounts of how people process story characters highlight that readers track a variety of features in narratives. This historically derives from Gernsbacher’s structurebuilding framework, an account that identifies the ways in which people build models for events (Gernsbacher, 1990). According to the framework, readers use the information provided in texts to build mental representations for the described events. The initially constructed representation, termed the foundation, is derived from the first descriptions offered in a story, forming the basis of a more complex mental or situation model. Subsequently encountered information consistent with this foundation is mapped onto the existing representation. Processing challenges occur when newly encountered text includes shifts or inconsistencies with information in the foundation. In these cases, readers may build new models to reflect new states of affairs in the text. For example, when a description shifts from discussing a protagonist to an antagonist’s plans, readers may shift to build a new foundation and thus mental model for the other character. This takes more time and resources to complete than does simply adding to an existing model. When this process spans multiple texts, the cognitive demands may increase further, with readers potentially needing to recall and reconcile a broader range of representations developed over varying amounts of time. The power of the structure-building framework is that it is amenable to empirical testing, with hypotheses derived directly from the presumed ease or difficulty of readers incorporating information into existing or newly formed models. For example, with a substantial shift in setting, we might predict that readers will take longer to read newly presented information, exhibit modest difficulty attempting to recollect information from the previous setting, and prefer to recall information from the hereand-now of the story rather than the previous sections of the text. Measures including reading times, recognition latencies, and memory recalls all confirm these predictions (Zwaan & Radvansky, 1998; Zwaan & Rapp, 2006). These core principles of the structure-building framework help provide a general account of how and when readers add to existing mental representations of text events, and/or build new representational structures. Based on these principles, researchers have identified particular dimensions of a text for which readers encode situation models. To date, five such dimensions have received substantial attention, additionally derived from theory and research in fields including literary theory, cognitive science, and discourse comprehension. Specifically articulated in one account derived from the structure-building framework, termed the event-indexing model (although it is more of a descriptive framework than a model), readers track information related to space, time, causality, characters, and character goals (e.g., Zwaan, Langston, & Graesser, 1995; Magliano, Zwaan, & Graesser, 1999). These dimensions are crucially related to story plot, as events take place in different locations, over time, involving protagonists, antagonists, and bystanders with intentions that help set other events in motion. Whenever new spatial or temporal settings, characters, or plot-related causal sequences are introduced, readers can construct new situation models to represent

306  •  Donovan and Rapp

those events, while also encoding information that aligns with existing understandings into already constructed models. A substantial body of research has demonstrated that readers track these various dimensions, both separately and as they necessarily interact with each other over the course of narratives (e.g., Albrecht & O’Brien, 1993; Morrow, Greenspan, & Bower, 1987; Rapp & Taylor, 2004). Several recent accounts attempt to describe the underlying mechanisms and processing consequences of tracking and updating these dimensions in memory, including event segmentation theory (Zacks, Speer, Swallow, Braver, & Reynolds, 2007), and the event horizon model (Radvansky, Krawietz, & Tamplin, 2011), which again align closely with the core propositions underlying the structure-building framework.

CONSTRUCTING CHARACTER MODELS Studies of readers’ tracking of characters have focused on a variety of features including character’s emotions (de Vega, Leon, & Diaz, 1996; Gernsbacher, Goldsmith, & Robertson, 1992; Gygax, Garnham, & Oakhill, 2004), the goals that characters seek to accomplish (Egidi & Gerrig, 2006; Foy & Gerrig, 2014; Magliano, Taylor, & Kim, 2005), empathy for characters (Komeda, Kawasaki, Tsunemi, & Kusumi, 2009; Komeda, Tsunemi, Inohara, Kusumi, & Rapp, 2013), expected character behaviors (Cook, Halleran, & O’Brien, 1998; O’Brien, Rizzella, Albrecht, & Halleran, 1998), the knowledge characters possess and share (Gerrig, Brennan, & Ohaeri, 2001), and character traits (Rapp, Gerrig, & Prentice, 2001). Across these various features, readers generate inferences and expectations for the characters that they read about in stories as they do with actual people they interact with and learn about in their daily dealings (Ross, 1977). For example, when we meet people, we often ascribe the behaviors and thoughts they exhibit to traits they might possess, such as their being friendly, angry, generous, or stingy (to name but a few possibilities). One example of the kinds of trait models that readers construct based on their encoding of character information comes from a study by Rapp et al. (2001). In their project, participants were asked to read stories that each included two separate episodes involving the same character. In the first episode of each story, participants were introduced to the focal character. For example: Philip was graduating from college. Each student received four tickets to invite parents and friends to the ceremony. Philip also had to buy a graduation gown and cap for the event. Everyone at the ceremony was happy and proud of the graduating class. Philip graduated with the highest honors, because he earned a 4.0 GPA. The final sentence of this episode was designed to encourage the inference that Philip is smart. This sentence contrasts with a different final sentence of the same length that other participants read, which did not encourage such a trait inference: The ceremony lasted for almost two hours, but it passed by quickly. After reading this first episode, participants were presented with a second episode involving the same characters. The goal was to determine whether participants would apply any potential trait inference from the earlier episode, derived from the

Updating of Character Information  •  307

final sentence they read, to generate predictions about future character behavior. For example: Philip enjoyed playing poker with a group of friends. One weekend most of the members of the group couldn’t attend. Instead of poker, Philip went over to his friend Greg’s house to play Trivial Pursuit. Philip had never played the game before, and Greg explained the game to him. Greg and Philip made a little cash bet to see who would win the game. At the conclusion of this second episode, participants were tasked with indicating whether they thought a final sentence accurately described what would happen next in the story. The final sentence was either consistent with the potentially encoded trait (e.g., “Philip won the game.”), or inconsistent with it (e.g., “Philip lost the game.”). Participants overwhelmingly responded that trait-consistent outcomes exemplifying traits encoded from the earlier episode were more likely to occur than were traitinconsistent outcomes. For instance, when participants read that Philip had graduated with highest honors and a 4.0 GPA in the first episode, they were more likely to conclude that Philip had won the game in the second episode than if they had read that the ceremony had lasted almost two hours in the first episode. These results indicate that readers use encoded models of characters to make future decisions about what might happen in later text episodes. A crucial question for this kind of work is whether readers spontaneously encode and apply trait models when they are not explicitly tasked with making decisions about future story events. To test this, Rapp et al. (2001) presented participants with the same two-episode stories, and rather than asking them to make explicit judgments of story outcomes, only measured reading times to outcome sentences for the stories (with those outcomes revised to be longer and better integrated into the unfolding stories). In line with expectations derived from the structure-building framework and event-indexing model, participants should take longer to read sentences inconsistent with encoded trait models, as compared to sentences consistent with them. Indeed, participants took longer to read a final sentence like “Philip couldn’t even guess the answers to most of the questions” after reading an earlier episode suggesting Philip was smart than after reading an earlier episode in which no such trait was associated with Philip. These reading time patterns further confirm the view that readers’ expectations for characters, based on previously encoded information, influence subsequent processing of the text. Rapp et al. (2001) also demonstrated that the traits readers encode are relatively specific to the behaviors encountered in the stories. Readers who encoded that Philip was smart believed he would later behave in a particularly intelligent way, but did not have analogous expectations about whether he might be a caring individual. This follow-up set of experiments is crucial for rejecting the possibility that readers might build general models of characters as positive or negative people, rather than more specific models derived from the information most relevant to the stories they read. Of course, as readers become more familiar with characters, they certainly could build more global models of good and bad characters, but at least given the brief kinds of information provided in these multi-episode stories, readers were more conservative in the models they built (but see Gygax et al., 2004, for possible

308  •  Donovan and Rapp

challenges to this idea). Across longer multi-text reading experiences, with more detailed character descriptions and events that illuminate different character traits, these models might, in contrast, also become more nuanced, such that readers may be able to make more fine-grained predictions.

UPDATING CHARACTER INFORMATION Beyond tracking information about characters in stories, readers must also add to or revise what they know about characters as new information is revealed, whether this information is provided in the same story or in a subsequent story (e.g., book series). When readers come to learn that a character they believed was miserly, authoritarian, and/or without redemption can actually change (e.g., Scrooge in A Christmas Carol; Darth Vader in Return of the Jedi), or that the behaviors of characters run counter to their apparent intentions (e.g., Nick Dunne’s wife Amy in Gone Girl), or that events reveal critical changes in the relationships between characters (e.g., Oedipus and his wife/mother in Oedipus Rex), they must update their models. Such updating requires not just adding new information to models of story characters, but also discounting earlier encodings. Updating as a process requires more attention and effort than does merely encoding information onto a consistent existing representation, and is not guaranteed (Kendeou & O’Brien, 2014). This might explain why readers can exhibit difficulty encoding or acknowledging changes in characters or events (O’Brien et al., 1998; Johnson & Seifert, 1994). Analogous failures occur as people fail to integrate news retractions and verified accounts into their understandings of historical events and scientific explanations (Rapp & Braasch, 2014). The difficulty of updating information in memory has been shown using a variety of methods and materials. One empirical demonstration is associated with readers’ apparent unwillingness to disregard earlier, false information in the face of explicit correction (Johnson & Seifert, 1994; Johnson & Seifert, 1998; Ecker, Lewandowsky, & Tang, 2010). While readers might endorse an alternative explanation as a viable cause of an event, their answers to probing questions often reflect and restate information that a retraction should have completely discounted. Thus, information encoded into memory, and associated with causes of events, remains available and influential even in the face of correction. This so-called continued influence effect also occurs with story characters’ descriptions. In a classic series of studies, participants were asked to read stories in which characters were associated with particular behaviors (Albrecht & O’Brien, 1993; Cook et al., 1998; O’Brien et al., 1998). For instance, participants might read a story containing the sentence “Mary, a health nut, had been a strict vegetarian for years.” In a subsequent episode, participants might learn that, counter to expectations, “Mary ordered a cheeseburger and fries.” Participants take longer to read that sentence after learning Mary was a vegetarian than after reading a sentence that did not set up that expectation. More surprisingly, that reading time pattern also emerged, albeit to a reduced degree, even after the potential behavior was followed up with a qualification (“Mary never stuck to her diet when she dined out with friends.”), or completely rejected as valid (“Mary recalled that she had been a health nut and a strict vegetarian for about ten years but she wasn’t anymore.”). So even in the face of contradictory

Updating of Character Information  •  309

claims coupled with reasonable explanations, readers exhibit a resistance to completely overwriting earlier encoded character models. It is worth noting here that such resistance runs counter to the kinds of claims offered in most accounts of text comprehension (e.g., structure-building, event-indexing, etc.) that contend readers update what they know to reflect the here-and-now of a narrative. The fact that participants are influenced by previous narrative information that is no longer valid indicates that updating is not as all-or-none as some accounts might conjecture or hope. So why is updating difficult? One consideration is that readers necessarily rely on early character descriptions as a foundation for constructing character models. Those initial models therefore inform any future information that is received. This aligns with the folk view that first impressions are difficult to change: When we meet someone, their earliest observed behaviors are those to which we will default to in expectations for future behaviors. Work in social psychology has similarly shown that people spontaneously construct trait inferences for people when they first meet and read about them, and that those inferences are strongly held in memory (Ross, 1977; Uleman, Hon, Roman, & Moskowitz, 1996). An added consideration is that information encoded into memory early on has more time to be rehearsed and integrated into other representations than does subsequently encoded information. These factors support so-called primacy effects, with earlier information privileged as compared to later information. At least one contradictory alternative to this account, or one that may co-occur with it, has received recent empirical support, focusing on recency effects (Ecker, Lewandowsky, Cheung, & Maybery, 2015). Consider that in some situations causes that are provided more recently win out over earlier causes, even in the face of counterevidence. And when counterevidence is provided, references are often made to previous explanations, reactivating them and increasing the likelihood they will be used on subsequent tasks. With respect to character models, this might mean that traits encoded into memory, even when rejected by subsequent accounts, nonetheless receive attention by being referenced in any retraction. Such reactivation of previously encoded information can result in the strengthening of that information. Referencing Mary’s food preferences, for example, reactivates earlier mentioned information that she is a vegetarian, making it again available and influential for subsequent decisions, despite being presented in a statement about how she no longer restricts her diet. Researchers have argued that difficulty with updating emerges, at least in part, due to information activating and reactivating in this way as a text unfolds (O’Brien, Cook, & Guéraud, 2010; Kendeou, Smith, & O’Brien, 2013). When readers learn about a character, they encode memory traces of the text into memory. According to a memory-based account, those traces remain available when related cues and concepts appear in the text, resulting in their reactivation. By this account, anything encoded into memory, including inaccurate and inappropriate models of characters, can become available again. Suggesting that earlier information is inappropriate or no longer relevant can thus actually serve to reactivate what should have been discounted. These accounts indicate that it proves difficult to revise what people know about characters, and that information that readers have encoded into memory about those characters can never be completely overwritten. Updating is therefore a distinct challenge, particularly if the kind of updating we are hoping for involves wholesale revision

310  •  Donovan and Rapp

of existing understandings. For that reason, when Sirius Black or Severus Snape are revealed as allies to Harry Potter, rather than adversaries, we might still retain some representation of them as potentially threatening characters to the heroes of the story. Obviously we can overcome the activation of those representations through careful thought and evaluation, since our decisions about what will take place in a story involve but are not entirely dependent upon the memory traces retrieved during comprehension. But retaining some of those discounted ideas does mean that previous understandings, while no longer true, can exert an influence on processing. It is precisely this issue that has led to a rich body of research on the most effective ways of encouraging updating. Much of this work involves examinations of text manipulations and constructions that encourage careful evaluation and reconsideration of the understandings and beliefs that people hold. One method that has received substantial research attention and support along these lines involves the use of refutation texts that call attention to an inaccurate idea and provide an alternative account substantiated with a useful explanation. Refutation materials are particularly popular in science classrooms, as students often reveal a variety of preconceptions that run counter to scientific findings and theories (Vosniadou & Brewer 1992; Chi, 2005). In many different projects, students who have been exposed to refutation texts exhibit better understandings of topics for which they previously held varied preconceptions, in contrast to exposure to nonrefutation texts (in which alternative explanations are provided but without connection to particular inaccurate ideas), and in contrast to a variety of other instructional methods (Guzetti, Snyder, Glass, & Gamas, 1993; Kendeou & van den Broek, 2005; Kendeou & van den Broek, 2007; Broughton, Sinatra, & Reynolds, 2010). Recently, we have demonstrated analogous benefits for the use of refutation materials with respect to history topics in non-fictional narrative descriptions as well (Donovan, Zhan, & Rapp, 2017). Findings from these studies are informative for considering situations in which readers exhibit more substantial (but again, never quite complete) updating of character models. These refutation approaches also align with two critical considerations for our focus in this chapter. First, application of these approaches often occurs across narrative episodes, chapters, and novels, as refutations are regularly situated in multitext experiences. Second, the features associated with refutations are precisely of the type that authors in their stories include to guide readers’ expectations and understandings of characters, including their traits, goals, and plans. In one project demonstrating the effectiveness of refutation approaches for encouraging updating of character information, participants learned that what they believed to be true about characters might be inaccurate (Rapp & Kendeou, 2007). Materials similar to those described earlier were employed here, which should be readily apparent in one of their first episodes: Philip was graduating from college. He was looking forward to the event. Each student received four tickets to invite parents and friends to the ceremony. Philip also had to buy a graduation gown and cap for the event. Everyone at the ceremony was happy and proud of the graduating class. Philip graduated with the highest honors, by earning a perfect 4.0 GPA. The next sentence of the story presented additional information about Philip. For some participants, the story continued in line with the view he is smart:

Updating of Character Information  •  311

He had been selected as the class valedictorian and he even got to give a speech. Other participants read a continuing statement designed to refute the notion Philip was smart: This was unusual for him, because in high school he got by with barely passing grades. Other participants read a statement designed to refute the trait that also included an explanation for why Philip might be mistakenly considered smart: This was unusual for him, and was a result of his father’s donations to the college. This statement not only argues against the trait, but also provides a useful account for why any potential inferences based on such a model of Philip being smart would be inappropriate. After reading one of these first episodes, participants read a second episode with Philip: The next week, Philip and his friends were going to play poker, as they did every weekend. However, most of the members of the group couldn’t attend. Instead of poker, Philip went to his friend Gina’s house to play Trivial Pursuit. Philip had never played the game before, and Gina explained the game to him. Gina and Philip made a little cash bet to see who would win the game. Participants were tasked with deciding whether the next sentence of the story adequately described what they thought would happen next. One such final sentence was “Philip was able to answer most of the questions quite easily,” which aligns with the notion that he is smart. Participants were more likely to agree with that statement after reading a traitconsistent version of the first episode than if they read either of the refutation versions of that episode. Refutations were thus generally effective at reducing participants’ use of the earlier encoded trait information for making subsequent predictions. In a second experiment, participants were not tasked with making an explicit judgment about the likelihood of that final sentence, but rather simply read the sentences after which they answered a general comprehension question about the story events to cover the purpose of the experiment. The goal was to determine whether and when participants exhibited difficulty, measured by reading slowdowns, if events ran counter to trait-based expectations. Participants slowed down to the final sentence “Philip was able to answer most of the questions quite easily” more so after reading the refutation that also included an explanation than after reading the refutation without it. Readers spontaneously updated their expectations that Philip was smart only after reading a refutation with an explanation, as exemplified by slowdowns to sentences consistent with a potentially instantiated trait model. In a third experiment using the same reading time methodology, participants were given instructions that asked them to carefully think about the story, and to vigilantly follow the characters in considering how the stories would unfold. With these instructions, participants exhibited evidence of updating generally. Reading slowdowns to trait-related final sentences now extended to both refutations with explanations, and refutations without them. Participants overall updated their trait models liberally when asked to track characters in the stories. These findings, and others like them, indicate that the refutations that are the most useful for encouraging people to update their

312  •  Donovan and Rapp

understandings of instructional content are also effective for helping people update their representations of characters. Updating is more likely when readers are not only told that previous information about a character is inappropriate or wrong, but also why that information might lead to an incorrect inference. Refutations, particularly those including explanations, may be especially useful in multi-text situations, in which readers may often have a broad range of character information to track and store in memory, potentially over long periods of time.

INFLUENCES ON UPDATING The previous discussion exemplifies an important issue for research on text comprehension generally, whether focused on narrative experiences or multiple-text presentations: Accounts should attempt to identify the particular features that support or restrict the likelihood of updating. For example, readers come to text experiences with a host of understandings, expectations, and beliefs. This prior knowledge is crucial in supporting the decoding of words, the construction of inferences, and the categorization of actions and behaviors into discernable events, to name but a few processes. The degree to which readers possess knowledge about a particular character, derived from reading previous stories, or from anticipations based on dust jacket descriptions, book reviews, word-of-mouth, or from other experiences (e.g., watching a James Bond film before reading a James Bond novel), influences the expectations readers have of how characters will behave. Knowing, for example, that our earlier friend Philip is about to participate on a Jeopardy television taping might encourage expectations that he will do well because, after all, he’s pretty smart. Readers’ prior knowledge represents a rich resource that will necessarily be recruited to support ongoing understandings of events and descriptions provided across text experiences involving the same characters and settings. And the more strongly held the expectations we have for a particular character’s behavior, the more of a challenge it should be to update that information in memory, even when confronted with refutations. If all of our experiences with Harry Potter indicate that he always seeks to do the right thing, we might have difficulty buying the notion that he could be corrupted in a future story. Information stored in prior knowledge, and derived from consistent experience over time, might be particularly resistant to updating. This also implies that it may be easier to update our representations of characters we are only just becoming familiar with both within and across texts. For example, we have only just met Sirius Black in the Harry Potter series, with the suggestion that he is a criminal, when we begin to find out he is actually working for good. This updating seems easy given we have relatively little experience with Sirius, and most of that is hearsay description rather than direct observations of his behavior. Readers may thus exhibit more flexibility in adding to and adjusting their representations of newly met characters rather than old familiar ones across multiple texts. There are a variety of ways in which prior knowledge, and different kinds of prior knowledge (e.g., declarative; procedural; etc.), might influence readers’ attempts to update memory. Here we focus on one that has received attention broadly with respect to its influences on cognitive processes and products, and that is specifically pertinent to models of characters across multiple-text experiences. When we encounter information, we may opt to consider the source providing it, whether that is a fictional character, some narrator, or even a real-world author. For example, individuals believe particular news outlets provide more reasonable, well-investigated accounts of

Updating of Character Information  •  313

current events than others; they feel particular authors are useful sources of information for particular topics; and they rely upon the testimony and opinions of particular friends and family while discounting others, given their expectations for how reasonable or well informed those people are. The credibility of a source is an important consideration for our comprehension and reliance on information. The influence of sources has been demonstrated in a variety of domains, by examining the characteristics most associated with trustworthy and untrustworthy sources (Eagly, Wood, & Chaiken, 1978), the kinds of cues that individuals attend to and that encourage or discourage consideration of sources (Priester & Petty, 1995; Tormala & Clarkson, 2007), and the pragmatic influence of source information on applied decision making (Bell & Loftus, 1989; Klettke, Graesser, & Powell, 2010). Source credibility is an issue that readers of fiction regularly wrestle with. Narrators may or may not be reliable, describing events, situations, and characters with particular biases, sometimes withholding or even being unaware of important considerations for unfolding plot. Characters in narratives can therefore be trustworthy sources of information, or alternatively, dubious informants whose machinations involve fooling protagonists (and the reader) into believing things they shouldn’t. These factors add a layer of complexity to the ease with which readers might develop models of characters. To consider the consequences of such complexity, let’s return to our earlier example of Philip. In the materials that we’ve considered thus far, our descriptions of him have been provided by an omniscient, unbiased narrator. But what if that narrator was someone whose opinion of Philip and his abilities was potentially unreliable? In experiments by Sparks and Rapp (2011), participants again read stories in which characters were described in a first episode, followed by a second episode to which a trait inference might be applied to generate expectations about the appropriateness of story events. In contrast to previous studies, participants were informed that the episodes were descriptions that emerged through conversations as part of a reporter’s project to write about small-town life. Early in the experiment, participants learned that the two informants interviewed by the reporter differed in their respective credibility. Here is a description of one of the informants: Zane Anderson has served as treasurer of River Village for 15 years. In the last election cycle, Zane convinced some of his campaign workers to solicit elderly voters for large donations. Zane then used the donations to buy himself a new sports car. Residents know that Zane is dishonest and untrustworthy. Participants learned the following information about the other informant: Quentin Carter has been the River Village fire chief for 25 years. Whenever someone’s home has been damaged by a severe storm, Quentin helps clean up the debris and pays for a portion of the repairs. Quentin is hardworking and willing to help those in need. Residents know that Quentin is honest and trustworthy. In these stories, the reporter, who was never linked to any credibility information so as to avoid being presented as more or less trustworthy (and in line with the omniscient narrator in the presented experiments), described the first episode as part of collected notes about the town’s populace. The stories unfolded in the following manner:

314  •  Donovan and Rapp

The next person in my notes is Philip Wright. Philip was graduating from college last June. He was looking forward to the event. Each student received four tickets to invite parents and friends to the ceremony. Philip also had to buy a graduation gown and cap for the event. He arrived at the arena that morning, mentally preparing to walk across the stage in front of everyone. Do you know what happened next? At this point, either Zane or Quentin provided the following information: Well, Philip graduated with the highest honors, by earning a perfect 4.0 GPA. He had been selected as the class valedictorian and he even got to give a speech. Because reliable Quentin or unreliable Zane provided the information, readers might believe the description to be more or less true, influencing their expectations for Philip in subsequent scenarios. This was tested by the reporter continuing the interview with a second episode: Okay, so the next week, Philip and his friends were going to play poker, as they did every weekend. However, most of the members of the group couldn’t attend. Instead of poker, Philip went to his friend Gina’s house to play Trivial Pursuit. Philip had never played the game before, and Gina explained the game to him. Gina and Philip made a little cash bet to see who would win the game. At this point, participants were presented with a final sentence that was either consistent with the idea Philip was smart (i.e., “Philip was able to answer most of the questions easily.”) or inconsistent with it (“Philip couldn’t even guess the answers to most of the questions.”). Their task was to decide whether that outcome was likely given what they had read. Importantly, this outcome was not associated with the reporter or either informant, so readers were asked to make their own judgments about how things would play out. The results indicated an important influence of credibility: When the source was reliable, participants were 65 percentage points more likely to agree with trait-consistent than trait-inconsistent story outcomes. However, when the source was unreliable, participants were only 22 percentage points more likely to agree with trait-consistent than trait-inconsistent outcomes. When trait information was provided by the unreliable source, agreement with trait-consistent outcomes was lower, and agreement with trait-inconsistent outcomes was higher. While the findings indicate an important influence of source credibility across separate text episodes, this influence may be rather limited. In other experiments, participants were not asked to make explicit decisions about story outcomes. Instead, their reading times for outcomes were recorded, in line with previously described experiments. If credibility was taken into account during readings of the episodes, we might expect participants to take longer to read different outcomes depending on informational sources. But this was not the case: Participants took longer to read trait-inconsistent than trait-consistent outcomes regardless of whether Zane or Quentin provided earlier information. Even when provided with reminders and additional time to consider the trustworthiness of sources, participants exhibited little in the way of a moment-by-moment effect of credibility. Thus, credibility plays an important role in readers’ encoding and application of trait models, as a function of whether they are motivated to make decisions and evaluations

Updating of Character Information  •  315

about unfolding stories, as shown when readers are asked to decide whether an outcome is likely given what they have read. However, without such motivations, credibility plays less of a role with readers treating information as viable regardless of the source (also see Andrews & Rapp, 2014; van Boekel, Lassonde, O’Brien, & Kendeou, 2017). Given the contingencies of reading stories, which involves tracking characters, considering plot, and a host of other comprehension-building activities, it is perhaps unsurprising that readers might not always consider credibility without additional prompting. This has important consequences for any propensity for updating trait models.

THE COMPLEXITY OF CHARACTERS Authors use a variety of tools to inspire particular responses and predictions in their readerships. And readers have a variety of schematic expectations and beliefs about how stories should play out given previous experiences with narrative fiction. These two contributors to comprehension, sometimes competing and sometimes aligned, create interesting situations for contemporary models of narrative comprehension to reckon with. Consider that characters are often more complex than a simple good or bad, smart or unintelligent, trait-consistent or trait-inconsistent dichotomy would suggest. Readers’ preferences for how they would like things to turn out (e.g., the protagonist will overcome adversity) influence the likelihood they will update character traits and events (Mensink & Rapp, 2011). Preliminary work on this topic indicates that readers revise their models for characters to incorporate what they hope will happen, even when such expectations would run counter to what story events and logical reasoning indicate is appropriate in the circumstances (Rapp & Gerrig, 2002, 2006; Rapp, Jacovina, Slaten, & Krause, 2014). Additionally, readers are usually interested in more nuanced characters than would be associated with a single trait or consistent pattern of behavior. Many of fiction’s most beloved heroes, including Sherlock Holmes, James Bond, and Wolverine from the X-Men, exhibit characteristics associated with ne’er-do-wells, including bad habits (e.g., drug addiction), immoral tendencies (e.g., misogyny), and socially undesirable behaviors (e.g., violent, irrational outbursts). In line with such nuanced associations, while readers might encode that a particular character is, for example, smart, recall this does not mean they will believe that smart characters should possess other positive characteristics (Rapp et al., 2001). Rather, the kinds of models readers seem to generate are specifically targeted to the kinds of behavioral and psychological evidence that they receive in character descriptions. And the previously described findings indicate that when readers do encode particular models of characters, the degree to which those models are amenable to change is a function of how much evidence is provided for traits, and how easy it is to assimilate new descriptions given previous evidence. What this means is that accounts being built from the studies reviewed in this chapter provide only a starting point, albeit an important one, for determining the cues that drive readers’ encoding and updating of complex character representations. Our understandings of readers’ mental representations of characters that populate fictional worlds, who often exhibit dynamic sets of traits and behaviors, should take as an analytical starting point theoretical understandings of how readers encode trait models that vary in their elaboration, malleability, and utility as stories unfold.

TO BE CONTINUED . . . The current volume examines readers’ experiences with multiple texts and sources. Readers’ preferred experiences with fiction, as exemplified by perusing the

316  •  Donovan and Rapp

New York Times Bestsellers’ list as but one potential indicator, involve learning about and following the exploits of characters across multiple books. Characters develop, encounter challenges, submit to weaknesses, and overcome struggles as plots unfold across story settings. Much of what we know about characters, and the kinds of representations that readers build about them, comes from work that has looked at single texts. Nevertheless, many of those experiments, including those reviewed in this chapter, align with the concerns and investments highlighted in multiple-text studies. For both, comprehension requires building coherence across episodes, which involves attempts to integrate such information to generate predictions and consider the relevance of previous content across scenarios and settings. However, in many ways the study of characters as described here using experimental stimuli can differ from our everyday experiences with multiple fictional texts. For example, readers of fiction sometimes have to wait a substantial amount of time before a new text starring their favorite protagonist is released, which could have consequences for their encoding and updating of characters. Readers are also sometimes aware (and critical) of differences across texts describing their heroes’ exploits, as when new authors take up the mantle to produce new adventures (e.g., with the James Bond series), or when stories ignore or reject previously described situations in the interest of plot (e.g., Sherlock Holmes’ death in “The Final Problem”), or even when authors seem to have forgotten what they described in previous books. And many experiences with fiction involve much longer, more involved texts that describe casts of characters in varied settings with complicated, convoluted plots. These differences highlight the need for evaluating whether accounts derived from the empirical evidence described in the extant literature are appropriate for considering multichapter and multi-novel predictions. To date there have only been a handful of studies testing readers’ processing of lengthier text presentations, with few if any looking across multiple fictional texts (e.g., McNerney, Goodwin, & Radvansky, 2011). So again, the work presented here offers a useful starting point for considering the ways in which readers construct and revise models of characters they meet in the stories they read. And of course we might consider other kinds of multi-story presentations that further complicate the materials and conditions studied thus far. Consider a reader who enjoys a Harry Potter novel opting to watch several of the movies before returning to the written series, or a regular viewer of the Dr. Who television show who decides to listen to radio play dramas starring the titular hero, or when a fan of the Batman comic has to wrestle with another relaunch of the character that jettisons previous continuity to fall in line with upcoming film storylines. How do readers reconcile these different depictions within and across media forms, which represent multi-text experiences provided by different sources with different goals describing different kinds of information? Work examining these realworld examples of “multiple-text presentations” will further inform theoretical accounts of how readers encode and update what they know about characters and their stories.

REFERENCES Albrecht, J. E., & O’Brien, E. J. (1993). Updating a mental model: Maintaining both local and global coherence. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 1061–1070. Andrews, J. J., & Rapp, D. N. (2014). Partner characteristics and social contagion: Does group composition matter? Applied Cognitive Psychology, 28, 505–517.

Updating of Character Information  •  317 Bell, B. E., & Loftus, E. F. (1989). Trivial persuasion in the courtroom: The power of (a few) minor details. Journal of Personality and Social Psychology, 56, 669–679. Britt, M. A., & Rouet, J. F. (2012). Learning with multiple documents: Component skills and their acquisition. In J. R. Kirby & M. J. Lawson (Eds.), Enhancing the quality of learning: Dispositions, instruction, and learning processes (pp. 276–314). New York: Cambridge University Press. Broughton, S., Sinatra, G., & Reynolds, R. (2010). The nature of the refutation text effect: An investigation of attention allocation. The Journal of Educational Research, 103, 407–423. Chi, M. T. H. (2005). Commonsense conceptions of emergent processes: Why some misconceptions are robust. Journal of the Learning Sciences, 14, 161–199. Cook, A. E., Halleran, J. G., & O’Brien, E. J. (1998). What is readily available during reading? A memory-based view of text processing. Discourse Processes, 26, 109–129. de Vega, M., Leon, I., & Diaz, J. M. (1996). The representation of changing emotions in reading comprehension. Cognition and Emotion, 10, 303–321. Donovan, A. M., Zhan, J., & Rapp, D. N. (2017). Supporting historical understandings with refutation texts. Manuscript submitted for publication. Eagly, A. H., Wood, W., & Chaiken, S. (1978). Causal inferences about communicators and their effect on opinion change. Journal of Personality and Social Psychology, 36, 424–435. Ecker, U. K. H., Lewandowsky, S., Cheung, C. S. C., & Maybery, M. T. (2015). He did it! She did it! No, she did not! Multiple causal explanations and the continued influence of misinformation. Journal of Memory and Language, 85, 101–115. Ecker, U. K. H., Lewandowsky, S., & Tang, D. T. W. (2010). Explicit warnings reduce but do not eliminate the continued influence of misinformation. Memory & Cognition, 38, 1087–1100. Egidi, G., & Gerrig, R. J. (2006). Readers’ experiences of characters’ goals and actions. Journal of Experimental Psychology: Learning, Memory, and Cognition, 6, 1322–1329. Foy, J. E., & Gerrig, R. J. (2014). Readers’ responses when characters act on completed goals: Impact of characters’ mental states and readers’ task focus. Discourse Processes, 51, 312–332. Gernsbacher, M. A. (1990). Language comprehension as structure building. Hillsdale, NJ: Erlbaum. Gernsbacher, M. A., Goldsmith, H. H., & Robertson, R. R. W. (1992). Do readers mentally represent characters’ emotional states? Cognition and Emotion, 6, 89–111. Gerrig, R. J., Brennan, S. E., & Ohaeri, J. O. (2001). What characters know: Projected knowledge and projected co-presence. Journal of Memory and Language, 44, 81–95. Guzetti, B., Snyder, T., Glass, G., & Gamas, W. (1993). Promoting conceptual change in science: A comparative meta-analysis of instructional interventions from reading education and science education. Reading Research Quarterly, 28, 117–155. Gygax, P., Garnham, A., & Oakhill, J. (2004). Inferring characters’ emotional states: Can readers infer specific emotions? Language and Cognitive Processes, 19, 613–639. Johnson, H. M., & Seifert, C. M. (1994). Sources of the continued influence effect: When misinformation in memory affects later inferences. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20, 1420–1436. Johnson, H. M., & Seifert, C. M. (1998). Updating accounts following a correction of misinformation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 1483–1494. Kendeou, P., & O’Brien, E. J. (2014). The knowledge revision components (KReC) framework: Processes and mechanisms. In D. N. Rapp & J. L. G. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp.  353–377). Cambridge, MA: MIT Press. Kendeou, P., Smith, E. R., & O’Brien, E. J. (2013). Updating during reading comprehension: Why causality matters. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 854–865. Kendeou, P., & van den Broek, P. (2005). The effects of readers’ misconceptions on comprehension of scientific text. Journal of Educational Psychology, 97, 235–245. Kendeou, P., & van den Broek, P. (2007). The effects of prior knowledge and text structure on comprehension processes during reading of scientific texts. Memory & Cognition, 35, 1567–1577. Klettke, B., Graesser, A. C., & Powell, M. B. (2010). Expert testimony in child sexual abuse cases: The effects of evidence, coherence, and credentials on juror decision making. Applied Cognitive Psychology, 24, 481–494. Komeda, H., Kawasaki, M., Tsunemi, K., & Kusumi, T. (2009). Differences between estimating protagonists’ emotions and evaluating readers’ emotions in narrative comprehension. Cognition and Emotion, 23, 135–151.

318  •  Donovan and Rapp Komeda, H., Tsunemi, K., Inohara, K., Kusumi, T., & Rapp, D. N. (2013). Beyond disposition: The processing consequences of explicit and implicit invocations of empathy. Acta Psychologica, 142, 349–355. Magliano, J. P., Taylor, H. A., & Kim, H.-J. J. (2005). When goals collide: Monitoring the goals of multiple characters. Memory & Cognition, 33, 1357–1367. Magliano, J. P., Zwaan, R. A., & Graesser, A. (1999). The role of situational continuity in narrative understanding. In H. Van Oostendorp & S. Goldman (Eds.), The construction of mental representations during reading (pp. 219–245). Mahwah, NJ: Erlbaum. McNerney, M. W., Goodwin, K. A., & Radvansky, G. A. (2011). A novel study: A situation model analysis of reading times. Discourse Processes, 48, 453–474. Mensink, M. C., & Rapp, D. N. (2011). Evil geniuses: Inferences derived from evidence and preferences. Memory & Cognition, 39, 1103–1116. Morrow, D. G., Greenspan, S. L., & Bower, G. H. (1987). Accessibility and situation models in narrative comprehension. Journal of Memory and Language, 26, 165–187. O’Brien, E. J., Cook, A. E., & Guéraud, S. (2010). Accessibility of outdated information. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36, 979–991. O’Brien, E. J., Rizzella, M. L., Albrecht, J. E., & Halleran, J. G. (1998). Updating a situation model: A memory based text processing view. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 1200–1210. Priester, J. R., & Petty, R. E. (1995). Source attributions and persuasion: Perceived honesty as a determinant of message scrutiny. Personality and Social Psychology Bulletin, 21, 637–654. Radvansky, G. A., Krawietz, S. A., & Tamplin, A. K. (2011). Walking through doorways causes forgetting: Further explorations. The Quarterly Journal of Experimental Psychology, 64, 1632–1645. Rapp, D. N., & Braasch, J. L. G., Eds. (2014). Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences. Cambridge, MA: MIT Press. Rapp, D. N., & Gerrig, R. J. (2002). Readers’ reality-driven and plot-driven analyses in narrative comprehension. Memory & Cognition, 30, 779–788. Rapp, D. N., & Gerrig, R. J. (2006). Predilections for narrative outcomes: The impact of story contexts and reader preferences. Journal of Memory and Language, 54, 54–67. Rapp, D. N., Gerrig, R. J., & Prentice, D. A. (2001). Readers’ trait-based models of characters in narrative comprehension. Journal of Memory and Language, 45, 737–750. Rapp, D. N., Jacovina, M. E., Slaten, D. G., & Krause, E. (2014). Yielding to desire: The durability of affective preferences. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40, 1419–1431. Rapp, D. N. & Kendeou, P. (2007). Revising what readers know: Updating text representations during narrative comprehension. Memory & Cognition, 35, 2019–2032. Rapp, D. N., & Taylor, H. A. (2004). Interactive dimensions in the construction of mental representations for text. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30, 988–1001. Ross, L. D. (1977). The intuitive psychologist and his short-comings: Distortions in the attribution process. In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 10, pp. 174–214). New York: Academic Press. Sparks, J. R., & Rapp, D. N. (2011). Readers’ reliance on source credibility in the service of inference generation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37, 230–247. Tormala, Z. L., & Clarkson, J. J. (2007). Assimilation and contrast in persuasion: The effects of source credibility in multiple message situations. Personality and Social Psychology Bulletin, 33, 559–571. Uleman, J. S., Hon, A., Roman, R. J., & Moskowitz, G. B. (1996). On-line evidence for spontaneous trait inferences at encoding. Personality and Social Psychology Bulletin, 22, 377–394. van Boekel, M., Lassonde, K. A., O’Brien, E. J., & Kendeou, P. (2017). Source credibility and the processing of refutation texts. Memory and Cognition, 4, 168–181. Vosniadou, S., & Brewer, W. F. (1992). Mental models of the earth: A study of conceptual change in childhood. Cognitive Psychology, 24, 535–585. Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009). Source evaluation, comprehension, and learning in internet science inquiry tasks. American Educational Research Journal, 46, 1060–1106. Wiley, J., & Voss, J. F. (1999). Constructing arguments from multiple sources: Tasks that promote understanding and not just memory for text. Journal of Educational Psychology, 91, 301–311. Wineburg, S. S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87.

Updating of Character Information  •  319 Zacks, J. M., Speer, N. K., Swallow, K. M., Braver, T. S., & Reynolds, J. R. (2007). Event perception: A mind/ brain perspective. Psychological Bulletin, 133, 273–293. Zwaan, R. A., Langston, M. C., & Graesser, A. C. (1995). The construction of situation models in narrative comprehension: An event-indexing model. Psychological Science, 6, 292–297. Zwaan, R. A., & Radvansky, G. A. (1998). Situation models in language comprehension and memory. Psychological Bulletin, 123, 162–185. Zwaan, R. A., & Rapp, D. N. (2006). Discourse comprehension. In M. Traxler & M. A. Gernsbacher (Eds.), Handbook of psycholinguistics, 2nd edition (pp. 725–764). San Diego, CA: Elsevier.

19

SELF-REGULATED LEARNING PROCESSES AND MULTIPLE SOURCE USE IN AND OUT OF SCHOOL Jeffrey Alan Greene, Dana Z. Copeland, Victor M. Deekens, and Rebekah Freed university of north carolina at chapel hill, usa

The advent of the Internet and other means of electronic dissemination of information has resulted in unprecedented access to, and proliferation of, multiple sources of information (Cisco, 2016). Searching through and making sense of this vast sea of information sources challenges the limits of human memory, attention, time, and expertise (Bromme & Goldman, 2014; Rapp, 2016). The sheer volume of sources available about any given topic requires learners who are able to navigate multiple texts, infer meaning from each of them, and then integrate that meaning into useful understanding and action. Doing so often involves complex cognitive and metacognitive processing (e.g., knowledge elaborations and judgments of learning) about multiple kinds of representations (e.g., text, pictures, video, simulations) of often abstract or complex topics (Azevedo, 2005). Further, these sources can vary in terms of the creators of the information (Macedo-Rouet, Braasch, Britt, & Rouet, 2013), the publication venue and context (e.g., website or newspaper, date of publication; Braasch, Bråten, Strømsø, Anmarkrud, & Ferguson, 2013), as well as who authored the content versus who is distributing it (e.g., when a website known for news publishes a paid advertisement article developed by a company). Finally, in the digital age, there are not only multiple sources of information; there are multiple curators and editors who determine what, when, and how information is disseminated, some of whom take their role seriously (e.g., academics), others who may not (e.g., peers in social media), and some who actively seek to promote fake information to make a profit (Silverman, 2016). Therefore, learners must become critical consumers of not only the information within and across multiple sources, but also the authors and distributors of that information (Alexander & the DRLRL, 2012; Goldman, Britt, Brown, Cribb, George, Greenleaf, Lee, & Shanahan, 2016; Greene, Yu, & Copeland, 2014; Leu, Forzani, Rhoads, Maykel, Kennedy, & Timbrell, 2015; Stanford History Education Group, 2016).

320

Self-Regulated Learning Processes  •  321

Such complex sourcing, information processing, and synthesis strategies, as well as the will to enact them, are sadly not natural, and must be learned, practiced, and often effortfully deployed when learners would rather be pursuing other goals (Azevedo, Johnson, Chauncey, & Graesser, 2011; Bennett, Maton, & Kervin, 2008; Braasch, Bråten, Strømsø, Anmarkrud, & Ferguson, 2013; Braasch, Lawless, Goldman, Manning, Gomez, & Macleod, 2009; Goldman, 2011; Greene, 2018). Research on such knowledge, skills, and dispositions has fallen within various intellectual traditions including multiple source use (Braasch et  al., this volume; Bråten, Britt, Strømsø, & Rouet, 2011; Britt, Richter, & Rouet, 2014; Britt, Rouet, & Dunkirk, this volume; Rouet & Britt, 2011), new literacies (Hartman, Hagerman, & Leu, this volume), epistemic cognition (Greene, Sandoval, & Bråten, 2016), and the focus of this chapter: self-regulated learning (SRL; Schunk & Greene, 2018; Zimmerman, 2013). Zimmerman and Schunk (2011) defined SRL as “the processes whereby learners personally activate and sustain cognitions, affects, and behaviors that are systematically oriented toward the attainment of personal goals” (p. 1). SRL knowledge, skills, and dispositions mediate relations between learners and their learning goals, including the effective use of multiple sources (Zimmerman, 2013). In this chapter, we review theoretical foundations of SRL and their relevance to multiple source use, and then synthesize relevant empirical literature. Given that there are other chapters in this Handbook that include detailed and informative reviews of the literature on the cognitive strategies involved in multiple source use, we chose to focus our discussion on the SRL processes involved in planning, monitoring, controlling, and evaluating the efficacy of those cognitive strategies. The reviewed empirical literature includes both studies involving multiple sources across documents (e.g., searching on the Internet) as well as studies involving multiple sources within a single document (e.g., refutational texts presenting arguments from multiple sources). Our review of the limited but informative corpus of SRL intervention studies involving multiple sources provides a springboard for the last section of this chapter, wherein we discuss implications for theory, research, and practice. We have several reasons to believe multiple source use literature would benefit from greater integration of SRL theory and findings. First, as Rouet, Britt, and Durik (2017) have noted, their REading as Problem SOLVing (RESOLV) model, designed to extend their MD-TRACE model (Rouet & Britt, 2011) to include the interpretative problemsolving nature of multiple source use, shares common features with models as SRL (e.g., Winne & Hadwin, 2008). Second, the substantial literature on the influence of cognitive, metacognitive, and epistemic cognition processing on multiple source use during learning (e.g., Bråten et al., 2011) can be usefully supplemented with SRL research on the relevant factors that occur before (e.g., task definitions, planning, motivation) and after (e.g., reflection) multiple source use. Likewise, there is a sizable literature on SRL interventions, which can be used to bolster the growing literature on interventions designed to foster effective use of multiple sources (see Section V of this volume). Finally, expanding the SRL connections to multiple source use research, and vice versa, reveals many promising directions and implications for future theory, research, and practice.

THEORETICAL BACKGROUND SRL models describe how people plan, enact, monitor, control, and self-evaluate the cognitive, metacognitive, behavioral, motivational, and affective aspects of active

322  •  Greene et al.

learning (Greene, 2018; Winne & Hadwin, 2008; Zimmerman, 2013). Fundamentally, what typifies self-regulated learning is a willingness and ability to inhibit typical or automatic learning processes (e.g., uncritically reading text) when circumstances require a different, more thoughtful approach. For example, casually scrolling through a social media feed to read about your family’s and friends’ latest adventures may be a relatively automatic process for most people, requiring little active critiquing or integrating of information across posts. On the other hand, when reading multiple social media posts about an important and controversial topic, such as a political debate about a policy that will affect one’s finances (e.g., taxes, health care), self-regulated learners will inhibit the automatic and relatively unreflective processes they typically enact for social media, and replace them with more active, thoughtful, and reflective processing. As such, learners who enact SRL are more proactive and aware of their learning processes than their peers who do not enact SRL, as well as more empowered to manage their learning in an agentic way. It is not feasible to enact SRL processing at every moment of every day, therefore effective self-regulation involves knowing when to invoke more effortful SRL in lieu of automatic processing, and being willing and able to do so (Greene, 2018). Though there are multiple SRL models, they share four premises (Pintrich, 2004). The first is that people are active participants in their learning. The second is that “learners can potentially monitor, control, and regulate certain aspects of their own cognition, motivation, and behavior as well as some features of their environments” (Pintrich, 2004, p. 387). The third is that there is some goal, criterion, or standard against which learners base judgments and upon which they make decisions. The fourth and final assertion is that the SRL process mediates the relationships between personal and contextual characteristics and performance. Researchers have organized SRL processing into three loosely ordered, chronological phases: before (i.e., Forethought), during (i.e., Performance), and after (i.e., Self-Reflection) learning (Zimmerman, 2013). In many instances students enact each phase in order, but sometimes phases can be and should be skipped or repeated. In the Forethought phase of SRL, learners engage in task analysis, which includes goal setting and strategic planning. In this phase, self-regulated learners are also actively aware of, and know how to modify, their self-motivation, beliefs, and values that may influence learning, such as whether they are interested in the learning task, their outcome expectations, or their beliefs about whether they can complete the learning task. During the Performance phase of SRL, learners engage in self-control and selfobservation while learning. Self-control encompasses focusing attention and concentrating on the task, self-instruction, and help-seeking to complete the learning task. During self-observation, learners engage in metacognitive monitoring and self-recording to measure their progress toward the learning goal. SRL involves enacting control when insufficient progress is being made, with effective learners having a wide variety of highutility strategies at their disposal (Dunlosky & Thiede, 2013). During the Self-Reflection phase of SRL, learners make self-judgments and have self-reactions as they reflect on their learning process. Self-judgments include causal attributions about the learning process (Weiner, 2010). The self-reactions learners make during the Self-Reflection phase include self-satisfaction or affective responses and adaptive or defensive reactions to their learning experience. The cyclical nature of SRL is such that the Self-Reflection phase naturally informs the Forethought phase of future learning tasks. For example, self-regulated learners will

Self-Regulated Learning Processes  •  323

determine whether they sufficiently met their goal and if not, they will modify aspects of their cognition, motivation, affect, behavior, or context to increase the likelihood of achieving similar goals in the future (e.g., choosing to spend more time finding a diverse set of sources to inform their understanding of controversial issues). The RESOLV model (Britt et al., this volume) incorporates numerous self-regulatory ideas into previous models of reading and source use such as the TRACE (Rouet, 2006) and MD-TRACE (Rouet & Britt, 2011) models. In the RESOLV model, readers translate internal and external resources, such as a teacher’s assignment and their own values, into a representation of the task, called a Context model. This aligns with Winne and Hadwin’s (2008) task definition phase. Next, Britt and colleagues described how readers use the Context Model to create a Task Model, composed of their own goals, plans, and values for those goals and plans, mirroring the goals and planning phase of Winne and Hadwin’s model. All of these processes are also encompassed in Zimmerman’s (2013) Forethought (i.e., Before Learning) phase, which illustrates how prior knowledge and motivation can influence the adaptive, goal-directed activity highlighted by the RESOLV model. Zimmerman’s next phase, Performance (i.e., During Learning), encompasses the kinds of metacognitive monitoring and control Britt and colleagues posited, including their Feelings of Knowing Evaluations, which are assessments of “one’s knowledge of a particular topic or question” (Rouet et al., 2017). In both SRL models and RESOLV, effective readers enact control when they realize that their current task progress, strategy use, or understanding are not meeting the standards they derived from their Task Model. SRL models place more emphasis upon processing during the Self-Evaluation (i.e., After Learning) phase than the RESOLV model, and how that processing can affect future reading and learning. Finally, the theoretical roots of SRL models, as well as the RESOLV model, include assumptions of idealized task performance and linear progression through the phases of learning. The most current versions of these models include explicit assumptions that processing, rather than being linear and coldly cognitive, is iterative, complex, and driven by motivation and emotion, often leading to intentional recursion among the phases, and revision of past phase products (e.g., Winne & Hadwin, 2008; Zimmerman, 2013). Clearly, the RESOLV model incorporates numerous aspects of Winne and Hadwin’s (2008) SRL model, and aligns well with other SRL models such as Zimmerman’s (2013) work. Likewise, many of the same factors posited to influence SRL have also been found to affect learners’ multiple source use, including task understanding (Bråten & Strømsø, 2003), effort and strategy use (Bråten et al., 2014), motivation and selfefficacy (Zimmerman, 2013; Bråten et al., 2013), affect and emotions (Bowler, 2010a), and epistemic beliefs and cognition (Bråten & Strømsø, 2003; Greene, Bolick, Jackson, Caprino, Oswald, & McVea, 2015). However, we argue that SRL models include several additional components, particularly in the Before and After phases that, when generalized to multiple source use, point to several promising directions for future research and implications for practice.

REVIEW OF LITERATURE ON SRL AND MULTIPLE SOURCE USE Learning and comprehending across multiple sources requires both higher and lowerlevel cognitive and metacognitive processes that can differ among individuals and

324  •  Greene et al.

even the domain or context of the task. In this literature review, we will focus on the metacognitive processes associated with the before, during, and after phases of SRL and their connections with multiple source use. Given the multiple source use literature, relevant individual difference variables can include, at the most basic level, wordlevel recognition (Bråten, Ferguson, Anmarkrud &, Strømsø 2013), and continue through individual differences in motivation (Bråten et  al., 2013), prior knowledge (Maier & Richter, 2014), and availability of strategies to increase comprehension of the material (Bråten et al., 2013; Maier & Richter, 2014) (see also Barzilai & Strømsø, this volume). A large portion of multiple source use research has involved studies of the flexible use of strategies employed to comprehend reading material, with an emphasis on metacognition and how such processing informs how learners monitor their progress toward task goals (e.g., Anmarkrud, McCrudden, Bråten, & Strømsø, 2013; Bråten & Strømsø, 2003; Cho & Afflerbach, this volume; Hagen, Braasch, & Bråten, 2014). Successful learners monitor and control their reading pathways and reading processes, as well as make evaluative judgments on content while using multiple sources (Cho, 2014; Coiro & Dobler, 2007). In addition, research on comprehension processes and strategy use has linked reader processes to broader models of cognition (e.g., construction-integration models; Kintsch, 1988). The RESOLV model, compared to MD-TRACE, includes an expansion of the variety of cognitive and motivational processing that occurs before learning, particularly in terms of findings from Winne and Hadwin’s early work (Rouet et al., 2017). Much more can be gained by incorporating ideas and findings from Winne and Hadwin’s later work (e.g., Winne & Hadwin, 2008), as well as from more contemporary models of self-regulated learning, including how motivation, emotion, calibration, and reflection operate during task definition, goal-orientation and planning, and performance (Efklides, 2011; Moos, 2014; Pintrich, 2004; Zimmerman, 2013). To review these contributions, we group them using Zimmerman’s (2013) phases: Before, During, and After Learning.

BEFORE PROCESSES Task Definition and Goal Setting Both the RESOLV and SRL models position task types, and learners’ constructed definitions of those tasks, as predictors of plans and enacted strategies (Britt et  al., this volume; Winne & Hadwin, 2008; Zimmerman, 2013). For example, learners may develop different strategies in response to different tasks. Bråten and Strømsø (2003) found that law students who studied multiple sources used different reading behaviors when reading for exam review compared to reading to keep up with lectures. When keeping up with a lecture, students used more memorization than when preparing for an exam. Conversely, when preparing for an exam, students used more elaboration of content, comprehension confirmation utterances, and evaluation of the content statements than when keeping up with lecture. Thus, the type of task learners engage in, and their definitions of those tasks, affect their study behaviors. However, effective task definitions require more than just an accurate idiosyncratic representation of the task, but also a sense of the potential solution paths or plans needed to complete the task (Lazonder & Rouet, 2008). Investigations of variance in task definitions before learning, and effects upon planning and solution paths, paint a

Self-Regulated Learning Processes  •  325

complex picture. For example, in their study of students using multiple sources from the same author within a single learning environment, Greene, Hutchison, Costa, and Crompton (2012) captured learners’ understanding and quality of task definitions through prompts to define tasks in learners’ own words. The quality of task definitions at pretest negatively predicted the frequency of monitoring processes, and positively predicted posttest performance. Thus, generating a high-quality task definition before engaging in a task is beneficial to learners. However, learners’ task definitions, on average, improved from pretest to posttest, supporting the argument that it is helpful when learners monitor, control, and improve their task definition as a result of processing during learning (Winne & Hadwin, 2008). Therefore, task definitions are not static; effective self-regulated learners will adapt those task definitions as needed when “during learning” processing indicates the need to do so (e.g., when learners realize the online history webquest they thought only required the gathering of information actually requires corroboration and scrutiny of sources). Learners make these adaptations both before and during learning, but their efficacy depends upon prior knowledge and calibration (Pieschl, Stahl, Murray, & Bromme, 2012). Prior Knowledge and Calibration Findings from the SRL literature support the presumed positive relationship between prior knowledge and the frequency, and utility, of planning, monitoring, and strategy use during learning (Greene, Costa, Robertson, Pan, & Deekens, 2010; Moos & Azevedo, 2008). Within the multiple source use literature, there is evidence that prior knowledge predicts the degree to which learners rely upon the expertise of the authors of the sources they encounter (McCrudden, Stenseth, Bråten, & Strømsø, 2016). Further, Coiro and Dobler (2007) found that several types of prior knowledge were positive predictors of learning on the Internet: prior topic knowledge, prior knowledge of search engines, and prior knowledge of text structure and informational website structures. These aspects of prior knowledge contributed to making meaning and understanding information across multiple sources. Other specific prior knowledge, such as meta-textual knowledge (i.e., knowledge of text structures and features such as linguistic cues, rhetorical devices, and headings), also serves as a powerful predictor of learning, allowing learners to efficiently navigate content online, locate needed information, and read in a targeted, goal-directed manner, rather than a non-linear one (Brand-Gruwel & Stadtler, 2011; Rouet & Coutelet, 2008; Rouet & Le Bigot, 2007). However, learners with low prior knowledge are often poorly calibrated in terms of judging what they do and do not know (Dunlosky & Thiede, 2013) as well as how much SRL will be needed to complete the task (DiFrancesca, Nietfeld, & Cao, 2016). This may explain why Pieschl et al. (2012, 2014) found that despite learners being able to adapt their plans and strategy use to differences in task complexity and goals, such adaptations were often not predictive of learning. If learners are poorly calibrated to their own knowledge, they may not know how to adapt best to varying task demands, and they may not know whether to rely on the expertise of the authors of the multiple sources they encounter (McCrudden et  al., 2016). Low prior knowledge, and subsequent effects upon the likelihood of self-regulation, are likely also moderated by students’ motivation (Winne & Hadwin, 2008). Therefore, as Britt and colleagues (this volume) posited, knowledge and motivation play key roles in readers’ construction

326  •  Greene et al.

of their Task Model, and the evidence reviewed here suggests internal and external factors interact in complex ways, resulting in an idiosyncratic, and sometimes inappropriate, understanding of what acceptable performance on the task looks like. Motivation Motivation plays a crucial role in enactment and sustained continuation within the task, particularly when using multiple sources. As Rouet and colleagues (2017) noted, readers’ values and interest in the task can determine what they choose to do, how they choose to do it, and how they do or do not persist through difficulties. Evidence from SRL research supports such claims. For instance, several studies have shown that interest and self-efficacy relate positively to multiple text comprehension. Bråten et al. (2014) found that individual interest in a topic positively predicted a measure of effort during a learning task, which subsequently predicted multiple text-level comprehension. In general, situational interest is positively related to the use of deeper level strategies during reading, and therefore higher levels of comprehension (Hidi & Renninger, 2006). Bowler (2010a) found a learner’s interest in a topic positively influenced information-seeking processes. In addition, Bowler (2010b) found that curiosity predicted not only what was read, but also the efficiency of information seeking. Learners may show curiosity for relevant material, but also for information on a topic that is not relevant to an assigned task. Such curiosity, directed toward irrelevant information, must be self-regulated for learners to focus successfully upon completing their task. Finally, Bråten et al. (2013) found that individual self-efficacy of a science topic positively predicted multiple text comprehension, but that task value did not predict comprehension of multiple texts. Therefore, the motivation factors that learners bring to a task are related to performance. Such complex relations between factors prominent before learning (e.g., individual interest, curiosity) and those active during learning (e.g., effort, strategy use, comprehension) highlight the importance of considering multiple use from an iterative, recursive perspective such as ones outlined in RESOLV and modern models of SRL (e.g., Efklides, 2011; Winne & Hadwin, 2008; Zimmerman, 2013).

DURING PROCESSES According to SRL research, competent readers are able to monitor and control their comprehension (Thiede & de Bruin, 2018). Britt and colleagues (this volume; Rouet et al., 2017) justifiably emphasize the role of Feeling of Knowing Experiences (FOKE) as a cue to enact control, such as changing strategies. Other metacognitive experiences such as judgments of understanding and ease of learning can also cue a need to enact control (Efklides, 2011). Self-regulated readers use various strategies to meet a variety of purposes and goals, as well as modify or change their reading goals as needed as they progress through the material (Minguela, Solé, & Pieschl, 2015). However, effective self-regulated monitoring and control processes may differ by domain, situation, or task (Greene et al., 2015). For example, learners engaging in a science task involving multiple sources in the same environment exhibited more frequent judgments of learning than learners engaged in a history task (Greene et al., 2015).

Self-Regulated Learning Processes  •  327

Bråten and Strømsø (2003) tracked students over time and found that during reading of multiple sources, students enacted control by moving from simpler strategy use, such as memorization, to increasingly more elaborative and organizational strategy use. Such control behaviors suggest that students were monitoring their learning. Monitoring for understanding in addition to problem detection and problem solving were three evaluative ways learners monitored their comprehension. Further, students who displayed a higher frequency of comprehension confirmation received the highest grades on their exam. However, one key consideration from the perspective of models of SRL is to understand contingent relations among strategies and monitoring processes, such as how enacting a sourcing strategy affects the likelihood and quality of subsequent monitoring of understanding and progress (Ben-Eliyahu & Bernacki, 2015). When the results of monitoring suggest a change in strategy use is needed, whether learners actually do so is a better predictor of learning than the presence of monitoring or strategy use alone (Binbasaran Tüysüzoğlu & Greene, 2015). Another aspect of SRL during learning that warrants investigation is attention to source information. As readers progress through multiple documents, they can engage in multiple evaluative processes indicative of source evaluation (Goldman et al., 2012), such as paying attention to source information embedded in the reading material. Attending to source information includes explicitly recognizing source information such as the author or publication, in addition to implicit source references that compare content in one source to another source (Strømsø, Bråten, Britt, & Ferguson, 2013). As such, learners are monitoring aspects of their cognition by attending to sources. Strømsø et al. (2013) found that learners using multiple sources both attend to and evaluate source information spontaneously, and in context. Attention to sources’ information did not always result in evaluation of that information, however. In addition, Gerjets, Kammerer, and Werner (2011) found that when given explicit instructions to evaluate multiple sources, learners improved on the quality of related evaluation criteria, changed their processing and strategy use during learning, and improved their information problem solving relative to the task. Clearly, the degree to which learners attend to, and self-regulate their attention to, source information can affect how they interact with and evaluate multiple sources.

AFTER PROCESSES There has been very little research on the processes learners use after reading or completing a multiple source use learning task. The roles of self-reflection and judgment are important to consider as readers and learners engage in multiple source use. Self-reflection and judgment can serve as feedback information that informs processing during current and future tasks (Zimmerman, 2013). Particularly during ill-structured tasks, such as reading multiple documents about a science topic with differing viewpoints, positive feedback may encourage continued motivation or volition to persist despite the challenges of the task. More specifically, Maier and Richter (2014) studied the relationship between text beliefs and the comprehension of multiple documents with conflicting information. Results indicated that providing readers with metacognitive strategies that allowed them to directly address differing beliefs within multiple documents, in addition to providing them positive feedback, produced enough support to motivate them to engage in documents with inconsistent

328  •  Greene et al.

beliefs and promote a stronger memory of the contents of their reading. This research supports the iterative nature of SRL, as feedback, delivered after learning, can affect motivation for subsequent learning tasks, even when that task is a recursive revisiting of prior texts (Zimmerman, 2013). The iterative nature of SRL benefits learners during multiple source use, as well as when they pause to evaluate their progress and use this feedback to determine subsequent performance. Further research is needed into other aspects of “after” learning processes, such as attributions and emotions when using multiple sources (Weiner, 2010). The RESOLV model could be usefully enhanced by incorporating these after learning processes.

INTERVENTIONS ON SRL DURING MULTIPLE SOURCE USE The expanding use of the Internet and technology as tools for learning and decision making has inspired multiple researchers to investigate how to enhance the metacognitive and self-regulatory skills of learners to improve their ability to learn in complex digital environments. Results from recent meta-analyses that measured the efficacy of SRL skills interventions (de Boer, Donker, & van der Werf, 2014; Dent & Koenka, 2016; Dignath & Büttner, 2008) have consistently demonstrated that classroom instruction focused explicitly on SRL intervention can increase students’ ability to regulate their own learning. However, despite the thoroughness of these reviews, none of these meta-analyses specifically described SRL interventions that targeted students learning across multiple sources, such as the Internet (for more on multiple source use interventions, see Section V of this volume). While none of these recent meta-analyses specifically described SRL intervention across multiple sources, Lazonder and Rouet (2008) incorporated multiple metacognition intervention studies in a special issue of Computers in Human Behavior. We selected three studies, specifically those authored by Stadtler and Bromme (2008), de Vries, van der Meij, and Lazonder (2008), and Kuiper, Volman, and Terwel (2008), for inclusion in this chapter as being representative of types of interventions conducted to enhance students’ use of self-regulatory skills, such as metacognition, when learning from multiple sources. Additional studies from Naumann, Richter, Christmann, and Groeben (2008), Raes, Schellens, De Wever, and Vanderhoven (2012), and Zhang and Quintana (2012) are also described. Stadtler and Bromme (2008) attempted to enhance undergraduate students’ selfregulation skills when learning about a medical topic outside of a school environment by conducting a direct intervention using a software tool they created, met.a.ware. Researchers provided participants with links to websites selected to represent a diverse set of both sources and perspectives about high cholesterol. Participants were asked to use the websites in order to provide advice to a fictional friend recently diagnosed with high cholesterol. The met.a.ware program was specifically designed to provide support for learning by enhancing students’ ability to conduct two distinct metacognitive processes: evaluating the quality of information and monitoring their personal understanding of the content. In support of this goal, met.a.ware software repeatedly prompted users to focus on their own cognitions, and also provided an external representation to support learners as they developed their own internal representation of the content. The software was used to scaffold non-expert learners’

Self-Regulated Learning Processes  •  329

self-regulatory skills by providing online prompts designed specifically to encourage learners to assess sources and monitor their own comprehension as they learned about a complex medical topic. Learners in the first two experimental conditions received either (a) evaluation prompts, which forced them to evaluate the website authors in terms of both credentials to provide information and interest in presenting unbiased information, or (b) monitoring prompts, which required the learners to report on their own understanding of the topic. A third experimental group received both types of prompts. Simultaneously, all members of the experimental groups received support from met.a.ware through learning-task-specific categorization tools that aided effective note-taking by providing relevant categories for classification. Learners in two separate control conditions did not receive evaluation or monitoring prompts and took notes either by hand or electronically. The results were mixed. On the factual knowledge and comprehension tests, only one experimental condition (i.e., monitoring prompts only) outperformed the control group. On the measure about knowledge of sources, participants who received evaluation prompts and those who received both evaluation and monitoring prompts outperformed the control conditions; however, those that received only monitoring prompts were not significantly different from the control condition. In a study with clearer findings than Stadtler and Bromme’s, de Vries, van der Meij, and Lazonder (2008) conducted two experiments in a classroom environment using a sample of elementary school students and a design-based research methodology to measure the effectiveness of two tools, a web-based portal and a worksheet, designed to supplant self-regulatory skills. Specifically, the tools were designed to enhance reflective web searching, which requires learners to activate prior knowledge and integrate it with new information to enhance understanding. Reflective web searching was described as having three phases: owning the question, interpreting and personalizing retrieved information, and adapting information. The researchers used a web portal and a worksheet to support self-regulatory skills and, simultaneously, used a collaborative classroom environment to enhance the personalization of knowledge and learning. The web-based portal was specifically designed for the learning task to ensure elementary-age students selected task-relevant sources that aided in the personalization and interpretation of information. Likewise, the worksheet was designed to enhance learners’ adaptation of new knowledge by requiring them to write down their own solutions. The authors concluded that the reflective web searching tool aided participants by limiting the search space and providing task-relevant cues to help them find and select the best sources. Simultaneously, the focus on developing a collaborative learning environment aided learners as they interpreted and personalized the information. As opposed to invoking SRL skills with an intervention as described by Stadtler and Bromme (2008) and de Vries and colleagues (2008), Kuiper, Volman, and Terwel (2008) described an 8-week intervention that focused specifically on teaching 5th-grade students self-regulatory skills during web searching. Using a multiple case study design focused on 12 pairs of students, Kuiper and colleagues investigated gains made in both content knowledge and Internet learning skills as students learned about healthy food from multiple sources. Students demonstrated improvement in their content specific knowledge, but at the conclusion of the intervention students

330  •  Greene et al.

remained inconsistent users of the multiple sources they encountered on the Internet. In general, students failed to enact planning before beginning the learning task and made only cursory evaluations of source quality and relevance. Similarly, Raes, Schellens, De Wever, and Vanderhoven (2012) sought to compare the effects of two different types of scaffolding on both the domain-specific knowledge and metacognitive awareness of high school students in an authentic classroom environment. Raes and colleagues created four conditions using two different types of scaffolding, technology-enhanced and teacher-enhanced, as students learned about global warming and climate change, a complex science topic. The four conditions were the control group, the technology-enhanced scaffolding-only group, the teacher-enhanced scaffolding-only group, and a group which received both teacher and technology-enhanced scaffolds. The students learned in dyads using a web-based learning platform during four lessons that were part of a four-week field study program. The technology-enhanced scaffolding consisted of on-screen prompts designed to encourage students to employ metacognitive processes they already knew, including instructions to evaluate the source of information, prompts to summarize content they had just read, or reminders to clearly define the task. These technology-enhanced prompts faded over time, becoming less directive. Teacher-enhanced scaffolding was implemented by graduate students who stimulated metacognitive processes, supplementing the classroom instructors. Participants in the group that received both teacher-enhanced scaffolding and technology-enhanced scaffolding as well as participants in the teacher-enhanced scaffolding-only group demonstrated domain-specific learning gains that were statistically significantly different from those in the control group. Outcomes for students in the technology-enhanced scaffolding group were not significantly different from the control group, indicating the value of the teacher-enhanced scaffolding for domain-specific knowledge gain in this study. Changes in metacognitive awareness, measured using a self-report instrument, indicated that technology-enhanced scaffolding was a more effective tool than teacher-enhanced scaffolding. Overall, Raes and colleagues found support for the value of multiple distributed types of scaffolding to both enhance content knowledge and SRL skills. Zhang and Quintana (2012) also investigated the role of SRL scaffolding in a naturalistic classroom setting using a software tool, Digital IdeaKeeper, which was designed to help students plan their inquiry, search for information, and analyze and synthesize information as they learned online about a scientific topic. The tool was designed to support four scaffolding strategies for online inquiry. First, it provided an integrated work space guiding students to conduct their inquiry using the Digital IdeaKeeper tool as opposed to planning or note-taking on paper. Second, it made the implicit learning activities that students often skip, including content and source evaluation, explicit by providing structure. Digital IdeaKeeper also explicitly supported planning and monitoring by requiring students to establish driving questions and keeping them visible as part of the planning module throughout the learning process. Finally, Digital IdeaKeeper was designed to reduce the cognitive workload often associated with online inquiry by creating logs of student activity and providing students access to these logs. This eliminated the requirement for students to track the mechanics of their search, freeing resources to focus on more meaningful learning processes.

Self-Regulated Learning Processes  •  331

In this study, eight pairs of 6th graders were tasked to select questions and then investigate these questions online over the course of approximately two weeks, culminating in the production of an essay and brochure on their chosen topics. Students conducted online inquiry in pairs during class time but, as this study was conducted in a natural setting, students were permitted to conduct research at other times. Student research was conducted in pairs but the final products were produced individually. The eight pairs of students were split into two groups. Four of the pairs conducted their inquiry using Digital IdeaKeeper while the other four pairs of students used Google and a paper notebook for their inquiry. Overall, Zhang and Quintana (2012) found support for the use of Digital IdeaKeeper scaffolding to enhance engagement, efficiency, and self-regulation. Specifically, with respect to SRL, students in the Digital IdeaKeeper group frequently monitored their progress using the structured planning module whereas those in the Google group rarely monitored. The structured environment also aided goal setting. Naumann, Richter, Christmann, and Groeben (2008) investigated the role of working memory capacity and reading comprehension as moderators of the efficacy of an intervention to enhance undergraduates’ use of cognitive and metacognitive strategies to learn the psychology of visual perception from multiple sources. Naumann and colleagues hypothesized that due to the cognitive resources required to enact both cognitive strategies (e.g., elaboration and organization) and metacognitive strategies (e.g., planning and monitoring), learners with lower working memory capacity, and learners with less routinized reading comprehension skills, would not benefit from strategy training. The authors found support for both hypotheses as students with higher working memory capacity and more routinized reading comprehension skills performed better on a written essay than those who scored lower on these measures, with the latter students underperforming a control group who did not receive the intervention. Naumann and colleagues concluded that these findings helped to explain why interventions that do not account for these factors may not lead to enhanced learning outcomes for all students. Taken together, these example studies indicate an interest from the field in understanding the role and effects of interventions targeting SRL skills. Collectively, the findings discussed in the selected studies indicate the value of direct instruction and targeted interventions to enhance SRL skills in multiple source environments both in and outside of classroom. Simultaneously, these findings create additional questions about the most efficacious ways to target SRL interventions across learner age range, academic contexts, and academic domain, highlighted by the challenges and mixed results that SRL interventions to date have generated. Additionally, Naumann and colleagues’ (2008) findings highlight the role individual differences such as reading comprehension, gender, or prior knowledge may play in the efficacy of SRL interventions.

IMPLICATIONS FOR THEORY, RESEARCH, AND PRACTICE Thankfully, the growing prominence of multiple source use skills as a key 21st-century competency has been matched by a commensurate increase in research into how to help people self-regulate their learning with multiple sources. The research reviewed in this chapter, and the work in this Handbook, indicate a strong foundation from

332  •  Greene et al.

which to build better theory, research, and practice in the field. Such reviews also indicate a number of promising directions for future work, with major areas of emphasis including: (1) the exploration of the role of motivation and affect, (2) investigations of the role of ego depletion (Baumeister, Vohs, & Tice, 2007), (3) the need for specific types of studies to understand domain, task, and context-specificity, and (4) the need for educators to invest resources into the development of interventions that foster the independent use and transfer of these essential SRL processes when learning with multiple sources. Motivation and Affect There has been promising work on the role of self-efficacy and interest in the ways in which people engage in multiple source use (e.g., Bråten et al., 2013, 2014). SRL research has long established that motivation predicts effort and engagement; indeed, it is unsurprising that the general “warming” trend in educational research (Pintrich, Marx, & Boyle, 1993) has been mirrored in research on multiple source use (Rouet et  al., 2017). Interesting directions for theory and research remain, including an expanded conceptualization of motivation and investigations of affective aspects of metacognitive experiences. As one example, research on motivation has begun expanding into metamotivation, or the ways people self-regulate their motivation to optimize their arousal in pursuit of goals (Miele & Scholer, 2016; Wolters & Benzon, 2013). Motivation during multiple source use, particularly in authentic contexts such as reviewing numerous sources for an integrative review paper for a college class, likely waxes and wanes over time. One likely predictor of persistence during multiple source use, beyond individual characteristics such as interest or self-efficacy, is learners’ efficacy at managing their motivation, such as bolstering their motivation when the task takes longer than expected. Likewise, affective responses to events during learning can inform persistence and performance (Efklides, 2011). Learners can experience many emotional responses to both the content being learned as well as to the results of monitoring progress toward goals (Boekaerts & Pekrun, 2015). These responses, and learners’ interpretations of them, can cue monitoring and control behavior, such as when a feeling of frustration precedes the conscious realization of misunderstanding (Efklides, 2011), and subsequent changes to strategy use (Binbasaran Tüysüzoğlu & Greene, 2015). Strong emotions can be invoked when learners interact with multiple conflicting sources, particularly those sources that contradict their own deeply held beliefs (e.g., Muis, Pekrun, Azevedo, Sinatra, Trevors, Meier, & Heddy, 2015). Models of multiple source integration would benefit from the incorporation of emotional responses, particularly in terms of how they affect the likelihood of continued progress through the steps outlined in the RESOLV model, versus foreclosure on the task. Self-Regulation of Academic and Social Goals Emotions and emotion regulation are key aspects of the broader psychological literature on self-regulation, of which SRL research is a part (Greene, 2018). SRL models focus upon learners’ pursuit of academic goals, but social and well-being goals are often equally if not more salient, and sometimes result in learners regulating in ways

Self-Regulated Learning Processes  •  333

that decrease their likelihood of succeeding academically in favor of maintaining their personal identity or interpersonal relations (e.g., doing poorly on a test to avoid being teased; Boekaerts & Pekrun, 2015; Perry, Hutchinson, Yee & Määttä, 2018). Models of SRL or multiple source use that focus only upon academic goals will likely be insufficient predictors of actual behavior and performance in authentic settings, due to the saliency of personal and interpersonal goals in those settings. The RESOLV model’s incorporation of values and interest represents an important step toward recognizing the role of individual differences in investment in tasks, goals, and the effort necessary to achieve them. Future research should investigate how individuals negotiate among academic, well-being, and social goals when using multiple sources. Likewise, challenges during multiple source use, such as when learners encounter contradictory views from family members and valued friends in their social media feed, can lead to strong emotional responses that can be taxing. Depletion models of self-regulation (Baumeister, Vohs, & Tice, 2007) suggest that people have a fixed amount of resources for handling such challenges. Frequent need to self-regulate in such circumstances can “deplete” those resources, decreasing the likelihood of successful self-regulation in similar future events. Such depletion aligns well with Rouet and colleagues’ (2017) assumptions regarding the limited nature of working memory resources, and the benefit–cost analysis that readers must conduct when faced with challenges during learning. The RESOLV could be profitably expanded by examining the role of self-regulatory depletion in the likelihood of continued engagement and learning when confronting challenging sources. Within-Subjects Research on Domain and Task-Specificity In both the fields of SRL and multiple source use, there is increasing interest in the ways that learner functioning can differ by content domain, task, and context (e.g., Bråten et al., 2008; Greene et al., 2015). Initial evidence from SRL research suggests that the predictive validity of learning strategies can differ by domain, but there may be more consistency in terms of learners’ use of metacognitive processing. Researchers risk overloading participants by asking them to work with multiple sources across multiple tasks, but within-subjects studies are the key to understanding the ways and degrees to which learners enact SRL during multiple source use differently across contexts. Such studies would begin to disentangle what aspects of SRL are person-specific versus those that are domain or task-specific (Greene et  al., 2015). These findings might also point to which aspects of SRL instruction are most likely to transfer across contexts (Sitzmann & Ely, 2011). Interventions to Foster Self-Regulated Learning with Multiple Sources The large pool of SRL intervention studies have allowed for meta-analytic investigations of the efficacious characteristics of those interventions (e.g., Dent & Koenka, 2016), but the pool of SRL intervention studies involving multiple sources is much smaller. Researchers and educators should collaborate, using design-based research methods, to develop and explore such interventions, iteratively building on what works in context and then testing the active ingredients of those interventions for generalization. Such work might also advance work in SRL in general, given that many

334  •  Greene et al.

SRL studies have involved limited numbers and kinds of sources (e.g., Minguela et al., 2015). Of course, such interventions must truly scaffold SRL, meaning they must involve teaching and modeling SRL, providing opportunities for students to practice and receive feedback, as well as then fading support as students show increased capacity for true self-regulation (van de Pol, Volman, & Beishuzien, 2010; Zimmerman, 2013). For a multiple source use SRL intervention to be considered effective, learners must be able to enact SRL after the intervention has faded (e.g., Raes et  al., 2012). Research on such adaptive, fading SRL interventions is ongoing (e.g., Azevedo, Johnson, & Burkett, 2015), but will be critical to helping learners acquire and apply SRL in and out of school and heavily supported contexts. Transfer to Out-of-School Contexts Finally, arguments for the situativity of education research findings (cf. Turner & Nolen, 2015) and the difficulties of fostering transfer from the classroom to the out-ofschool context present important challenges for how to help learners effectively enact SRL when working with multiple sources (Järvenoja, Järvelä, & Malmberg, 2015). Given the societal issues mentioned at the beginning of this chapter, researchers and educators must focus on developing SRL and multiple source use interventions that influence learners’ actions and understanding both inside and outside of school. The key to such work may be the degree to which those interventions can mirror authentic multiple source use contexts, and invoke the cognitive, motivational, and emotional responses typical of such contexts, so students have the best chance to connect what is learned in the classroom to what they do in the rest of their lives (Immordino-Yang & Damasio, 2007).

CONCLUSION In sum, the clear connections between models of SRL and multiple source use reveal many useful coherences, with supportive research findings, as well as several promising directions for future research in both fields. There are clear opportunities for further work on what happens before and after multiple source use, as well as the role of calibration and self-regulation during learning. Likewise, intervention research in both fields suggests some common best practices, including direct instruction of knowledge and skills followed by guided practice, support, feedback, and fading. Finally, iterative, moment-to-moment interactions between cognition, metacognition, motivation, and affect can have dramatic effects on how learners use multiple sources, inside and outside of the classroom, and there is much more to understand about those contingent relations, and how best to shape them to prepare learners for the challenges of the information-rich world of the 21st century.

REFERENCES Alexander, P. A., & the Disciplined Reading and Learning Research Laboratory (2012). Reading into the future: Competence for the 21st century. Educational Psychologist, 47(4), 259–280. Anmarkrud, O., McCrudden, M., Bråten, I., & Strømsø, H (2013). Task-oriented reading of multiple documents: online comprehension processes and offline products. Instructional Science, 41, 873–894.

Self-Regulated Learning Processes  •  335 Azevedo, R. (2005). Using hypermedia as a metacognitive tool for enhancing student learning? The role of selfregulated learning. Educational Psychologist, 40(4), 199–209. Azevedo, R., Johnson, A. M., & Burkett, C. (2015). Does training of cognitive and metacognitive regulatory processes enhance learning and deployment of processes with hypermedia? Proceedings of the 37th Annual Meeting of the Cognitive Science Society (pp. 136–141). Austin, TX: Cognitive Science Society. Azevedo, R., Johnson, A., Chauncey, A., & Graesser, A. (2011). Use of hypermedia to assess and convey selfregulated learning. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance (pp. 102–121). New York: Taylor & Francis Group. Baumeister, R. F., Vohs, K. D., & Tice, D. M. (2007). The strength model of self-control. Current Directions in Psychological Science, 16(6), 351–355. Ben-Eliyahu, A., & Bernacki, M. L. (2015). Addressing complexities in self-regulated learning: a focus on contextual factors, contingencies, and dynamic relations. Metacognition and Learning, 10(1), 1–13. Bennett, S., Maton, K., & Kervin, L. (2008). The ‘digital natives’ debate: A critical review of the evidence. British Journal of Educational Technology, 39(5), 775–786. Binbasaran Tüysüzoğlu, B., & Greene, J. A. (2015). An investigation of the role of contingent metacognitive behavior in self-regulated learning. Metacognition & Learning, 10, 77–98. Boekaerts, M., & Pekrun, R. (2015). Emotions and emotion regulation in academic settings. In L. Corno & E. M. Anderman (Eds.), Handbook of educational psychology (pp. 76–90). New York: Routledge. Bowler, L. (2010a). Talk as a metacognitive strategy during the information search process of adolescents. Information Research, 15(4), 1–17. Bowler, L. (2010b). The self-regulation of curiosity and interest during the information search process of adolescents. Journal of the American Society for Information Science and Technology, 61(7), 1332–1344. Braasch, J. L., Bråten, I., Strømsø, H. I., Anmarkrud, Ø., & Ferguson, L. E. (2013). Promoting secondary school students’ evaluation of source features of multiple documents. Contemporary Educational Psychology, 38(3), 180–195. Braasch, J. L., Lawless, K. A., Goldman, S. R., Manning, F. H., Gomez, K. W., & Macleod, S. M. (2009). Evaluating search results: An empirical analysis of middle school students’ use of source attributes to select useful sources. Journal of Educational Computing Research, 41(1), 63–82. Brand-Gruwel, S., & Stadtler, M. (2011). Solving information-based problems: Evaluating sources and information. Learning and Instruction, 21, 175–179. Bråten, I., Anmarkrud, O., Brandmo, C., & Strømsø, H. (2014). Developing and testing a model of direct and indirect relationships between individual differences, processing and multiple text comprehension. Learning and Instruction, 30, 9–24. Bråten, I., Britt, M. A., Strømsø, H., & Rouet, J.-F. (2011). The role of epistemic beliefs in the comprehension of multiple expository texts: Toward an integrated model. Educational Psychologist, 46(1), 48–70. Bråten, I., Ferguson, L., Anmarkrud, O., & Strømsø, H. (2013). Prediction of learning and comprehension when adolescents read multiple texts: The roles of word-level processing, strategic approach, and reading motiva­ tion. Reading and Writing: An Interdisciplinary Journal, 26, 321–348. Bråten, I., & Strømsø, H. (2003). A longitudinal think-aloud study of spontaneous strategic processing during the reading of multiple expository texts. Reading and Writing: An Interdisciplinary Journal, 16, 195–218. Bråten, I., Strømsø, H., & Samuelstuen, M. (2008). Are sophisticated students always better? The role of topic-specific personal epistemology in the understanding of multiple expository texts. Contemporary Educational Psychology, 33, 814–840. Britt, M. A., Richter, T., & Rouet, J.-F. (2014). Scientific literacy: The role of goal-directed reading and evaluation in understanding scientific information. Educational Psychologist, 49(2), 104–122. Bromme, R., & Goldman, S. R. (2014). The public’s bounded understanding of science. Educational Psychologist, 49(2), 59–69. Cho, B-Y. (2014). Competent adolescent readers’ use of Internet reading strategies: A think-aloud study. Cognition and Instruction, 32(3), 253–289. Cisco (2016). Cisco global cloud index: Forecast and methodology: 2015–2020. Retrieved from: www. cisco.com/c/dam/en/us/solutions/collateral/service-provider/global-cloud-index-gci/white-paperc11-738085.pdf. Coiro, J., & Dobler, E. (2007). Exploring the online reading comprehension strategies used by sixth-grade skilled readers to search for and locate information on the Internet. Reading and Research Quarterly, 42(2), 214–257.

336  •  Greene et al. de Boer, H., Donker, A. S., & van der Werf, M. P. C. (2014). Effects of the attributes of educational interventions on students’ academic performance: A meta-analysis. Review of Educational Research, 84(4), 509–545. Dent, A. L., & Koenka, A. C. (2016). The relation between self-regulated learning and academic achievement across childhood and adolescence: A meta-analysis. Educational Psychology Review, 28(3), 1–50. de Vries, B., van der Meij, H., & Lazonder, A.W. (2008) Supporting reflective web searching in elementary schools. Computers in Human Behavior, 24, 649–665. DiFrancesca, D., Nietfeld, J. L., & Cao, L. (2016). A comparison of high and low achieving students on selfregulated learning variables. Learning and Individual Differences, 45, 228–236. Dignath, C., & Büttner, G. (2008). Components of fostering self-regulated learning amongst students: A metaanalysis on intervention studies at primary and secondary school level. Metacognition and Learning, 3, 231–264. Dunlosky, J., & Thiede, K. W. (2013). Four cornerstones of calibration research: Why understanding students’ judgments can improve their achievement. Learning and Instruction, 24, 58–61. Efklides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model. Educational Psychologist, 46(1), 6–25. Gerjets, P., Kammerer, Y., & Werner, B. (2011). Measuring spontaneous and instructed evaluation processing during Web search: Integrating concurrent thinking-aloud protocols and eye-tracking data. Learning and Instruction, 21(2), 220–231. Goldman, S. R. (2011). Choosing and using multiple information sources: Some new findings and emergent issues. Learning and Instruction, 21(2), 238–242. Goldman, S. R., Braasch, J. L., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47(4), 356–381. Goldman, S. R., Britt, M. A., Brown, W., Cribb, G., George, M., Greenleaf, C., Lee, C. D., & Shanahan, C. (2016). Disciplinary literacies and learning to read for understanding: A conceptual framework for disciplinary literacy. Educational Psychologist, 51(2), 219–246. Greene, J. A. (2018). Self-regulation in education. New York: Routledge. Greene, J. A., Bolick, C. M., Jackson, W. P., Caprino, A. M., Oswald, C., & McVea, M. (2015). Domain-specificity of self-regulated learning processing in science and history digital libraries. Contemporary Educational Psychology, 42, 111–128. Greene, J. A., Costa, L. C., Robertson, J., Pan, Y., & Deekens, V. (2010). Exploring relations among college students’ prior knowledge, implicit theories of intelligence, and self-regulated learning in a hypermedia environment. Computers & Education, 55, 1027–1043. Greene, J. A., Hutchison, L. A., Costa, L.-J., & Crompton, H. (2012). Investigating how college students’ task definitions and plans relate to self-regulated learning processing and understanding of a complex science topic. Contemporary Educational Psychology, 37, 307–320. Greene, J. A., Sandoval, W. A., & Bråten, I. (2016). An introduction to epistemic cognition. In J. A. Greene, W. A. Sandoval, & I. Bråten (Eds.), Handbook of epistemic cognition (pp. 1–15). New York: Routledge. Greene, J. A., Yu, S., & Copeland, D. Z. (2014). Measuring critical components of digital literacy and their rela­ tionships with learning. Computers & Education, 76, 55–69. Hagen, A. M., Braasch, J. L., & Bråten, I. (2014). Relationships between spontaneous not-taking, self-reported strategies and comprehension when reading multiple texts in different task conditions. Journal of Research in Reading, S1, S147–S157. Harris, K. R., Alexander, P., & Graham, S. (2008). Michael Pressley’s contributions to the history and future of strategies research. Educational Psychologist, 43(2), 86–96. Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111–127. Immordino-Yang, M. H., & Damasio, A. (2007). We feel, therefore we learn: the relevance of affective and social neuroscience to education. Mind, Brain, and Education, 1, 3–10. Järvenoja, H., Järvelä, S., & Malmberg, J. (2015). Understanding the process of motivational, emotional and cognitive regulation in learning situations. Educational Psychologist, 50(3), 204–219. Kintsch, W. (1988). The role of knowledge in discourse comprehension: A construction-integration model. Psychological Review, 95, 163–182. Kuiper, E., Volman, M., & Terwel, J. (2008). Integrating critical web skills and content knowledge: Development and evaluation of a 5th grade educational program. Computers in Human Behavior, 24, 666–692.

Self-Regulated Learning Processes  •  337 Lazonder, A. W., & Rouet, J.-F. (2008). Information problem solving instruction: Some cognitive and metacognitive issues. Computers in Human Behavior, 24, 753–765. Leu, D. J., Forzani, E., Rhoads, C., Maykel, C., Kennedy, C., & Timbrell, N. (2015). The new literacies of online research and comprehension: Rethinking the reading achievement gap. Reading Research Quarterly, 50(1), 37–59. Macedo-Rouet, M., Braasch, J. L., Britt, M. A., & Rouet, J. F. (2013). Teaching fourth and fifth graders to evaluate information sources during text comprehension. Cognition and Instruction, 31(2), 204–226. Maier, J., & Richter, T. (2014). Fostering multiple text comprehension: How metacognitive strategies and motivation moderate the text-belief consistency effect. Metacognition Learning, 9, 51–74. McCrudden, M. T., Stenseth, T., Bråten, I., & Strømsø, H. I. (2016). The effects of topic familiarity, author expertise, and content relevance on Norwegian students’ document selection: A mixed methods study. Journal of Educational Psychology, 108(2), 147–162. Miele, D. B., & Scholer, A. A. (2016). Self-regulation of motivation. In K. Wentzel & D. B. Miele (Eds.), Handbook of motivation at school (2nd Ed.) (pp. 363–384). New York: Routledge. Minguela, M., Solé, I., & Pieschl, S. (2015). Flexible self-regulated reading as a cue for deep comprehension: Evidence from online and offline measures. Reading and Writing, 28(5), 721–744. doi: 10.1007/ s11145-015-9547-2. Moos, D. (2014). Setting the stage for the metacognition during hypermedia learning: What motivation constructs matter? Computers & Education, 70, 128–137. Moos, D. C., & Azevedo, R. (2008). Self-regulated learning with hypermedia: The role of prior domain knowledge, Contemporary Educational Psychology, 33, 270–298. Moos, D., & Marroquin, E. (2010). Multimedia, hypermedia, and hypertext: Motivation considered and reconsidered. Computers in Human Behavior, 26, 265–276. Muis, K. R., Pekrun, R., Azevedo, R., Sinatra, G. M., Trevors, G., Meier, E., & Heddy, B. (2015). The curious case of climate change: Epistemic emotions mediate relations between epistemic beliefs, learning strategies and learning outcomes. Learning and Instruction, 39, 168–183. Naumann, J., Richter, T., Christmann, U., & Groeben, N. (2008). Working memory capacity and reading skill moderate in the effectiveness of strategy training in learning from hypertext. Learning and Individual Differences, 18, 197–213. Perry, N. E., Hutchinson, L., Yee, N., & Määttä, M. (2018). Advances in understanding young children’s selfregulation for learning. In. D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd Ed.) (pp. 457–472). New York: Routledge. Pieschl, S., Stahl, E., & Bromme, R. (2008). Epistemological beliefs and self-regulated learning with hypertext. Metacognition Learning, 3, 17–37. Pieschl, S., Stahl, E., Murray, T., & Bromme, R. (2012). Is adaptation to task complexity really beneficial for performance? Learning and Instruction, 22, 281–289. Pieschl, S., Stahlmann, F., & Bromme, R. (2014). High school students’ adaptation of task definitions, goals and plans to task complexity: The impact of epistemic beliefs. Psychological Topics, 23(1), 31–52. Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16(4), 385–407. Pintrich, P. R., Marx, R., & Boyle, R. (1993). Beyond cold conceptual change: The role of motivational beliefs and classroom contextual factors in the process of conceptual change. Review of Educational Research, 63, 167–199. Raes, A., Schellens, T., De Wever, B., & Vanderhoven, E. (2012). Scaffolding information problem solving in web-based collaborative inquiry learning. Computers & Education, 59, 82–94. Randi, J., Grigorenko, E. L., & Sternberg, R. J. (2005). Revisiting definitions of reading comprehension: Just what is reading comprehension anyway? In S. E. Israel, C. C. Block, K. L. Bauserman, & K. KinnucanWelsch (Eds.), Metacognition in literacy learning: Theory, assessment, instruction, and professional development (pp. 19–39). Mahwah, NJ: Lawrence Erlbaum Associates. Rapp, D. N. (2016). The consequences of reading inaccurate information. Current Directions in Psychological Science, 25(4), 281–285. Rouet, J.-F. (2006). The skills of document use: From text comprehension to web-based learning. New York: Routledge. Rouet, J.-F., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text. Greenwich, CT: Information Age.

338  •  Greene et al. Rouet, J.-F., & Coutelet, B. (2008). The acquisition of document search strategies in grade school students. Applied Cognitive Psychology, 22, 389–406. Rouet, J.-F., & Le Bigot, L. (2007). Effects of academic training on metatextual knowledge and hypertext navigation. Metacognition & Learning, 2, 157–168. Rouet, J.-F., Britt, M. A., & Durik, A. M. (2017). RESOLV: Readers’ representation of reading contexts and tasks. Educational Psychologist, 52, 200–215. Schunk, D. H., & Greene, J. A. (2018). Handbook of self-regulation of learning and performance (2nd Ed.). Routledge: New York. Silverman, C. (November 2016). This analysis shows how fake election news stories outperformed real news on Facebook. BuzzFeedNews. Retrieved from: www.buzzfeed.com/craigsilverman/viral-fake-election-newsoutperformed-real-news-on-facebook?utm_term=.uaDVYBQqy#.koKLK1lBd. Sitzmann, T., & Ely, K. (2011). A meta-analysis of self-regulating in work-related training and educational attainment: What we know and where we need to go. Psychological Bulletin, 137(3), 421–442. Stadtler, M., & Bromme, R. (2008). Effects of the metacognitive computer-tool met.a.ware on the web search of laypersons. Computers in Human Behavior, 24, 716–737. Stanford History Education Group (November 2016). Evaluating information: The cornerstone of civic online reasoning. Robert R. McCormick Foundation. Retrieved from: https://sheg.stanford.edu/upload/ V3LessonPlans/Executive%20Summary%2011.21.16.pdf. Strømsø, H., Bråten, I., Britt, M. A., & Ferguson, L. E. (2013). Spontaneous sourcing among students reading multiple documents. Cognition and Instruction, 3(2), 176–203. Thiede, K. W., & de Bruin, A. (2018). Self-regulated learning in reading. In. D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd Ed.) (pp. 124–137). New York: Routledge. Turner, J. C., & Nolen, S. B. (2015). Introduction: The relevant of the situative perspective in educational psychology. Educational Psychologist, 50(3), 167–172. van de Pol, J., Volman, M., & Beishuzien, J. (2010). Scaffolding in teacher-student interaction: A decade of research. Educational Psychology Review, 22, 271–296. Weiner, B. (2010). The development of an attribution-based theory of motivation: A history of ideas. Educational Psychologist, 45(1), 28–36. Winne, P. H., & Hadwin, A. F. (2008). The weave of motivation and self-regulated learning. In D. Schunk & B. Zimmerman (Eds.), Motivation and self-regulated learning: Theory, research, and applications (pp. 297–314). Mahwah, NJ: Erlbaum. Wolters, C., & Benzon, M. (2013). Assessing and predicting college students’ use of strategies for the selfregulation of motivation. Journal of Experimental Education, 18, 199–221. Zhang, M., & Quintana, C. (2012). Scaffolding strategies for supporting middle school students’ online inquiry processes. Computers & Education, 58, 181–196. Zimmerman, B. J. (2013). From cognitive modeling to self-regulation: A social cognitive career path. Educational Psychologist, 48(3), 135–147. Zimmerman, B. J., & Schunk, D. H. (2011). Self-regulated learning and performance: An introduction and overview. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance (pp. 1–12). New York: Taylor & Francis Group.

Section V

Multiple Source Use Interventions

20

EFFECTS OF INSTRUCTIONAL CONDITIONS ON COMPREHENSION FROM MULTIPLE SOURCES IN HISTORY AND SCIENCE Jennifer Wiley university of illinois at chicago, usa

Allison J. Jaeger temple university, usa

Thomas D. Griffin university of illinois at chicago, usa

The purpose of this chapter is to review theory and research on the effects of instructional manipulations and conditions on comprehension of events or phenomena in science and history from multiple sources. Throughout this chapter the terms text, source, and document are generally used interchangeably to refer to a body of information that is presented in some way that denotes it as an entity distinct from other bodies of information that are presented alongside it. This could be done by having bodies of information on separate physical pages, within separate windows of a browser, accessed via different web links, and so on. Different studies have employed stimuli ranging from websites to excerpts of magazine or journal articles, news briefs, or passages from books, which may or may not explicitly include authorship or other referential information. They can include text, tables, or graphics, exclusively or in combination. Within the domain of history, there is a long tradition of using multiple sources as part of the inquiry process, with the goal for such inquiry being the development of an account of how or why things may have happened in the past (Wiley & Ash, 2005; Wineburg, 1991). This multiple-source inquiry process is the central activity for many professional historians, and primary evidence for various historical accounts of events typically comes from other existing documents. The multiple-source inquiry

341

342  •  Wiley et al.

process is also part of disciplinary instruction in History, most prominently as part of Advanced Placement (AP) courses in which students learn to respond to documentbased questions. In contrast, the central task of professional scientists is answering questions via their own direct observations of events within the context of an experiment. Integrating information from multiple documents is essential to how scientists determine what is already known, what hypotheses are in need of testing, and what methods to use in their own research. However, these earlier stages of the scientific process involving multiple-document inquiry have received far less attention as part of disciplinary instruction in science. Typically, students are merely presented with the final conclusions from prior scientific work, with more focus upon experimental data collection processes. With the emergence of the Internet, the relevance of comprehending from multiple sources has become ever more pertinent across a wide variety of topics and disciplines. Today, the most common method that lay people use to gain an understanding of historical events or scientific phenomena is via Internet searches, displacing the practice of using a bound and printed encyclopedia set or textbook as a reference. Starting with early research on multiple-source comprehension that primarily emerged from work in history, researchers have explored several types of instructional manipulations including altering the features of the inquiry task that is given (such as being asked to write a narrative or an argument); changing features of the task environment (such as the format of the source documents or features of the document set); and varying the instructional context (such as having students engage in a particular activity or training prior to engaging in a multiple-source inquiry task). These three broad categories continue to represent the main types of manipulations that have been studied in the literature. Further, this literature now includes both studies that have explored the comprehension of historical events from multiple-source inquiry tasks, as well as studies that have explored the development of understanding about scientific phenomena. To outline the remainder of this chapter, the next section provides an overview discussing the kinds of processes that are theoretically involved in multiple-source comprehension, and articulates the challenges that readers face when they attempt to engage in multiple-source inquiry tasks. Then, the main section of the chapter provides a summary of empirical research that has attempted to explore multiple-source comprehension processes using manipulations of the inquiry task, the task environment, and the instructional context. Research from studies in both history and science are included in an attempt to outline possible similarities and differences. A final section concludes the chapter with a theoretical synthesis derived from the empirical review, a discussion of future directions for research, and practical implications of this work.

THEORETICAL BACKGROUND Engaging in inquiry using multiple sources is a complex activity because it requires all the processes necessary for comprehending individual informational sources, plus an additional set of processes that become particularly important when readers are confronted with information from more than one source. According to theories of text comprehension, understanding even a single informational text requires the

Effects of Instructional Conditions  •  343

construction of multiple levels of representation (Graesser, Singer, & Trabasso, 1994; Kintsch, 1998). At the most basic level, a reader creates a surface representation, which generally consists of a fleeting episodic trace capturing the exact words and format of the source. At the next level of processing, the reader attempts to develop a textbase representation. This is essentially a propositional representation of the ideas presented in each clause or sentence. Basic word and sentence-level reading processes contribute to the construction of the textbase. In addition, as part of the comprehension of informational text, the reader must attempt to develop yet another level of representation, referred to as the situation model. On this level, the reader attempts to connect ideas between the sentences that appear in the text, and connect ideas in the text with prior knowledge, to develop a coherent understanding of the content that is being described. When readers are confronted with more than a single source from which to obtain information, then the situation becomes more complicated. Comprehending events or phenomena from multiple sources instead of a single source necessitates representation of information about the nature or origin of the various documents (e.g., who wrote it, for what purpose was it written), and relations among the documents (e.g., the presence of corroborating or conflicting statements). In these situations, the term “source” is commonly used in a more restricted manner than the way it was defined at the outset of this chapter. Specifically, the term “source” can also refer to “information about individuals and organizations that create and publish textual content, including information about when, where, and for what purpose the content is created and published” (Bråten & Braasch, this volume). According to the MD-Trace framework (Rouet & Britt, 2011), this information is captured by the intertext model. Further, there needs to be a level of representation that reflects the understanding derived from integrating information across multiple sources. Beyond developing representations of each individual source, the reader needs to develop a documents model (Perfetti, Rouet, & Britt, 1999) or an integrated model to serve as the representation of the readers’ understanding of the situation or phenomenon being described (Britt & Rouet, 2012). In a multiple-source context, it is this level that best represents a reader’s understanding of the content, or the mental model that has been constructed about the phenomenon that is the focus of inquiry (Wiley & Voss, 1999; Wiley et al., 2009). The development of this understanding is also critically influenced by the reader’s interpretation of the task and the processing they should engage in to achieve it (Wiley, Griffin, & Thiede, 2005). As described by Rouet and Britt as part of the MD-Trace framework (2011; Britt & Rouet, 2012), the task model includes what the goals and sub-goals are for reading and writing (e.g., Why are the sources being read? What is the goal for the inquiry task?). In addition to containing the reader’s goals, the task model subsumes an activity model that informs the reader about which specific actions, procedures, or behaviors one might engage in to reach those goals during the inquiry task. The activity model is a representation of how to complete the task (What does completion of the task entail? How should I engage in this task?). The extent to which readers might develop an integrated model or intertext model in multiple-document contexts will be partially determined by the contents of the task model, including the representation of task goals and the activity model. Although different reading, writing, or processing goals may affect learning in single-source

344  •  Wiley et al.

contexts, they become even more critical to consider in multiple-source contexts. With a single source, the construction of a good-enough situation model can often be achieved by representing the original text’s intended purpose, structure, or argument in memory. However, when tasked with comprehending an event or phenomenon from multiple sources, the reader must actively impose selection, organization, and transformation to construct a mental model that integrates the information from different texts, rather than more passively creating distinct traces for each individual text (Wiley & Voss, 1996, 1999; Wiley et  al., 2009). Often, some or even all of the individual documents a learner has available were written for a purpose other than addressing the questions relevant for the learner’s goals. The learner must selectively re-purpose the information in the documents. A reader’s task goals and activity model will determine the extent to which each reader will engage in the process of evaluating the individual texts; selecting, organizing, and transforming relevant information; and re-assembling what is selected into a new coherent model. Thus, the interpretation of the task—and the extent to which readers understand that they need to actively engage in constructive processing, attempt to build connections across ideas, and try to form a coherent, integrated model of the phenomena—are critical determinants of multiple-source comprehension (Britt & Rouet, 2012; Wiley & Voss, 1996, 1999). In general, individuals face several challenges when engaging in multiple-source inquiry activities. The primary dependent variables that have been used to assess effective multiple-source comprehension include analyses of the quality of the written responses that are generated as part of the inquiry activity, as well as performance on tests of comprehension of the content following an inquiry activity. In addition, measures of sourcing (i.e., consideration of information about when, where, and for what purpose a source was created), source evaluation (i.e., the ability to discriminate reliable from unreliable sources), or corroboration behaviors are sometimes examined. Research suggests that students are generally unlikely to spontaneously attend to the source of information and note it in their written responses. Students also typically fail to engage in critical evaluation or corroboration of sources. Further, many students tend to engage in superficial representation of the information they read, rather than actively selecting and transforming information to construct a coherent, integrated mental model of the phenomenon. This affects both the quality of their essays and their performance on comprehension tests. These difficulties are the primary issues that instructional manipulations have been designed to address.

SUMMARY OF EMPIRICAL RESEARCH The main areas of research in this literature have focused on how features of the inquiry task, the task environment, and the instructional context provided prior to engaging in multiple-source inquiry tasks can affect performance. Inquiry Task Manipulations Studies that fall under this heading feature manipulations in the reading and/or writing tasks that are assigned to students as the main task for the multiple-document inquiry activity. Previous work has shown that the writing prompt can alter performance during multiple-document inquiry tasks.

Effects of Instructional Conditions  •  345

For example, Wiley and Voss (1996, 1999) found that prompting undergraduates to write arguments from information presented in multiple sources about the causes of the Irish Potato Famine (a period of mass starvation, disease, and emigration in Ireland between 1845 and 1852) led to more connected essays, and better comprehension of the material, than when students were prompted to write narratives from the same information presented in a single source. Students prompted to write arguments also wrote essays that contained a higher proportion of transformed sentences—that is, sentences that combined information from the texts in a new way—rather than borrowing sentences directly from the texts, or adding sentences that contained no information from the texts. Wiley (2001) extended these findings, showing that greater benefits were seen when students composed an argument from multiple sources using a two-window browser with additional instructions about why they were being given two windows, versus when they composed a narrative from sources presented in a single-window browser. Wiley (2001) and Hemmerich and Wiley (2002) also extended these findings to a document set on a scientific topic and found benefits for writing an argument about the causes of the eruption of Mt. St. Helens in the state of Washington in the United States in 1980 versus writing a narrative, while Wiley et al. (2009) found benefits for writing arguments about the causes of the eruption over writing a description. Le Bigot and Rouet (2007) demonstrated better essay quality from argument than summary writing tasks for a document set on the topic of social influence, and Naumann, Wechsung, and Krems (2009) found better-quality essays about the Panamanian Revolution (Panama’s separation from Colombia in 1903) in a condition which they referred to as “argumentative,” where participants were instructed to “form an opinion about a controversy on the Panama topic,” versus a condition they referred to as “narrative,” where participants were instructed to “write a description of the events.” Stadtler, Scharrer, Skodzik, and Bromme (2014) found that an argumentation prompt led to more balanced coverage of a medical controversy than a summarization prompt. Their argumentation prompt was: Read the texts attentively to write an argument afterward. The first part of an argument provides a comparative overview of the standpoints and supporting arguments brought forward by the different authors. The second part consists of your own personal view regarding the authors’ positions, which should stand on justified grounds. Their summary prompt was: “Read the texts attentively to write a summary afterward. A summary is a clearly laid-out overview of the essential contents. Hence, it should inform about what the nine texts are about.” Maier and Richter (2016) also found that argumentation prompts led to more balanced reading of sources on either side of a controversy (“build a justified point of view whether or not electromagnetic radiation from cell phones is hazardous”) than summarization prompts (“memorize as many facts as possible”). Their argument instruction also included the request to critically evaluate text information to judge the plausibility of arguments. On the other hand, writing prompts that encourage students to discuss their own opinions in an essay can often lead to poorer comprehension of the materials and lower-quality essays. Monte-Sano and De La Paz (2012) had students engage in an inquiry task about the Cold War and found that a situated prompt to write a letter arguing about what is wrong with Communism led to lower-quality essays that were

346  •  Wiley et al.

less likely to attend to the reconciliation of historical perspectives than prompts that instructed high school students to consider the historical context, compare across documents, or consider the causes of the Cold War. Similarly, Wiley, Steffens, Britt, and Griffin (2014) found both college and high school students wrote better essays about the Panamanian Revolution when prompted to “write an argument about the factors that contributed to the Revolution” than when prompted to “write an argument about the extent to which Teddy Roosevelt and his administration were responsible for the Revolution.” Note that while these prompts did not specifically ask students to report their opinions, they may have steered students toward taking a stance on a question of subjective moral evaluation (e.g., “what is wrong with,” who was “responsible for”) rather than a question of fact. Although these studies did not explicitly use the term “opinion” in their prompts, the inherently subjective and personal or ideological nature of the questions used in these studies may have led to poorer essays in terms of addressing the factual matters surrounding the topic. Stahl, Hynd, Britton, McNish, and Bosquet (1996) explicitly manipulated whether AP high school students received opinion-based reading and writing prompts during an inquiry task about the Vietnam War and the Gulf of Tonkin incident that took place in the waters off the coast of Vietnam in 1964, drawing the United States more directly into the war. They found that students failed to integrate multiple documents or engage in sourcing or corroboration behaviors, regardless of whether they were asked to “write a description” or “write about your opinions regarding” the Gulf of Tonkin incident and resolution. However, students asked to “describe” stuck very close to the information in the texts, and primarily relied on a single text. Students who were asked to write their opinion tended to move away from the original texts, including more generalizations not tied to any single text and more subjective evaluative statements in their writing, and they did so without providing grounding from the texts. In this case, the instruction to “write about your opinions regarding” political actions and policies of governments seemed to steer learners toward their own subjective moral judgments about who was “wrong” or “at fault.” In some cases, argument-based prompts can lead to poorer outcomes than prompts that encourage students to summarize the information provided in a set of documents. This may be especially problematic when students are fairly naïve to argumentation. When students are given instruction in the skills of argument writing or reasoning from evidence, then clear benefits are seen from argument writing tasks, even among younger students (De La Paz, 2005; Klein & Rose, 2010; Klein & Samuels, 2010; Reisman, 2012). Alternatively, sometimes no differences in learning outcomes have been found due to manipulations in writing prompts with younger students (Strømsø, Bråten, & Britt, 2010), while other times interactions have been seen between task manipulations and individual differences among students in thinking dispositions, reading skills, or prior knowledge. Griffin, Wiley, Britt, and Salas (2012) found that both individual differences in thinking dispositions and reading skill contributed to middle school students’ ability to learn when prompted to write an essay “explaining how and why recent patterns of global temperature are different from what has been observed in the past.” In this study, the disposition that was measured was CLEAR thinking (Commitment to Logic, Evidence, and Reasoning) which assessed the extent to which students place value and importance

Effects of Instructional Conditions  •  347

on reasoning about evidence when forming and revising beliefs. Climate change, at least in the United States, is surrounded by political controversy and thus subject to strong personal and emotional biases. A dispositional commitment to evidence-based reasoning could help ensure that a student adopts the correct task model of understanding the science surrounding climate change, rather than one of trying to selectively focus on information one can use to defend political opinions on the issue. Gil, Bråten, Vidal-Abarca, and Strømsø (2010a) found that only undergraduates with high prior knowledge about the topic experienced better learning outcomes (as measured by a post-inquiry inference verification test) from an argument writing prompt that included writing about their opinion (“Imagine that you have to write a brief report to other students where you express and justify your personal opinion about how climate changes may influence life on Earth and what are the causes of climate changes”) than a summarization prompt (“Imagine that you have to write a brief report to other students that summarizes how climate changes may influence life on Earth and what are the causes of climate changes”). Otherwise, students who were assigned to write summaries wrote more transformed and integrated essays, and performed better on the inference verification test. Similarly, Bråten and Strømsø (2009) found that only undergraduates with more sophisticated topic-specific epistemologies (readers who considered knowledge about climate change to be tentative) experienced better learning outcomes from an argument writing task. Gil, Bråten, Vidal-Abarca, and Strømsø (2010b) found that epistemologically naïve undergraduates, again using a climate-change-specific epistemology scale, had worse learning outcomes from the opinion writing prompt than the summarization prompt. As in Gil et al. (2010a), students’ essays were more transformed and integrated in the summarization condition than the opinion-based argument condition. Other manipulations have varied whether learners received any intertextual inquiry prompt at all, in comparison to more text-specific prompts. Britt and Sommer (2004) had undergraduates read a document set on the Panamanian Revolution and manipulated whether they responded to “macro-questions” which required attention to relationships across texts to explain what happened and why, or “micro-questions” which required readers to pay attention to specific details within individual texts. Participants who received the macro-questions outperformed those responding to micro-questions when asked to construct a timeline from memory, which the authors employed as a measure of the ability to integrate event information across multiple documents. Cerdán and Vidal-Abarca (2008) gave undergraduates a document set related to bacteria resistance and found that engaging in an essay task which prompted readers to make intertext connections led to better understanding of the content than short essay questions that prompted readers to consider each text individually. Similarly, Wiley et al. (2014) found that middle school students who completed an intertext timeline activity that required integration of event information from across multiple documents before writing an essay explaining the factors that caused the Panamanian Revolution tended to perform better on an inference verification task than students who did not engage in the timeline activity first. Taken together, this body of research has demonstrated that, across both science and history domains, tasks which prompt students to generate an argument or consider intertextual connections can foster deeper comprehension from multiple documents

348  •  Wiley et al.

than tasks that prompt students to simply describe or summarize what they have read. More specifically, argumentation tasks have led to students including more causes and connections in their essays, more coverage of information across documents in their essays, and more balanced online reading behaviors. Similarly, activities such as engaging in macro-level or intertext tasks while reading can also benefit multiple-document comprehension. On the other hand, this research also suggests that argument tasks can be less effective than more descriptive tasks when writing prompts encourage students to discuss their own opinions; or when students are young, have little prior knowledge about the topic, have weak reading skills, or are epistemologically naïve. Manipulations of Features of the Task Environment Studies that fall under this heading include manipulations that alter the context in which the task is completed including whether the document set contains primary documents, policy arguments, or conflicting information; whether the information is presented in a single chapter versus in separate documents; and manipulations in presentation format such as whether documents are viewed in a multiple-window environment, or in a fixed order. Some of the earliest work on multiple-source comprehension investigated whether presenting sources as a set of multiple documents, as compared to a single document, would improve comprehension (Wiley & Voss, 1996, 1999). While it might be assumed that processing information from multiple sources would be more demanding, Wiley and Voss found that undergraduates who received information about the Irish Potato Famine in a multiple-source format exhibited better understanding of the relations between concepts on verification tasks, and wrote more connected essays, as compared to readers who received the same information as a single textbook-like source. They also wrote essays that contained a higher proportion of transformed sentences, and from this evidence the authors argued that the multiple-source format prompted individuals to engage in more constructive processing while writing. In contrast, while writing within a single-source format, students tended to rely on the loose and implied temporal connections that already existed in the text and “borrowed” or copied more sentences directly from the texts into their essays. Wiley (2001) extended this work to online sources and found that writing arguments from multiple sources presented within two side-by-side windows encouraged more comparison, corroboration, and integration of ideas across texts versus a single-window browser. In the context of studies testing the effectiveness of the Sourcer’s Apprentice tool, Britt and Aglinskas (2002) also compared single and multiple-source presentation. The Sourcer’s Apprentice environment provides students with a tutorial in sourcing, corroboration, and contextualization (based on the heuristics described by Wineburg, 1991). It also prompts students to complete notecards for each of the sources in a multiple-document set. In the final experiment in this paper, high school students learned about the Homestead Steel Strike, a serious labor dispute that took place in 1892 near Pittsburgh, Pennsylvania in the United States. Documents were presented either within the Sourcer’s Apprentice environment, or participants read the information presented as a single textbook-like document. They found that essays in the Sourcer’s Apprentice/Multiple Documents condition were more integrated and were also perceived as being of higher quality when graded by history teachers.

Effects of Instructional Conditions  •  349

Nokes, Dole, and Hacker (2007) also compared the use of multiple sources to traditional textbooks in the context of a 15-day high school unit on the major events in United States History during the 1920s and 1930s, which covered topics such as the Great Depression, foreign affairs at the time, and the African American movement known as the Harlem Renaissance. In a fully crossed design, students also either received instruction designed to help them learn historical content (content instruction) or instruction designed to help them develop skills in sourcing, corroboration, and contextualization (heuristics instruction). They found that students who read multiple sources performed better on a test of history content than students who read the information in traditional history textbook format, regardless of the type of instruction they received. In addition, all students completed a pre-post multipledocument essay task (on either the Battle of Lexington, which was the first military engagement of the United States Revolutionary War in 1775, or the Pullman Strike, which was a national railroad strike in the United States that began in Chicago, Illinois in 1894). Performance on this task revealed that source evaluation and corroboration skills improved most with multiple-source heuristics instruction. In a more recent study, Stadtler, Scharrer, Brummernhenrich, and Bromme (2013) examined the impact of presentation format on the integration of conflicting scientific information about health risks from high cholesterol levels. Undergraduates were presented with either four separate websites by four authors containing two pairs of opposing views, or were presented with a single website stating only one name as the author. Students in the multiple-source condition correctly verified more conflicting points of information than students in the single-source condition. Students in the multiple-source condition also noted more points of conflict in their essays than students in the single-source condition. In another format manipulation, Naumann et al. (2009) varied whether documents about the Panamanian Revolution had to be read in a fixed order versus the reader being able to select when to read each document. Undergraduates given argumentative writing tasks profited from being able to choose the order of the documents they wished to read, while students given narrative writing tasks benefited from static presentation formats. The choice of which documents are given to students also matters in several ways. Providing students with a pre-determined document set rather than having them engage in their own Internet search has been shown to increase the amount of time spent on “meaning-making” activities (Cho, 2014). Also, the Internet is rife with unreliable sources on just about every topic. Wiley, Ash, Sanchez, and Jaeger (2011) contrasted a set of pre-selected documents about the causes of volcanic eruptions that contained unreliable sources with a set containing only reliable sources. The latter led to more accurate essays, while the former led to essays that included more misconceptions. The types of documents in a set can also impact sourcing behaviors. Rouet, Britt, Mason, and Perfetti (1996) manipulated whether a document set about the Panamanian Revolution included primary source documents. Their results revealed that undergraduates who received primary sources were more likely to cite sources in their essays. Paxton (2002) found that reading an introductory text with a visible author versus an anonymous author led to more sourcing behavior in the essays of AP high school students who wrote about the murder of Julius Caesar from a set of sources.

350  •  Wiley et al.

Document sets can also vary in whether they present opposing viewpoints. Braasch, Rouet, Vibert, and Britt (2012) manipulated whether undergraduates were given brief news reports containing discrepant information or consistent information. They found that students who were given discrepant information used more source information in their essays and demonstrated better source memory compared to students who were given all consistent information. Barzilai and Eshet-Alkalai (2015) manipulated whether undergraduates were exposed to converging or conflicting blog posts about desalination. They found an interaction between the consistency manipulation and the epistemological sophistication of the students. Students who endorsed more sophisticated, evaluative epistemologies (as opposed to less sophisticated absolutist or multiplist epistemologies) were able to write better arguments in the conflicting-sources condition. Other recent work has demonstrated that the presence of policy-related documents can also have an impact on multiple-source comprehension. A study by Blaum, Griffin, Wiley, and Britt (2017) manipulated whether middle school students received a policyrelated document as part of a document set. Students in both conditions were instructed to “write an essay explaining how and why recent patterns in global temperature are different from what has been observed in the past.” Results revealed that students who received policy documents (related to proposed changes in energy regulations) produced essays that included fewer core concepts and causal connections that addressed the inquiry question than students who did not receive policy documents. A pilot study with high school students showed the same result. These results suggest that opinion-based or policy-based prompts can be harmful, especially if the topic or the nature of the question steers students toward personal opinions that inherently must go beyond evidence and textual information and encourages reliance upon subjective values or feelings. Overall, this body of literature indicates that the manner in which the tasks or information is presented is important. For instance, this work demonstrates that presenting scientific or historical information in a multiple-document format can benefit learning and comprehension more than presenting the same information in a single document or textbook-like format. Across these studies, when the information was presented in a multiple-document format, students produced essays that were more integrated, included more connections and transformed sentences, demonstrated better source evaluation and corroboration, and addressed more points of conflict. Additionally, some work has demonstrated that the order in which documents are presented can matter: developing an argument seems to benefit from being allowed to access the documents in any order whereas accessing documents in a fixed order may be more beneficial for developing a thorough description. The type of documents included in the set can also matter. For instance, sets including documents that are less reliable or less relevant for understanding the important causal information underlying a phenomenon can harm comprehension and lead to lower-quality essays. The presence of discrepant or inconsistent information can also impact comprehension and essay quality, but this may interact with other variables such as epistemology. Manipulations of Instructional Context Studies that fall under this heading include interventions that occurred over several weeks within classrooms. It also includes studies that have used more targeted

Effects of Instructional Conditions  •  351

instructional manipulations in computer-based tutorials, laboratory experiments, and smaller-scale classroom studies completed within one or two sessions. Several multi-week instructional interventions have been implemented in high school and middle school history classrooms. For example, De La Paz (2005) tested the benefits of an extended unit on historical reasoning strategies. As part of this unit, middle school students engaged in a mock trial, which was intended to help them understand that there can be varying interpretations of an event. After the mock trial, the teacher introduced historical reasoning strategies that included information about how historians use information within a source to determine its usefulness and trustworthiness, information about comparing the details of one source to another to develop corroboration and notice conflicting views, and information about how to plan and compose argumentative essays. Students in control classes did not receive instruction on either historical reasoning or argument writing strategies, but did read the same social studies texts. After the 12-day unit, students in both conditions completed a multiple-document inquiry task on the U.S.–Mexican war of 1846–8 and wrote an opinion essay about whether the United States or Mexico was responsible for instigating the war. Students who received the intervention included more arguments and correct historical information in their essays than students in the control class. A version of this instructional unit was also adapted for use with older students (De La Paz & Felton, 2010). Results from the high school study replicated and extended previous work, showing that students in the intervention condition wrote essays with more elaborated claims and rebuttals, as well as more document citations. De La Paz and colleagues (2017) further explored the effects of a year-long intervention in middle school classrooms and again demonstrated improvement in the argumentation in student essays compared to students who did not receive the intervention. As mentioned previously, Nokes et al. (2007) conducted a study on another intervention, involving a 15-day history unit on the events and trends of the United States in the 1920s and 30s with high school students. Half of the classrooms received instruction that was focused on content learning while half received instruction focused on heuristics (corroboration, contextualization, and sourcing). Students who received heuristics instruction were more likely to engage in source evaluation and corroboration behaviors on the post-unit multiple-document task than students who received content instruction, especially if they had received the instruction with multiple sources rather than textbooks. Reisman (2012) studied the effects of a longer-term intervention using the Reading-Like-a-Historian (RLH) curriculum. The RLH curriculum focuses on training students how to use document sets to engage in historical inquiry. Students in the RLH condition received between 36 and 50 Document-Based Lessons over the course of six months of instruction in 11th-grade history, while the control condition received typical textbook instruction. Students receiving the RLH curriculum outperformed students in the traditional textbook condition in both sourcing and factual knowledge gains. Follow-up analyses further revealed that the RLH intervention was especially beneficial for struggling readers. Klein and Rose (2010) and Klein and Samuels (2010) studied a year-long intervention for middle school students in science. Half of the students were taught to write arguments or explanations as part of content-area instruction. At the end of the year,

352  •  Wiley et al.

all students completed a multiple-document inquiry task on plate tectonics, nutrition, or the circulatory system. Students who received argument or explanation instruction throughout the school year learned more from the inquiry tasks than did students in comparison classes. Follow-up analyses suggested that the instructional manipulation most strongly affected argument genre knowledge, which in turn accounted for variance in science learning. While these multi-week classroom interventions have been shown to be effective in improving sourcing and content learning, their broad scope, variability in implementation, and inclusion of many different manipulations simultaneously make it difficult to make attributions about the cause of any improvements. The effects of shorter-term, more targeted instructional manipulations have also been investigated, primarily in the context of computer-based tutorials completed within one or two sessions. As previously mentioned, the Sourcer’s Apprentice environment was designed to support behaviors of sourcing, contextualization, and corroboration by training students on each of these heuristics through short tutorials, and then supporting their application during an inquiry task by requiring students to complete notecards. Britt and Aglinskas (2002) found high school students who used the tool as part of instruction outperformed students who received traditional instruction in sourcing on a transfer task on a new topic. Britt, Wiemer-Hastings, Larson, and Perfetti (2004) found that providing tailored feedback on sourcing (or the lack thereof) in essays written using the Sourcer’s Apprentice environment can further increase sourcing behavior in a sample of undergraduates and reduce unsourced usage of original texts. Building on the work with Sourcer’s Apprentice, Wiley et al. (2009) developed a similar unit (SEEK) which included four key behaviors that were found to be important for successful engagement in multiple-document inquiry tasks in science. These included attending to the source of the information, evaluating the evidence that was presented, developing an explanation for the phenomena, and integrating new information with prior knowledge. In the SEEK training condition, undergraduates completed a practice inquiry task about low-carbohydrate diets. They were given information about which aspects of sources to consider when attempting to construct an explanation as part of an inquiry task, prompted to evaluate the sources, and provided with feedback on their evaluations. They were also presented with the protocol of an example “peer” who modeled these behaviors. The comparison group completed the same inquiry task on low-carbohydrate diets, but were not provided with the SEEK instructional materials. In a second session, both groups completed a second inquiry activity where they were tasked with understanding the causes of the eruption of Mt. St. Helens. Students in the SEEK condition demonstrated better learning from the inquiry activity than students in the comparison condition, included fewer erroneous causes in their essays, and wrote more integrated causal essays. They were also better able to discriminate between reliable and unreliable sources. An alternative version of SEEK training was tested by Graesser et al. (2007), but it was not as effective. The main difference between the two versions was that the ineffective version did not include the example student protocol. Another instructional study provides further support for the importance of peer examples. Instead of including a single peer model, Braasch, Bråten, Strømsø, Anmarkrud, and Ferguson (2013) used a contrasting-cases approach. Specifically,

Effects of Instructional Conditions  •  353

they presented high school students with two examples of peers engaging in evaluating sources in relation to answering the question whether cell phones pose a health risk. The “better” student protocol included more sophisticated strategies such as checking the information about the source including author information, the venue of the information, and the date of publication. The “poorer” student protocol contained less sophisticated strategies such as checking to see if the title of the article contained the key words they were using. In the experimental condition, students were asked to read through the two example student accounts and compare the strategies used. Control classes received no contrasting case examples or instruction on sourcing strategies. In a second session, students in both conditions engaged in a multiple-document inquiry task on a new topic: the causes of El Nino weather patterns. Students given the contrasting-cases instruction included more scientific concepts from more useful sources in their essays, and were also better able to discriminate between more and less reliable documents. The results of this study in combination with the work from Wiley et al. (2009) and Graesser et al. (2007) suggest that providing students with instruction in how to effectively evaluate sources not only improves their ability to detect unreliable information, but can also facilitate the development of mental models of the content. Note that this intervention and the SEEK environment are the two interventions where the content area was science rather than history, and both presented sets of documents that included unreliable sources that presented unscientific information. Interventions in history have not had this explicit feature, but rather tend to use sources that vary more in terms of perspective and involvement with the historical events. In addition to prompting students to evaluate sources, the SEEK training developed by Wiley et al. (2009) prompted readers to engage in explanation as they read. A series of studies with undergraduates learning from a document set about electricity (Linderholm, Kwon, & Therriault, 2014; Linderholm, Therriault, & Kwon, 2014) manipulated the role of explanation more specifically. In Linderholm, Therriault, and Kwon (2014), students in the control condition were told to “read for comprehension” while students in the experimental condition were told, “As you read the following texts, attempt to explain how circuits work to yourself. Imagine, for example, that you might need to explain the concept of how circuits work to a group of students in a science class.” Students who were prompted to engage in explanation while reading did better on a comprehension test. Two follow-up studies by the same authors (Linderholm, Kwon, & Therriault, 2014) tested for the effectiveness of more elaborate instructional conditions. In one condition, participants received a definition for selfexplanation and were urged to use it during reading: Self-explanation is the process of explaining each sentence of a text using previous text information or your background knowledge to better understand the text. That is, by self-explaining each sentence, you will rephrase it in a way that makes more sense to you or attempt to understand the text by filling in the “reason” for the statement using previous text information or your own background knowledge. In a second condition, participants received both this definition as well as exposure to the experimenter modeling rephrasing following three sentences from an expository text on another topic (obesity). Again, these conditions were contrasted with a

354  •  Wiley et al.

control condition that was simply told to read for comprehension. Both follow-up studies only found benefits of the simpler instructional condition (providing just the definition) over the control condition on test performance. The modeling condition did not result in performance that significantly differed from the control condition. Moreover, both of these studies were only able to show a benefit of the simpler instruction on factual questions, but not on questions that required integration across texts to answer. Thus, the general finding of these studies was that although the instructions did in some cases help to improve memory for the texts, they did not seem to be effective at supporting deeper comprehension or understanding of the topic from a multiple-document inquiry task. It is possible that the effectiveness of explanation instructions for promoting understanding from multiple documents may have varied as a function of whether readers were allowed to re-read the texts in each study. The follow-up studies (Linderholm, Kwon, & Therriault, 2014) explicitly prevented readers from returning to prior pages in their booklets, whereas this restriction did not appear in the earlier study (Linderholm, Therriault, & Kwon, 2014), nor was it the case in Wiley et  al. (2009) where participants had access to all texts throughout reading and writing. In addition, providing students with a short definition and examples focusing on rephrasing or paraphrasing text may not be enough for them to form an appropriate activity model of what good explanation entails. The results of another recent study suggest that readers may benefit from a different kind of instruction. Jaeger, Griffin, Britt, and Wiley (2015) developed a pre-reading instruction that informed students that good explanations in science involve considering multiple, linked causes. In their study, all middle school students were given a clear task goal, prompted to use the information from the documents to explain how and why recent patterns in global temperature are different from what has been observed in the past, and encouraged to make connections. However, half the students who received the pre-reading instruction demonstrated better coverage of and more connectedness among key causal concepts in their essays, as well as better performance on a comprehension test. In contrast with earlier studies by Linderholm and colleagues, this work shows that instruction in explanation-based processing can improve comprehension in a multiple-document context. These results suggest that even if students understand the general goal of the inquiry task, they may not have a clear understanding of the types of processing that are required to achieve those goals. For example, students might assume that they should seek out the one most probable cause discussed in a particular text rather than find and integrate all the relevant causal factors into a coherent explanation. Importantly, a brief lesson illustrating the multi-causal nature of scientific explanations (using an unrelated topic as the example) yielded notable improvements in learning from multiple documents. Overall, this area of research demonstrates that the instruction students receive before engaging in inquiry tasks is a critical factor to consider for multiple-document inquiry learning activities. In both extended, classroom-based interventions and shorter-term studies, research has shown that instruction which focuses on developing skills in reasoning and evaluation can foster deeper comprehension and lead to better student essays from multiple-document inquiry tasks. Research in this area

Effects of Instructional Conditions  •  355

has also demonstrated that instructional contexts that prompt students to engage in comparison or explanation, particularly contexts that incorporate examples of desired behaviors, can promote more effective learning from multiple-document contexts by supporting the generation of more appropriate activity models.

SUMMARY OF RESEARCH FINDINGS, CURRENT CHALLENGES, AND PRACTICAL IMPLICATIONS Research has sought to identify the conditions and interventions that facilitate learning of both content and inquiry skills within multiple-document environments. Learning can be greater than with traditional single-textbook presentations, but not always. There has been increasing attention paid to individual differences, such as those in prior knowledge and epistemology, and how these interact with varying features of multiple-document inquiry environments. Although most work has been done with undergraduates, the research suggests that less sophisticated readers may need support to learn effectively from multiple-document inquiry tasks. Students’ perceptions of their task are important for how they engage with multiple sources. Their perceptions are shaped by complex interactions between their a priori assumptions and beliefs about comprehension processes (their activity model), their perceptions of their task goals, and the specifics of any particular multiple-source activity that inform their task model (the instructions they are told, the support they are given, the context in which the sources are presented, etc.). Studies that have manipulated the features of the inquiry task that is given have supported the theoretical importance of the task model, and have highlighted that a learner’s task model can be sensitive to subtle cues triggered by a single word in the inquiry prompt. Studies that have manipulated features of the task environment, such as changing the type of documents included in a set (e.g., including a policy-related document), have also highlighted how sensitive the task model is to subtle changes. Manipulations of the task environment have also shown that the sub-goals and activities a learner engages in (e.g., sourcing and corroboration) can be impacted by how the documents are presented. This suggests that learners may often lack a well-developed activity model that would guide them to consciously implement basic sub-goals during inquiry tasks, and instead depend on contextual cues that make different sub-goals salient. Long-term instructional interventions and smaller-scale training studies have demonstrated that it is possible to help students develop more appropriate task goals and activity models that are represented with sufficient enough abstraction to be transferable to new inquiry tasks. On the whole, learning appears to be greatest when students are instructed to construct and justify what they think is the best explanation of a scientific phenomenon or historical event. However, there are caveats to this. If tasked with constructing explanations rather than narratives, then students appear to benefit from presentation formats that allow them some control over how they navigate between the documents. Also, the exact wording of the prompt is critical. Opinion prompts can be harmful, especially if the topic or the nature of the question steers students toward personal opinions that inherently must go beyond evidence and textual information, and encourages students to rely upon their subjective values or feelings. Even prompts

356  •  Wiley et al.

that mention “argument” may lead to similar problems in such contexts. In science, this means being clear about whether the task is one of scientific understanding or policy advocacy. Presenting an opinion or argument about the causes or effects of climate change is fundamentally different from presenting an opinion or argument about what society should do to deal with the issue. Features of the inquiry question or documents could steer students toward a policy-focus at the expense of comprehending scientific phenomena. Policy opinions are a type of attitude preference that have an inherent basis in affective and personal morals that can produce different reactions to texts than knowledge-based beliefs about factual matters (Wolfe & Griffin, 2017). For example, the oft misinterpreted belief-polarization effect actually showed that people’s policy attitudes about laws allowing capital punishment became more extreme in the opposite direction of the attitude-conflicting texts they read. In contrast, their beliefs on the factual matter of crime deterrence became less extreme and changed in the direction of belief-conflicting texts (Lord, Ross, & Lepper, 1979). Thus, inquiry tasks need to take care in considering whether the prompt and the texts relate to preferences or opinions, versus matters of fact. In history, the opportunity for confusion may be even greater, because the phenomenon to be understood entails people acting in ways that impact others, which often triggers moral concerns that blur the line between understanding causes of events and assigning ethical blame for their negative consequences. There is a critical distinction between what the causes of a war were and whether it should have occurred. If students are to be encouraged to “take a side” as part of any inquiry task, it needs to be made clear to them what exactly they are taking a side about. Nevertheless, the “is/ought” or “factual/ethical” distinction might manifest itself differently in science and history. The fact that science-related interventions tend to be more focused on training students to selectively rely on information from trustworthy sources raises another issue. In history, primary sources are critical for developing an understanding of the events related to the phenomena to be understood. However, the nature of primary sources is that they represent viewpoints from individuals who may have played different roles or had access to different information related to the target phenomenon. Primary historical sources from authors with biased interests are often the most important for developing an understanding of the events and the motives of those who caused them. Primary source authors sometimes lay ethical blame on other parties, yet this can reveal those authors’ own motivations that are critical to understanding the causes of their actions that produced some focal event. In contrast, sources in which authors display strong personal, ethical, or ideological biases on scientific questions are often viewed as untrustworthy and as a source of misconceptions that are to be avoided in favor of more objective and reliable sources. For example, authors of original articles reporting the results of scientific research are not expected to have or discuss their personal stake in evidence they are presenting. As such, sourcing may play different roles in science and history, which implies that instruction in sourcing skills may need to be tailored to the different types of sources that are used in different disciplines. One beneficial aspect of multiple-document inquiry tasks is that they provide an opportunity for students to engage in active or constructive comprehension processes. With only a single source, the mental representation of the text can be achieved fairly passively by representing the original text’s intended purpose, structure, or argument

Effects of Instructional Conditions  •  357

in memory. However, when tasked with comprehending an event or phenomenon from multiple documents, the reader must actively impose selection, organization, and transformation to construct a representation that integrates the information from different texts. To answer an inquiry question, the learner must also selectively repurpose the information from the original documents. For many studies, the primary evidence for the reader having engaged in constructive processing during the inquiry activity comes from detailed analysis of the inquiry product. Although trace methods such as think-aloud protocols, navigation logs, or scanpaths are generally thought of as measures that can reflect the online processing of information, a variety of measures of essay quality have been developed with the goal of assessing the extent to which students attempted to integrate and transform information as they wrote. One measure has been the incidence of connections and connectives included in each essay (Britt & Aglinskas, 2002; Voss & Wiley, 1997; Wiley & Voss, 1999). This serves as an index of the extent to which students attempted to connect or integrate ideas that were previously unconnected in the original texts. Students who demonstrate better understanding of the material on comprehension tests tend to write essays that have more connected ideas and more causal connections (Voss & Wiley, 1997, 2000; Wiley, 2001; Wiley & Voss, 1999). Another measure of constructive activity considers the origin of information included in the essay, and the extent to which students plagiarize or copy information from the original texts. In one approach, each sentence is scored as to whether it contains a connection between idea units that were presented separately in the reading materials. The content of each sentence is classified into one of three categories: transformed, added, or borrowed (Wiley & Voss, 1996, 1999). Sentences that combine some presented information with a new claim or fact, or that integrate two bits of presented information that were not previously connected, are classified as transformed. A sentence is coded as added when it contains only novel information. Sentences that are taken directly from, or are paraphrased from, the original material are classified as borrowed. Students who demonstrate better understanding of the material on comprehension tests tend to write essays that contain a lower proportion of borrowed or copied sentences (Voss & Wiley, 1997, 2000; Wiley, 2001; Wiley & Voss, 1999). Similarly, using automated plagiarism detection techniques, Britt et al. (2004) identified the use of unsourced copied material and excessive quoting as two primary deficiencies when students compose essays from multiple sources. Also using another automated approach, the findings of Foltz, Britt, and Perfetti (1996) suggested that similarity to an expert model is more important for essay quality than similarity to the original texts. Finally, Wiley et al. (2017) found a significant negative correlation between plagiarism scores and explanation quality. This suggests that the quality of the inquiry product suffers as students fail to transform information and fail to engage in constructive activity as they write from multiple documents. Constructive activity during inquiry is also reflected by the extent to which the student essay is responsive to the inquiry prompt and re-purposes text from the documents, rather than using segments of text as written for their original purpose. When the documents do not contain verbatim phrases that can be used to directly address the inquiry prompt, students who merely copy text ideas without translation or integration will tend to write essays that focus on the purpose of the original documents rather

358  •  Wiley et al.

than the inquiry prompt. In Wiley et al. (2017), whether student essays addressed the inquiry prompt predicted the quality of essays as well as learning outcomes. In fact, in that study the document set was deliberately designed so that each document was originally written for a different purpose that could not be used to directly address the inquiry prompt. Many studies have designed their document sets and inquiry prompts in such a way, as this is a useful method that helps the researcher to distinguish between essays that merely copy versus those that re-purpose and integrate the information. Regardless of whether online trace measures are collected, or whether measures of constructive processing are derived from an analysis of inquiry products such as essays, a key for future research is that studies endeavor to include both measures of processing and of learning outcomes so that it can be better understood what particular behaviors lead to success, and which activities need to be encouraged or supported to promote effective learning from multiple-document inquiry tasks. An additional important direction for future research is to continue to integrate work using experimental manipulations, processing measures, and assessments of learning outcomes, with the exploration of individual differences. Several findings suggest that individual differences in epistemic dispositions may affect learning from multiple-document inquiry tasks by prompting different students to have different task models (Bråten et  al., 2011; Griffin et  al., 2012). Readers who are disposed to using evidence and reasoning to update their beliefs may adopt a very different task model from those who do not. Similarly, students with more sophisticated epistemic beliefs may view a multiple-document inquiry task as an exercise in corroboration, seeking coherence, and looking for evidence to support claims, whereas students with less sophisticated epistemic beliefs may see the goal of an inquiry task as simply finding the “right” answer verbatim within the documents. A final interesting question for future research is discovering the conditions under which engaging in multiple-document inquiry activities might serve to alter students’ epistemic beliefs and thinking dispositions. Multiple-document inquiry tasks provide an opportunity to help students not only gain a richer understanding of content, but also develop skills that are important for seeking, selecting, evaluating, re-purposing, and integrating information from various sources. Such skills are vital to life-long learning outside of classrooms. However, students need help developing these skills. If simply thrown into a multiple-document environment without proper scaffolds, they may learn even less content than from traditional textbook- and lecture-based instruction. Researchers and educators must carefully consider the nature of the documents provided; the training of behaviors like sourcing, corroborating, constructing arguments, and explaining; and the wording of the inquiry question so that it may prompt constructive behaviors like integrating information in order to develop an argument or explanation, without inviting subjective personal opinions on non-factual issues. Without these considerations, students may fail to reap the benefits of learning from multiple-document inquiry activities.

AUTHOR NOTE The authors were supported by grants from the National Science Foundation (DUE 1535299) and the Institute of Education Sciences (R305A160008) during the preparation of this chapter. All opinions expressed herein are those of the authors and do not necessarily reflect those of the funding agencies.

Effects of Instructional Conditions  •  359

REFERENCES Barzilai, S., & Eshet-Alkalai, Y. (2015). The role of epistemic perspectives in comprehension of multiple author viewpoints. Learning and Instruction, 36, 86–103. Blaum, D., Griffin, T. D., Wiley, J., & Britt, M. A. (2017). Thinking about global warming: Effect of policy-related documents and prompts on learning about causes of climate change. Discourse Processes, 54, 303–316. Braasch, J. L. G., Bråten, I., Strømsø, H. I., Anmarkrud, Ø., & Ferguson, L. E. (2013). Promoting secondary school students’ evaluation of source features of multiple documents. Contemporary Educational Psychology, 38, 180–195. Braasch, J. L. G., Rouet, J. F., Vibert, N., & Britt, M. A. (2012). Readers’ use of source information in text comprehension. Memory & Cognition, 40, 450–465. Bråten, I., & Strømsø, H. I. (2009). Effects of task instruction and personal epistemology on the understanding of multiple texts about climate change. Discourse Processes, 47, 1–31. Bråten, I., Britt, M. A., Strømsø, H. I., & Rouet, J.-F. (2011). The role of epistemic beliefs in the comprehension of multiple expository texts: Towards an integrated model. Educational Psychologist, 46, 48–70. Britt, M. A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20, 485–522. Britt, M. A., & Rouet, J.-F. (2012). Learning with multiple documents: Component skills and their acquisition. In M. J. Lawson & J. R. Kirby (Eds.), The quality of learning: Dispositions, instruction, and mental structures (pp. 276–314). New York, NY: Cambridge University Press. Britt, M. A., & Sommer, J. (2004). Facilitating textual integration with macro-structure focusing tasks. Reading Psychology, 25, 313–339. Britt, M. A., Wiemer-Hastings, P., Larson, A., & Perfetti, C. A. (2004). Automated feedback on source citation in essay writing. International Journal of Artificial Intelligence in Education, 14, 359–374. Cerdán, R., & Vidal-Abarca, E. (2008). The effects of tasks on integrating information from multiple documents. Journal of Educational Psychology, 100, 209–222. Cho, B. Y. (2014). Competent adolescent readers’ use of internet reading strategies: A think-aloud study. Cognition and Instruction, 32, 253–289. De La Paz, S. (2005). Effects of historical reasoning instruction and writing strategy mastery in culturally and academically diverse middle school classrooms. Journal of Educational Psychology, 97, 139–156. De La Paz, S., & Felton, M. K. (2010). Reading and writing from multiple source documents in history: Effects of strategy instruction with low to average high school writers. Contemporary Educational Psychology, 35, 174–192. De La Paz, S., Monte-Sano, C., Felton, M., Croninger, R., Jackson, C., & Piantedosi, K. W. (2017). A historical writing apprenticeship for adolescents: Integrating disciplinary learning with cognitive strategies. Reading Research Quarterly, 52, 31–52. Foltz, P. W., Britt, M. A., & Perfetti, C. A. (1996). Reasoning from multiple texts: An automatic analysis of readers’ situation models. In G. W. Cottrell (Ed.), Proceedings of the 18th annual conference of the Cognitive Science Society (pp. 110–115). Mahwah, NJ: Erlbaum. Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H. I. (2010a). Summary versus argument tasks when working with multiple documents: Which is better for whom? Contemporary Educational Psychology, 35, 157–173. Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H. I. (2010b). Understanding and integrating multiple science texts: Summary tasks are sometimes better than argument tasks. Reading Psychology, 31, 30–68. Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101, 371–395. Graesser, A. C., Wiley, J., Goldman, S. R., O’Reilly, T., Jeon, M., & McDaniel, B. (2007). SEEK Web tutor: Fostering a critical stance while exploring the causes of volcanic eruption. Metacognition and Learning, 2, 89–105. Griffin, T. D., Wiley, J., Britt, M. A., & Salas, C. (2012). The role of CLEAR thinking in learning science from multiple-document inquiry tasks. International Electronic Journal of Elementary Education, 5, 63–78. Hemmerich, J., & Wiley, J. (2002). Do argumentation tasks promote conceptual change about volcanoes? Proceedings of the 24th annual conference of the Cognitive Science Society (pp.  453–458). Hillsdale, NJ: Erlbaum. Jaeger, A. J., Griffin, T. D., Britt, M. A., & Wiley, J. (2015, July). Making connections: Improving student learning about climate change. Paper presented at the 25th Annual Meeting of the Society for Text & Discourse, Minneapolis, MN.

360  •  Wiley et al. Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge: Cambridge University Press. Klein, P. D., & Rose, M. A. (2010). Teaching argument and explanation to prepare junior students for writing to learn. Reading Research Quarterly, 45, 433–461. Klein, P. D., & Samuels, B. (2010). Learning about plate tectonics through argument-writing. Alberta Journal of Educational Research, 56, 196–217. Le Bigot, L., & Rouet, J. F. (2007). The impact of presentation format, task assignment, and prior knowledge on students’ comprehension of multiple online documents. Journal of Literacy Research, 39, 445–470. Linderholm, T., Kwon, H., & Therriault, D. J. (2014). Instructions that enhance multiple-text comprehension for college readers. Journal of College Reading and Learning, 45, 3–19. Linderholm, T., Therriault, D. J., & Kwon, H. (2014). Multiple science text processing: Building comprehension skills for college student readers. Reading Psychology, 35, 332–356. Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37, 2098–2109. Maier, J., & Richter, T. (2016). Effects of text-belief consistency and reading task on the strategic validation of multiple texts. European Journal of Psychology of Education, 31, 479–497. Monte-Sano, C., & De La Paz, S. (2012). Using writing tasks to elicit adolescents’ historical reasoning. Journal of Literacy Research, 44, 273–299. Naumann, A. B., Wechsung, I., & Krems, J. F. (2009). How to support learning from multiple hypertext sources. Behavior Research Methods, 41, 639–646. Nokes, J. D., Dole, J. A., & Hacker, D. J. (2007). Teaching high school students to use heuristics while reading historical texts. Journal of Educational Psychology, 99, 492–504. Paxton, R. J. (2002). The influence of author visibility on high school students solving a historical problem. Cognition and Instruction, 20, 197–248. Perfetti, C. A., Rouet, J.-F., & Britt, M. A. (1999). Towards a theory of documents representation. In H. van Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 99–122). Mahwah, NJ: Erlbaum. Reisman, A. (2012). Reading like a historian: A document-based history curriculum intervention in urban high schools. Cognition and Instruction, 30, 86–112. Rouet, J.-F., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 19–52). Greenwich, CT: Information Age. Rouet, J.-F., Britt, M. A., Mason, R. A., & Perfetti, C. A. (1996). Using multiple sources of evidence to reason about history. Journal of Educational Psychology, 88, 478–493. Stadtler, M., Scharrer, L., Brummernhenrich, B., & Bromme, R. (2013). Dealing with uncertainty: Readers’ memory for and use of conflicting information from science texts as function of presentation format and source expertise. Cognition and Instruction, 31, 130–150. Stadtler, M., Scharrer, L., Skodzik, T., & Bromme, R. (2014). Comprehending multiple documents on scientific controversies: Effects of reading goals and signaling rhetorical relationships. Discourse Processes, 51, 93–116. Stahl, S. A., Hynd, C. R., Britton, B. K., McNish, M. M., & Bosquet, D. (1996). What happens when students read multiple source documents in history? Reading Research Quarterly, 31, 430–456. Strømsø, H. I., Bråten, I., & Britt, M. A. (2010). Reading multiple texts about climate change: The relationship between memory for sources and text comprehension. Learning and Instruction, 20, 192–204. Voss, J. F., & Wiley, J. (1997). Developing understanding while writing essays in history. International Journal of Educational Research, 27, 255–265. Voss, J. F., & Wiley, J. (2000). A case study of developing historical understanding via instruction: The importance of integrating text components and constructing arguments. In P. Stearns, S. Wineburg, & P. Seixas (Eds.), Knowing, teaching and learning history (pp. 375–389). New York, NY: NYU Press. Wiley, J. (2001). Supporting understanding through task and browser design. In Proceedings of the 23rd annual conference of the Cognitive Science Society (pp. 1164–1169). Hillsdale, NJ: Erlbaum. Wiley, J., & Ash, I. (2005). Multimedia learning in history. In R. Mayer (Ed.), The Cambridge handbook of mul­ timedia learning (pp. 375–391). New York, NY: Cambridge University Press. Wiley, J., Ash, I. K., Sanchez, C. A., & Jaeger, A. (2011). Clarifying readers’ goals for learning from expository science texts. In M. McCrudden, J. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 353–374). Greenwich, CT: Information Age.

Effects of Instructional Conditions  •  361 Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009). Source eval­ uation, comprehension, and learning in Internet science inquiry tasks. American Educational Research Journal, 46, 1060–1106. Wiley, J., Griffin, T. D., & Thiede, K. W. (2005). Putting the comprehension in metacomprehension. Journal of General Psychology, 132, 408–428. Wiley, J., Hastings, P., Blaum, D., Jaeger, A. J., Hughes, S., Wallace, P., Griffin, T. D., & Britt, M. A. (2017). Different approaches to assessing the quality of explanations following a multiple-document inquiry activ­ ity in science. International Journal of Artificial Intelligence in Education, 27, 758–790. Wiley, J., Steffens, B., Britt, M. A., & Griffin, T. D. (2014). Writing to learn from multiple-source inquiry activi­ ties in history. In G. Rijlaarsdam (Series Ed.), P. Klein, P. Boscolo, L.C. Kirkpatrick, & C. Gelati (Vol. Eds.), Studies in Writing: Vol.28, Writing as a learning activity (pp. 120–148). Leiden, Netherlands: Brill. Wiley, J., & Voss, J. F. (1996). The effects of “playing” historian on learning in history. Applied Cognitive Psychology, 10, 63–72. Wiley, J., & Voss, J. F. (1999). Constructing arguments from multiple sources: Tasks that promote understand­ ing and not just memory for text. Journal of Educational Psychology, 91, 301–311. Wineburg, S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87. Wolfe, M. B., & Griffin, T. D. (2017). Beliefs and discourse processing. In M. F. Schober, D. N. Rapp, & M. A. Britt (Eds.), Handbook of discourse processes (2nd ed.) (pp. 295–314). New York, NY: Routledge.

21

LEARNING TO READ WHILE READING TO LEARN The Central Role of Multiple Documents in Two Instructional Programs Lowry Hemphill wheelock college, usa

Catherine Snow harvard graduate school of education, usa

The many chapters in this volume document the growing importance in the 21st century of nurturing students’ abilities to consider multiple sources of information in developing an argument, making a decision, and navigating the complexities of key social, civic, and scientific questions. We present in this chapter a description of two closely related literacy programs – Word Generation (WordGen) developed for the general population of US 4th–8th graders (10–14-year-olds), and the Strategic Adolescent Reading Intervention (STARI) designed for struggling readers in grades 6–8 (12–14-year-olds). Though the programs are distinct, they operate on very similar principles. They share a theory of action that posits engagement, active learning, background knowledge, and perspective-taking as key drivers of literacy outcomes, and both programs provide students with multiple documents related to central issues or questions, with support for processing those documents in purposeful ways. Our goal in this chapter is to use these two educational programs to display both the practicalities of achieving multiple-documents literacy in urban schools serving many students considered to be at risk for poor literacy outcomes, and the unique potential of multiple-document curricula for fostering more complex literacy abilities. We see multiple documents as serving a variety of purposes, both essential and instrumental, in both programs under consideration here. First, our definition of adequate literacy skills for adolescents coping with the demands of citizenship in a digital, information-rich environment locates analyzing, critiquing, evaluating, and

362

Learning to Read While Reading to Learn  •  363

synthesizing information from text as central processes (Goldman & Snow, 2015); these more complex reading processes can be applied to single documents, but are much more likely to be deployed if the reader encounters different documents that offer varied genres of information and distinct perspectives on a topic (Ferguson, Bråten, Strømsø, & Anmarkrud, 2013; Wiley & Voss, 1999). Second, skilled literacy involves an increased role for the reader in using background knowledge, often acquired through past reading, to elaborate upon and bridge gaps in what an individual text can convey (Kintsch, 1998). Carefully designed text sets can help build up reader understanding so that the documents themselves rather than the reader’s accumulated cultural capital become a primary resource for deeper understanding. Finally, through creating a larger role for readers’ own sense-making and evaluative efforts, multiple-document curricula can help overcome the declines in reading engagement that often characterize less advantaged learners in the later stages of schooling. A central assumption in our work is that active engagement in authentic literacy tasks is the best mechanism for promoting students’ comprehension skills. US teachers have been given very few tools to support reading comprehension other than teaching reading strategies. While students may benefit from being reminded to think about what they are reading, to summarize at regular intervals, and to reread if confused, such supports are inadequate to build robust comprehension skills. We argue that students learn to comprehend by working actively on text, and that active text processing can best be promoted by structuring an authentic purpose, e.g., solving a problem or engaging in a debate. If the student is focused, for example, on building evidence to support one position in a debate, if the available texts are accessible, and if the task is appropriately scaffolded, then students can comprehend successfully and in the process further build their comprehension skills. Exposure to multiple sources is not a universal experience for students even in secondary school, however, and in lessons that include multiple sources, the focus is often on extracting isolated facts, rather than analyzing, critiquing, and synthesizing, practices that are characteristic of skilled disciplinary reading (Litman et al., 2017). Without access to curricula that integrate reading and content instruction, teachers too often simply tell students what is in their texts. Lecturing about content, distributing summary worksheets, or using videos and PowerPoint decks can shortcut the need for students with limited comprehension skills to struggle with text (Chandler-Olcott, Doerr, Hinchman, & Masingila, 2015; Greenleaf & Valencia, 2016). Unfortunately, though such approaches may indeed transmit some information and enable coverage of the prescribed content, they do not help students develop the literacy skills needed to become independent readers or learners. In other words, students may learn what is in the civics text, but end up unable nonetheless to function as citizens because they can’t read about politics in newspapers or online sources with comprehension. In addition, because students are exposed only to the teacher’s summary of disciplinary content, they miss out on developing the critical thinking and reasoning skills that are forefronted when readers must make sense of multiple sources. We have incorporated multiple documents into programs designed for students with a particular set of educational needs and challenges – a group quite different from those with whom much prior work on the processing of multiple documents has been

364  •  Hemphill and Snow

carried out. Empirical research on multisource literacy has typically focused on older, more committed readers whose basic comprehension skills are largely secure and who are, for the most part, motivated to engage with academic texts (e.g., Braasch, Goldman, & Wiley, 2013; Bråten & Strømsø, 2010; Strømsø, Bråten, & Britt, 2010). Low-income students in early adolescence, however, show a higher incidence of reader traits that can affect the ability to read and reason successfully with multiple sources. Word recognition difficulties, found in as many as a third of struggling adolescent readers, can limit readers’ ability to devote cognitive resources to reasoning about text (Bråten, Ferguson, Anmarkrud, & Strømsø, 2013; Samuelstuen & Bråten, 2005) and can disrupt the construction of a literal textbase, a prerequisite for building adequate understandings (Kintsch, 1998; van Dijk & Kintsch, 1983). In addition, middle school students in low-income communities often show substantial gaps in their school-related background knowledge, such as knowledge about literary genres, scientific phenomena, and historical events (Heafner & Fitchett, 2015). Topic-related background knowledge may be especially important for multiple-source reading because of its role in helping to determine textual relevance and importance and in supporting inference generation, key processes in multisource literacy (Rydland, Aukrust, & Fulland, 2012). Finally, low-income students characteristically show declining levels of reading motivation as they move into adolescence (McKenna, Kear, & Ellsworth, 1995; Unrau & Schlackman, 2006). Reading motivation appears to contribute to students’ meaning-making with and across academic texts (Anmarkrud & Bråten, 2009), supporting effort and persistence. In designing a multiple-source literacy program for younger, at-risk readers, we concluded that it was therefore critical to incorporate texts at moderate levels of challenge, to design supports for building topic-related background knowledge, and to create tasks and learning contexts that sustain reading motivation. A small number of existing curricula attempt to engage younger and less advantaged readers with text sets using tasks that emphasize disciplinary reading and reasoning. The Seeds of Science, Roots of Reading Curriculum, for example, presents 7–11-yearolds with multiple texts on a common topic, guided by a scientific inquiry framework. After participation in Seeds of Science readings, peer discussion, literacy strategies support, and hands-on inquiry activities, 9–10-year-olds have shown greater gains in science content knowledge and vocabulary than peers who received typical textbook instruction (Cervetti, Barber, Dorph, Pearson, & Goldschmidt, 2012). Using a similar approach, Concept-Oriented Reading Instruction (CORI) has documented stronger reading gains for upper elementary students receiving text-rich science and literacy instruction as opposed to traditional, single-textbook reading instruction (Guthrie et al., 2009). Middle school students exposed to CORI history units, which integrate instruction in summarizing and inferencing with highly varied history texts, have shown superior growth in nonfiction reading comprehension (Guthrie & Klauda, 2014). Disciplinary practices for reading and reasoning across multiple sources in history, particularly contextualizing and sourcing (Wineburg, 1991), have been successfully implemented with students as young as 10–12 years old (VanSledright, 2002). An emphasis on controversy and argumentation, more common in history curricula for older adolescents (e.g., Nokes, Dole, & Hacker, 2007; Reisman, 2012), appears to enhance the impact of multiple-source curricula for younger students. Focused on argumentation and debate, a multitext history curriculum has been developed for

Learning to Read While Reading to Learn  •  365

11–14-year-olds (Wissinger & De La Paz, 2016). Students who received instruction on argumentation structures and participated in discussion of debatable key questions – e.g., was the U.S. justified in going to war with Mexico in 1846? – showed greater gains in historical content knowledge and academic writing than students who were simply exposed to multiple historical sources. Three features distinguish past efforts at developing effective multiple-source curricula for school-age children and younger adolescents: careful selection and adaptation of texts for readability and student interest; active inquiry tasks that motivate individual sense-making and disciplinary reasoning; and scaffolding for the processes of reading and responding to text. A fourth feature, present in some but not all effective multiple-source curricula, is a focus on controversy (Ferguson, Bråten, Strømsø, & Anmarkrud, 2013), guiding both text selection (e.g., texts that present competing or contrasting perspectives) and task design (e.g., partner or small-group discussion, argumentation and debate activities). Empirical work on existing multiple-source curricula suggests that they hold promise for improving students’ content area knowledge, an essential resource for building up deeper understandings of what is read, and for fostering improvement in overall reading abilities, particularly when implemented for longer periods of time, with texts matched to students’ reading abilities, and accompanied by forms of scaffolding that promote successful comprehension. In this chapter we describe our curricular materials so as to display how content area learning and more sophisticated literacy skills can be supported simultaneously through the use of multiple texts. One of the distinguishing features of WordGen and STARI is a focus on ensuring text accessibility even to high-needs students. We viewed text accessibility as multidimensional, promoted by the management of vocabulary challenge, decodability, necessary background knowledge, and text length as well as by scaffolding. As an example of how the curricula addressed text accessibility, before reading the more challenging young adult novels in each STARI unit, students read short, highly readable, topical passages to develop the phrasing, reading rate, background knowledge, and stamina needed to access longer texts. Students also learned morphological analysis skills important for decoding and understanding words like ineligible, detained, and impractical in unit texts. WordGen was implemented in classrooms with a wide array of reading levels, ranging from well below to above the actual grade level, but typically serving a high proportion of reluctant and unsuccessful readers. Thus, the WordGen texts were brief and catchy, represented multiple genres, and provided lots of readily retrievable information. In addition, WordGen lessons directly taught the meanings of key unit words. For example in a unit focused on the controversy over nutrition standards for school lunches in the US, students worked with the key words nutrition, effective, campaign, and eliminate. WordGen lessons also developed morphological analysis skills for linking and extending key unit words such as nutrition→nutritional→nutritious.

CONTEXT Word Generation and STARI were both launched in response to requests from an urban school district for help with middle-grades literacy development, and subsequently developed with collaboration from teachers in that and other urban districts

366  •  Hemphill and Snow

serving many struggling readers. In these settings, the challenges of beginning reading instruction – ensuring phonological awareness and accurate reading of highly frequent and of decodable words – were for the most part adequately met. The worries about student progress emerged at about age 9, after the basics of learning the reading and spelling rules of English had been mastered by most students. Teachers complained that students could read the words, but did not understand their texts, and thus could not learn academic content from reading those texts. The primary tool students were offered to improve their understanding was reading comprehension strategies, but these turned out to be inadequate. The new literacy challenges of the middle and later grades are, of course, multifaceted. They include new, more complex texts, a higher proportion of expository texts about unfamiliar topics, and new tasks, notably the task of learning through reading. Though good decoding and adequate fluency are prerequisite to meeting these challenges, it was clear that additional skills were also needed. Our research team formulated a set of hypotheses about key skills that are newly called upon in the middle grades but rarely explicitly taught: academic language, perspectivetaking, and argumentation. We hypothesized that skills in these three domains would explain variation in reading comprehension over and above skills of decoding and fluency. Indeed, a test of this hypothesis (LaRusso et al., 2016) demonstrated that all three of the hypothesized skill domains predicted reading comprehension outcomes for young adolescents, with academic language explaining the greatest amount of variance. As a consequence, we attended carefully to the need to provide opportunities to practice these skills in the development of our curricular materials. The use of multiple documents turned out to be crucial in addressing all of them. Academic language is a source of great frustration for students with weak reading skills, because it is unfamiliar and may seem impenetrable. We wanted students to process texts that presented information in academic language, but we also needed those texts to be accessible. Accordingly, we introduced the central idea and key concepts for each unit in texts written in conversational language, then piggy-backed the more formal presentation on the accessible version (see below for examples). Perspective-taking is similarly promoted when students confront two different points of view in adjacent texts on the same topic, or different points of view expressed by participants in a conversation presented as part of the curriculum. Explicitly contrasting the perspectives within or across texts raises awareness of perspective and of the language forms that signal perspective. In addition, tasks in which students are paired up to explain to one another their understanding of a single text often generate somewhat different readings, and the need to accommodate to and perhaps resolve alternative reader perspectives in the discussion. Finally, sophisticated reasoning requires accumulating and evaluating evidence, most powerfully from different sources. STARI and WordGen share a general theory of change. Both programs develop engagement by embedding reading tasks in the presentation of discussable issues related to students’ own lives. Both programs promote peer talk, in part to give students opportunities to practice explaining their own thinking, and in part to build engagement, because students enjoy peer activities. Both programs invented mechanisms through which students could acquire the background knowledge needed for

Learning to Read While Reading to Learn  •  367

reading core texts, and promoted careful reading of those texts by requiring students to develop personal stances on them and to provide textual evidence in support of their personal stances. Finally, both programs created opportunities for students to learn and engage in discourse practices needed for discussion and debate. Differences between the programs in target audience and use had implications for how texts were scaffolded. STARI, designed for struggling readers, responded to those students’ need for improved fluency by including short nonfiction texts at different reading levels, with routines for repeated reading that built up reading rate and stamina. STARI directly taught more challenging letter-sound patterns and more basic morphological analysis strategies (e.g., locating base words), while WordGen, serving smaller proportions of struggling readers, focused more on grade-level morphological analysis skills such as nominalizing suffixes.

WORD GENERATION’S USE OF MULTIPLE TEXTS The design of each Word Generation unit incorporates multiple texts relevant to the central theme or dilemma. The various texts were crafted or selected to serve a number of pedagogical purposes: •• To allow access for students at a range of reading levels, by including relatively easy as well as harder passages •• To introduce challenging topics with student-relevant analogs before moving to more authentic historical or scientific texts •• To provide material designed to support analysis of different perspectives on the central topic, and to support comparison of alternative perspectives on a particular issue •• To ensure that the language forms (academic vocabulary and academic language features such as nominalizations, discourse markers, and relative clauses) being focused on occurred in a variety of rich semantic contexts. We focus here on one typical WordGen unit, a 7th-grade science unit, to display the range of texts included and their intended role in the instructional activities. Science Generation 7.4: Populations in Balance The goal of SciGen unit 7.4 is to provide opportunities for students to learn about population ecology, by presenting information on a number of different specific cases: the overpopulation of tree snakes in Guam, the effects of pollution on the oyster population in Chesapeake Bay, the relation between numbers of hawks and of rodents in farmland, and the effects of forest fires and of hunting in promoting species strength. The unit begins with a Reader’s Theater conversation among a fictional group of 7th-grade students and Rob, the cousin of one of them, who is a Navy pilot on leave. In WordGen Reader’s Theater activities, students are presented with background information about the unit topic and are introduced to different possible stances or perspectives on the topic. Students have the option of simply reading the conversation or taking on individual speaking roles.

368  •  Hemphill and Snow

Elena: Rob: Miles: Rob: Miles: Rob: Elena: Rob:

I hope I’m not being rude by asking, but were you in Afghanistan? Nope, I’m stationed in Guam. Guam? Where’s Guam? It‘s kind of near Hawaii. Must be a nice place. It’s okay, but it’s full of snakes. No way! I hate snakes! No kidding. Especially the brown tree snakes; they’re everywhere! They’re in the trees, in the bushes . . . Heck, I even found one in my bed once. Elena: Do not tell me that. Rachel: Can’t people just get rid of the snakes? Rob: They’re trying. A lot of people use snake traps or dogs but nothing seems to work. You’re not going to believe this, but just before I came here the Navy had me fly a helicopter over the jungle, and folks were chucking dead mice stuffed with poison out of the helicopter. The idea was to kill the snakes by having them consume the dead mice. Elena, Miles, WHAT? and Rachel:  Elena: Did that really work? Do snakes eat dead mice? I thought they just ate live ones. Rob: I knew you’d be interested in that, Rachel. Check out the newspaper article I brought you. It’s there on the dash. The topic and some relevant information having been introduced through the conversation, a more academic treatment is provided by the newspaper article, which Rachel reads aloud (initial paragraphs only quoted here). The goal of the Reader’s Theater is to make the issue engaging and accessible for students, but the newspaper article provides experience reading about the issue in a more formal and academic register and further develops topic-relevant background knowledge.

Government Drops Dead Poisoned Mice from Sky in Another Attempt to Kill Tree Snakes The U.S. Department of Agriculture experimented with a new method to reduce the population of the invasive brown tree snake. They put Tylenol tablets into dead mice, attached the dead mice to long strips of paper, and dropped them out of helicopters into the jungle. Acetaminophen is a chemical used in Tylenol and other drugs. It’s safe for humans in normal doses but it is poisonous to snakes. This environmental disturbance is due to snakes brought to Guam from the Solomon Islands by the military after World War II. Since then, the brown tree snake population has exploded. Government officials say Guam now has one of the densest populations of snakes in the world.

Subsequently, students are given a brief authoritative text to read, with instructions to use the information in a writing task, and then to generate examples of naturally occurring vs. human-created and beneficial vs. harmful ecological disturbances:

Learning to Read While Reading to Learn  •  369

Are Ecological Disturbances Good or Bad? It depends. The situation you read about on Guam is an example of a disturbance of an ecosystem. A disturbance is when something happens that messes up the normal way an ecosystem functions. We tend to think that disturbances are bad, and in the case of the snakes on Guam, it does indeed look like a difficult situation that is causing serious problems. But in other cases, ecosystems depend on disturbances to maintain their health. Let’s consider forest fires. In California, redwood trees don’t get killed by forest fires. In fact, the tiny cones on the tree (like pine cones) drop their seeds during fires because that’s the best time for the seeds to sprout into new trees. When a fire clears away all the dense underbrush, seeds have a much better chance of growing into trees. Only then does sunlight shine all the way to the forest floor. The entire ecosystem depends on fire. Pretty amazing. The illustrations below are like a timeline of a fire. Try to complete the captions to explain how fire benefits the redwood tree and its ecosystem.

Next, yet another text focusing on disturbance in an ecosystem is introduced, this one about a different context entirely: the influence of pollution in Chesapeake Bay on the oyster trade. Four brief paragraphs about the decline of the oyster population are supplemented by an aerial photo of the Bay, a histogram documenting the decline, and an illustrated graphic of the Chesapeake Bay food web, further extending students’ content knowledge. Exposure to these varied text types offers different entry points to key unit content while also reflecting disciplinary content standards in science, which include making sense of data displays, diagrams, maps, and direct observations. The debate in this unit is structured as a Town Hall meeting called to discuss the competing demands of chicken farmers whose livestock is being preyed upon by hawks and grain farmers whose grain gets infested with rodents if hawks are not available to keep their numbers in check. The information for the debate is introduced in the form of a cartoon (see Figure 21.1), which is then discussed and expanded upon using data on hawk and rodent sightings that is provided but has to be plotted by the students in order to see the pattern. Immediate debate preparation is supported by explicit statements of different participants’ perspectives (see Figure 21.2). As is clear from these texts, the focus here is articulating contrasting perspectives and defending them using a single set of facts and texts. We consider the debate itself yet another text, an oral one, made available for reflection through use of a rubric on which students rate their own and their classmates’ participation and contributions. The classroom debate, a feature of each WordGen unit, promotes active learning and highlights perspective-taking. The debate preparatory notes and debate itself constitute preparation for the final science text in this unit, the essay students are guided to write on the last day. Instructions for the essay reintroduce the issue of controlling potentially invasive species: You are a customer service representative at Island Airways. You just received this email from a customer who is complaining about missing his flight.

Figure 21.1  Cartoon Included in SciGen Unit 7.4 to Provide Background Information Explaining the Relationship Between Populations of Chicken Hawks and Rodents.

Figure 21.2  Explicit Statements of Different Stakeholders’ Perspectives on Killing Hawks, in Preparation for the SciGen Unit 7.4 Classroom Debate.

372  •  Hemphill and Snow

Dear Island Airways, I am irate! I made it to the Guam airport in just enough time to catch my flight to Hawaii. However, I was forced to comply with some stupid inspection rule. Airport officials wanted to see if I had fresh fruit in my luggage. I told them I didn’t, but they went ahead and looked through my stuff anyway. When I told them that I would miss my flight, they just shook their heads and kept searching through my stuff. They said that they were especially worried about insect eggs getting off the island. Ridiculous! What nonsense! I want my money back and I’m never flying your crazy airline again!

Your boss has asked you to respond in a respectful way to help this customer understand the importance of these inspections to protect the populations on other islands. Wow! You have a tough job, teaching science to an angry customer. Best of luck to you! The final texts considered in the unit, then, are those the students produce themselves, expressing their own understanding of the issues. During the course of this week-long unit delivered by a science teacher, students are also exposed to additional texts relevant to the theme of population ecology in math class (processing estimates of how much different states in the Chesapeake Bay watershed contribute to water pollution), social studies (considering a variety of positions on whether hunting is a benign or harmful disturbance), and English language arts (evaluating a variety of analogies between organisms and human-made artifacts).

STARI’S USE OF MULTIPLE TEXTS In similar fashion, each STARI unit incorporates multiple texts relevant to a central topic such as conflicting demands of family responsibilities and school on teenagers, or the debate over immigration policy. Inquiry and discussion are organized around an essential question for each unit, e.g., What makes a family? or Who can be an American? Varied texts were written or selected: •• To build motivation and task persistence by embedding skills practice with decoding and fluency in cognitively challenging and provocative content •• To build struggling readers’ confidence, stamina, and topic familiarity through work with shorter texts that scaffold subsequent engagement with longer texts •• To further engage reluctant readers through texts that vary in readability, genre, visual support, and length •• To support analysis of different perspectives, both on the central unit topics (e.g., Unit 2.4, should immigration be controlled?) and within individual unit texts (Should schools close for Muslim holidays like Eid? Should the immigrant family in the novel Ask Me No Questions (Budhos, 2007) pay for forged documents to avoid deportation?) STARI unit 1.2, What makes a family?, illustrates the varied texts and activity within the STARI curriculum design. The unit centers on a young adult novel, Locomotion (Woodson, 2003), describing a young boy’s experience in foster care after his parents are killed in a devastating fire. The narrative in Locomotion unfolds through a series of poems written by the fictional character, Lonnie, as his class studies genres of poetry

Learning to Read While Reading to Learn  •  373

and tries their hand at poetry writing. Through poetic self-expression, Lonnie works through sadness and begins to build a new life in the foster family. Background knowledge relevant for comprehending Locomotion included understanding the foster care system, house fires, sickle cell disease, Lonnie’s favorite poet, Langston Hughes, and aspects of the New York City setting. Unit fluency passages built highly specific content understandings on these topics. At the end of each short passage, students were challenged to develop individual perspectives on passage content through “mini-debates.” In this fluency passage example, students learned information about Lonnie’s Brooklyn neighborhood and then were asked to contrast their own perspective on the content with the perspective of a peer reading partner:

Figure 21.3  Fluency Passage in STARI Unit 1.2 to Provide Background Information About the Setting of the Unit Novel.

374  •  Hemphill and Snow

To prepare for the complexities of the unit novel, students also read a nonfiction book about family structure (Who Are These People? Coping with Family Dynamics; Fallon, 2010), and selections from a poetry anthology for young adults (Poetry Speaks Who I Am: Poems of Discovery, Inspiration, Independence and Everything Else; Paschen & Raccah, 2010). In activities designed to promote active learning, students created their own texts (poems about a vivid personal memory and narratives about work experiences) that complemented unit texts. Classroom-wide unit debates built interest in specific textual content, promoted active learning, and motivated students to construct personal stances on what was read. A short story, “First Job” (Soto, 1993), launched the unit and introduced unit themes about work and mutual obligations within families. The character Alex in “First Job” takes on chores for a neighbor along with babysitting for his younger brother but manages to make a mess of the combined responsibilities. The unhappy outcome of the fictional story is echoed in a case study featured in the unit’s nonfiction book, in which a young adolescent takes on paid work to help his family but loses out on peer social life. Students read a more optimistic personal account of young teens working from the news program Story Corps and a disturbing news story about teen worker deaths on a Midwestern grain farm owned by Monsanto. Workbook activities prompted students to both note relevant details and construct personal perspectives as they read each text:

Figure 21.4  Prompt in STARI Unit 1.2 for Students to Construct Personal Perspectives on the Gary Soto Short Story, “First Job.”

Figure 21.5  Prompt in STARI Unit 1.2 for Students to Note Key Details and Construct Personal Perspectives on the Nonfiction Text, “Risks of Working.”

378  •  Hemphill and Snow

A debate question on which students might legitimately disagree (Teens working, a good idea or a bad idea?) challenged the class to return to these varied texts (fictional short story, nonfiction text, personal account, news report). In support of debate team presentations, students took notes on specific details from the texts (e.g., Noe, the author of the Story Corps personal account, was proud when his mother could use his earnings for household necessities) and synthesized specific information for their team presentations.

EFFECTIVENESS Despite the promise of multiple-source curricula for developing content knowledge and overall literacy abilities, larger-scale impact studies, particularly using random assignment, have rarely been attempted. WordGen has been evaluated by randomly assigning schools in four districts to implement the program or serve as controls. Levels of implementation varied widely, and were strongly associated with size of impacts. Students in classrooms in which 70% of the curriculum was covered showed substantial improvements in academic vocabulary, academic language, perspectivetaking, and reading comprehension (Jones et  al., under review). We interpret the strong effects of dosage as reflecting, in part, the value to students of access to multiple texts on each of the unit topics. STARI’s impacts were also assessed through a clinical trial in four low-performing school districts (Kim et al., 2017). Students who scored below proficient on state reading assessments were randomly assigned to either receive the year-long STARI curriculum or another literacy program implemented in their school. Students in both intervention and comparison groups were assessed at the beginning and the end of the school year with the Reading Inventory and Scholastic Evaluation (RISE; O’Reilly, Sabatini, Bruce, Pillarisetti, & McCormick, 2012; Sabatini, Bruce, Steinberg, & Weeks, 2015). The RISE was used to evaluate student progress on multiple aspects of reading skill that contribute to reading for understanding: decoding, morphology, vocabulary, sentence structure, reading fluency, and comprehension. At baseline, at the beginning of the school year, there were no significant differences between intervention and comparison groups on any of the RISE subtests (all p values less than .05). By the end of the 2013–14 school year, comparison students made very small gains in reading comprehension, morphology, and sentence processing. Comparison students also made only small gains in efficiency of basic reading comprehension, word recognition, and vocabulary. STARI participants showed greater growth than comparison students on end-of-year measures of word recognition (Cohen’s d = 0.20), morphological awareness (Cohen’s d = 0.18), and efficiency of basic reading comprehension (Cohen’s d = 0.21). STARI students also showed greater gains in sentence processing (Cohen’s d = 0.15), vocabulary (Cohen’s d = 0.16), and reading comprehension (Cohen’s d = 0.08), although these differences did not reach statistical significance. Areas in which STARI students showed the most progress relative to comparison students included recognizing words, analyzing word structure in complex words, reading fluently, and building a basic understanding of what was read. Although the effect sizes from the clinical trials of WordGen and STARI are comparatively modest, they reflect large-scale implementation of the programs by regular classroom teachers in low-performing schools and districts. Effect sizes for STARI in basic reading comprehension, fluency, and decoding equal or exceed those reported in meta-analyses of effective adolescent reading interventions (Wanzek et al., 2013) and, as for WordGen, were larger in classrooms where teachers were able to implement

Learning to Read While Reading to Learn  •  379

more of the curriculum. After a year of participation in STARI’s multiple-source curriculum, students substantially reduced the gap between themselves and classmates who had already scored proficiently on state reading assessments. Substantial differences in teachers’ ability to implement all or most of the STARI and WordGen curricula point to unmet needs for ensuring greater support at the school level and for in-depth teacher professional development. Encouraging diverse responses to text, promoting peer conversation about text, and scaffolding argumentation are novel practices for most content area teachers. Developers of multiple-source curricula need to plan robust supports for teachers to implement these new kinds of classroom literacy activities.

IMPLICATIONS FOR INSTRUCTION AND INTERVENTION The use of multiple documents relevant to a particular topic or unit might be thought of as appropriate for advanced students or those with relatively well-developed reading skills. Our goal in this chapter was to show that the use of multiple documents is feasible even with struggling, nonproficient readers, and in fact that a multipledocuments approach has particular value for such readers, offering the opportunity to promote engagement, to provide texts at various complexity levels, and to display an array of perspectives. The success of both the programs discussed here in promoting students’ literacy skills confirms both the feasibility and the effectiveness of the multiple-documents approach. The students we work with, just like those from more advantaged families attending better resourced schools, desperately need preparation to engage in analytic and critical reading of the cacophony of texts available to any citizen of the 21st century. That preparation requires the use of multiple texts and engagement in tasks that reveal the connections and contrasts among them. We introduced text sets and multiple-document types in STARI and in WordGen because manipulating such multiple texts is a key element of 21st-century literacy, made explicit as an expectation in the US Common Core State Standards (CCSS; National Governors Association Center for Best Practices & Council of Chief School Officers, 2010) and operationalized in the new assessments aligned to the CCSS. Less advantaged readers, along with their more successful peers, need opportunities to develop multiple text literacy for both academic success and for civic competence. Furthermore, we see the presence of multiple documents as an introduction to literacy skills central to English language arts, science, and social studies as disciplines. Though neither of the programs discussed here promises in-depth introductions to disciplinary literacy, they both expose students, via the use of multiple documents, to a few of the basic principles of disciplinary literacy: that literature evokes multiple interpretations, that understanding historical events requires resolving differing accounts offered by a wide variety of actors, and that science is a process of approaching the truth by accumulating knowledge through comments on and critiques of prior work. All these disciplines require multiple texts and multiple perspectives, so we model that for middle-grade students in our curricula. Beyond their value in promoting awareness of disciplinary literacy practices, multiple texts are crucial components of the kind of authentic literacy tasks that move students’ reading abilities forward. In both WordGen and STARI, activities with multiple texts created opportunities for student engagement and critical reasoning, the often missing ingredients in skilled reading in late childhood and adolescence.

380  •  Hemphill and Snow

ACKNOWLEDGMENTS The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education through Grant R305A090555 (Catherine Snow, Principal Investigator) to the Strategic Education Research Partnership (SERP), and through Grant R305F100026, awarded to SERP as part of the Reading for Understanding Research Initiative. The opinions expressed are those of the authors and do not represent the views of the Institute of Education Sciences or the U.S. Department of Education. We thank the collaborating school districts and school personnel, as well as the participating teachers and students.

REFERENCES Anmarkrud, Ø., & Bråten, I. (2009). Motivation for reading comprehension. Learning and Individual Differences, 19(2), 252–256. doi:10.1016/j.lindif.2008.09.002. Braasch, J. G., Goldman, S. R., & Wiley, J. (2013). The influences of text and reader characteristics on learning from refutations in science texts. Journal of Educational Psychology, 105(3), 561–578. doi:10.1037/ a0032627. Bråten, I., Ferguson, L. E., Anmarkrud, Ø., & Strømsø, H. I. (2013). Prediction of learning and comprehension when adolescents read multiple texts: The roles of word-level processing, strategic approach, and reading motivation. Reading and Writing, 26(3), 321–348. doi:10.1007/s11145-012-9371-x. Bråten, I., & Strømsø, H. I. (2010). When law students read multiple documents about global warming: Examining the role of topic-specific beliefs about the nature of knowledge and knowing. Instructional Science, 38(6), 635–657. doi:10.1007/s11251-008-9091-4. Budhos, M. (2007). Ask me no questions. New York: Athenaeum. Cervetti, G. N., Barber, J., Dorph, R., Pearson, P. D., & Goldschmidt, P. G. (2012). The impact of an integrated approach to science and literacy in elementary school classrooms. Journal of Research in Science Teaching, 49(5), 631–658. doi:10.1002/tea.21015. Chandler-Olcott, K., Doerr, H. M., Hinchman, K. A., & Masingila, J. O. (2015). Bypass, augment, or integrate: How secondary mathematics teachers address the literacy demands of standards-based curriculum materials. Journal of Literacy Research, 47(4), 439–472. Fallon, M. (2010). Who are these people? Coping with family dynamics. Minneapolis, MN: ABDO Publishing. Ferguson, L. E., Bråten, I., Strømsø, H. I., & Anmarkrud, Ø. (2013). Epistemic beliefs and comprehension in the context of reading multiple documents: Examining the role of conflict. International Journal of Educational Research, 62, 100–114. http://dx.doi.org/10.1016/j.ijer.2013.07.001. Goldman, S. R., & Snow, C. E. (2015). Adolescent literacy: Development and instruction. In A. Pollatsek & R. Treiman (Eds.), The Oxford handbook of reading (pp. 463–478). Oxford: Oxford University Press. Greenleaf, C., & Valencia, S. (2016). Missing in action: Learning from texts in subject-matter classrooms. In D. A. Appleman & K. A. Hinchman (Eds.), Adolescent literacies: A handbook of practice-based research (pp. 235–256). New York: Guilford. Guthrie, J. T., & Klauda, S. L. (2014). Effects of classroom practices on reading comprehension, engagement, and motivations for adolescents. Reading Research Quarterly, 49(4), 387–416. Guthrie, J. T., McRae, A., Coddington, C. S., Klauda, S. L., Wigfield, A., & Barbosa, P. (2009). Impacts of comprehensive reading instruction on diverse outcomes of low- and high-achieving readers. Journal of Learning Disabilities, 42(3), 195–214. doi:10.1177/0022219408331039. Heafner, T. L., & Fitchett, P. G. (2015). An opportunity to learn US history: What NAEP data suggest regarding the opportunity gap. The High School Journal, 98(3), 226–249. doi:10.1353/hsj.2015.0006. Jones, S. M., LaRusso, M., Kim, J., Kim, H. Y., Selman, R., Uccelli, P., . . . Snow, C. (under revision). Experimental effects of Word Generation on vocabulary, academic language, perspective taking, and reading comprehension in high poverty schools. Journal of Research in Educational Effectiveness. Kim, J., Hemphill, L., Troyer, M., Thomson, J., Jones, S., LaRusso, M., & Donovan, S. (2017). Engaging struggling adolescent readers to improve reading skills. Reading Research Quarterly. 10.1002/rrq.171. Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, MA: Cambridge University Press.

Learning to Read While Reading to Learn  •  381 LaRusso, M., Kim, H. Y., Selman, R., Uccelli, P., Dawson, T., Jones, S., . . . Snow, C. (2016). Contributions of academic language, perspective taking, and complex reasoning to deep reading comprehension. Journal of Research on Educational Effectiveness, 9(2), 201–222. doi:10.1080/10508406.2016.1256809. Litman, C., Marple, S., Greenleaf, C., Charney-Sirott, I., Bolz, M. J., Richardson, L. K., . . . Goldman, S. R. (2017). Text-based argumentation with multiple sources: A descriptive study of opportunity to learn in secondary English language arts, history, and science. Journal of the Learning Sciences, 26(1), 79–130. doi: 10.1080/10508406.2016.1256809. McKenna, M. C., Kear, D. J., & Ellsworth, R. A. (1995). Children’s attitudes toward reading: A national survey. Reading Research Quarterly, 30(4), 934–956. doi:10.2307/748205. National Governors Association Center for Best Practices & Council of Chief State School Officers (2010). Common Core State Standards for English language arts and literacy in history/social studies, science, and technical subjects. Washington, DC: National Governors Association and Council of Chief State School Officers. www.corestandards.org/assets/CCSSI_ELAStandards.pdf. Nokes, J. D., Dole, J. A., & Hacker, D. J. (2007). Teaching high school students to use heuristics while reading historical texts. Journal of Educational Psychology, 99(3), 492–504. doi:10.1037/0022-0663.99.3.492. O’Reilly, T., Sabatini, J., Bruce, K., Pillarisetti, S., & McCormick, C. (2012). Middle school reading assessment: Measuring what matters under a RTI framework. Reading Psychology, 33(1–2), 162–189. doi:10.1080/027 02711.2012.631865. Paige, D. D. (2011). Engaging struggling adolescent readers through situational interest: A model proposing the relationships among extrinsic motivation, oral reading proficiency, comprehension, and academic achievement. Reading Psychology, 32(5), 395–425. doi:10.1080/02702711.2010.495633. Paschen, E., & Raccah, D. (2010). (Eds.). Poetry speaks who I am: Poems of discovery, inspiration, independence and everything else. Naperville, IL: Sourcebooks. Reardon, S. F., Valentino, R. A., & Shores, K. A. (2012). Patterns of literacy among U.S. students. The Future of Children, 22(2), 17–37. doi:10.1353/foc.2012.0015. Reisman, A. (2012). Reading like a historian: A document-based history curriculum intervention in urban high schools. Cognition and Instruction, 30(1), 86–112. doi:10.1080/07370008.2011.634081. Rydland, V., Aukrust, V. G., & Fulland, H. (2012). How word decoding, vocabulary and prior topic knowledge predict reading comprehension. A study of language-minority students in Norwegian fifth grade classrooms. Reading and Writing, 25(2), 465–482. doi:10.1007/s11145-010-9279-2. Sabatini, J. P., Bruce, K., & Steinberg, J., & Weeks, J. (2015). SARA reading components tests, RISE forms: Technical adequacy and test design (ETS RR-15-32). Princeton, NJ: Educational Testing Service. doi:10.1002/ets2.12076. Samuelstuen, M. S., & Bråten, I. (2005). Decoding, knowledge, and strategies in comprehension of expository text. Scandinavian Journal of Psychology, 46, 107–117. Soto, G. (1993). Local news. San Diego, CA: Harcourt. Strømsø, H. I., Bråten, I., & Britt, M. A. (2010). Reading multiple texts about climate change: The relationship between memory for sources and text comprehension. Learning and Instruction, 20(3), 192–204. Unrau, N., & Schlackman, J. (2006). Motivation and its relationship with reading achievement in an urban middle school. The Journal of Educational Research, 100(2), 81–101. van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic Press. VanSledright, B. A. (2002). Fifth graders investigating history in the classroom: Results from a researcherpractitioner design experiment. The Elementary School Journal, 103(2), 131–160. Wanzek, J., Vaughn, S., Scammacca, N. K., Metz, K., Murray, C. S., Roberts, G., & Danielson, L. (2013). Extensive reading interventions for students with reading difficulties after grade 3. Review of Educational Research, 83(2), 163–195. doi:10.3102/0034654313477212. Wiley, J., & Voss, J. F. (1999). Constructing arguments from multiple sources: Tasks that promote understanding and not just memory for text. Journal of Educational Psychology, 91(2), 301–311. doi:10.1037/0022-0663.91.2.301. Wineburg, S. S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–77. Wissinger, D. R., & De La Paz, S. (2016). Effects of critical discussions on middle school students’ written historical arguments. Journal of Educational Psychology. 108(1), 43–59. doi:10.1037/edu0000043. Woodson, J. (2003). Locomotion. New York: Puffin

22

PROMOTING MULTIPLE-TEXT COMPREHENSION THROUGH MOTIVATION IN THE CLASSROOM John T. Guthrie university of maryland college park, usa

INTRODUCTION The main purpose of this chapter is to address the relationships of motivational and cognitive processes in reading with special reference to instructional contexts. Although the linkages of motivation to reading have been deeply investigated, these investigations have assumed that many motivations are relatively similar in how they influence cognitive processes of reading. The predominant view is that motivations drive literacy activities or other academic behaviors such as effort and strategy use, which in turn increase students’ achievement (Eccles & Wang, 2012). An assumption is that motivations influence cognitions in relatively similar fashions. In this chapter, the hypothesis is proposed that motivations are differentially associated with diverse cognitions depending on the characteristics of the motivational and cognitive processes involved. The proposal is termed the alignment hypothesis.

DIFFERENTIATED COGNITIVE PROCESSES OF READING A preliminary condition for explicating the alignment hypothesis is that cognitive processes in reading are heterogeneous in characteristics that influence their acquisition, functioning, and interrelationships. For the present purposes, we will discuss the cognitive processes shown in empirical research to be contributing to text and multipletext comprehension. Although we do not attempt to be exhaustive, we will consider the cognitive reading domain to include the following: working memory, word recognition, sentence comprehension, complex comprehension including text-knowledge integration, and synthesizing multiple-text knowledge (Walczyk et  al., 2007). These cognitive processes have been conceptualized and empirically shown to vary along several dimensions related to lower-order and higher-order cognitive aspects of reading.

382

Promoting Multiple-Text Comprehension  •  383

ATTRIBUTES OF READING PROCESSES Automaticity Lower-order cognitive reading processes are often shown to become automated as students gain reading proficiency. For example, the letter–sound associations entailed in word recognition become automated in the sense that they do not require explicit attention, which frees cognitive capacity for comprehension of word meanings. In contrast, higher-order cognitive processes such as the strategy of summarizing what has been learned from text interaction are usually deliberate (Cromley & Azevedo, 2007). In other words, automaticity is likely to characterize letter-sound associations but not online summarizing for proficient readers at, for example, the grade four level. However, when confronted with a challenging, unknown word, a fifth grader who has substantially automated word recognition may deliberately analyze the unknown word into syllabic constituents, retrieve sounds for them, and pronounce the word slowly to identify it. Likewise, proficient fourth graders who self-regulate their reading may typically summarize as they read a dense science text on an unknown topic (Guthrie et al., 2004). However, these two instances of self-regulated reading are the exception to frequent occurrence of automatic processing during reading in early elementary school. Consequently, many researchers concur that automaticity will occur for the lower-order processes such as letter-sound association but not for the higherorder processes such as online summarizing during reading (Lai, Benjamin, Schwanenflugel, & Kuhn, 2014).

TEXTUAL CHARACTERISTICS A second attribute of reading cognition that differentiates lower-order from higherorder processing is the nature of the text stimulus. Texts for which processing is often automated are letters, words, and phrases. In contrast, texts that are rarely automated include paragraphs, documents such as maps or directions, and multiple-text amalgams such as science texts in magazines or books (Bråten, Ferguson, Anmarkrud, & Strømsø, 2013). Textual characteristics influence the complexity of the cognitive system used to process the stimulus. Processing of a letter, for example the letter t, at the beginning of a word is relatively simple. Its sound is usually “t” as in top, although occasionally the sound may be “th” as in thick. However, the range of sounds for a letter is constrained, leading to cognitive benefits for repeated exposure. A complex paragraph, in comparison, has few limits in its possible constituency of letters, words, or meanings. Therefore, we classify word recognition as lower-order and complex paragraph comprehension as higher-order cognitive processes. For our present purposes, we distinguish between complex comprehension which demands extensive use of substantive knowledge and simple comprehension, such as reading a familiar story, which requires language processing but not deliberate use of complex declarative knowledge. Such language processing may well be automated in proficient readers, whereas deep knowledge use is deliberate in all except possibly experts in their fields.

384  •  John T. Guthrie

DIFFERENTIATED MOTIVATIONS RELATED TO READING Having outlined a set of dimensions that differentiate lower-order processes from the higher-order processes in the cognitive system of reading, we next propose several dimensions that differentiate a limited set of motivations frequently implicated in reading. In this review, we will attempt to characterize two pairs of motivations, consisting first of self-efficacy from socio-cognitive theory (Bandura & Schunk, 1981) and expectations from expectancy value theory (Wigfield & Eccles, 2002). The second pair is intrinsic motivation drawn from self-determination theory (Ryan & Deci, 2009) and valuing drawn from expectancy value theory (Wigfield & Eccles, 2002). The rationale for grouping the pairs in this fashion is that they share common attributes which will be proposed. We expect that each pair empirically correlates more highly with each other than with members of the other pair. Self-efficacy refers to an optimism that the individual will perform well on a specific task in a future circumstance. A recent definition from Bandura is that “self-efficacy [refers to] people’s beliefs in their capacities [which] vary across domains and situational conditions [rather than being] a general trait” (Bandura, 2012, p. 13). In early reading, a task may consist of reading a list of grade three-level words aloud. When a researcher attempts to measure self-efficacy by asking “Will you be a good reader next year?”, the third grader will situate the request. She will interpret it to mean word reading rather than story comprehension or symbolic interpretation of text. In a similar manner, expectations for the future drawn from expectancy value theory are constrained to a defined subset of possible future endeavors. Expectations have been defined as behaviors that an individual intends to perform or believes that she is likely to perform in a context such as a math course or the sport of baseball. Specifically, leading researchers have stated that “expectancies for success [are] children’s beliefs about how well they will do on upcoming tasks, either in the immediate or longer term future” (Wigfield & Eccles, 2000, p. 70). Typical questionnaires contain such questions as “Do you think you will take a math course in the future?” or “Do you think you will play baseball next year?” Empirically, the likelihood of participating in baseball is better predicted by questioning whether a person will participate in baseball than whether a person will participate in a sport. The concrete referent for the activity is more predictive than the more global referent. We are not suggesting that self-efficacy and expectations are identical. However, these two constructs share the attribute that the individual believes she will successfully perform given tasks in a constrained domain. Next, we consider intrinsic motivation. Ryan and Deci (2000, p. 56) stated that “intrinsic motivation is defined as the doing of an activity for its inherent satisfactions rather than some separate consequence.” Many investigators refer to intrinsic motivation as the enjoyment of an activity, or doing something for its own sake. For elementary students, it is often represented as fun. Our pairing of motivation constructs place valuing in the same set as intrinsic motivation. The definition of value in expectancy value theory refers to attaching personal significance to an activity through believing that it is important, useful, or inherently interesting. Wigfield and Eccles (2000, p. 74) stated that “components of achievement values [include] attainment value, or importance, intrinsic value, utility value or usefulness of the task, and cost.” Such value is typically placed on such activities as

Promoting Multiple-Text Comprehension  •  385

attending school, reading, or being a musician. It is superimposed on a domain rather than an artifact in that domain. One does not value a single math problem, a page in a book, or a musical note, although one may find them intriguing. Studies of valuing follow this perception by referring to valuing of reading, math, school, or sports, which are more general than individual tasks within any single domain.

ATTRIBUTES OF MOTIVATION PROCESSES Task Referent Versus Process Referent A first commonality of self-efficacy and expectancy is the referents for them. Selfefficacy in reading does not refer to optimism for reading any text at any time, but rather is contextualized within concrete tasks. Empirically this is evidenced by studies showing that measures of self-efficacy can easily be constrained, for example, “I am good at reading science information books,” rather than being more general, for example, “I am good at reading.” The constrained version correlates higher with proficiency on the specific task of comprehending science text than the more general version. Self-efficacy can become quite specialized. In a canonical correlation analysis of motivations for reading information books in school, self-efficacy emerged as a distinct motivational construct after a general motivation composite was extracted (Ho & Guthrie, 2013). Self-efficacy/expectancy is task specific rather than domain general. Our proposal is that the motivational constructs of intrinsic motivation and valuing rely on the individual’s experiential process. To determine intrinsic motivation for an activity, an individual reflects on whether the experience of participating in the activity was enjoyable. The process referent is an affectively salient experience rather than a fact about success. A typical questionnaire for intrinsic motivation refers to enjoying, finding pleasure, or being interested in an activity domain. In the same vein, we suggest that valuing is also based on an experiential process referent. If participating in an activity is perceived as important, central, or useful to oneself, it is said to be valued. One cannot know whether one values an activity such as reading without experiencing it as important or trivial, as personally meaningful or alien. Evidence for this experiential referent derives from the finding that valuing correlates more highly with holding mastery goals than performance goals (Martin & Elliot, 2016).

AFFECTIVE SALIENCE Another dimension used to differentiate motivations is affective salience. It is almost definitional that intrinsic motivation and valuing are affectively strong and positive. Among adolescents, valuing and intrinsic motivation correlated quite highly, and this correlation was significantly higher than the correlation of intrinsic motivation and self-efficacy or valuing and self-efficacy (Wolters, Denton, York, & Francis, 2014). In questionnaires, the probes of intrinsic motivation or valuing address the affective qualities of enjoyment, use, importance, or benefit of reading to the individual. In happiness research, school-related positive emotions correlated highly with “zest’ and “love of learning,” which are akin to intrinsic motivation; in contrast,

386  •  John T. Guthrie

those positive school affects were less well correlated with “hope,” which is akin to self-efficacy (Weber, Wagner, & Ruch, 2016). Driven by an appraisal of prior success, a person may possess a self-efficacy/expectation that is high, medium, or low in affect. For example, the individual may have a low estimate of success for an activity that is boring or unimportant, in which case the person will have no affect regarding the level of self-efficacy. Evidence for the difference in affective salience is supported by the finding that the personality traits of “openness” and “agreeableness” among 10-year-olds correlated significantly more highly with intrinsic motivation than selfefficacy (McGeown et al., 2014). Because they are information based, self-efficacy/ expectations are affectively more neutral than intrinsic motivation/valuing, which are more grounded in emotionally significant experiences.

COGNITION–MOTIVATION ALIGNMENT HYPOTHESIS The alignment hypothesis for motivations and cognitive processes in reading has two parts. First, it states that higher-order reading cognitions will correlate more highly with the motivations of intrinsic motivation and valuing than the motivations of selfefficacy and expectations. Second, it states that lower-order cognitions will correlate more highly with motivations of self-efficacy and expectations than the motivations of intrinsic motivation and valuing. To refine the alignment hypothesis, a crucial differentiation in contexts for reading should be noted. One context that is widely studied in reading motivation is the personally significant real-world environment, consisting mainly of out-of-school reading, reading for enjoyment in free time during school, and designated independent reading activities. The other circumstance for reading is the academic context consisting of reading for courses or school work. At least two studies have shown that motivation-cognition associations differ in these two contexts. Researchers have investigated the two contexts explicitly among Flemish fifth graders (Naeghel, Keer, Vansteenkiste, & Rosseel, 2012). A path model for recreational reading showed that self-efficacy was associated with reading engagement, which was associated with reading comprehension. Likewise, valuing in the recreational context, which was termed autonomous motivation, was correlated with reading engagement, which was associated with reading comprehension. However, in a path model for academic reading, self-efficacy performed the same way as in recreational reading, but valuing did not.

Table 22.1  Cognitive–Motivational Alignment Hypothesis.

Cognitive constituents of reading Attributes of lower and higher-order reading cognitions Reading motivations Attributes of motivations

Reading

Processes

Lower-order cognitive processes Working memory; decoding; word recognition; sentence comprehension High automaticity; simple text stimulus; processing complexity low

Higher-order cognitive processes: Passage comprehension; multiple-text comprehension; literacy engagement Low automaticity; complex text stimulus; processing complexity high

Self-efficacy/Expectancy Intrinsic motivation/Valuing Informational referent; affective low Experiential referent; affect high

Promoting Multiple-Text Comprehension  •  387

Valuing for academic reading was not associated with reading engagement, although engagement was associated with reading comprehension. The key difference was that in the recreational context valuing was positively associated with reading comprehension, whereas in the academic context valuing was not associated with reading comprehension. Confirming this finding, investigators reported the motivations of third and fifth graders reading either for school or for personal enjoyment (Cox & Guthrie, 2001). A composite of valuing and intrinsic motivation correlated significantly more highly with amount of reading for personal enjoyment than amount of reading for school. This finding was sustained even when controls for previous achievement and strategy use were employed. Collectively these studies suggest that achievement is more highly associated with valuing for real-world reading than valuing for academic reading. It is possible that valuing reading in the personal, real-world environment is more deeply rooted in the individual’s identity than valuing in the academic context. Consequently, real-world valuing generates stronger goals, more persistence, and deeper comprehension than academic valuing. Using an expectancy value framework, investigators reported that elementary students’ general self-efficacy predicted number of courses taken in secondary school more highly than amount of leisure reading in elementary school. However, leisure reading in secondary school was predicted more highly by their intrinsic motivation in elementary school than the number of courses taken in secondary school (Durik, Vida, & Eccles, 2006). On this basis, we expect that complex reading comprehension may be more highly associated with intrinsic motivation and valuing than self-efficacy and expectations in personally significant, real-world contexts. However, complex comprehension may be more highly associated with selfefficacy/expectancy than intrinsic motivation/valuing in academic contexts. Academic context may be constructed by an investigator through anchoring motivations in science texts or information texts read for school. The notion of academic context extends to topics such as biodiversity, astronomy, or math that are studied in school, but are not often experienced or explored by children or students until they become knowledgeable professionals in such a domain. We expect to see higher-order comprehension in science texts to be more highly associated with expectancy and selfefficacy anchored to science reading than intrinsic motivation and valuing similarly qualified into science. The rationale for this proposed alignment is grounded in a functional view of the roles of motivation in cognitive reading acquisition. Basic processes are imperative. To learn reading, students must gain proficiency in rapid word recognition, for instance. It is reasonable that students who have a drive to succeed in word recognition, and who can accurately detect their current level of task success, will be at an advantage in learning. Such drive and detection are integral to self-efficacy and expectancy. For higher-order processes, these motivations remain beneficial but are less predominant. After basic skills are learned to a functional level, there are diverse texts, myriad topics, and multiple interpretations possible in the reading domain. For complex comprehension, strategy use, and evaluative interpretation of text, students need a guidance system for selecting among options. In such a situation students benefit from a set of interests, preferences, and personal identifications that enable them to select. Students and citizens pursue topics and select cognitive strategies that are beneficial to their social, cultural, and intellectual needs and aspirations. Such pursuits

388  •  John T. Guthrie

are steered by intrinsic motivation and valuing which energize the press for advancement and navigation of the options available. In brief, it is reasonable that both sets of motivations, self-efficacy/expectancy and intrinsic motivation/valuing, should play distinctive, but coordinated and functional roles in the growth of reading expertise.

ASSESSING THE COGNITIVE–MOTIVATION ALIGNMENT HYPOTHESIS To test the alignment hypothesis, we have conducted simple secondary analyses of existing studies. These analyses usually consisted of comparisons of the correlations of two motivations with a cognitive measure. For instance, for middle school students, we compared the correlation of self-efficacy and simple text comprehension, which was moderately high, with the correlation of intrinsic motivation and simple comprehension, which was significantly lower (Ho & Guthrie, 2013). For our purposes, we inferred a difference between the correlations only if the computed difference using the numbers in each sample was statistically significant. Higher-Order Reading Processes and Motivation Complex Text Comprehension A study of a nationally representative sample of eighth-grade students in the U.S. showed that complex reading comprehension was more highly associated with a measure that merged valuing with effort in general, recreational contexts than a measure of self-efficacy that emphasized perceived competence. With controls for previous achievement, socioeconomic status, and other motivation variables, valuing was dramatically higher than self-efficacy in predicting complex reading comprehension (Guthrie, Wigfield, Metsala, & Cox, 1999). For reading in an academic context, the alignment hypothesis expects that selfefficacy will exceed valuing in correlating with complex comprehension, as stated previously. Several studies investigated motivations of self-efficacy/expectations and intrinsic motivation/valuing in relation to science text reading in an academic context. These studies have anchored the items in motivation questionnaires with such qualifiers as “natural science texts” or “information books at school.” In a study of tenth-grade students, multiple-text comprehension was significantly correlated with self-efficacy anchored in reading natural science texts but comprehension was not significantly associated with valuing in this context (Bråten et al., 2013). Confirming this result, when the motivational constructs were qualified into the academic context of “reading information books for school,” an investigation of seventh graders showed that complex information text comprehension was more highly correlated with self-efficacy than valuing. In a conceptual replication within that study, information text comprehension correlated more highly with perceived difficulty (the inverse of self-efficacy) than devaluing (the inverse of valuing) (Guthrie, Klauda, & Ho, 2013). Several investigators have directly compared the association of lower and higherorder cognitive reading processes to reading motivations including self-efficacy, intrinsic motivation, and valuing. In one study, reading comprehension of social

Promoting Multiple-Text Comprehension  •  389

studies texts among ninth graders was related to self-efficacy and valuing which were qualified by referencing them to “understanding books and texts.” Because this qualification was not specific to a subject matter such as history, or a context such as school, this referencing may be considered general rather than academically specific. Results showed that reading comprehension that demanded inferencing and understanding main ideas correlated significantly higher with valuing than self-efficacy, which is consistent with the alignment hypothesis. The higher-order reading processes were more closely linked to valuing than self-efficacy, whereas the achievement outcome of grades linked more highly to self-efficacy than valuing (Anmarkrud & Bråten, 2009). In a related study, a general measure of intrinsic motivation, which was not anchored to a specific text type, subject matter, or context, correlated higher with complex comprehension than working memory, which is expected by the alignment hypothesis (Andreassen & Bråten, 2010). When reading in an academic context was examined by referencing motivations for “text that I read in natural science,” researchers found a relatively low but statistically significant correlation of .18 between individual interest and multiple-text comprehension (Bråten, Anmarkrud, Brandmo, & Strømsø, 2014). This illustrates the weakness of the association of intrinsic motivation and achievement in an academic context. A second investigation of multiple-text comprehension and motivation in the academic context of “reading science texts” showed no significant difference between reading comprehension and valuing or self-efficacy (Bråten et  al., 2013). However, the study had a relatively low number of participants, limiting its power for this comparison purpose. Among first and second graders, passage comprehension correlated significantly with valuing, but it was not significantly correlated with self-efficacy (Cartwright, Marshall, & Wray, 2016). Thus, where comparisons are feasible and sensitive, the alignment hypothesis is confirmed for the association of reading motivations and reading comprehension including multiple-text comprehension. Multiple-Text Comprehension Multiple-text comprehension may be distinguished from complex text comprehension by a range of textual, processing, and outcome characteristics, which are addressed more fully by Bråten, Braasch, and Salmerón (in press). Briefly, one model proposes several phases (Leu, Kinzer, Coiro, Castek, & Henry, 2013). To find information on a topic, for example, the reader will need to a) generate key words that return useful websites, b) read a set of links returned from the search engine to infer which websites might be useful, and c) skim and scan information presented within the websites. In a next step, the reader decides upon which information is reliable by critically evaluating the available information (based on accuracy, reliability, potential biases, and so forth. The individual then synthesizes information deemed useful for answering the question into a coherent understanding of what was read. Finally, the reader communicates the constructed response to an intended audience. The extensive roles of diverse cognitive strategies in multiple-text comprehension were shown by Guthrie and colleagues (1998) experimentally. In their investigation, students who were taught strategies of questioning, searching, summarizing, and organizing knowledge were superior to control students in multiple-text comprehension and conceptual knowledge transfer in an extended performance assessment.

390  •  John T. Guthrie

Contributions of interest to multiple-text comprehension have been reported (Bråten et  al., 2014; Strømsø & Bråten, 2009; Strømsø, Bråten, & Britt, 2010). For example, Strømsø and Bråten (2009) found that topic interest, referring to students’ self-reported individual interest in the topic of climate change, uniquely explained variance in multiple-text comprehension when several cognitive variables were controlled. In another study, fifth graders’ individual interest in biology predicted strategy use in learning from science books and conceptual knowledge about the biological principles of adaptation (Alao & Guthrie, 1999). It should be noted that neither of these investigations controlled for other motivations such as self-efficacy or the quality of students’ task engagement. In path modeling of variables influencing multiple-text comprehension, Bråten and colleagues (2014) showed that individual interest in understanding science texts contributed indirectly to multiple-text comprehension with controls for prior knowledge, deep strategies, and effort. However, in a different path model of multiple-text comprehension controlling for multiple motivations including self-efficacy, Guthrie and colleagues (2013) reported that intrinsic motivation for reading science books did not uniquely contribute to engagement or complex text comprehension. Instead, selfefficacy and perceived difficulty contributed to text comprehension when controlled for other motivations. Although interest and intrinsic motivation are not identical, this suggests that these interest-related variables are correlated with self-efficacy, which is a strong predictor of multiple-text comprehension in academic contexts. Both the Bråten et al. (2014) path model and the Guthrie and Klauda (2016) literature review concur that the causal flow appears to be from knowledge and motivations—to effort in task engagement—to text and multiple-text comprehension proficiency. The relative roles of different motivations contributing to multiple-text comprehension have been addressed. Concurring with the alignment hypothesis, the association of self-efficacy and multiple-text comprehension was higher than the association of value and multiple-text comprehension in an academic context (Bråten et al., 2013). Consistent with this finding, it was found that in an academic context, self-efficacy and perceived difficulty correlated higher than value and devaluing with complex text comprehension, a state reading test (omnibus measure of text comprehension), and grades two years later (Rosenzweig & Wigfield, 2017). Engagement in Reading According to the alignment hypothesis, engagement in reading is expected to be associated with valuing and intrinsic motivation more highly than self-efficacy or expectancy. This is due to the individual’s reliance on volitional strategies for finding and pursuing texts, and metacognitive strategies for comprehending the texts sufficiently to enjoy or identify with them. Inspecting a national sample of secondary school students, investigators found that a measure of reading engagement consisting of self-report (frequency in reading a variety of text types in a real-world, outof-school context) was more highly correlated with a measure of valuing and effort than a measure of self-efficacy (Guthrie et  al., 1999). This finding was controlled for previous achievement, socioeconomic status, and other reading motivations. Investigating middle school students’ engagement in reading information books in school, engagement correlated significantly higher with intrinsic motivation than

Promoting Multiple-Text Comprehension  •  391

self-efficacy, and reading engagement correlated higher with valuing than selfefficacy (Guthrie et al., 2009). Using undermining motivation variables with middle school students, investigators found that engagement was negatively correlated with devaluing (inverse of valuing) more highly than perceived difficulty (inverse of selfefficacy) (Guthrie et al., 2013). In a brief longitudinal analysis of reading social studies texts, investigators reported that across two time periods, diverse measures of reading engagement correlated higher with valuing than self-efficacy on three of four occasions, and reading engagement correlated significantly higher with devaluing (inverse of valuing) than perceived difficulty (inverse of self-efficacy) on four of four occasions (Guthrie & Klauda, 2014). Thus, the alignment hypothesis that reading engagement is more highly associated with valuing than self-efficacy was confirmed across several time periods and diverse measures (affirming and undermining) of motivation constructs. This finding is robust across academic and recreational contexts. Effects of motivations on achievement are often mediated by reading engagement. For engagement operationalized as amount of reading among students in grades 2–3, the intrinsic motivational construct of “reading involvement” was highly associated with reading amount, which in turn was associated with word recognition, sentence comprehension, and text comprehension. Also, the effects of intrinsic motivation on achievement were mediated by engagement measured by reading amount (Stutz, Schaffner, & Schiefele, 2016). Further, researchers have reported that the effect of intrinsic motivation on reading comprehension is fully mediated by reading engagement in the form of reading amount (Schaffner & Schiefele, 2016). This mediation effect is stronger for the outcome of comprehension than word recognition, which is compatible with the alignment hypothesis. Among middle school students, effortful behavior in school, which is akin to reading engagement, correlated higher with intrinsic motivation than self-efficacy. Similarly, disruptive behavior was inversely associated with intrinsic motivation more highly than self-efficacy (Shin & Ryan, 2014). It should be noted that the link of engagement to achievement is robust in most circumstances throughout schooling. Most persuasive in this regard is the wealth of evidence that across ages 9 to adult, amount of print exposure correlates with understanding complex text and the comprehension-driving construct of world knowledge in an extensive meta-analysis (Mol & Bus, 2011). Thus, it is reasonable to suggest that valuing and intrinsic motivation energize engagement, possibly by fostering self-regulation and volitional strategies, to generate growth in reading comprehension (Wigfield et al., 2008). Lower-Order Reading Processes and Motivation An important aspect of the alignment hypothesis is the proposition that lowerorder reading cognitions are expected to correlate more highly with self-efficacy or expectancy motivations than intrinsic motivation or valuing. Within the class of lower-order reading cognitions, we include working memory, word recognition, oral reading fluency, and simple comprehension. Recall that the rationale is that self-efficacy and expectancy are characterized by their strong relation to successful performance, likelihood of contributing to automaticity of cognitive processes, and dependency on information rather than affective experience. Relatively few

392  •  John T. Guthrie

studies made direct comparisons needed to most strongly test the hypothesis, but several investigations report highly related findings. Word Recognition Among basic cognitive processes, word recognition is often highly correlated with reading comprehension (Tannenbaum, Torgesen, & Wagner, 2006). Consequently, motivations for word recognition are integral to the alignment hypothesis. One investigation showed that among students in grades 3–5, intrinsic motivation correlated significantly with text comprehension, but intrinsic motivation did not correlate significantly with word recognition efficiency measured by the TOWRE test (Liebfreund & Conradi, 2016), which directly confirms the alignment hypothesis. Corroborating this result, investigators reported that first graders’ word identification measured by the Woodcock Johnson test correlated significantly with self-efficacy, but did not correlate significantly with intrinsic motivation. While those results were based on student self-reports of motivation, teacher perceptions of students’ motivation concurred that word identification correlated higher with students’ self-efficacy than intrinsic motivation (Coddington & Guthrie, 2009). Likewise, for primary students, word recognition and word attack skills on the Woodcock Johnson test correlated significantly with self-efficacy but not with valuing, although the difference was not statistically significant (Cartwright et al., 2016). Among older students in fifth grade, word recognition correlated higher with self-efficacy than valuing, although the difference did not attain statistical significance. However, word recognition correlated significantly higher with perceived difficulty (inverse of self-efficacy) than valuing (Guthrie et al., 2009). It has been observed that cognitive reading variables often correlate more highly with undermining constructs of motivation than affirming versions (Guthrie et  al., 2013). Undermining constructs are conceptualized and measured to correlate negatively with achievement, whereas affirming motivations are proposed to correlate positively with achievement. For example, whereas valuing (“Reading is important to me”) is positively correlated with reading comprehension, devaluing (“Reading is not important to me”) is negatively correlated with reading comprehension. A comparison of the affirming and undermining variables shows that the associations of undermining variables with dis-engagement or achievement are higher than the associations of affirming variables with engagement or achievement. One likely reason is that the affirming ones are more subject to social desirability, which introduces error. This statistically significant correlational advantage can be observed in matrices of multiple motivations as reported by Guthrie and Klauda (2014). Reading Fluency Reading fluency is a relatively lower-order cognitive process that is subject to automaticity. Consequently, the alignment hypothesis proposes that fluency will correlate higher with self-efficacy/expectancy than intrinsic motivation/valuing. In this review, we include the speed of reading a word list aloud, speed of reading prose aloud, and the prosody of reading text orally as three operational definitions of reading fluency that semi-independently contribute to reading comprehension (Klauda & Guthrie, 2008).

Promoting Multiple-Text Comprehension  •  393

One investigation reported that among fifth graders, fluency according to the Woodcock Johnson measure, correlated more highly with perceived difficulty (selfefficacy inverted) than intrinsic motivation, and fluency measured as prosody of reading aloud correlated higher with perceived difficulty than intrinsic motivation (Guthrie et al., 2009). Among seventh graders, a similar result was observed. Reading fluency correlated significantly with self-efficacy (and negatively) with perceived difficulty, but did not correlate significantly with valuing, devaluing, or intrinsic motivation (Ho & Guthrie, 2013). It should be noted that the undermining constructs were not weak in the study, because perceived difficulty and devaluing both correlated significantly and positively with reading avoidance, which is a negative form of engagement. These findings confirm the alignment hypothesis. Simple Comprehension In the set of lower-order cognitions of reading, we include simple text comprehension. This construct refers to reading simple declarative fiction text that may be understood with relatively limited inferencing or reasoning. Basic language skills of sentence comprehension suffice to provide a reasonable level of proficiency. These language processes are often automated by age 9. The Gates-MacGinitie comprehension test is an example of a measure of simple comprehension. In contrast, multiple-text and multi-genre tasks of the kind used in PISA or PIRLS demand complex comprehension including reasoning and inter-textual integration. For primary students who are acquiring lower-order cognitive processes in reading, we expect that reading proficiency will correlate more highly with self-efficacy than intrinsic motivation or valuing. In one study of 7–8-year-olds in England, competency in sentence comprehension correlated with self-efficacy, whereas this competency did not correlate with intrinsic motivation. Among 10–12-year-olds, competency, which was measured by simple passage comprehension, correlated with both self-efficacy and intrinsic motivation although the former correlation was higher (Logan & Medford, 2011). Embracing a suite of basic reading processes, one study reported a moderately high correlation of simple comprehension with self-efficacy for processes including word recognition, sounding out long words, and identifying syntactic relations. Simultaneously, the study reported a significantly lower correlation of simple comprehension and self-efficacy for reading eighteen different text types such as employment applications and poetry (Shell, Murphy, & Bruning, 1989). Simple comprehension correlated higher with self-efficacy for the most basic reading skills than for comprehension of real-world texts. In a related finding, investigators reported that general intrinsic motivation correlated lower with working memory than with complex text comprehension (Andreassen & Bråten, 2010), which is consistent with the alignment hypothesis. Finally, among first graders the Woodcock Johnson reading comprehension measure correlated significantly with valuing in reading but did not correlate significantly with self-efficacy, although the difference was not statistically significant (Cartwright et al., 2016). For students 13 years of age, simple comprehension as shown on the GatesMacGinitie has correlated higher with self-efficacy than either valuing or intrinsic motivation in a variety of investigations, confirming the alignment hypothesis

394  •  John T. Guthrie

(Ho & Guthrie, 2013). It should be noted that for ninth graders, comprehension of social studies text in a school context was more highly correlated with general measures of valuing than self-efficacy (Anmarkrud & Bråten, 2009). It seems likely that the comprehension task in this investigation required use of background knowledge, inferencing, reasoning, and text integration. Such characteristics qualify the task as complex rather than simple comprehension. Consequently, the task was associated more with valuing than self-efficacy, which is in accord with the alignment hypothesis. It should be noted that one study reported a significantly higher correlation between complex text comprehension and self-efficacy than valuing (Solheim, 2011), which does not concur with the alignment hypothesis. However, the reliability of the valuing construct was quite low, and valuing did not correlate with any variable in the study other than self-efficacy, raising doubts about the strength of any null inference from this variable in this data set.

INSTRUCTIONAL PRACTICES RELATED TO THE ALIGNMENT HYPOTHESIS In this chapter, we address instruction or intervention in text comprehension through motivation by beginning with several assumptions. First, we assume as a higher-order form of reading, multiple-text comprehension relies on lower-order cognitive systems. We further suppose that instruction in multiple-text comprehension can and should be taught in grades 3–5 because higher-order processes needed for complex text can be learned in a rudimentary form at this age (Guthrie et al., 2004). To apply the reading motivation–cognition alignment hypothesis to issues of instruction, we propose that optimal instruction for lower-order cognitive processes should emphasize support for motivations of self-efficacy/expectations while also including moderate support for intrinsic motivation/valuing. Similarly, we propose that instruction and intervention for higher-order processes in secondary school (academic context) should emphasize self-efficacy/expectations while placing a moderate priority on intrinsic motivation/valuing. In the middle elementary grades of 3–5, we propose an instructional balance with an emphasis on both self-efficacy/expectations and intrinsic motivation/valuing because both lower-order and higher-order cognitive processes are emergent and recreational reading is being acquired.

INSTRUCTION IN LOWER-ORDER READING PROCESSES In reading, lower-order cognitive processes include working memory, which is sufficiently powerful that it correlates with reading comprehension throughout the elementary grades even when processes of background knowledge and fluency are controlled (Tiffin-Richards & Schroeder, 2015). However, working memory is not easily modified with instruction. Although extensive training can increase working memory for words or abstract visual shapes temporarily among adults, the effect disappears within several weeks (Jaeggi, Buschkuehl, Shah, & Jonides, 2014), suggesting that interventions to improve working memory for young children are not advisable. For primary students, strong experimental evidence from multiple studies shows that an emphasis on self-efficacy support during early reading instruction serves to

Promoting Multiple-Text Comprehension  •  395

increase self-efficacy as well as word recognition (Schunk & Mullen, 2009). Efficacysupportive intervention has focused on helping students set realistic goals and provided task-specific feedback for learning. Whether other motivations such as intrinsic motivation increase under these conditions has not been deeply investigated. Multiple studies verify that providing competence support in the form of feedback on progress and helping students set realistic goals increases self-efficacy for the domain of the activity (Schunk & Zimmerman, 2007). Furthermore, reading comprehension instruction for struggling readers in fourth grade was most effective for students who had relatively higher self-efficacy at the outset of instruction (Cho et al., 2015). Competence support may also appear in the form of using content texts, such as science trade books, that are readily decodable and enable students to derive knowledge of the world while learning reading skills (Guthrie, McRae, & Klauda, 2007). These findings confirm the expectation from the alignment hypothesis that instruction emphasizing self-efficacy for young or struggling learners can effectively raise achievement and self-efficacy simultaneously.

INSTRUCTION IN A BALANCE OF LOWER AND HIGHER-ORDER READING PROCESSES For students in grades 3–5 the alignment hypothesis applied to instruction is that a balance of emphasis on self-efficacy/expectancy and intrinsic motivation/valuing will be optimal for acquisition of both lower-order and higher-order reading processes. Preliminary evidence is provided by a meta-analysis of research on Concept Oriented Reading Instruction (CORI). Providing a balance of support for both motivation sets, the effect size of CORI across several studies was .49 for raising self-efficacy (five studies), and .29 for reducing perceived difficulty (two studies). Simultaneously, effect sizes were .47 for increasing curiosity (five studies), .31 for preference for challenge (three studies), and .29 for enjoyment of reading (three studies), showing that constructs related to intrinsic motivation were facilitated as well as self-efficacy. Effect sizes showed increase in reading achievement at several cognitive levels including word recognition .75 (two studies), oral reading fluency .59 (two studies), standardized test .91 (five studies), and multiple-text comprehension .93 (seven studies) (Guthrie et al., 2007). In brief, instruction balanced to support both self-efficacy and intrinsic motivation increased those motivations in similar degrees, and facilitated growth of multiple-text comprehension which was taught with direct strategy instruction. A different study showed that the CORI effects on simple reading comprehension and use of multiple-text comprehension strategies were fully mediated by the amount and depth of students’ reading engagement which depended directly on their motivations (Wigfield et al., 2008). Benefits of CORI for growth of multipletext comprehension was partially mediated by motivation-engagement, due in part to the science knowledge structures taught in the CORI unit that may have transferred to the assessment. Furthermore, instruction designed to emphasize fifth-grade students’ self-directed and collaborative literacy activities increased self-efficacy (termed reading self-concept) and a composite of intrinsic motivation and valuing (termed valuing), although an achievement measure was not included in the study (Marinak Malloy, Gambrell, & Mazzoni, 2015).

396  •  John T. Guthrie

INSTRUCTION FOR HIGHER-ORDER READING PROCESSES The alignment hypothesis suggests that self-efficacy/expectations should be emphasized in secondary school due to its prominent role in correlating with complex comprehension in academic contexts. However, explicit support for self-efficacy in secondary instruction is relatively rare. In a meta-analysis of 74 studies of motivation interventions containing 92 effect sizes, self-efficacy was not identified as a construct that served as the sole target of an intervention (Lazowski & Hulleman, 2015), although self-efficacy was used in studies employing multiple theoretical perspectives. In this meta-analysis, valuing was identified as a target motivation in seven intervention studies, but self-efficacy/expectancy not isolated for analysis. In contrast, valuing and intrinsic motivation have been used and evaluated in secondary instructional research. For secondary and college students, several studies have reported the benefits of providing a “valuing rationale” for reading an uninteresting text. In two studies, the valuing rationale described how information in the text would provide immediate professional benefit for prospective teachers. Groups receiving a valuing rationale showed enhanced behavioral engagement (close attention to reading) and increased conceptual comprehension of information text compared to no-rationale groups (Jang, 2008). In another study, asking college students to find usefulness and applicability in text increased students’ comprehension in both laboratory and classroom settings compared to control conditions (Hulleman, Godes, Hendricks, & Harackiewicz, 2010). Relevant for intrinsic motivation, interventions with secondary students that emphasized the relevance of text to students’ interests (termed intrinsic framing) increased students conceptual learning more than instruction emphasizing getting a good test score (extrinsic framing) (Vansteenkiste, Simons, Lens, Soenens, & Matos, 2005). Combining these investigations, it is evident that emphasizing valuing/intrinsic motivation in secondary school increases the targeted motivations and achievement. However, none of these investigations included or compared the effects of instruction with supports for self-efficacy, and consequently the instructional implications of the alignment hypothesis cannot be tested from these findings. It is well established that for secondary students, mastery-oriented instruction which appears to emphasize valuing increases academic achievement more than performance-oriented instruction which appears to emphasize self-efficacy (Maehr & Zusho, 2009). However, these instructional interventions have not explicitly attempted to influence motivations directly. Therefore, the crucial test of the alignment hypothesis has not been conducted for instruction in higher-order reading processes. At present, we have no direct comparison of whether academic achievement in higherorder reading processes is increased more by instruction targeting valuing/intrinsic motivation or by instruction targeting self-efficacy/expectations. For struggling students in middle school, instruction in cognitive reading strategies had a positive experimental effect on motivations of self-efficacy and intrinsic motivation, but not social motivation. Cognitive benefits were observed in students’ self-reported strategy use, employment of multiple-text documents, and problem solving (Cantrell, Almasi, Rintamaa, Pennington, & Buckman, 2014), although standardized reading test scores did not increase.

Promoting Multiple-Text Comprehension  •  397

In conclusion, available evidence is consistent with the implications of the alignment hypothesis for instructional and intervention practices. For primary students, reading instruction emphasizing self-efficacy in the context of teaching reading competencies will increase lower-order processes of reading more effectively than instruction that does not provide explicit self-efficacy support. Whether this effect is higher for struggling readers than on-grade readers is unknown. For students in elementary grades 3–5, balanced instruction emphasizing self-efficacy/expectancy and intrinsic motivation/valuing facilitates growth of those motivations concurrent with lower-order cognitive processing and multiple-text comprehension. However, instructional research on motivation support in secondary contexts has not investigated self-efficacy or expectations sufficiently to compare findings with investigations on intrinsic motivation or valuing. Consequently, further research is needed on this aspect of the alignment hypothesis.

RESEARCH NEEDS The cognitive–motivational alignment hypothesis in reading was initially confirmed by existing studies reported in this review. However, a more exhaustive review should be undertaken, with quantitative, meta-analytic comparisons. Beyond that, new research comprehensive of all the alignment constructs considered simultaneously would be beneficial. Within the hypothesis, proposed attributes of the cognitive and motivational constructs need empirical undergirding. Finally, the instructional implications of the alignment hypothesis should be investigated with experimental intervention designs.

REFERENCES Alao, S., & Guthrie, J. T. (1999). Predicting conceptual understanding with cognitive and motivational variables. Journal of Educational Research, 92, 243–253. Andreassen, R., & Bråten, I. (2010). Examining the prediction of reading comprehension of different multiple choice tests. Journal of Research in Reading, 33, 263–283. Anmarkrud, Ø., & Bråten, I. (2009). Motivation for reading comprehension. Learning and Individual Differences, 19, 252–256. Bandura, A., (2012). On the functional properties of perceived efficacy revisited. Journal of Management, 38, 9–44. Bandura, A., & Schunk, D. H. (1981). Cultivating competence, self-efficacy, and intrinsic interest through proximal self-motivation. Journal of Personality and Social Psychology, 41, 586–598. Bråten, I., Anmarkrud, Ø., Brandmo, C., & Strømsø, H. I. (2014). Developing and testing a model of direct and indirect relationships between individual differences, processing, and multiple-text comprehension. Learning and Instruction, 30, 9–24. Bråten, I., Braasch, J. L. G., & Salmerón, L. (in press). Reading multiple and non-traditional texts: New opportunities and new challenges. In E. Moje, P. Afflerbach, P. Enciso, & N. Lesaux (Eds.), Handbook of reading research (Vol. V). New York: Routledge. Bråten, I., Ferguson, L. E., Anmarkrud, Ø., & Strømsø, H. I. (2013). Prediction of learning and comprehension when adolescents read multiple texts: The roles of word-level processing, strategic approach and reading motivation. Reading and Writing, 26, 321–348. Cantrell, S., Almasi, J., Rintamaa, M., Pennington, J., & Buckman, D. (2014). The impact of supplemental instruction on low achieving adolescents’ reading engagement. The Journal of Educational Research, 107, 36–58.

398  •  John T. Guthrie Cartwright, K., Marshall, T., & Wray, E. (2016). A longitudinal study of the role of reading motivation in primary students’ reading comprehension: Implications for a less simple view of reading, Reading Psychology, 37, 55–91. Cho, E., Roberts, G., Capin, P., Roberts, G., Miciak, J., & Vaughn, S. (2015). Cognitive attributes, attention, and self-efficacy of adequate and inadequate responders in a fourth grade reading intervention. Learning Disabilities Research & Practice, 30, 159–170. Coddington, C. S., & Guthrie, J. T. (2009). Teacher and student perceptions of boys’ and girls’ reading motivation. Reading Psychology, 30, 225–249. Cox, K., & Guthrie, J. T. (2001). Motivational and cognitive contributions to children’s amount of reading. Contemporary Educational Psychology, 26, 116–131. Cromley, J., & Azevedo, R. (2007). Testing and refining the direct and indirect inferential mediation model of reading comprehension. Journal of Educational Psychology, 99, 311–325. Durik, A., Vida, M., & Eccles, J. (2006). Task values and ability beliefs as predictors of high school literacy choices: A developmental analysis. Journal of Educational Psychology, 98, 382–393. Eccles, J., & Wang, M. (2012). Part I commentary: So what is student engagement anyway? In S. Christensen, A. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp.  133–149). New York: Springer. Guthrie, J. T., & Klauda, S. L. (2014). Effects of classroom practices on reading comprehension, engagement, and motivation of adolescents. Reading Research Quarterly, 49, 387–416. Guthrie, J. T., & Klauda, S. L. (2016). Engagement and motivational processes in reading. In P. Afflerbach (Ed.), Handbook of individual differences in reading (pp. 41–54). New York: Routledge. Guthrie, J. T., Klauda, S. L., & Ho, A. (2013). Modeling the relationships among reading instruction, motiva­ tion, engagement, and achievement for adolescents. Reading Research Quarterly, 48, 9–26. Guthrie, J. T., McRae, A., Coddington, C., Klauda, S. L., Wigfield, A., & Barbosa, P. (2009). Impacts of compre­ hensive reading instruction on diverse outcomes of low- and high-achieving readers Journal of Learning Disabilities, 42, 195–214. Guthrie, J. T., McRae, A. C., & Klauda, S. L. (2007). Contributions of Concept-Oriented Reading Instruction to knowledge about interventions for motivations in reading. Educational Psychologist, 42, 237–250. Guthrie, J. T., Van Meter, P., Hancock, G. R., McCann, A., Anderson, E., & Alao, S. (1998). Does ConceptOriented Reading Instruction increase strategy-use and conceptual learning from text? Journal of Educational Psychology, 90, 261–278. Guthrie, J. T., Wigfield, A., Barbosa, P., Perencevich, K. C., Taboada, A., Davis, M. H., Scafiddi, N. T., & Tonks, S. (2004). Increasing reading comprehension and engagement through Concept-Oriented Reading Instruction. Journal of Educational Psychology, 96, 403–423. Guthrie, J. T., Wigfield, A., Metsala, J., & Cox, E. (1999). Motivational and cognitive predictors of text compre­ hension and reading amount. Scientific Studies of Reading, 3, 231–256. Ho, A. N., & Guthrie, J. T. (2013). Patterns of association among multiple motivations and aspects of achievement in reading. Reading Psychology, 34, 101–147. Hulleman, C. S., Godes, O., Hendricks, B. L., & Harackiewicz, J. M. (2010). Enhancing interest and performance with a utility value intervention. Journal of Educational Psychology, 102, 880–895. Jaeggi, S., Buschkuehl, M., Shah, P., & Jonides, J. (2014). The role of individual differences in cognitive training and transfer. Memory & Cognition, 42, 464–480. Jang, H. (2008). Supporting students’ motivation, engagement, and learning during an uninteresting activity. Journal of Educational Psychology, 100, 798–811. Klauda, S. L., & Guthrie, J. T. (2008). Relationships of three components of reading fluency to reading comprehension. Journal of Educational Psychology, 100, 310–321. Lai, S., Benjamin, R., Schwanenflugel, P., & Kuhn, M. (2014). The longitudinal relationship between reading fluency and reading comprehension skills in second-grade children. Reading & Writing Quarterly, 30, 116–138. Lazowski, R., & Hulleman, C. (2015). Motivational interventions in education: A metaanalytic review. Review of Educational Research, 20, 1–39. Leu, D. J., Kinzer, C.K., Coiro, J., Castek, J., & Henry, L. A. (2013). New literacies: A dual-level theory of the changing nature of literacy, instruction, and assessment. In D. E. Alvermann, N. J. Unrau, & R. B. Ruddell (Eds.), Theoretical models and processes of reading (6th ed., pp.  1150–1181). Newark, DE: International Reading Association.

Promoting Multiple-Text Comprehension  •  399 Liebfreund, M., & Conradi, K. (2016). Component skills affecting elementary students’ informational text comprehension. Reading and Writing, 29, 1141–1160. Logan, S., & Medford, E. (2011). Gender differences in the strength of association between motivation, competency beliefs and reading skill. Educational Research, 53, 85–94. Maehr, M., & Zusho, A. (2009). Achievement goal theory: The past, present, and future. In K. Wentzel & A. Wigfield (Eds.), Handbook of motivation at school (pp. 77–105). New York: Routledge. Marinak, B., Malloy, J., Gambrell, L., & Mazzoni, S. (2015). Me and my reading profile. The Reading Teacher, 69, 51–62. Martin, A., & Elliot, A. (2016). The role of personal best (PB) and dichotomous achievement goals in students’ academic motivation and engagement: A longitudinal investigation. Educational Psychology, 36, 1285–1302. McGeown, S., Putwain, D., Simpson, E., Boffey, E., Markham, J., & Vince, A. (2014). Predictors of adolescents’ academic motivation: Personality, self-efficacy and adolescents’ characteristics. Learning and Individual Differences, 32, 278–286. Mol, S., & Bus, A. (2011). To read or not to read: A meta-analysis of print exposure from infancy to early adulthood. Psychological Bulletin, 137, 267–296. Naeghel, J., Keer, H., Vansteenkiste, M., & Rosseel, Y. (2012). The relation between elementary students’ recreational and academic reading motivation, reading frequency, engagement, and comprehension: A selfdetermination theory perspective. Journal of Educational Psychology, 104, 1006–1021. Rosenzweig, E., & Wigfield, A. (2017). What if reading is easy but unimportant? How students’ patterns of affirming and undermining motivations for reading information texts predict different reading outcomes. Contemporary Educational Psychology, 48, 133–148. Ryan, R. M., & Deci, E. L. (2000). Intrinsic and extrinsic motivation: Classic definitions and new directions. Contemporary Educational Psychology, 25, 54–67. Ryan, R. M., & Deci, E. L. (2009). Promoting self-determined school engagement: Motivation, learning, and well-being. In K. Wenzel & A. Wigfield (Eds.), Handbook of motivation at school (pp. 171–195). New York: Routledge. Schaffner, E., & Schiefele, U. (2016). The contributions of intrinsic and extrinsic reading motivation to the development of reading competence over summer vacation. Reading Psychology, 37, 917–941. Schunk, D., & Mullen, C. (2009). Self-efficacy as an engaged learner. In K. R. Wenzel & A. Wigfield (Eds.), Handbook of motivation at school (pp. 291–237). New York: Routledge. Schunk, D., & Zimmerman, B. (2007). Influencing children’s self-efficacy and self-regulation of reading and writing through modeling. Reading & Writing Quarterly, 23, 7–25. Shell, D., Murphy, C., & Bruning, R. (1989). Self-efficacy and outcome expectancy mechanisms in reading and writing achievement. Journal of Educational Psychology, 81, 91–100. Shin, H., & Ryan, A. (2014). Early adolescent friendships and academic adjustment: Examining selection and influence processes with longitudinal social network analysis. Developmental Psychology, 50, 2462–2472. Solheim, O. (2011). The impact of reading self-efficacy and task value on reading comprehension scores in different item formats. Reading Psychology, 32, 1–27. Strømsø, H. I., & Bråten, I. (2009). Beliefs about knowledge and knowing and multiple-text comprehension among upper secondary students. Educational Psychology, 29, 425–445. Strømsø, H. I., Bråten, I., & Britt, M. A. (2010). Reading multiple texts about climate change: The relationship between memory for sources and text comprehension. Learning and Instruction, 18, 513–527. Stutz, F., Schaffner, E., & Schiefele, U. (2016). Relations among reading motivation, reading amount, and reading comprehension in the early elementary grades. Learning and Individual Differences, 45, 101–113. Tannenbaum, K., Torgesen, J., & Wagner, R. (2006). Relationships between word knowledge and reading comprehension in third grade children. Scientific Studies of Reading, 10, 381–398. Tiffin-Richards, S., & Schroeder, S. (2015). The component processes of reading comprehension in adolescents. Learning and Individual Differences, 42, 1–9. Vansteenkiste, M., Simons, J., Lens, W., Soenens, B., & Matos, L. (2005). Examining the motivational impact of intrinsic versus extrinsic goal framing and autonomy-supportive versus internally controlling communication style on early adolescents’ academic achievement. Child Development, 76, 483–501. Walczyk, J. J., Wei, M., Griffith-Ross, D. A., Cooper, A. L., Zha, P., & Goubert, S. E. (2007). Development of the interplay between automatic processes and cognitive resources in reading. Journal of Educational Psychology, 99, 867–887.

400  •  John T. Guthrie Weber, M., Wagner, L., & Ruch, W. (2016). Positive feelings at school: On the relationship between students’ character strengths, school-related affect, and school functioning. Journal of Happiness Studies, 17, 341–355. Wigfield, A., & Eccles, J. S. (2000). Expectancy value theory of achievement motivation. Contemporary Educational Psychology, 25, 68–81. Wigfield, A., & Eccles, J. S. (2002). The development of competence beliefs, expectancies for success, and achievement values from childhood through adolescence. In A. Wigfield & J. S. Eccles (Eds.), Development of achievement motivation (pp. 91–120). San Diego, CA: Academic Press. Wigfield, A., Guthrie, J. T., Perencevich, K. C., Taboada, A., Klauda, S. L., McRae, A., et al. (2008). The role of reading engagement in mediating effects of reading comprehension instruction on reading outcomes. Psychology in the Schools, 45, 432–445. Wolters, C., Denton, C., York, M., & Francis, D. (2014). Adolescents’ motivation for reading: Group differences and relation to standardized achievement. Reading and Writing, 27, 503–533.

23

INSTRUCTION TO PROMOTE INFORMATION PROBLEM SOLVING ON THE INTERNET IN PRIMARY AND SECONDARY EDUCATION A Systematic Literature Review Saskia Brand-Gruwel welten institute, research centre for learning, teaching, and technology open university of the netherlands

Johan L.H. van Strien faculty of social and behavioural sciences, utrecht university, the netherlands

INTRODUCTION Students are increasingly confronted with tasks that present an information-based problem. An information-based problem can be defined as a problem that only can be solved by gathering information. When facing such problems, students most likely conduct a search on the Internet. They need to find useful information from reliable sources and process and integrate this information in order to formulate a complete and correct solution. Effectively and efficiently solving such information-based problems requires knowledge of tools such as search engines or literature databases (Brand-Gruwel, Wopereis, & Vermetten, 2005; Rosman, Mayer, & Krampen, 2016) and domain-specific knowledge (Lucassen & Schraagen, 2013; Salmerón, Kammerer, & García-Carrión, 2013). This must be combined with a critical attitude to correctly judge the relevance and quality of information sources (Kammerer, Bråten, Gerjets, & Strømsø, 2013). Given the ease with which students browse on the Internet, it is tempting to believe they automatically develop the skills needed to solve information-based problems without any explicit instruction. However, this is not the case (see Brand-Gruwel & Gerjets, 2008;

401

402  •  Brand-Gruwel and van Strien

Kirschner & Van Merriënboer, 2013). While some aspects of information literacy, such as operational skills (Van Deursen & Van Dijk, 2009), might indeed develop quickly, research shows that information-problem solving (IPS) is underdeveloped in students of all ages, especially the skill of evaluating and selecting trustworthy sources and information (Frerejean, Van Strien, Kirschner, & Brand-Gruwel, 2016; Van Deursen & Van Diepen, 2013; Walraven, Brand-Gruwel, & Boshuizen, 2009). This is also in line with research on sourcing. Sourcing concerns explicit attention to and use of source information to interpret the trustworthiness of documents (Strømsø & Bråten, 2014). Research on spontaneous sourcing during multiple-document reading indicates that both high-school students and undergraduates show low interest in source information and seldom consider such information helpful for interpretative or evaluative purposes (e.g., Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012; Walraven et al., 2009). Skills concerning the evaluation of sources and information (i.e., sourcing), which are required for solving information-based problems, do not develop sufficiently from mere exposure to online search tasks, making instruction necessary. This chapter presents a systematic review of instructional materials and programs that intend to foster students’ IPS skills, especially the skill of evaluating the trustworthiness of sources and information. Although skills are underdeveloped in students of all ages, this review focuses on students in primary and secondary education. This is because research conducted in primary and secondary education showed that students hardly evaluated sources and information and that instruction for these children is highly needed (Walraven et al., 2009). Using Merrill’s (2009) first principles of instruction, we focus our review on which instructional design principles are used in developing effective instructional materials or programs. In the next section, we elaborate on the construct of information-problem solving and discuss the instructional design principles presented by Merrill (2009).

INFORMATION-PROBLEM SOLVING There are different prominent models describing the skills needed to solve information problems or the process of IPS (Eisenberg & Berkowitz, 1990; Kuhlthau, 1993; Stripling & Pitts, 1988). However, these models are not primarily based on situations in which students use the Internet to search for information. The importance of judging the trustworthiness of sources and information on the Internet has increased over the years. At the same time, the use of search engines and the ability to generate search terms are specific for searching information on the Internet. People can post all kinds of information on the Internet, and facts and opinions alternate in social media. These aspects are addressed in the Information-Problem Solving on the Internet (IPS-I) model of Brand-Gruwel, Wopereis, and Vermetten (2005) and further developed by Frerejean, Velthorst, Van Strien, Kirschner, and Brand-Gruwel (under review). The IPS-I model describes the process and the skills that are involved in solving information-based problems. When confronted with an information-based problem, the student has to define the problem, determine which information is needed, and formulate a clear and concise question. This question often contains the core concepts that can subsequently be used as search terms in the search engine. Critical evaluation of each result on the search engine results page (SERP) by, for instance, looking at the URL and the summary is necessary to select relevant and reliable sources. By looking at characteristics such as the topic, the author, and the publication date, the student determines if the source is trustworthy and useful. Sources will be selected and the corresponding

Information Problem Solving  •  403

information will be processed more deeply, compared with learners’ own knowledge and information from other sources, and further judged with respect to trustworthiness. Contrasting and comparing information from different sources, as well as drawing on prior knowledge, help learners make further decisions concerning trustworthiness. This process of contrasting and comparing will result in what Rouet and Britt (2011) called an Intertext Model. How this process of reading multiple information sources works has been studied in much multiple-source comprehension research (see Bråten, Stadtler, & Salmerón, in press). When sufficient information has been processed, the student integrates the studied information to formulate an answer to the question and presents the solution to the problem. The regulation of the process is of importance, because orientation, monitoring, and steering of the process ensure that it becomes iterative and more effective and efficient. Figure 23.1 gives an overview of the skills elaborated in the IPS-I model by Brand-Gruwel et al. (2005).

DESIGN PRINCIPLES TO FOSTER INFORMATION-PROBLEM SOLVING Research has repeatedly shown that these IPS skills on the Internet are underdeveloped at all levels of education (Walraven et  al., 2009), as well as overestimated by students (Frerejean et al., 2016; Kirschner & Van Merriënboer, 2013; Rosman, Mayer, & Krampen, 2014). This underlines the necessity to provide IPS instruction. However, while this necessity is often acknowledged, schools and teachers are poorly equipped to design effective IPS instruction. To design instructional materials to foster IPS skills, design principles can help make deliberate design choices. Given that IPS can be characterized

Figure 23.1  Five-Step Systematic Approach to Information-Problem Solving (Frerejean et al., under review; based on Brand-Gruwel et al., 2005).

404  •  Brand-Gruwel and van Strien

as a complex generic or higher-order skill, instructional measures to support this kind of skill should be called upon. Contemporary instructional design theories, such as the four-component instructional design (4C-ID) (Van Merriënboer & Kirschner, 2013) and the first principles of instructional design (Merrill, 2009), can therefore be very helpful. Merrill distinguishes five principles that will be used to analyze the interventions studied in this literature review. Concisely, they can be described as follows: First, learning is promoted when learners are engaged in solving real-world problems (task-centered principle). This principle stresses the importance of using whole tasks in authentic settings, because this promotes transfer and motivates students (Van Merriënboer & Kirschner, 2013). Whole tasks refer to tasks where skills are learned and practiced in authentic settings and to the full extent. For IPS skills this means that all sub-skills are addressed in the task, ranging from defining the problem to the completing the product. This is in contrast to part tasks in which, for instance, only generating search terms or formulating search questions is practiced. Second, learning is promoted when existing knowledge is activated as a foundation for new knowledge (activation principle). It is well known that activating prior knowledge makes students better able to link learned concepts to already existing knowledge (e.g., Gurlitt & Renkl, 2010). Third, learning is promoted when new knowledge is demonstrated for the learner (demonstration principle). Different studies have shown that worked-out examples and modeling are beneficial to learning (e.g., Van Gog, Paas, & Van Merriënboer, 2008). Fourth, learning is promoted when new knowledge is applied by the learner (application principle). Students need to apply skills and learn during task performance. Support and fading of support are therefore important. Fifth, learning is promoted when new knowledge is integrated into the learner’s world (integration principle). This aspect is important to foster transfer of learned skills. Transfer can be seen as the ability to apply skills in contexts other than the context in which the skills are learned. Applying the skills in different contexts and using different authentic tasks in the learning environment are of importance (Van Merriënboer & Kirschner, 2013). These five principles are widely studied, used, and accepted in education. Therefore, they can also be considered an adequate framework when analyzing specific instructions to foster IPS skills. The research questions we address in this chapter are: What kinds of interventions have been conducted that aim at promoting the component skills of ISP-I in primary and secondary education, and what kinds of results have been achieved with these interventions? To what extent have the interventions taken Merrill’s first principles of instructional design into account? To answer these questions a review study has been conducted.

METHOD Criteria for Inclusion Aiming at a systematic review, the following criteria were used for inclusion of studies: (1) Empirical research focusing on investigating the effects of instruction to foster IPS skills among students, (2) articles in which an overview was given of the problem areas and skills mastered by students, (3) articles dated after 2000 (because of the popularity of the Internet), (4) studies must be conducted in primary and secondary education, and (5) studies published in English in peer-reviewed journals.

Information Problem Solving  •  405

Search Strategy A systematic literature search was conducted within the academic databases Academic Search Elite, PsycINFO, ERIC, and Psychology and Behavioral Sciences Collection. Two blocks of search terms were used and combined. The first block comprised the terms information-problem solving, sourcing, and source evaluation. The second block comprised the terms instruction, training, and intervention. The references of the articles found were used to search for new articles. The articles were sorted according to grade and only studies conducted in primary and secondary education were included. Identification of Relevant Studies The initial search yielded 39 unique results, which were scanned for possible inclusion based on the formulated criteria. Fifteen articles matched those criteria. Subsequent scanning of the reference lists of the 15 articles yielded eight additional articles. Given these 23 articles, the next step was to select the studies conducted in primary and secondary education (up to grade 12). As a result, 13 articles were included in our systematic literature review (see references with an asterisk in the reference list). Analyses and Description The 13 articles were scanned to get an overview of their characteristics in terms of (1) type of education, (2) if the instruction was embedded in the school’s curriculum or independent of it (i.e., a stand-alone), (3) if the intervention was classroom-based or computer-based (i.e., delivered online), (4) the type of skill(s) included as dependent variable(s), and (5) if there was a focus on transfer. Appendix 23.1 gives an overview of all the studies including the above-mentioned aspects. Four articles focused on elementary education (grades 4, 5, and 6); one article describing two studies focused in one of its experiments on grade 6 (elementary school) and in the other experiment on grades 7 and 8 (secondary education). Eight studies were conducted in secondary education (grades 7–12). When an article described more than one study or experiment, we always focused on the last one, because the instruction was refined after the first study and the last study tested the effects of an updated version. The articles by Britt and Aglinskas (2002), Gerjets and Hellenthal-Schorr (2008), and De Vries, Van der Meij, and Lazonder (2008) included more than one study. Britt and Aglinskas revised the Sourcer’s Apprentice. Gerjets and Hellenthal-Schorr revised the CIS-WEB and, because of the small number of participants, their first study could be seen as a pilot. De Vries and colleagues used a design-based approach and made substantial revisions after the first cycle.

RESULTS In addressing the research questions, we make a distinction between studies conducted in elementary/primary and secondary education. We first present an overview of the studies by describing their dependent variables and the effects of the interventions. Afterwards, we discuss the interventions in light of Merrill’s principles.

406  •  Brand-Gruwel and van Strien

Primary Education Four studies were conducted in primary education. The study of Kroustallaki, Kokkinaki, Sideridis, and Simos (2015) was conducted in 5th and 6th grade, used an experimental and a control group, and measured the effects of a stand-alone, classroom-based intervention on students’ query formulation, information selection, credibility evaluation, affect, and achievement goals. The focus was on searching and selecting sources and information. The intervention had a positive effect on all IPS skills measured. Also, a positive effect was found on the affective variables. The study of Kuiper, Volman, and Terwel (2008) used a classroom-based, multiple-case design and was conducted in 5th grade. Effects of embedded training on knowledge of the subject of healthy food and on knowledge of searching for and evaluating web information were studied. Knowledge of searching for and evaluating web information was measured with a questionnaire. The lessons were observed and field notes were taken, which led to rich data. The effect of the study showed that students from two of the four participating schools improved significantly on self-reported searching and evaluation. Furthermore, it was concluded that students showed knowledge gain with regard to both the subject of healthy food and web skills. However, most students appeared to be inconsistent web users, who did not always act upon their knowledge. The study of Macedo-Rouet, Braasch, Britt, and Rouet (2013) was conducted in 4th and 5th grade and was an embedded classroom-based intervention. Effects of this short intervention on students’ number of correct answers on source and knowledge questions were studied, taking students’ reading ability into account. The goal of the intervention was to encourage students to identify the sources, establish links between sources and content, and assess the competence of each source with respect to the topic of the text. The focus was on both sourcing and comprehension. Effects showed that especially the less skilled readers benefitted from the intervention. The study of De Vries et al. (2008) was a design experiment that involved embedded computer-based training. Fifth and sixth graders participated in the study, which investigated the effects on ownership, interpretation and personalization, and adaptation. Ownership concerned ownership over the research question. Interpretation and personalization concerned the interpretation of new information and creation of links to prior knowledge. Adaptation indicated that students were able to translate the information into their own words. These concepts were related to formulating the problem and search question, the search and selection of sources and information, and the possessing of information. It was found that ownership was established, and that interpretation and personalization, measured by correctness of the answers, were improved. However, because students merely cut and pasted the information, adaptation was not observed. Characteristics of the Interventions in Terms of Merrill’s First Principles Table 23.1 gives an overview of the four studies and the principles that were used, based on descriptions included in the introduction and method sections of the articles. The Table displays whether each principle was used and provides some elaboration. Appendix 23.1 further describes each intervention. The study of Kroustallaki et al. (2015) was consistent with almost all the principles for instructional design, as was the study by Kuiper et al. (2008). The effects of these

Information Problem Solving  •  407 Table 23.1  Merrill’s Principles Used in the Intervention Conducted in Primary Education. Study

Duration training

First Principles of Merrill Task-centered Activation

Kroustallaki 3 × 45 min Students et al. (2015) worked on inquiry tasks

Stimulated questions to relate to students’ experiences Kuiper et al. 8 × 1.5 / 2 All tasks were Each lesson (2008) hours on the topic of started with a discussion healthy food. It is not clear if in which students’ all tasks were knowledge whole tasks, but in the last and skills lessons whole were addressed real-world tasks were used Macedo30 minutes Students Rouet et al. read texts (2013) with different opposing views

Demonstration Application Students practiced the tasks focusing on the modeled skills Students Teachers worked on instructed on the web skills in the tasks in pairs the beginning of each lesson, and modeled skills during the lesson

Integration

Modeled and explained the skills

A sample text was read out loud to the students; two types of questions were discussed

De Vries 3 × 2 hours Students et al. (2008) worked in teams on an inquiry project

The last lessons were aiming at integration of the learned skills, but did not require transfer

Students read texts and answered questions concerning source and knowledge and discussed the answers Students worked on their project using a webbased portal

studies were also promising, with effects found for knowledge and skills concerning searching and selecting sources and information. The intervention of Macedo-Rouet et al. (2013) lasted only 30 minutes and focused on one specific aspect (sourcing). This study had an effect for less skilled readers. The study of De Vries et al. (2008) focused on skills application. One of the findings of this study was that students continued to cut and paste information to formulate their answers. All studies focus on selection of trustworthy information and sources, although the studies of De Vries et al. (2008) and Kuiper et al. (2008) also addressed the other IPS skills. Secondary Education Nine interventions were conducted in secondary education. Gerjets and HellenthalSchorr (2008) conducted an experiment in 7th and 8th grade, evaluating the effects of a web-based training program called CIS-WEB on students’ declarative knowledge and search performance. The intervention focused on insight into the structure of information problems, as well as on search and selection of sources and information.

408  •  Brand-Gruwel and van Strien

The experiment used a pretest-posttest control group design. The intervention was considered successful because experimental students’ knowledge and search performance improved relative to controls. The intervention of Argelagós and Pifarré (2012) was conducted in 7th and 8th grade and lasted two years. It involved web-based training targeting the skills included in the IPS-I model of Brand-Gruwel et al. (2005). The study used a pretest-posttest control group design. Results showed that students in the control group outperformed those in the experimental group at pretest. The most salient result concerned the skill ‘defining the problem’, with repeated measures showing a significant difference between the two conditions. Although the students in the control condition scored higher at pretest, the gain from pre- to posttest was significantly larger in the experimental group. Furthermore, after training, the students in the experimental condition displayed significantly better task performance than those in the control condition. Raes, Schellens, De Wever, and Vanderhoven (2012) instructed students in 9th and 10th grade in science content and IPS skills (based on Brand-Gruwel et al., 2005) by combining a web-based environment called WISE with teacher scaffolds. Effects on domain-specific knowledge and metacognitive awareness were measured. The latter was measured by asking students to report their perceptions about their use of metacognitive and strategic actions after accomplishing an IPS task. Effects were positive, with especially the combination of the web-based environment and teacher scaffolding during task performance leading to high performance in terms of knowledge gain and metacognitive awareness. The study of Mason, Junyent, and Tornatora (2014) focused on students’ evaluation of sources and comprised a short-term intervention in which students were instructed in how to evaluate sources. One group of 9th-grade students participated in the SEEK intervention designed by Wiley et al. (2009) and one group of students just performed the inquiry tasks. It was found that students in the intervention condition made fewer visits to the least reliable sites, rank-ordered more accurately the most reliable sites, and justified their rank-ordering of the most and least reliable sources. Still, they were not able to produce an overall more accurate rank-ordering of all the sites that were used (students were asked to rank the sites on reliability and the accuracy was measured). However, students in the experimental condition produced an overall more accurate rank-ordering than those in the control condition on a transfer task. In the Braasch, Bråten, Strømsø, Anmarkrud, and Ferguson (2013) intervention, students in the last year of upper secondary school were instructed in judging multiple documents using sophisticated evaluation strategies focusing on source features. The intervention presented contrasting cases illustrating such sophisticated strategies in contrast to less useful strategies focusing on content alone. The study showed that students in the intervention condition included more scientific concepts from the more useful documents, discriminated better between more and less useful documents, and were far more likely to use source features as a basis for this discrimination compared to control students. The study of Walraven, Brand-Gruwel, and Boshuizen (2013) used a pretestposttest control group design. Students in 9th grade participated in embedded training consisting of 15 lessons in which they learned how to evaluate sources and information found on the Internet. Effects on evaluation of a SERP, sources (websites), and

Information Problem Solving  •  409

information were determined in the contexts of history (lessons were embedded in history class) and biology (in order to measure transfer). It was found that students in the experimental group scored significantly better than the control students at posttest. Furthermore, scores in regard to the evaluation of sources improved over time in the experimental condition, but not in the control condition. Also, transfer effects were found. Walraven, Brand-Gruwel, and Boshuizen (2010) studied two different interventions aiming at teaching ninth graders IPS skills, especially evaluation of sources and information. To foster transfer, both interventions focused on the generic aspects of the skills. The two interventions, termed the ‘high-road intervention’ and the ‘rich representation intervention’, respectively, were embedded in 15 lessons. Results showed that both interventions led to improvement in the evaluation of websites and information. In general, the same results were found on the transfer tasks. In the study of Caviglia and Delfino (2016), intervention students were trained in IPS skills, and the training was based on inquiry-based learning. The final evaluation showed that students (14 and 15 years old) who participated in the intervention scored significantly higher on the PISA literacy tests. Also, these students developed a positive attitude toward studying at school, and the results were promising in terms of school performance. Thus, the intervention students continued to the next grade whereas only 20% of the control students did so (which was similar to the country average). Finally, the study of Britt and Aglinskas (2002) was conducted in 11th grade, and students were instructed in how to evaluate sources and information. In a quasiexperimental setting, students in the experimental condition used the computer-based Sourcer’s Apprentice (SA) environment to learn about sourcing. The results showed that the SA group improved from pretest to posttest, whereas the control group answered fewer items correctly on the posttest. Characteristics of the Interventions in Terms of Merrill’s First Principles The interventions conducted in secondary education are described in Appendix 23.1. Table 23.2 gives an overview of these studies in terms of Merrill’s first principles of instructional design. The intervention studies addressed different IPS skills, but all focused on the evaluation and selection of trustworthy information and sources. Some studies took a whole-task approach and studied the effects on the constituent skills involved. All interventions involved instructional design principles described by Merrill. Although the duration of the training differed among the interventions, they were generally successful in promoting the skills that they targeted. Of note is the fact that even brief training seemed to make a difference. All studies except Gerjets and Hellenthal-Schorr used whole tasks in more or less authentic settings, or inquiry tasks. Furthermore, when applying the skills during training, students were well supported by worksheets, visualizations, feedback, discussions, etc. In most studies, prior knowledge also was activated. However, the skills were demonstrated only in a few studies. Also, only a few studies measured transfer effects. These effects were in general positive.

6 months, Whole tasks were used, 50 hours connection to real world

Two days Training involved situated problem solving in an authentic context

Caviglia & Delfino (2016)

Britt & Aglinskas (2002)

Both interventions used whole tasks

Students were asked about the strategies they used when evaluating sources Students’ knowledge was activated at the beginning of the lessons Students’ prior knowledge was activated in both interventions

Integration

Transfer was measured, and authentic contexts were used

Interventions were set up to teach the generic aspects of the skills to stimulate transfer

Transfer was measured

After reading the instructional Transfer was measured material, students applied the skills during task performance Students got an assignment in which Transfer was measured they applied the learned strategies

While carrying out the inquiry activities, students were supported

While performing the web quests, students were supported by worksheets and prompts

When learning the skills, worksheets and questions with feedback were provided

Application

When applying the skills, students were supported by process worksheets High-road intervention: students were supported by process worksheets. Rich representation intervention: students were supported by visualizations Students practiced with a focus Not clear if on searching and evaluation. instruction Discussions were an important part involved of the intervention demonstration Skills were When applying the skills, students modeled by using were supported in different ways in examples using the formulated principles

Students studied two protocols illustrating source evaluation

Tasks involved judging source evaluation strategies in multiple-document comprehension Different whole IPS tasks

Braasch One et al. (2013) session of 60 minutes Walraven 15 lessons et al. (2013) of 50 minutes Walraven 15 lessons et al. (2010) of 50 minutes

Demonstration Worked-out examples and visualizations were used

12 x 45 min

Gerjets & HellenthalSchorr (2008) Argelagós & Pifarré (2012)

Activation

Prior knowledge was recapitulated at the beginning of each module 15 web Each web quest comprised Students were asked quests/in a whole task to activate prior total 60 knowledge in the web hours environments Raes et al. Multiple Inquiry-based activities One of the scaffolds (2012) sessions concerned the activation of prior knowledge Mason One Inquiry task et al. (2014) session

Duration First Principles of Merrill training Task-centered

Study

Table 23.2  Merrill’s Principles Used in the Interventions Conducted in Secondary Education.

Information Problem Solving  •  411

DISCUSSION In this chapter, we reviewed studies in which students in primary and secondary education were trained in IPS, specifically in skills concerning the evaluation of sources and information. Our research questions were: What kinds of interventions have been conducted that aim at promoting the component skills of ISP-I in primary and secondary education, and what kinds of results have been achieved with these interventions? To what extent have the interventions taken Merrill’s first principles of instructional design into account? Considering the growth of social media and the amount of information available on the Internet, it is important to constantly check facts, read critically, and evaluate the trustworthiness of sources and information when learning from multiple documents. And, as already stated, the IPS skills important to deal with this overload of information are often underdeveloped in children in primary and secondary education (Walraven et al., 2009). It can be concluded that the interventions we reviewed were generally successful and resulted in gains in the knowledge and skills that were targeted. Given their importance in daily life, the underdevelopment of such skills, and the fact that they can be improved through instruction, suggest that further efforts to teach students IPS skills, especially skills concerning the evaluation of sources and information found on the Internet, are needed. Another observation is that some studies focused on all the constituent IPS skills using a whole-task approach and that other studies just focused on skills concerning the evaluation of sources and information. An important issue concerns how these different approaches to designing instruction can be integrated to foster students’ learning. These different approaches stem from different lines of research. In particular, it seems important to bring together research on IPS and research on multiple-source discourse comprehension (Goldman & Brand-Gruwel, in press). Regarding design principles (Merrill, 2009), all studies, albeit to different extents, used a task-centered approach in which they presented tasks that students face in daily (school) life. Most studies used an inquiry-based approach where students worked on projects involving different aspects of IPS. Using a task-centered approach is of importance because the more authentic the task, the more transfer can be expected. The principle of activation was also used in most interventions. Activating prior knowledge is of importance because new knowledge can then be integrated into the already existing schemata (Alvermann, Smith, & Readence, 1985). However, few studies seemed to use the demonstration principle. Demonstrating the process by using worked-out examples or modeling examples can help students acquire new skills because the underlying cognitive processes are unraveled and exposed (e.g., Van Gog et al., 2008). The principle of applying the learned skills was used in all the interventions. Thus, during application, students were supported by prompts, process worksheets, scaffolds, discussions, or cognitive feedback, and in some cases this support faded. Such support can help students build on their knowledge base and internalize a systematic problem-solving approach. Support and feedback are powerful instructional means to support learning (Hattie & Timperley, 2007). However, the results of this review and the interpretation of which aspects are most effective should be made with caution, because effect sizes were not taken into consideration.

412  •  Brand-Gruwel and van Strien

The interventions we reviewed had different durations. Some were short and others were embedded in the curriculum and lasted longer. Although all interventions to some extent were successful, the transfer of skills was seldom directly targeted in the interventions and seldom assessed. However, it could be argued that the trained skills are generic in nature. According to more recent instructional design theories (e.g., Van Merriënboer & Kirschner, 2013), it is important to train skills in different contexts to make sure that procedural knowledge and a systematic approach to problem solving are learned and internalized. Future research should not only focus on the design of instruction in IPS, but also address the transfer of skills and try to foster students’ critical thinking skills. Moreover, Bråten, Britt, Strømsø, and Rouet (2011) discussed the importance of students’ beliefs about knowledge and knowing (i.e., epistemic beliefs) for their evaluation of sources and information, and Van Strien, Kammerer, Brand-Gruwel, and Boshuizen (2016) recently showed that students’ attitudes can impact their evaluation behavior. Studies focusing on instruction should also take such individual difference factors into account. Furthermore, task characteristics should be considered. For example, finding simple facts requires different skills than solving more complex problems. Finally, this review has highlighted that instruction in IPS skills, especially the skill to evaluate sources and information, is essential in the 21st century. Therefore, teachers and curriculum designers need to seriously consider how to embed such instruction in daily school practices to support children’s development toward becoming critical citizens.

REFERENCES Alvermann, D. E., Smith, L. C., & Readence J. E. (1985). Prior knowledge and the comprehension of compatible and incompatible text. Reading Research Quarterly, 20, 420–436 Argelagós, E., & Pifarré, M. (2012). Improving information problem solving skills in secondary education through embedded instruction. Computers in Human Behavior, 28, 515–526. Braasch, J. L. G., Bråten, I., Strømsø, H. I., Anmarkrud, Ø., & Ferguson, L. E. (2013). Promoting secondary school students’ evaluation of source features of multiple documents. Contemporary Educational Psychology, 38, 180–195. Brand-Gruwel, S., & Gerjets, P. (2008). Instructional support for enhancing students’ information problem solving ability. Computers in Human Behavior, 24, 615–622. Brand-Gruwel, S., Wopereis, I., & Vermetten, Y. (2005). Information problem solving by experts and novices: Analysis of a complex cognitive skill. Computers in Human Behavior, 21, 487–508. Bråten, I., Britt, M. A., Strømsø, H. I., & Rouet, J.-F. (2011). The role of epistemic beliefs in the comprehension of multiple expository texts: Towards an integrated model. Educational Psychologist, 46, 48–70.508. Bråten, I., Stadtler, M., & Salmerón, L. (in press). The role of sourcing in discourse comprehension. In M. F. Schober, M. A. Britt, & D. N. Rapp (Eds.), Handbook of discourse processes (2nd ed.). New York: Routledge. Britt, M. A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20, 485–522. Caviglia F., & Delfino M. (2016). Foundational skills and dispositions for learning: An experience with Information Problem Solving on the Web. Technology, Pedagogy and Education, 25, 487–512. De Vries, B., Van der Meij, H., & Lazonder, A. W. (2008). Supporting reflective Web searching in elementary schools. Computers in Human Behavior, 24, 649–666. Eisenberg, M. B., & Berkowitz, R. E. (1990). Information problem-solving: The big six skills approach to library and information skills instruction. Norwood, NJ: Ablex. Frerejean, J., van Strien, J. L. H., Kirschner, P. A., & Brand-Gruwel, S. (2016). Completion strategy or emphasis manipulation? Task support for teaching information problem solving. Computers in Human Behavior, 62, 90–104.

Information Problem Solving  •  413 Frerejean, J., Velthorst, G., Van Strien, J. L. H., Kirschner, P. A., & Brand-Gruwel, S. (under review). Embedded instruction for information problem solving: Effects of a whole task approach. Gerjets, P., & Hellenthal-Schorr, T. (2008). Competent information search in the World Wide Web: Development and evaluation of a web training for pupils. Computers in Human Behavior, 24, 693–715. Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381. Goldman, S., & Brand-Gruwel, S. (in press). Learning from multiple sources in a digital society. In F. Fischer, C. E. Hmelo-Silver, P. Reimann, & S. R. Goldman (Eds.), International handbook of the learning sciences. New York: Routledge. Gurlitt, J., & Renkl, A. (2010). Prior knowledge activation: How different concept mapping tasks lead to substantial differences in cognitive processes, learning outcomes and perceived self-efficacy. Instructional Science, 38, 417–433. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77, 81–112 Kammerer, Y., Bråten, I., Gerjets, P., & Strømsø, H. I. (2013). The role of Internet-specific epistemic beliefs in laypersons’ source evaluations and decisions during Web search on a medical issue. Computers in Human Behavior, 29, 1193–1203. Kirschner, P. A., & Van Merriënboer, J. J. G. (2013). Do learners really know best? Urban legends in education. Educational Psychologist, 48, 169–183. Kroustallaki, D., Kokkinaki, T., Sideridis, G. D., & Simos, P. G. (2015). Exploring students’ affect and achievement goals in the context of an intervention to improve web searching skills. Computers in Human Behavior, 49, 156–170. Kuhlthau, C. (1993). Seeking meaning: A process approach to library and information services. Norwood, NJ: Ablex. Kuiper, E., Volman, M., & Terwel, J. (2008). Integrating critical Web skills and content knowledge: Development and evaluation of a 5th grade educational program. Computers in Human Behavior, 24, 666–692. Lucassen, T., & Schraagen, J. M. (2013). The influence of source cues and topic familiarity on credibility evaluation. Computers in Human Behavior, 29, 1387–1392. Macedo-Rouet, M., Braasch, J. L., Britt, M. A., & Rouet, J.-F. (2013). Teaching fourth and fifth graders to evaluate information sources during text comprehension. Cognition and Instruction, 31, 204–226. Mason L., Junyent A. A., & Tornatora M. C. (2014). Epistemic evaluation and comprehension of web-source information on controversial science-related topics: Effects of a short-term instructional intervention. Computers & Education 76, 143–157. Merrill, M. D. (2009). First principles of instruction. In C. M. Reigeluth & A. Carr (Eds.), Instructional design theories and models: Building a common knowledge base (Vol. III). New York: Routledge. Raes, A., Schellens, T., De Wever, B., & Vanderhoven, E. (2012). Scaffolding information problem solving in web-based collaborative inquiry learning. Computers & Education, 59, 82–94. Rosman, T., Mayer, A.-K., & Krampen, G. (2014). Combining self-assessments and achievement tests in information literacy assessment: Empirical results and recommendations for practice. Assessment & Evaluation in Higher Education, 40, 1–15. Rosman, T., Mayer, A.-K., & Krampen, G. (2016). On the pitfalls of bibliographic database searching: Comparing successful and less successful users. Behaviour & Information Technology, 35, 106–117. Rouet, J.-F., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Relevance instructions and goal-focusing in text learning (pp. 19–52). Greenwich, CT: Information Age. Salmerón, L., Kammerer, Y., & García-Carrión, P. (2013). Searching the Web for conflicting topics: Page and user factors. Computers in Human Behavior, 29, 2161–2171. Stripling, B., & Pitts, J. (1988). Brainstorms and blueprints: Teaching library research as a thinking process. Littleton, CO: Libraries Unlimited. Strømsø, H. I., & Bråten, I. (2014). Students’ sourcing while reading and writing from multiple documents. Nordic Journal of Digital Literacy, 9, 92–111. Van Deursen, A. J. A. M., & van Diepen, S. (2013). Information and strategic Internet skills of secondary students: A performance test. Computers & Education, 63, 218–226. Van Deursen, A. J. A. M., & van Dijk, J. A. G. M. (2009). Using the Internet: Skill related problems in users’ online behavior. Interacting with Computers, 21, 393–402.

414  •  Brand-Gruwel and van Strien Van Gog, T., Paas, F., & Van Merriënboer, J. J. G. (2008). Effects of studying sequences of process-oriented and product-oriented worked examples on troubleshooting transfer efficiency. Learning and Instruction, 18, 211–222. Van Merriënboer, J. J. G., & Kirschner, P. A. (2013). Ten steps to complex learning: A systematic approach to four-component instructional design (2nd ed.). New York: Routledge. Van Strien, J. L. H., Kammerer, Y., Brand-Gruwel, S., & Boshuizen, H. P. A. (2016). How attitude strength biases information processing and evaluation on the web. Computers in Human Behavior, 60, 245–252. Walraven, A., Brand-Gruwel, S., & Boshuizen, H. P. A. (2009). How students evaluate information and sources when searching the World Wide Web for information. Computers & Education, 52, 234–246. Walraven, A., Brand-Gruwel, S., & Boshuizen. H. P. A. (2010). Fostering transfer of websearchers’ evaluation skills: A field test of two transfer theories. Computers in Human Behavior, 26, 716–728. Walraven, A., Brand-Gruwel, S., & Boshuizen, H. P. A. (2013). Fostering students’ evaluation behaviour while searching the internet. Instructional Science, 41, 125–146. Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I., & Hemmerich, J. (2009). Source evaluation, comprehension, and learning in Internet science inquiry tasks. American Educational Research Journal, 46, 1060–1160.

Dependent variables

•• Quasiexperimental design (experimental and control group) •• 5th and 6th graders •• No transfer measured •• N = 96

•• Multiple case study design •• 5th graders •• No transfer measured •• N = 82

•• Query formulation •• Information selection •• Credibility evaluation •• Affect •• Achievement goals

•• Knowledge of the subject of healthy food •• Knowledge of searching and evaluating web information

Kuiper, Volman, & Terwel (2008)

Design

Kroustallaki, Kokkinaki, Sideridis, & Simon (2015)

Primary Education / Elementary school

Author

•• Embedded, classroom-based program on the subject, ‘healthy food’ •• Characteristics: all students practiced the same exercises at the same time and a great focus on class discussion, skills instruction, and collaboration between students

•• Stand-alone, classroom-based training based on the Information-Problem Solving Internet Model (ISP-I) •• Goal 1st session: to be able to select appropriate search terms and revise them when necessary (class discussion, modeling/explaining, practice) •• Goal 2nd session: to be able to skim the text, identify various elements of text structure and read selectively to locate relevant information (class discussions about the central topic, instructor explored selected web sites and demonstrated online reading strategies, students worked in pairs to locate appropriate information) •• Goal 3rd session: to use different criteria to judge trustworthiness of information and sources (lesson started with a question to gauge students’ knowledge of the subject, students worked as a group to discuss the characteristics of credible information and created a list of evaluation criteria in the form of a question)

Instruction

APPENDIX 23.1

8 weekly lessons of 1.5–2 hours each

Three 45-min sessions

Duration

(Continued)

•• All classes showed improvement on knowledge of healthy food •• Although students mostly used single or multiple search terms, they sometimes typed in the whole assignment •• While reading web texts, students mostly used scanning strategies and only sometimes used the menu or links on a website

•• Experimental group showed significant growth throughout the course on all searching skills. Practice alone (control group) did not lead to better searching. Gender and grade were not significant predictors •• Students searching at home for school assignments used significantly more evaluation criteria to judge the quality of information at baseline •• The more students tried to develop their skills, the more positive and less negative affect they experienced during information searching

Effect

Design

•• Pretest / posttest control group design •• 4th and 5th graders •• No transfer measured •• N = 96

Dependent variables

•• Number of correct answers on source questions and knowledge questions •• Reading ability taken into account

Author

MacedoRouet, Braasch, Britt, & Rouet (2013)

Appendix 23.1  (Continued)

•• Embedded, classroom-based intervention to encourage students to identify the sources, establish links between sources and content, and assess the knowledge of each source with respect to the topic of the text •• First step: to have students think of the reasons to accept or reject someone’s advice or opinion •• Second step: students read a text and answer questions •• Third step: students explained the topic of the text they had just read to make sure students had correctly identified all the characters in the story and their opinions •• Fourth step: students were asked who they thought was more knowledgeable about the text topic and why •• Fifth step: discussion •• Students went through steps 1 to 5 once while completing the practice task, then two times (steps 3 to 5 only) while completing two of the four experimental tasks

•• The first five lessons focused on acquiring web searching, reading, and evaluating skills, the last three lessons aimed at integrating the web skills and specific content knowledge •• The last three lessons the students were supposed to search the web for specific information and had to compose their own text based on that information

Instruction

One session of 30 minutes

Duration

•• Less skilled comprehenders displayed significantly better adjusted posttest performance after the intervention •• Skilled comprehenders did not benefit from the intervention •• Less skilled comprehenders in the interventional group were more likely to select an authoritative source at least once compared to those in the control group; skilled comprehenders in the intervention group performed statistically similar to those in the control group

•• ‘Non-reading’ often occurred while scanning web texts, especially in the form of ignoring relevant headings •• With regard to web evaluating skills, students never questioned the reliability of a specific website. They sometimes explicitly paid attention to the usefulness of a website

Effect

•• Ownership •• Interpretation and personalization •• Adaptation

Gerjets & HellentalSchorr (2008)

•• Declarative knowledge •• Search performance

Secondary education

De Vries, Van der Meij, & Lazonder (2008)

•• Pretest / posttest control group design •• 7th and 8th grade •• No transfer measured •• N = 61

•• Design experiment •• 5th and 6th graders •• No transfer measured •• 16 groups of 3–4 students

1. Representation of the WWW as an information environment: Basic knowledge about the Internet, the WWW, and search systems on the web. 2. Information problems: Basic knowledge about information problems, their potential sub-goals and complex sub-goal structures. 3. Localization of information on a website: Structure of websites and auxiliary tools.

•• Embedded, web-based training program called CIS-WEB, combined with classroom teaching •• Six training modules:

•• Embedded, computer-based program •• The portal was built as a hierarchy of main topics, and subtopics representing concepts in the domain. The portal contained 75 pages and about 246 links to websites •• Additional support was given by a sitemap with a hyperlinked overview •• The worksheet was an A4 paper containing three sections to write down the question, the provisional answer, and final answer. For each new web visit, one worksheet could be used

12 lessons of 45 minutes

3 lessons of 2 hours

(Continued)

•• CIS-WEB had a positive impact on declarative knowledge acquisition, both for search-irrelevant and search-relevant facts •• Search performance improved for all four sub-goal structures

•• Ownership: The observations showed that in both classrooms, web use was successfully integrated in the learning task •• Interpretation and personalization: The portal played a critical role in helping children to select relevant information. Furthermore, significantly more correct answers were formulated •• Adaptation: Examination of the answers on the worksheets showed that most answers were adoptions with only minor textual adjustments

Process:

Argelagós & Pifarré (2012)

•• Task performance

Product:

•• Define problem •• Info. search •• Scanning / processing •• Organizing / presenting •• Appropriateness of search terms and selected results

Dependent variables

Author

Appendix 23.1  (Continued)

•• Pretest / posttest control group design •• 7th and 8th grade •• No transfer measured •• N = 48

Design

•• Embedded, web-based program •• The IPS web-based activities (web-quests) were devoted both to training IPS skills and learning curricular contents •• Each activity consisted of an authentic learning task completed in approximately four sessions •• Students worked in pairs to perform the IPS activities collaboratively •• Principles: embedding, structuring (welldefined objective, structure), supporting (IPS scaffolds)

•• Classroom teaching, work in dyads and individual work •• Instructional methods: worked-out examples, symbolic and iconic visualizations, interactive multiple-choice questions with feedback, working sheets and exercises

4. Localization of a website: Methods for retrieving websites. 5. Selecting an information provider: Evaluation and selection of an information provider with regard to credibility and actuality. 6. Identifying sub-tasks of information problems: Breaking complex information problems into sub-tasks and selecting strategies to solve these sub-tasks.

Instruction

First year: nine activities of four sessions of 1 hour In total 36 hours Second year: six activities of four sessions of 1 hour In total 24 hours

Duration

•• Experimental students outperformed controls on defining the problem •• A trend that experimental students searched more efficiently, needed fewer attempts, and invested more effort in ‘scanning and processing the information’ •• The experimental group obtained significantly higher appropriateness scores with respect to ‘search terms’ and ‘selected results’ from the SERP •• Experimental students displayed better task performance than controls

Effect

•• Two-by-two factorial quasiexperimental •• design •• Factors: techn. scaffolds and teacher scaffolds •• 9th and 10th grade •• No transfer measured •• N = 347

•• Pretest / posttest control group design •• 9th grade •• Transfer measured •• N = 134

•• Domain-specific knowledge •• Metacognitive awareness (knowledge about cognition, regulation of cognition)

•• Navigation patterns •• Source evaluation •• Information comprehension •• Controlling for: •• Prior knowledge •• Argumentative reasoning •• Topic beliefs •• Reading comprehension •• Internet use •• Achievement in science

Raes et al. (2012)

Mason et al. (2014)

•• A stand-alone instruction called SEEK consisting of three pages of declarative information about how to evaluate the reliability of a website and the veracity of its content •• The material explained that there are three main criteria to assess how reliable or trustworthy websites are: Who is the author? How reliable is the information? How well does the site explain the information? Procedures for answering questions about each of these criteria were also provided. The declarative information material was accompanied by instructions for the specific session (basic inquiry task) •• In addition, a worksheet for each site with questions about the author, information, and explanation was provided •• After reading the provided information, students performed a search task using a Google-like SERP.

•• Embedded, web-based program called WebBased Inquiry Science Environment (WISE) combined with classroom teaching •• Focus on science content and IPS (i.e., search, select, gather, and use web info as evidence to support claims and answers) •• Students worked in pairs on inquiry activities and were supported by scaffolds that were subsequently faded •• In the teacher-enhanced conditions, the teacher interacted with groups of students to monitor their IPS process, e.g., asking questions to stimulate students’ reflection, probing students’ thinking, and asking questions to help them clarify and elaborate on ideas •• In the conditions without teacher-enhanced scaffolding, students received no metacognitive and strategic prompting. One session of 70 minutes A second session for measuring transfer

Multiple sessions, not further specified

(Continued)

•• Experimental students spent longer reading the most authoritative electronic resources •• Experimental students rank-ordered more accurately the least reliable sources, and justified their rank-ordering •• Experimental students produced an overall more accurate rank-ordering of the eight sites

Transfer task

•• Experimental students made fewer visits to the least reliable sites •• Experimental students rank-ordered more accurately the most reliable sites, and justified their rank-ordering •• Experimental students were not able to produce an overall more accurate rankordering of the sites

Basic inquiry task

•• Students in the three experimental conditions significantly outperformed students in the control condition without scaffolds •• After the intervention, all students reported a higher knowledge of cognition, but no differences were found between conditions. Still, pairwise comparisons suggested that the condition combining teacher-enhanced and technology-enhanced scaffolding significantly outperformed the control condition without scaffolds •• Pairwise comparisons showed that both the condition with technology-enhanced scaffolds and the condition with teacherand technology-enhanced scaffolds significantly differed from the control condition on regulation of cognition

Design

•• Quasiexperimental posttest only control group •• 12th grade •• Transfer measured •• N = 130

•• Pretest / posttest control group design •• 9th grade •• Transfer measured •• N = 101

Dependent variables

•• Concepts in essays •• Rank-order decisions •• Rank-order justifications (content-based, source-based)

•• Evaluation of SERP •• Evaluation of websites and information •• Learning outcome

Author

Braasch et al. (2013)

Walraven et al. (2013)

Appendix 23.1  (Continued)

•• Embedded classroom intervention to teach students how to evaluate search results, information, and sources when searching for information on the WWW in a history context •• Used different kinds of tasks and support in a variety of settings to foster transfer •• Goal of the first lesson was to confront students with incorrect, false, and biased information and have them think about the importance of evaluating information

•• Embedded, classroom-based instruction, using a ‘contrasting-cases’ approach illustrating two plausible alternative strategies for evaluating multiple scientific documents from the Internet: a set of less sophisticated paired with a set of more sophisticated strategies •• Students were asked to reflect on the different kinds of strategies one might use when working with multiple documents •• They then studied the strategies that two students from a different secondary school reported when they researched multiple texts retrieved from the Internet. Students explained why some strategies were better than others. This was also discussed in dyads. Finally, a classroom discussion took place

Instruction

15 lessons of 50 minutes

One session of 60 minutes

Duration

•• Use of criteria for evaluating SERP: A significant interaction effect between ‘time’ and ‘condition’ was not caused by an increase in scores for the experimental classes, but by a decrease in scores for the control class •• Use of criteria for evaluating information and sources (websites): scores for the experimental classes increased over time, while scores for the control class decreased

Basic task (history)

•• Experimental students included more scientific concepts from the more useful documents compared to control students. However, number of erroneous concepts included from less useful documents did not differ between groups •• Experimental students displayed significantly better discrimination between documents •• Content-based justifications were similarly prevalent for students in both conditions •• Experimental students were far more likely to mention source features in their justifications than control students •• Experimental students more frequently used words indicating a consideration of document trustworthiness in their justifications •• The skills were trained using the topic ‘cell phone radiation’ and the posttest concerned the topic ‘weather conditions in the Pacific’. This could be seen as transfer

Effect

Walraven et al. (2010)

•• Evaluation of SERP •• Evaluation of websites

•• Pretest / posttest with two conditions •• 9th grade •• Transfer measured •• N = 84

•• Two different embedded classroom interventions on World War II •• Students in both conditions received a reader on IPS and how to evaluate search results, information, and sources •• High-road intervention: focused on the evaluation of search results, information, and sources, and was embedded in and linked to the whole process of IPS. Students worked on several information problems during the lessons. Together with complex whole tasks, they received a process worksheet, providing a stepby-step plan to solve their information problem •• Rich representation intervention: rich knowledge representations were visualized using mind map techniques to visualize criteria that can be used for evaluation of SERPs, sources, and information. Criteria were addressed to give students insight into how those criteria are interlinked, and should be used. Students discussed the representations and what they learned from them •• The critical difference between the programs lay in the guidance provided by the worksheets in the high-road intervention and the discussions about representations of criteria and their relations in the rich representation intervention

•• Students received process worksheets with the assignments for the next lessons. The questions on the sheets were linked to the phases of IPS •• In the first three lessons, the focus was on defining information problems, the next three lessons on the scanning of information phase, and so on •• Students worked on different tasks and answered questions during the different phases of the IPS process. At the end of every lesson teachers and students had a discussion on evaluation criteria. The goal of these discussions was to develop a mind map or knowledge structure 15 lessons of 50 minutes

(Continued)

•• Students’ use of criteria for evaluating search results (SERP) improved in the rich representation condition but decreased in the high-road condition •• Both programs had a positive effect on students’ evaluation behavior; that is, students evaluated more sources in regard to reliability and usability •• Students in both conditions evaluated search results using the title or the summary of the information. They evaluated information by questioning if the information could be used to solve the task, but the information was hardly evaluated regarding reliability. The websites (sources) were seldom evaluated

Transfer

•• Students in both conditions slightly improved their use of criteria for evaluating information and sources (websites)

Basic

•• Evaluation of the biology SERP showed an interaction effect not caused by an increase in scores for the experimental condition, but by a decrease in scores for the control condition •• The average score on the final exam was higher in the experimental classes than for the control class

Transfer task (biology):

Design

•• Elements of design-based research •• Experimental – control group design •• 8th grade (14/15 years old) •• No transfer measured •• N = 23

•• Pretest / posttest control group design •• 11th grade •• No transfer measured •• N = 23

Dependent variables

•• Reading comprehension •• Performance and attitude to learning

•• Sourcing skills

Author

Caviglia & Delfino (2016)

Britt & Aglinskas (2002)

Appendix 23.1  (Continued)

•• Embedded, computer-based intervention called the Sourcer’s Apprentice (SA); taught sourcing, contextualization, and corroboration in the context of researching a historical controversy •• Principle 1: Taught through situated problem solving. The application modeled skilled sourcing by prompting. Principle 2: The SA supported expert representations. Principle 3: By decomposing the skills and representing these elements in the interface, SA scaffolded students. Principle 4: The SA supported transfer in two complementary ways. Principle 5: Provided explicit instruction. Principle 6: Students were presented with challenging goals (to motivate engagement)

•• Classroom-based instruction, embedded in the curriculum of Italian language teaching •• Class activities were organized according to this sequence: (1) introduction to the information problem to be solved (10 minutes); (2) IPS activity in the computer lab with one PC per student (90 minutes); (3) discussion of the results (10–15 minutes) •• Students worked alone, with occasional assistance from the teacher and experiment leaders, to encourage them to be individually active and to make it easier to monitor their progress over time •• Students started from a problem; after searching the web, reading, and reflecting, they had to hand in written answers containing their solutions, plus explanations of their search paths and their justifications for their •• answer •• Weaker students received extra coaching and personalized goals to help them formulate better search queries

Instruction

Two days of exposure to the Sourcer’s Apprentice

50 hours in 6 months

Duration

•• Mixed ANOVA showed no main effect for condition or test occasion •• There was a significant interaction between condition and test occasion •• The SA group improved from pretest to posttest, whereas the control group performed poorer at posttest than at pretest

•• The final evaluation showed that students in the experimental group scored significantly higher on the PISA literacy test •• Students developed a positive attitude toward studying at school •• More participants in the intervention group continued to the next grade than did students in the control group

Effect

Section VI

Assessment of Multiple Source Use

24

COMPLEMENTARY METHODS FOR ASSESSING ONLINE PROCESSING OF MULTIPLE SOURCES Lucia Mason and Elena Florit university of padova, italy

INTRODUCTION The purpose of this chapter is to critically review the various methods that have been used to assess the online processing of multiple sources. To deal with assessment implies addressing theoretical issues, as assessment is closely related to theory. This chapter first describes what an effective use of multiple sources implies in terms of online processing, with reference to the conceptual model that currently guides research in this area of investigation. The description of the model underlines the complexity of the processes which users of multiple sources of information are involved in when reading them. The chapter then analytically examines each of the methods used to assess online processes in this field of research. For each method, theoretical justification and empirical evidence will be discussed with the aim of highlighting potentials and limitations. Some of the reviewed methods have a long tradition in educational research, while others have been used only recently, or even represent a new potential approach to the study of multiple-document literacy in school and academic settings. The link between online processes and offline outcomes of reading multiple sources is highlighted. The chapter ends with overall implications for theory, research, and practice. Four clarifications are necessary to better understand this contribution. The first is that our review is not intended to be a systematic review based on the inclusion criteria, search strategies, and article selection that such a review requires. Rather, our review is a presentation of underpinnings and results of several methods, as well as of their implications for assessing online processing of multiple sources. Given the aim of the review, we have first done a search of online methodological procedures and techniques in the PsycInfo database. Four methods were identified: Note-taking, thinking aloud, reading time, and eye movements. A promising approach which has very recently been applied to the study of multiple-text comprehension—that is,

425

426  •  Mason and Florit

physiological measurement—has also been considered in the review. We have then identified some relevant examples of studies that used one or more of these methods, which are representative of their potential advantages in extending our knowledge about the use of information sources. This review is therefore not a complete and exhaustive review of all studies that used the identified methods, but rather a review of illustrative studies to highlight both strengths and weaknesses of each online method. The second clarification concerns the polysemy of the term source. As noted by Goldman and Scardamalia (2013), even within a single article, source may have multiple referents. In this chapter source has two referents: in a broader sense, the term refers to an information resource (e.g., a website that provides information on a topic), whereas, in a more restricted sense, the term refers to metadata about a text or document (e.g., the author, publication date, or any other type of information about an information resource). Moreover, we also clarify that we use the term sourcing to refer to the “processes of identifying and representing metadata”, in accordance with Goldman and Scardamalia (2013, p. 259). The third clarification is that the current review refers to studies that focus on multiple information resources, either printed or digital. That is, students may have processed multiple sources of information on paper, as has mainly been the case until recently, or in digital format. The latter may include reading online multiple sources when navigating on the Internet, or reading sources that have been stored locally by creating an offline environment in which the webpages are presented through a browser with clickable links, thus appearing as a Google search output page, although sources are in fact offline. The fourth clarification is that we will only review process methods that have been used concurrently, not retrospectively, to task performance.

PROCESSING MULTIPLE SOURCES: WHAT IS INVOLVED? From the last years of primary school onward, students very often face multiple sources of information, especially when they use the Internet to learn more about an unfamiliar topic. In this regard, the Internet is unparalleled for the opportunities it offers in terms of a huge number of sources on any topic. What does it mean to process multiple sources on a given issue or question? The Multiple-Document Task-based Relevance Assessment and Content Extraction (MD-TRACE) model (Rouet & Britt, 2011; see also Britt & Rouet, this volume) describes the processes underlying the use of multiple sources or documents with the aim of carrying out a particular task, for instance answering questions, composing a short essay, or preparing a visual presentation. This model describes the use of a set of documents as an unfolding, iterative cycle through five main processing steps. The first step is the task model construction. Based on their interpretations of the reading task, readers form a representation of the expected outcomes, which results in a set of goals to achieve and plans to achieve them. The second step concerns the assessment of information needs against the task demands. Readers may recognize that they do not possess the required topic familiarity to start composing an essay, for instance, thus they need to look for information from outside sources. The third step implies that readers select several documents on the basis of their relevance to the established goal, and then read and evaluate their content. During the processing of multiple documents, which may be influenced by a number of individual characteristics (e.g., prior

Complementary Methods  •  427

knowledge, reading comprehension ability, or epistemic beliefs), readers may identify similarities and differences between the ideas proposed in each document in order to form a comprehensive, integrated representation. The fourth step concerns the construction or updating of a task product, that is, the outcome of the task in terms of an essay or presentation. The fifth and last step is the assessment of the product quality against the task goals. If the product is not evaluated as satisfactory, readers may need to recycle through the previous stages. For this chapter, the third step is of greatest interest, with readers mainly involved in selecting, reading, and evaluating information. If selection occurs on the basis of the relevance of information to the specific goal of multiple-source use, reading and evaluation may be influenced by the reliability of the sources. Consequently, a source evaluated as scarcely reliable because of low author expertise may be processed for a shorter time even if it is relevant to the reading goal. On the other hand, a source that appears highly reliable because of high author expertise may receive more attention and be considered useful. Information from reliable sources can then be integrated. However, the two first steps of the MD-TRACE model are also of interest for this chapter. In the first step, readers set up the goals of the reading activity and consider the procedures necessary to accomplish them. The established goals influence the successive processes of information searching, processing, and use. In the second step, readers determine if, and to what extent, they may rely on internal resources, for example what they already know about the topic, or if they need to rely on external resources to accomplish their reading goals. The two last steps are of less interest in this chapter as they concern task products. Still, they are related to the previous steps. The fourth step, the response to the task, is influenced, at least to some extent, by the previous processing of information sources (McCrudden, Stenseth, Bråten, & Strømsø, 2016), and the last step involves deciding whether to terminate or continue recycling through previous steps. In sum, studies that aim to investigate online processing of multiple sources necessarily have to deal with the various steps, directly or indirectly. In reviewing their methods in the light of the extant literature, we will also highlight the specific steps that each of them is most suitable to capture. The next sections will present online process methods that have been used concurrently to task performance, such as notetaking, thinking aloud, reading time, eye movements, and physiological measures.

NOTE-TAKING A method that has been used for a long time to assess online processing of singletext comprehension is note-taking. Notes are supposed to be traces of information processing, and students are commonly seen taking notes while reading or during lectures. Note-taking is considered to be a strategy that may facilitate text comprehension. Why? Kiewra and colleagues (Kiewra, 1991; Kiewra et  al., 1991) investigated the functions and techniques of note-taking among university students reading single texts. These authors posited that note-taking has two main functions: encoding information and external storage of information. The mechanisms involved in note-taking are verbatim repetition or generative processing, which are associated with note quality. Generative processing implies making connections between the concepts or ideas read in a text. This processing underlies the quality of notes that

428  •  Mason and Florit

make a difference in text comprehension. In other words, it is the manipulation of textual content or the depth of information processing that explains the effectiveness of note-taking (Slotte & Lonka, 1999). From a methodological point of view, notes are qualitatively analyzed for content to identify various categories of information processing. Inter-rater reliability should be established to support the reliability of the procedure. Validity can also be established in relation to other measures, for example self-report questionnaires. Only very few studies, however, have investigated the quality of notes taken during multiple-text reading. Two investigations were carried out by Kobayashi (2009a, b) in the context of reading conflicting texts about English education in elementary schools in Japan. The first study examined the performances of university students who used note-taking as an external strategy together with highlighting and underlining. Students read to form an opinion or to find intertextual relations. Findings revealed the effectiveness of external-strategy use for the construction of intertextual relations when students’ reading goal was to find such relations (Kobayashi, 2009a). In the second investigation, the use of the same external strategies was examined while students read two conflicting texts on the topic of the previous study. It emerged that producing summary notes while reading had both indirect (through recall of intratextual arguments) and direct positive effects on the comprehension of intertextual relations, which is an indicator of deeper multiple-text comprehension (Kobayashi, 2009b). In both these studies, readers were explicitly instructed to use external strategies. More recent research has focused on spontaneous note-taking in relation to multipletext comprehension. A study by Hagen, Braasch, and Bråten (2014) investigated spontaneous notes taken by undergraduates while reading a set of multiple texts on climate change with one of two reading goals: to write an argument or to write a summary. Findings showed that notes reflecting attempts to integrate information within and across texts were related to comprehension and self-reported strategies when reading to write an argument. Notes of a higher quality were those reflecting intertextual elaboration that combined information from two or more texts, or notes referring to students’ background knowledge in relation to information from at least two texts. Lower-quality notes were paraphrases with no or little transformation of text content. The quality of students’ notes was associated with cross-text elaboration, as revealed by the reported strategies, and both intra- and intertextual inferential performance. Thus deeper processing of the relationships between concepts and ideas in each text, and between concepts and ideas across texts, seemed to underlie better multiple-text performance. As pointed out by Chi (2009), the very nature of a constructive activity is the transformative manipulation of the material to be learned, which reaches beyond the given information. In the context of multiple-text comprehension, readers are deeply involved in constructive activity when they find connections between texts and generate a comprehensive and integrated representation of conflicting information. Advantages Note-taking is a relatively simple and easy method to use in academic and school settings, and it requires no apparatus. In addition, note-taking has high ecological validity as it can be implemented in the naturalistic context framed by the specific

Complementary Methods  •  429

task (Kobayashi, 2009a, b). Considering the MD-TRACE model (Rouet & Britt, 2011), traces of information processing that characterizes its third step can be found in students’ notes, for instance, traces of source selection and evaluation in terms of reliability, and traces of content transformation to form a coherent intertextual representation based on similarities and differences between the ideas expressed across texts. Critical Issues A prerequisite for note-taking to be effective is that students must be familiar with this method; that is, to know how and why to take notes during reading. Younger students, in particular, need to be instructed in the method first, and be familiar with it. However, even when familiarity is not a problem, not enough traces of source evaluation and information transformation may be identified in notes, especially among younger students at primary and middle school levels. Therefore, it is preferable to supplement note-taking with other methods (Hagen et al., 2014).

THINKING ALOUD The method most used to assess online processing of multiple sources is thinking aloud. Individuals are asked to freely express everything that comes to their mind while they are dealing with multiple documents. The theoretical foundation for using verbal protocols as data was described clearly by Ericsson and Simon (1980). Pressley and Afflerbach (1995) highlighted the validity and relevance of this process method to understand performance in reading tasks. When participants are instructed to verbalize (but not to explain) what they are currently thinking, “a direct trace is obtained of the heeded information, and hence, an indirect one of the internal stages of the cognitive process” (Ericsson & Simon, 1980, p. 220). Verbalization processes externalize information that is currently in working memory, without making additional demands. The structure and course of the task processes are not altered by verbalizations, which may only slightly reduce the speed of task performance (Ericsson & Simon, 1980). Thinking aloud is usually audio-recorded and transcribed verbatim to generate verbal protocols. These are qualitatively analyzed; that is, protocols are segmented and segments are grouped into categories according to the purpose of the thinkingaloud analysis. For example, if the purpose is to identify the criteria used to evaluate the reliability of a set of sources, the segments of verbal protocols are assigned to different categories of criteria. Inter-rater reliability should also be established to support the reliability of the procedure. We identified two specific areas of research that adopted thinking aloud as a method to investigate online processing of multiple sources: (1) research on processing strategies and (2) research on epistemic cognition. We will review the relevant literature in each of these research areas to illustrate advantageous uses of thinking aloud. Thinking Aloud in Research on Processing Strategies In one of the first studies in their systematic research into multiple-text comprehension, Strømsø, Bråten, and Samuelstuen (2003) examined seven law students reading

430  •  Mason and Florit

naturalistic materials (textbooks and codes of law) over one academic year. They considered three types of sources: primary endogenous sources represented by the texts students were currently reading, secondary endogenous sources explicitly mentioned in the current text, and exogenous sources located outside the current task context (e.g., notes from lectures or previous readings). Thinking-aloud protocols were segmented and the categorization process led to the identification of memorization, elaboration, organization, and monitoring strategies. Memorization and organization were used particularly during the processing of the current text, while monitoring and elaboration emerged when focusing on sources external to the current text. Longitudinally, students focused their strategic processing of the various materials less on text-internal and more on text-external sources over time. The use of organization and elaboration strategies was associated with higher grades at the end of the academic year. Processing strategies were also investigated by Anmarkrud, McCrudden, Bråten, and Strømsø (2013) in a study focused on university students’ discrimination of more and less relevant information while they read multiple contrasting documents about the controversial issue of potential health risks associated with the use of mobile phones. The authors examined whether the students used different processing strategies while attending to more and less relevant information on the topic. First, students’ judgments of relevance were extracted from their thinking-aloud protocols, specifically from the comments made regarding a sentence or group of sentences that expressed opinions about their instrumental value in relation to the task instruction. This instruction asked participants to study the documents in order to provide a friend with informed advice about the question of health risks posed by the use of mobile phones. Second, the authors identified various processing strategies in the verbal protocols, with strategies referring to comments made by readers to control or modify their comprehension while reading a sentence or group of sentences. The strategies identified were categorized into three groups: linking, monitoring, and evaluation. Findings revealed that when the students processed information judged as more relevant, they made more connections between the current information and information found in other documents. Although they also used more evaluation and monitoring strategies while reading more relevant compared to less relevant information, the difference was not statistically significant. Moreover, the ability to appropriately identify relevant information and to evaluate less relevant information was positively related to the quality of post-reading essays. These findings were interpreted with reference to standards of coherence (van den Broek, Bohn-Gettler, Kendeou, Carlson, & White, 2011); that is, the criteria readers use to determine the level of their comprehension. The fact that participants used more linking strategies while processing more compared to less relevant information indicates that they had different standards of coherence for the two types of text information (Anmarkrud et al., 2013). The processing of multiple sources by undergraduates, as revealed in their thinkingaloud protocols, was also investigated by Goldman, Braasch, Wiley, Graesser, and Brodowinska (2012). Better and poorer learners’ sense-making and evaluation processes were compared while reading webpages about the causes of volcanic eruption. The analysis of the protocols was based on parsing comments into events, which were then coded into processes. Eight processing strategies were identified: repetition,

Complementary Methods  •  431

self-explanation, surface connection, irrelevant association, prediction, monitoring, information and source evaluation, and navigation (description of movements within or across pages). Better learners were more involved in self-explaining and monitoring of reliable sites than were poorer learners. The former also verbalized more about their moves from page to page while reading reliable Internet pages. In another thinking-aloud study, Strømsø, Bråten, Britt, and Ferguson (2013) focused on undergraduates’ spontaneous attention to and use of source information when reading multiple documents on the issue of mobile phones. Verbal protocols were coded according to three dimensions: whether the identified statements explicitly or implicitly referred to sources, whether students referred to source information about the current source or source information included in the current document, and the type of activity indicated in utterances, such as attention, predicting, interpreting, and evaluating. Spontaneous attention to source information was positively related to the active processing of the information in terms of making predictions, interpretations, and evaluations. In addition, spontaneous sourcing varied depending on text characteristics, so that vivid style and the presence of controversial information influenced it positively. In a more recent investigation, Barzilai, Tzadok, and Eshet-Alkalai (2015; see Barzilai & Strømsø, this volume) observed university students dealing with four blogposts on the topic of seawater desalination in Israel. Each blog-post included the author’s name, profession, and affiliation. Thinking aloud was used to reveal spontaneous sourcing. Verbal protocols were segmented in comments or sets of comments that were related to specific source information. As in the Strømsø et al. (2003) study, three types of sources were identified: the current blog-post, other blog-posts in the set, and other sources not included in the set given to participants. In addition, three types of sourcing activity emerged from the thinking aloud. The first was source representations; that is, the readers’ descriptions of sources. The second was source-content links, and the third was source-source links. Source-source links involved comparisons between source claims and positions, as well as source reliability. Results showed a great variability in sourcing practices, ranging from lack of sourcing while reading the blog-posts to detailed source representations, source-content links, and sourcesource links. More implicit than explicit source comments were identified, probably because of the absence of sources embedded in the documents. That is, the inclusion of such sources in the body of the text might support their identification during participants’ processing of the reading materials. Moreover, participants who were highly involved in sourcing generated significantly more complex arguments about the topic than participants who sourced little, or not at all. Thinking Aloud in Research on Epistemic Cognition During Multiple-Source Processing Most thinking-aloud studies have been carried out to investigate epistemic cognition or epistemic beliefs when processing multiple information sources. In accordance with Greene, Sandoval, and Bråten (2016), we consider epistemic cognition to refer to unique mental processes that are activated when the object of cognition is the epistemic; that is, when it concerns knowledge and issues related to knowing. The term

432  •  Mason and Florit

beliefs refers to “an influence upon, and result of, cognitive processing” (Greene et al., 2016, p. 5). A combined perspective has also been proposed, considering epistemic beliefs as the content of the processes of epistemic cognition. In other words, the processes of epistemic cognition draw on beliefs about knowledge and knowing when individuals reason and make decisions about epistemic issues (Sinatra, 2016). Until relatively recently, research on beliefs about the nature of knowledge and the knowing process was mainly carried out using self-reports. The use of this methodology was promoted by Schommer (1990), who in her seminal work developed a belief questionnaire from which subsequent questionnaires were derived. This line of research was criticized for its decontextualized nature, since important features of the context, such as characteristics of tasks, contents, materials, and resources, were not taken into account when assessing beliefs about knowledge and knowing (Greene, Azevedo, & Torney-Purta, 2008; Greene & Yu, 2014; Louca, Elby, Hammer, & Kagey, 2004; Sandoval, 2005). Moving toward more contextualized measurement of epistemic cognition, scholars have examined beliefs about knowledge and knowing within concrete learning scenarios, in particular when students evaluate and comprehend information from multiple sources. In this way, epistemic beliefs in action—that is, spontaneous reflections about the information accessed in various documents—are targeted. Thinking aloud has therefore been considered a valid and useful method for measuring epistemic beliefs in more naturalistic and context-sensitive ways (Mason, 2016). Mason and associates were among the first scholars who examined university (Mason, Boldrin, & Ariasi, 2010) and high school students’ (Mason, Ariasi, & Boldrin, 2011) spontaneous activation of epistemic beliefs while they read and evaluated sources and information. These studies used the controversial topic of health risks associated with the use of mobile phones. In both studies protocol analyses revealed spontaneous activation of beliefs about the knowledge accessed on the websites, especially about the credibility of digital sources and how to know from online information, that is, about the criteria for justification of knowledge. University students who spontaneously generated more sophisticated reflections about the reliability of the sources (who to believe) and the veracity of the information (what to believe) learned more than those who only evaluated the sources (Mason et al., 2010). A similar pattern of results emerged for high school students, whose argumentative reasoning skills were also related to web-based learning about the topic (Mason et al., 2011). The youngest students involved in a thinking-aloud study were the Israeli sixthgrade participants in Barzilai and Zohar’s (2012; see Barzilai & Strømsø, this volume) research, who dealt with two dilemmas during online inquiry tasks (whether chocolate is healthy and whether fish farms harm the coral reef). Verbal protocols revealed that students evaluated the trustworthiness of only 39% of the websites they read, and only about half of the students explicitly mentioned criteria for the evaluation of source trustworthiness. These criteria appealed to authority, expertise, and the presence of scientific evidence, as well as to bias. Barzilai and Zohar (2012) also identified students’ epistemic perspectives according to Kuhn, Cheney, and Weinstock’s (2000) framework, including absolutist, multiplist, and evaluativist perspectives. Contrary to their expectations, the authors did not find relations between these students’ perspectives and source evaluation as revealed in the thinking-aloud protocols. However, evaluativist students

Complementary Methods  •  433

were better at integrating sources than their absolutist counterparts. That is, evaluativist students did better when identifying multiple websites’ points of view, comparing them, and using them to form an argument. It should be pointed out that in this study, the authors did not use the neutral type of thinking aloud instruction mentioned above, as proposed by Ericsson and Simon (1980), but a thinking-aloud approach that elicited students’ self-explanations (e.g., “Why did you enter this website?”, “What is your opinion of this website?”). On the one hand, the prompts were instrumental in revealing epistemic thinking about the online sources accessed and in offering rich data. On the other hand, they might have raised students’ awareness about the sources and increased their comprehension (Barzilai & Zohar, 2012). The thinking-aloud method was also applied by Ferguson, Bråten, and Strømsø (2012) in a study with undergraduates who read multiple sources about the issue of mobile phones. Verbal protocols were segmented into utterances and the analysis allowed a more articulated and fine-grained categorization of epistemic beliefs about the justification of knowledge, which is essential when processing multiple sources. This new categorization became the reference for current research. Specifically, students appealed to justification of knowledge by authority (source reputability), personal justification (personal opinion and previous experience and knowledge), and different sources (cross-checking, comparing, and corroborating). Thinking-aloud protocols also provided evidence of the theoretically assumed mechanisms of epistemic change as students spontaneously verbalized cognitions about epistemic doubt, volition, and resolution strategies (Bendixen & Rule, 2004). The first implies questioning and discrediting one’s current views about knowledge and knowing. The second concerns intention and commitment to invest in a sustained effort to overcome the epistemic doubt. Finally, resolution strategies include various types of reflection to resolve the doubt. Using multiple conflicting sources on the same debated topic of mobile phones, Bråten, Ferguson, Strømsø, and Anmarkrud (2014) also examined whether epistemic cognition about the justification of knowledge was related to sourcing (i.e., explicit reference to source information and source-content links) and argumentation as revealed in undergraduates’ essays. Verbal protocols based on students’ thinking aloud while reading were segmented into utterances and coded into the three dimensions of justification of knowledge, that is, justification by authority, personal justification, and justification by multiple sources. Findings showed that beliefs in justification through corroboration of multiple sources positively predicted students’ sourcing and argumentation in essays over and above their prior topic knowledge. The aforementioned studies indicate that more advanced forms of epistemic cognition about the process of knowing, which may be related to better source processing, may be required in the challenging reading task context of comprehending and integrating multiple documents (Bråten, Britt, Strømsø, & Rouet, 2011). Advantages Thinking aloud is a method with high ecological validity because it is context sensitive and allows for naturalistic inquiries framed by the completion of specific tasks. It provides unparalleled rich material about processing without any research apparatus (only an audio recorder is needed); thus, it largely contributes to an in-depth

434  •  Mason and Florit

understanding of mechanisms underlying performance (Cote, Goldman, & Saul, 1998; Pressley & Afflerbach, 1995). The outcomes of all these studies highlight that multiple-source comprehension relies on crucial processes activated during strategic reading. Considering the MD-TRACE model (Rouet & Britt, 2011), thinking aloud can provide empirical support for the first, second, and third steps included in the model. Depending on specific task instructions, verbal protocols may offer explicit comments about the goals (first step), the need for further information (second step), or about processing at certain points of time (third step). As reviewed above, studies have documented that thinking-aloud protocols offer comments about the selection of information on the basis of relevance to the reading goal, as well as about the strategies adopted when trying to understand the various sources and building a coherent representation from multiple documents (third step). Furthermore, evaluation of information in terms of source reliability and content accuracy according to more or less advanced criteria (third step) has been identified in verbal protocols generated during the reading of multiple documents. Critical Issues Thinking aloud is not as simple and easy to use in academic and school settings as notetaking because it requires more time and familiarity with the procedure. Although it has been proven valid for examining reading processes, it also has some evident shortcomings. First, it is intrusive, at least to some extent, and interference with the ongoing cognitive processes cannot be excluded (Veenman, Van Hout-Wolters, & Afflerbach, 2006). Second, it can only capture reflections students are aware of. Thus, because the procedure only permits investigation of conscious reading processes, quick and unconscious reading processes cannot be captured by thinking-aloud data. Third, verbalizations may have one meaning for the participant and another for the interpreter, which can result in misinterpretation of a reading process. Fourth, it is difficult to use with younger students who may need continuous prompts and requests for explanations, which are likely to alter the ongoing cognitive processes. Given the limitations of thinking aloud, it is essential to gain converging evidence from a less intrusive and more objective measure (e.g., reading time; Kendeou & van den Broek, 2007). Yet, despite the limitations, thinking aloud can be considered one of the most suitable and simplest methods to assess online processing of multiple sources.

READING TIME The time spent reading conflicting documents on a topic has also been used as an objective and quantitative online measure. Specifically, reading time has been considered as a proxy for the effort invested during a reading task. From a theoretical point of view, longer reading times may be indicators of higher engagement with the documents, once it has been determined that basic reading skills do not matter and individual differences in reading are taken into account (Schroeder, 2011; Stine-Morrow, Miller, Gagne, & Hertzog, 2008). The role of reading time as an indicator of effort was examined in a study with upper secondary school students who read multiple sources about sun exposure and health (Bråten, Anmarkrud, Brandmo, & Strømsø, 2014). The overall aim of the study was to

Complementary Methods  •  435

test a model linking individual differences, processing, and multiple-text comprehension. It emerged that reading time was predicted by individual interest in comprehending science texts and epistemic beliefs concerning justification of knowledge by multiple sources. Reading time as a proxy for effort was also related to self-reported deeper-level processing strategies and predicted multiple-text comprehension as assessed by shortessay answers to open-ended questions (Bråten et al., 2014). Reading time in relation to information relevance and trustworthiness was addressed in a study by Salmerón, Kammerer, and García-Carríon (2013), including university students who read multiple sources about greenhouse gas emissions. Participants were provided with two search engine results pages (SERPs). One presented search results listed according to the relevance and completeness of information, while the other presented them in the reverse order. SERP type had an effect on reading times with respect to topic relevance. That is, with the reversed SERP, students spent more time on irrelevant information (presented at the top) but less time on relevant and complete information (presented at the bottom). No effects of SERP type emerged for trustworthiness, which was the other characteristic of information about the topic. The role of background knowledge was also shown, as longer reading times were registered for the relevant and trustworthy search results among participants with higher prior knowledge of the topic, irrespective of SERP position. Reading times and navigation behavior were measured in three intervention studies aimed at improving source evaluation skills. In the second experiment reported in Wiley et al. (2009; see also Wiley, Jaeger, & Griffin, this volume), undergraduates were given instructions on how to evaluate source information by means of the SEEK instructional unit, where SEEK is an acronym for Source, Evaluation, Explanation, and Knowledge. The SEEK instructions were first applied to an inquiry task dealing with the evaluation of websites about a low-carbohydrate diet, and transfer was tested on an inquiry task about the causes of volcanic eruption. Results indicated that for the practice inquiry task, non-instructed students were more likely to reread the sources in a non-selective way or return to unreliable sites, whereas SEEK-instructed participants were more likely to reread reliable sites selectively. For the transfer task, there were no significant differences between instructed and non-instructed students with respect to time spent reading either reliable or unreliable sources. The number of visits to reliable sites by the two groups of participants was also quite similar. However, SEEK-instructed students discriminated better between reliable and unreliable sites in evaluating the Internet sources and constructed a better intertextual model in learning from them. In a study by Mason, Junyent, and Tornatora (2014), upper secondary school students instructed in source evaluation by means of the SEEK instructional approach spent a longer time reading more reliable sources on genetically modified food than did non-instructed students. This outcome was also corroborated, to some extent, by more appropriate rank-ordering of more and less reliable sources by the instructed students, as well as by their use of more advanced criteria for source evaluation and better surface and deeper-level intertextual comprehension of the online pages. Finally, in a study by Kammerer, Amann, and Gerjets (2015), adults without any university level education participated in an intervention on source evaluation that was based on a combination of declarative information, concrete examples, and interactive exercises with feedback. They performed two search tasks, one in the context

436  •  Mason and Florit

of the intervention and one to assess its effects. The search tasks used mock Google SERPs including objective, subjective, and commercial sites. The objectives of the study included an examination of the predictability of Internet-specific epistemic beliefs for navigation behavior in web search task 1, and an examination of the effects of source evaluation on navigation behavior in web search task 2. A positive relation emerged between beliefs in justification of knowledge by multiple sources and time spent on reliable, objective webpages, as well as between beliefs in Internet as a reliable source of information and time spent on reliable, objective webpages. In addition, the instructed students spent longer on objective sites in search task 2, were more likely to make post-reading decisions in accordance with those pages, and were more certain about their decision than were non-instructed students. Advantages Reading time is an objective measure of engagement in processing reading materials. It can be considered an indicator of effort invested in attending to information provided in multiple sources when potentially interfering variables have been controlled (Salmerón et al., 2013). In addition, reading time can be easily registered using simple software. Therefore, it is a methodology that can be applied to examine the third step of the MD-TRACE model described by Rouet and Britt (2011), in particular the processing of document information. In this regard, it should be pointed out that in all the aforementioned studies, reading-time data were in the expected direction and coherent with other performance or self-report data. Critical Issues However, reading time must be interpreted in relation to other variables that can be person-related, such as cognitive (e.g., reading skills) and motivational characteristics (e.g., individual or topic interest), task-related (e.g., to write an argumentative or descriptive text on the topic), content-related (e.g., degree of conflict between the sources), or context-related (e.g., training or no training for source evaluation). This means that even more than other online measurements, reading-time data need to be triangulated with data from other methodologies for a rigorous interpretation of their meaning (Kendeou & van den Broek, 2007).

EYE MOVEMENTS In the last decade, eye-tracking methodology has also been adopted in research on multiple-text comprehension. The theoretical foundation for this quantitative method is that saccades (eye movements) and attentional shifts are necessarily linked. Specifically, two related theoretical assumptions underlie the method (Just & Carpenter, 1980). The first is that information processing is not postponed but takes place while the information is encountered. The second is that individuals process in the mind the visual information that is currently being fixated by the eyes. Eye movements are, therefore, considered a strong online indicator of the cognitive processes involved in reading tasks (Rayner, 1998).

Complementary Methods  •  437

The unique contribution of eye-tracking methodology is the monitoring of the time course of text processing. Fixation times can be analyzed at a fine-grained level. Hyönä and colleagues (Hyönä, Lorch, & Rinck, 2003; Hyönä & Nurminen, 2006) examined two distinct and subsequent stages of global text processing, as reflected in the eyefixation records: first-pass reading and second-pass reading. First-pass reading is the summed duration of all fixations that land on a text segment (sentence, paragraph, or the entire text) when it is read for the first time. It includes both forward fixations and backward fixations. First-pass reading is considered to reflect the initial, more automatic processing of information. In addition, this index of the first encounter with a text may reflect some breakdown in comprehension when it includes long regressions. Second-pass reading is the summed duration of all fixations after the first-pass reading. More specifically, second-pass reading reflects the total look-back fixation time, which is the summed duration of all fixations returning to a text segment that has already been processed after its initial reading. In addition, sequences of eye movements can be analyzed to examine attempts at integrating information that is not adjacent. Second-pass reading is considered to reflect a delayed and more purposeful processing of information. One of the first eye-tracking studies in the research field of web searching was carried out by Gerjets, Kammerer, and Werner (2011). In this study, university students dealt with three different search results about which of two diets—low fat or low carbohydrate—promotes healthy and long-lasting weight loss. Both eye-fixation and thinking-aloud data (concurrent to reading) were collected. The former was expected to reveal implicit cognitive processes and the latter explicit information processing as expressed in utterances. In regard to the verbal protocols, half of the students were instructed just to verbalize everything that came to mind, while the other half were explicitly asked to evaluate the information sources and were instructed to verbalize the evaluation criteria applied to select search results and assess webpages. The authors measured the number of fixations and total fixation times on some areas of interest. Longer fixation times on search results, as well as a higher number of fixations and longer fixation times on specific pieces of information (e.g., source information, publishing information, user ratings), were considered potential indicators of the effect of instructions on visual behavior. Thus, these parameters were expected to indicate that participants tried to evaluate the quality of search results and the accessed webpages. Eye-tracking data revealed no significant differences between the instructed and noninstructed students for the number of gazes or fixation times on search results, or for the processing of specific information about sources or publishing. However, the students who were asked to evaluate search results and web information had more fixations and longer fixation times on “user ratings” (considered quality-related information) and expressed more quality-related evaluative criteria. Eye-tracking data were therefore in line with the concurrent thinking-aloud data. In another study, Kammerer and Gerjets (2012) used eye-movement data to examine the influence of search interface and epistemic beliefs on university students’ source evaluation during a web search about two controversial therapies for a disease. Two search interfaces were used: one was a standard, Google-like list interface; the other was a tabular interface in which the search results were grouped into three categories—objective websites (scholarly websites or websites providing

438  •  Mason and Florit

neutral, fact-based information), subjective websites (forum websites providing personal opinions and experiences), and commercial websites (websites of health farms or pharmaceutical companies sponsoring their treatments)—to facilitate source evaluation. Fixation times revealed that students who interacted with the tabular interface allocated less visual attention to the commercial sites than did students who received the list interface. This finding was consistent with data showing that the tabular interface led students to select objective sites more frequently and commercial sites less frequently than the list interface. In a more recent study, eye-tracking methodology was used to investigate whether contradictions between two websites—one of which was biased for its commercial purpose—led students to pay more attention to “about us” information that was presented at the bottom of the page in Experiment 1 and was accessible by a click in Experiment 2 (Kammerer, Kalbfell, & Gerjets, 2016). In both experiments university students who read contrasting information about a fictitious nutritional supplement spent a longer overall time on “about us” and used this information more often in their written reports than students who read consistent information. In addition, those who read contrasting information evaluated the commercial site as less trustworthy and convincing, as revealed by verbal protocol data collected during reading in Experiment 1, and by site ratings in Experiment 2. In these eye-tracking studies total fixation times were computed without specifying the time course of information processing. In a study carried out by Mason, Pluchino, and Ariasi (2014), a more fine-grained analysis was performed considering indices of first-pass and second-pass reading or inspection. The aim of the study was to investigate implicit source evaluation, that is, whether information received differential attention depending on the reliability of the online source. University students were given four webpages with verbal and graphical information about the universal validity of the central dogma of molecular biology. Each webpage was made up of five types of information: a headline, two texts, and two pictures. The webpages did not vary with respect to relevance and usefulness; all provided essentially the same information useful for understanding both the normal, unidirectional nature of the transfer of genetic information and the reverse of this transfer that occurs in the case of retroviruses. However, the four pages differed with respect to reliability, as they were taken from an institutional source, Wikipedia, a popular online science magazine, and a private site with no information about the author. Findings showed that during the immediate, more automatic processing, as revealed by the first-pass reading, students spent more time processing the pictures displaying more and less familiar information that was included in the most reliable source. No differences emerged for the delayed, less automatic second-pass reading. These outcomes indicate quick and efficient implicit source evaluation. In addition, a moderating role of topic-specific epistemic beliefs was observed as students with more availing beliefs about knowledge (knowledge is complex and based on evidence) paid more visual attention to the information presented in pages at an intermediate level of reliability, which presumably demanded more discernment (Mason et al., 2014). Differences in processing were observed only for the pictures, not for the texts about the dogma of molecular biology. This can be interpreted by referring to research on students’ processing in multimedia learning. Although instructional pictures may not be well attended

Complementary Methods  •  439

to by students, they tend to believe that illustrations are more easily processed than texts to form a rough idea of the information conveyed (Schroeder et al., 2011). Advantages Eye-tracking methodology is unique in examining the time course of information processing, providing objective and detailed data about the distribution of visual attention (i.e., where, how long, and in what order attention is allocated to instructional materials; van Gog & Scheiter, 2010). As such, it offers unparalleled insight into cognitive processes, including those that are not necessarily accessed by introspection as they occur quickly and in an automated fashion (Rayner, 1998). This seems particularly relevant in research on processing multiple sources because fast and immediate processing may be indicative of implicit selection and evaluation. In addition, more fine-grained analyses of delayed processing may reveal attempts to integrate text content (Mason, Tornatora, & Pluchino, 2015). Therefore, all three processes included in the third step of the MD-TRACE model (Rouet & Britt, 2011)—selection, processing, and integration—can be revealed by eye-tracking data. In addition, modern technology makes eye trackers non-intrusive as they do not require a chin rest and allow relative freedom of head movement, which means they can be used more easily in the school context. Moreover, eye tracking is not only a process measure in itself, as illustrated in the aforementioned studies, but also an instructional tool to model desired processing behavior (e.g., Mason, Pluchino, & Tornatora, 2015). A gaze replay of a model who looks first at source information and then processes the content might be an effective indirect way to make students aware of the importance of source details in dealing with multiple documents on the same topic. Future research can explore this eye-tracking affordance. Critical Issues Eye-tracking data are not self-explaining and often require substantial interpretation by the researcher to better understand the observed visual behavior (van Gog & Scheiter, 2010). Therefore, they must be supplemented with other process measures, such as concurrent or retrospective verbal protocols, and outcome measures, such as learning performance. A methodological triangulation with other data sources (concerning person, task, or content variables) is needed to increase the validity of eyemovement data (Kammerer & Gerjets, 2012; Salmerón, Gil, Carrión, & Bråten, 2016). In the aforementioned eye-tracking studies, eye-fixation data were validated to some extent by data obtained with other methodologies. Finally, eye-tracking apparatuses are relatively expensive, which may hinder use on a large scale.

A NOVEL APPROACH: PHYSIOLOGICAL MEASURES A novel, promising approach that has been applied in only a couple of studies of singletext comprehension uses physiological indices as process measures during reading. These may include, for example, skin conductance level, heart rate, and respiratory sinus arrhythmia (Kreibig & Gendolla, 2014). Physiological indices reflect responses

440  •  Mason and Florit

of the autonomic nervous system, which includes the sympathetic (SNS) and parasympathetic (PNS) branches. An increase in sympathetic activity is expressed in cardiac acceleration—that is, increased heart rate—reflecting a higher level of arousal. An increase in electrodermal activity also indicates sympathetic response. Activation of the sympathetic nervous system is associated with increased task difficulty and emotion (Kreibig & Gendolla, 2014). Parasympathetic activity involves greater influence of the vagal tone on the heart, which leads the cardiac system to decelerate and not become overexcited, an essential function for adaptation and self-regulation (Porges, 2007). As it is associated with a calm state and self-regulation, cardiac vagal tone may be considered a psychophysiological correlate of the cognitive effort required to execute a complex task. A study by Daley, Willett, and Fischer (2014) indicated that university students’ respiratory sinus arrhythmia (cardiac vagal tone) during reading predicted singletext comprehension. Better readers were students who had an initial decrease in vagal tone after hearing the instructions and then an increase in vagal tone, reflecting a calm state as a function of the appraisal and engagement process. A study by Scrimin, Patron, Ruli, Altoè, and Mason (under review) examined the psychophysiological correlates of single-text comprehension in relation to two reading goals, that is, to read to perform well and gain course credit, or to read for oneself. Findings revealed that university students’ heart rate (arousal) decreased during reading and increased during the testing phase. Positive, non-threatened appraisal (regulation) during reading was associated with better performance, which was obtained by students with a reading goal to perform well. The use of physiological parameters of activation and regulation can also be applied to research on online processing of multiple sources. Both arousal indicated by heart rate or electrodermal activity and cognitive effort indicated by cardiac vagal tone can be traced during multiple-document reading. Psychophysiological measures can therefore add insight into the complex dynamics including the processing and integration of document information described in the third step of the MD-TRACE model. Such dynamics may involve emotional arousal relative to text content and individual emotional reactivity, as well as appraisal and regulation that are conducive to states of calm and concentration required by complex cognitive performance. Only one study has adopted a psychophysiological approach to investigate the role of emotional arousal, as reflected in electrodermal activity, in middle school students’ multiple-text comprehension on a debated issue (Mason, Scrimin, Tornatora, & Zaccoletti, 2017). Emotionally low-reactive students outperformed high-reactive students with respect to deeper, intertextual comprehension, specifically with respect to references to source information and formation of source-content connections. Presumably, high physiological arousal and a more intense negative affective state acted as impairments for the complex cognitive task of constructing a coherent and integrated representation of a set of documents. Interestingly, the aforementioned indices of autonomic nervous system responses are acquired using relatively simple and inexpensive apparatuses that can easily be used in school and academic contexts. Obviously, physiological measures must be supplemented by data from other measures (Mason et al., 2017; Mason, Scrimin, Tornatora, Suitner, & Moè, 2018). Research on the role of affective engagement in

Complementary Methods  •  441

multiple-source comprehension (see Alexander & List, this volume) can benefit from the knowledge of micro-level psychophysiological correlates of its complex dynamics.

IMPLICATIONS FOR THEORY, RESEARCH, AND PRACTICE Assessment of online processing of multiple sources is particularly demanding because the task includes an iterative cycle of several steps, as the MD-TRACE model describes (Rouet & Britt, 2011). The recursive characteristic of the processes involved in functional reading and use of a set of documents highlights the complexity of resources and cognitive activities that must operate to accomplish the task demands. The various methods reviewed in the previous sections are essential tools for process-oriented studies of multiple-text comprehension. Table 24.1 synthesizes our view that each of them is in principle more suitable for assessing some processes than others when dealing with a variety of documents differing in quality, which often present conflicting perspectives on the same issue. It is evident that one single method cannot tap all processing involved in multipletext comprehension, given the task complexity (see Goldman et al., this volume). Such complexity emphasizes the need for theoretically grounded assessment that relies on mixed methods. Hence, to unravel these processes in more detail, one method (e.g., eye tracking) should supplement another (e.g., verbal protocols). The combination of more traditional and innovative assessment approaches, including qualitative and quantitative methodologies, as well as analyses at different levels, is likely to offer an increasingly rich picture of what underlies one of the most frequent reading activities in the information society. Table 24.2 synthesizes the strengths and weaknesses of each of the methods reviewed in this chapter. Table 24.1  Overview of the Functions of the Various Methods for Assessing Online Processing of Multiple Sources in Relation to the MD-TRACE Model (Rouet & Britt, 2011).

Step 1: Task model construction Step 2: Assessment of information need Step 3: Selection Processing Integration of information Step 4: Task product construction Step 5: Assessment of product quality

Notetaking

Thinking aloud

Reading time

Eye movements

Physiological measures

Yes

Yes

No

No

No

Yes

Yes

No

No

No

Yes Yes Yes

Yes Yes Yes

Yes Yes No

Yes Yes Yes

No Yes No

No

No

No

No

No

No

No

No

No

No

442  •  Mason and Florit Table 24.2  Overview of the Strengths and Weaknesses of the Various Methods for Assessing Online Processing of Multiple Sources. Strengths

Weaknesses

Note-taking

−− Simple and easy to use −− Ecologically valid −− No apparatus required

Thinking aloud

−− Ecologically valid −− Rich traces of processing −− No apparatus required

Reading time

−− Objective measure −− Requires simple apparatus −− Unique objective and detailed measures of the time course of processing −− Allow fine-grained analyses of processing −− Potentially usable to model desired processing −− Micro-level objective measures of emotional arousal −− Micro-level objective measures of emotional regulation −− Require relatively simple and inexpensive apparatus

−− Requires some familiarity with the method −− There may be insufficient traces of processing −− Intrusive to some extent −− Not easy to use with young students −− Captures only conscious processing −− Verbalizations may have different meanings for verbalizer and interpreter −− Must be interpreted in relation to other variables −− Not self-explaining −− Need to be triangulated −− Require still expensive apparatus

Eye movements

Physiological measures

−− Need to be triangulated −− Require familiarity with uncommon software in educational research

Assessment of multiple-source processing itself may contribute to theory development, refining the descriptive model that guides assessment in a bidirectional relationship between theory and method. On the one hand, theory has methodological implications and needs empirical support to move beyond mere speculation; on the other hand, a method needs theoretical grounding to be more than just a technical tool. Despite the progress that has been made, there is still a need for research on assessment of online processing of multiple sources. Scholars need valid and reliable methods that capture the various aspects of the task in relation to different reading contexts. The ultimate aim is to take advantage of available complementary methods that tap intra- and inter-individual variations. In this regard, widening the age range and academic skills of the samples will not only contribute to our understanding of the developmental aspects, which are still neglected in the literature, but will also be beneficial for both theory and research in the area more generally. Assessment of online processing of multiple sources is also of educational importance. Currently, students search the web daily to learn more about unfamiliar topics for school assignments, typically being connected through a smart phone, tablet, or laptop. However, this does not imply that they have the declarative and procedural knowledge to access and evaluate digital sources when seeking online information, as research has indicated (e.g., Barzilai & Zohar, 2012; Walraven, Brand-Gruwel, & Boshuizen, 2009; see also Hartman, Hagermann, & Leu, this volume). Analytical and

Complementary Methods  •  443

critical processing of information presented across texts is therefore more crucial than in the past when editors and publishing companies controlled the accuracy and relevance of information in traditional printed media. Thus, motivated and effortful reasoning based on engagement with multiple sources is of paramount importance. In this regard there is a particular need for interventions to promote multiplesource comprehension and use, as this volume also documents. Updated knowledge obtained from the application of various methods for measuring source processing is useful for two main reasons. First, assessment is needed to focus educational interventions on the processes or activities whose activation or execution is more difficult. In particular, it is needed to provide an informed basis for guidance and scaffolding. Second, assessment is essential to capture various aspects of source processing before and after the implementation of educational interventions, that is, to test their effectiveness in promoting digital reading and quality learning with multiple documents on complex topics.

ACKNOWLEDGMENTS This research was supported by a grant to the first author (CPDA158418) from the University of Padova.

REFERENCES Anmarkrud, Ø., McCrudden, M. T., Bråten, I., & Strømsø, H. I. (2013). Task-oriented reading of multiple documents: Online comprehension processes and offline products. Instructional Science, 41, 873–894. doi:10.1007/s11251-013-9263-8. Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online sources. Cognition and Instruction, 30, 39–85. doi:10.1080/07370008.2011.636495. Barzilai, S., Tzadok, E., & Eshet-Alkalai, Y. (2015). Sourcing while reading divergent expert accounts: Pathways from views of knowing to written argumentation. Instructional Science, 43, 737–766. doi:10.1007/ s11251-015-9359-4. Bendixen, L. D., & Rule, D. C. (2004). An integrative approach to personal epistemology: A guiding model. Educational Psychologist, 39, 69–80. doi:10.1207/s15326985ep3901_7. Bråten, I., Anmarkrud, Ø, Brandmo, C., & Strømsø, H. I. (2014). Developing and testing a model of direct and indirect relationships between individual differences, processing, and multiple-text comprehension. Learning and Instruction, 30, 9–24. doi:10.1016/j.learninstruc.2013.11.002. Bråten, I., Britt, M. A., Strømsø, H. I., & Rouet, J. F. (2011). The role of epistemic beliefs in the comprehension of multiple expository texts: Toward an integrated model. Educational Psychologist, 46, 48–70. doi:10.108 0/00461520.2011.538647. Bråten, I., Ferguson, L. E., Strømsø, H. I., & Anmarkrud, Ø. (2014). Students working with multiple conflicting documents on a scientific issue: Relations between epistemic cognition while reading and sourcing and argumentation in essays. British Journal of Educational Psychology, 84, 58–85. doi:10.1111/bjep.12005. Chi, M. T. H. (2009). Active-constructive-interactive: A conceptual framework for differentiating learning activities. Topics in Cognitive Science, 1, 73–105. doi:10.1111/j.1756-8765.2008.01005.x. Cote N., Goldman S. R., & Saul, E. U. (1998). Students making sense of informational text: Relations between processing and representation. Discourse Processes, 25, 1–53. doi:10.1080/01638539809545019. Daley, S., Willett, J. B., & Fischer, K. W. (2014). Emotional response during reading: Physiological responses predict real-time reading comprehension. Journal of Educational Psychology, 106, 132–143. doi:10.1037/ a0033408. Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87, 215–251. doi:10.1037/0033-295X.87.3.215.

444  •  Mason and Florit Ferguson, L. E., Bråten, I., & Strømsø, H. I. (2012). Epistemic cognition when students read multiple documents containing conflicting scientific evidence: A think-aloud study. Learning and Instruction, 22, 103–120. doi:10.1016/j.learninstruc.2011.08.002. Gerjets, P., Kammerer, Y., & Werner, B. (2011). Measuring spontaneous and instructed evaluation processes during web search: Integrating concurrent thinking-aloud protocols and eye-tracking data. Learning and Instruction, 21, 220–231. doi:10.1016/j.learninstruc.2010.02.005. Goldman, S. R., Braasch, J. L. G, Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381. doi:10.1002/RRQ.027. Goldman, S. R., & Scardamalia, M. (2013). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities, Cognition and Instruction, 31, 255–269. doi:10.1080/10824669.2013.773217. Greene, J. A., Azevedo, R., & Torney-Purta, J. (2008). Modeling epistemic and ontological cognition: Philosophical perspectives and methodological directions. Educational Psychologist, 43, 142–160. doi:10.1080/00461520802178458. Greene, J. A., Sandoval, W. A., & Bråten, I. (2016). An introduction to epistemic cognition. In J. A. Greene, W. A. Sandoval, & I. Bråten (Eds.), Handbook of epistemic cognition (pp. 1–15). New York: Routledge. Greene, J. A., & Yu, S. B. (2014). Modeling and measuring epistemic cognition: A qualitative re-investigation. Contemporary Educational Psychology, 39, 12–28. doi:10.1016/j.cedpsych.2013.10.002. Hagen, Å. M., Braasch, J. L. G., & Bråten, I. (2014). Relationships between spontaneous note-taking, selfreported strategies and comprehension when reading multiple texts in different task conditions. Journal of Research in Reading, 37, 141–157. doi:10.1111/j.1467-9817.2012.01536.x. Hyönä, J., Lorch, R. F., & Rinck, M. (2003). Eye movement measures to study global text processing. In J. Hyönä, R. Radach, & H. Deubel (Eds.), The mind’s eye: Cognitive and applied aspects of eye movement research (pp. 313–334). Amsterdam: Elsevier. Hyönä, J., & Nurminen, A.-M. (2006). Do adult readers know how they read? Evidence from eye movement patterns and verbal reports. British Journal of Educational Psychology, 97, 31–50. doi:10.1348/0007126 05x53678. Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87, 329–354. doi:10.1037/0033-295X.87.4.329. Kammerer, Y., Amann, D. G., & Gerjets, P. (2015). When adults without university education search the Internet for health information: The roles of Internet-specific epistemic beliefs and a source evaluation intervention. Computers in Human Behavior, 48, 297–309. doi:10.1016/j.chb.2015.01.045. Kammerer, Y., & Gerjets, P. (2012). Effects of search interface and Internet-specific epistemic beliefs on source evaluations during Web search for medical information: An eye-tracking study. Behaviour & Information Technology, 31, 83–97. doi:10.1080/0144929X.2011.599040. Kammerer, Y., Kalbfell, E., & Gerjets, P. (2016). Is this information source commercially biased? How contradictions between web pages stimulate the consideration of source information. Discourse Processes, 53, 430–456. doi:10.1080/0163853X.2016.1169968. Kendeou, P., & van den Broek, P. (2007). The effects of prior knowledge and text structure on comprehension processes during reading of scientific texts. Memory & Cognition, 35, 1567–1577. doi:10.3758/BF03193491. Kiewra, K. A. (1991). Aids to lecture learning. Educational Psychologist, 26, 37–53. doi:10.1207/ s15326985ep2601_3. Kiewra, K. A., Dubois, N. F., Christian, D., McShane, A., Meuerhoffer, M., & Roskelley, D. (1991). Effects of repetition on recall and note-taking: Strategies for learning from lectures. Journal of Educational Psychology, 83, 120–123. doi:10.1037/0022-0663.83.1.120. Kobayashi, K. (2009a). Comprehension of relations among controversial texts: Effects of external strategy use. Instructional Science, 37, 311–324. doi:10.1007/s11251-007-9041-6. Kobayashi, K. (2009b). The influence of topic knowledge, external strategy use, and college experience on students’ comprehension of controversial texts. Learning and Individual Differences, 19, 130–134. doi:10.1016/j.lindif.2008.06.001. Kreibig, S., & Gendolla, G. H. E. (2014). Autonomic nervous system measurement of emotion in education and achievement settings. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 625−642). New York: Routledge. Kuhn, D., Cheney, R., & Weinstock, M. (2000). The development of epistemological understanding. Cognitive Development, 15(3), 309–328. doi:10.1016/S0885-2014(00)00030-7.

Complementary Methods  •  445 Louca, L., Elby, A., Hammer, D. M., & Kagey, T. (2004). Epistemological resources: Applying a new epistemological framework to science instruction. Educational Psychologist, 39, 57–68. doi:10.1207/s15326985 ep3901_6. Mason, L. (2016). Psychological perspectives on measuring epistemic cognition. In J. A. Greene, W. A., Sandoval, & I. Bråten (Eds.) Handbook of epistemic cognition (pp. 375–392). New York: Routledge. Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action: Spontaneous reflections about knowledge and knowing during online information searching and their influence on learning. Learning and Instruction, 21, 137–151. doi:10.1016/j.learninstruc.2010.01.001. Mason, L., Boldrin, A., & Ariasi, A. (2010). Searching the Web to learn about a controversial topic: Are students epistemically active? Instructional Science, 38, 607–633. doi:10.1007/s11251-008-9089-y. Mason, L., Junyent A. A., & Tornatora, M. C. (2014). Epistemic evaluation and comprehension of web-source information on controversial science-related topics: Effects of a short-term instructional intervention. Computers & Education, 76, 143–157. doi:10.1016/j.compedu.2014.03.016. Mason, L., Pluchino, P., & Ariasi, N. (2014). Reading information about a scientific phenomenon on webpages varying for reliability: An eye-movement analysis. Educational Technology Research & Development, 62, 663–685. doi:10.1007/s11423-014-9356-3. Mason, L., Pluchino, P., & Tornatora, M. C. (2015). Eye-movement modeling of text and picture integration during reading: Effects on processing and learning. Contemporary Educational Psychology, 41, 172–187. doi:10.1016/j.cedpsych.2015.01.004. Mason, L., Scrimin, S., Tornatora, M. C., & Zaccoletti, S. (2017). Emotional reactivity and comprehension of multiple online texts. Learning and Individual Differences, 58, 10–21. doi:10.1016/j.lindif.2017.07.002 Mason, L., Scrimin, S., Tornatora, M. C., Suitner, C., & Moè, A. (2918). Internet source evaluation: The role of implicit associations and psychophysiological self-regulation. Computers & Education, 119, 59–75. doi:10.1016/j.compedu.2017.12.009 Mason, L., Tornatora, M. C., & Pluchino, P. (2015). Integrative processing of verbal and graphical information during re-reading predicts learning from illustrated text: an eye-movement study. Reading and Writing, 28, 851–872. doi:10.1007/s11145-015-9552-5. McCrudden, M. T., Stenseth, T., Bråten, I., & Strømsø, H. I. (2016). The effects of topic familiarity, author expertise, and content relevance on Norwegian students’ document selection: A mixed methods study. Journal of Educational Psychology, 108, 147–162. doi:10.1037/edu0000057. Porges, S. W. (2007). A phylogenetic journey through the vague and ambiguous Xth cranial nerve: A commentary on contemporary heart rate variability research. Biological Psychology, 74, 301–307. doi:10.1016/j. biopsycho.2006.08.007. Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading. Hillsdale, NJ: Erlbaum. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372–422. doi:10.1037/0033-2909.124.3.372. Rouet, J.-F., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 19–52). Greenwich, CT: Information Age. Salmerón, L., Gil, L., Carrión, A., & Bråten, I. (2016). Is eye-tracking useful to assess sourcing in multiple documents? Paper presented at the SIG meeting on Online Measures of Learning Processes of the European Association for Research on Leaning and Instruction, Oulu, Finland. Salmerón, L., Kammerer, Y., & García-Carríon, P. (2013). Searching the Web for conflicting topics: Page and user factors. Computers in Human Behavior, 29, 2161–2171. doi:10.1016/j.chb.2013.04.034. Sandoval, W. A. (2005). Understanding students’ practical epistemologies and their influence on learning through inquiry. Science Education, 89, 634–656. doi:10.1002/sce.20065. Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82, 498–504. doi:10.1037/0022-0663.82.3.498. Schroeder, S. (2011). What readers have and do: Effects of students’ verbal ability and reading time components on comprehension with and without text availability. Journal of Educational Psychology, 103, 877–896. doi:10.1037/a0023731. Schroeder, S., Richter, T., McElvany, N., Hachfeld, A., Baumert, J., Schnotz, W., . . . Ulrich, M. (2011). Teachers’ beliefs, instructional behaviors, and students’ engagement in learning from texts with instructional pictures. Learning and Instruction, 21, 403–415. doi:10.1016/j.learninstruc.2010.06.001. Scrimin, S., Patron, E., Ruli, E., Altoè, G., & Mason, L. (under review). Psychophysiological correlates of text comprehension in relation to reading goals.

446  •  Mason and Florit Sinatra, G. M. (2016). Thoughts on knowledge about thinking and knowledge. In J. A. Greene, W. A. Sandoval, & I. Bråten, I. (Eds.), Handbook of epistemic cognition (pp. 479–491). New York: Routledge. Slotte, V., & Lonka, K. (1999). Review and process effects of spontaneous note-taking on text comprehension. Contemporary Educational Psychology, 24, 1–20. doi:10.1006/ceps.1998.0980. Stine-Morrow, E. A. L., Miller, L. M. S., Gagne, D. D., & Hertzog, C. (2008). Self-regulated reading in adulthood. Psychology and Aging, 23, 131–153. doi:10.1037/0882-7974.23.1.131. Strømsø, H. I., Bråten, I., Britt, A. M., & Ferguson, L. E. (2013). Spontaneous sourcing among students reading multiple documents. Cognition and Instruction, 31, 176–203. doi:10.1080/07370008.2013.769994. Strømsø, H. I., Bråten, I., & Samuelstuen, M. (2003). Students’ strategic use of multiple sources during expository text reading: A longitudinal think-aloud study. Cognition and Instruction, 21, 113–147. doi:10.1207/ S1532690XCI2102_01. van den Broek, P., Bohn-Gettler, C., Kendeou, P., Carlson, S., & White, M. J. (2011). When a reader meets a text: The role of standards of coherence in reading comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 123–139). Greenwich, CT: Information Age. van Gog, T., & Scheiter, K. (2010). Eye tracking as a tool to study and enhance multimedia learning. Learning and Instruction, 20, 95–99. doi:10.1016/j.learninstruc.2009.02.009. Veenman, M. V. J., Van Hout-Wolters, B. H. A. M., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and Learning, 1, 3–14. doi:10.1007/ s11409-006-6893-0. Walraven, A., Brand-Gruwel, S., & Boshuizen, H. P. A. (2009). How students evaluate information and sources when searching the World Wide Web for information. Computers & Education, 52, 234–246. doi:10.1016/j. compedu.2008.08.003. Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009). Source evaluation, comprehension, and learning in Internet science inquiry tasks. American Educational Research Journal, 46, 1060–1106. doi:10.3102/0002831209333183.

25

SCENARIO-BASED ASSESSMENT OF MULTIPLE SOURCE USE John Sabatini, Tenaha O’Reilly, Zuowei Wang, and Kelsey Dreier educational testing service, usa

PROLOGUE This book is a testament to the increasing role and importance of multiple source use in everyday and academic literacy activities in the 21st century. How should we introduce the topic of multiple sources here? For most readers, we need not, because this chapter is not their first literacy stop in the volume, so the topic has been adequately covered in other sources. As authors, we are keenly aware that our readers already may have developed mental models and critical stances that will influence their understanding and interpretation of the content we are about to present to them. No one is a tabula rasa. In the spirit of multiple source literature, we can only wonder whether the reader’s aims correspond to the intended aims of this chapter, whether the text is relevant for their purpose for reading, and whether the mental models formed from reading prior chapters conflict or otherwise influence their interpretations of ours.

OVERVIEW OF THE CHAPTER Our plan is as follows. First, we present a recent and widely accepted conceptualization of 21st-century constructs for understanding of single and multiple-text sources, the MD-TRACE model (Rouet & Britt, 2011), as an analytic framework for thinking about assessment of multiple source use. In doing so, we acknowledge that the citations across this entire volume represent key sources that should inform a comprehensive assessment framework, but space limitations preclude integrating more of them into this chapter. Next, we describe, explain, and justify the use of scenario-based assessments (SBAs) as an approach to measuring multiple source use. The knowledge, skills, strategies,

447

448  •  Sabatini et al.

and dispositions necessary for proficiency in multiple source use pose a challenge to traditional assessment designs. On the one hand, multiple-choice paradigms that have privileged discrete, independent items and tasks tend to make it difficult to elicit the complex, cross-source, cognitive activities that are core to multiple source processing (Sabatini, Petscher, O’Reilly, & Truckenmiller, 2015). On the other hand, traditional constructed response (mostly essay-based) and performance assessments have analogous challenges and limitations related to reliability, bias, and added value (Hift, 2014; Kafer, 2002; Lukhele, Thissen, & Wainer, 1994). Specifically, most exemplars of this approach in use today, such as the College and Work Readiness Assessment (or CWRA+) (Council for Aid to Education, 2017), combine multiple source reading and writing skills that culminate in a single, complex, writing task. We view the cluster of techniques that comprise SBAs as an alternative approach to assessment design that can be used to mediate or overcome many of the challenges and limitations associated with measuring multiple source use. We discuss different conceptualizations of SBAs in research by describing several of the most developed research programs and exemplars of SBAs. We also discuss desired consequences of using SBA approaches to impact learning and instruction, and how they are being applied in operational testing programs.

MULTIPLE SOURCE USE CONSTRUCTS: WHAT IS IT AND HOW CAN WE ASSESS IT? Multiple sources are not a new phenomenon to reading instruction or even reading assessment. For example, conducting a literature review, which requires accessing, evaluating, and synthesizing multiple sources, is a staple activity taught in schools. The Advanced Placement history exam test routinely includes a “document-based” free response section that has students analyze and synthesize a set of multiple documents in order to explain a key historical event (College Board, 2017). What has changed is the volume and diversity of sources available, as a result of the advent of the Internet and World Wide Web (Leu, Kinzer, Coiro, & Cammack, 2004). In turn, renewed interest in cognitive research on document use has informed more complex, multi-source models (Kirsch & Mosenthal, 1990; Mosenthal & Kirsch, 1991; Rouet, 2006). This has led to the need to expand the construct coverage expected in the assessment of reading comprehension expertise to take into account the wide repertoire of flexible and differentiated processes that are needed to achieve complex task goals that require examining multiple-text sources (Rouet & Britt, 2011; Sabatini, O’Reilly, & Deane, 2013). In this paper, we draw inspiration from the PISA reading framework in defining multiple source comprehension. Multiple sources are defined here as any collection of texts that have been written by a single author (or co-authors), but published across multiple time points, or any collection of texts written by different authors. These could be dynamic (e.g., hyperlinked) or static texts, printed texts, emails, blogs, webpages, or other digital sources. They can also include multimedia such as audio files, pictures, and videos. The diversity of text type and modality, coupled with an increased level of access to multiple sources, has placed additional demands on attention and resource allocation for the 21st-century reader.

Scenario-Based Assessment  •  449

During single-text comprehension tasks, some of the processes involved in multiple source comprehension are not called upon or are less complex to deploy, as a single source “should” be internally coherent in terms of its goals, arguments, and intended audience. While single texts may introduce controversies, they are usually written from one perspective and the related information is within close proximity. In contrast, when reading multiple sources, the reader usually has a specific goal directed at answering an overarching question, of which, only some sources are relevant or select elements of a source. Importantly, the reader’s goals may differ from the author’s intended goal for writing, requiring additional processes to identify, select, and interpret information relevant to the reader’s aims. In multiple source processing, readers need to find information that is relevant to their goal, evaluate it for credibility, and corroborate sources to achieve their aims. Sources may support some points, while other sources may contradict each other. While some of these processes are required in single-source reading, the source author often provides guidance regarding integrating, synthesizing, or representing conflicting information for the reader, presenting it from a single point of view, or explaining when different points of view are being represented. While the demand for multiple source skills has increased with wide access and use of the Internet, how are multiple source skills assessed? To address this, we need to understand the goals and purposes of assessments. Assessments are used for many purposes, but chief among these are 1) to evaluate whether and how much individuals have learned or achieved in a domain; 2) to predict how well or whether individuals can apply what they know and can do in a context of use; or 3) to aid or guide instructional decisions and learning. For the first purpose, it may be sufficient to examine recall of taught knowledge and skills in a relatively decontextualized manner. In this case, the student is expected to act independently and no support for learning is provided. Conclusions drawn from such an assessment would indicate whether the student has learned factual or maybe even conceptual content. However, when one is assessing the application of skills when thinking or reasoning about content learned, then a different assessment strategy may be required. One could ask the learner to complete a complex and integrated task (e.g., write an essay that evaluates and integrates multiple sources), such that skills are called upon to engage and solve a novel problem. Further, for inferences about depth of understanding in a context of use (the second purpose of assessment), ecological validity in the assessment strengthens the validity of claims that individuals can apply what they have learned and that the construct has been adequately measured. That is, we might ask whether the scores produced by a decontextualized assessment transfer to more realistic settings. In addition, when we want to inform future learning (the third purpose), we may want to know whether a low score on a complex task, such as an essay, essentially means the student did not have any of the sub-skills that feed into the more complex task. In this chapter, we describe a new type of assessment called Scenario-Based Assessment (SBA) that provides both a framework for assessing multiple source use and takes a step toward ecological validity in assessments, in that it establishes credible literacy purposes or goals for the individual, and a structured, sequenced set of tasks or subgoals toward achieving those purposes. This also aligns such assessments better with contemporary models of discourse processing and reading comprehension

450  •  Sabatini et al.

(e.g., Kintsch, 1998; Rouet, 2006; Rouet & Britt, 2011). In this way, SBAs can sample both complex, integrated performances (Can a student evaluate and integrate information independently?) and some of the key sub-skills that support those performances (Can an assessment support and measure partial skill development, e.g., evaluate the credibility of a website?). The SBAs reviewed in this chapter take a further step, attempting to reflect what Bennett (2010) calls “assessments as learning”, that is, assessments that serve as models of or aids to learning and instruction.

MD-TRACE MODEL: DECONSTRUCTING THE CONSTRUCT OF MULTIPLE SOURCE USE As stated earlier, one of the goals of assessment is to measure a student’s independent performance on a task. Another goal of assessment is to support learning that may also involve obtaining information on whether a student has mastered any of the subcomponents of the more complex skill. To achieve this second aim, it would be useful to deconstruct and identify the essential elements of multiple source use so that assessments can be designed to measure the key sub-skills. To illustrate how multiple sources fit into assessment contexts, we employ a well-known, cognitive framework of multiple source processing, specifically the Multiple Documents – Text-based Relevance Assessment and Content Extraction (MD-TRACE) model (Rouet & Britt, 2011). To summarize, the MD-TRACE model is composed of internal and external resources, and cognitive activities described as a set of steps or processes. The external resources consist of the external task requirements; search devices, source material, and text organizers; document contents; and readergenerated products (such as notes, summaries, or essays). The internal resources consist of the task model (the internal representation of the external task requirements) and the documents model (the representation that is the product of the cognitive operations). These internal resources are moderated by prior knowledge, reading/ search skills, and self-regulation skills. The MD-TRACE cognitive activities are decomposed into five interacting processing steps: 1) create and update a task model; 2) assess information needs; 3a) assess item relevance; 3b) process text contents; 3c) create/update a documents model; 4) create/update a task product; 5) assess whether the product meets the task goals. The authors note that the steps may occur out of order or in parallel in actual task performance. Embedded in each of these steps are multiple cognitive activities, plans, evaluations, and decisions.

FROM COGNITIVE MODEL TO ASSESSMENT DESIGN Now is a good moment to step back and remind ourselves of the relevant goals of assessment, in contrast to a cognitive model. A cognitive model is a description or explanation of a process, in this case multiple source processing. An assessment is an information gathering tool. So, while the MD-TRACE model is a detailed description of the reading sub-skills, processes, and strategies necessary to use multiple sources to achieve a goal, it is precisely the question of whether, or to what extent, a student possesses each sub-skill that the assessments we design seek to uncover.

Scenario-Based Assessment  •  451

In outcome tests, one is first and foremost interested in whether the individual has the relevant cognitive knowledge, skills, strategies, and disposition (hereafter simplified to cognitive skills). Typically, one derives a score that represents a point on a continuum of proficiency. Ideally, participants are required to apply their cognitive skills in task and text sets that are similar to or at least predictive of performance in applied, real-world settings. Steps to minimize the influence of construct irrelevant variance or bias are taken, for example, trying to reduce the influence of background knowledge by using “familiar topics”, using standard administration or scoring procedures, and avoiding sensitive topics that could disadvantage some groups or individuals. Constraints are put in place to ensure reliable, consistent scoring including the use of multiple-choice items, restricting the search and use of outside documents, creating effective scoring rubrics, and conducting training sessions for raters to ensure high inter-rater reliability of constructed responses (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014). Collectively, the focus of these measurement standards is to ensure that the product of comprehension, defined here as the responses and the scores derived from them, yield reliable and valid inferences of the test taker’s relative proficiency on the construct of interest. Of far less interest in traditional testing paradigms, is the process by which the test taker responds, or which steps, activities, or processes (or sub-skills) yielded the correct versus incorrect responses. In other words, traditional assessment is concerned more about whether a student is proficient, and less about how or why a particular student received a score, or whether a student was able to do parts of a more complex task.

THE ASSESSMENT PARADOX: HOW TO MEASURE AND SUPPORT MULTIPLE SOURCE USE The above discussion illustrates the discrepancy between what is valued in cognitive models of reading and what is measured by traditional reading assessments. Theoretical models such as the MD-TRACE place emphasis on the growing importance of multiple source use in today’s digital world. The model outlines the key skills and processing steps that proficient readers need to follow to successfully undertake the metacognitive, evaluative, and integrative mindset for 21st-century multiple source reading environments. In short, the model presents a stance on what is important to measure, and underscores the importance of the component processes that are applied in achieving proficient performance. In contrast, traditional assessment paradigms prioritize measuring reading ability in an efficient and cost-effective way. In most current tests, this has meant measuring student ability to comprehend single passages in isolation. Passages are chosen to not demand much topical, background knowledge and the passages are not intended to be related to one another. There is no overarching goal for reading other than to answer the questions accurately (Rupp, Ferne, & Choi, 2006), and the questions are also assumed to be independent of each other. This is not to say that traditional assessments of reading ability are not valid for some inferences of proficiency; the assessments typically have strong psychometric properties and the scores are predictive of success on a number of metrics. However, this paradigmatic design poses significant constraints in adapting to a changing, multiple source literacy environment.

452  •  Sabatini et al.

The key challenge for assessment designers, then, is how to assess modern constructs (and sub-constructs) of reading including constructs such as multiple source use, while simultaneously providing information that is psychometrically sound. It would also be of value to gather evidence of the process that leads to the final goal (e.g., understanding the goal, assess document relevance), preferably without the need to parse each sub-skill into discrete, decontextualized items. In SBA designs, both process and product can be considered (O’Reilly & Sabatini, 2013; Sabatini et al., 2013). One can apply cognitive and learning science insights in the assessment design with an aim of enhancing the assessment’s instructional relevance and construct coverage. SBAs have the potential to enhance such construct and instructional utility and are especially well suited for multiple source assessment, over traditional testing paradigms. In the remainder of the chapter, we describe three well-developed research programs that have pioneered the development of SBAs and how they address the construct of multiple source use.

THE GLOBAL, INTEGRATED SCENARIOBASED ASSESSMENT (GISA) APPROACH The scenario-based, reading comprehension assessments which we call global, integrated scenario-based assessments (GISA) can be useful for achieving a number of such construct and process goals, while maintaining psychometric integrity. While the approach was not designed to explicitly measure multiple source use, many of the design features can be used to both support and measure many of the key elements of multiple source comprehension. O’Reilly and Sabatini (2013) defined SBAs as a collection of techniques that allow test designers to structure tasks and items to enhance construct coverage using valid task designs, with a goal of enhancing instructional value. Unlike traditional assessments that present texts and tasks in a discrete manner, the GISA approach to SBAs organize the texts and tasks into units of integrated activities, rather than a collection of stand-alone items. This approach to structuring and sequencing makes the SBA inherently amenable to assessing the steps of multiple source use.

KEY FEATURES OF GISA The key features of the GISA SBA approach include: 1) an initial goal and context for reading; 2) a collection of thematically related sources; 3) a set of simulated social agents (i.e., simulated peers, teacher); and 4) a set of techniques to model good reading habits, while simultaneously providing opportunities for students to demonstrate their partial skill development. In addition, because reading is an integrated activity, what we call performance moderators are also included in the design. The performance moderators of background knowledge, motivation, metacognition/selfregulation, and reading strategies are not directly considered a part of the construct, but may impact reading performance (O’Reilly & Sabatini, 2013). In many GISA assessments, background knowledge is directly measured and we look for indicators of student motivation. Reading strategies are often incorporated into GISA forms as specific reading tasks (e.g., summary, graphic organizer).

Scenario-Based Assessment  •  453

HOW THE FEATURES ARE ASSEMBLED IN AN ASSESSMENT CONTEXT How are these features implemented? In a typical GISA form, students are given a specific goal for reading a collection of thematically related materials (e.g., should your school adopt a clean energy program). The sources are chosen to be diverse in terms of format (e.g., blog, email, website, textbook passage), depth (e.g., comprehensive, selective), trustworthiness (e.g., reliable or unreliable sources), and accuracy (e.g., contains errors or misconceptions). This diversity is not only used to set the stage for multiple source use, but also to engage students in critical thinking, encourage perspective taking, and broaden students’ awareness and appreciation of genre similarities and differences. Learning new ideas across different text formats and contexts is also designed to promote transfer by not restricting the conditions of learning to a single source. While the sources are diverse on a number of dimensions, they are all related to each other at some level in connection to the reading goal. Tasks and items in an SBA require the students to engage in both traditional “single-text” forms of reading (e.g., identify key ideas, draw local inferences) and more demanding multiple source tasks. Multiple source tasks may require integrating and synthesizing cross-textual information, evaluating source utility and trustworthiness, identifying and potentially resolving conflicting claims and evidence, or making decisions about how to apply text content in new situations or contexts (transfer). In GISA forms, all of these activities are thematically related to achieving the larger, scenario goal. These features of GISA SBAs enable designs that extend beyond the traditional, discrete item paradigms of reading assessment, better aligning the assessment with modern models of goal-driven, multiple source processing accounts of reading (e.g., Magliano, McCrudden, Rouet, & Sabatini, 2018; Rouet & Britt, 2011). In addition, GISA forms are designed to model good reading habits, as well as provide opportunities for test takers to demonstrate partial skill development (e.g., the stages in the MD-TRACE model).

ILLUSTRATING MULTIPLE SOURCE USE THROUGH A GISA EXAMPLE: CONNECTING THE MD-TRACE MODEL TO ASSESSMENT To illustrate how some of these features work together, we briefly describe the structure and sections (tasks) of a GISA form on the topic of community gardens, and how these GISA sections roughly correspond to stages of the MD-TRACE model (see Table 25.1). While the GISA form was designed to measure constructs that go beyond multiple source use, it does cover many aspects of the key sub-skills involved in multiple source use. Table 25.1 includes the section number, name, and intended key function. The key function describes what the section was designed to do, which includes, and may go beyond, multiple source use constructs. The table also includes a column that identifies the elements of the MD-TRACE that roughly correspond to the GISA sections of the community gardens assessment. According to the MD-TRACE model, students undertake multiple activities in the course of reading and understanding multiple source texts. Some of these are strategic in nature, and may involve interactions outside of text comprehension in a

454  •  Sabatini et al. Table 25.1  Overview of the Structure of the GISA Community Gardens Assessment Including the Key Stages of the Assessment, Their Basic Function, and the Relationship to the MD-TRACE Model. GISA section number

GISA section name

1

Introduction

2 3

4 5 6 7

Key function of the GISA section

Rough correspondence to relevant MD-TRACE stage

To provide the goal and sub-goals of Create and update a task model the assessment; provide an overview of the task product; and to introduce the simulated peers and teacher Background To measure students’ knowledge on Internal resources are moderated knowledge the topic of the sources by prior knowledge, reading/ assessment search skills, and self-regulation Text 1: The To build up students’ knowledge Process single-text contents; build controversy of the topic and to assess basic prior knowledge understanding; to model selfregulation through peer dialogue Web links To assess students’ ability to evaluate Assess information needs the relevance of source material Assess item relevance Text 2: To deepen students’ understanding Process text contents Community of the issues and to assess learning; Update task model gardens multiple source comprehension Create/update a documents model Text 3: To present a counterargument; Process text contents Opposing view multiple source comprehension Update documents model Produce a flyer Culminating task that communicates a Create/update a task product position and supports it with evidence

more restricted sense (e.g., defining a task model, defining information needs, assessing item relevance, and assessing whether a task product meets task requirements). Other activities may involve skills characteristic of a traditional, single-source reading construct (processing text content), while others require critical thinking and written communication (e.g., building a documents model and creating a task product). How does the GISA SBA align with this model?

SECTION 1: SETTING UP THE TASK MODEL – WHAT ARE STUDENTS SUPPOSED TO DO AND PRODUCE? The community gardens assessment is a 45-minute, computer-delivered assessment designed for use with 5th to 6th-grade students. In the introductory section of the assessment, students are given the goal of helping decide whether to build a community garden on a vacant lot. Subgoals include: find out more about community gardens; decide if they support the construction of the community garden; and prepare a flyer to inform community members of their recommendation. During this process, simulated peers are introduced who help define task goals, support the test taker by providing hints, or provide stimuli that need to be evaluated by the test taker. In the introductory section, the test taker is also introduced to a simulated teacher who will provide guidance as the test taker and simulated peers “work” on the community gardens project.

Scenario-Based Assessment  •  455

In the MD-TRACE model, this first section of the GISA corresponds to the “create a task model” step (Table 25.1, row 1). Note that the task model goals and subgoals are structured and organized for the students in this scenario. We do not directly assess students’ ability to formulate a complex task model, though future SBA forms could target those multiple source sub-skills in the design.

SECTION 2: MEASURING BACKGROUND KNOWLEDGE – WHAT DO STUDENTS KNOW ABOUT THE TOPIC? The thematic nature of a scenario-based assessment could be considered a limitation of the design. This is because students enter a scenario with different levels of prior knowledge about the topic, and with different levels of skill in performing each of the component tasks. This variability can cause students with very different skills profiles to perform similarly. For instance, a student with high knowledge of the topic but weak ability to integrate information from multiple sources might produce a final product of similar quality as a student who had low initial knowledge on the topic, but strong text integration skills (e.g., a quick learner of a new domain). The GISA approach is designed to capture information about different phases of this complex process, making it easier to develop skills profiles that suggest hypotheses about why students did well or poorly on a specific GISA form. For example, background knowledge is not in and of itself part of multiple-text reading comprehension, but it is a performance moderator (in the GISA framework and in MD-TRACE), since some students will enter a task with high or low knowledge, and may learn the content presented more or less quickly and completely as the task unfolds. To address the potential limitation of the thematic nature of GISA, the form is designed to take variability in levels of knowledge into account. Therefore, at the beginning of the assessment, students’ background knowledge is measured. In this case, the background knowledge concerns the topics of community gardens and farming in general. This step provides the test user with some evidence to determine whether the students had sufficient knowledge to understand the topic, or whether they had so much knowledge that the assessment is essentially a test of knowledge, rather than a test of comprehension. In addition, some of the background questions will be answered in the text of subsequent passages. This feature also allows the assessment to measure whether the students learn passage content from reading versus recalling it from prior knowledge (see Table 25.1, row 2).

SECTION 3: BUILDING UP STUDENTS’ UNDERSTANDING – SINGLE-SOURCE COMPREHENSION The GISA form is also sequenced to build background knowledge up over the course of the assessment (Table 25.1, row 3). For instance, an initial text introduces the topic of community gardens. To support (and evaluate) test takers’ understanding of key ideas in this introductory text, we present a sequence of tasks that probes their global understanding. In addition, to support low-knowledge readers, the peers engage in a dialogue that explains what a community garden is. Students are also introduced to the controversy that drives the scenario goal: some groups want to build a playground on the vacant lot; others want to build a community garden on the lot.

456  •  Sabatini et al.

Students may have weak, single-text reading skills, or may be inclined to put as little effort into reading as is feasible given the task they are assigned. The GISA SBA addresses motivation by providing a more meaningful, and scaffolded, task sequence, while measuring single-text reading comprehension with tasks (such as summarization) that also encourage students to build the deeper knowledge they will need during task integration as the assessment unfolds. For example, in a fourpart sequence, we gather information on whether test takers can write a summary independently, whether they can detect if a fellow student’s summary violates one of the guidelines, whether they can locate where in the summary the violation occurred, and whether they can fix the error. Such sequencing is useful for identifying what parts of the more complex task a student could or could not do. Collectively, this section is related to the “process text contents” part of the MD-TRACE model, but also provides clues as to whether test takers have developed some accuracy in their mental model of the single text, when later they are required to integrate this knowledge into a multiple source use task where it is applied (Gil, Bråten, Vidal-Abarca, & Strømsø, 2010). The above sequence is intended to measure basic single-text understanding (Table 25.1, row 3). Other tasks in the section are designed to help test takers formulate their argument, and can be thought of as fostering/assessing multiple source processing. For instance, the test taker is asked to complete a graphic organizer (a reading strategy) that outlines who supports a particular position and whether the text provides information to make the position judgment. Requiring that students recognize who supports what position aligns with assessing multiple source use.

SECTION 4: EVALUATING WEB LINKS – ASSESS ITEM RELEVANCE After the introductory text and tasks are presented, the test taker is given a list of simulated websites that contain the URL and a short description similar to what one would find in a typical search engine output (Table 25.1, row 4). Some of the sites are relevant to the task goal and others are not. The test taker is asked to identify the relevant sources, as well as engage in some perspective taking tasks. A subsequent task asks the test taker to identify the parts of an actual website that are useful toward their goal.

SECTION 5: GAINING A DEEPER UNDERSTANDING OF THE TOPIC – UPDATE TASK MODEL AND CREATE A DOCUMENTS MODEL Next, test takers are given a second, detailed article that explains more about community gardens, what they are, why people create them, and their relation to real-world problems, such as food deserts. This second text is key for building an argument supporting the creation of a community garden (Table 25.1, row 5). Items in this section measure understanding through the use of graphic organizers, identifying causes and effects, and identifying correct and incorrect paraphrases of key information. Also part of this section are items tapping knowledge of key vocabulary that was formerly presented in the background knowledge section. Here, we measure if students learned

Scenario-Based Assessment  •  457

formerly unknown terms (or whether they knew terms prior to taking the assessment). To support argumentation, another task asks students to classify statements that support a position.

SECTION 6: OPPOSING VIEWPOINTS AND COUNTERARGUMENT– UPDATE DOCUMENTS MODEL The subsequent task presents a third text that opposes the building of the community garden and offers reasons for building a playground in its place (Table 25.1, row 6). This section requires the students to “process text contents” and “update their documents model” (in the MD-TRACE process) by presenting information that is not consistent with prior texts. Here, the test taker can compare the conflicting arguments across multiple sources.

SECTION 7: PRODUCE A FLYER – CREATE A TASK PRODUCT The culminating task requires the test takers to complete parts of a flyer (Table 25.1, row 7). In particular, they are asked to take a position and provide reasons that justify the position. In theory, test takers’ decision to take a side should be based on their ability to weigh the evidence on both sides of the argument as they consider the information across multiple sources. The first task requires the test taker to provide this information in a constructed response format (open ended), after which the test taker is given a second attempt, this time with selected responses. This sequence allows less skilled readers to demonstrate their partial skill development. The task is related to the “create/update a task product” part of the MD-TRACE model. In sum, this example from the community gardens GISA form was used to illustrate many of the features of SBAs and how they can be applied to measuring aspects of multiple source use. While multiple source measurement was not the sole construct targeted in the assessment, it does include tasks that call upon most of the processing steps of the MD-TRACE model. At the same time, it probes and monitors student performance on single-source texts, to help identify and distinguish single versus multiple source skill strengths versus weaknesses. We believe that the set of GISA SBA features, coupled with performance moderators, enables richer construct coverage of goal-driven, single and multiple source comprehension than discrete item, multiplechoice or single, culminating writing task test designs. While SBAs do take a lot of thought to design and implement, the extra effort may pay off and they are easier to build once initial designs are developed.

PROPERTIES OF GISA While there is much more work to be done, we have created over 20 SBAs for students in grades pre-K through 12th grade. Most of these SBAs have multiple source use tasks similar to those in the community garden form, though the difficulty and support for students vary across developmental levels. We have piloted them in several states and locales with large numbers of students. Our analyses reveal that the assessments demonstrate adequate psychometric properties including reliability,

458  •  Sabatini et al.

variability of scores, and appropriateness for the intended population (O’Reilly Weeks, Sabatini, Halderman, & Steinberg, 2014; Sabatini, Halderman, O’Reilly, & Weeks, 2016; Sabatini, O’Reilly, Halderman, & Bruce, 2014a, 2014b). We have also created a vertical scale among the GISA forms, which allows comparisons across grade levels and forms. In other words, even if students take different GISA forms, their scores can still be compared to each other, thanks to the vertical scale. This is useful in pre/post-intervention designs and to explore changes in development over time. In short, we believe the scenario-based assessment is a feasible way to measure reading comprehension inclusive of multiple source use.

APPLICATIONS OF GISA Elements of the GISA approach, including measurement of multiple source use, have been operationally implemented in the PISA reading literacy assessments. The triennial Programme for International Student Assessment (PISA) surveys 15-year-old students around the world, assessing the extent to which students near the end of compulsory education have acquired key knowledge and skills that are essential for full participation in modern societies. The PISA Reading Literacy Framework discusses and cites the GISA framework and designs of SBA tasks have been developed for use in the 2018 implementation of PISA, where reading literacy is the main cognitive focus.

OTHER SCENARIO-BASED ASSESSMENT RESEARCH PROGRAMS To our knowledge, there are only a limited number of other research programs currently investigating the use of scenario-based assessment of reading and writing that could also be considered measures of multiple source processing. We summarize the research of each program, with strong emphasis on how they define and operationalize SBAs to address elements of multiple source constructs. We do not review the work of Goldman and colleagues in this chapter as the research is covered elsewhere in this book (Goldman, Blair, & Burkett, this volume). The ORCA (Online Research and Comprehension Assessment) Project As a response to the proliferation of new literacies, the ORCA, Online Research and Comprehension Assessment, was created. ORCA was developed under a new literacies perspective and defines online reading comprehension as a “web‐based problem‐ solving inquiry process involving skills and strategies for locating, critically evaluating, synthesizing, and communicating information on the Internet” (Coiro, 2011, p. 352). The strategies involved in the forms are summarized by the acronym LESC, which stands for reading to locate, reading to evaluate, reading to synthesize, and reading and writing to communicate; several of the processes described by “LESC” align to the cognitive activities of the MD-TRACE model. Three types of ORCA forms were originally created, and each form was designed with a problem-solving scenario related to human biology, a subject familiar to most 7th-grade students, who provided the majority of the assessment’s sample population. The initial forms, ORCA-Open and ORCA-MC (multiple choice), were piloted first

Scenario-Based Assessment  •  459

in 7th-grade classrooms (Coiro & Dobler, 2007). The ORCA-Closed form was created and piloted later, based on the iSkills assessment (Katz, 2007) and the digital literacy assessment in PISA (Organisation for Economic Co-operation and Development, 2011). ORCA-Open lets students navigate the open internet while ORCA-Closed, similar to a scenario-based assessment, requires students to write a report within a simulated online environment that includes several internet capabilities like instant messaging, search engines, emails, and several web pages. ORCA-MC was created similarly and used the same content, except that all student response types were multiple-choice items. After extensive pilot research, the researchers decided that the ORCA-Closed and multiple choice had adequate feasibility and psychometric properties for continued development (Leu et al., 2014). The ORCA-Closed forms had students use multiple source skills in a scenario-based environment. Students were given a clear purpose – a problem-solving task related to issues in human biology, such as asthma, decorative contact lenses, or safe music volume levels. The overall task was presented in a Facebook-like interface, which involved multiple sources including a feed, instant messaging, and emailing. Students were given 45 minutes to use a simulated search engine, “Gloogle”, to locate, evaluate, synthesize, and communicate (LESC) information from multiple sources in order to complete the overall task. Students were given smaller scenario-related tasks throughout the test that helped lead to the final response. In the ORCA-MC forms, students were given a similar scenario, except instead of having free-range across sources, they were guided through several key stopping points that were aimed to also use students’ LESC skills. The scenarios presented in both ORCA-Closed and ORCA-MC required students to examine multiple sources from the “Gloogle” pseudo search engine. The scenarios called upon several multiple source skills such as purposeful reading, searching sources, evaluating sources, and perspective taking. General Properties of ORCA The ORCA-Closed and ORCA-MC demonstrate adequate reliability and validity, although ORCA-Closed had slightly higher reliability (Leu et al., 2014). The results from the forms indicate that offline and online reading comprehension skills each contributed to performance on reading tasks, which, following the theory of new literacies, suggests online reading comprehension involves skills beyond offline reading comprehension (Coiro, 2011). However, it was also found that the majority of students are not skilled in online reading comprehension, especially the ability to critically evaluate the information, which was the most difficult for students compared to the three other online reading skills assessed (Forzani & Maykel, 2013). One might infer that the multiple source evaluation and integration skills required in the ORCA tasks contributed to the challenges students had with the online reading skills, though that was not the authors’ research focus. Applications of ORCA The primary design elements of the ORCA designs have been incorporated into the ePIRLS design (Mullis & Martin, 2015). The Progress in International Reading Literacy Study (PIRLS) is conducted on a regular five-year cycle on a population of children

460  •  Sabatini et al.

in their fourth year of formal schooling. The ePIRLS was designed as an extension to the traditional paper-booklet PIRLS reading measures to assess reading in an online environment. The ePIRLS assessment consists of four, school-based, online reading tasks, each involving two to three different websites, with any student completing two of the four tasks in a 40-minute session. Thus, an SBA approach to multiple source measurement as represented in an online computer environment construct is being operationally implemented in the ePIRLS program. The Educational Testing Service (ETS) CBAL Initiative Another SBA approach that has an extensive history of development is called CBAL™, or Cognitively Based Assessment of, for, and as Learning. CBAL is a research initiative funded by ETS, aimed at creating “a model for an innovative K-12 assessment system that documents what students have achieved, facilitates instructional planning, and is considered by students and teachers to be a worthwhile educational experience in and of itself” (Bennett, 2010, p. 70). The aforementioned GISA SBAs owe their origins to research commenced as part of the CBAL initiative. Despite similarities between GISA and CBAL assessments, CBAL assessments place more emphasis on using assessments to facilitate learning. CBAL assessments not only serve as a documentation of what students have learned (of learning), but also help teachers with instructional planning (for learning), and provide a model for students to follow when learning a content area (as learning). While the ability to use multiple sources is one assessment target in GISA, multiple sources of information serve as a learning opportunity for students to learn the content area. This is because in reality, no content area can be learned with a single source of information. In other words, multiple sources are an inherent feature of learning. In CBAL assessments, materials from multiple sources are organized by cognitive models, including competency models and associated learning progressions, which were derived from cognitive and learning sciences research in English Language Arts (Deane, Sabatini, & O’Reilly, 2012), mathematics, and science domains. During CBAL assessments, students participate in extended scenario-based tasks that are created by modeling high-quality teaching practices that have been shown to improve classroom learning. These assessment scenarios not only help students learn the content while they go through the assessment, but also set up good examples for teachers to make their own instructional plans. By using materials from multiple sources, CBAL assessments also have advantages over traditional assessments in terms of the intended consequences of testing. Under the pressure of traditional high-stakes testing, teachers may focus their instruction on the test content. Although this may improve student performance on the test itself, it does not generalize to the content domain. This problem is partially mitigated in CBAL assessments, because the assessments are based on domain-specific competency models (e.g., Liu, Rogat, & Bertling, 2013; O’Reilly & Sheehan, 2009). The competency models include key practices, strategies, and habits of mind, and CBAL assessments are developed to represent these processes with scenarios that students may experience in real life. By using scenarios that involve multiple sources of information, the CBAL assessments promote learning gains that can potentially generalize to the broader content domain.

Scenario-Based Assessment  •  461

Multiple source materials are prepared for CBAL assessments through key practices of the related content area. A key practice is a class of activities in which students use their skills to carry out complex tasks within a purposeful social context (Deane et al., 2015). The CBAL ELA key practices includes 1) basic literacy skills such as reading, writing, listening, and speaking, 2) model building skills such as building and sharing knowledge (O’Reilly, Deane, & Sabatini, 2015), and 3) applied practices such as conducting inquiry and research (Sparks & Deane, 2015) and discussing and debating ideas (Deane & Song, 2015). Participating in key practices allows students to gain competence in a domain (Deane et al., 2015). The concept of a key practice originates from the social constructivist view of learning (Vygotsky, 1978), which emphasizes the importance of social interaction in cognitive development. Thus, key practices often involve interaction with peers and teachers. Following this reasoning, multiple source materials are developed by considering the activities of key practices. In short, the ETS CBAL initiative is aimed at creating the next generation assessments that not only measure student learning, but also facilitate it (Bennett, 2010). A typical CBAL SBA 1) provides a realistic purpose, 2) sequences tasks to follow hypothesized learning progressions of a domain and thus provides support for student performance, and 3) includes texts and information coming from multiple sources. CBAL assessments use scenarios to organize multiple assessment materials that reflect key practices in related content areas. Below we provide an example to illustrate these features. CBAL Example: SBA of Argumentation In CBAL SBAs that target argumentation skills (Bennett, Deane, & van Rijn, 2016), students are asked to write on controversial issues such as whether advertising to children under age 12 should be banned in the U.S. Following the learning progression of argumentation proposed by Deane and Song (2015), the assessment target four elements: 1) understand the issue, 2) consider positions, 3) create and evaluate arguments, and 4) organize and present arguments. First, students are presented with several source materials related to the topic and are asked to summarize the materials in preparation for using these documents in writing an argumentative essay (Gil et al., 2010). Following the summarization task, students work on an argument classification task, which requires them to classify people’s positions based on the reasons/ evidence they provide. The third task is an evidence classification task, which asks students to determine whether a piece of evidence supports or weakens a claim. This task also evaluates students’ ability to create and evaluate arguments. The fourth task requires students to write an argument essay. The task provides students access to the source documents reviewed in earlier tasks, along with a writer’s checklist to help them write the essay. This task addresses students’ ability to understand the issue, to create and evaluate arguments, and to organize and present arguments. Finally, a fifth task requires students to write a few sentences to identify logical flaws in example arguments on the issue. Across the portfolio of CBAL ELA SBA forms, the use of multiple sources is a staple design element. Students are typically provided with several document sources on the scenario topic. Initially, each source is accompanied by tasks probing specific learning progressions that target understanding of the individual text. This may be in

462  •  Sabatini et al.

the form of writing a summary, answering questions about key content, or evaluating claims and evidence. As the SBA progresses (typically in a second, 40-minute session), the multiple sources become foundational and are made available in completing an extended constructed response essay task such as writing an argument, framing a proposal, or creating a research synthesis. Applications of CBAL Several of the primary elements of the CBAL ELA designs have been incorporated into NAEP reading blocks. The National Assessment of Educational Progress (NAEP), sometimes referred to as the nation’s report card, is conducted on a regular cycle with a United States nationally representative sample of 4th, 8th, and 12th-grade students. Scenario-based tasks derived from principles and prototypes of CBAL ELA forms have been adapted for subsequent implementation in NAEP, which uses tablet delivery of assessments for the first time in 2018. Thus, an SBA approach to multiple sources measurement as represented in CBAL-style tasks, consistent with the NAEP reading framework, is being operationally implemented in the NAEP assessment program.

IMPLICATIONS FOR RESEARCH AND PRACTICE While still in its infancy as a research program, we see several key lines of research and application with respect to multiple source assessment, scenario-based assessment approaches, and their intersection. First, continued theoretical research is warranted to clarify what is common versus unique in processing of multiple sources in comparison to single-source analogs, with careful attention to how individual differences in ability and other characteristics may interact with performance. Assessments are best when targeted (with respect to construct) and efficient (with respect to time, cost, and effort). Multiple source assessment is likely to impact efficiency, so it is helpful to know when or whether inferences can be made from single to multiple source tasks, and vice versa. Also, within the multiple source construct, which elements are most valuable as targets of assessment and are there contingent relations among elements? Ideally, learning and instructional approaches to enhancing multiple source processing will be documented, as these serve as models for assessment scenarios. Future research is, of course, needed to clarify which features or techniques of SBAs are necessary or effective in achieving their intended goals of enhancing construct relevant processing, versus those that are ineffective or sources of construct irrelevant variance. SBA techniques and research are in their infancy – and agreed-upon definitions or descriptions of SBA elements are still emerging. In order for SBAs to be used at scale, research must demonstrate practicality, utility, and efficiency, along with psychometric reliability and efficiency (Haertel, 1999). Future applications of SBAs that are designed to model and inform instructional programs (e.g., GISA and CBAL) need to be evaluated to see whether they foster intended consequences; that is, they facilitate effective instructional approaches to multiple source use. One promise of SBAs is that they provide models of instruction and applications of reading skills. But a history of teaching to the test has mostly yielded unintended, negative consequences to instruction and learning (Au, 2007; Jones & Egley, 2004), so there is much research to be done to reverse this trend.

Scenario-Based Assessment  •  463

SUMMARY In this chapter, we have reviewed the application of multiple source constructs in scenario-based assessment approaches. We referenced the MD-TRACE model as a basis for analyzing how scenario-based assessment designs encourage multiple sources processing. We also reviewed three prominent research programs that are developing and evaluating SBAs – GISA, ORCA, and CBAL. We noted how each of these research programs has been influential in impacting the test approaches of national and international testing programs. We hope that this chapter encourages other research and test development programs to experiment with SBA design, theory, and research.

ACKNOWLEDGMENTS The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305F100005 to the Educational Testing Service as part of the Reading for Understanding Research (RFU) Initiative. The opinions expressed are those of the authors and do not represent the views of Educational Testing Service, the Institute, or the U.S. Department of Education. We want to thank Kim Fryer, Jim Carlson, Paul Deane, and Jesse Sparks for helpful comments and editorial assistance.

REFERENCES American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). NCME: Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Au, W. (2007). High-stakes testing and curricular control: A qualitative metasynthesis. Educational Researcher, 36, 258–267. Bennett, R., Deane, P., & van Rijn, P. (2016). From cognitive-domain theory to assessment practice. Educational Psychologist, 51, 82–107. Bennett, R. E. (2010). Cognitively based assessment of, for, and as learning (CBAL): A preliminary theory of action for summative and formative assessment. Measurement, 8, 70–91. Coiro, J. (2011). Predicting reading comprehension on the Internet: Contributions of offline reading skills, online reading skills, and prior knowledge. Journal of Literacy Research, 43, 352–392. Coiro, J., & Dobler, E. (2007). Exploring the online reading comprehension strategies used by sixth‐grade skilled readers to search for and locate information on the Internet. Reading Research Quarterly, 42, 214–257. College Board (2017). The AP United States History Exam. Retrieved from: http://apcentral.collegeboard.com/ apc/members/exam/exam_information/2089.html. Council for Aid to Education (2017). K-12 Faculty or Administrator CWRA+ Sample Instrument. Retrieved from: http://cae.org/education-professionals/k12-faculty-or-administrator/cwra-sample-instrument. Deane, P., Sabatini, J., Feng, G., Sparks, J.R., Song, Y., Fowles, M., . . . Foley, C. (2015). Key practices in the English language arts (ELA): Linking learning theory, assessment, and instruction (Research Report No. RR-15-17). Princeton, NJ: Educational Testing Service. Deane, P., Sabatini, J., & O’Reilly, T. (2012). English language arts literacy framework. Princeton, NJ: Educational Testing Service. Retrieved from http://elalp.cbalwiki.ets.org/Table+of+Contents. Deane, P., & Song, Y. (2015). The key practice, discuss and debate ideas: conceptual framework, literature review, and provisional learning progressions for argumentation (Research Report No. RR-15-33). Princeton, NJ: Educational Testing Service. Forzani, E., & Maykel, C. (2013). Evaluating a representative state sample of Connecticut seventh-grade students’ ability to critically evaluate online information. CARR Reader, 10, 23–37. Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H. (2010). Summary versus argument tasks when working with multiple documents: Which is better for whom? Contemporary Educational Psychology, 35, 157–173.

464  •  Sabatini et al. Haertel, E. (1999). Validity arguments for high‐stakes testing: In search of the evidence. Educational Measurement: Issues and Practice, 18, 5–9. Hift, R. (2014). Should essays and other “open-ended”-type questions retain a place in written summative assessment in clinical medicine? BMC Medical Education, 14, 249. Jones, B. D., & Egley, R. J. (2004). Voices from the frontlines: Teachers’ perceptions of high-stakes testing. Education Policy Analysis Archives, 12(39), 1–34. Kafer, K. (2002, December 1). High-poverty students excel with direct instruction. Heartlander Magazine. Retrieved from www.heartland.org/news-opinion/news/high-poverty-students-excel-with-direct-instruction. Katz, I. (2007). Testing information literacy in digital environments: ETS’s iSkills assessment. Information Technology and Libraries, 26, 3–12. Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, UK: Cambridge University Press. Kirsch, I., & Mosenthal, P. (1990). Exploring document literacy: Variables underlying the performance of young adults. Reading Research Quarterly, 25, 5–30. Leu, D., Kinzer, C., Coiro, J., & Cammack, D. (2004). Toward a theory of new literacies emerging from the Internet and other information and communication technologies. In R. B. Ruddell & N. Unrau (Eds.), Theoretical models and processes of reading (pp.  1570–1613). Newark, DE: International Reading Association. Leu, D., Kulikowich, J., Sedransk, N., Coiro, J., Liu, C., Cui, W., . . . Maykel, C. (2014, April). The ORCA project: Designing technology-based assessments for online research, comprehension, and communication. Paper presented at the annual meeting of the American Educational Research Association, Philadelphia, PA. Liu, L., Rogat, A., & Bertling, M. (2013). A CBAL™ science model of cognition: Developing a competency model and learning progressions to support assessment development. (Research Report No. RR-13-54). Princeton, NJ: Educational Testing Services. Lukhele, R., Thissen, D., & Wainer, H. (1994). On the relative value of multiple‐choice, constructed response, and examinee‐selected items on two achievement tests. Journal of Educational Measurement, 31, 234–250. Magliano, J. P., McCrudden, M. T., Rouet, J.-F., & Sabatini, J. (2018). The modern reader: Should changes to how we read affect research and theory? In M. F. Schober, M. A. Britt, & D. N. Rapp (Eds.), Handbook of discourse processes (2nd ed., pp. 342–361). New York: Routledge. Mosenthal, P., & Kirsch, I. (1991). Toward an explanatory model of document literacy. Discourse Processes, 14, 147–180. Mullis, I., & Martin, M. (Eds.). (2015). TIMSS & PIRLS International Study Center. Chestnut Hill, MA: Boston College. O’Reilly, T., Deane, P., & Sabatini, J. (2015). Building and sharing knowledge key practice: What do you know, what don’t you know, what did you learn? (Research Report No. RR-15-24). Princeton, NJ: Educational Testing Service. O’Reilly, T., & Sabatini, J. (2013). Reading for understanding: How performance moderators and scenarios impact assessment design (Research Report No. RR-13-31). Princeton, NJ: Educational Testing Service. O’Reilly, T., & Sheehan, K. M. (2009). Cognitively based assessment of, for, and as learning: A framework for assessing reading competency. (Research Report No. RR-09-43). Princeton, NJ: Educational Testing Service. O’Reilly, T., Weeks, J., Sabatini, J., Halderman, L., & Steinberg, J. (2014). Designing reading comprehension assessments for reading interventions: How a theoretically motivated assessment can serve as an outcome measure. Educational Psychology Review, 26, 403–424. Organisation for Economic Co-operation and Development (2011). PISA 2009 results: Students online: Digital technologies and performance. Retrieved from https://www.oecd.org/pisa/pisaproducts/48270093.pdf. Rouet, J.-F. (2006). The skills of document use: From text comprehension to web-based learning. Mahwah, NJ: Erlbaum. Rouet, J.-F., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. In M. T. McCrudden, J. P. Magliano, & G. J. Schraw (Eds.), Text relevance and learning from text (pp. 19–52). Charlotte, NC: Information Age. Rupp, A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23, 441–474. Sabatini, J., Halderman, L., O’Reilly, T., & Weeks, J. (2016). Assessing comprehension in kindergarten through third grade. Topics in Language Disorders, 36, 334–355. Sabatini, J., O’Reilly, T., & Deane, P. (2013). Preliminary reading literacy assessment framework: Foundation and rationale for assessment and system design (Research Report No. RR-13-30). Princeton, NJ: Educational Testing Service.

Scenario-Based Assessment  •  465 Sabatini, J., O’Reilly, T., Halderman, L., & Bruce, K. (2014a). Broadening the scope of reading comprehension using scenario-based assessments: Preliminary findings and challenges. L’Année psychologique, 114, 693–723. Sabatini, J., O’Reilly, T., Halderman, L., & Bruce, K. (2014b). Integrating scenario‐based and component reading skill measures to understand the reading behavior of struggling readers. Learning Disabilities Research & Practice, 29, 36–43. Sabatini, J., Petscher, Y., O’Reilly, T., & Truckenmiller, A. (2015). Improving comprehension assessment for middle and high school students: Challenges and opportunities. In K. Santi & D. Reed (Eds.), Improving reading comprehension for middle and high school students (pp. 119–152). Baltimore, MD: Springer Literacy Series. Sparks, J. R., & Deane, P. (2015). Cognitively based assessment of research and inquiry skills: Defining a key practice in the English language arts (Research Report No. RR-15-35). Princeton, NJ: Educational Testing Service. Vygotsky, L. (1978). Mind in society: The development of higher mental process. Cambridge, MA: Harvard University Press.

26

ASSESSMENT OF MULTIPLE RESOURCE COMPREHENSION AND INFORMATION PROBLEM SOLVING Susan R. Goldman, Alyssa Blair, and Candice M. Burkett department of psychology and learning sciences research institute university of illinois, chicago, usa

Citizens of the 21st century need to be able to critically read and evaluate the burgeoning information resources relevant to personal, academic, and professional life. Yet, there is ample evidence that the literacy levels of the majority of graduating high school students are inadequate for these purposes (Carnegie Council on Advancing Adolescent Literacy, 2010; NAEP, 2009; OECD, 2013). Responses to this gap call for greater emphasis in formal schooling on analysis and critical reading in the context of complex argumentation and problem-solving tasks (Council of Chief State School Officers, 2010; Next Generation Science Standards Lead States, 2013). Indeed the very concept of a Handbook on multiple sources attests to the burgeoning interest in and efforts to research how people locate, process, critique, evaluate, and employ information available in a variety of media and modalities, including spoken and printed verbal words as well as static and dynamic visual images. Increasingly, electronic technologies “host” the information and may soon supplant traditional paper-based publication venues. In the spirit of these everevolving forms and formats in which information “resides” we prefer to use the term resource as compared to source, document, or text, all of which occur in the literature on the topic of this Handbook. As well, adopting the term resource allows greater referential clarity with respect to the information resource per se as compared to its attributes such as author or publisher, which are sometimes referred to as sources. We use the term sourcing processes to refer to efforts to identify attributes of an information resource (Goldman & Scardamalia, 2013). There are numerous challenges to developing instruction that creates opportunities for students to develop critical reading and analytic competencies with multiple

466

Multiple Resource Comprehension  •  467

information resources and to designing assessments that measure what they have learned (Afflerbach, Cho, & Kim, 2015; Goldman et al., 2016; Lee & Spratley, 2010). These challenges include adequate conceptualization of the knowledge and skills that underlie critical reading to accomplish authentic disciplinary tasks, developmental progressions in them, and aspects of resource complexity that are not amenable to quantification and computational approaches (Goldman & Lee, 2014). Attention to these challenges is critical if we are to develop assessments that are authentic, reliable, and informative with respect to what individuals know and can do with multiple information resources. In this chapter we describe an approach to addressing challenges associated with assessing complex comprehension with multiple information resources, including traditional texts, digital texts, and multimedia visuals. By complex comprehension we mean the processes and outcomes of working with multiple information resources for purposes of decision making, problem solving, perspective taking on the human condition or other forms of robust learning needed for successful functioning in academic, personal, and professional life. The processes and outcomes of interest are consistent with previously articulated calls for greater attention to deep learning (Pellegrino & Hilton, 2012), the generation of knowledge (Scardamalia & Bereiter, 2006), and the construction of interpretations, arguments, and explanations (Goldman et al., 2016). Many of the chapters in this Handbook address these processes and outcomes. However, they do so from the perspective of contributing to a normative knowledge base about discourse processing, reading comprehension, and/or information problem solving. That is, the research addresses questions about the effects of manipulating characteristics of information resources and tasks on processes and their outcomes, with data analyzed to determine whether “mean differences” among conditions are reliable statistically. For the most part, conclusions are stated in terms of the effect(s) of whatever was manipulated (e.g., characteristics of the task instructions or the expertise of the author of the information resource) on the average performance of a group of individuals. The performance examined varies depending on the research questions but generally indexes some aspect of access, use, or evaluation of the resource (Bråten, Stadtler, & Salmerón, in press). As well, this type of research may look at whether, and how, effects of manipulations are moderated by various pre-existing individual differences (e.g., domain or topic knowledge, epistemic commitments, attitudes and beliefs about the topic). Increasingly, mediational analyses are used to explore effects of manipulations on processing strategies to gain insights into the mechanisms by which the manipulations affect performance. Overall, this body of work is exceptionally valuable for building a high-quality knowledge base about the domain of comprehension and problem solving with multiple (as well as single) information resources. However, a normative knowledge base per se is insufficient for purposes of the design of assessment at the individual level. In the next section we provide a definition of assessment and discuss an approach to the design of assessment, Evidence-Centered Design (ECD) (Mislevy & Haertel, 2006; Mislevy, Steinberg, & Almond, 2003). Key to the ECD process is the specification of the knowledge, skills, and competencies that constitute proficient performance in the domain and what might be observable evidence of an individual’s proficiency with respect to these domain components. We argue that when informed by the normative knowledge base, the ECD process can

468  •  Goldman et al.

move us in the direction of assessments that are informative with respect to what individuals know and can do in multiple information resource comprehension and problem-solving situations.

ASSESSMENT: A PROCESS OF REASONING FROM EVIDENCE In its most general sense, assessment can be conceptualized as a process of reasoning from evidence for purposes of making some claim (Pellegrino, Chudowsky, & Glaser, 2002). In the case of multiple resource comprehension and problem solving, claims often refer to internal thinking and reasoning processes that are typically not open to inspection. Accordingly, we need to generate ways to make the internal observable and from what is observed make inferences to support claims about the internal processes. Critical to the validity of our inferences from the observable to the nonobservable is the alignment among what Pellegrino et al. (2002) referred to as three vertices of an assessment triangle: the cognition we want to make claims about, the observables we gather as a basis for those claims, and the interpretive inferences we make to warrant the observables as evidence for that which we cannot observe directly. The normative knowledge base contributes to defining the cognition vertex through empirical findings and theoretical models and to identifying potentially useful observables from among the performance measures employed in the research. Comprehension and problem solving with multiple information resources involve many strategies and processes and a variety of types of background knowledge, all of which must be coordinated, monitored, and evaluated to complete a task successfully. These process and knowledge components and their interrelationships constitute the cognition we might want to make claims about. As well, the relevance and importance of different strategies, processes, and knowledge are constrained by the nature and type of task, information resources, and task outcome or product that constitute the assessment situation. It is critical to identify the cognitive components about which claim(s) are to be made, and what evidence is intended to provide the basis for those claims. Furthermore, the soundness of the inferences and reasoning justifying the evidence as support for the claim must be evaluated, especially in consideration of the task situation in which the evidence is elicited. The essential issue is whether the affordances of the assessment situation permit the individual to provide evidence of the intended cognitive component. While this may seem obvious, it is surprisingly easy to generate assessment situations that involve multiple cognitive components, thereby making it difficult if not impossible to make unambiguous claims about any one component from what is observed. To illustrate, we pose a thought experiment based on an assessment scenario that although fictitious has features typical of course assessments. At the end of a semester, the instructor of an entry-level Italian class wishes to determine what his college class can comprehend when reading in Italian. The students are asked to read about the economy in two different regions of Italy. They are provided with two texts, one about each of the regions. The author of each text is indicated, along with the information that each owns a property development company in the region. The task instructions indicate that students are to write a short essay (in Italian) indicating where they would

Multiple Resource Comprehension  •  469

prefer to live and why. The instructor assigns a comprehension proficiency score to each student based on the content of the essay, with the highest comprehension scores assigned to essays that justified the preference through a comparison of features of the two regions. Given the task, text set, and the scoring criteria, it seems fairly obvious that much more than comprehension of Italian is reflected in the essay score. At a minimum the rhetorical demands of writing an essay in Italian are reflected in the score that the instructor is using to index comprehension. Beyond this confounding of comprehension and production, one might be tempted to make claims about sourcing processes but we need to ask whether these are valid inferences from these essays given the task instructions and affordances of the texts. That is, if student A’s essay attributed information to the author of each piece but student B’s did not, would it be reasonable to conclude that A was more “source aware” than B? Furthermore, would it be reasonable to conclude that students understood little Italian if they simply wrote that they preferred one of the regions because of some characteristics not even mentioned in the texts with which they were provided? Can we claim that because these students apparently relied on their prior knowledge they comprehended the Italian texts less well than students who copied several sentences from each passage into their essays but failed to provide a reason why they preferred one region over the other? And what about the texts themselves – is enough information provided to allow students to reach a reasonable conclusion? Should the task instructions indicate that they may not be able to make a decision based on the information provided, in which case they should indicate what else they would want to know? This might be reasonable if the assessment was intended to provide evidence pertinent to understanding relationships between economy and geography but as a measure of comprehension of the language per se, probably not so much. We intend this thought experiment to illustrate that the design of assessments demands systematic analyses of task affordances along with a clear articulation of the construct(s) about which the assessment is intended to yield evidence. It is intended to illustrate the many “conceptual tangles” that can arise when designing assessments that unequivocally warrant the inferences drawn about what an individual knows and can do. A systematic analytic approach to assessment design is essential especially when multiple types of knowledge, processes, and materials are involved, as they are in the case of comprehension and problem solving with multiple information resources. The ECD process exemplifies just such a systematic analytic approach (Mislevy & Haertel, 2006; Mislevy et al., 2003). When used appropriately it can help assessment designers, including the instructor in our thought experiment, avoid the conceptual tangles illustrated in the thought experiment scenario. The ECD process proved extremely useful in prior work on multiple resource comprehension in science and history disciplines precisely because it demanded high levels of precision and clarity regarding the constructs of interest, observations that provided information about these constructs, and the inferences that connected the observations to claims about proficiency on the construct (e.g., Goldman, Lawless, & Manning, 2013; Goldman et al., 2012). In the next section we elaborate on the ECD process to illustrate its utility in clarifying and structuring the design of assessments.

470  •  Goldman et al.

The ECD Process The ECD process has three major components: the Student Model, the Task Model, and the Evidence (or Interpretive) Model. The Student Model defines the cognition – the knowledge, skills, and practices that constitute competence in the domain. These are expressed in claim–evidence statements. Claim statements articulate what students should know and be able to do (e.g., The student makes use of author information in the sourcing process). Evidence statements linked to specific claims indicate what observables students would need to produce in support of inferences about competencies (e.g., The student work includes information about the credibility of the author or efforts to determine the credibility of the author). Task Models articulate the characteristics of assessment situations that are used to generate observations in the form of student work (e.g., characteristics of information resources, instructions, medium of presentation and response). The Evidence Model specifies how to interpret the observations with respect to claims about students. (See Lawless, Goldman, Gomez, Manning, & Braasch [2012] and Goldman et al. [2013] for elaboration.) Current theoretical frameworks and empirical research on comprehension and problem solving with multiple information resources contribute most directly to articulating the Student and Task Models. For example, in a recent comprehensive review of multiple source research, Bråten et al. (in press) characterized the research as addressing three dimensions of source processing: awareness, evaluation, and use. From the ECD perspective, these three dimensions might define three components of the Student Model about which we want to gather evidence pertinent to claim statements such as The student is aware of resource attributes . . .; The student evaluates resources . . .; The student uses resources . . . . Stated in these ways as claim statements, however, they are too vague to specify what observations would warrant inferences about awareness, evaluation, or use. In other words, these claim statements need to be iteratively unpacked to the point where the answers allow for specification of the student work that would provide evidence from which interpretive inferences about cognition could be drawn. The ECD process suggests using question frames of the following type to unpack claim statements: What is meant by resource attributes?, What is meant by aware?, What is meant by evaluates?, What is meant by uses? The results of the unpacking process are statements that connect claims to specific observables in some student work product. Sometimes the process can lead assessment designers to set aside claims about some cognitive components in favor of others because of the pragmatic feasibility of observables from work products. For example, unpacking processes for awareness, evaluation, and use could result in three claim–evidence statements differentiated by critical defining features of each construct. 1. Awareness: The student work includes at least one mention of the author of a resource. 2. Evaluate: The student work includes information about the credibility of author(s) or efforts to determine the credibility of author(s). 3. Use: The student work includes information that is attributed to the author of a resource.

Multiple Resource Comprehension  •  471

Although similar, the three claim–evidence statements make clear what needs to be observed in a work product – what there needs to be evidence of – to make inferences about each of the three constructs and how the observed evidence would differ. The Task Model specifies the parameters of assessment activities that could provide opportunities for students to generate the work defined in the Student Model. Elements of assessment activities include the task instructions, time frame, characteristics of the information resources (e.g., topical content, number, form/media, relationship of content within and among sources, i.e., complementary/consistent, contradictory) and product (e.g., answer questions from memory, write an explanation or prepare a poster using the information resources). The Evidence Model specifies what observations would be made, what data would be gathered, and how it would be coded and scored in service of gathering evidence relevant to the specified claims. An illustration of the value of the ECD process is provided in Goldman et  al. (2013) where they described how they used the ECD process to guide the design of multiple source comprehension assessments for students in the age range of 10–13 years. Their work began by considering what theory and research indicated about cognitive components of multiple source comprehension when they began the work (circa 2007). They then used the ECD process to develop Student, Task, and Evidence models for two components that seemed important to separate for assessment purposes: a Sourcing component and a Comprehension component. (See Lawless et al., 2012.) For the comprehension component they developed claim and evidence statements for analysis, synthesis, and integration processes. Specifying the Task Model led these researchers to a conscious decision to develop a text set comprised of three verbal texts that contained information that was complementary (as compared to contradictory). This text set allowed inferences to be made from student essays (the work product) about synthesis and integration, but not about detection of contradictions or how the presence of contradictions impacted integration efforts. Note that a different Task Model would be required for claim and evidence statements about integration for resources that contained conflicting information. Thus, the ECD process helped the researchers design a Task Model that generated student work that could provide evidence of synthesis and integration under the particular constraints and affordances of the text set. Usefulness of information resources was assessed in the Task Model for the Sourcing component. (See Goldman et al. [2013] and Lawless et al. [2012] for elaboration and details on the assessment design process.) Critical in the assessment design process used by Goldman, Lawless, and colleagues was the careful consideration of the extant theoretical and empirical research in defining the three vertices of the assessment triangle: what cognition to assess, how to observe the cognition, and how to interpret the observations with respect to inferences about the cognition. The ECD process necessarily proceeds from a consideration of theoretical and empirical research relevant to the domain to be assessed. In the next section, we consider what current theoretical and empirical research suggests about components of the domain of multiple information resource comprehension and problem solving that may be important to assess, informative ways to observe these

472  •  Goldman et al.

components, and how to interpret these observations with respect to the knowledge, skills, and competencies of the individual.

WHAT TO ASSESS AND HOW TO ASSESS IT Over the past 10 to 15 years, theoretical frameworks and empirical research in the domain of multiple information resource comprehension and problem solving has explored and refined the basic components outlined in early work on this topic (e.g., Perfetti, Britt, & Georgi, 1995; Rouet, 2006; VanSledright, 2002; Wineburg, 1991). Our review is necessarily brief and presented from the perspective of what current theory and research in this domain suggest about what cognition might be important to assess and how we might assess it. Theoretical Frameworks A number of frameworks have been proposed for purposes of specifying the components of a domain model for multiple source comprehension and information problem solving (e.g., Brand-Gruwel, Wopereis, & Walraven, 2009; Lawless et  al., 2012; Perfetti, Rouet, & Britt, 1999; Rouet, 2006). The Multiple Document Taskbased Relevance Assessment and Content Extraction (MD-TRACE) model of Rouet and Britt (2011; Britt, Rouet, & Braasch, 2013) and the information problem solving with Internet (IPS-I) model of Brand-Gruwel et al. (2009) reflect some of the most well developed of these prior efforts. Generally speaking the models indicate five essential components: interpreting the task, gathering/identifying resources, sourcing processes for evaluating relevance and reliability, analysis/synthesis/integration within and across resources, and application of the information to generate a task product. Although often described linearly, researchers discuss the importance of ongoing evaluation and monitoring of the “output” of each component that may produce revisiting other components of the process to, for example, reinterpret what the task is asking, or search for additional or different resources. Detailed models have also been proposed that focus on sourcing processes, such as the ContentSource Integration (CSI) model (Stadtler & Bromme, 2014) and the DiscrepancyInduced Source Comprehension (D-ISC) model (Braasch & Bråten, 2017; Braasch, Rouet, Vibert, & Britt, 2012). Recently, in an effort to produce a general description of this class of models, Goldman and Brand-Gruwel (in press) attempted to bring together various models emanating from discourse comprehension and learning from text research with those emerging in the field of information problem solving. The integrative framework they proposed was intended not to replace but to generalize and capture commonalities across prior efforts. As well, the framework attempted to emphasize the dynamic interactivity and iterative cycles of search and processing inherent in authentic inquiry with multiple information resources, but which to date have garnered only a small share of attention from researchers and prior theoretical models. Figure 26.1 reproduces the framework proposed in Goldman and Brand-Gruwel (in press). The components of the domain model serve as the cognitive constructs that we might wish to assess at an individual level and thus as starting points for generating the ECD Student, Task, and Evidence Models using the “unpacking” process described earlier.

Multiple Resource Comprehension  •  473

Sense-Making, Problem Solving Processes

Integrated Representation

Learner Attributes

Learner Attributes Information resources

Task/Task interpretation

Information Resources: Search and Select

Information resources

Information resources

Product/Task interpretation

Information resources

Figure 26.1  Representation of Components and Processes for the Domain of Comprehension and Information Problem Solving with Multiple Information Resources.

In Figure 26.1, the leftmost portion reflects task interpretation and resource gathering/selecting processes, including sourcing processes and initial evaluation of resources with respect to relevance and reliability in particular. The “cloud” of learner attributes represents what the learner brings to the situation, including prior knowledge, epistemic cognitions, beliefs, attitudes, dispositions, skills, practices, strategies, and heuristics applicable to comprehension and problem solving. These attributes are “active” and accessible throughout the activity. The arrows between Task/Task interpretation and Resource Search and Selection components reflect the mutual impact that the two have on each other. That is, the results of initial examination of resources may help redefine, redirect, and revise the search process. The center portion of Figure 26.1 refers to activities involved in making sense of the information resources that are selected or provided for doing the task. Sensemaking processes and Integrated representations are shown in “clouds” because these go on “in the head of the learner,” so to speak, as they analyze, synthesize, and integrate information judged to be relevant to the task. The arrows between and among the information resources reflect cross-resource comparison and contrast, including further use of sourcing processes. Integrated representations result from the application of sense-making and problem-solving processes, including information search and selection, basic (e.g., decoding, word recognition, parsing, resonance, simple inferences) and complex (e.g., analysis within and across sources, interpretation, reasoning; synthesis within and across sources; sourcing processes; critique and evaluation) comprehension processes. The rightmost section of Figure 26.1 refers to processes invoked in creating a product or outcome that meets the task specifications, and a modified (hopefully) constellation of learner attributes. We indicate with two-way arrows iterative back and forth between and among components.

474  •  Goldman et al.

It should be clear from Figure 26.1 that there are many components that could be assessed and multiple measures that could be used for assessment. ECD provides a systematic analytical process for moving from a domain model like that depicted in Figure 26.1 to Student, Task, and Evidence Models that specify assessments that are informative with respect to what an individual knows and can do in the domain. The design of assessment thus proceeds from the normative knowledge base, i.e., the extant theoretical and empirical work in the domain, to assessments that are informative at the individual level. Research on Multiple Information Resources In this section, we provide an admittedly selective review of empirical research to identify components and relations among components, research tasks, behavioral measures, and analytic methods that inform the what and the how of the Student, Task, and Evidence Models. We first review the “how” – methods and measures – and then turn to findings regarding the “what” – domain components and learner attributes identified in Figure 26.1. Prototypical Paradigms and Methods A prototypical approach to research studies in multiple source comprehension research is to manipulate the content and/or source attributes and examine the effects of these manipulations on sense-making processes and the integrated representation that is constructed. Effects are examined through behavioral indicators of processing derived from data collected concurrent with sense-making and from the products of sense-making (e.g., qualities of outcome products, performance on recognition memory tasks, and post-task ratings of information resources). PROCESSING

There are three primary ways in which investigators have assessed processing “while it is happening”: navigation logs, eye-tracking, and concurrent think-alouds. A fourth measure, reading time, is often derived from navigation logs or eye-tracking data. Each of these measures provides a “real-time trace” of processing activity from which researchers derive dependent measures. Navigation logs and eye-tracking are particularly useful for deriving measures of amount of time spent on different information resources or parts of a resource, and sequence data within and between information resources and inspection patterns including frequency of accessing and returning to previously accessed information resources. Think-alouds are less useful for time data but also provide sequence and inspection pattern data and may in addition provide insight into readers’ goals, meaning making, and evaluations of content and other attributes of information resources. Typically, these online measures are the basis of claims about the search for, and selection of, information resources, as well as reading processes during sense-making. They are often analyzed in relation to intentionally manipulated content and attributes of information resources as well as to post-reading products such as recalls, summaries, or essays (e.g., Strømsø, Bråten, Britt, & Ferguson, 2013).

Multiple Resource Comprehension  •  475 TASK PRODUCTS AND MEASURES

The outcomes of sense-making processes are assumed to be represented internally, referred to as the integrated representation in Figure 26.1. These are “made visible” through task products and outcome tasks, often some form of recognition or recall memory task. Multiple choice and sentence verification are commonly used recognition tasks; these typically test for information that was explicitly presented in the information resources as well as information that can be inferred from what was presented, or inferences (Royer, Carlo, Dufresne, & Mestre, 1996). Inferences often draw on information within a particular resource but especially when interest is in cross-text synthesis, draw on information that was presented in different information resources (e.g., intertextual inference verification; Bråten, Strømsø, & Britt, 2009). Response options for verification tasks may be true/false or include rating scales that ask participants to estimate how confident they are about the information or sometimes about their response (e.g., Braasch & Goldman, 2010). Recall memory tasks are typically some form of constructed response task, such as a written essay (descriptive, explanatory, or argumentative) or decision plus justifications. Sometimes the audience is explicit (e.g., a friend or acquaintance) and other times the audience is left vague. The content of constructed responses is often “traced back” to particular information resources so researchers can make inferences about information selection in relation to content and sourcing attributes. RATINGS OF INFORMATION RESOURCES

In addition to process and product measures, some research studies have participants provide ratings of the information resources as a means of assessing participants’ evaluations and judgments of, for example, reliability, trustworthiness, and usefulness. As well, ratings are sometimes collected for purposes of manipulation checks. For example, a researcher may want to test the hypothesis that participants will rely on more as opposed to less reliable resources when solving a science task. The researcher operationalizes reliability by providing the type of publication in which the resource appeared, with some indicated as having been published in peer-reviewed scientific journals and others in popular press and life-style magazines. After they have completed the science task, participants may be asked to rate the reliability of each of the information resources. If those that the researcher designated as more reliable align with the ratings of the participants, the researcher has greater confidence that the manipulation was perceived by the participants as the researcher intended. (See Wiley et al., 2009 for an example of this.) LEARNER ATTRIBUTES

Learner attributes reflect individual differences among learners in what they bring to the task (e.g., knowledge, epistemic cognition, attitudes and beliefs, working memory). Attributes such as these have been examined as moderators of the effects of various manipulations on processes and products and are typically assessed prior to beginning research tasks through surveys, multiple choice, or open-ended elicitation methods depending on the dimension of individual differences.

476  •  Goldman et al.

In terms of topics and topic knowledge the vast majority of multiple resource comprehension and problem-solving work has focused on historical topics (Perfetti et al., 1995; Rouet, Britt, Mason, & Perfetti, 1996; Wineburg, 1991) or socioscientific controversies such as climate science and health/medical treatments (e.g., Bråten, Ferguson, Strømsø, & Anmarkrud, 2014; Kammerer, Amann, & Gerjets, 2015; Stadtler & Bromme, 2008; Stadtler, Scharrer, Macedo-Rouet, Rouet, & Bromme, 2016). In some cases the topics involve understanding technically complex phenomena, such as the economic and environmental impact of seawater desalination in Israel (Barzilai, Tzadok, & Eshet-Alkalai, 2015), volcanic eruptions (Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012; Wiley et al. 2009), or El Niño weather patterns (Braasch, Bråten, Strømsø, Anmarkrud, & Ferguson, 2013). Consistent with findings for single-text comprehension and problem-solving research, individuals who know a lot about the topic (e.g., experts and those with high prior knowledge) compared to those who know less (e.g., novices and those with low prior knowledge) exhibit more strategic search, selection, and sense-making processes and produce higher-quality arguments and other types of task products in multiple information resource situations (BrandGruwel, Kammerer, van Meeuwen, & van Gog, in press; Brand-Gruwel, Wopereis, & Vermetten, 2005; Bråten, Strømsø, & Salmerón, 2011; Lucassen, Muilwijk, Noordzij, & Schraagen, 2013; Salmerón, Kammerer, & García-Carrión, 2013; Wineburg, 1991). Major Findings Researchers from information problem-solving traditions have tended to focus more on the search and selection components of the domain model depicted in Figure 26.1 whereas researchers coming from the multiple-document comprehension perspective have tended to focus on sense-making processes, relationships between task instructions and outcome products, and between task instructions, processes (including sourcing processes), and outcomes. Increasingly, both traditions are examining various learner attributes in relation to comprehension and problem solving with multiple information resources. SEARCH AND SELECTION OF INFORMATION RESOURCES

Patterns of resource inspection and selection derived from navigation logs indicate that participants tend to access more frequently and to rate as more reliable and trustworthy sites listed at the top of a search engine results page (SERP) as compared to those further down the list, regardless of whether the list is researcher-provided or participant-generated (Keane, O’Brien, & Smyth, 2008; Pan et  al., 2007; Kammerer & Gerjets, 2012, 2014; Salmerón et  al., 2013). The order effect is so powerful that when experimenters have reverse-ordered sites on a SERP so that the top links are the least reliable pages, participants still rate these sites as more reliable and subsequently use more of the information from those sites in subsequent argumentation tasks (Kammerer & Gerjets, 2014). Interestingly, the “first listed” effect appears to be an artifact of list formats. When participants were provided with site information in a columnar tab-like interface, they showed fewer visual fixations on commercial search results and selected more objective sites for use in argumentative summaries than when these same sites were arrayed in a typical SERP list interface (Kammerer &

Multiple Resource Comprehension  •  477

Gerjets, 2012). These effects of display format were especially strong for participants who indicated a belief in the correctness of information on the Internet (Kammerer & Gerjets, 2014). Participants are also aware of, although they do not always use, resource attribute information in selecting information resources from among those returned on a SERP or provided by the researcher. That is, although participants can identify when content experts are the authors of information resources, they tend to base judgments of trustworthiness, reliability, and usefulness for the task on characteristics of the content rather than characteristics of the authors (Braasch et al., 2009; Bråten, Braasch, Strømsø, & Ferguson, 2015; Bråten et al., 2009, 2011; Macedo-Rouet, Braasch, Britt, & Rouet, 2013). However, a recent finding indicated more value placed on resources authored by experts when study participants were planning presentations on topics unfamiliar to them (McCrudden, Stenseth, Bråten, & Strømsø, 2016). Furthermore, trustworthiness judgments were found to be higher for seemingly more “authoritative” information resources such as textbooks and official documents than for newspapers and information attributed to commercial agents, especially for those participants with high prior knowledge of the topic (Bråten et al., 2009, 2011, 2015). As well, trustworthiness evaluations positively predicted frequency of citation in essays and performance on intertextual inference verification tasks (Bråten, Salmerón, & Strømsø, 2016). It is unclear whether these trustworthiness judgments reflect content or evaluations of resource attributes because in many of these studies the judgments were made after some outcome task or task product has been completed. INFORMATION RESOURCE PROCESSING AND MENTAL REPRESENTATION

Research studies generally find that more strategic and elaborative processing of information resources is associated with better performance on outcome tasks (e.g., Anmarkrud, Bråten, & Strømsø, 2014; Goldman et al., 2012). The occurrence of more strategic processing has been shown to be related to features of content, resource attributes, and learner attributes. For example, one consistent finding across several studies is that when there are discrepancies present in the content of the materials, people are more likely to show evidence of sourcing processes, as well as better memory for the resource from which particular information came, resource attributes, and author-content links. This has been demonstrated for discrepancies within a single resource (Braasch et al., 2012; Kammerer, Kalbfell, & Gerjets, 2016; Keck, Kammerer, & Starauschek, 2015; Steffens, Britt, Braasch, Strømsø, & Bråten, 2014) as well as between multiple resources (Kammerer et al., 2016; Lund, Bråten, Brante, & Strømsø, 2017; Saux et al., 2017). The discrepancy effect has also been found when reading and providing summaries for shorter and longer narrative texts (Rouet, Le Bigot, Pereyra, & Britt, 2016). Other studies have demonstrated interactions between the information content, memory for attributes of the resource, and the consistency of the content with participants’ beliefs. In brief, memory for the surface text of information resources as well as resource attributes is better for belief inconsistent texts whereas memory at situation model levels is better for belief-consistent texts (Braasch, Bråten, Britt, Steffens, & Strømsø, 2014; Bråten et al., 2016; Maier & Richter, 2013).

478  •  Goldman et al. LEARNER ATTRIBUTES

In addition to topic knowledge and expertise, participants’ epistemic cognition in the topic area as well as their beliefs about the nature of intelligence and their own selfefficacy affect their approaches to multiple information resources. Generally speaking, individuals who endorse the importance of corroborating information across multiple information resources display more explicit attention to the attributes of resources during search, in their processing, and in performance on post-reading memory and information production tasks (Bråten et al., 2014; Kammerer et al., 2015). Likewise, individuals who endorse incremental theories of intelligence, i.e., intelligence is malleable, consider resource trustworthiness in judgments of usefulness of information resources to a greater degree than those who endorse a fixed view of intelligence (Braasch, Bråten, Anmarkrud, & Strømsø, 2014). Barzilai and colleagues have found that those who hold the belief that knowledge is subjective, uncertain, and justified by personal experience tend to rely on their own prior conceptions in a topic area rather than carefully consider the perspectives and content of information resources. In contrast, they found that people who ascribe to the belief that knowledge is constructed by people within a particular perspective but is anchored by evidence and shared standards are more likely to consider viewpoints of resource authors and integrate multiple viewpoints in their arguments (Barzilai & Eshet-Alkalai, 2015; Barzilai et al., 2015; Barzilai & Zohar, 2012). There have also been several studies that explore Internet-specific beliefs. For example and not surprisingly, individuals who indicate stronger as compared to those who indicate weaker beliefs in the reliability of web-based information depend more on the order in which sites are listed on SERPs, spend less time and processing on resource attributes, and produce less balanced arguments in their task products (e.g., Kammerer, Bråten, Gerjets, & Strømsø, 2013). Although learner attributes are often examined from the perspective of pre-existing moderators of performance in experimental tasks, it is as yet indeterminate whether and how cognitive performance and attitudes and beliefs are related to one another. As indicated in Figure 26.1, learner attributes result from as well as provide input to multiple information resource comprehension and problem solving but content knowledge has been the learner attribute that has received the most consistent attention in research. Efforts to Improve Comprehension and Problem Solving with Multiple Information Resources There have been a number of lab-based and classroom-based instructional intervention efforts, the majority of which show the malleability of sourcing processes. For instance, in samples of college (Wiley et  al., 2009) and adolescent (Macedo-Rouet et al., 2013) students, instruction geared toward improving students’ resource evaluation competencies was successful in improving application of reliability criteria. Similarly, Mason, Junyent, and Tornatora (2014) provided instruction on evaluating the authoritativeness and accuracy of web resources. The subsequent navigation patterns of those who participated in the instruction compared to those who did not indicated fewer visits to the least reliable sites with post-task evaluations reflecting increased discrimination of reliable from unreliable sites.

Multiple Resource Comprehension  •  479

With a somewhat different focus, Stadtler et al. (2016) implemented a classroom instructional program intended to increase students’ consideration of information about the resource (resource attribute information) to make decisions about science-based controversies. Students who participated in the program increased in the likelihood of agreeing with pertinent expert positions rather than low-pertinent expert positions on the controversies and were more likely to cite pertinence and expertise in justifying their agreement decisions.

IMPLICATIONS FOR THE DESIGN OF ASSESSMENTS Several consistent findings are particularly helpful in prioritizing what components of the domain model we might wish to assess and observable behavioral indicators that might be useful in assessing them. However, the normative research and theory does not prescribe assessment designs any more than physics prescribes the design of a bridge or airplane. There are many choices and decisions to be made along the way. We have presented the ECD process as a means of going from the normative theory and research to the design of assessment. We have tried to clarify how important components of the domain model might be expressed in the Student Model so that claims can be made about them based on observable evidence. Task Models specify situational constraints under which the claims can be made. Evidence Models indicate the validity of the claim and measurement properties of the assessment. (For contemporary discussion of validity, see Pellegrino, DiBello, & Goldman, 2016.) In the remainder of this chapter we discuss several of the issues that can arise in using the ECD process to move from the normative theory and research to an assessment. An additional issue concerns the purpose of assessment, i.e., why the assessment is being designed. A common distinction is between summative assessment of learning and formative assessment for learning. However, proper consideration of the distinctions and their implications for assessment is beyond the scope of this chapter. (For discussion of the purposes of assessment, see Pellegrino et al., 2002.) Furthermore, although this is a critical issue in designing assessments, especially for educational practitioners, it is orthogonal to the process used to move from normative theory and research to the design of assessment. Emergent Issues The normative research literature suggests that discrepancies in the content information within and among information resources plays an important role in deeper or more extensive consideration of resource attributes. Thus claims about this component of the domain model seem to be good starting points for the design of assessments that are informative with respect to what individual students know and can do. At the same time, theoretical analyses and empirical studies in different disciplines (e.g., science versus literary reading versus history) indicate that discrepancies have different implications with respect to the role their detection plays in sourcing processes. For example, discrepancies in results between replications in an experiment might not reflect issues related to resource attributes but rather stimulate closer examination of error of measurement and reliability of methods along with statistical processes used in the studies. In literary reading, discrepancies or disjunctures are less relevant

480  •  Goldman et al.

for sourcing processes but are of great importance as cues to the reader to go beyond literal meaning and explore potential figurative interpretations of a text. However, in history the heuristics of experts place great importance on resource attributes in reconciling or interpreting discrepancies detected between, for example, two diary accounts written at the time of a historical event. Thus, developing claim–evidence statements and the Task Models for student work that will generate evidence relevant to the claim require careful analyses in the specific discipline of the domain model for multiple information resources. Likewise, the need for these kinds of disciplinespecific analyses suggests future work that would contribute to the normative knowledge base for disciplinary inquiry as well as multiple information resource comprehension and problem solving. (See Goldman et al., 2016 for discussion.) There are also design considerations related to the Task Model for the generation of student work and the choice of what student work is informative for particular claims. For example, one important difference among research studies on multiple resources is whether or not the task product is produced with versus without access to the information resources. In much of the research to date, task products are generated with the resources absent. Yet, that is rarely the way in which we work with multiple information resources. We more often freely move back and forth between them. If the information resources are not present or accessible the role of memory must be acknowledged in the claim statement as the evidence reflects work produced based on whatever internal mental representation resulted from sense-making processes operating on the information resources. These are not claims about using information “at hand” and may reflect different competencies than those associated with working from information resources present in “real time” and consultable during production tasks. Another aspect of the Task Model is the parameters for the composition of the information resources. The parameters need to be aligned with both the task instructions and the claim–evidence statement with which the Task Model is associated. For example, if the claim statement is about students being able to integrate information across a set of resources to create a cause–effect chain, the resource set must not do it for the student by explicitly providing the cause–effect chain. Although this may seem obvious, in translating parameters into actual materials to be used by students, misalignment problems of this type are not infrequent. There are many less obvious issues of this ilk that arise in the design of task situations. In the review of the normative knowledge base we noted a number of different process and product “dependent” measures and types of tasks that have been used. Each one has its strengths and weaknesses, especially when assessments are intended for use by teachers. Most of the process measures we discussed are labor intensive to code and score and/or require sophisticated software applications. These are unlikely to be useful as “just in time” assessments that can inform decision making in real time. Thus, for use in contexts such as classrooms or other settings in which decisions rest on knowing what individuals know, it will be important to consider analogs of both process and product measures that have been so informative in the research. (See Magliano, Hastings, Kopp, Blaum, & Hughes, this volume.) Future developments in learning analytics may also yield ways to generate informative assessment data from navigation log or eye-tracking data in real time. In the meantime, we have seen written “talk to the

Multiple Resource Comprehension  •  481

text” (Schoenbach, Greenleaf, & Murphy, 2012) used as a means of generating analogs of think-alouds that can be discussed and examined by students and teachers. In closing, we want to point out that the design of assessment can reveal issues that need theoretical and empirical attention. For example, there are only a few case studies that attempt to characterize an individual’s strategies and processes over the time course of a comprehension or problem-solving task using multiple information resources (Cho, 2014; Cho & Afflerbach, 2015; Goldman et  al., 2012; Hartman, 1995) or that looked at the incorporation of additional information into an integrated representation over a period of weeks or longer (Perfetti et al., 1995). Furthermore, we need research that helps us understand learning trajectories linked to students’ opportunities to learn the knowledge, skills, and disciplinary practices that are needed for critical processing and use of the multiple information resources that are ubiquitous in our global society.

FURTHER READING Bråten, I., Stadtler, M., & Salmerón, L. (in press). The role of sourcing in discourse comprehension. In M. F. Schober, M. A. Britt, & D. N. Rapp (Eds.), Handbook of discourse processes (2nd ed.). New York: Routledge. Goldman, S. R., Lawless, K. A., & Manning, F. (2013). Research and development of multiple source comprehension assessment. In M. A. Britt, S. R. Goldman, & J. F. Rouet (Eds.), Reading: From words to multiple texts (pp. 180–199). New York: Routledge. Lawless, K. A., Goldman, S. R., Gomez, K., Manning, F., & Braasch, J. (2012). Assessing multiple source comprehension through Evidence-Centered Design. In J. P. Sabatini, T. O’Reilly, & E. R. Albro (Eds.), Reaching and understanding: Innovations in how we view reading assessment (pp. 3–18). Lanham, MD: R & L Education. Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25(4), 6–20. Pellegrino, J. W., DiBello, L. V., & Goldman, S. R. (2016). A framework for conceptualizing and evaluating the validity of instructionally relevant assessments. Educational Psychologist, 51, 59–81.

REFERENCES Afflerbach, P., Cho, B. Y., & Kim, J. Y. (2015). Conceptualizing and assessing higher-order thinking in reading. Theory Into Practice, 54, 203–212. Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. Barzilai, S., & Eshet-Alkalai, Y. (2015). The role of epistemic perspectives in comprehension of multiple author viewpoints. Learning and Instruction, 36, 86–103. Barzilai, S., Tzadok, E., & Eshet-Alkalai, Y. (2015). Sourcing while reading divergent expert accounts: Pathways from views of knowing to written argumentation. Instructional Science, 43, 737–766. Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online sources. Cognition and Instruction, 30, 39–85. Braasch, J. L. G., & Bråten, I. (2017). The Discrepancy-induced source comprehension (D-ISC) model: Basic assumptions and preliminary evidence. Educational Psychologist, 52, 167–181. Braasch, J. L. G., Bråten, I., Anmarkrud, Ø., & Strømsø, H. I. (2014). Incremental theories of intelligence predict multiple documents comprehension. Learning and Individual Differences, 31, 11–20. Braasch, J. L. G., Bråten, I., Britt, M. A., Steffens, B., & Strømsø, H. (2014). Sensitivity to inaccurate argumenta­ tion in health news articles: Potential contributions of readers’ topic and epistemic beliefs. In D. N. Rapp & J. L. G. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 117–138). Cambridge, MA: MIT Press. Braasch, J. L. G., Bråten, I., Strømsø, H. I., Anmarkrud, Ø., & Ferguson, L. E. (2013). Promoting second­ ary school students’ evaluation of source features of multiple documents. Contemporary Educational Psychology, 38, 180–195.

482  •  Goldman et al. Braasch, J. L. G., & Goldman, S. R. (2010). The role of prior knowledge in learning from analogies in science texts. Discourse Processes, 47, 447–479. Braasch, J. L. G., Lawless, K. A., Goldman, S. R., Manning, F., Gomez, K. W., & MacLeod, S. (2009). Evaluating search results: An empirical analysis of middle school students’ use of source attributes to select useful sources. Journal of Educational Computing Research, 41, 63–82. Braasch, J. L. G., Rouet, J.-F., Vibert, N., & Britt, M. A. (2012). Readers’ use of source information in compre­ hension. Memory & Cognition, 40, 450–465. Brand-Gruwel, S., Kammerer, Y., van Meeuwen, L., & van Gog, T. (in press). Source evaluation of domain experts and novices during Web search. Journal of Computer Assisted Learning. Brand-Gruwel, S., Wopereis, I., & Vermetten, Y. (2005). Information problem solving by experts and novices: Analysis of a complex cognitive skill. Computers in Human Behavior, 21(3), 487–508. Brand-Gruwel, S., Wopereis, I., & Walraven, A. (2009). A descriptive model of Information Problem Solving while using Internet. Computers & Education, 53, 1207–1217. Bråten, I., Braasch, J. L., Strømsø, H. I., & Ferguson, L. E. (2015). Establishing trustworthiness when students read multiple documents containing conflicting scientific evidence. Reading Psychology, 36(4), 315–349. Bråten, I., Ferguson, L. E., Strømsø, H. I., & Anmarkrud, Ø. (2014). Students working with multiple conflicting documents on a scientific issue: Relations between epistemic cognition while reading and sourcing and argumentation in essays. British Journal of Educational Psychology, 84, 58–85. Bråten, I., Salmerón, L., & Strømsø, H. I. (2016). Who said that? Investigating the Plausibility-Induced Source Focusing assumption with Norwegian undergraduate readers. Contemporary Educational Psychology, 46, 253–262. Bråten, I., Stadtler, M., & Salmerón, L. (in press). The role of sourcing in discourse comprehension. In M. F. Schober, M. A. Britt, & D. N. Rapp (Eds.), Handbook of discourse processes (2nd ed.). New York: Routledge. Bråten, I., Strømsø, H. I., & Britt, M. A. (2009). Trust matters: Examining the role of source evaluation in students’ construction of meaning within and across multiple texts. Reading Research Quarterly, 44, 6–28. Bråten, I., Strømsø, H. I., & Salmerón, L. (2011). Trust and mistrust when students read multiple information sources about climate change. Learning and Instruction, 21, 180–192. Britt, M. A., Rouet, J. F., & Braasch, J. L. G. (2013). Documents experienced as entities: Extending the situation model theory of comprehension. In M. A. Britt, S. R. Goldman, & J. F. Rouet (Eds.), Reading: From words to multiple texts (pp. 160–179). New York: Routledge. Carnegie Council on Advancing Adolescent Literacy (CCAAL). (2010). Time to act: An agenda for advancing adolescent literacy for college and career success. New York: Carnegie Corporation of New York. Cho, B. Y. (2014). Competent adolescent readers’ use of Internet reading strategies: A think-aloud study. Cognition and Instruction, 32, 253–289. Cho, B. Y., & Afflerbach, P. (2015). Reading on the Internet. Journal of Adolescent & Adult Literacy, 58, 504–517. Council of Chief State School Officers (2010). Time to act: An agenda for advancing adolescent literacy for college and career success. New York: Carnegie Corporation of New York. Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381.ldman, S. R., & Brand-Gruwel, S. (in press). Learning from multiple sources in a digital society. In F. Fischjer, C. E. Hmelo-Silver, S. R. Goldman, & P. Reimann (Eds.), International handbook of the learning sciences. New York: Routledge. Goldman, S. R., Britt, M. A., Brown, W., Cribb, G., George, M., Greenleaf, C., Lee, C. D., Shanahan, C., & Project READI (2016). Disciplinary literacies and learning to read for understanding: A conceptual frame­work for disciplinary literacy. Educational Psychologist, 51, 219–246. Goldman, S. R., Lawless, K. A., & Manning, F. (2013). Research and development of multiple source compre­ hension assessment. In M. A. Britt, S. R. Goldman, & J. F. Rouet (Eds.), Reading: From words to multiple texts (pp. 180–199). New York: Routledge. Goldman, S. R., Lawless, K. A., Pellegrino, J. W., Braasch, J. L. G., Manning, F. H., & Gomez, K. (2012). A tech­ nology for assessing multiple source comprehension: an essential skill of the 21st Century. In M. Mayrath, J. Clarke-Midura, & D. H. Robinson (Eds.). Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research (pp. 171–207). Charlotte, NC: Information Age. Goldman, S. R., & Lee, C. D. (2014). Commentary on text complexity: State of the art and the conundrums it raises. Elementary Education Journal, 115, 290–300.

Multiple Resource Comprehension  •  483 Goldman, S. R., & Scardamalia, M. (2013). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities. Cognition and Instruction, 31, 255–269. Hartman, D. K. (1995). Eight readers reading: The intertextual links of proficient readers reading multiple passages. Reading Research Quarterly, 30, 520–561. Kammerer, Y., Amann, D. G., & Gerjets, P. (2015). When adults without university education search the Internet for health information: The roles of Internet-specific epistemic beliefs and a source evaluation intervention. Computers in Human Behavior, 48, 297–309. Kammerer, Y., Bråten, I., Gerjets, P., & Strømsø, H. I. (2013). The role of Internet-specific epistemic beliefs in laypersons’ source evaluations and decisions during Web search on a medical issue. Computers in Human Behavior, 29(3), 1193–1203. Kammerer, Y., & Gerjets, P. (2012). Effects of search interface and Internet-specific epistemic beliefs on source evaluations during Web search for medical information: An eye-tracking study. Behaviour & Information Technology, 31(1), 83–97. Kammerer, Y., & Gerjets, P. (2014). The role of search result position and source trustworthiness in the selec­ tion of web search results when using a list or a grid interface. International Journal of Human-Computer Interaction, 30(3), 177–191. Kammerer, Y., Kalbfell, E., & Gerjets, P. (2016). Is this information source commercially biased? How contradic­ tions between web pages stimulate the consideration of source information. Discourse Processes, 53, 430–456. Keane, M. T., O’Brien, M., & Smyth, B. (2008). Are people biased in their use of search engines? Communications of the ACM, 51(2), 49–52. Keck, D., Kammerer, Y., & Starauschek, E. (2015). Reading science texts online: Does source information influence the identification of contradictions within texts?. Computers & Education, 82, 442–449. Lawless, K. A., Goldman, S. R., Gomez, K., Manning, F., & Braasch, J. (2012). Assessing multiple source comprehension through Evidence Centered Design. In J. P. Sabatini, T. O’Reilly, & E. R. Albro (Eds.), Reaching an understanding: Innovations in how we view reading assessment (pp. 3–18). Lanham, MD: R & L Education. Lee, C. D., & Spratley, A. (2010). Reading in the disciplines: The challenges of adolescent literacy. New York: Carnegie Corporation of New York. Lucassen, T., Muilwijk, R., Noordzij, M. L., & Schraagen, J. M. (2013). Topic familiarity and information skills in online credibility evaluation. Journal of the American Society for Information Science and Technology, 64, 254–264. Lund, E. S., Bråten, I., Brante, E. W., & Strømsø, H. I. (2017). Memory for textual conflicts predicts sourcing when adolescents read multiple expository texts. Reading Psychology, 38, 417–437. Macedo-Rouet, M., Braasch, J. L. G., Britt, M. A., & Rouet, J. F. (2013). Teaching fourth and fifth graders to evaluate information sources during text comprehension. Cognition and Instruction, 31, 204–226. Maier, J., & Richter, T. (2013). Text belief consistency effects in the comprehension of multiple texts with conflicting information. Cognition and Instruction, 31, 151–175. Mason, L., Junyent, A. A., & Tornatora, M. C. (2014). Epistemic evaluation and comprehension of web-source information on controversial science-related topics: Effects of a short-term instructional intervention. Computers & Education, 76, 143–157. McCrudden, M. T., Stenseth, T., Bråten, I., & Strømsø, H. I. (2016). The effects of topic familiarity, author expertise, and content relevance on Norwegian students’ document selection: A mixed methods study. Journal of Educational Psychology, 108, 147–162. Mislevy, R., & Haertel, G. (2006). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25, 6–20. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). Focus article: On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3–62. National Assessment of Educational Progress (NAEP) (2009). NAEP 2008 trends in academic progress (NCES 2009–479). Prepared by Rampey, B. D., Dion, G. S., & Donahue, P. L. for the National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education, Washington, D.C. Next Generation Science Standards Lead States (2013). Next generation science standards: For states, by states. Washington, D.C.: National Academies Press. Organisation for Economic Co-operation and Development (OECD) (2013). PISA 2012: Results in focus. Paris: OECD. Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007). In Google we trust: Users’ decisions on rank, position, and relevance. Journal of Computer Mediated Communication, 12, 801–823.

484  •  Goldman et al. Pellegrino, J. W., Chudowsky, N., & Glaser, R., (Eds.) (2002). Knowing what students know: The science and design of educational assessment. Board on Testing and Assessment, Center for Education. Division of Behavioral and Social Sciences and Education. Washington, D.C.: National Academy Press. Pellegrino, J. W., DiBello, L. V., & Goldman, S. R. (2016). A framework for conceptualizing and evaluating the validity of instructionally relevant assessments. Educational Psychologist, 51, 59–81. Pellegrino, J. W., & Hilton, M. L. (Eds.) (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century. National Research Council. Committee on Defining Deeper Learning and 21st Century Skills. Washington, D.C.: The National Academies Press. Perfetti, C. A., Britt, M. A., & Georgi, M. (1995). Text-based learning and reasoning: Studies in history. Hillsdale, NJ: Erlbaum. Perfetti, C. A., Rouet, J. F., & Britt, M. A. (1999). Toward a theory of documents representation. In H. van Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 99–122). Mahwah, NJ: Erlbaum. Rouet, J. F. (2006). The skills of document use: From text comprehension to Web-based learning. Mahwah, NJ: Erlbaum. Rouet, J. F., & Britt, M. A. (2011). Relevance processes in multiple documents comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 19–52). Charlotte, NC: Information Age. Rouet, J. F., Britt, M. A., Mason, R. A., & Perfetti, C. A. (1996). Using multiple sources of evidence to reason about history. Journal of Educational Psychology, 88, 478–493. Rouet, J. F., Le Bigot, L., Pereyra, G., & Britt, M. A. (2016). Whose story is this? Discrepancy triggers readers’ attention to source information in short narratives. Reading and Writing, 29, 1549–1570. Royer, J. M., Carlo, M. S., Dufresne, R., & Mestre, J. (1996). The assessment of levels of domain expertise while reading. Cognition and Instruction, 14, 373–408. Salmerón, L., Kammerer, Y., & García-Carrión, P. (2013). Searching the Web for conflicting topics: Page and user factors. Computers in Human Behavior, 29, 2161–2171. Saux, G., Britt, A., Le Bigot, L., & Vibert, N., Burin, D., & Rouet, J. F. (2017). Conflicting but close: Readers’ integration of information sources as a function of their disagreement. Memory and Cognition, 45, 151–167. Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp.  97–118). New York: Cambridge University Press. Schoenbach, R., Greenleaf, C., & Murphy, L. (2012). Reading for understanding: How reading apprenticeship improves disciplinary learning in secondary and college classrooms (2nd ed.). San Francisco, CA: Jossey-Bass. Stadtler, M., & Bromme, R. (2008). Effects of the metacognitive computer-tool met.a.ware on the web search of laypersons. Computers in Human Behavior, 24, 716–737. Stadtler, M., & Bromme, R. (2014). The content-source integration model: A taxonomic description of how readers comprehend conflicting scientific information. In D. N. Rapp & J. L. G. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 379–402). Cambridge, MA: The MIT Press. Stadtler, M., Scharrer, L., Macedo-Rouet, M., Rouet, J. F., & Bromme, R. (2016). Improving vocational students’ consideration of source information when deciding about science controversies. Reading and Writing, 29, 705–729. Steffens, B., Britt, M. A., Braasch, J. L. G., Strømsø, H. I., & Bråten, I. (2014). Memory for scientific arguments and their sources: Claim-evidence consistency matters. Discourse Processes, 51, 117–142. Strømsø, H. I., Bråten, I., Britt, M. A., & Ferguson, L. E. (2013). Spontaneous sourcing among students reading multiple documents. Cognition and Instruction, 31, 176–203. VanSledright, B. (2002). Confronting history’s interpretive paradox while teaching fifth graders to investigate the past. American Education Research Journal, 39, 1089–1115. Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009). Source evaluation, comprehension, and learning in Internet science inquiry tasks. American Educational Research Journal, 46, 1060–1106. Wineburg, S. S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87.

27

ASSESSING ONLINE COLLABORATIVE INQUIRY AND SOCIAL DELIBERATION SKILLS AS LEARNERS NAVIGATE MULTIPLE SOURCES AND PERSPECTIVES Julie Coiro university of rhode island, usa

Jesse R. Sparks educational testing service, usa

Jonna M. Kulikowich the pennsylvania state university, usa

Learners in today’s knowledge society have access to an overwhelming, ever-increasing amount of information. The Internet has become a universal source of information with new content increasing more than five exabytes every day (Internet Live Stats, 2016). Sorting through, processing, learning from, and effectively using this information demands a wide range of online research and inquiry skills. To answer questions and solve problems, individuals must rely on their own knowledge and ability to analyze and construct meaning from the tremendous amount of information with which they are confronted each day (Leu, Kinzer, Coiro, Castek, & Henry, 2013). In addition, on the Internet, answers to open-ended problems are rarely found from a single source. Online readers encounter multiple and diverse sources with different purposes and varied quality. To effectively integrate and reconcile competing perspectives while making sense of controversial issues (those about which there may be multiple viewpoints), learners require skills in organizing, evaluating, analyzing, comparing, contrasting, and integrating information drawn from multiple documents (Britt & Rouet, 2012; Goldman, Lawless, & Manning, 2013). Thus, the ability to recall and summarize single texts is insufficient to effectively use online texts. Further, to truly understand complicated societal issues, as students navigate across multiple sources, they must also move beyond their own perspective to build

485

486  •  Coiro et al.

an understanding reflecting multiple perspectives (Barzilai & Zohar, 2012; Schwarz & Asterhan, 2010). Interacting with others from around the world while exploring online sources almost guarantees that learners will regularly encounter opinions and ideologies different from their own. Developing the capacity to deliberate, or deal productively with these differences, is necessary to understand complex global issues or simple interactions with others (Murray, Xu, & Woolf, 2013). Despite the importance of these online inquiry and deliberation skills, students have difficulty planning, monitoring and regulating their inquiry processes when reading within and across multiple documents (Kuiper, Volman, & Terwel, 2005; Zhang & Quintana, 2012). In addition to revealing a range of online locating challenges (Knight & Mercer, 2015), studies indicate that many students engage with online sources in a superficial and uncritical manner (Coiro, Coscarelli, Maykel, & Forzani, 2015; Stanford History Education Group [SHEG], 2016), and they fail to see connections within and across different types of sources (Barzilai & Zohar, 2012). Some research suggests that collaborating with peers might help students work through the complexities of inquiry (see, for example, Duschl & Osborne, 2002; Liu & Hmelo-Silver, 2010). However, many also recognize that working jointly with others may complicate tasks, due to factors such as limitations in students’ task-relevant background knowledge and problems with social dynamics (Chung, Leet, & Liut, 2013; Kreijns, Kirschner, & Jochems, 2003). The act of partnering students, by itself, does not necessarily ensure collaboration (Häkkinen & Mäkitalo-Siegl, 2007; King & Rosenshine, 1993). Thus, to better understand the relations among collaboration, online inquiry and deliberation, it is important to define, characterize, and develop valid and reliable ways of simultaneously capturing evidence of these interacting competencies. This volume speaks to the body of work emerging in this area, and, as we review in this chapter, several assessments have been designed to measure one or more of these competencies. However, to our knowledge, no single assessment has been created to assess online inquiry, collaboration, and social deliberation as an integrated performance during group-based tasks. The purpose of this chapter is three-fold. First, we synthesize key theories and research that inform our vision of how online collaborative inquiry and social deliberation work together in digital learning environments. Second, we review current efforts to assess these constructs, including details from our emerging work under the National Assessment of Educational Progress (NAEP) Survey Assessment Innovations Laboratory (SAIL) Initiative, which is dedicated to exploring potential innovations in large-scale assessment by leveraging cognitive science research and new technologies. Finally, we outline key challenges associated with work in this area as well as implications for educational research and practice.

THEORETICAL BACKGROUND Online Inquiry Our vision of online collaborative inquiry and social deliberation is based on three interconnected lines of theory and research. First, we draw on both inquiry-based theories of multiple-document comprehension (Goldman et al., 2013; Rouet & Britt, 2011)

Assessing Collaborative Inquiry Skills  •  487

and a new literacies perspective of online reading comprehension (Leu, Kinzer, Coiro, & Cammack, 2004; Leu et al., 2013). More specifically, models of multiple-document comprehension involve: 1) task interpretation; 2) searching to gather information; 3) evaluating information resources’ relevance and reliability; 4) analyzing, synthesizing, and integrating within and across multiple documents; and 5) applying information to achieve task goals (Goldman et al., 2013; Rouet & Britt, 2011). These models also include a metacognitive self-regulation component that permits evaluation of whether task goals have been satisfied, allowing learners to adjust their inquiry as needed. Challenges associated with identifying, connecting, and integrating relevant text content with source information (e.g., author expertise and affiliation, publication venue) make inquiry across multiple documents especially complex (Britt & Rouet, 2012). A new literacies perspective of online reading comprehension considers additional challenges as learners engage with multiple documents on the Internet. Online readers engage in inquiry while using Internet texts and tools to accomplish at least five complex practices: generating important questions, navigating to locate online information, evaluating information critically, synthesizing information across multiple media, and reading and writing to communicate learned information. These online reading practices require new literacy skills and strategies not accounted for in more general models of multiple-document comprehension, such as accessing search engines, generating reasonable search terms, navigating multilevel websites, monitoring navigational pathways, and managing a lack of uniform standards across online sources (Afflerbach & Cho, 2009; Coiro, 2011; Coiro & Dobler, 2007; Leu et al., 2013). For our purposes, we use the term online inquiry to represent inquiry-based practices associated with both multiple-document comprehension and online reading comprehension. Consequently, assessment tasks should actively engage learners in both multiple-document and online reading practices to elicit evidence of the knowledge and skills required for competence in online inquiry.

COLLABORATIVE LEARNING A second area that informs our conception of online collaborative inquiry and social deliberation is collaborative learning. Increasingly, students are expected to be able to work effectively in groups and apply their problem-solving skills in a range of social situations (Organisation for Economic Co-operation and Development [OECD], 2017). As a result, there is growing interest in assessing collaboration as a critical 21stcentury skill (Griffin & Care, 2015; von Davier, 2017). We conceptualize collaborative learning during online inquiry as involving “groups of learners working together to solve a problem, complete a task, or create a product” (Laal & Laal, 2012, p. 491). During inquiry, social interaction and collaborative discussion can play an important role in helping to construct new ideas and solve problems (see Salomon & Perkins, 1998; Wertsch, 1991). Peer collaboration, in particular, has been shown to increase students’ conceptual understanding of subject matter (Liu & Hmelo-Silver, 2010) while providing opportunities to develop group cohesion around shared conceptions of an issue (Roschelle & Teasley, 1995; Schwartz, 1995). Thus, collaborating during online inquiry may foster deeper understanding of content across multiple documents as well as partners’ ability to build a shared representation of their

488  •  Coiro et al.

learning across diverse perspectives, as compared to individual inquiry that does not require one’s thinking to be shared and negotiated with others. Notably, many earlier studies of peer collaboration emphasized cognitive aspects of problem solving while neglecting the role that social dynamics play in the success of any peer-to-peer partnership (Kreijns et al., 2003). More recently, however, researchers have begun to acknowledge and articulate the mutually important roles that cognitive processes and social interactions (e.g., negotiating ideas, regulating goals) play in collaborative problem solving (CPS; see, for example, Griffin & Care, 2015; Kirschner & Erkens, 2013; Liu, Hao, von Davier, Kyllonen, & Zapata-Rivera, 2015), including their value as a source of evidence of students’ collaborative skills. Small-scale studies have characterized various interaction patterns that occur as students work together to solve information problems while reading across online sources (see Castek, Coiro, Guzniczak, & Bradshaw, 2012; Coiro, Castek, Sekeres, & Guzniczak, 2014; Kiili, Laurinen, Marttunnen, & Leu, 2012). Findings suggest that while some students benefitted from working with a partner, others did not. These conflicting findings illustrate the need to better understand collaborative interaction (see Andrews & Rapp, 2015; Kuhn, 2015) and its relationship to learning in online networked spaces (see Foster, 2009). For our purposes, we use the term online collaborative inquiry to represent interactions among group members during online inquiry.

SOCIAL DELIBERATION A third area that informs our work assumes the key role that argumentation (Nussbaum, 2008) and social deliberation (Murray, 2013) play in students’ deep-level understanding of content and learning. Argumentation, defined as an attempt to increase or decrease the acceptability of one or more ideas by reasoning (van Eemeren, Grootendorst, & Henkemans, 1996), lies at the heart of human thinking, and is essentially a social activity (Baker, 2002). Skillful argumentation incorporates the ability to articulate one’s own argument while considering the merits of counterarguments from more than one perspective (Kuhn & Udell, 2003). Like Schwarz and Asterhan (2010), our interests lie in capturing processes involved when two or more people engage in reasoned dialogue to consider multiple perspectives around a claim (see Baker, 2002). Specifically, we focus on reasoning that involves the questioning, clarification, explanation, justification, and elaboration of ideas as peers work together to co-construct solutions (see Kruger, 1993). These reasoning processes are particularly important when students explore open-ended questions with many alternative solutions and stakeholder views (Marttunen & Laurinen, 2006). One possible goal of argumentation is that of deliberation, a process by which an individual works to understand multiple perspectives on an issue to support informed decision-making (Murray, 2013). Individuals can deliberate, or engage in careful consideration of ideas on their own, as part of decision-making or problem-solving activities. When two people work together and they encounter discrepant ideas or differences of opinion, social deliberation helps them build a shared knowledge

Assessing Collaborative Inquiry Skills  •  489

representation of these differences while working toward a reasonable solution on which both individuals can agree (Fearon, 1998; Murray, 2013). According to Clark and Brennan (1991), collaborative partners engage in a wide range of communicative interactions, including: building a shared representation of their understanding of multiple ideas; understanding each other’s roles; understanding the abilities and perspectives of each partner; mutually tracking the transfer of information and feedback between partners; and monitoring progress toward a common solution. During social deliberation, collaboration is characterized by these capacities in the context of perspective taking, perspective seeking, perspective monitoring, and perspective weighing (Murray et al., 2013). In contrast to traditional argumentation approaches that focus on fostering students’ use of logical, objective, and well-supported explanations and efficient solutions (see Andriessen, Baker, & Suthers, 2003), Murray and colleagues emphasize the inherently social and intersubjective skills needed for mutual understanding and mutual recognition as part of deliberation (Murray et al., 2013). Discussion that involves these co-constructive deliberation skills can foster positive educational outcomes (higher test scores; Kawashima-Ginsburg, 2013) and develop more democratic citizens (Graseck, 2009; Hess, 2009). Moreover, engaging in argumentation that moves beyond persuasion (seeking to promote certain opinions in a win or lose format) toward social deliberation (for the purpose of reaching consensus) positively influences the quality of reasoning processes and written argumentation products (Felton, Crowell, & Liu, 2015) as well as the amount of content learned (Felton, Garcia-Mila, Villarroel, & Gilabert, 2015). Nevertheless, Schwarz and Asterhan (2010) point out current shortcomings in being able to systematically capture and characterize reasoning processes involved in argumentation and social deliberation. To address the challenges in simultaneously assessing these processes, some have begun to capture argumentative reasoning through analyses of jointly constructed argumentative maps and essays (see Coiro & Kiili, 2016; Schwarz & de Groot, 2007) or direct observation (Osborne, Erduran, & Simon, 2004). However, more work in this area lies ahead (Mislevy, 2016).

ASSESSMENTS OF ONLINE INQUIRY, COLLABORATION, AND SOCIAL DELIBERATION The previous section of this chapter introduced and defined key terms relevant to online inquiry, collaboration, and social deliberation. Next, we review how this set of competencies has been measured in current assessments. Finally, we describe a digital environment that seeks to integrate these measurement approaches in the context of a single assessment. But first, we briefly highlight the importance of principled approaches to developing performance-based assessments of complex online cognitive and social processes. Designing Assessments Designing quality assessments of student learning begins with consideration of three critical elements: cognitively grounded theories of domain-relevant skills and

490  •  Coiro et al.

competencies and their development, situations wherein those skills might be demonstrated or observed, and a process for meaningfully interpreting evidence from those observations (Pellegrino, Chudowsky, & Glaser, 2001). This interconnected “assessment triangle” corresponds to three components of the conceptual assessment framework within Evidence-Centered Design (ECD; Mislevy, Steinberg, & Almond, 2003): the student model, or the set of knowledge, skills, and proficiencies to be assessed; the evidence model, behaviors that provide evidence of the targeted proficiencies (including scoring rules); and the task model, situations that afford opportunities to elicit and observe those behaviors. Alignment among target constructs, tasks, and evidence is necessary to ensure valid assessments that permit sound inferences about students. Mislevy (2016) argues that as technology advances, the term assessment “encompasses an expanding range of purposes, targets of inference, contexts of use, forms of activity, and sources of evidence” (p. 265) that necessitate expanded notions of what to consider when designing valid educational measures. Specifically, new technologies offer promising ways to prompt and to capture evidence of digital interactions as students work to obtain, evaluate, integrate, and use information within rich, performance-based assessments of complex cognitive constructs like online inquiry. Additionally, sociocognitive extensions of ECD explicitly account for measurement of collaborative processes like social deliberation (Kerr, Andrews, & Mislevy, 2016; Mislevy, 2016). We now review contemporary efforts to assess students’ proficiency with online collaborative inquiry and social deliberation. Review of Current and Emerging Assessments While to date, no single assessment has been created to assess online collaborative inquiry and social deliberation as an integrated performance, several assessments measure one or more aspects of these competencies. Table 27.1 summarizes several efforts to assess online inquiry and collaboration skills in the 21st century, with an emphasis on K-12 populations (see Sparks, Katz, & Beile, 2016 for a review of related digital information literacy assessments for higher education). For each assessment, Table 27.1 describes its: 1) purpose and target population; 2) delivery format; 3) available forms, tasks, and items; 4) length; 5) content domains or disciplinary contexts; and 6) use of ECD (or other principled approaches). We also note whether the following competencies are measured: online reading comprehension, multiple-document use, collaborative inquiry, and social deliberation. Of these, online reading and multipledocument use are considered individual cognitive skills, while collaborative inquiry and social deliberation represent social or sociocognitive skills. Broadly, the assessments fall into two categories: those emphasizing individual, cognitive skills involved in online inquiry (Table 27.1, rows 1–7), and those emphasizing individual cognitive and social skills required for successful collaborative problem solving (CPS; rows 8–9). We briefly summarize the constructs, task designs, and evidence collected by available assessments within each category. The review concludes by describing a new project under the NAEP SAIL initiative that builds on those assessment efforts to measure individual online inquiry (row 10) and to simultaneously capture elements of online collaborative inquiry and social deliberation (row 11).

Purpose

Population

Delivery

Forms, Tasks, Items

Length

Web-based

Paperand-pencil (Middle and high school); Web-based (Collegelevel) Web-based

Middle school, High school

Middle school, High school, College

Grade 7

Measures ability to critically evaluate online information in the context of reasoning about civic issues, including analysis of information presented on social media and in advertisements

Scenarios measure processes and products of online reading comprehension and research skills, including the abilities to locate, evaluate, synthesize, and communicate research results

Online Research & Comprehension Assessment (ORCA; Leu, Kulikowich, Sedransk, & Coiro, 2009)

Council for Aid to Education’s College and Work Readiness Assessment + (CWRA+; Council for Aid to Education, 2015)

Assessment of Civic Online Reasoning (Stanford History Education Group, 2016)

Web-based

Measures multiple-document comprehension skills in the context of disciplinary inquiry tasks (evidence-based argumentation) in history, science, and literature Measures critical thinking, problem solving, and written communication skills by requiring students to analyze, evaluate, and apply information from multiple documents in a written argumentation task

Project READI (Goldman et al., this volume)

Grades 6 to 12

4 SBTs with 16 items; 2 versions: simulations and CR items (ORCA-Closed); SR (multiple choice) items (ORCA-MC)

Multiple forms each with Performance Task (4–9 texts for high school, fewer for middle school) and 25 SR items 56 tasks across 15 types (5 per grade level); SR, CR items

Multiple SBTs (2+ per domain) using SR, Rating, ECR items

Civics

Varies

ELA, Average 25 (ORCA-MC) Science to 45 minutes (ORCAClosed)

NO; experts develop and review tasks

Science, Social Sciences

90 minutes (60-minute performance task, 30-minute SR section)

YES

NO; “design thinking”

YES

Uses ECD?

Literature, Science, History

Domain

45 minutes

Individual Assessments of Online Reading Comprehension, Online Inquiry, and/or Multiple-Document Use

Assessment

Table 27.1  Review of Assessments.

(Continued)

ORC* MDU (except for Middle School) *Middle and high school read static versions of online documents; College reads live websites ORC MDU OCI (Simulated)

ORC* MDU *Limited; students read static versions of online documents

ORC MDU

Constructs Assessed?

International assessment measures reading to acquire and use information in online contexts, including locating explicit information, making inferences, interpreting and integrating, and evaluating websites Summative assessments measure ELA research and inquiry skills, including multiple documents and perspectives, with tasks aligned to hypothesized learning progressions based on cognitive learning sciences literature Summative scenario-based assessments measure foundational and applied reading comprehension skills and strategies, including building knowledge from multiple texts

ePIRLS: Progress in International Reading Literacy Study (Mullis & Martin, 2015)

Web-based

Web-based

Web-based

Grade 4

Grades 6 to 8

Pre-K to Grade 12

Programme for International Student Assessment: Collaborative Problem Solving (PISA-CPS; OECD, 2017)

Measure individual students’ collaborative problemsolving skills in situations where they work together to achieve consensus with virtual collaborators using simulated conversations using a branching selected-response format

15-year-olds (international)

Web-based

Delivery

Population

Individual Assessments of Collaborative Problem Solving

GISA (Sabatini et al., this volume)

CBAL Research & Inquiry (Sparks & Deane, 2015)

Purpose

Assessment

Table 27.1  (Continued)

45 minutes

Academic and civic contexts

ELA, Literature, Science, Social Sciences

ELA, Science, Social Sciences

ELA, Science, Social Sciences

80 minutes (40 per task)

90 minutes

Domain

Length

6 units; each unit 5, 10, 15, or 20-minute contains chattasks like SR items and interactive items

20+ SBTs using SR, CR, ECR items

2 SBTs using SR, CR, ECR items

4 SBTs with 2–3 websites and comprehension items

Forms, Tasks, Items

YES

YES

YES

NO; leverages PIRLS framework

Uses ECD?

ORC* MDU CPS (Simulated) *Limited (some use of hyperlinks and digital texts)

ORC MDU OCI (Simulated)

ORC MDU OCI (Simulated)

ORC MDU OCI (Simulated)

Constructs Assessed?

International assessment measures 11–15-yearadolescents’ individual proficiency olds with collaborative problem solving (international) (CPS) and learning through digital networks (LDN), including cognitive and social proficiencies like perspective taking and mutual understanding

Web-based platform enabling realtime remote collaboration via text chat

11 CPS tasks developed, 2 of 3 developed LDN tasks retained; tasks require use of digital tools

Prototype tools and tasks capture products and processes of individual multiple-document inquiry (planning, locating, evaluating, synthesizing, and communicating) in an extended scenario Adapt an online multipledocument inquiry task to enable two students to collaborate remotely, affording opportunities to explore the measurement of social deliberation

Web-based platform for delivering scenariobased tasks Web-based platform enabling realtime remote collaboration via text and audio/video chat

Grade 8

Grades 8 to 10

90 minutes

120 minutes

1 SBT with SR, Rating, CR, ECR items

1 SBT with SR, Rating, CR, ECR items

Varies

ELA, Social Sciences (U.S. History)

ELA, Science, Social Sciences (History)

Contexts include academic content or abstract reasoning puzzles

YES

YES

ORC MDU OCI SD

ORC MDU OCI (Simulated)

ORC (LDN) YES (CPS MDU (LDN) only); LDN CPS uses Berkeley Evaluation and Assessment Research (BEAR) assessment system

Note: ORC = Online reading comprehension, MDU = Multiple-document use, CPS = Collaborative problem solving, OCI = Online collaborative inquiry, (Simulated) = Collaboration is simulated with avatars, SD = Social deliberation, ECD = Evidence-Centered Design, SBT = Scenario-based tasks; ELA = English language arts, SR = Selected response, CR = Constructed response, ECR = Extended constructed response, LDN = Learning in Digital Networks.

NAEP SAIL Online Collaborative Inquiry & Social Deliberation in Virtual Worlds (SAIL-OCISD; Coiro, Sparks, Kiili, & Castek, 2016)

NAEP SAIL Virtual Worlds for ELA Inquiry (SAIL-ELA; Sparks, 2014)

NAEP SAIL Assessments of Individual/Collaborative Online Inquiry and Social Deliberation

Assessment and Teaching of 21st Century Skills (ATC21S; Griffin & Care, 2015)

494  •  Coiro et al.

ASSESSMENTS OF ONLINE INQUIRY Assessments designed to measure individual students’ online inquiry skills range from established testing programs to ongoing research and development efforts. These include the Online Research and Comprehension Assessment (ORCA; Leu, Kulikowich, Sedransk, & Coiro, 2009), Progress in International Reading Literacy Survey’s ePIRLS (Mullis & Martin, 2015), Cognitively Based Assessment of, for, and as Learning (CBAL™; Sparks & Deane, 2015), Global Integrated ScenarioBased Assessment (GISA; Sabatini, O’Reilly, Wang, & Dreier, this volume), Project READi (for Reading, Evidence, and Argumentation in Disciplinary Instruction; Goldman et al., this volume), College and Work Readiness Assessment (CWRA+; Council for Aid to Education, 2015), and the Assessment of Civic Online Reasoning (SHEG, 2016). Constructs Across assessments, questions about digital texts seek to measure cognitive skills related to locating, evaluating, synthesizing, and communicating information, particularly from multiple documents. Content domains span literary analysis, science inquiry, and social sciences, with assessments covering all grade levels. Both ORCA and ePIRLS simulate dynamic web environments requiring navigation, while others provide static representations of websites or more traditional text types. Some assessments tap integrated performances spanning multiple competencies (e.g., write a source-based argumentative essay), but most involve some sequencing or task decomposition (e.g., identifying multiple perspectives and judging source reliability), which permits estimations of proficiency with or relations among component skills. Tasks With some exceptions, the assessments use scenario-based formats (see Sabatini et al., this volume) in which students must use information from provided sources to address an overarching inquiry question or task goal. Avatars, or virtual characters, often serve to engage students in simulated collaborative inquiry by delivering task instructions, sequencing, hints, and feedback. Several assessments require students to demonstrate their ability to make use of research results in a culminating task by communicating it to others in an extended constructed-response (CR) task. Evidence Most assessments use automated scoring when possible to maximize reliability and cost-effectiveness. Typically, student responses on traditional selected-response (SR) or CR item types are evaluated, rather than scoring on the basis of students’ solution processes. Logfiles may capture student actions, but only some assessments score them (e.g., ORCA). Extended CR tasks are scored by human raters, automated writing evaluation systems, or a combination. Notably, many assessments explicitly used ECD in the development process (e.g., Goldman et al., 2013).

Assessing Collaborative Inquiry Skills  •  495

ASSESSMENTS OF COLLABORATIVE PROBLEM SOLVING Fewer assessments of collaborative problem solving exist, but two notable large-scale international efforts include the Programme for International Student Assessment’s 2015 Collaborative Problem Solving assessment (PISA-CPS; OECD, 2017) and assessments developed for the Assessment and Teaching of 21st Century Skills (ATC21S™; Griffin & Care, 2015). Both programs measure the CPS skills of individual students, working in real or simulated collaborative contexts. Constructs These assessments measure an individual’s social and cognitive processes during collaborative tasks. For example, the PISA-CPS Framework defines the construct as a matrix crossing individuals’ collaborative competencies with their cognitive problemsolving skills. The two ATC21S modules measure separate constructs, but both involve collaboratively using digital information to solve problems; the Learning in Digital Networks (LDN) module taps online reading and document use, while the CPS module emphasizes reasoning and joint decision-making. Perspective taking and establishing and maintaining mutual understanding are critical components across these three assessments. Tasks Scenario-based tasks are used to elicit individual students’ collaborative problemsolving skills. While PISA-CPS relies on simulated collaboration with avatars to assess students’ CPS skills as they interact via selected-response chat (i.e., choose an appropriate reply) and by actions taken in the task environment (e.g., clicking to view digital resources) to achieve group goals (e.g., determine a suitable field trip location), ATC21S’s CPS tasks engage dyads of real students in real-time collaboration as they communicate using text-based chat. Domain-specific and domaingeneral exercises are employed. Sometimes, tasks provide each participant with unique information and dyads must share relevant information via chat to complete the task; perspective taking is especially critical in these asymmetric tasks. For the ATC21S LDN tasks, small teams must assign roles, communicate via digital notebook, and build knowledge from multiple digital texts within scientific inquiry or literary interpretation scenarios. Evidence Both approaches use automated scoring based on logfiles that capture the unfolding collaborative process and related products. Tasks are automatically scored based on student actions (e.g., selecting appropriate feedback for a virtual collaborator), which enables estimation of individual proficiency levels on CPS competencies or social and cognitive readiness. ATC21S also applies simple natural language processing to identify keywords or numeric entries in students’ chat contributions. The LDN module employs both human and automated scorings.

496  •  Coiro et al.

ASSESSMENTS OF ONLINE COLLABORATIVE INQUIRY AND SOCIAL DELIBERATION As noted earlier, no single assessment in our review addresses the full breadth of the construct of online collaborative inquiry and social deliberation. However, emerging research under the NAEP SAIL research initiative has proposed to develop just such an assessment. Next, we describe this project as it evolved in two phases, informed by previous assessments we reviewed: first, the development of a virtual world platform to assess individuals’ performance in online inquiry, and second, a two-player expansion of this virtual world designed to simultaneously capture elements of online collaborative inquiry and social deliberation in a single assessment prototype. Online Inquiry in Virtual Worlds The SAIL Virtual World for ELA (SAIL-ELA; Sparks, 2014) aims to develop a flexible “virtual world” platform for eliciting evidence of individual students’ information gathering, processing, and evaluation skills in the context of multiple-document inquiry in English language arts (ELA). The SAIL-ELA platform is designed to measure students’ online inquiry skills as they carry out extended investigations requiring coordination of multiple critical reading and reasoning skills. Informed by prior domain and student modeling approaches (Goldman et al., 2013; Sparks & Deane, 2015), the SAIL-ELA inquiry construct integrates students’ reading, writing, and critical thinking into several sub-skills: 1) define information needs; 2) locate sources; 3) evaluate sources; 4) process, analyze, and synthesize sources; and 5) communicate research results. Given NAEP’s sampling of students at 4th, 8th, and 12th grades, we focus on middle school (Grade 8). Importantly, SAIL-ELA connects to but extends beyond competencies in the NAEP Reading framework (National Assessment Governing Board, 2012), which may help extend traditional reading assessments to better capture inquiry-based reading in digital environments. Tasks SAIL-ELA scenarios feature an overarching narrative context and goal. The virtual world depicts a town with various locations (e.g., university library, Internet café, public spaces) populated with digital documents (websites, database results) or nonplayer avatars (experts, laypeople) serving as resources. Tasks begin with a Setup phase, wherein students are introduced to the virtual world, characters, and task goals, and begin planning their research (e.g., identifying the best first step, narrowing their focus). In Free Roam, students freely explore available resources, engaging in self-regulated information gathering, processing, and evaluation activities, which are captured via responses to questions (SR and CR items) and the actions taken (or not). After retrieving sufficient resources, the Conclusion requires an overall response to the inquiry task (e.g., a source-based argument) that synthesizes the collected resources. Evidence The SAIL-ELA platform includes a suite of digital inquiry tools (cf. Zhang & Quintana, 2012) that elicit evidence of a specific aspect of the student model (e.g., virtual Library and

Assessing Collaborative Inquiry Skills  •  497

Internet Search environments simulate online reading and locating contexts). These tools elicit evidence of students’ inquiry skills from both responses representing the products of inquiry (i.e., answers to items, search engine queries), and observable data in digital logfiles including action sequences and keystroke logging. SAIL-ELA is specifically designed to permit flexible, self-regulated information gathering, which provides an opportunity to examine and characterize relationships among students’ information-gathering patterns and final scores. Preliminary cognitive labs suggest this less-constrained approach yields valuable information about students’ proficiency with aspects of multiple-document inquiry. Ongoing research aims to clarify the utility and validity of this approach. Online Collaborative Inquiry and Social Deliberation in Virtual Worlds The Online Collaborative Inquiry and Social Deliberation in Virtual Worlds project (SAIL-OCISD; Coiro, Sparks, Kiili, & Castek, 2016) is the only assessment designed to investigate issues related to all three sets of competencies articulated in this chapter – online inquiry, collaboration, and social deliberation. Specifically, we extend the SAILELA platform to develop a collaborative scenario-based inquiry task. Like ATC21S, we envision dyads of students engaging in real-time remote collaboration to address an overarching inquiry question by gathering, processing, and evaluating information gathered from multiple sources in the virtual world. However, in contrast to the individual-level assessments of CPS skills previously described, we aim to characterize group-level performance and individuals’ contributions to joint processes and outcomes. While online collaborative inquiry and social deliberation could be explored with groups of varied sizes, we focus on dyads in particular to lessen the complexities involved with modeling interactions among groups of three or more learners (see von Davier & Halpin, 2013; von Davier, 2017). Extending the SAIL-ELA student model, we specified a model of collaborative skills in the context of online inquiry tasks, informed by the theoretical and empirical work previously reviewed in this chapter. Specifically, we developed a mapping between cognitively focused problem-solving skills and subcomponents of the SAILELA student model and between socially focused CPS competencies and phases of the SAIL-ELA task model. Thus, SAIL-OCISD tasks can measure both cognitive and social constructs, using the same tools and resources as in single-user SAIL-ELA tasks. However, in SAIL-OCISD, dyads will cooperate to conduct inquiry activities and to construct a joint task product as they communicate via text or audio/video chat. Collaborative actions and discourse are elicited via text-based prompts (e.g., discuss your ideas with your partner before submitting a joint response) presented in a digital “wrapper” around the inquiry task (see Liu et al., 2015). This unique SAIL-OCISD assessment environment enables capturing of rich evidence of students’ online collaborative inquiry and social deliberation skills, with structured prompts that permit capturing initial individual responses as well as subsequent jointly negotiated responses (Liu et al., 2015). Evidence from team discourse in response to prompts can be captured via coding schemes and explicitly incorporated into the evidence model. This approach follows extensions of ECD designed to incorporate evidence from in-task actions and collaborator discourse extracted from logfiles (cf. Kerr et al., 2016). Ultimately, this project aims to develop an assessment prototype that effectively elicits collaborative dialogue during online inquiry.

498  •  Coiro et al.

IMPLICATIONS FOR THEORY, RESEARCH, AND PRACTICE This review lays the groundwork for thinking about how to define, capture, and differentiate collaborative processes and dialogic interactions between groups of students engaged in online inquiry and social deliberation. In contrast to working individually, the collaborative experience requires students to make their reasoning explicit to their partner, which can yield opportunities to expose and correct misunderstandings or misconceptions (Coiro et al., 2014; Kuhn, 2015). Further, efforts to develop online collaborative inquiry tasks can elicit additional evidence of students’ deliberative reasoning, through dialogue and interaction, that provides opportunities to assess both individual cognitive reasoning (i.e., online inquiry, regulation, and deliberation) and co-constructive, social deliberation skills (i.e., negotiation, perspective taking, consensus building). Continued theoretical and empirical research is warranted to clarify how students’ abilities to efficiently locate, critically evaluate, synthesize, regulate, and deliberate relate to their success at online inquiry across multiple documents. Understanding whether and how students’ abilities to engage in argumentation and social deliberation play a role in negotiating and understanding diverse perspectives during online inquiry is also important, given growing interest in understanding and measuring these 21st-century skills. Our review also illustrates that existing and emerging assessments provide tangible examples of how technology can facilitate online collaborative inquiry while also capturing and analyzing process and product data. However, multiple challenges remain (von Davier, 2017). Constructs such as social deliberation may require new assessment designs and psychometric evaluation of scores in which group achievement is considered collectively apart from individual contributions to learning. To date, largescale international assessments have estimated the proficiency of individual students, working in (real or simulated) collaborative contexts (Griffin & Care, 2015; OECD, 2017). Future research should explore how to disentangle individual from group-level contributions and how to handle dependencies in the data (von Davier & Halpin, 2013). Given the NAEP context of group-level score reporting, SAIL-OCISD plans to report at the group level (i.e., aggregating across dyads), rather than at the individual student level. However, some individual-level performance measures will be captured and used to interpret and characterize group-level performances (e.g., more and less successful dyads). Accounting for various student characteristics (e.g., gender, prior knowledge, proficiency level) and situational features (e.g., item types, prompt design, communication modality) that might influence successful collaboration (see Kreijns et al., 2003) is an important objective for future research to ensure that collaborative assessments are fair and valid for all students. Finally, to move from research into practice, practitioners should work collaboratively with assessment designers to inform decisions about which kinds of reporting output is most useful to help educators diagnose gaps in skill development, link to effective instructional practices, and measure progress over time. This will ensure that assessment scores not only have strong psychometric properties, but that they can also be considered as primary outcomes in intervention-based research initiatives. The combined psychometric quality of scores coupled with evidence of instructional treatment efficacy can, in time, provide stakeholders with theoretically sound

Assessing Collaborative Inquiry Skills  •  499

and actionable information that supports learning and the development of students’ online collaborative inquiry and social deliberation skills.

REFERENCES Afflerbach, P., & Cho, B. (2009). Identifying and describing constructively responsive comprehension strategies in new and traditional forms of reading. In S. Israel & G. Duffy (Eds.), Handbook of reading comprehension research (pp. 69–90). Mahwah, NJ: Erlbaum. Andrews, J. J., & Rapp, D. N. (2015). Benefits, costs, and challenges of collaboration for learning and memory. Translational Issues in Psychological Science, 1, 182–191. Andriessen, J., Baker, M., & Suthers, D. (Eds.) (2003). Arguing to learn: Confronting cognitions in computer supported collaborative learning environments (pp. 1–25). Boston, MA: Kluwer Academic. Baker, M. (2002). Forms of cooperation in dyadic problem-solving. In P. Salembier & H. Benchekroun (Eds.), Cooperation and complexity in sociotechnical systems (Vol. 16, pp. 587–620). Paris: Hermes. Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: Evaluating and integrating online sources. Cognition and Instruction, 30, 39–85. Britt, M. A., & Rouet, J.-F. (2012). Learning with multiple documents: Component skills and their acquisition. In M. J. Lawson & J. R. Kirby (Eds.), Enhancing the quality of learning: Dispositions, instruction, and learning processes (pp. 276–314). Cambridge, UK: Cambridge University Press. Castek, J., Coiro, J., Guzniczak, L., & Bradshaw, C. (2012). Examining peer collaboration in online inquiry. The Educational Forum, 76, 479–496. Chung, C., Leet, C., & Liut, C. (2013). Investigating F2F peer interaction patterns in collaborative web discovery tasks. Journal of Computer Assisted Learning, 29, 188–206. Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. Resnick, J. Levine, & S. Teasley (Eds.), Perspectives on socially shared cognition (pp. 127–149). Hyattsville, MD: American Psychological Association. Coiro, J. (2011). Predicting reading comprehension on the Internet: Contributions of offline reading skills, online reading skills, and prior knowledge. Journal of Literacy Research, 43, 352–392. Coiro, J., Castek, J., Sekeres, D., & Guzniczak, L. (2014). Comparing third, fourth, and fifth graders’ collaborative interactions while engaged in online inquiry. Journal of Education, 194(2), 1–16. Coiro, J., Coscarelli, C., Maykel, C., & Forzani, E. (2015). Investigating criteria seventh graders use to evaluate the quality of online information. Journal of Adolescent and Adult Literacy, 58, 546–550. Coiro, J., & Dobler, E. (2007). Exploring the online comprehension strategies used by sixth-grade skilled readers to search for and locate information on the Internet. Reading Research Quarterly, 42, 214–257. Coiro, J., & Kiili, C. (2016). Digital scaffolds for evaluating and synthesizing argumentative online texts. Unpublished final report. International Literacy Association’s Elva Knight Research Award. Coiro, J., Knobel, M., Lankshear, C., & Leu, D. J. (Eds.) (2008). Handbook of research in new literacies. Mahwah, NJ: Erlbaum. Coiro, J., Sparks, J. R., Kiili, C., & Castek, J. (2016). Capturing dimensions of collaboration and deliberation with multiple-source inquiry tasks. NAEP SAIL Research funded by the U.S. Department of Education, National Center for Education Statistics. Council for Aid to Education. (2015). CWRA+ Technical FAQs. New York, NY: Author. Retrieved from http:// cae.org/images/uploads/pdf/CWRA_Plus_Technical_FAQs.pdf. Duschl, R., & Osborne, J. (2002). Supporting and promoting argumentation discourse. Studies in Science Education, 38, 39–72. Fearon, J. D. (1998). Deliberation as discussion. In J. Elster (Ed.), Deliberative democracy (pp.  44–68). Cambridge, UK: Cambridge University Press. Felton, M., Crowell, A., & Liu, T. (2015). Arguing to agree: Mitigating the effects of my-side bias through consensus-seeking dialogue. Written Communication, 32, 317–331. Felton, M., Garcia-Mila, M., Villarroel, C., & Gilabert, S. (2015). Arguing collaboratively: Argumentation discourse types and their potential for knowledge building. British Journal of Educational Psychology, 85, 372–386. Foster, J. (2009). Understanding interaction in information seeking and use as a discourse: A dialogic approach. Journal of Documentation, 65, 83–105.

500  •  Coiro et al. Goldman, S. R., Lawless, K., & Manning, F. (2013). Research and development of multiple source comprehension assessment. In M. A. Britt, S. R. Goldman, & J.-F. Rouet. (Eds.), Reading from words to multiple texts (pp. 180–199). New York, NY: Routledge. Graseck, S. (2009). Teaching with controversy. Educational Leadership, 67(1), 45–49. Griffin, P., & Care, E. (Eds.) (2015). Assessment & teaching of 21st century skills: Methods and approach. New York, NY: Springer. Häkkinen, P., & Mäkitalo-Siegl, K. (2007). Educational perspectives on scripting CSCL. In F. Fischer, I. Kollar, H. Mandl, & J. M. Haake (Eds.), Scripting computer-supported collaborative learning: Cognitive, computational, and educational approaches (pp. 263–271). New York, NY: Springer. Hess. D. (2009). Controversy in the classroom: The democratic power of discussion. New York, NY: Routledge. Internet Live Stats. (2016). Google search statistics. Retrieved from www.internetlivestats.com/googlesearch-statistics. Kawashima-Ginsburg, K. (2013). Do discussion, simulations, and debate boost NAEP civics performance? Medford, MA: The Center for Information and Research on Civic Learning and Engagement. Kerr, D., Andrews, J. J., & Mislevy, R. J. (2016). The in-task assessment framework for behavioral data. In A. A. Rupp & J. P. Leighton (Eds.), The Wiley handbook of cognition and assessment: Frameworks, methodologies, and applications (pp. 472–507). West Sussex, UK: Wiley. Kiili, C., Laurinen, L., Marttunen, M., & Leu, D. (2012). Working on understanding during collaborative online reading. Journal of Literacy Research, 44, 448–483. King, A., & Rosenshine, B. (1993). Effects of guided cooperative questioning on children’s knowledge construction. Journal of Experimental Education, 61, 127–148. Kirschner, P. A., & Erkens, G. (2013). Toward a framework for CSCL research. Educational Psychologist, 48, 1–8. Knight, S., & Mercer, N. (2015). The role of exploratory talk in classroom search engine tasks. Technology, Pedagogy and Education, 24, 303–319. Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social collaboration in computersupported collaborative learning environments: A review of the research. Computers in Human Behavior, 19, 335–353. Kruger, A. (1993). Peer collaboration: Conflict, cooperation, or both? Social Development, 2, 165–182. Kuhn, D. (2015). Thinking together and alone. Educational Researcher, 44, 47–53. Kuhn, D., & Udell, W. (2003). The development of argument skills. Child Development, 74, 1245–1260. Kuiper, E., Volman, M., & Terwel, J. (2005). The web as information resource in K-12 education: Strategies for supporting students in searching and processing information. Review of Educational Research, 75, 285–328. Laal, M., & Laal, M. L. (2012). Collaborative learning: What is it? Procedia – Social and Behavioral Sciences, 31, 491–495. Leu, D. J., Kulikowich, J., Sedransk, N., & Coiro, J. (2009). Assessing online reading comprehension: The ORCA project. Research grant funded by U.S. Department of Education, Institute of Education Sciences. Leu, D. J., Jr., Kinzer, C. K., Coiro, J., & Cammack, D. (2004). Toward a theory of new literacies emerging from the Internet and other ICT. In R. B. Ruddell & N. Unrau (Eds.), Theoretical models and processes of reading, (5th ed., pp. 1568–1611). Newark, DE: International Reading Association. Leu, D. J., Jr., Kinzer, C. K., Coiro, J., Castek, J., & Henry, L. A. (2013). New literacies: A dual-level theory of the changing nature of literacy, instruction, and assessment. In R. B. Ruddell & D. Alvermann (Eds.), Theoretical models and processes of reading. (6th ed., pp. 1150–1181). Newark, DE: International Reading Association. Liu, L., Hao, J., von Davier, A., Kyllonen, P., & Zapata-Rivera, D. (2015). A tough nut to crack: Measuring collaborative problem solving. In Y. Rosen, S. Ferrara, & M. Mosharraff (Eds.), Handbook of research on technology tools for real world skill development (pp. 344–359). Hershey, PA: ICI Global. Liu, L., & Hmelo-Silver, C. E. (2010). Conceptual representation embodied in hypermedia: An approach to promoting knowledge co-construction. In M. S. Khine & I. M. Saleh (Eds.), New science of learning: Cognition, computers and collaboration in education (pp. 341–356). New York, NY: Springer. Marttunen, M., & Laurinen, L. (2006). Collaborative learning through argument visualisation in secondary school. In S. N. Hogan (Ed.), Trends in learning research (pp.  119–138). New York, NY: Nova Science Publishers. Mislevy, R. J. (2016). How developments in psychology and technology challenge validity argumentation. Journal of Educational Measurement, 53, 265–292. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3–67.

Assessing Collaborative Inquiry Skills  •  501 Mullis, I. V. S., & Martin, M. O. (Eds.). (2015). PIRLS 2016 Assessment framework, 2nd Edition. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. Murray, T. (2013). Toward defining, justifying, measuring, and supporting social deliberative skills. Retrieved from http://socialdeliberativeskills.org/documents/AIED2013-MetacogWorkshopMurraySubm.pdf. Murray T., Xu X., Woolf B. P. (2013). An exploration of text analysis methods to identify social deliberative skill. In H. C. Lane, K. Yacef, J. Mostow, & P. Pavlik (Eds.), Artificial intelligence in education. AIED 2013. Lecture Notes in Computer Science, vol. 7926. Berlin: Springer. Retrieved from https://link.springer.com/ chapter/10.1007/978-3-642-39112-5_119. National Assessment Governing Board. (2012). Reading framework for the 2013 National Assessment of Educational Progress. U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. Washington, DC: NAGB. Nussbaum, E. M. (2008). Collaborative discourse, argumentation, and learning. Contemporary Educational Psychology, 33, 345–359. Organisation for Economic Co-operation and Development. (2017). PISA 2015 collaborative problem solving framework. Paris, France: OECD Publishing. Retrieved from www.oecd.org/pisa/pisaproducts/Draft%20 PISA%202015%20Collaborative%20Problem%20Solving%20Framework%20.pdf. Osborne, J., Erduran, S., & Simon, S. (2004). Enhancing the quality of argument in school science. Journal of Research in Science Teaching, 41, 994–1020. Pellegrino, J. W., Chudowsky, N., & Glaser, R. (2001). Knowing what students know. Washington, DC: National Academy Press. Roschelle, J., & Teasley, S. (1995). The construction of shared knowledge in collaborative problem solving. In C. E. O’Malley (Ed.), Computer supported collaborative learning. Heidelberg, Germany: Springer. Rouet, J.-F., & Britt, M. A. (2011). Relevance processes in multiple document comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Relevance instructions and goal-focusing in text learning (pp. 19–52). Greenwich, CT: Information Age Publishing. Salomon, G., & Perkins, D. N. (1998). Individual and social aspects of learning. Review of Research in Education, 23, 1–24. Schwartz, D. L. (1995). The emergence of abstract representations in dyad problem solving. The Journal of the Learning Sciences, 4, 321–354. Schwarz, B. B., & Asterhan, C. (2010). Argumentation and reasoning. In K. Littleton, C. P. Wood, & J. Kleine Staarman (Eds.), International handbook of psychology in education (pp. 137–176). Bingley, UK: Emerald Group. Schwarz, B. B., & De Groot, R. (2007). Argumentation in a changing world. The International Journal of Computer-Supported Collaborative Learning, 2, 297–313. Sparks, J. R. (2014). Investigating the use of virtual worlds to measure information processing and evaluation in English language arts. NAEP SAIL Research funded by U.S. Department of Education, National Center for Education Statistics. Sparks, J. R., & Deane, P. (2015). Cognitively based assessment of research and inquiry skills: Defining a key practice in the English language arts (Research Report 15-35). Princeton, NJ: Educational Testing Service. Sparks, J. R., Katz, I. R., & Beile, P. M. (2016). Assessing digital information literacy in higher education: A review of existing frameworks and assessments with recommendations for next-generation assessment (Research report 16-32). Princeton, NJ: Educational Testing Service. Stanford History Education Group [SHEG]. (2016). Evaluating information: The cornerstone of civic online reasoning – Executive Summary. Retrieved from https://sheg.stanford.edu/upload/V3LessonPlans/ Executive%20Summary%2011.21.16.pdf. van Eemeren, F. H., Grootendorst, R., & Henkemans, F. S. (1996). Fundamentals of argumentation theory: A handbook of historical backgrounds and contemporary developments. Mahwah, NJ: Erlbaum. von Davier, A. A. (2017). Collaborative educational assessments [Special issue]. Journal of Educational Measurement, 54(1). von Davier, A. A., & Halpin, P. F. (2013). Collaborative problem solving and the assessment of cognitive skills: Psychometric considerations (Research Report 13-41). Princeton, NJ: Educational Testing Service. Wertsch, J. (1991). Voices of the mind: A sociocultural approach to mediated action. Boston, MA: Harvard University Press. Zhang, M., & Quintana, C. (2012). Scaffolding strategies for supporting middle school students’ online inquiry processes. Computers & Education, 58, 181–196.

28

COMPUTER-BASED ASSESSMENT OF ESSAYS BASED ON MULTIPLE DOCUMENTS Evaluating the Use of Sources Joseph P. Magliano northern illinois university, usa

Peter Hastings depaul university, usa

Kristopher Kopp arizona state university, usa

Dylan Blaum northern illinois university, usa

Simon Hughes depaul university, usa

Many personal, professional, and academic literacy activities require one to read multiple documents (texts, graphs, videos), extract content, and represent it in an integrated mental model, which has been referred to as a documents model (OECD, 2008; Rouet & Britt, 2011; Strømsø, Bråten, & Britt, 2010). As such, curricular standards emphasize the need for students to acquire and master the skills that are necessary to successfully learn from, use, integrate, and write about information represented across multiple documents (Achieve, 2013; Council of Chief State School Officers, 2010). Accordingly, there has been an increased level of research interest in the psychological processes and skills that support the use of multiple documents, as evidenced by this volume. The skills necessary to use and write about multiple documents are complex

502

Computer-Based Assessment of Essays  •  503

and involve challenges not often reflected in learning and writing about a single document (Anmarkrud, Bråten, & Strømsø, 2014; Rouet & Britt, 2011). To elaborate, imagine an academic situation where a student is asked to write an explanatory essay about a scientific process, which is not fully described in any one document at the student’s disposal. Figure 28.1 represents such a situation. To write the essay, the student must identify information about the source, read each document, identify content relevant to the task within each document, extract that content, connect the relevant information extracted from each document to relevant information in the other documents, use that information to describe the process they are trying to explain, and identify the sources of their ideas in the essay. Doing these things can be particularly challenging when the documents are each written for a purpose that may not directly align with the task the student is using the documents to complete (Magliano, McCrudden, Rouet, & Sabbatini, in press). From the standpoint of a teacher or researcher, the essays produced by students are the external artifacts available to assess the extent that they were able to successfully engage in these processes. While tasks of this nature are challenging for students, evaluating and scoring essays and providing feedback are also complex and time consuming for teachers and researchers. Grading essays is always a daunting task and

Task Introduced

Task Model

Document 1

Document 2

Document

Title1

Title2

Title3

Source1

Source2

Source3

Documents Model Explanation of Process

Essay

Source1 Source2 Source3

Figure 28.1  A Graphic Depiction of the Nature of a Multiple Documents Task.

504  •  Magliano et al.

especially doing it such that feedback is provided in a timely fashion (Magliano & Graesser, 2012; Shermis & Burstein, 2013). There are a variety of dimensions that could be considered for assessment while scoring essays like spelling, grammar, cohesion, etc. (e.g., Magliano & Graesser, 2012), but in this chapter we are concerned about evaluating the content that reflects what was learned and used from a set of documents. In this context, one needs to evaluate how successful students (or participants) were at extracting information from the documents, synthesizing information, and indicating where their ideas came from. The challenge of assessing essays for such tasks may deter instructors from assigning them to students. However, over the past two decades there have been substantial advances in the application of natural language processing (NLP) techniques to support the analyses of student essays (Magliano & Graesser, 2012; Graesser & McNamara, 2011). Generally, NLP refers to a wide range of computational approaches that are used to analyze the content, structure, and intended meaning of texts. For example, computer programs have been developed to accurately identify the syntactic structure of sentences (Chen & Manning, 2014), the phrases in a paragraph which refer to the same objects (Clark & Manning, 2016), and the location of answers to questions (Morales, Premtoon, Avery, Felshin, & Katz, 2016). These advances can be brought to bear to create systems that are devoted to evaluating essays based on multiple documents (Hastings, Hughes, Magliano, Goldman, & Lawless, 2012; Hughes, Hastings, Britt, Wallace, & Blaum, 2015; Wiley et al., 2017). We adopt a perspective that assessments in general should be grounded in theory (Mislevy, 1993; Pellegrino & Chudowsky, 2003), and in this case theories associated with understanding and learning from multiple documents (Rouet & Britt, 2011). As such, while we have emphasized the application of these tools in an educational context in this introduction, they could also be a boon for research on learning from multiple documents (Hastings et al., 2012; Higgs, 2016; Hughes et al., 2015; Wiley et al., 2017). In this chapter, we first discuss features of essays based on multiple documents that are important to assess as delineated by theories of text comprehension and task-oriented reading (Rouet, 2006; Rouet & Britt, 2011). We focus on situations in which essays are based on a preselected set of documents rather than situations in which students find their own texts. To our knowledge, most existing systems were developed to address the former situation. Moreover, most research on learning and writing based on multiple documents reflects this situation (e.g., Anmarkrud et al., 2014; Blaum, Griffin, Wiley, & Britt, 2017; Wiley & Voss, 1999). We then discuss promising approaches to computer-based assessment of essays, and what is needed to develop and test systems specifically designed for essays based on multiple documents. We identify several challenges for developing these systems that are grounded in theory, research, and practical problems associated with automatic assessment of the use of multiple documents in essays. We present research on existing scoring systems that are illustrative of these approaches, but it is important to note that this area of research is in its early stages and there are only a few studies that involve the automatic scoring of essays. We conclude with a discussion of important directions for further development and testing of automatic grading systems for essays based on multiple documents.

Computer-Based Assessment of Essays  •  505

THEORETICAL PERSPECTIVES ON WHAT SHOULD BE ASSESSED Evidence-centered approaches to academic assessment specify that these assessments should be grounded in relevant theories from cognitive science (Mislevy, 1993; Pellegrino & Chudowsky, 2003; Pellegrino, Chudowsky, & Glaser, 2001). Specifically, theories of cognition associated with an academic activity should be used to identify constructs that are assessed. To this end, in this section we describe relevant theoretical perspectives of task-oriented reading (Rouet, 2006) and learning from multiple documents (Britt & Rouet, 2012). Based on these theories, we identify factors that should be assessed for essays based on multiple documents and some of the challenges for doing so when one is evaluating them with or without the aid of computational systems. Theoretical Constructs Relevant to Evaluating Essays In any reading situation, a person is reading in order to complete a task (Graesser, Singer, & Trabasso, 1994; Snow & The Rand Reading Study Group, 2002). Even people who are reading for pleasure are reading with the basic goal of understanding and hopefully enjoying the story or information they are reading. Purposeful reading has been described as task-oriented reading (McCrudden & Schraw, 2007; Rouet, 2006; Rouet & Britt, 2011; Vidal-Abarca, Mañá, & Gil, 2010). Task-oriented reading elicits goal-directed behaviors and strategies, which will vary based on the task the reader is trying to complete (McCrudden & Schraw, 2007). In Figure 28.1, the task orients the reader to the content from the documents for which they should allocate their attentional resources. Consider a situation in which the hypothetical task depicted in Figure 28.1 involves having to read a set of documents in order to generate a causal explanation for a physical process (e.g., Why are tsunamis destructive? How does coral bleaching occur? How can releasing carbon into the atmosphere lead to a rise in global temperature?). Rouet and Britt (2011) provided a framework for how task-oriented reading may happen in a multiple document reading situation, specifically the Multiple Documents – Task-based Relevance and Content Extraction (MD-TRACE) model. The MD-TRACE model describes how readers interpret a given task and create goals and strategies for completing that task. The goals and behaviors that readers will complete are part of their task model. As people read documents, they identify information that is relevant for the task they are trying to complete (McCrudden & Schraw, 2007). This information may be different than the information that is important to the underlying message of the document (McCrudden, Magliano, & Schraw, 2011; McCrudden & Schraw, 2007). As such, the overall message of a single document may not be relevant to the reader’s task. Rather, some parts of that message may be used by the reader to complete the given task. Figure 28.1 represents this situation. In each document, only the information demarcated with a symbol is actually relevant for the student to complete the task of writing an explanatory essay based on the hypothetical prompt. Ideally, in this document set, students would be able to identify and extract the content germane to the task and discriminate it from that content which is not (Rouet & Britt, 2011).

506  •  Magliano et al.

While students are reading each individual document, they must be able to identify this information and begin constructing a documents model with information from multiple documents connected together (Rouet & Britt, 2011). After constructing the documents model, a reader will update the documents model regularly as they read through each document and encounter relevant information to complete their task. The situation reflected in Figure 28.1 involves constructing an explanation, and so the documents model would reflect a student’s understanding of the explanatory processes. Finally, a reader will create a task product, the “Essay” shown in Figure 28.1, and then check whether that product satisfies their task goal. As students create their task product, they create and continually update a product model, which is a mental representation of what they have written. Initially, the product model is likely to have a high level of overlap with the task model, but as individuals write the document, they likely construct a mental representation of that document that is distinct from the task model. That is, the product model contains a representation of the content of the essay that likely is akin to a mental model for a text (Rouet & Britt, 2011). To assess whether their product has met their task goal, students will compare their product model with the goal created in their task model. This assessment is essential because as the writing process starts, the information that students decide to put into their writing product may deviate from what is necessary to meet their task goal. Optimally, the task, documents, and product models should be conceptually linked, which is depicted in Figure 28.1. In Figure 28.1, no product model is shown because it is assumed that the essay is completed. As such, the essay is an external representation of the product model. If the task were to write an explanatory essay, one would need to build a mental representation of the information from the documents with regard to the goals related to the task. The task model would affect how information is selected, processed, and presented in the documents model (McCrudden, Magliano, & Schraw, 2011; McCrudden & Schraw, 2007; Rouet & Britt, 2011). Moreover, the task model should affect how content from the documents is integrated into a mental model within the document space. For example, if the task is to identify a causal explanation for a physical event (e.g., Why are tsunamis destructive?), the documents model would optimally contain a sequence of events that are connected via causal relationships (Blaum et al., 2017). However, if the task were to write an argumentative essay about some issue (e.g., Write an argument about various preventative steps that should be taken to lessen damage from tsunamis), then the documents model will be structured around claims, reasons, and potentially counter-arguments. The essay that is produced by students should be indicative of the task, documents, and product models (Rouet, 2006).

DIMENSIONS OF ESSAYS THAT SHOULD BE EVALUATED This chapter emphasizes evaluation that pertains to the use of the documents in a set provided to the student. First, it is important to assess the extent to which content was used from the different documents. However, students could summarize the relevant content, but do so in a manner that does not reflect the goal of the essay (e.g., describe a causal process, compare and contrast positions about an issue, write an argumentative essay that supports a particular position). As such, a second issue to consider is whether the content is conveyed in a manner that reflects the task at hand. An ideal essay should reflect the relevant content in a manner that is consistent

Computer-Based Assessment of Essays  •  507

with the prompt (e.g., the explanation of a process), rather than being constrained by how the content was conveyed in the documents. A third issue to consider is sourcing (Britt & Aglinskas, 2002; Rouet & Britt, 2011; Wineburg, 1991). Sourcing refers to activities that involve evaluating the reliability of the sources, and documenting how the sources were used in the essay (Wineburg, 1991). Monitoring source reliability is important because, although the information may be relevant to the task, that information may not be true. Monitoring source reliability involves assessing the author’s expertise on the subject, the outlet where it is published, the intent of the author, and possibly the date of the publication (Bråten, Strømsø, & Britt, 2009; Britt & Aglinskas, 2002; Wiley et al., 2009; Wineburg, 1991). This is important when, for example, there may be conflicting information from two sources. In such cases, it would be better to use information from a trustworthy source. Unfortunately, most people are not overly sensitive to the author (Britt & Aglinskas, 2002; Claassen, 2012) or vigilant about keeping track of the authors when processing a set of documents without training (Bråten et  al., 2009; Britt & Rouet, 2012). Nonetheless, teachers and researchers should be sensitive to whether students are drawing upon the sources in a document set appropriately, sufficiently transforming that content to meet the task and avoid plagiarism, and following protocols for indicating where the ideas came from.

WHAT IS NEEDED TO EVALUATE ESSAYS BASED ON MULTIPLE TEXTS We are considering a situation in which there is a defined document set given to students or participants that they are to use to write an essay (e.g., Blaum et al., 2017; Wiley & Voss, 1999). This is different than a situation in which students self-select texts that are unknown to an instructor. We are restricting this discussion to a closeddocuments-set context because the automatic assessment protocols that have been developed to date reflect that situation. Based on the discussion above, we identify what an instructor or researcher needs to implement a writing task based on multiple documents. We describe these here because they are also germane to developing systems that automatically evaluate essays. 1. An essay prompt. Prompts should ideally be specific enough to support the development of a specific task model that allows the reader to process the texts in a strategic fashion. Moreover, these prompts should clearly state the instructor’s desired structure for the essay, whether it is a causal prompt, argumentative prompt, compare and contrast, etc. 2. A documents set. Texts need to be selected that contain content that can be used to write the essay. A decision needs to be made as to the extent that the purposes of the documents are aligned with the essay prompt. The more divergent the intentions of the authors of the documents are from the essay prompt (i.e., the texts were written to convey points different from how their contents should be used in the essay), the more challenging it is for the student to identify relevant content from the documents. The more divergent the documents are from each other, the more difficult it is for the student to integrate information between documents, but the easier it is to identify where the information came from.

508  •  Magliano et al.

3. A documents model. A representation that contains an idealized specification of 1) the content from the documents that is relevant to the prompt, 2) how that content should be linked together to address the prompt (e.g., bridging inferences that explicitly link ideas in a manner consistent with the prompt), 3) how that information is transformed and synthesized across multiple documents (i.e., ways in which students may transform ideas in the texts to address the prompt), and 4) where that information came from. 4. A scoring rubric. A set of dimensions used to objectively evaluate the contents of a product model and how well it satisfies the task given by the essay prompt. 5. A protocol for delivering feedback. Feedback that is provided should be timely, appropriate, and targeted to address any deficiencies detected by the scoring rubric. The purpose of the feedback should be to help individuals modify and improve their task and product models, and therefore, ultimately, the quality of their essays. This list reflects not only what is needed when one develops and implements a writing task that involves multiple documents that will be evaluated by an instructor or researcher, but also situations that would involve automated systems. These five dimensions have important challenges to overcome. However, before we discuss these challenges, it is important to understand how prevalent approaches to the automatic assessment of student-constructed responses work.

PROMISING APPROACHES IN AUTOMATED ESSAY ASSESSMENT There are many different NLP techniques that have been developed for a wide range of purposes. In this section we describe the major categories of approaches and present some of the more prominent specific techniques within those categories, including the supporting resources that are required to use them. We then describe some example applications of their use in the evaluation of multiple source use in essays (Hastings et al., 2012; Higgs, 2016; Hughes et al., 2015; Wiley et al., 2017). General Overview of Scoring Systems There are several core features of any system that is designed to assess multiple document use, of which most correspond to the five features that are necessary for a multiple documents essay task that are described above. Some features are a requirement of all systems, whereas other features constitute options for the developers. In this section, we discuss some of these features. The first key features are semantic benchmarks, which are critical for assessing content that should be in an essay as delineated by the documents model. Many computational systems that evaluate constructed responses (e.g., essays, answers to open-ended questions, think-aloud protocols) do so by comparing those responses to semantic benchmarks (Magliano & Graesser, 2012). Semantic benchmarks can reflect a variety of things, such as idealized responses (e.g., Magliano et al., 2011), a set of responses that vary in quality (Foltz, Gilliam, & Kendall, 2000), or content specified by theory or an ideal model as being important for the response (e.g., Magliano & Millis, 2003;

Computer-Based Assessment of Essays  •  509

Magliano et  al., 2011; Hastings et  al., 2012). If the system is designed to detect the extent that students identify the sources of their ideas, then semantic benchmarks could be created that reflect how the students are instructed to do this (e.g., list authors’ names). The underlying assumption is that the more essays have semantic overlap with the semantic benchmarks, the more those benchmarks are reflected in the essays. As an example, consider the situation reflected in Figure 28.1. Three semantic benchmarks could be constructed for the relevant content from the three texts that should be included in the documents model (i.e., benchmarks corresponding to the bolded symbols in each document box). Essays that would have a high level of semantic overlap with all three semantic benchmarks would be considered more compliant with the prompt than those that have low overlap with one or more benchmarks. A second feature is the grain size of the text and benchmarks. To compare an essay to semantic benchmarks, one must determine the grain size of what is being compared. The grain size can vary from the whole document, to groups of sentences, to single sentences, to individual words or phrases. Comparing an essay to semantic benchmarks that reflect entire texts allows a general assessment of the extent that the contents of the texts are manifested in the essay, but does not allow one to assess the extent that specific content is included. On the other hand, smaller grain sizes may allow for some level of precision in determining what specific content from a text is actually manifested in the essay, but might not provide a good holistic evaluation. Importantly, parsing essays into smaller units of analysis is essential for determining if content across a document set is integrated into the essay. Consider Figure 28.1, which depicts a situation that requires students to integrate content from the three documents. As such, an essay that would contain the content, but in a compartmentalized fashion (e.g., text 1 content is described, then text 2, and finally text 3, rather than as is depicted in the explanatory process), would not be compliant with the prompt. However, an assessment system based on text-level grain size would not be able to distinguish essays that are compartmentalized from those that demonstrate integration. We contend that successful systems for scoring essays based on multiple, pre-designated documents require a grain size of comparison smaller than the essay or texts in the documents set. A third feature that is essential for system development is a corpus of human-scored essays based on a closed set of prompts and document sets, where the scoring is based on a rubric (as discussed above). That is, the corpus should reflect the exact prompt(s) and document set(s) that will be used once the system is developed. The essays must be scored on the dimensions that the system will be designed to evaluate. There is no specific number or type of features that should be scored, but ideally they should be delineated by theory (Mislevy, 1993; Pellegrino & Chudowsky, 2003; Pellegrino et al., 2001). According to the MD-TRACE model (Rouet & Britt, 2011), the scoring system could be constrained by 1) evidence of sensitivity to the essay prompt (i.e., the task model), 2) evidence that it reflects the ideal documents model (both content and how that content should be integrated), and 3) the extent that students have identified the sources of their ideas. That said, the systems that have been developed to date (and are described below) have primarily scored the essays on the extent that they reflected integrated content, rather than being sensitive to the essay prompt (beyond overlap with the documents model) and explicit sourcing.

510  •  Magliano et al.

Another potential aspect of human scoring is the annotation of the essays such that specific content is linked to idea units (clauses, sentences, groups of sentences, etc.) (Hastings et al., 2012; Hughes et al., 2015; Wiley et al., 2017). Consider Figure 28.1. While not always necessary for scoring systems (for example, where the goal is not to determine coverage of the sources by the essays), essays could be annotated such that content in the essays is specified as being semantically connected to sentences in the texts that are identified as being important to the documents model. Annotation is particularly valuable when machine learning algorithms are used to train a system to identify if essays contain content that is aligned with the documents model. As such, a fourth feature of scoring systems is training based on machine learning. Machine learning refers to computer algorithms that automatically learn from data (Mitchell, 1997; Russell & Norvig, 2010). Annotated essays and documents are required for training because they provide the data used to train the system. Specifically, the system learns to classify a variety of ways that content from the text can be presented by students. For example, consider Table 28.1, which shows two idea units from a document set on coral reefs and a sample of protocols that reflect how participants described those ideas unites in the context of writing an essay on the process of coral bleaching (Kopp et al., 2016). Protocols 1a and 1b reflect a close paraphrase, whereas protocols 5a and 5b reflect some degree of transformation of the original text content. The more an idea from a text is transformed by the students, the more challenging it is to computationally determine the source of that idea. As such, creating a semantic benchmark that reflects the variety of ways that a sample of students can express relevant content from the text can be useful, and is essential in any system that is trained to detect the variety of ways ideas can be expressed. This is an issue that is of concern for the computational analysis of student-constructed responses in general, Table 28.1  Example Text Idea Units, Participant Essay Idea Units, and LSA Cosines Between the Two. Original Idea From Document

Protocols

LSA Cosines

“. . . ocean water temperatures increase by 3 to 5 degrees Fahrenheit”

Protocol 1a: “When the water temperature increases. . . ”

0.83

Protocol 2a: “The higher the water temperatures. . . ” Protocol 3a: “The water temperature was around two to three degrees higher than normal. . . ” Protocol 4a: “. . . allows for the water to become very warm.” Protocol 5a: “. . . a large spike in heat. . . ” Protocol 1b: “. . . upsets the balance for a healthy coral reef.”

0.83 0.87

Protocol 2b: “. . . declines the coral‘s overall health.” Protocol 3b: “This ultimately deteriorates the corals overall health.” Protocol 4b: “. . . providing large risks to the health and lives of the corals.” Protocol 5b: “. . . brings the imbalance in the corals from keeping them thriving.”

0.85 0.85

“. . . upsets the balance necessary for coral health.”

0.77 0.48 0.77

0.81 0.39

Computer-Based Assessment of Essays  •  511

whether they be essays (e.g., Hastings et al., 2012), think-aloud/self-explanation protocols (Millis, Magliano, Todaro, & McNamara, 2007), or question–answer protocols. While some systems for analyzing student-constructed responses do not require the system to be trained to detect overlap between essays and semantic benchmarks (e.g., Magliano et al., 2011), some systems have relied on machine learning to do so (e.g., Hastings et al., 2012; Millis et al., 2007). Below we will describe two studies that have explored the extent to which training benefits essay scoring in the context of multiple documents (Hastings et al., 2012; Hughes et al., 2015). The final feature of scoring systems that we highlight is scoring categories. Developers have to determine exactly what aspects will be scored, the data for evaluating those aspects, an operationalization of the dimensions (i.e., the scores), and if feedback is provided to users (teachers, students), protocols for delivering those scores. There are no hard and fast rules on developing scoring criteria, as they are ideally constrained by theory, research questions, and practical considerations (e.g., curricular decisions, constraints on what can be assessed). For example, the Reading Strategy Assessment Tool (RSAT) developed by Magliano and Millis (Magliano et al., 2011) was designed to score think-aloud protocols. They identified two types of inferences (i.e., bridging and elaborative inferences) as a scoring dimension because they were delineated by theory to be important for comprehension. Scoring dimensions for analyzing essays based on multiple documents should be dovetailed with the dimensions that were identified as being important for scoring the essays by human coders in the scoring rubric. While we mention the importance of developing feedback for the user, the nature of that feedback depends on the user. To date, the systems that have been developed to score essays based on multiple documents have been developed for research purposes and therefore protocols for delivering feedback to teachers and students have not been developed.

COMPUTATIONAL TOOLS FOR ANALYZING ESSAY CONTENT In this section, we describe techniques for evaluating the content of the essays, parsing essays, and training systems to evaluate essays for specific content. Approaches for Analyzing Semantic Content There are two general approaches for comparing student responses to semantic benchmarks, which is typically in the service of identifying the content of an essay, but could also be used to identify the extent that students explicitly identify the sources of their ideas. The first is keyword matching and regular expressions (Magliano & Graesser, 2012; Hastings et  al., 2012). The simplest indication that something in an essay was derived from a particular source is that it uses the same unique words to describe whatever that is. By “unique” we are referring to words that occur in one of the given sources, but not in the others. Simple scanning techniques can search essays for important terms (keywords) or consecutive words (known as n-grams) that comprise the semantic benchmarks created for scoring the essays. If the benchmarks reflect the different sources, this search can be used to identify the relevant source.

512  •  Magliano et al.

Ideally, however, students will transform the content of the essays because they are instructed to convey it in their own words. As such, there can be a “family” of ways in which content could be expressed by students (see Table 28.1). An alternative to keyword matching is identifying a set of expressions or patterns that might reflect the different ways in which semantic benchmarks can be conveyed. These are normally referred to as regular expressions (Aho, 1990). Regular expressions provide a way of specifying keyword strings that include variants. For example, the expression, “increas(ing|ed) fresh water” can match two different key phrases, “increasing fresh water” or “increased fresh water.” By combining regular expressions, one can specify key phrases in a way that is rich, powerful, and concise. All modern programming and scripting languages include built-in mechanisms or libraries for searching for regular expressions. (Keywords can be treated as simple regular expressions.) A second approach for evaluating essays against semantic benchmarks involves the use of a more general (i.e., not customized to a particular task) high-dimensional vector or semantic space (Magliano & Graesser, 2012), such as Latent Semantic Analysis (LSA; Landauer & Dumais, 1997), Hyperspace Analogue to Language (HAL: Lund, Burgess, & Atchley, 1995), holographic models (Jones, Kintsch, & Mewhort, 2006), and word2vec (Mikolov, Sutskever, Chen, Corrado, & Dean, 2013). Because these techniques create vector representations of words and/or documents, they are commonly referred to as Vector Space Models (VSM) of Semantics (Turney & Pantel, 2010). A number of these VSM models ignore word order, in which case they are referred to as bag-of-words models. All of these models are based on the Distributional Hypothesis of word meanings, which holds that words which occur in similar contexts tend to have similar meanings (Firth, 1957). Approaches like LSA and HAL start with the creation of a co-occurrence matrix that reflects the extent that words co-occur across a large set of (possibly domain-specific) texts. (The website http://lsa.colorado.edu contains a number of previously developed spaces reflecting different topics and ranges of texts.) The matrix generally contains thousands of words and the frequency at which they co-occur across thousands (or more) of texts. With LSA, a dimensionality reduction technique, Singular Value Decomposition, is used to reduce the number of dimensions from thousands to typically 100–500. Word “meanings” are represented as vectors within the semantic space. Similarity of words can be simply computed by calculating the proximity of the vectors of the words, typically using the geometric cosine, which, in practice, varies from 1.0 (semantically identical) to near 0 (completely unrelated). For example, using the Colorado LSA space representing general reading up to the first year of college, “tsunami” and “wave” have a cosine of .76, whereas “tsunami” and “mountain” have a cosine of −.01, indicating that the first pair are semantically very close in the semantic space and the second pair are essentially unrelated. Representations for groups of words (sentences, paragraphs, texts) are computed by a simple combination of the vectors for the words in the groups. A semantic space approach can thus be used in a multi-document setting by using it to compare the sentence (or words or paragraphs) of a new text to the original source documents and/ or semantic benchmarks to identify those with a sufficiently close “fit” (typically using an empirically determined cosine threshold). One advantage of the semantic space approach over keyword matching is that it is sensitive to semantic distance (as reflected in the example from the Colorado LSA

Computer-Based Assessment of Essays  •  513

space above) and therefore does not require one to develop dictionaries of synonyms or a family of regular expressions. However, the uses of keyword matching, regular expressions, and high-dimensional spaces to analyze constructed responses are not mutually exclusive, and there are examples of hybrid systems (e.g., Graesser et  al., 2004). In fact, it has been argued that systems should rely on both approaches to compare constructed responses to semantic benchmarks whenever possible (Magliano & Graesser, 2012). Approaches for Parsing Sentences in Essays Semantic space methods can be used at different levels of granularity. They can be used to compare entire essays, or paragraphs, or sentences, or words. But they do not provide information about the relationships between, for example, the words in a sentence. There are situations in which parsing essays is advantageous in the computerbased assessment of essays. Parsing involves determining the structure of the sequence of words in sentences and the phrases within them. For sentences, one may apply a syntactic grammar (often along with semantic constraints) to determine the phrasal structure. The structure is normally viewed as a tree, normally with the main verb as its root, and the phrasal attachments as the branches. From the syntactic structure, the specific semantic relationships between the components of a sentence can be derived. This type of analysis is often necessary to gain a clearer understanding of the meaning of a text. For example, in the sentence, “The woman kissed the man,” a bag-of-words semantic space approach would not be able to determine who is doing the kissing because the order of the words is not evaluated. Both semantic space methods (when applied at the sentence level) and parsingbased methods benefit from the segmentation of a text into sentences. This is normally easy to accomplish, especially with the use of punctuation (assuming it is reliable and assuming exceptions like abbreviations are taken into account). If the semantic benchmarks for a task are relatively specific (as is depicted in Figure 28.1), the system may be more accurate in detecting them if the essay text is parsed, because then it can compare concepts from the benchmarks to representations of specific phrases. Parsing is also necessary when the different documents in a set describe the same entities, but with different relationships between them. For example, in the document set used in Hastings et al. (2012), one document described how the advent of the trains allowed Chicago to become a transportation hub, whereas another described how trains made it easy for people to move to Chicago. Discriminating these different roles of trains in Chicago population growth might be improved by automatically analyzing the clauses in the student essays that describe trains. The presence of nouns that are strongly associated with the concept “trains” (e.g., train, locomotive, tracks, etc.) provides semantic cues that these sentences are associated with that concept. However, the verbs associated with the roles of trains should be indicative of the events associated with them from the documents that the students are describing (e.g., Zwaan, Langston, & Graesser, 1995). These verbs can provide cues to causal and situational cohesion (McNamara, Graesser, McCarthy, & Cai, 2014), and therefore could be useful in detecting the extent that students are linking ideas in the essay in a manner consistent with the documents model.

514  •  Magliano et al.

As noted above, many semantic space systems do not consider word order, and one reason that semantic space methods are so popular is that parsing can be a notoriously difficult task. Besides ungrammaticalities or partial phrases introduced by the writer, the biggest problems are due to the inherent ambiguities of human languages. Many, if not most, words have multiple senses; phrases and discourse elements can be combined in different ways; and ambiguities at lower levels multiply the ambiguity at higher levels. For example, if a word has two possible interpretations, and that word is in a prepositional phrase that can attach to another element in the sentence in three different ways, that leads to six different interpretations of the sentence. This ambiguity can make parsing computationally very intensive and make it very difficult to determine the intended meaning. Recent research into this problem has reduced it substantially, by using probabilistic grammars that are learned from real-world texts (e.g., Chen & Manning, 2014). These grammars take into account the likelihood of the various combinations, and only pursue the most likely. The existence of very large annotated corpora has also allowed these grammars to include semi-semantic information like dependencies, which go beyond the basic syntactic relationships between words. Dependencies indicate which word in each phrase is the root and the types of relationships between the roots and the other words. For example, “advmod” indicates that the word is an adverbial modifier to a root verb, an “amod” is an adjectival modifier, and an “agent” dependency indicates the performer of a verb’s action (de Marneffe et al., 2013). The best known of these systems is the Stanford Natural Language Processing Group’s CoreNLP system (available from http://nlp.stanford.edu/software). Custom Machine Learning Methods Some types of machine learning require no special annotation (i.e., where the researcher designates the meaning of semantic units in a training corpus). This is called unsupervised learning. In this section, however, we focus on supervised learning approaches to assessing multiple documents use, where the training data has been coded by human coders to indicate whether it does or does not fall into particular categories. For example, assume a task for which researchers have developed a causal model, which includes all of the main concepts and the desired causal relationships between them. (The next section provides a more detailed description of such a situation.) The training data would consist of a large number of example (student) texts (at least 200, preferably many more) in which human coders have identified specifically where the different concepts (from different sources) and connections between them are mentioned. For this task, it is useful to use an annotation tool like brat (available from http://brat.nlplab.org/index.html). Then a machine learning algorithm such as Support Vector Machines, Logistic Regression, or Neural Networks can be used to infer from the examples how to identify the concepts and relationships in a new text (Mitchell, 1997; Russell & Norvig, 2010).

EXAMPLE STUDIES USING NLP TO STUDY MULTIPLE DOCUMENTS PROCESSING In this section we discuss four studies that have used NLP systems to study multiple documents processing.

Computer-Based Assessment of Essays  •  515

Using NLP Tools to Identify Source Content Hastings et  al. (2012) conducted a study to test different methods of automatically assessing the content of essays written from multiple documents, specifically: LSA, identification of keyword phrases (n-grams) with machine learning, and the machine learning technique called Support Vector Machines (SVMs), which learns the best separation of texts into classes (here, whether they contain specific concepts from the ideal products model) based on the words that occur in them. In their study, 460 essays were collected from students in grades 5 through 8. These students were asked to use three text-based documents to answer a question about why people moved to Chicago between 1830 and 1930. A documents model was created to identify sentences from the document set that were relevant and how they should be integrated in the essay to answer this question. Human coders segmented the student essays into sentences and then identified the extent to which each sentence reflected the ideas specified in the documents model. In the LSA approach, the semantic benchmarks were simply the human annotations that reflected which sentences in the document set corresponded to the parts of the idealized product model. LSA cosines were computed between each sentence in the essays and the semantic benchmarks and a threshold was determined to indicate if the sentences were similar enough (the cosine was high enough) for a model concept to be considered present in the essay. The machine learning approaches used 10-fold cross-validation, learning from a randomly chosen 90% of the essays, and testing on the other 10%, averaging performance over 10 iterations. Performance of the systems was calculated by comparing their identification of concepts from the products model to those of the human annotators. Overall, the LSA approach most closely matched the human judgments over the entire set of concepts when the frequency of the concepts was taken into account. However, some of the concepts relied on connections between sentences in a single document or across documents. In these cases (especially the latter), the LSA approach performed poorly because it was based on sentence-to-sentence comparisons. The overall performance of SVM was similar to that of the LSA approach, but it suffered when concepts occurred rarely in the corpus of essays. This is a typical problem for machine learning approaches; the frequency of the answer in the training set affects the frequency that the answer is given during testing. For detecting content that made connections between documents, the n-gram learning method performed best. It was trained on each concept individually, so it was unaffected by the frequency of occurrence of the concept, and, unlike the LSA approach, it did not rely on matching specific sentences in the source documents. Hughes et al. (2015) developed an automatic coding system to assess the overall quality of causal essays written from multiple documents based on the content and structure of the essay. Students were given a set of five documents about the topic of coral bleaching or skin cancer (each set had four text documents and one graph) and asked to write about what causes the scientific phenomenon they read about. Human coders scored and tagged the essays for key content and the causal links made connecting the content. Based on the amount of content and number of connections students made, their essays were sorted into four different quality categories: no core content from the documents; some core content, but no connections between any content; some core content, but only one connection linking content; or both

516  •  Magliano et al.

content and at least two consecutive connections structurally linking the content. This task is especially difficult because any misclassification on the component tasks (identifying concepts and causal connections) affects performance on the holistic assessment. A machine learning system was trained on a subset of essays which had been tagged by human coders. Once trained, the system could place an essay in its appropriate quality category with moderate success, with a Krippendorff’s alpha correspondence with human coders of 0.56 for essays about coral bleaching and 0.47 for essays about skin cancer. If the neighboring quality category was included (indicating that the system was not far off in its assessment), then the accuracy was 85% and 88% respectively for the two topics. Using NLP Tools to Assess Explanation Quality and Student Understanding Wiley et  al. (2017) built on the machine learning-based research described above and compared it to other methods of evaluating multiple source use with the goal of assessing how well each method accounts for both student understanding of the information from multiple documents and the quality of the explanatory essays that they created. The participants in this study were 178 middle and high school students. The students were given seven short documents on topics related to global temperature change. One document gave general background information, five described related main topics like ice ages and the carbon cycle, and one was a graph of carbon dioxide concentrations over the last 400,000 years. The students were asked to read the documents and then write an essay (with the documents present) to explain “how and why recent patterns in global temperature are different from what has been observed in the past.” None of the source documents was sufficient by itself to create a complete answer to the prompt. The essays were annotated by human scorers as described above, and quality categories were derived from the coding of concepts and connections. After writing the essay, the students were given an 18-question multiple-choice inference verification test to assess their understanding as indicated by the connections and inferences they made within and between the documents. Along with the machine learning method described above (Hughes et al., 2015), this study analyzed metrics derived from two “off-the-shelf” tools. LSA was used in two ways. Following Ventura et al. (2004), an ideal essay similarity score was computed by comparing the student’s entire essay to an idealized essay that was constructed from two highly rated peer essays. Following Britt, Wiemer-Hastings, Larson, and Perfetti (2004), plagiarism scores were calculated by comparing each student essay sentence to each sentence of the source documents. If the maximum cosine was above 0.75, the sentence was deemed to be copied from the source. The plagiarism score for an essay was the percentage of its sentences that were marked as copied. Coh-Metrix was also used, which is an online tool (available at http://cohmetrix. com) providing 108 indices of a variety of aspects of readability, cohesion, and complexity of texts (McNamara et al., 2014). From these, three metrics were computed: causality, cohesion, and lexical diversity. The causality score operationally was based on the number of causal verbs and particles. The cohesion score was based on LSA cosines between paragraphs in the essays. If there was only one paragraph in an essay, cosines between adjacent sentences were computed instead. The lexical diversity score

Computer-Based Assessment of Essays  •  517

was the type-to-token ratio of the essay, which is the number of unique content words which appeared in the essay divided by the number of occurrences of those words. These metrics, along with some basic descriptive features of the essays, were entered into simultaneous regression equations to see how well they predicted the overall essay quality scores and student understanding of the documents as indicated by the inference verification test. For essay quality, the unique significant predictors of variance, predicting a combined 49% of the variance, were the number of concepts identified by the machine learning model, the LSA plagiarism score, the LSA comparison to the idealized essay, and the Coh-Metrix cohesion score. The unique significant predictors of variance in student understanding, predicting a combined 23% of the variance, were the number of concepts identified by the machine learning model, the LSA ideal essay similarity score, and the Coh-Metrix causality, cohesion, and lexical diversity scores. This study has a number of interesting conclusions and implications for future research. One is that automatic methods of assessing explanatory essay quality are feasible, but that is especially so with hybrid models that combine a number of different types of factors (Magliano & Graesser, 2012). Surprisingly, the basic text features of essay length, responsiveness to the prompt, and presence of citations were not found to be related to essay quality. This has implications for studying writing processes; the lack of predictive power of citations could indicate that students who referred more to the source documents were focusing more on knowledge-telling rather than knowledge-transforming (Wiley & Voss, 1999). It also suggests that more research is needed on the machine learning approach to identifying connections between concepts. Current approaches are limited in the extent to which they take into account discourse features like anaphora which can play a large role in explanations. Using NLP Tools to Study How Texts are Processed in a Multiple Documents Task While the emphasis of this chapter is on the development of systems for analyzing essays based on multiple documents, NLP tools can also be used to study how texts are processed in a multiple documents task. For example, Higgs (2016) was interested in assessing if integration across documents happens during reading or after reading (i.e., while engaged in the writing task). She manipulated the specificity of a reading goal that was either general or emphasized a specific topic in the texts, namely how tsunamis occur and why they are sometimes destructive. Participants first read the texts silently under the three instructions (read to understand, read to learn about tsunamis, and read to explain why tsunamis can be destructive). None of the texts were specifically about tsunamis and why they can be destructive, but an explanation for that could be derived from content across the three texts. They then re-read the texts and thought aloud at target locations. A documents model was created by the experimenters that reflected how content across the texts could be used to answer a causal prompt about the text (i.e., Why are tsunamis so destructive?). Sentences were selected for the think-aloud prompts that afforded integration of content in the documents model. These sentences were chosen because readers should make connections to other sentences in the same text and across texts in the document set. Inspired by studies that have used NLP tools to analyze think-aloud protocols (Magliano & Millis, 2003; Magliano et  al., 2011), Higgs used LSA to analyze the

518  •  Magliano et al.

think-aloud protocols to assess if they reflected intra-text (i.e., within text) or intertext (i.e., across texts) integration. Specifically, she compared the verbal protocols to two semantic benchmarks: one that reflected the most important ideas in the texts with respect to their topic (which was not specific to the causal prompt) and another benchmark that reflected content from the three texts that was in the documents model. Higgs found that cosines were higher for the important ideas benchmark than the documents model benchmark under the general comprehension instruction, but were the same under the two specific reading goal instructions. She concluded that specific task instructions led readers to process content semantically aligned with the prompt more closely than the general instruction (e.g., McCrudden & Schraw, 2007). A second analysis involved developing two benchmarks associated with content in the documents model. Specifically, an intra-text benchmark was constructed that reflected the content in the documents model that was in the text that was being read when the verbal protocols were produced. An inter-text benchmark reflected content in the documents model from other texts (i.e., not the text that was currently being read when the protocols were produced). Higgs found that under the goal to explain why tsunamis are destructive, the LSA cosines were higher for the intra-text benchmark than the inter-text benchmark, but there were no differences under the other two instructions (and the cosines were relatively small). She concluded that under read-to-explain instructions, participants focused on making connections to content aligned with those instructions in the text that was currently being read, but did not make connections to other documents in the set. Importantly, specific task instructions (both to learn about tsunamis and read to explain) did lead to evidence of greater inter-text integration in the recall protocols (as evidenced by human coding of the protocols) than the general instruction. Higgs (2016) concluded that integration across documents likely occurred during the writing task rather than during reading.

CHALLENGES IN DEVELOPING AUTOMATED ESSAY EVALUATION SYSTEMS In the first section of this chapter, we discussed challenges instructors and researchers face when grading and evaluating essays based on multiple documents. The goal of this section is to discuss how these challenges need to be addressed to develop a system that can be used to automatically analyze essays based on multiple documents. For some of these problems there are good solutions, and other problems have not yet been solved. Level of Semantic Overlap Between Documents in a Document Set The level of semantic overlap between documents in a document set can vary. For example, the documents used in Higgs (2016) were all written on different topics (i.e., the role plate tectonic shifts play in earthquakes, how tsunamis form, and the Fukushima Nuclear power plant disaster), but each provided part of the causal process specified by the question prompt (“Explain how tsunamis can be destructive”). On the other hand, the documents used in Hastings et al. (2012) each explained one aspect of how Chicago grew in population during the latter half of the 19th century (i.e., the availability of jobs in Chicago, economic problems in the south, Chicago becomes a

Computer-Based Assessment of Essays  •  519

transportation and shipping hub). As such, there may be stronger cues to facilitate integration within the documents used in Hastings et al. (2012) than Higgs (2016). There is a tradeoff one needs to consider in terms of having documents with sufficient semantic overlap to support integration when students are writing their essays but with sufficient semantic dissimilarity to be able to computationally detect which documents are being used in the essays (Hastings et al., 2012). It is easier to integrate content across documents in an essay when those documents describe similar events than when they describe different events (Kurby, Britt, & Magliano, 2005). On the other hand, the more semantic overlap that there is between documents, the harder it is to develop a system that can accurately detect the extent that content from the different documents that is specified in the documents model is present in the essays. This can be particularly challenging when the documents include graphs to be interpreted by students. In Table 28.1, protocol 5a represents the idea present in the given idea unit from the text, but the document set also included a line graph which included temperature changes over time. While it is possible, and even likely, that the student used the word “spike” to indicate the visual cue on the graph, it is difficult to tell for certain whether the student was transforming the idea from the text or using information present in the graph on a separate document. Research is needed to specify the right balance between overlap that affords integration and maximizes source detection. However, a first question should be, “How well can human judges distinguish the source of the concepts that the students write about?” With enough examples, modern computational methods are very good at distinguishing sources. But they cannot perform well if their training data (annotated essays) are unreliable. Student Transformation of Content from the Documents As discussed above, ideally students should transform the content from the documents such that it is described in their own words. However, given the propensity of students to closely paraphrase source documents and even “write” by cutting and pasting, a number of essay grading systems have developed protocols to detect plagiarism (Foltz et al., 2000). However, in this chapter we want to emphasize the challenge of students transforming content into their own words. For example, consider Table 28.1. It would obviously be much easier to determine the semantic overlap of the source text with protocol 1a rather than with protocol 5a. Both protocols are paraphrases of the source texts in that they convey the same idea (temperature increases). Protocol 1a contains many of the words in the source text, but protocol 5a does not. As such, cosines based on the Colorado LSA space are very different. This is exactly the situation in which the development of regular expressions and machine learning can be used to train a system to recognize the different ways students can produce content from a document set. For instance, from an annotated corpus, a machine learning model could detect features that are commonly associated with expressing a certain concept in text. In addition, unsupervised machine learning approaches to modeling semantics, such as LSA and word2vec, can make use of semantic information acquired from large external corpora to map the annotated essay text into a semantic space, and then supervised machine learning models can be trained on this representation. This allows the system to recognize other ways of expressing these ideas that were not observed in the annotated corpus.

520  •  Magliano et al.

Dedicated Versus General Semantic Spaces If one is using high-dimensional spaces, such as LSA, an important consideration is whether one needs to develop a dedicated semantic space that covers the topics in a document set. Many systems that have been developed to code student-constructed responses have relied on general spaces, such as the Colorado TASA (Touchstone Applied Science Associates) LSA spaces (e.g., Foltz et al., 2000; Hastings et al., 2012; Kintsch, Caccamise, Franzke, Johnson, & Dooley, 2007; Magliano & Millis, 2003), which were built from corpora including representative texts that might have been seen up to 3rd, 6th, 9th, and 12th grades, or college. However, if the documents in a set describe relatively novel or specialized topics, then those specialized word senses might not be represented in the general semantic space. For example, Higgs used texts describing geological processes associated with tsunamis (Higgs, 2016). While she used LSA to analyze the think-aloud protocols, many of the words in the documents were not represented in the semantic space (e.g., subduction). As such, LSA was not sensitive to their presence. There are two options present to the developers. The first is to develop a dedicated high-dimensional space (Kurby et  al., 2003). This is a time-consuming process that not only involves developing the space (i.e., collecting a large amount of domain-specific text and creating the space from it), but testing its validity. A second option is to rely on a hybrid system that uses both semantic spaces and keyword matching/regular expressions. Given the time-consuming nature of developing and testing a new semantic space, we advocate using hybrid systems (Magliano & Graesser, 2012). Detecting the Relationships Between Content Documents models specify both content that should be in the essays and the relationships between them. While we have developed protocols for assessing the presence of specific content, as discussed above, there are significant challenges to assessing the relationships between content, as specified in the documents model. Often students may explicitly state the idea units in their essays, but they do not always explicitly state the relationship between the ideas. Without explicit markers, it can be quite challenging for an automated system to detect these relationships. Additionally, students will connect ideas across sentences. For example, a student might write, “Sometimes weather changes and trade winds decrease. This causes ocean temperatures to increase.” It is challenging to determine exactly the antecedent for the referent “this.” A human may be able to intuit that it refers to the decrease of trade winds by applying her general world knowledge, but a computer doesn’t have that luxury. Developing Systems That Can Generalize To date, most systems that have been developed to analyze essays based on multiple documents have been specialized systems that are specific to a document set and essay prompt. While it is possible to develop a system that can handle different essay prompts for the same document set, there are significant challenges to developing a system that is generalized enough to handle any document set.

Computer-Based Assessment of Essays  •  521

Challenges to Providing Feedback None of the systems that have been developed to analyze essays based on multiple documents are designed to provide feedback to users (students or teachers). As such, this has not been a primary focus of this chapter. Nonetheless, we point this out as a challenge to overcome. Of course, there are systems that provide feedback to users about their writing in other contexts (e.g., Dai, Raine, Roscoe, Cai, & McNamara, 2011; Kintsch et al., 2007). The nature of that feedback is determined by the pedagogical goals. Given that those goals can vary, we do not belabor the point here. Rather, our goal is to acknowledge that this is a dimension that developers need to consider.

CONCLUSIONS AND FUTURE DIRECTIONS This chapter is best seen as a primer for developing systems that can support the automatic assessment of essays which are based on multiple documents rather than a chapter that specifies the technical features of these systems. The systems that have been discussed in this chapter pertain to the evaluation of essays based on a closed set of documents with an emphasis on evaluating document use. We have discussed the features of such tasks (i.e., document sets, prompts, documents models, scoring rubrics), dimensions of systems designed to evaluate essays (i.e., semantic benchmarks, specified grain size of content that is assessed, human scoring, and training), promising approaches for addressing these features (keyword matching and regular expressions, high-dimensional semantic spaces, automatic parsing, and machine learning), and finally challenges for implementing systems (optimizing overlap between documents, evaluating transformed content, developing dedicated semantic spaces, creating systems that can be generalized, and providing feedback). To date, we know of only two systems that have been designed to evaluate essays based on multiple documents (Hastings et al., 2012; Hughes et al., 2015), and as such, more research is needed to learn how to best develop these systems. Based on what we have learned in developing these systems, and the challenges raised in the last section, we conclude by identifying key areas that warrant more research to support the development of automatic scoring systems for multiple document use. One pressing issue discussed above is learning how to optimize the level of semantic overlap between documents to maximize the ability to detect their use and provide sufficient semantic scaffolds to afford using those documents to address the task specified in the prompt. The insights gained into this issue arose when developing systems for studies in which this issue was not under consideration (Hastings et al., 2012; Hughes et al., 2015; Wiley et al., 2017). Given that integration is facilitated by the extent that documents in a set discuss similar events (e.g., Kurby et al., 2005), we envision studies that systematically manipulate the level of semantic overlap at the level of events. Criteria would need to be determined for evaluating both the success at which students are able to integrate texts and the success of the automatic evaluation of document use. A second pressing issue is developing approaches to determine the extent that content reflects the relationships specified in the documents model (e.g., causal, logical, and argumentative). Evaluating the extent that explicit content from the documents is in the essay is a relatively less challenging problem because there are

522  •  Magliano et al.

explicit semantic relationships that can be assessed between the essay and documents (i.e., content from the essays can be compared to semantic benchmarks reflecting content from the texts). It is possible to develop a system that can determine explicit linguistic markers of semantic relationships, such as the use of appropriate connectives (logical, temporal, and causal connectives) between important idea units in the essays. However, students may leave these out and rely on conveying the relationships implicitly. One possible solution was described above in the context of Hughes et al. (2015). They developed an assessment protocol to detect relationships using machine learning algorithms. They relied on the gold standard of human judges to classify essays as to the extent that they conveyed important causal relationships necessary to address the essay prompt, and the system was trained to detect the ways that these were conveyed in natural language. While this approach is viable, a large corpus of training texts is required, and certainly more research is needed to identify the optimal strategies for determining if an essay conveys important concepts and relationships between them, as delineated by a documents model. Magliano and Graesser (2012) argued that systems should ideally be developed that use multiple approaches for evaluating the relationships between student-generated content and semantic benchmarks, and the results of Wiley et al. (2017) support this. We have outlined several promising approaches in this chapter, but more research needs to be conducted to learn how to optimally combine the different approaches. One method would be to “let the data decide,” using a type of machine learning ensemble method called stacking (Wolpert, 1992), where the system learns how to balance different classification methods for different essays. A third issue involves the potential to develop systems that can generalize to new prompts and document sets. This is indeed one goal of using machine learning to train systems to evaluate essays. However, given the idiosyncratic nature of the mapping between prompts, documents, documents models, and products for any given task, we see this as one of the more serious challenges raised in the last section. Any solution to this problem would require near-human level understanding of the texts. There are some advanced machine learning methods that use deep learning and unsupervised training (e.g., “zero-shot learning”; Norouzi et al., 2013), but these require extremely large corpora to have a chance at success, and are very much an area for future research. Another potential future direction would be to be able to automatically assess the extent to which source reliability is considered when writing essays from sources that vary with regard to author credibility. Consider, for example, the 2016 election cycle in the United States and the preponderance of unreliable sources reporting “fake news.” It is important for any consumer of information to be able to deal appropriately with varying levels of credibility. Especially in educational writing contexts, students are expected to explicitly identify sources. Future work could build on previous research which was effective at automatically identifying sourcing in texts (and the lack thereof) and also tutored students to improve their sourcing (Britt et al., 2004). Extensions to this research could focus on identifying statements that indicate the writer’s evaluation of the sources. Of course, this is challenging because of many factors. First, evaluating reliability requires that the information seeker has prior knowledge of what makes a good vs. poor

Computer-Based Assessment of Essays  •  523

source (akin to a schema for knowing where to find reliable information), and second, it would depend on a system that would be able to identify when/where unreliable information was present. Another issue to consider is the extent that approaches have to be developed in a manner that is sensitive to the language in which the texts are conveyed. Systems that rely on keyword matching, regular expressions, and machine learning can be readily applied to just about any language system. However, systems that rely on high-dimensional spaces, such as LSA and HAL, require one to build the semantic spaces based on a large sample of documents (e.g., Landauer & Dumais, 1997). As such, the semantic spaces that support these systems have to be built to support the linguistic contexts where the essay coding systems will be applied, but thankfully this is a viable endeavor (León, Olmos, Escudero, Cañas, & Salmerón, 2006; Olmos, León, Escudero, & Jorge-Botana, 2011). Finally, while our emphasis has been on the automatic evaluation of essays, we argue that the automatic assessment of other kinds of constructed responses, such as think-aloud and question–answering protocols (e.g., Magliano et al., 2011), has a utility in research on multiple document use. Higgs (2016) provided a proof of concept for this claim. Her study used LSA to compare think-aloud protocols to semantic benchmarks derived from a documents model that enabled her to evaluate if readers tended to make intra and inter-text connections when reading documents. She found that readers tended to make connections within texts to information delineated as important in the documents model more so than connections across texts. Integration across texts likely happened after the initial readings of the documents in the set. In summation, we hope that this chapter provides cognitive, learning, and educational scientists information about the tools and approaches they need to develop systems that have utility both in research and educational contexts. These systems will afford the use of essays and other student-constructed responses to study multiple document use in the context of task-oriented reading (e.g., Rouet & Britt, 2011). However, as we have emphasized, they will also likely lead to the development of tools that could eventually be integrated into classroom use. Considerably more research is needed for that outcome to be realized and we hope that this chapter provides a foundation for this work.

REFERENCES Achieve (2013). Next generation science standards. Washington, DC: National Academies Press. Aho, A. V. (1990). Algorithms for finding patterns in strings. In J. van Leeuwen (Ed.), Handbook of theoretical computer science, volume A: Algorithms and complexity (pp. 255–300). Cambridge, MA: The MIT Press. Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. Blaum, D., Griffin, T. D., Wiley, J., & Britt, M. A. (2017). Thinking about global warming: Effect of policy-related documents and prompts on learning about causes of climate change. Discourse Processes, 54, 303–316. Bråten, I., Strømsø, H. I., & Britt, M. A. (2009). Trust matters: Examining the role of source evaluation in students’ construction of meaning within and across multiple texts. Reading Research Quarterly, 44, 6–28. Britt, M. A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20, 485–522. Britt, M. A., & Rouet, J.-F. (2012). Learning with multiple documents: Component skills and their acquisition. In M. J. Lawson & J. R. Kirby (Eds.), The quality of learning (pp. 276–314). Cambridge, UK: Cambridge University Press.

524  •  Magliano et al. Britt, M. A., Wiemer-Hastings, P., Larson, A., & Perfetti, C. (2004). Using intelligent feedback to improve sourcing and integration in students’ essays. International Journal of Artificial Intelligence in Education, 14, 359–374. Chen, D., & Manning, C. D. (2014). A fast and accurate dependency parser using neural networks. Proceedings for Conference on Empirical Methods on Natural Language Processing. Retrieved from http://emnlp2014. org/papers/emnlp2014-proceedings.pdf. Claassen, E. (2012). Author representation in literary reading. Utrecht, The Netherlands: John Benjamins. Clark, K., & Manning, C. D. (2016, November). Deep reinforcement learning for mention-ranking coreference models. Paper presented at the Conference on Empirical Methods on Natural Language Processing, Austin, Texas. Council of Chief State School Officers. (2010). The common core standards for English language arts and literacy in history/social studies and science and technical subjects. Washington, DC: National Governors Association for Best Practices. Retrieved from www.corestandards.org. Dai, J., Raine, R. B., Roscoe, R., Cai, Z., & McNamara, D. S. (2011). The Writing-Pal tutoring system: Development and design. Journal of Engineering and Computer Innovations, 2, 1–11. de Marneffe, M. C., Connor, M., Silveira, N., Bowman, S. R., Dozat, T., & Manning, C. D. (2013, August). More constructions, more genres: Extending Stanford dependencies. Paper presented at the International Conference on Dependency Linguistics, Prague, The Czech Republic. Firth, J. R. (1957). A synopsis of linguistic theory 1930–1955. Studies in linguistic analysis. Oxford: Philological Society (pp. 1–32). Reprinted in F. R. Palmer (Ed.) (1968). Selected papers of J.R. Firth 1952–1959. London: Longman. Foltz, P. W., Gilliam, S., & Kendall, S. (2000). Supporting content-based feedback in on-line writing evaluation with LSA. Interactive Learning Environments, 8, 111–127. Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H., Ventura, M., Olney, A., & Louwerse, M. M. (2004). AutoTutor: A tutor with dialogue in natural language. Behavior Research Methods, 36, 180–193. Graesser, A. C., & McNamara, D. S. (2011). Computational analyses of multilevel discourse comprehension. Topics in Cognitive Science, 3, 371–398. Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehen Psychological Review, 101, 371–395. Hastings, P., Hughes, S., Magliano, J. P., Goldman, S. R., & Lawless, K. (2012). Assessing the use of multiple sources in student essays. Behavior Research Methods, 44, 622–633. Higgs, K. P. (2016). Task specificity and multiple document integration (Unpublished doctoral dissertation). Northern Illinois University, DeKalb, Illinois. Hughes, S., Hastings, P., Britt, M. A., Wallace, P., & Blaum, D. (2015). Machine learning for holistic evaluation of scientific essays. In C. Conti, N. Hefferman, A. Mitrovic, & M. F. Verdejo (Eds.), Artificial intelligence in education (pp. 165–175). New York: Springer. Jones, M. N., Kintsch, W., & Mewhort, D. J. K. (2006). High-dimensional semantic space accounts of priming. Journal of Memory and Language, 55, 534–552. Kintsch, E., Caccamise, D., Franzke, M., Johnson, N., & Dooley, S. (2007). Summary Street®: Computer-guided summary writing. In T. K. Landauer, D. M. McNamara, S. Dennis, & W. Kintsch (Eds.), Latent semantic analysis (pp. 263–277). Mahwah, NJ: Erlbaum. Kopp, K., Rupp, K., Blaum, D., Wallace, P., Hastings, P., & Britt, M. A. (2016, November). Assessing the influence of feedback during a multiple document writing task in science. Poster presented at the Annual Meeting of the Psychonomic Society, Boston, MA. Kurby, C. A., Britt, M. A., & Magliano, J. P. (2005). The role of top-down and bottom-up processes in betweentext integration. Reading Psychology, 26, 335–362. Kurby, C. A., Wiemer-Hastings, K., Ganduri, N., Magliano, J. P., Millis, K. K., & McNamara, D. S. (2003). Computerizing reading training: Evaluation of a latent semantic analysis space for science text. Behavior Research Methods, 35, 244–250. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104, 211–240. León, J. A., Olmos, R., Escudero, I., Cañas, J. J., & Salmerón, L. (2006). Assessing short summaries with human judgments procedure and latent semantic analysis in narrative and expository texts. Behavior Research Methods, 38, 616–627. Lund, K., Burgess, C., & Atchley, R. A. (1995). Semantic and associative priming in a high-dimensional semantic space. In J. D. Moore & J. F. Lehman (Eds.), Proceedings of the Seventeenth Annual Conference of the Cognitive Science Society (pp. 660–665). Mahwah, NJ: Erlbaum.

Computer-Based Assessment of Essays  •  525 Magliano, J. P., & Graesser, A. C. (2012). Computer-based assessment of student-constructed responses. Behavior Research Methods, 44, 608–621. Magliano, J. P., McCrudden, M. T., Rouet, J. F., & Sabbatini, J. (in press). The modern reader: Should changes to how we read affect research and theory? In M. F. Schober, M. A. Britt, & D. N. Rapp (Eds.), Handbook of discourse processes (2nd ed.). New York: Routledge. Magliano, J. P., & Millis, K. K. (2003). Assessing reading skill with a think-aloud procedure. Cognition and Instruction, 21, 251–283. Magliano, J. P., Millis, K. K., The RSAT Development Team, Levinstein, I., & Boonthum, C. (2011). Assessing comprehension during reading with the reading strategy assessment tool (RSAT). Metacognition and Learning, 6, 131–154. McCrudden, M. T., Magliano, J. P., & Schraw, G. (Eds.) (2011) Text relevance and learning from text. Greenwich, CT: Information Age. McCrudden, M. T., & Schraw, G. (2007). Relevance and goal-focusing in text processing. Educational Psychology Review, 19, 113–139. McNamara, D. S., Graesser, A. C., McCarthy, P., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge, UK: Cambridge University Press. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L Bottou, & M Welling (Eds.), Proceedings of Advances in Neural Information Processing Systems (pp.  3111–3119). La Jolla, CA: Neural Information Processing Systems Foundation. Millis, K. K., Magliano, J. P., Todaro, S., & McNamara, D. S. (2007). Assessing and improving comprehension with latent semantic analysis. In T. Landauer, D. S. McNamara, S. Dennis, & W. Kintsch (Eds.), The handbook of latent semantic analysis (pp. 207–225). Mahwah, NJ: Erlbaum. Mislevy, R. J. (1993). Foundations of a new test theory. In N. Frederikson, R. J. Mislevy, & I. I. Bejar (Eds.), Tests theory for a new generation of tests. Hillsdale, NJ: Erlbaum. Mitchell, T. (1997). Machine learning. Columbus, OH: McGraw-Hill. Morales, A., Premtoon, V., Avery, C., Felshin, S., & Katz, B. (2016). Learning to answer questions from Wikipedia Infoboxes. In Proceedings of the 2016 Conference on Empirical Methods on Natural Language Processing. Stroudsburg, PA: Association for Computational Linguistics. Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome, A., & Dean, J. (2013). Zero-shot learning by convex combination of semantic embeddings. Retrieved from https://arxiv.org/pdf/1312.5650.pdf. OECD (2008). PISA 2009 assessment framework: Key competencies in reading, mathematics and science. Paris: OECD. (Retrieved August 5, 2010 from www.oecd.org.) Olmos, R., León, J. A., Escudero, I., & Jorge-Botana, G. (2011). Using latent semantic analysis to grade brief summaries: Some proposals. International Journal of Continuing Engineering Education and Life Long Learning, 21, 192–209. Pellegrino, J. W., & Chudowsky, N. (2003). The foundations of assessment. Interdisciplinary Research and Perspectives, 1, 103–148. Pellegrino, J. W., Chudowsky, N., & Glaser, R. (Eds.). (2001). Knowing what students know: The science of design of educational assessment. Washington, DC: National Academy of Sciences. Rouet, J. F. (2006). The skills of document use: From text comprehension to web-based learning. Mahwah, NJ: Erlbaum. Rouet, J. F., & Britt, M. A. (2011). Relevance processing in multiple document comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Text relevance and learning from text (pp. 19–52). Greenwich, CT: Information Age. Russell, S., & Norvig, P. (2010). Artificial intelligence: A modern approach. Upper Saddle River, NJ: Pearson. Shermis, M. D., & Burstein, J. (Eds.) (2013). Handbook of automated essay evaluation: Current applications and new directions. New York: Routledge. Snow, C., & the RAND Reading Study Group. (2002). Reading for understanding: Towards a R&D program for reading comprehension. Santa Monica, CA: RAND. Strømsø, H. I., Bråten, I., & Britt, M. A. (2010). Reading multiple texts about climate change: The relationship between memory for sources and text comprehension. Learning and Instruction, 20, 192–204. Turney, P. D., & Pantel, P. (2010). From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37, 141–188. Ventura, M. J., Franchescetti, D. R., Pennumatsa, P., Graesser, A. C., Hu, G. J., & Cai, Z. (2004). Combining computational models of short essay grading for conceptual physics problems. In J. C. Lester, R. M. Vicari,

526  •  Magliano et al. & F. Paraguac (Eds.), Proceedings of the Intelligent Tutoring Systems Conference (pp. 423–431). Berlin: Springer. Vidal-Abarca, E., Mañá, A., & Gil, L. (2010). Individual differences for self-regulating task-oriented reading activities. Journal of Educational Psychology, 102, 817–826. Wiley, J., Goldman, S., Graesser, A., Sanchez. C., Ash, I., & Hemmerich, J. (2009). Source evaluation, comprehension, and learning in internet science inquiry tasks. American Educational Research Journal, 46, 1060–1106. Wiley, J., Hastings, P., Blaum, D., Jaeger, A. J., Hughes, S., Wallace, P., & Britt, M. A. (2017). Different approaches to assessing the quality of explanations following a multiple-document inquiry activity in science. International Journal of Artificial Intelligence in Education, 23, 1–33. Wiley, J., & Voss, J. F. (1999). Constructing arguments from multiple sources: Tasks that promote understanding and not just memory for text. Journal of Educational Psychology, 91, 301–311. Wineburg, S. S. (1991). On the reading of historical texts: Notes on the breach between school and the academy. American Educational Research Journal, 28, 495–519. Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5, 241–259. Zwaan, R. A., Langston, M. C., & Graesser, A. C. (1995). The construction of situation models in narrative comprehension: An event-indexing model. Psychological Science, 6, 292–297.

29

REFLECTIONS AND FUTURE DIRECTIONS Jason L. G. Braasch university of memphis, usa

Matthew T. McCrudden victoria university of wellington, new zealand

Ivar Bråten university of oslo, norway

This Handbook surveys ongoing international efforts to understand, describe, and explain multiple source use in numerous disciplines and contexts by individuals with different characteristics. When we conceptualized this Handbook, our main goal was to provide a comprehensive overview of research on multiple source use that describes when, how, and why people use multiple sources. To do this, we invited researchers from different disciplines and theoretical orientations. Drawing upon multiple perspectives enabled us to showcase important theoretical and empirical insights into multiple source use in both formal and informal contexts. We also did this to afford opportunities for cross-fertilization across disciplines and to broaden readers’ knowledge base of multiple source use. Thus, this Handbook may help readers discern conceptual relationships across different disciplines and theoretical orientations on multiple source use. As such, we hope it will help readers see similarities, and potential differences, in how theory and research in this area can be advanced and applied. In this final chapter we first revisit and address the six core questions posed in Chapter 1. The goal is to synthesize concepts presented within each of the six sections. We then discuss five future directions that we identified when synthesizing and analyzing ideas presented across the six sections. Finally, we complete the chapter with a general conclusion.

527

528  •  Braasch et al.

REFLECTING ON CORE QUESTIONS POSED IN CHAPTER 1 In our introductory chapter (Chapter 1), we posed six questions to frame the current state of the field on multiple source use. In this section, we revisit these core questions. For organizational purposes, we mainly focus on each section separately in addressing the questions, although there are many connections across the sections. Core Question #1: How Do Theories Describe Macro- and Micro-Level Processes of Multiple Source Use? What Particular Cognitive and Non-Cognitive Activities Guide Individuals as They Construct Understandings from Multiple Sources? Several chapters identify processes that occur before, during, and after multiple source use. As a starting point, individuals are supposed to have a purpose or goal for using multiple sources, which can be self-generated (e.g., reading to decide whether a particular food is “healthy”) or provided (e.g., by a teacher), that guides performance (Britt, Rouet, & Durik, this volume; McCrudden, this volume). At this initial stage, cognitive (e.g., topic knowledge, task familiarity) and affective factors (e.g., attitudes, situational interest, task value) (Britt et al., this volume; Greene, Copeland, Deekens, & Freed, this volume; Guthrie, this volume; List & Alexander, this volume) may influence individuals’ orientation to the task. Individuals then use their goals to guide moment-by-moment processes, such that they tend to process information that is more relevant to their goals differently than information that is less relevant to their goals. These moment-by-moment processes support construction of mental representations of the information of the texts. In relation to the Documents Model Framework, several authors theorize that multiple source users ideally create two types of interconnected representations (Barzilai & Strømsø, this volume; Bråten & Braasch, this volume; Britt et al., this volume; List & Alexander, this volume; Richter & Maier, this volume; Wiley, Jaeger, & Griffin, this volume). The integrated mental model is an elaborated, coherent model of the situation, topic, or phenomenon discussed by the texts. In this sense, people construct a mental model by connecting text information to prior knowledge, and by making connections amongst ideas within and across different texts. The intertext model represents links between source feature information and content information (Author A made Statement A), or links between source features including support (Author A corroborates what Author B said) or discrepancy (Author A contradicts what Author B said). Information accessed from multiple sources must be evaluated in a number of ways to support the construction of these mental representations, including the extent to which it is relevant to current reading goals (McCrudden, this volume), is interesting (List & Alexander, this volume), is consistent with prior attitudes, knowledge, and beliefs (Richter & Maier, this volume; Wegener, Patton, & Haugtvedt, this volume), and is provided by a credible source (Hartman, Hagerman, & Leu, this volume; McCrudden, this volume). If information sufficiently meets these criteria, individuals will consider it useful, and employ meaning-making strategies to process the ideas and concepts deeply enough such that they become incorporated into the current mental model (Cho, Afflerbach, & Han, this volume; Hartman et al., this volume). When information is consistent with prior attitudes, knowledge, or beliefs, as well as with previously read text(s), it may be more readily incorporated into readers’ mental models (Braasch & Bråten, 2017; Richter & Maier, this volume; Wegener et al., this volume).

Reflections and Future Directions  •  529

However, if information contradicts prior attitudes, knowledge, beliefs, or the content of previously read text(s), people may experience a failure in epistemic monitoring (Richter & Maier, this volume), cognitive conflict (Bråten & Braasch, this volume), or perception of a threat (Salmerón, Kammerer, & Delgado, this volume; Wegener et al., this volume). As a result, they may strategically address the discrepancy by choosing to disregard or actively refute information conveyed by a less credible source (Wegener et  al., this volume), to construct a more balanced representation of the controversy (Richter & Maier, this volume), or to use source features in efforts to organize their mental representation, including inferences about the author’s knowledge or the publication venues’ intentions (Bråten & Braasch, this volume). Effective inquiry requires that people self-monitor understandings as they develop over the process of interacting with multiple sources (Barzilai & Strømsø, this volume), and their progress toward completing the task goals (Greene et al., this volume). If attempts at meeting current goals are not successful, effective individuals self-regulate by flexibly applying new strategies in hopes of resolving gaps between what they know so far, and what they still seek to discover (Cho et al., this volume; Greene et al., this volume). However, if thresholds for sufficient understandings are met, individuals will infer that multiple source use goals have been completed. Thus, one fundamental assumption stemming from theories of multiple source use is that people have subjective thresholds they use to decide whether they have sufficiently understood the information (Britt et al., this volume; List & Alexander, this volume; van den Broek & Kendeou, 2015). The consequence of these processes is a mental representation that individuals presumably use in the current and potentially future contexts. Core Question #2: What Individual Characteristics Support (or Constrain) Multiple Source Use? The multiple source use processes described in response to core question #1 pertain to resources that are external to the individual (e.g., task instructions, the types of documents that are available or accessible) as well as individuals’ internal resources (e.g., prior domain knowledge, reading abilities, self-regulation skills). Some individual differences can promote effective multiple source use. Barzilai and Strømsø (this volume) comprehensively identify several types of factors that guide multiple source use including cognitive, metacognitive, motivational-affective, and socio-cultural abilities and experiences. Some of these factors are more stable, such as pre-existing epistemic beliefs (Barzilai & Strømsø, this volume), whereas other factors can change more flexibly while interacting with multiple sources, such as metacognitive and self-regulatory skills and strategies (Cho et al., this volume; Greene et al., this volume). Conversely, other individual differences can impede effective multiple source use. For example, the challenges typically developing readers face when reading multiple sources via the Internet, such as locating sources or shifting attention across multiple sources, are amplified for readers with dyslexia (Anmarkrud, Brante, & Andresen, this volume). Individuals also commonly evaluate belief-consistent information more favorably than belief-inconsistent information, which can even override the quality of the information and its relevance to the task (McCrudden, this volume; Richter & Maier, this volume). Indeed, researchers have identified many individual differences that relate to multiple source use. However, far less is known about potentially more complex ways that multiple interacting factors contribute to multiple source use. Future research

530  •  Braasch et al.

should advance in this direction to more adequately examine unique and interactive contributions of individual differences to multiple source use. Core Question #3: Are There Discipline-Specific (e.g., History Versus Science Learning) and Domain-General Multiple Source Use Processes? Section III of this Handbook focuses on multiple source use in history, science, mathematics, and literature. Across the four disciplines, these authors indicated that effective users of multiple sources rely on their prior knowledge to evaluate and interpret concepts, events, themes, or statements expressed by multiple sources, which can help them select relevant information, and actively incorporate it into their developing mental models. Nevertheless, how people use prior knowledge substantially differs across these disciplines. In literature, effective individuals use general world knowledge and specific experiences to extract common themes across multiple sources, including those reflecting the human condition (Bloome, Kim, Hong, & Brady, this volume). In science, effective individuals use prior knowledge to evaluate authors’ data collection methods and the degree to which the empirical results support the main claims (Tabak, this volume). In history, effective individuals attempt to corroborate different pieces of evidence across texts (Fox & Maggioni, this volume). In mathematics, proofs of theorems in texts are often considered definitive, requiring little additional scrutiny. Weber (this volume) stresses that, once a proof has been established, people rarely expend effort considering alternative pieces of evidence, as is the primary concern for people using multiple sources in history. There are also discipline-specific strategies for attending to, representing, evaluating, and using source features found within texts. In science, individuals use source features to judge credibility, and regulate their reading accordingly. A greater familiarity with an author or a greater knowledge about a publication venue’s vetting processes, for example, can guide expectations about rigor and trustworthiness (Tabak, this volume). As stated above, in mathematics, people typically consider proofs of theorems as definitive, but authors of theorems occasionally make errors (Weber, this volume). As such, Weber (this volume) states that multiple source users in mathematics also engage in sourcing strategies to evaluate the trustworthiness of mathematical statements. Thus, it appears that science and mathematics readers use somewhat similar sourcing processes to verify knowledge claims. In history, effective individuals think about primary (written contemporaneously with the relevant historical event) and secondary texts (an interpretation of the event after it has occurred) as socially constructed artifacts, written by particular authors with unique perspectives, intentions, and biases (Britt, Rouet, & Braasch, 2013). Fox and Maggioni (this volume) highlight that acquiring new knowledge from historical texts is fundamentally a process of understanding the authors’ “voices” when forming a mental model of the event(s) described across texts. Core Question #4: How Do People Use Multiple Sources in Non-Academic Contexts? In What Ways Are the Processes Similar to or Different from Those Used in Academic Contexts? A primary characteristic of multiple source use in non-academic contexts is that individuals most often choose to seek out information that addresses their particular

Reflections and Future Directions  •  531

needs, rather than having a task (and potentially a subset of information) provided to them. This commonly involves more open-ended inquiry via the Internet in which people formulate their own goals for reading. Salmerón et al. (this volume) identify three key phenomena that affect multiple source use within non-academic contexts, including a link’s position in a search engine result (e.g., accessing links near the top of the page), individuals’ dispositions to access links that are aligned with their existing views, and preference for personal narratives over quantitative evidence from a scientific study. These tendencies, of course, are also driven by familiarity with and knowledge about the topic, and individuals’ decisions about whether to evaluate and rely upon the credibility of the source when determining whether to use information (Bromme, Stadtler, & Scharrer, this volume). Thus, similar to multiple source use in academic contexts, self-regulation in multiple source use in non-academic contexts is related to how effectively people appraise information and sources (Greene et al., this volume). Interestingly, but perhaps not surprisingly, memory-based processes that support mental model construction when reading in academic and nonacademic contexts appear to be indistinguishable (Donovan & Rapp, this volume). Ultimately, there is much more room for research on similarities and differences between multiple source use in academic and non-academic contexts (as addressed in future direction #4). Core Question #5: What Types of Tasks and Interventions Promote Successful Multiple Source Use? Authors in Section V describe several interventions that promote efficient, effective multiple source use. Wiley and colleagues (this volume) extensively review research on the effects of inquiry tasks and the task environment on multiple source use in history and science. Some research provided in the chapter involved manipulating features of the task environment, including whether the same information was presented in one versus across multiple sources, whether the reader could choose the order with which the texts were presented, the types of documents that were available, and whether a discrepancy existed across the sources. Several aspects (e.g., viewing multiple texts in multiple windows, presence of discrepancies) promote a deeper comprehension of the information provided by multiple sources. All four chapters in Section V focus on instructional contexts that take place within K-12 classrooms (Brand-Gruwel & van Strien; Guthrie; Hemphill & Snow; Wiley et  al.). Broadly speaking, the interventions described in this section, both short and long in duration, are quite promising in promoting multiple source use. This is true when the classroom interventions focus on a set of skills related to multiple source literacy and learning, motivation in multiple source use, or problemsolving using resources from the Internet. This is also the case if an intervention focuses on a more specific sub-skill, such as source evaluation while searching the Internet. Based on this work, it is clear that students need and also can benefit from instruction that targets skills and strategies for dealing with multiple sources. Thus, Section V has the potential to provide useful information for K-12 classroom practitioners concerning the educational applicability of theory and research in the field of multiple source use.

532  •  Braasch et al.

Core Question #6: How Can Different Kinds of Assessments Provide Insight into the Component Processes and Products of Multiple Source Use? The authors in Section VI present and discuss innovative approaches to measuring the component processes and products associated with multiple source use. Measures of online processing – e.g., navigation logs, eye movement patterns, notes taken, and think-aloud utterances produced as information is inspected – and offline processing – e.g., written essays, multiple-choice responses, website ratings – can be used to inves­ tigate how readers evaluate and integrate information from multiple sources (Mason & Florit, this volume; Wiley et al., this volume). Building on an evidence-centered design perspective (Mislevy, Steinberg, & Almond, 2003), Goldman, Blair, and Burkett (this volume) argue that assessments focusing on processes or products of multiple source use need to involve connections amongst three elements. That is, multiple source use assessments should include clear articulation of the constructs of interest, identification of the types of observable(s) readers need to demonstrate to provide adequate evidence of competency, and inferences that specify how these observables can be taken as evidence for their competency (i.e., their mastery of the constructs of interest). Several authors in Section VI raise questions about the adequacy of current assessments for capturing multiple source use in and out of school. For example, assessments that focus on individual performance may not adequately capture readers’ collaborative construction of understandings from multiple texts. To address this issue, Coiro, Sparks, and Kulikowich (this volume) developed the Online Collaborative Inquiry and Social Deliberation in Virtual Worlds project, which targets pairs of readers’ remote collaboration as they engage in multiple source use (e.g., gathering, processing, and evaluating texts). Another limitation is the ability to assess decision making in real time (Goldman et al., this volume; Magliano, Hastings, Kopp, Blaum, & Hughes, this volume; Mason & Florit, this volume). To address this issue, Magliano et al., for example, discuss promising automated approaches for scoring essays with respect to readers’ abilities to extract and synthesize information from a discrete set of multiple sources, while also assessing readers’ propensities to indicate where their ideas came from (sourcing). Finally, Sabatini, O’Reilly, Wang, and Dreier (this volume) discuss scenario-based assessments as an alternative, more ecologically valid, and contextualized approach to assessment design. Scenario-based assessments establish an authentic reading purpose (e.g., to help decide whether to build a community garden on a vacant lot) and present readers with tasks and resources, including multiple texts, that they can use to achieve those purposes. Taken together, the innovative assessment approaches described above have the potential to provide a “window” into the processes and products of multiple source use. As all Section VI chapters would attest, assessment design is (and rightfully should be) inextricably linked to theories of multiple source use. As theoretical models are refined and further articulated in future work, such changes should also “downstream” into refinements to multiple source use assessments.

DIRECTIONS FOR FUTURE RESEARCH In analyzing and synthesizing concepts presented in the sections of this Handbook, additional unanswered questions emerged. In this section, we discuss five larger

Reflections and Future Directions  •  533

themes for future directions that span several sections. For each theme, we focus on central prescriptions for future research on multiple source use. Future Direction #1: Future Research Should Continue to Investigate Individuals’ Moment-by-Moment Multiple Source Use Decisions, and Relate Them to Multiple Source Comprehension Previous research on moment-by-moment processes and subsequent mental models has illuminated the underpinnings of multiple source comprehension. For example, associations between think-aloud comments produced during reading and measures of integration within essays generated after reading have afforded inferences about the nature of mental representations of multiple sources (Anmarkrud, Bråten, & Strømsø, 2014). Studies like these highlight specific reading behaviors that lead to more- versus less-successful outcomes, thereby increasing awareness of the kinds of activities that warrant encouragement and support in multiple source use (Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012). One direction for future research may be to incorporate multiple measures of online processing (e.g., complementing online navigation patterns with post-reading, cued interviews) in efforts to more completely describe how reading processes unfold in real time (see Jarodzka & Brand-Gruwel, 2017, for a recent example). Moreover, replications of patterns using online and offline measures across different samples, task instructions, topics, materials, and cultures could further verify inferences about the generalizability of processes underlying multiple source use, or identify boundary conditions. In addition, mixed methods may have advantages for further investigating and explaining different facets of multiple source use (McCrudden, Stenseth, Bråten, & Strømsø, 2016; Wiley et al., this volume). Future Direction #2: Future Research Should Investigate Additional Moderators and Mediators of Multiple Source Use Researchers have predominantly independently used experimental (e.g., Wiley & Voss, 1999) or individual difference approaches (e.g., Bråten, Ferguson, Strømsø, & Anmarkrud, 2013) to investigate multiple source use. Additionally, some researchers have used both experimental manipulations (e.g., reading task) and one or more individual difference measures (Bråten & Strømsø, 2010; Maier & Richter, 2013) in the same study. Future research should incorporate more complex designs to investigate additional interactive effects between experimentally manipulated variables (e.g., texts, tasks) and multiple individual difference variables in the same study. Such an approach could provide useful information about moderators and mediators of previously established effects. For example, Barzilai and Eshet-Alkalai (2015) measured readers’ epistemic beliefs and asked them to read either blog posts that contained consistent or inconsistent information on a socio-scientific topic. As in earlier research (Braasch, Rouet, Vibert, & Britt, 2012), when the text contained inconsistent information, readers directed more attention toward source features; however, this was only the case for readers who believed that knowledge claims should be justified through critical evaluation and consideration of evidence. Approaches like these enable researchers to gain deeper insights into the nature of multiple source use, and help them pose new questions about variables that mediate

534  •  Braasch et al.

and moderate multiple source use. For instance, what particular strategies mediate the relationships between task instructions and multiple source use? How might prior knowledge or interest moderate these effects? Future work including additional mediators and moderators could benefit understandings of mechanisms of multiple source use, and what particular reader characteristics promote them (see Bråten, Anmarkrud, Brandmo, & Strømsø, 2014, for a recent example). Such information could prove useful for expansions and refinements of theoretical models. In a similar vein, research has provided examples where instructional manipulations and interventions designed to promote multiple source use were not comparably beneficial for all readers. That is, certain affective states, knowledge, skills, strategies, and dispositions appear to moderate the benefits of multiple source use interventions. For example, Gil, Bråten, Vidal-Abarca, and Strømsø (2010) found that students with higher prior knowledge displayed more integrated understandings of conflicting information presented by multiple sources as a function of writing an argument. Students with lower prior knowledge, however, appeared to be more hindered than helped by argument task instructions. In another example, Macedo-Rouet, Braasch, Britt, and Rouet (2013) conducted a brief classroom-based intervention to encourage students to identify source features, establish source-content links, and assess the competence of each source with respect to the reading topic. The intervention produced better comprehension and sourcing relative to a control condition amongst less skilled readers, but more skilled readers performed well independently of the intervention. Of course, it may never be possible to construct a “one size fits all” intervention that benefits all readers. However, future research could more deeply explore particular reader characteristics that constrain or promote the success of different kinds of multiple source use interventions. Such studies could more clearly identify the parameters of multiple source use interventions, while also encouraging adaptations of current interventions described in Section V. Adaptations could offer a broader impact on readers from diverse backgrounds. Future Direction #3: Future Research Should Investigate the Different Ways that Individuals Interpret Tasks Provided to Them, the Ways They Re-Purpose Information to Suit Their Goals, and How These Processes Constrain or Promote Multiple Source Use The chapters in this Handbook consistently emphasize that reading tasks (whether assigned by others or reader-generated), goals, and purposes all importantly guide multiple source use. Although theoretical models indicate that individuals can differently interpret the same academic reading assignment (Britt et  al., this volume), empirical studies could be conducted to better understand how different task interpretations constrain or promote multiple source use. Assessments will need to be sensitive enough to capture potentially subtle differences in task interpretation, how those interpretations guide multiple source use, and ultimately whether certain task interpretations lead to greater success than others. Having people generate think-aloud protocols, in particular, could offer insight into initial specifications of task goals, and their dynamic, subjective adjustments during multiple source use experiences (Britt et al., this volume; Greene et al., this volume; Hartman et al., this volume). Such approaches could elucidate how updated goals change the nature of the processes in

Reflections and Future Directions  •  535

which people engage (e.g., in creating new task goals, information previously considered irrelevant might become relevant) (McCrudden, this volume). It is also important to note that although authors provide information for particular purposes, these may not always align with the information consumer’s goals (DeSchryver, 2014; McCrudden, this volume; Wiley et al., this volume). Thus, future research might also investigate the success with which individuals selectively repurpose information from multiple sources to satisfy their task goals. Of course, more dynamic conceptions of task goals and information re-purposing only add layers of increased complexity to theoretical models of multiple source use. However, these dynamic changes undoubtedly occur within authentic contexts. Accordingly, theories may require revision to more adequately characterize multiple source use as it happens “in the wild.” Future Direction #4: Theories Need to Consider How Multiple Layers of Contexts Contribute to Multiple Source Use Future work should continue to explore when, how, and why a much broader spectrum of contexts influences multiple source use, perhaps in interactive ways. Speaking metaphorically, multiple source use contexts can be viewed as concentric circles that vary in terms of proximity to the reader. The most proximal is the immediate context in which multiple source use takes place. Wiley et al. (this volume) document several contextual factors specific to the reading environment that can affect multiple source use including the types of sources that are available, their order of presentation (fixed versus reader-selected), and presentation format (viewing one window versus multiple windows at a time). These more proximal contextual factors can differentially contribute to the success with which individuals use multiple sources. At a slightly larger grain size of environmental context, multiple source use occurs in academic (K-12 classrooms) and non-academic contexts (reading at home for pleasure). The latter scenario, of course, is much less constrained than the former. However, even within academic contexts, the amount and explicitness of instruction addressing best practices for multiple source use impact comprehension and learning (Brand-Gruwel & van Strien, this volume; Wiley et al., this volume). Thus, just like for more proximal aspects of context, larger environmental contextual factors, and other broad socio-cultural contexts such as families, ethnicities, and cultures, may differentially contribute to readers’ approaches and successes in using multiple sources (Loughlin & Alexander, 2016). At an even larger contextual grain size, multiple source use typically occurs within – and is shaped by – the greater context of online information technologies. Internet algorithms play a large – and somewhat invisible – role in a person’s access to and presumably use of multiple sources (Hartman et  al., this volume). They greatly restrict what the person “would want to see,” and in what order, based on prior subjective Internet use experiences. The opposite is also true: The same algorithms filter out what information a person would “not want to see” as well, in essence creating “filter bubbles” (Pariser, 2012). Thus, although diverse information may be available on the Internet, people may only access a narrow subset of information given the Internet has created personalized, potentially homogenized, multiple source experiences for them.

536  •  Braasch et al.

As future work continues to explore the ways that a broader spectrum of contexts influences multiple source use, it might uncover important interactions. Additional studies could investigate whether more formal academic contexts elicit a different set of processes relative to more informal contexts within the same individuals. For example, students might evaluate source feature information differently when they complete an assignment for biology class than when they look up health information on the web to make a decision about which medication to take. For the biology assignment, they may read the texts in the order in which they were presented within the search engine output, opening and closing one window at a time until the assignment has been completed. By contrast, the same readers might be much more selective about which sources to even open from the output when they read to make a decision about their personal health, keeping multiple windows open at one time to compare information provided by each text. In this example, additional research could elucidate whether and in what ways readers can flexibly use different knowledge, skills, strategies, and dispositions dependent on the nature of the reading context. Additional empirical investigations could also shed light on particular aspects of theories that may require revision to account for the various ways that different kinds of contexts constrain multiple source use, in unique and interactive ways. For an extended discussion of the importance of contextual factors in reading, see Britt, Rouet, and Durik (2018). Future Direction #5: Theories and Empirical Research Have Not Adequately Addressed Multiple Source Use from Developmental Perspectives Much of the research on multiple source use is a collection of independent studies, many of which were conducted using samples of proficient college student readers (Goldman et al., 2012; Wiley & Voss, 1999). Fewer studies of multiple source use have employed cross-sectional designs, e.g., comparing high school to university students (e.g., Rouet, Britt, Mason, & Perfetti, 1996), or high school students to expert readers (e.g., Wineburg, 1991). Although the development of multiple source use – and the knowledge, skills, strategies, and dispositions that undergird effective performance – is an undoubtedly important area to study, research addressing how multiple source use evolves over time is scant (see, however, Perfetti, Britt, & Georgi, 1995; Strømsø, Bråten, & Samuelstuen, 2003). Thus, future research on the development of multiple source use should preferably employ longitudinal designs. Such designs could provide further insight into the developmental nature of readers’ selection of texts, their processing and evaluation of information they find across longer time periods, and the ways they apply information gleaned from multiple sources to complete various kinds of tasks. In addition, research employing a developmental perspective with younger readers could identify at what age readers begin to capably engage in multiple source use, which could inform how and when early interventions might best support progress.

CONCLUSION The information universe expands and diversifies with every second that passes. Consumers of information must be adept in knowing when, how, and why they should engage in various activities, and how these contribute to effective multiple source use.

Reflections and Future Directions  •  537

As Handbook editors, we sought to provide a comprehensive overview of the landscape of research on multiple source use. Although this Handbook is a testament to the major advances in theory development and empirical investigations of multiple source use, as well as the educational practices that are most supportive of it, there is still clearly much more work to be done. We hope that the ideas found within this Handbook promote dialogue between scholars from different disciplines and perspectives as we continue toward a unified goal: optimizing multiple source use for people from diverse backgrounds.

REFERENCES Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. Barzilai, S., & Eshet-Alkalai, Y. (2015). The role of epistemic perspectives in comprehension of multiple author viewpoints. Learning and Instruction, 36, 86–103. Braasch, J. L. G., & Bråten, I. (2017). The discrepancy-induced source comprehension (D-ISC) model: Basic assumptions and preliminary evidence. Educational Psychologist, 52, 167–181. Braasch, J. L. G., Rouet, J. F., Vibert, N., & Britt, M. A. (2012). Readers’ use of source information in text comprehension. Memory & Cognition, 40, 450–465. Bråten, I., Anmarkrud, Ø., Brandmo, C., & Strømsø, H. I. (2014). Developing and testing a model of direct and indirect relationships between individual differences, processing, and multiple-text comprehension. Learning and Instruction, 30, 9–24. Bråten, I., Ferguson, L. E., Strømsø, H. I., & Anmarkrud, Ø. (2013). Justification beliefs and multipledocuments comprehension. European Journal of Psychology of Education, 28, 879–902. Bråten, I., & Strømsø, H. I. (2010). Effects of task instruction and personal epistemology on the understanding of multiple texts about climatechange.Discourse Processes, 47, 1–31. Britt, M. A., Rouet, J.-F., & Braasch, J. L. G. (2013). Documents experienced as entities: Extending the situation model theory of comprehension. In M. A. Britt, S. R. Goldman, & J. F. Rouet (Eds.), Reading: From words to multiple texts (pp. 160–179). New York: Routledge. Britt, M. A., Rouet, J.-F., & Durik, A. M. (2018). Literacy beyond text comprehension: A theory of purposeful reading. New York: Routledge. DeSchryver, M. (2014). Higher order thinking in an online world: Toward a theory of web-mediated knowledge synthesis. Teachers College Record, 116, 1–44. Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H. I. (2010). Summary versus argument tasks when working with multiple documents: Which is better for whom? Contemporary Educational Psychology, 35, 157–173. Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381. Jarodzka, H., & Brand-Gruwel, S. (2017). Tracking the reading eye: Towards a model of real-world reading, Journal of Computer Assisted Learning, 33, 193–201. Loughlin, S. M., & Alexander, P. A. (2016). Individual differences relations and inter-relations: Reconciling issues of definition, dynamism, and development. In P. Afflerbach (Ed.), Handbook of individual differences in reading: Reader, text, and context (pp. 377–393). New York: Routledge. Macedo-Rouet, M., Braasch, J. L. G., Britt, M. A., & Rouet, J.-F. (2013). Teaching fourth and fifth graders to evaluate information sources during text comprehension. Cognition and Instruction, 31, 204–226. Maier, J., & Richter, T. (2013). Text-belief consistency effects in the comprehension of multiple texts with conflicting information. Cognition and Instruction, 31, 151–175. McCrudden, M. T., Stenseth, T., Bråten, I., & Strømsø, H. I. (2016). The effects of author expertise and content relevance on document selection: A mixed methods study. Journal of Educational Psychology, 108, 147–162. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). Focus article: On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3–62. Pariser, E. (2012). The filter bubble: How the new personalized web is changing what we read and how we think. New York: Penguin Books.

538  •  Braasch et al. Perfetti, C. A., Britt, M. A., & Georgi, M. C. (1995). Text-based learning and reasoning: Studies in history. Hillsdale, NJ: Erlbaum. Rouet, J.-F., Britt, M. A., Mason, R. A., & Perfetti, C. A. (1996). Using multiple sources of evidence to reason about history. Journal of Educational Psychology, 88, 478–493. Strømsø, H. I., Bråten, I., & Samuelstuen, M. S. (2003). Students’ strategic use of multiple sources during expository text reading: A longitudinal think-aloud study. Cognition and Instruction, 21, 113–147. van den Broek, P., & Kendeou, P. (2015). Building coherence in web-based and other non-traditional reading environments: Cognitive opportunities and challenges. In R. J. Spiro, M. DeSchryver, M. S. Hagerman, P. M. Morsink, & P. Thompson (Eds.), Reading at a crossroads? Disjunctures and continuities in current conceptions and practices (pp. 104–114). New York: Routledge. Wiley, J., & Voss, J. F. (1999). Constructing arguments from multiple sources: Tasks that promote understanding and not just memory for text. Journal of Educational Psychology, 91, 301–311. Wineburg, S. S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73–87.

INDEX

academic language 366, 367, 378 accuracy 40, 119, 408, 453; historical sources 208, 210; online sources 126, 288; strategic processing 138 activation principle 404, 407, 410, 411 activity models 343, 344, 354, 355 advertising 71, 79–80 affect 107, 332, 334; see also emotions affective engagement 37–39, 43, 44–45, 46–48, 50, 440–441 affective salience 385–386 African-Americans 261 agency 289 algorithms 67–68, 272, 289, 510, 514, 522, 535 alignment hypothesis 382, 386–397 ambivalence 86–87, 88 annotation 510 application principle 404, 407, 410, 411 argumentation 355–356, 358, 366, 506; argument quality 80–81, 85, 88; conflicting information 194; epistemic cognition 433; history 205, 206, 210, 212, 216, 217–218, 351, 364–365; individual differences 108, 109; literature and language arts 255; RESOLV 24–25, 28, 29, 30–31; scenariobased assessment 456–457, 461–462; skills 432; social deliberation 488–489; strategic processing 103; synthesis 67; task environment 349; task manipulations 345–348; think-aloud protocols 431; topic knowledge 476; validation 151, 155, 156–157, 160–161 assessment 10–11, 425–446, 532; CAEM 51; collaborative inquiry and social deliberation 11, 486–487, 490, 492–493, 495, 496–499; computerbased 11, 504, 508–523, 532; conceptualization of 468–469; design 479–481, 489–490; essays

503–504, 505–514; Evidence-Centered Design 11, 467–468, 469–474, 479, 490, 494, 497; multiplesource curricula 378–379; online inquiry 487, 490, 491–492, 494, 496–499; paradox of 451–452; purpose of 449, 479; scenario-based 11, 447–448, 449–450, 452–463, 494, 497, 532; task interpretation 534; theoretical frameworks 472–474 Assessment and Teaching of 21st Century Skills 493, 495 Assessment of Civic Online Reasoning 491, 494 attention 125–126, 143–144, 439 attitudes 19, 38–39, 43, 145; ambivalent 86–87, 88; CAEM 49, 51; cognitive dissonance 290, 295; history 217; learner attributes 473, 475; persuasion 79–91; policy 356; prior 102–103, 528–529; provenance information 276; selective exposure 296; source evaluation 412 authenticity 3 authors 279, 293, 320, 356; credentials 4, 40, 123, 137, 229, 276, 329; expertise 110, 142, 175–177, 178, 179–180, 277, 427, 477, 507; history 530; mathematics 242, 244–246, 248, 250, 530; science 223, 225, 228–229 automatic assessment 11, 504, 508–523, 532 automaticity 383, 386, 392 autonomous model 255 autonomy 239–240, 241, 251 behavioral dispositions 43, 44 beliefs 7, 38, 145, 154; belief-polarization effect 356; cognitive dissonance 290; false 49, 158–159, 163; learner attributes 473, 475, 478; prior 102–103, 109–110, 153, 161, 187–188, 191–192, 289–290, 528–529; provenance information 276;

539

540  • Index self-beliefs 106–107; self-regulated learning 322; text-belief consistency 103, 152, 155–163, 171, 180, 327–328, 477, 529; text relevance 180; see also epistemic beliefs benefit-cost analysis 20, 28, 30 benevolence 274 bias 4, 42, 226; assessment methods 448; cognitive processes 38–39; confirmation 156, 290, 296; Elaboration Likelihood Model 82–83; history 207; instructional supports 110; mathematics 249; prior beliefs 103, 290; strategic processing 138; validation 163 blogs 118, 222, 232, 279, 294, 296, 350, 431 C3 Framework for Social Studies State Standards 212–214, 218 cessation 47, 49, 51 characters, fictional 303–319 chunking 91, 92 CIS-WEB 405, 407–408, 417–418 cognition, need for 49, 89, 91, 107 Cognitive Affective Engagement Model (CAEM) 6, 34–35, 39–51, 107 cognitive differences 101–103 cognitive dissonance 290, 295, 296 cognitive load 90, 111, 330 cognitive overload 122, 143–144 cognitive processes 6, 17–18, 34, 135; automaticity 383; biased 38–39; CAEM 51; collaborative inquiry 488; eye movements 436, 439; history 208; literature and language arts 9, 255–258, 261–263; motivation 10, 382, 386–397; multimedia learning 121; personal epistemology 145; RESOLV 19–20, 324; self-regulated learning 321, 323–324; think-aloud protocols 434; see also processing; validation cognitive products 43–44, 51 cognitive skills 451 Cognitively Based Assessment of, for, and as Learning (CBAL) 460–462, 492, 494 Coh-Metrix 516–517 coherence 17–18, 122, 430; conflicting information 186, 187, 188, 192–193, 195; epistemic beliefs 358; fiction 316; validation 162, 163 collaboration 64 collaborative inquiry 11, 128, 217, 418, 485–501, 532 College and Work Readiness Assessment (CWRA+) 448, 491, 494 Common Core State Standards (CCSS) 145, 212, 379 competencies 4, 460, 467, 489–490, 532; see also skills complexity 69–70, 71, 292, 386, 441 comprehension 151–152, 225, 342–343; assessment 11, 459, 466–484; assumptions 355; collaborative inquiry 486–487; complex 382, 383, 388–389, 391, 393, 394, 467; definition of 99; Document

Model 35; fiction 316; individual differences 99–116; instructional conditions 341–361; integration 123; interest 326; motivational processes 10, 386–387, 388–390, 393–394; multiple-source curricula 365, 378; note-taking 428; reading time 435; science 226; self-regulated learning 324, 326, 327, 331; simple 383, 388, 391, 393–394; single texts 17–18, 118–119; skills 363; strategic processing 140–141, 145; task manipulations 347–348; validation 152–163 computer-based assessment 11, 504, 508–523, 532 Concept-Oriented Reading Instruction (CORI) 364, 395 confirmation bias 156, 290, 296 conflicting information 8, 184–201, 217, 534; eyetracking studies 438; GISA 453, 457; note-taking 428; provenance information 277; sourcing 278, 507; task environment 349, 350 conjectures 239, 247, 248 connections/connectives 357 consistency 103, 152, 155–163, 180, 278, 350, 477, 529; see also inconsistencies constructive-integrative processing 137, 139–142, 144, 146 Content-Source Integration (CSI) 472 context 57–58, 67–68, 70, 535–536; conflicting information 193–194, 195; history 207; instructional 350–355; reading 100, 386–387; social 255–256, 261, 262 Context Model 18–19, 20–21, 26–29, 323 contextualization 209, 210–211, 216, 217, 348, 351, 352 continued influence effect 308 controversial topics 365, 485; conflicting information 184–186, 188–190, 193, 195; personal narratives 293; prior beliefs 102–103; science 479; social media 322; user comments 295; validation 151–152, 154, 155, 161, 163 corroboration 55–56, 134, 344, 358, 449; C3 Framework 213; CAEM 40–41, 47, 50; conflicting information 190; epistemic beliefs 358; epistemic cognition 433; history 209, 210–212, 216, 217, 351; instructional context 351, 352; learner attributes 478; socio-cultural identities 108; task environment 348, 349, 350, 355; task manipulations 346 credentials 4, 40, 123, 137, 229, 276, 329 credibility 3, 8, 42, 280, 449, 522, 528; conflicting information 195; Elaboration Likelihood Model 82, 83, 85; Evidence-Centered Design 470–471; fiction 313–315; mathematics 250; online sources 62, 123, 143, 286, 291, 531; persuasive messages 92; provenance information 276; science 228, 530; strategic processing 7, 143; text relevance 168–169, 175–180

Index  •  541 critical-analytical processing 137–138, 139–140, 142–143, 144, 146 critical stance 63–64, 65; CAEM 43, 44, 46–47, 48; history 249; science 223 cross-textual linkages 137, 141–142, 215–216 cultural ideologies 261, 262, 263 cultural relevance 217 curation 66, 67 curiosity 326 data collection 240 decisions 297, 533; decision thresholds 20, 21; processing 19, 20, 22–24, 27–32 decoding 1, 119, 141, 366, 473; dyslexic readers 120, 126–127; STARI 365, 372, 378 default stance 42–43, 44–45, 47, 49, 50, 51 deliberation 11, 486, 488–489, 490, 493, 496–499 demonstration principle 404, 407, 410, 411 depletion models 333 design principles 403–404, 406–407, 409–410, 411 developmental perspectives 536 Digital IdeaKeeper 330–331 digital literacy 108, 128 digital texts 60–61, 279, 494 disciplinary concepts 212 disciplinary literacy 379 discrepancies 8, 91, 186–187, 350, 477, 479–480; see also conflicting information; inconsistencies Discrepancy-Induced Source Comprehension (D-ISC) 8, 185–190, 194–195, 472 Discrepancy Motives Model (DMM) 7, 81, 83, 85–86, 87–88, 92 disengagement 43, 44, 46, 47, 48, 392 document, definition of a 99 Document Model (DM) 35, 56, 100, 134, 151, 257, 502, 528; D-ISC model 185–186, 187; engaged responses 41, 42; essay assessment 506, 508, 515, 520; ideal representation 270; MD-TRACE 19, 36, 343, 450; NLP tools 517, 518, 523; RESOLV 21–22; sourcing 278 domain knowledge 119 dyslexia 7, 117–132, 529 Educational Testing Service (ETS) 460–462 elaboration 7, 46, 80, 81–83, 84–91; individual differences 103; science 227; self-regulated learning 324; think-aloud protocols 430 Elaboration Likelihood Model (ELM) 7, 80, 81–83, 85, 89, 92, 288 embedded sources 195, 228, 270 emotions 107, 196, 306, 323, 324, 332, 440 engagement 41–45, 48, 50, 362; multiple-source curricula 379; reading 363, 386–387, 390–391, 434, 436; struggling readers 127–128; see also affective engagement

ePIRLS 459–460, 492, 494 epistemic beliefs 277, 323, 358, 412, 533; conflicting information 192–194, 195–196; eye-tracking studies 437, 438; individual differences 104, 108, 111, 529; reading time 435, 436; think-aloud protocols 431–433 epistemic cognition 277, 278, 321, 323, 431–433, 473, 475, 478 epistemic criteria and standards 104–105 epistemic monitoring 7, 152, 155–156, 157, 161, 529; see also validation epistemic strategies 111 epistemic Stroop paradigm 153–154 epistemological beliefs 155, 156, 160 error-detection paradigm 154–155 essays 345–346, 349, 357–358, 503–504, 505–514, 533 ethnic minorities 107–108 evaluative stance 63–64, 233n1; CAEM 43, 45–46, 47; science 223, 226, 228–232 event-indexing model 305–306, 307 Evidence-Centered Design (ECD) 11, 467–468, 469–474, 479, 490, 494, 497 Evidence Model 470–471, 474, 479, 490 exemplification theory 290, 293 expectancy value theory 384–385 expectations 384, 386, 387–388, 394, 396, 397 expert sources 85, 89, 104 expertise 102, 110, 142, 277, 427, 477; history 211; literature and language arts 262, 263; online sources 291, 293; prior knowledge 325; science 272, 273, 479; source trustworthiness 274; sourcing 507; text relevance 175–177, 178, 179–180; think-aloud protocols 432 explanation-based processing 354 eye-tracking studies 290–291, 425, 436–439, 441–442, 474, 480; CAEM 51; conflicting information 189–190; validation 153, 154–155 Facebook 65, 66 “fake news” 55, 71, 522 false beliefs 49, 158–159, 163 false information 157–159, 292, 308; see also misinformation familiarity with topic 178, 277, 531 feedback 106–107, 327–328, 395; essay assessment 503–504, 508, 511, 521; information-problem solving 411 feeling of knowing evaluation 20, 23, 323, 326 fiction 9, 254, 303–319, 372–374 fluency 101, 119, 366, 367; CORI 395; motivation 391, 392–393; STARI 372, 373, 378 gender 61, 108–109 geometry 250

542  • Index global integrated scenario-based assessments (GISA) 11, 452–458, 460, 492, 494 goals 134, 136, 427, 449, 528–529; goal-focusing model 170; IPS-I 36; literature and language arts 255, 262, 263; product model 506; RESOLV 20–23, 26–30, 31–32; scenario-based assessment 449; self-efficacy 395; self-regulated learning 324–325, 332–333, 487; strategic processing 138, 140–141, 145–146; task goals 343–344, 355, 534–535; text-belief consistency 160–161; text relevance 169–171, 180; validation 155, 162, 163; valuing 387 Google 67–68, 426 grain size 509, 535 heart rate 439–440 heuristics 206, 210–211, 215, 259, 480; instruction 349, 351, 352; learner attributes 473; online sources 288–289, 292 history 8, 10, 205–220, 255, 341–342, 356, 448, 530; argument task 25; curricula 364–365; discrepancies 480; epistemic strategies 111; Evidence-Centered Design 469; historians 209–212, 249; information-problem solving 408–409, 420; instructional context 351, 353; refutation texts 310; self-regulated learning 326; task environment 348–350; task manipulations 345–346, 347–348; topic knowledge 476 hyperlinks 137, 142, 289, 292 Hyperspace Analogue to Language (HAL) 512, 523 hypertext 48, 60, 62, 122, 124 identification of ideas 137, 139 identity 107–108 ideological model 255–256, 259, 261 importance of text 173–175 inconsistencies 153–157, 159–160, 162, 278, 327–328, 533; see also discrepancies individual differences 7, 61, 355, 358, 529–530; CAEM 49; cognitive 101–103; conflicting information 192–193, 194, 195, 196; future research 533; goals 333; metacognitive 103–106; motivational and affective 106–107; multiple document comprehension 99–116; self-regulated learning 324, 331; socio-cultural 107–109; strategic processing 145 inferences 119, 121–122, 146, 224, 394, 529; assessment 468, 473, 475; background knowledge 364; fiction 306–307, 312; history 209; inference verification task 347; mental models 169; misinformation 158; trait 309 informal learning 151 information access 100, 142, 143, 290–292, 294, 297 information accumulation 46, 47 Information Foraging Theory 287–288

information identification 124, 141, 206, 503, 505; see also sourcing information location 137, 139, 141 information overload 4, 411; see also cognitive overload information-problem solving (IPS) 10, 36–37, 286–287, 401–422, 476 Information Problem Solving on the Internet (IPS-I) model 36–37, 402–403, 408, 415, 472 information resources 118, 224–225, 270, 426; assessment 473, 480; research on 474–479; science 221–222, 227–228, 232–233; see also resources; source information scent 287, 292, 297 information self-sufficiency 143 initiating task 42, 43–44 inquiry 356–358, 531; CBAL assessments 461; history 212–214, 215, 218, 341–342; informationproblem solving 409, 411; task manipulations 344–348; see also collaborative inquiry instruction 341–361, 443, 466–467, 478–479, 531; CAEM 50; conflicting information 196–197; epistemic strategies 111; history 214, 215–216, 217, 218; individual differences 110–111; information-problem solving 401, 402, 403, 404–412; literature and language arts 254, 256–257; mathematics 250–251; multiple-source curricula 362–381; reading processes 394–397; scenario-based assessment 462; self-regulated learning 328–331, 333–334; source evaluation 412; strategic processing 145, 146; synthesis 63, 70, 71; validation 163; variability in intervention effectiveness 534 Integrated Model 35, 100, 224–225, 528; MD-TRACE 343; RESOLV 19, 21, 26–31; science 227, 229, 232; sourcing 228 integration 4, 18, 57, 151, 169, 394; assessment 471, 472, 517, 518–519, 521; C3 Framework 213; conflicting information 196; Document Model 186; GISA 453; information-problem solving 407, 410; Internet information 122, 123, 124, 126–127; literature and language arts 258; MD-TRACE 100; strategic processing 141; transfer of skills 404 integrity 274 intentions 136, 170–171, 173, 207, 294 interactivity 289 interdisciplinarity 5, 297 interest 37–38, 43, 390, 396; CAEM 44, 49, 51; individual differences 106, 109; RESOLV 333; self-regulated learning 326 Internet 1, 34, 50, 99, 342, 448; access to the 69; algorithms 535; assessment methods 426, 442–443; cessation 47; collaborative inquiry 11, 485–501; definition of source 118; dyslexic readers 117, 122–123, 124–128; gender differences

Index  •  543 109; historical sources 214; informal learning 151; information-problem solving 10, 36, 286–287, 401–422; multimodality 121; non-academic use 9, 285–302, 531; online meaning construction 55–72, 134, 135; ORCA 458–460; proliferation of information 320; provenance information 279; relevance 169; science information 221–222, 269, 272; self-regulated learning 328; socioeconomic status 108; sourcing challenges 3–4; strategic processing 135–146; unreliability 349; see also search engines; social media; websites Intertext Model 35, 100, 134, 224–225, 257, 270, 528; information-problem solving 403; MD-TRACE 343; RESOLV 19, 21; science 227, 232; sourcing 228; strategic processing 141 intertextuality 134, 141–142, 190; literature and language arts 255, 257, 259–260; note-taking 428, 429 intrinsic motivation 384–397 journals 40, 475; mathematics 239, 242–244, 248, 250; science 222–224, 249 justification 104, 238, 239, 433 keyword matching 511, 512–513, 520, 523 knowledge 123, 168, 431–433, 468; attitudes and beliefs 145; declarative 278–279, 280, 383, 407, 417; disciplinary 110; Elaboration Likelihood Model 80; Evidence-Centered Design 467; GISA 452, 455, 456; history 206, 211, 213, 530; information-problem solving 404, 408, 410; learner attributes 475–476, 478; low-income students 364; mathematics 239–241; MD-TRACE 19, 36; metacognitive 103–105; RESOLV 21, 31; science 230–231, 233, 272–274; Situation Model 152; strategic processing 137, 139, 140; student views 63, 64; validation 154, 155; see also epistemic beliefs; epistemic cognition; prior knowledge language: academic 366, 367, 378; natural language processing 504, 508–523; processing 383, 393; socialization 261 language arts 8–9, 212, 254–266, 379, 460 Latent Semantic Analysis (LSA) 512, 515, 516–518, 519–520, 523 laypeople 290, 292, 342; provenance information 274, 275–280; science 221–222, 226–231, 270 learner attributes 473, 475–476, 478 learning 71, 145, 355; assessment 479, 481; CBAL 460–462; collaborative 487–488; deep 32, 123, 467, 522; design principles 404; literature and language arts 258; multimedia 121–122, 126, 128; outcomes 358; RESOLV 31; self-regulated 9, 144, 320–338, 383, 487

lexical pathway 120 link-selection behavior 290–291 linking strategies 103 literacy 10, 58, 69, 366; CBAL 461; conflicting information 185; information-problem solving 409; multisource 362–363, 364, 379; scientific 221, 232–233 literature 8–9, 254–266, 530 longitudinal designs 536 low-income students 364 machine learning 510–511, 514–517, 519, 522–523 MAIN model 289 majority-minority status of source 86 mathematics 8, 212, 238–253, 460, 530 meaning 1, 2, 17–18, 134–135; definition of reading 133–134, 136; online meaning construction 55–72, 122; provenance 271; single texts 118–119; Social-Interactive-Texts framework 260–261; strategic processing 7, 138, 139, 140, 144–145; surface model 224 memory 172–173, 477, 480, 531; accessibility of attitudes in 87, 88; conflicting information 187, 188–189, 191, 193; individual differences 110; knowledge activation 152–153; persuasive messages 80, 82, 91; RESOLV 22; sourcing 278; structure-building framework 305; tasks 475; text importance 174–175; think-aloud protocols 430; trait inferences 309; updating 304, 308, 309–310; validation 155, 159; see also working memory mental models 4, 35, 100, 137, 151, 344, 506, 528; coherent 169; conflicting information 186; discipline-specific strategies 232; GISA 456; instructional context 353; narratives 305; source credibility 180; strategic processing 138, 141; validation 156, 157, 159; see also Context Model; Document Model; Integrated Model; Intertext Model; Task Model message repetition 79–80, 90 metacognition 49, 62; conflicting information 192–193; engagement in reading 390; GISA 452; individual differences 103–106, 109, 110, 529; information-problem solving 408; laypeople 277; online synthesis 63–64; self-regulated learning 321–324, 327–328, 330–331, 333–334, 487; strategic processing 145; validation 155, 156, 162, 163 metacognitive-reflective processing 138, 139–140, 143–144, 146 metacomprehension 154 metadata 2, 99, 426 metatextual information 222, 224, 270, 271, 279, 325; see also metadata met.a.ware 328–329 misinformation 157–159, 184; see also false information

544  • Index modality 289 monitoring 105–106, 139; conflicting information 192–193; epistemic 7, 152, 155–156, 157, 161, 529; metacomprehension 154; self-regulated learning 322, 326–327, 329, 331; strategic processing 138, 140, 143–145; think-aloud protocols 430–431 motivation 10, 38–39, 382, 384–397; CAEM 44, 50, 51; Discrepancy Motives Model 83, 85; Elaboration Likelihood Model 82, 83; GISA 452, 456; individual differences 106–107, 110; reading 37, 61, 364; resistance of attitudes to change 89–90, 91; RESOLV 21, 31, 32; self-regulated learning 322–323, 324, 325–326, 332, 334; STARI 372 multimedia learning 121–122, 126, 128 multimodality 55, 59, 60–61, 66–67, 68, 121, 448 Multiple-Document Task-based Relevance Assessment and Content Extraction (MD-TRACE) model 6, 19, 36–37, 43, 56, 100, 343, 505; assessment 441, 447, 450–451, 472; eye movements 439; GISA 453–454, 455, 456; individual differences 111; note-taking 428–429; physiological measures 440; processing steps 426–427, 450; reading time 436; scoring systems 509; self-regulated learning 323; think-aloud protocols 434 multiple source use: assessment 425–446, 448–449, 466–484, 504, 507–508, 514–518, 521, 532; cold perspectives 35–37; conflicting information 184–201; controversial topics 151–152; core questions 528–532; definition of 2; definition of multiple source reading 134–135; digital age 320; dyslexic readers 124–128; fiction 303, 304, 316; framework for 224–225; future research 532–536; history 205–220, 341–342; individual differences 99–116; literacy skills 362–363; literature and language arts 254–266; models 35–36; motivation 389–390; non-academic use 285–302, 530–531; research on 2–4, 34; scenario-based assessment 452–463; science 221–222, 225–233, 342; self-regulated learning 321, 323–334; strategic processing 135–146; strategies 123; text relevance 175, 178–181; validation 159–163; warm perspectives 37–39; wider context 9 mush model 41, 42 National Assessment of Educational Progress (NAEP) 462, 486, 493, 496–497, 498 natural language processing (NLP) 504, 508–523 navigation 285, 287–288, 292, 297, 478, 487; assessment 474, 494; dyslexic readers 122–123, 124–126; hypertext 48; navigability 289; reading time 435–436; think-aloud protocols 431 need for cognition (NFC) 49, 89, 91, 107

New Literacies 6, 56–58, 63–65, 67–68, 72, 321, 486–487 news articles 222, 226, 228–232, 270, 295, 350 non-linear reading 223, 225, 226, 229 non-routine processing 19, 22, 23–24, 28–31, 32 note-taking 62–63, 103, 217, 425, 427–429, 441–442 Online Research and Comprehension Assessment (ORCA) 458–460, 491, 494 parsing sentences 513–514 peer review 242–243, 245–246 peers 352–353, 366, 487–488 performance moderators 452, 455, 457 persistence 37, 45, 51, 332, 364, 372, 387 personal accounts 290, 293–294 personal narratives 293–294, 296 perspective-taking 366, 369, 378, 467, 489, 495 persuasion 7, 79–91 phonological skills 120, 366 physiological measures 10, 425–426, 439–441, 442 plagiarism 357, 516–517, 519 plausibility 152, 156, 157, 161, 277–278; PlausibilityInduced Source Focusing 8, 185, 187–188, 190–192, 195; Plausibility Judgements in Conceptual Change 188; provenance information 271, 272–273, 275–277; validation 159 policy opinions 356 policy-related documents 350 postmodernism 60 power 63 prejudice 90 previewing 223, 225, 229 primacy effects 90–91, 309 primary sources 206–208, 214–217, 349, 356, 430, 530 print texts 58–59, 146 prior knowledge 49, 61, 119, 528–529, 530; assessment 469, 473; conflicting information 187–188, 192, 194, 534; evaluation of search results 291; fiction 312; GISA 455; history 208, 209, 211; individual differences 100–101, 102, 109–111; information-problem solving 402, 404, 410–411; information scent cues 292; Internet information 117; literature and language arts 258; MD-TRACE 450; multiple representations 122; provenance information 275, 276; reflective web searching 329; science 226, 227, 228–229; self-regulated learning 324, 325–326; Situation Model 152, 343; strategic processing 137, 140, 145; task manipulations 346–347, 348; text-belief consistency 160; topic knowledge 476; validation 153, 157 problem-solving 19, 110, 396, 467; assessment 468, 473; collaborative 488, 490, 492–493, 495;

Index  •  545 information-problem solving 10, 36, 286–287, 401–422, 476 processing 3, 119, 122, 321, 344; assessment 442–443, 448, 474–475, 480; CAEM 46, 51; conflicting information 189, 194; deep 36–37, 46; dyslexic readers 128; explanation-based 354; eye movements 437, 438–439; future research 533; IPS-I 36; language 383, 393; learner attributes 478; MD-TRACE 100, 426–427, 450; measures of 357, 358; note-taking 427–428; persuasive messages 80–92; provenance information 271; resistance of attitudes to change 89, 90; RESOLV 19, 20, 22–24, 27–32; self-regulated learning 323; strategic 103, 110, 111, 133–150, 161, 477; surface-level 46, 140; task environment 348; think-aloud protocols 430–431; validation 155, 156–157, 161, 162–163; see also cognitive processes product model 506 Programme for International Student Assessment (PISA) 108, 393, 409, 422, 448, 458, 492, 495 Progress in International Reading Literacy Study (PIRLS) 393, 459–460, 492, 494 Project READi 491, 494 prompts 345–348, 355–358; essay assessment 505, 507–508, 509, 520, 522; information-problem solving 411; SAIL-OCISD 497 proofs 239, 240, 241–248, 250–251, 530 provenance 269–284 purpose 57, 58–59, 70, 225 re-purposing 535 Reader/Writer-Texts framework 255, 256–259, 261–263 readers 57–58, 61–64, 70, 170–171 REading as Problem SOLVing (RESOLV) model 6, 18–32, 100; self-regulated learning 321, 323–324, 326, 328, 332–333 reading context 386–387 reading, definition of 133–134, 136 Reading Inventory and Scholastic Evaluation (RISE) 378 reading paths 138, 139, 141, 144, 324, 386 Reading Strategy Assessment Tool (RSAT) 511 reading time 171–173, 425, 434–436, 441–442, 474 reasoning 346–347, 351, 354, 366, 488; assessment 468, 473; collaborative inquiry 498; critical 379; deductive 251; social deliberation 489 recency effects 90–91, 309 refutations 310–312 regular expressions 511–512, 513, 519, 520, 523 relationships between content 520, 521–522 relevance 3, 8, 168–183, 427; algorithms 67; assessment 473; background knowledge 364; conflicting information 195; cultural 217;

historical sources 208; Internet information 36, 123, 126, 288, 291; motivation 396; reading time 435; resistance of attitudes to change 89, 90–91; RESOLV 23, 28–29 reliability 3–4, 389, 427, 477, 507; assessment 448, 473, 475, 522–523; epistemic beliefs 432; eye movements 438; historical sources 215, 255; information-problem solving 408, 419; mathematics 239–240, 246–247; note-taking 428, 429; online sources 62, 70, 126, 169, 286, 291, 478; ORCA 459; scenario-based assessment 457–458; strategic processing 137, 138, 144; text access 45; think-aloud protocols 429, 434 religion 108, 191–192 remix 66, 68 representations 17, 34–35, 100, 224, 356–357, 528; conflicting information 186, 188; integrated 473; literature and language arts 257–258; multiple 121–122, 126–127, 208–209, 257, 342–343; narratives 305–306; product model 506; RESOLV 19–21, 31–32; see also mental models reputation 242, 244–246, 248, 250 resistance to change 88, 89–91, 187 resources 100–101, 466, 529; cognitive 156, 159, 161, 331, 364; MD-TRACE 19, 427, 450, 454; RESOLV 20, 21, 31, 323; see also information resources responses to tasks 39–42 responsivity 136, 146 results-methods-conclusions coordination 229–231 RI-Val model 162 routine processing 19, 22–23, 28–29, 32, 153 SAIL Online Collaborative Inquiry and Social Deliberation in Virtual Worlds (SAIL-OCISD) 493, 497, 498, 532 SAIL Virtual World for ELA (SAIL-ELA) 493, 496–497 scaffolding 71, 110, 196, 215, 358, 443; GISA 456; history 217; information-problem solving 408, 411, 419; multiple-source curricula 365, 379; self-regulated learning 328–329, 330, 331, 334; STARI 372 scanning 36, 46, 223, 287, 291, 389, 415 scenario-based assessment (SBA) 11, 447–448, 449–450, 452–463, 494, 497, 532 science 8, 10, 221–237, 342, 356, 530; CBAL assessments 460; conflicting information 193; controversial topics 479; Evidence-Centered Design 469; information-problem solving 408–409, 419, 421; instructional context 351–353, 354; interest 390; literacy skills 379; mathematics compared with 240, 249, 250; motivation 387, 388, 389; public engagement with 269–270, 271–275, 277–280; Seeds of Science curriculum

546  • Index 364; self-regulated learning 326; summary task 25; task environment 349–350; task manipulations 347–348; topic knowledge 476; WordGen 367–372 scoring systems 451, 508–511, 521 search 50, 110; eye movements 437; informationproblem solving 36, 403, 406–408, 415–419, 422; learner attributes 478; reflective web searching 329; science information 269, 272; strategic processing 137, 139, 141; struggling readers 124–125, 128 search engine result pages (SERPs) 476–477, 478; information-problem solving 402, 408–409, 419, 420–421; non-academic use 276, 277, 287, 288, 290–291, 294–295, 296; reading time 435–436 search engines 3, 62, 65, 286, 487; algorithms 67–68; information-problem solving 287, 401, 402; multiple-text comprehension 389; ORCA 459; struggling readers 124–125, 128 secondary sources 206–208, 214, 216–217, 430, 530 Seeds of Science, Roots of Reading Curriculum 364 SEEK 352, 353, 408, 419, 435 selective exposure 294–296 self-assessment 135, 140 self-beliefs 106–107 self-determination theory 384 self-efficacy 37, 106, 193, 478; instruction 394–397; motivation 384, 385–397; self-regulated learning 323, 326, 332 self-evaluation 323 self-explanation 103, 106, 353, 431 self-monitoring 138, 139, 144 self-reflection 322, 327 self-regulation 9, 49, 100–101, 320–338, 487, 529; automaticity 383; conflicting information 193; engagement in reading 391; GISA 452; individual differences 529; MD-TRACE 450; non-academic use 531; strategic processing 143, 144 self-sufficiency 143 semantic benchmarks 508–509, 510–513, 515, 518, 522, 523 semantic spaces 512–513, 514, 520, 523 sense-making 363, 473, 474, 476 sentence parsing 513–514 shifting 125–126 single texts 17–18, 118–119, 121, 257, 343–344, 356–357, 449; conflicting information 188–189, 191, 193; GISA 455–456; task environment 348; text relevance 177–178, 180; “traditional” 146 Situation Model 18, 118–119, 152, 257, 270, 343; history 255; MD-TRACE 19; narratives 305–306; single texts 344; strategic processing 141; validation 157, 158–159, 160 skills 64, 136, 358, 502–503; assessment of 449–454, 461, 489–490, 496–497; CAEM 51; digital

reading 124; Evidence-Centered Design 467; information-problem solving 36, 401–402, 403, 406–412; learner attributes 473; literacy 362–363, 366; MD-TRACE 19; metacognitive 103, 105–106; phonological 120, 366; social deliberation 489; strategy knowledge 100–101; transfer of 404, 409, 410, 412, 421 skimming 29, 30, 62, 229, 232, 389 SNIF-ACT 2.0 model 288, 292, 297 social constructivism 461 social curation 66, 67 social deliberation 11, 486, 488–489, 490, 493, 496–499 social interaction 286, 461, 487, 488 Social-Interactive-Texts framework 255, 259–263 social media 47, 279, 294, 320, 333, 411; informationproblem solving 402; science 222, 232; selfregulated learning 322; synthesis 65–67 social practices 71, 255–256, 262, 263 social psychology 7, 79–95, 309 socio-cognitive theory 384 socio-cultural differences 107–109 socioeconomic status (SES) 90, 108 source, definition of a 2, 35, 99, 118, 184, 206, 239, 270, 343, 426 source evaluation 36–37, 40, 42, 344; C3 Framework 212–213; CAEM 45, 46–47, 49, 51; conflicting information 190, 196; epistemic beliefs 412, 432; eye movements 438; GISA 453; informationproblem solving 402, 408–411, 420–421; instructional context 351–353; laypeople 279–280; literature and language arts 257; note-taking 429; online sources 123–124, 126, 286, 287, 288–289, 292–294; reading time 435–436; science 223; self-regulated learning 327; strategic processing 142; student profiles 48; task environment 349, 350; text relevance 169, 178, 180; think-aloud protocols 430–431 source judgement 138, 139, 142, 143 Sourcer’s Apprentice 348, 352, 405, 409, 422 sourcing 3–4, 91, 123, 134, 151, 321, 358, 466; assessment 469, 471, 472–473, 507, 522; conflicting information 189, 194–195, 196; definition of 184, 206, 426; epistemic beliefs 192; epistemic cognition 433; history 209, 210–212, 215, 217, 349, 356; individual differences 103; information-problem solving 402, 407, 422; instructional context 351–352; mathematics 238–239, 240, 241–251, 530; measures of 344; provenance information 269, 271, 278, 279; science 228, 240, 356; self-regulated learning 327; socio-cultural identities 108; strategic processing 137, 139, 142, 143; task environment 348, 355; task manipulations 346; think-aloud protocols 431

Index  •  547 spelling 122–123, 124–125 stereotyping 90 storytelling 259–261 Strategic Adolescent Reading Intervention (STARI) 362, 365–367, 372–379 strategic processing 7, 103, 110, 111, 133–150, 161, 477 strategies 123, 134–135, 389; assessment 468, 481; definition of 135–136; epistemic 105, 111; GISA 452; information-problem solving 420; learner attributes 473; metacognitive 162, 163; online synthesis 61–64; ORCA 458; science 232, 272–274; self-regulated learning 323, 324, 326–327, 331, 333; strategy knowledge 100–101; think-aloud protocols 430 Stroop paradigm 153–154 structure-building framework 305–306, 307 Student Model 470–471, 474, 479, 490 subject-matter knowledge 119 summary tasks 109, 157, 161; conflicting information 194; RESOLV 24–25, 29–30; scenario-based assessment 456, 461–462; task manipulations 345–348 Support Vector Machines (SVMs) 514, 515 surface model 224, 227, 232 synthesis 57–58, 68–72, 321; assessment 471, 472; GISA 453; multiple-text comprehension 389; online 56, 59, 60, 61–64, 66–67, 487 task-centered approach 404, 407, 410, 411 task definition 324–325 task environment 348–350, 355, 531 task instructions 170, 171–173, 193–194, 469, 476, 518, 534 task manipulations 344–347, 355 Task Model 257, 355, 480, 506; assessment methods 441; Evidence-Centered Design 470–471, 474, 479, 490; GISA 454–455; individual differences 358; MD-TRACE 36, 43, 343, 426, 450, 505; RESOLV 19, 21, 22–23, 26–29, 323; science 224, 225, 227, 232; self-regulated learning 325–326 task-oriented reading 505 teachers 50, 214, 218, 254, 363, 379 technologies 57–58, 64–67, 70, 490 text 57–58, 59–61, 70, 133–134; cognitive processes 383; definition of a 35; dimensions 305; literature and language arts 254; online 55; Reader/Writer-Texts framework 255, 256–259, 261–263; relevance 168–183; Social-InteractiveTexts framework 255, 259–263; text-visual coordination 231–232 text access 45–46, 49 text accessibility 365 text-belief consistency 103, 152, 155–163, 171, 180, 327–328, 477, 529

text importance 173–175 text relevance see relevance Textbase 18, 21, 118, 257, 258, 343, 364 textual analysis 138, 139, 142, 143 theory 250, 425, 442, 504, 505–506 think-aloud protocols 146, 357, 425, 429–434, 441–442, 474, 480–481, 533; automatic assessment 523; CAEM 45–46, 48, 51; conflicting information 189–190, 191–192; eye-tracking studies 437; history 210–211, 214, 215; metacognitive skills 105; NLP tools 517–518; science 222–223, 224; scoring 511; strategic processing 103; task goals 534; text relevance 171 topic knowledge 119, 476 trait models 306–307, 311, 314–315 trustworthiness 49, 135, 151, 475, 477; epistemic strategies 105; GISA 453; historical sources 208, 210, 255, 351; information-problem solving 401–402, 407, 409, 411, 415, 419; learner attributes 478; mathematics 530; online sources 288; provenance information 275, 276; reading time 435; science 228, 273, 274, 356, 530; search engine results 291; source credibility 175, 179– 180, 313, 314; strategic processing 137, 138, 144; student responses 40, 41–42; text access 45–46; think-aloud protocols 432; web page evaluation 292–293 truth 272, 273 Twitter 41–42, 65–66 updating 9, 304, 308–315, 316 usefulness 178–179, 276, 288, 351, 396, 475, 477, 478 user comments 295 validation 3, 7, 82–83, 92, 138, 151–167 validity 142, 153, 273, 449, 459 values 107–108, 145, 226, 250, 322, 326, 333 valuing 384–385, 386–396, 397 Vector Space Models (VSM) 512 verbalization see think-aloud protocols visual information 227, 231–232, 438–439 vocabulary 119, 120, 365, 378, 456–457 Web-Based Inquiry Science Environment (WISE) 408, 419 web page evaluation 292–293 websites 65, 118, 291, 487; assessment 494; eye-tracking studies 437–438; GISA 456; information-problem solving 408–409, 417–418, 420–421; multiple-text comprehension 389; reading time 435–436; selective exposure 296; self-regulated learning 328–329; trustworthiness 105, 432–433; see also Internet whole tasks 404, 409, 411 Wikipedia 40, 438

548  • Index Word Generation (WordGen) 362, 365–372, 378–379 word recognition 119, 364, 378, 382, 473; automaticity 383; CORI 395; dyslexia 119–120; engagement in reading 391; individual differences 101, 324; motivation 387, 391, 392

working memory 101–102, 119, 121, 382; alignment hypothesis 386; conflicting information 192; dyslexic readers 120, 122–124, 125, 127; instruction 394; learner attributes 475; motivation 389, 391, 393; self-regulated learning 331; validation 155; verbalization 429