Assessing Student Learning: A Common Sense Guide [Third Edition] 9781119426929, 1119426928

Citation preview

ASSESSING

student LEARNING

ASSESSING

student LEARNING A Common Sense Guide THIRD EDITION

Linda Suskie



Copyright © 2018 by John Wiley & Sons, Inc. All rights reserved. Portions of Chapter 15 were previously published by Routledge. Copyright © 2017 from Handbook on Measurement, Assessment, and Evaluation in Higher Education by Charles Secolsky and D. Brian Denison. Reproduced by permission of Taylor and Francis Group, LLC, a division of Informa plc. Published by Jossey-Bass A Wiley Brand One Montgomery Street, Suite 1000, San Francisco, CA 94104-4594—www.josseybass.com No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax 978-646-8600, or on the Web at www.copyright.com. Requests to the publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax 201-748-6008, or online at www.wiley.com/go/ permissions. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. Readers should be aware that Internet Web sites offered as citations and/or sources for further information may have changed or disappeared between the time this was written and when it is read. Jossey-Bass books and products are available through most bookstores. To contact Jossey-Bass directly call our Customer Care Department within the U.S. at 800-956-7739, outside the U.S. at 317-572-3986, or fax 317-572-4002. Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-ondemand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com. Library of Congress Cataloging-in-Publication Data Names: Suskie, Linda, author. Title: Assessing student learning : a common sense guide / by Linda Suskie. Description: Third edition. | San Francisco, CA : Jossey-Bass, [2018] | Includes bibliographical references and index. Identifiers: LCCN 2017042567 | ISBN 9781119426868 (pdf) | ISBN 9781119426929 (epub) | ISBN 9781119426936 (pbk.) Subjects: LCSH: College students—Rating of. | Educational tests and measurements. Classification: LCC LB2336 .S87 2018 | DDC 378.1/662—dc23 LC record available at https://lccn.loc.gov/2017042567 Cover Design: Wiley Printed in the United States of America third edition PB Printing

10 9 8 7 6 5 4 3 2 1

Contents List of Tables

vii

List of Lists

ix

List of Figure List of Exhibits Preface to the Third Edition

xiii xv xix

Introduction 1

Part 1

Understanding Assessment Chapter 1  What Is Assessment?

7

Chapter 2  The Many Settings for Student Learning and Assessment

15

Chapter 3  What Are Effective Assessment Practices?

23

Part 2 Laying a Foundation for Assessment Success Chapter 4  Learning Goals: A ­ rticulating What You Most Want ­ Students to Learn

39

Chapter 5  Designing Curricula to Help Students Learn What’s ­Important

63

Chapter 6  How Will Your Evidence of S­ tudent Learning be Used?

85

Chapter 7  Planning Assessments in Academic Programs

93

Chapter 8  Planning Assessments in General Education, Co-curricula, and Other Settings

105

Part 3 Building a Pervasive, Enduring Culture of Evidence and Betterment Chapter 9  Guiding and Coordinating Assessment Efforts

117

Chapter 10  Helping Everyone Learn What to Do

127

Chapter 11  Supporting Assessment Efforts

139 v

Contents

Part 4

Chapter 12  Keeping Assessment Cost-Effective

149

Chapter 13  Collaborating on Assessment

165

Chapter 14  Valuing Assessment and the People Who Contribute

175

The Assessment Toolbox Chapter 15  Designing Rubrics to Plan and Assess Assignments

189

Chapter 16  Creating Effective Assignments

205

Chapter 17  Writing Multiple-Choice and Other Objective Tests

215

Chapter 18  Assembling Evidence of S­ tudent Learning into Portfolios

235

Chapter 19  Selecting Published Instruments

245

Chapter 20  Other Assessment Tools

259

Chapter 21  Assessing the Hard-to-Assess

273

Part 5 Understanding and Using ­Evidence of Student Learning Chapter 22  Setting Meaningful Standards and Targets

287

Chapter 23  Summarizing and Storing Evidence of Student Learning305 Chapter 24  Analyzing Evidence of Student Learning

319

Chapter 25  Sharing Evidence of Student Learning

333

Chapter 26  Using Evidence of S­ tudent Learning to Inform Important Decisions

355

References 367 Index 377

vi

List of Tables Table 1.1

Traditional Versus Contemporary Ways of Thinking About Assessment 9

Table 2.1

Relevant Chapters of This Book for Each Learning Setting

20

Table 4.1

Examples of Effectively Expressed Learning Goals

58

Table 7.1

Common Assessment Tools

96

Table 12.1 Error Margins of Various Sample Sizes

159

Table 12.2 Sample Sizes Needed from Small Groups for a 5% Error Margin

160

Table 14.1 Common Root Causes of Foot-Dragging on Assessment

177

Table 16.1 Examples of Learning Assessment Techniques

209

Table 20.1 Student Definitions of Leadership Before and After Participating in a Leadership Development Program

263

Table 20.2 Examples of Rating Scales

265

Table 22.1 Perspectives for Comparing Evidence of Student Learning

296

Table 23.1 A Tally of Assessments of Students by Their Peers

308

Table 23.2 A Tally of Assessments of Students by Their Peers Presented with Percentages

309

Table 23.3 Biology Test Outcomes Mapped Back to the Test Blueprint

312

Table 24.1 How to Document Evidence of Key Traits of Effective Assessment Practices 326 Table 24.2 Examples of Item Discrimination Results

329

Table 24.3 Selected Outcomes from the National Survey of Student Engagement for Rodney College Seniors

331

vii

List of Lists List 1.1

The Four-Step Teaching-Learning-Assessment Process

8

List 3.1

Statements of Principles of Good Assessment Practice

24

List 3.2

Steps to Ensure That Evidence of Student Learning Is Useful and Used

25

List 3.3

Examples of Direct Evidence of Student Learning

26

List 3.4

Examples of Indirect Evidence of Student Learning

27

List 3.5

Examples of Assessment Errors and Biases

33

List 3.6

Strategies to Minimize Assessment Errors and Biases

33

List 4.1

Habits of Mind

45

List 4.2

Transferrable Skills Valued by Employers

46

List 4.3

Resources for Identifying Potential Learning Goals

51

List 4.4

Taxonomies for Learning Goals

52

List 4.5

Examples of Discussion Topics Regarding Learning Goals

54

List 5.1

Information That Can Be Used to Help Ensure That a Curriculum Is Responsive

66

List 5.2

Strategies to Add More Intensive Study of a Key Learning Goal

81

List 5.3

Strategies to Improve a Program or General Education Curriculum 83

List 7.1

Questions to Address in an Assessment Plan

98

List 7.2

Questions to Ask About Assessment Efforts

103

List 8.1

Suggestions for Assessing General Education Learning Goals

109

List 9.1

Examples of the Benefits of Assessment

119

List 10.1

Seminal Books on Assessing Student Learning

128

List 10.2

Journals and Other Publications That Address Assessing Student Learning in Higher Education

129

List 11.1

Assessment Tasks That Technologies Can Help With

143

List 11.2

Questions to Ask Vendor References About Assessment Technologies 146

List 11.3

Potential Assessment Resource Needs

146 ix

List of Lists

x

List 12.1

Where to Look for Assessment Ideas

151

List 12.2

Examples of Evidence of Student Learning That May Already Be on Hand

152

List 12.3

Is It Worth Taking Extra Steps to Minimize Assessment Errors and Biases and Increase Consistency?

156

List 12.4

Four Situations Where Samples May Make Sense

158

List 13.1

Strategies to Involve Part-Time Adjunct Faculty in Assessment

168

List 13.2

Examples of How Students Can Engage in Assessment

170

List 13.3

The Delphi Method for Achieving Consensus on Key Learning Goals

173

List 14.1

Strategies to Value, Respect, and Reward Efforts to Improve Teaching 180

List 14.2

Incentives and Rewards That Recognize and Honor Assessment Efforts

181

List 14.3

How College Leaders Can Foster a Culture of Assessment

184

List 15.1

Benefits of Well-Crafted Rubrics

195

List 15.2

Questions to Help Identify What You’re Looking for in Student Work

198

List 15.3

Examples of Continuums for Rubric Performance Levels

200

List 16.1

Examples of Assignments Beyond Essays, Term Papers, and Research Reports

208

List 16.2

Questions to Address in a Prompt for an Assignment

212

List 16.3

Strategies to Counter Plagiarism

213

List 17.1

Tips for Writing Challenging Rather Than Trick Questions

222

List 17.2

Tips for Writing Good Multiple-Choice Questions

223

List 17.3

Tips for Writing Good Interpretive Exercises

227

List 17.4

Tips for Writing Good Matching Items

229

List 17.5

Tips for Writing Good True-False Items

231

List 17.6

Tips for Writing Good Completion or Fill-in-the-Blank Items

232

List 17.7

Information to Include in Test Directions

233

List 18.1

Why Use Portfolios?

237

List 18.2

Questions to Consider as You Plan a Portfolio Assessment

238

List 18.3

Examples of Items That Might Be Included in a Portfolio

239

List 18.4

Suggestions to Keep Portfolio Assessments Manageable

240

Questions to Address in Portfolio Guidelines to Students

240

List 18.6

Examples of Prompts for Student Reflection on a Portfolio

241

List 19.1

Examples of Published Instruments for Student Learning Assessment in Higher Education

247

List 19.2

Resources for Identifying Potential Published Instruments

249

List 19.3

Useful Information on Potential Published Instruments

250

List 19.4

Questions to Ask About Instruments with Little Published Information 255

List 19.5

Deciding If a Published Instrument Is Right for You

List 20.1

Examples of Prompts for Reflection on a Learning Experience262

List 20.2

Tips for Focus Groups and Interviews

270

List 21.1

Factors Affecting Participation in Add-On Assessments

280

List 22.1

Resources for Identifying Potential Peer Colleges

292

List 22.2

Sources of External Insight on Potential Standards

299

List 23.1

Models for Calculating Overall Rubric Scores

309

List 23.2

A Summary of Qualitative Participant Feedback on an Assessment Workshop 315

List 24.1

How to Determine the Discrimination of Test Items

329

List 25.1

Formats for Sharing Summaries and Analyses of Student Learning Evidence

337

List 25.2

Venues for Announcing and Sharing Summaries and Analyses of Student Learning Evidence

347

List 25.3

Tips to Engage Audiences in Face-to-Face Discussions of Student Learning Evidence

352

List 26.1

Strategies That Help College Students Learn

358

List 26.2

Using Student Learning Evidence Fairly, Ethically, and Responsibly 364

List of Lists

List 18.5

257

xi

List of Figure Figure 1.1 Teaching, Learning, and Assessment as a Continuous Four-Step Cycle

9

xiii

List of Exhibits Exhibit 5.1

Template for a Three-Column Curriculum Map for a Course Syllabus 77

Exhibit 5.2

Template for a Four-Column Curriculum Map for a Course Syllabus 78

Exhibit 5.3

Curriculum Map for a Hypothetical Certificate Program

Exhibit 7.1

An Example of a Completed Chart for Monitoring Assessment Progress Across a College

100

Exhibit 7.2

A Rating Scale Rubric for Evaluating College-Wide Student Learning Assessment Processes

102

79

Exhibit 10.1 A Rubric for Providing Feedback on Assessment Plans and Reports

134

Exhibit 10.2 A Template for an Annual Program Assessment Report

136

Exhibit 15.1 A Checklist for Safe Culinary Practices

190

Exhibit 15.2 A Structured Observation Guide for a One-Act Play

193

Exhibit 15.3 A Student Essay on Making Community Service a Graduation Requirement 204 Exhibit 17.1 A Test Blueprint for an Exam in an Educational Research Methods Course

219

Exhibit 17.2 Multiple-Choice Questions on Assessment Concepts

225

Exhibit 17.3 An Example of an Interpretive Exercise

227

Exhibit 17.4 Matching Items from a Nursing Research Methods Test

230

Exhibit 18.1 A Reflection Sheet for Individual Portfolio Items from a Graduate Course on Assessment Methods

241

Exhibit 18.2 A Rubric for Assessing Portfolios from a Graduate Course on Assessment Methods

242

Exhibit 20.1 A Prompt for a Reflective Paper on an Internship

264

Exhibit 20.2 An Exit Survey for Students Completing a BS in Computer Information Systems

267

Exhibit 20.3 A Self-Assessment of Library Skills, with a (Fictitious) Student’s Responses

268 xv

List of Exhibits

xvi

Exhibit 21.1 A Rating Scale Rubric for Assessing Fellow Group Members

279

Exhibit 22.1 Rubric Results for a Hypothetical Assessment of Written Communication Skills

289

Exhibit 23.1 A Scored Rubric for a Research Report in Speech-Language Pathology/Audiology 311 Exhibit 25.1 A Poorly Designed Table

343

Exhibit 25.2 An Improved Version of the Table in Exhibit 25.1

344

Exhibit 25.3 Results of a Rubric Assessing Writing

345

Exhibit 25.4 A Pie Graph of the Qualitative Feedback in List 23.2

346

To my husband Steve, for his unflagging support for everything I’ve done, including this book To our children, Melissa and Michael And to everyone in higher education who believes, as I do, that one of the answers to today’s problems is for everyone to get the best possible education

xvii

Preface to the Third Edition W

hen Jim Anker, the publisher of the first edition of this book, approached me about writing a second edition in 2008, I figured that I’d update the references and a few chapters and be done. The first edition was based, after all, on an enduring common sense approach to assessment that hadn’t changed materially since the first edition was published in 2004. Ha! I ended up doing a complete reorganization and rewrite. Fast-forward to 2017. With the second edition now eight years old, it was clearly time for an update. But the second edition had been very successful, so I again figured I’d update the references and a few chapters and be done. Ha! Once again, this is a complete reorganization and rewrite of the previous edition. Why the rewrite? As I started work on this edition, I was immediately struck by how outdated the second edition had become in just a few short years. When I wrote the second edition, AAC&U’s VALUE rubrics were largely untested, the National Institute for Learning Outcomes Assessment was just getting started, and the Degree Qualifications Profile didn’t exist in the United States. Learning management systems and assessment information management systems were nowhere near as prevalent or sophisticated as they are today. More broadly, the higher education community has largely moved from getting started with assessment to doing assessment. But a lot of assessment to date hasn’t been done very well, so now we’re starting to move from doing assessment to doing it meaningfully. Truly meaningful assessment remains a challenge, and this third edition aims to address that challenge in the following ways: An increased emphasis on useful assessment. In earlier editions, I placed a chapter on using assessment results at the end, which made chronological sense. But many faculty and administrators still struggle to grasp that assessment is all about improving how we help students learn, not an end in itself, and that assessments should be planned with likely uses in mind. So I have added a second chapter on using assessment results to the beginning of the book. And throughout the book I talk not about “assessment results” but about “evidence of student learning,” which is what this is really all about. Greater attention to building a culture in which assessment is useful and used. Getting colleagues on board remains a stubborn issue. Two chapters on this in the second edition have been expanded to six, including new chapters on guiding and xix

Preface to the Third Edition

coordinating assessment, helping everyone learn what to do, keeping assessment cost-effective, and making assessment collaborative. An enhanced focus on the many settings of assessment, especially general education and co-curricula. Faculty and administrators are looking for more guidance on how to assess student learning in specific settings such as the classroom, general education curricula, undergraduate and graduate programs, and co-curricular experiences. The second edition provided little of this guidance and, indeed, did not draw many distinctions in assessment across these settings. A thorough treatment of assessment in each setting is beyond the scope of this book, of course. But this edition features a new chapter on the many settings of assessment, and several chapters now include discussions on applying the concepts in them to specific settings. Call-out boxes to introduce assessment vocabulary. The jargon of assessment continues to put off faculty and staff as well as graduate students who use this as a textbook. I opened the second edition with a chapter that was essentially a glossary but overwhelmed graduate students. In my 2014 book, Five Dimensions of Quality: A Common Sense Guide to Accreditation and Accountability, I introduced higher education vocabulary with call-out boxes called Jargon Alerts. That feature was well received, so in this edition I’ve eliminated the glossary chapter and instead sprinkled Jargon Alert boxes throughout. A new focus on synthesizing evidence of student learning into an overall picture of an integrated learning experience. Assessment committees and administrators today are often inundated with assessment reports from programs and units, and they struggle to integrate them into an overall picture of student learning, in part because learning itself is not yet an integrated experience. The idea that assessment should be part of an integrated learning experience is now a theme addressed throughout the book. More immediate attention to learning goals. The second edition discussed learning goals about a third of the way through the book. Because learning goals are the drivers of meaningful assessment, the chapter on them now appears earlier in the book. A new chapter on curriculum design. One of the major barriers to effective assessment and the use of student learning evidence is poor curriculum design, so I’ve added a whole new chapter on this. More thorough information on planning assessment processes. There are now two chapters instead of one. The first one, on planning assessments of program learning goals, provides a framework, and the second one applies that framework to planning assessments in general education, co-curricula, and other settings.

xx

A new chapter on assessing the hard-to-assess. The former chapter on assessing attitudes and values is now two chapters – one on miscellaneous assessment tools and one on how to assess the hard-to-assess.

Preface to the Third Edition

New frameworks for rubric design and setting standards and targets. In 2016 I researched and wrote a chapter, “Rubric Development,” for the second edition of the Handbook on Assessment, Measurement, and Evaluation in Higher Education (Secolsky & Denison, 2017). My research changed my thinking on what an effective rubric looks like, how it should be developed, and how standards and targets should be set.

New resources. Many new assessment resources have emerged since the second edition was published, including books, models, published instruments, technologies, and research. Perhaps the most important new resources are the widely used VALUE rubrics published by the Association of American Colleges & Universities (www.aacu.org) and the many white papers published by the National Institute for Learning Outcomes Assessment (www.learningoutcomesassessment .org). This edition introduces readers to these and other valuable new resources. And, yes, I did update the references as I originally envisioned!

xxi

Introduction I

nterest in assessing student learning at colleges and universities – and the need to learn how to do it – skyrocketed in the last two decades of the twentieth century and continues to grow in the twenty-first century. The higher education community is increasingly committed to creating learning-centered environments in which faculty and staff work actively to help students learn, and the assessment of student learning is essential to understanding and gauging the success of these efforts. In the United States and elsewhere, accreditors and other quality assurance agencies require colleges and academic programs to assess how well students are achieving key learning goals. These trends have created a need for straightforward, sensible guidance on how to assess student learning.

Purpose and intended audience Many years ago, someone commented on the value of my workshops to the “But how do we do it?” crowd. That phrase has stayed with me, and it is the root of this book. Yes, we in higher education are theorists and scholars, with an inherent interest in whys and wherefores, but there are times when all we need and want is simple, practical advice on how to do our jobs. Providing that advice is the purpose of this book. Assessing Student Learning: A Common Sense Guide is designed to summarize current thinking on the practice of assessment in a comprehensive, accessible, and useful fashion for those without formal experience in assessing student learning. Short on background and theory and long on practical advice, this is a plainspoken, informally written book designed to provide sensible guidance on virtually all aspects of assessment to four audiences: Assessment newcomers, experienced assessment practitioners, faculty and administrators involved in student learning, and students in graduate courses on higher education assessment.

Scope and treatment: A common sense approach to assessment This book is called A Common Sense Guide because its premise is that effective assessment is based on simple, common sense principles. Because each college and learning experience is unique and therefore requires a somewhat unique approach to assessment, this book presents readers not with a prescriptive cookbook

1

Introduction

approach but with well-informed principles and options that they can select and adapt to their own circumstances. This book is also based on common sense in that it recognizes that most faculty do not want to spend an excessive amount of time on assessment and are not interested in generating scholarly research from their assessment activities. The book therefore sets realistic rather than scholarly standards for good practice. It does not expect faculty to conduct extensive validation studies of the tests they write, for example, but it does expect faculty to take reasonable steps to ensure that their tests are of sufficient quality to generate fair and useful evidence of student learning, and it provides very practical suggestions on how to do that. This book also minimizes the use of educational and psychometric jargon. For instance, while it discusses reliability and validity, it avoids using those terms as much as possible. Jargon Alert boxes are sprinkled throughout to help readers understand the vocabulary of higher education assessment. This book is also unique in its comprehensive scope, although it is not (as my husband reminded me when I was in despair over ever finishing the first edition) an encyclopedia. If you’d like to learn more, every chapter cites additional resources to explore.

Using this book

2

Assessment in higher education is still a nascent discipline. The science of educational testing and measurement is little more than a century old, and many of the ideas and concepts presented here have been developed only within the last few decades. Assessment scholars and practitioners still lack a common vocabulary or a widely accepted definition of what constitutes good or best assessment practices. As a result, a few may disagree with some of the ideas expressed here. As you hear conflicting ideas, use your own best judgment – your common sense, if you will – to decide what’s best for your situation. Assessment newcomers who want to gain a general understanding of all aspects of assessment will find that the book’s five parts take them sequentially through the assessment process: Understanding assessment, planning the assessment process, getting everyone on board, choosing and developing appropriate tools, and understanding and using student learning evidence. More experienced assessment practitioners will find the book a helpful reference guide. Plenty of headings, lists, tables, and cross-references, along with a thorough index, will help them find answers quickly to whatever questions they have. Anyone involved in student learning, including faculty who simply want to improve assessments within their classes or student development staff who want to improve assessments in their co-curricular experiences, will find much of the book

Introduction

of interest. Table 2.1 in Chapter 2 suggests especially relevant chapters for various assessment settings. Faculty and staff teaching professional development workshops and graduate courses in assessment will find this book a useful textbook or resource. Each chapter concludes with questions and exercises for thought, discussion, and practice. No answer key is provided, because these are mostly complex questions with no simple answers! Often the conversation leading to the answers will reinforce learning more than the answers themselves.

Acknowledgments Some of the material in this book is adapted from my earlier book, Questionnaire Survey Research: What Works (Suskie, 1996), published by the Association for Institutional Research. I am grateful to the Association for permission to adapt this material. Some material in Chapter 15 is adapted from my book chapter “Rubric Development” in the Handbook on Measurement, Assessment, and Evaluation in Higher Education (Secolsky & Denison, 2017), by permission of Taylor and Francis Group, LLC, a division of Informa plc. I am grateful to Taylor and Francis for permission to adapt this material. I also thank my daughter Melissa for writing the deliberately less-than-sterling essay in Exhibit 15.3. This book would not be in your hands without the work, support, and input of many people, including assessment practitioners and scholars, faculty and staff. Over the last 15 years, I have worked with literally thousands of faculty and staff at colleges and universities across the United States and throughout the world. Their questions, comments, thoughts, and ideas have pushed me to research and reflect on issues beyond those in the second edition, and this led to much of the new material in this edition. When I was contemplating this third edition, I emailed several hundred colleagues for their thoughts and ideas on a new edition, and dozens responded with extraordinarily thoughtful input. Assessment people are the nicest, friendliest, and most supportive people in the world! I particularly want to acknowledge, with deep gratitude, Elizabeth Barkley, Cynthia Howell, Claire Major, and Susan Wood, who reviewed drafts of the entire manuscript and offered wise counsel and suggestions. Altogether I am incredibly grateful to these unsung colleagues for their willingness to share so much with me.

About the author Linda Suskie is an internationally recognized consultant, writer, speaker, and educator on a broad variety of higher education assessment and accreditation topics. Her most recent book is Five Dimensions of Quality: A Common Sense Guide

3

Introduction

4

to Accreditation and Accountability (2014). Her experience includes serving as a Vice President at the Middle States Commission on Higher Education, Associate Vice President for Assessment & Institutional Research at Towson University, and Director of the American Association for Higher Education’s Assessment Forum. Her more than 40 years of experience in higher education administration include work in accreditation, assessment, institutional research, strategic planning, and quality management, and she has been active in numerous professional organizations. Linda has taught graduate courses in assessment and educational research methods, as well as undergraduate courses in writing, statistics, and developmental mathematics. She holds a bachelor’s degree in quantitative studies from Johns Hopkins University and a master’s in educational measurement and statistics from the University of Iowa.

Part 1

Understanding Assessment

Some valuable ideas you’ll find in this chapter Assessment is simply deciding what we want students to learn and making sure they learn it.

CHAPTER 1

What Is Assessment?

Assessment is a cousin of traditional empirical research. Assessment today is based on research on effective teaching strategies in higher education.

7

W

Understanding Assessment

hile the term assessment can be used broadly – we can assess the achievement of any goal or outcome – in this book, the term generally refers to the assessment of student learning. Many assessment practitioners have put forth definitions of student learning assessment, but the best one I’ve heard is in the Jargon Alert box. It’s from Dr. Jane Wolfson, a professor of biological Jargon Alert! sciences at Towson University (personal Assessment communication, n.d.). It suggests that Assessment is deciding what we want our students student learning assessment has three to learn and making sure they learn it. fundamental traits. 1. We have evidence of how well our students are achieving our key learning goals. 2. The quality of that evidence is good enough that we can use it to inform important decisions, especially regarding helping students learn. 3. We use that evidence not only to assess the achievement of individual students but also to reflect on what we are doing and, if warranted, change what we’re doing.

Assessment is part of teaching and learning Assessment is part of a four-step process of helping students learn (List 1.1). These four steps do not represent a one-and-done process but a continuous four-step cycle (Figure 1.1). In the fourth step, evidence of student learning is used to review and possibly revise approaches to the other three steps (see Jargon Alert on closing the loop), and the cycle begins anew. List 1.1  The Four-Step Teaching-Learning-Assessment Process 1. 2. 3. 4.

Establish clear, observable expected goals for student learning Ensure that students have sufficient opportunities to achieve those goals Systematically gather, analyze, and interpret evidence of how well student learning meets those goals Use the resulting information to understand and improve student learning

Jargon Alert! Closing the Loop Closing the loop is the fourth step of the teachinglearning-assessment cycle. In the fourth step, evidence of student learning is used to understand and improve student learning by improving the other steps in the cycle: Establishing learning goals, ensuring sufficient learning opportunities, and assessing learning.

8

If the cycle in Figure 1.1 looks familiar to you, it’s the Plan-Do-Check-Act cycle of business quality improvement popularized by Deming (2000): Plan a process, do or carry out the process, check how well the process is working, and act on the information obtained during the Check step to decide on improvements to the process, as appropriate.

1. Establish Learning Goals

3. Assess Student Learning

What Is Assessment?

2. Provide Learning Opportunities

4. Use the Results

CHAPTER 1

Figure 1.1:  Teaching, Learning, and Assessment as a Continuous Four-Step Cycle

Comparing traditional and current approaches to assessment Faculty have been assessing student learning for centuries, often through written and oral examinations. How do today’s approaches to assessment differ from traditional approaches? Table 1.1 summarizes some key differences between traditional and contemporary ways of thinking about assessment. Table 1.1:  Traditional Versus Contemporary Ways of Thinking About Assessment Traditional Approaches: Assessment is. . .

Contemporary Approaches: Assessment is. . .

Planned and implemented without consideration of learning goals, if any even exist

Carefully aligned with learning goals: The most important things we want students to learn (Chapter 4)

Often focused on memorized knowledge

Focused on thinking and performance skills (Chapter 4)

Often poor quality, simply because faculty and staff have had few formal opportunities to learn how to design and use effective assessment strategies and tools

Developed from research and best practices on teaching and assessment methodologies (Chapters 3 and 26)

Used only to assess and grade individual students, Used to improve teaching, learning, and student with decisions about changes to curricula success as well as to assign grades and otherwise and pedagogies often based on hunches and assess individual students (Chapters 6 and 26) anecdotes rather than solid evidence Used only in individual course sections; not connected to anything else

Viewed as part of an integrated, collaborative learning experience (Chapter 2)

Not used to tell the story of our successes; stories are told through anecdotes about star students rather than broader evidence from representative students

Used to tell our story: What makes our college or program distinctive and how successful we are in meeting societal and student needs (Chapter 25)

9

Understanding Assessment

Comparing assessment and grading Obviously there is a great deal of overlap between the tasks of grading and assessment, as both aim to identify what students have learned. There are two key differences, however. The first is that the grading process is usually isolated, involving only an individual faculty member and an individual student. Assessment, in contrast, focuses on entire cohorts of students, and it often considers how effectively many people, not just an individual faculty member, are collectively helping them learn. The second difference between grading and assessment is that they have different purposes. The main purpose of grades is to give feedback to individual students, while assessment has three broader purposes discussed in Chapter 6: Ensuring and improving educational quality, stewardship, and accountability. Grades alone are usually insufficient to achieve these purposes for several reasons. Grades alone do not usually provide meaningful information on exactly what ­students have and haven’t learned.  We can conclude from a grade of B in an organic chemistry course, for example, that the student has probably learned a good deal about organic chemistry. But that grade alone cannot tell us exactly what aspects of organic chemistry she has and has not mastered. Grading and assessment criteria may differ.  Some faculty base grades not only on evidence of what students have learned, such as tests, papers, presentations, and projects, but also on student behaviors that may or may not be related to course learning goals. Some faculty, for example, count class attendance toward a final course grade, even though students with poor attendance might nonetheless master course learning goals. Others count class participation toward the final grade, even though oral communication skills aren’t a course learning goal. Some f­ aculty ­downgrade assignments that are turned in late. Under these grading p ­ ractices, ­students who do not achieve major learning goals might nonetheless earn a fairly high grade by playing by the rules and fulfilling other less-important grading criteria. ­Conversely, students who achieve a course’s major learning goals might nonetheless earn a poor grade if they fail to do the other things expected of them. To better sync grading and assessment criteria, add a professionalism learning goal (Chapter 4) or develop a competency-based curriculum (Chapter 5). Grading standards may be vague or inconsistent.  While many faculty base ­assignment and course grades on carefully conceived learning goals and s­ tandards, others may base grades on inconsistent, imprecise, and idiosyncratic criteria. ­Faculty may say they want students to learn how to think critically, for example, but base grades largely on tests emphasizing factual recall. Faculty teaching sections

10

What Is Assessment?

Grades do not reflect all learning experiences.  Grades give us information on student performance in individual courses or course assignments (Association of American Colleges & Universities, 2002), but they do not provide information on how well students have learned key competencies, such as critical thinking or writing skills, over an entire program. Grades also do not tell us what students have learned from ungraded co-curricular experiences.

CHAPTER 1

of the same course may not agree on common standards and may therefore award ­different grades to similar student performance. Sometimes individual grading standards are so vague that a faculty member might, in theory, award an A to a ­student’s work one day and a B to identical work a week later.

Do grades have a place in an assessment effort?  Of course they do! Grades can be useful, albeit indirect (Chapter 3), and therefore insufficient evidence of s­ tudent learning. Although grades are often too holistic to yield useful information on strengths and weaknesses in student learning, they can be a good starting point for identifying potential areas of concern. DFIW rates – the proportions of s­ tudents earning a D, F, Incomplete, or Withdrawal in a course – can identify potential ­barriers to student success.

Grades can be especially useful if courses, assignments, and learning activities are purposefully designed to help students achieve key learning goals (Chapters 5 and 16) by using tools such as test blueprints (Chapter 17) or rubrics (Chapter 15).

Comparing assessment and scholarly research Assessment, while a cousin of scholarly research, differs in its purpose and therefore in its nature (Upcraft & Schuh, 2002). Traditional scholarly research is commonly conducted to test theories, while assessment is a form of action research (see Jargon Alert) conducted to inform one’s own practice – a craft-based rather than scientific approach (Ewell, 2002). The four-step teaching-learningassessment cycle of establishing learning goals, providing learning opportunities, assessing student learning, and using evidence of student learning mirrors the four steps of action research: Plan, act, observe, and reflect. Assessment, like any other form of action research, is disciplined and Jargon Alert! systematic and uses many of the Action Research methodologies of traditional research. Action research is a distinct type of research whose But most faculty and staff lack the time purpose is to inform and better one’s own practice rather than make broad generalizations. and resources to design and conduct rigorous, replicable empirical research 11

Understanding Assessment

studies of student learning. They instead aim to keep the benefits of assessment in proportion to the time and resources devoted to them (Chapter 12). If you design your assessments reasonably well and collect corroborating evidence (Chapter 21), your evidence of student learning may be imperfect but will nonetheless give you information that you will be able to use with confidence to make decisions about teaching and learning.

Comparing assessment and evaluation Is assessment a synonym for evaluation? It depends on the definition of evaluation that is used. Evaluation may be defined as using assessment information to make an informed judgment  on matters such as whether students have achieved the learning goals we’ve established for them, the relative strengths and weaknesses of teaching strategies, or what changes in learning goals and teaching strategies are appropriate. Under this definition, evaluation is the last two steps of the teaching-learning-assessment process: Interpreting student learning evidence (part of Step 3) and using it (Step 4). This definition points out that student learning evidence alone only guides us; it does not dictate decisions to us. We use our best professional judgment to make appropriate decisions. This definition of evaluation thus reinforces the ownership that faculty and staff have over the assessment process. Evaluation may be defined as determining the match between intended and actual outcomes.  Under this definition, evaluation is virtually synonymous with the third step of the teaching-learning-assessment cycle. Evaluation may be defined as investigating and judging the quality or worth of a program, project, or other endeavor.  This defines evaluation more broadly than assessment. We might evaluate an employee safety program, an alumni program, or a civic project designed to reduce criminal recidivism. While assessment focuses on how well student learning goals are achieved, evaluation addresses how well all the major goals of a program are achieved. An anthropology program, for example, might have goals not only for student learning but also to conduct anthropological research, provide anthropological services to local museums, and conduct its affairs in a cost-effective manner. An evaluation of the program would consider not only student learning but also research activities, community service, and cost-­ effectiveness.

12

What Is Assessment?

Just as assessment and evaluation of student learning are sometimes considered synonymous, so are assessment and measurement of student learning. But many people have a relatively narrow conception of measurement, thinking of it as placing something on a quantitative scale akin to a yardstick. This book avoids the term measurement, because assessment is much broader than this conception. Assessment may generate qualitative as well as quantitative evidence of student learning (Chapter 20); it may generate categorical evidence as well as evidence that can be placed on a scale (Chapter 23); and it does not have the precision that images like a yardstick imply (Chapter 24).

CHAPTER 1

Comparing assessment and measurement

Time to think, discuss, and practice 1. Compare the traits of traditional and contemporary assessment practices in Table 1.1 with those you’re aware of at your college. Do people at your college largely practice traditional or contemporary approaches to assessment? Can you think of any ways to make contemporary approaches more pervasive? 2. If anyone in your group has already conducted an assessment of student learning, ask them to share what was done. Then discuss how, if at all, the assessment would have been done differently if it had been approached as scholarly research.

13

CHAPTER 2

The Many Settings for Student Learning and Assessment Some valuable ideas you’ll find in this chapter Liberal education has nothing to do with political views; it stems from liber, the Latin word for “free”, meaning not obliged to a particular vocation. The best learning experiences are integrated and developed collaboratively.

15

Understanding Assessment

S • • • • •

tudent learning takes place in many settings and on many levels. The setting in which learning takes place can affect: Our learning goals for students How the curriculum is designed to help students achieve those learning goals How student achievement of those learning goals is assessed How assessment is organized and supported How and by whom the resulting evidence of student learning is used

Students Student-level assessment assesses how well individual students have achieved course learning goals (Chapter 4) through tests, quizzes, assignments, and the like. Its primary purpose is to give students feedback and grades on their learning. This is the kind of assessment that faculty have done for literally thousands of years.

Classes (course sections) Class-level assessment assesses how well an entire course section or class of students has achieved course learning goals. Its primary purpose is to enable the faculty member to reflect on and improve student learning in that class and/or in subsequent offerings of the same class. Class-level assessments look at the same evidence as student-level assessment, but in a different way. Evidence for all students in a class or course section is tallied (Chapter 23) to get an overall picture of how well the entire class achieved course learning goals. A faculty member might tally, for example, how many students got each question right on a midterm or how many students earned each possible score on each rubric criterion. These tallies take time to compile, of course, but looking at tallies for just a few key assignments, particularly those completed toward the end of the course, can generate plenty of helpful insight.

Courses Course-level assessment assesses how well all students in a multi-section course have achieved course learning goals. Its primary purpose is to enable the faculty teaching those sections to collaboratively reflect on and improve student learning. Faculty engaged in course-level assessment collaborate to identify a core set of course learning goals that they agree are essential to future success, perhaps in a subsequent course, a subsequent program, a career, or simply in life. (Individual faculty members may, of course, address additional learning goals of their own choosing beyond this common core.) The faculty then collaborate to identify how best to assess the core learning 16

Academic programs

Program-level assessment looks at how well all students in a program have Institutions of higher education go by many achieved program-level learning goals. names: College, university, institute, seminary, But what is a program? The answer is and academy, among others. Many people in higher education, including many accreditors, surprisingly complicated. It depends use the generic term institution to refer to all in part on the credential being offered. of these entities. But that term can also be a bit off-putting. Students in the United States do not Most colleges in the United States offer “go to institution”; they go to college. So this courses leading to a degree, certificate, book uses college as a generic term to refer to a two- or four-year college, university, or any other or diploma. institution of higher education. Most bachelor’s degree curricula have two components: A major field of study and a general education curriculum. The major field of study is generally considered an academic program. Certificate, diploma, and graduate degree curricula generally consist of a program of study without a general education curriculum. So in these cases the courses leading to these awards constitute an academic program. Associate degrees are where things get complicated. There are two kinds of associate degrees: those that prepare students for immediate entry into a career, and those that prepare students to transfer into a bachelor’s degree program. Career associate degrees typically include about 15 credits of general education requirements. Transfer associate degrees typically include about 45 credits of general education requirements, with the remaining 15 credits taken as electives or in a concentration, specialization, emphasis, or track. So what constitutes an associate degree program? Does a career associate degree program include those 15 or so credits of general education requirements or not? Does a transfer associate degree program consist of the 45 credits of general education requirements, the 15-credit concentration or track, or both? If a college is public, it may be bound by a state or system definition of a program, sometimes apt, sometimes not (Nunley, Bers, & Manning, 2011). If a college is not bound in this way, it has flexibility in defining what constitutes a program. If you have this flexibility, you can define an associate degree program in a way that students, faculty, staff, and other constituents will find meaningful and useful. Institution

Settings for Student Learning and Assessment

Jargon Alert!

CHAPTER 2

goals that they have agreed upon. Faculty might, for example, agree to use a shared rubric to grade the final course assignment, or to include a shared set of five questions on the final exam. They then summarize evidence across sections to get an overall picture of student achievement of the core course learning goals. If they see areas of weakness across sections, they work collaboratively to identify ways to address them.

17

Understanding Assessment

Keep in mind the three purposes of assessment discussed in Chapter 6: Ensuring and improving educational quality, stewardship, and accountability. Also keep in mind that, as discussed in Chapter 4, program learning goals should be addressed repeatedly, with increasing rigor, throughout the program. Define an associate degree program in a way that helps you identify meaningful program learning goals and allows you to use evidence of student achievement of those goals to improve student learning in the program, be effective stewards of college resources, and be accountable to college and program constituents. Also consider what is on your graduates’ transcripts and diplomas, which often note a credential such as Bachelor of Science in X or Associate of Arts in Y. In these cases, X or Y is the program. For transfer associate degrees, it may make the most sense to think of the entire associate degree as the program whose learning goals are Jargon Alert! being assessed, with the 15 or so credits beyond the general education requirements Liberal Arts as a concentration, specialization, track, or The liberal arts are those studies addressing knowledge, skills, and competencies that cross emphasis of the program whose learning disciplines, yielding a broadly educated, wellgoals might also be assessed. Program-level rounded individual. The term liberal comes from liber, the Latin word for “free”; in the Middle Ages, assessment is discussed further in Chapter 7. a liberal arts education was for the free individual, as opposed to an individual obliged to enter a particular trade or profession. Today many people use the term liberal arts and sciences or simply arts and sciences to make clear that the liberal arts comprise study of the sciences as well as the arts and humanities. The Association of American Colleges & Universities (AAC&U), a leading advocate of liberal arts education, refers to liberal arts as a liberal education (Humphreys, 2013).

Jargon Alert! General Education General education is the part of a liberal arts education that is shared by many or all undergraduates at a college (Humphreys, 2013). It may go by another name such as liberal arts core or core curriculum. General education curricula traditionally comprised study of arts, humanities, social sciences, natural sciences, and mathematics, although today’s general education curricula are increasingly diverse. This book refers to general education as a curriculum rather than a program; it uses the term program to refer to studies that lead to a degree, certificate, or other credential.

18

General education curricula and other college-wide learning experiences Most U.S. colleges incorporate study of the liberal arts into undergraduate curricula through general education requirements (see Jargon Alerts). General education assessment assesses how well all undergraduates have achieved general education learning goals. Faculty teaching courses that address a particular general education learning goal collaborate to decide how to assess that learning goal. They then aggregate evidence across courses to get an overall picture of how well students have achieved that learning goal. General education assessment is discussed further in Chapter 8. Some colleges have college-wide learning requirements or expectations

Co-curricular Experiences Co-curricular experiences are out-of-classroom experiences that help students achieve meaningful learning goals in concert with academic study (Suskie, 2015). Examples include the following: • Athletics • Collaborative learning experiences • Cultural experiences such as lectures, exhibits, and performances • Field experiences such as internships, practicums, and clinicals

Co-curricular experiences Student learning takes place outside as well as within academic curricula. Wherever student learning and development is supposed to happen, there should be goals for that learning and assessments to see how well students are achieving those goals. Assessment of co-curricular experiences (see Jargon Alert) is discussed further in Chapter 8.

• First-year experiences

Assessment as part of an

• Learning communities

integrated, collaborative learning

• Service learning experiences • Student activities such as clubs

Settings for Student Learning and Assessment

Jargon Alert!

CHAPTER 2

beyond general education, such as a required service learning experience or an expectation that ethical reasoning skills are developed in all academic programs as well as in the general education curriculum. Assessment of institutional learning goals looks at how well students have achieved college-wide learning goals in settings beyond general education. How institutional learning goals are assessed depends on how and where they are addressed in the college’s curricula, as discussed in Chapter 5.

experience

Student learning is deeper and more lasting when students can see connections among their learning experiences (List 26.1 in Chapter 26). Learning experiences should therefore be purposefully designed as coherent, integrated, and collaborative, building upon and reinforcing one another (Chapter 5). The need for coherence, integration, and collaboration affects each part of the teaching-learning-assessment process (Chapter 1). • Learning goals should be integrated across settings, with appropriate relationships among general education, program, course, and co-curricular learning goals as discussed in Chapter 4. • Curricula in any setting should be designed to ensure that every student, regardless of the choices he or she makes in selecting a course of study, has ample opportunity to achieve every key learning goal of that curriculum (Chapter 5). • Learning goals, curricula, and assessments should be designed collaboratively, drawing from participants in all relevant sectors of the college community (Chapter 13).

• Student support services such as tutoring, advisement, counseling, health and safety, and library

19

✓

✓ ✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓ ✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

Co-curricular Experience

✓

✓

✓

✓

✓

✓

✓

General Education or Other College-Wide Experience

✓

✓

Academic Program

✓

Course

✓

Class (Course Section)

If You Are Assessing in This Setting

Table 2.1:  Relevant Chapters of This Book for Each Learning Setting

15. Designing Rubrics to Plan and Assess Assignments

14. Valuing Assessment and the People Who Contribute

13.  Collaborating on Assessment

12.  Keeping Assessment Cost-Effective

11.  Supporting Assessment Efforts

10.  Helping Everyone Learn What to Do

9. Guiding and Coordinating Assessment Efforts

8. Planning Assessments in General Education, Co-curricula, and Other Settings

7. Planning Assessments in Academic Programs

6. How Will Your Evidence of Student Learning Be Used?

5. Designing Curricula to Help Students Learn What’s Important

4. Learning Goals: Articulating What You Most Want Students to Learn

3.  What Are Effective Assessment Practices?

You May Find These Chapters Helpful

✓ ✓ ✓ ✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓ ✓

✓

✓

✓ ✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

26. Using Evidence of Student Learning to Inform Important Decisions

25.  Sharing Evidence of Student Learning

24.  Analyzing Evidence of Student Learning

23. Summarizing and Storing Evidence of Student Learning

22. Setting Meaningful Standards and Targets

21.  Assessing the Hard-to-Assess

20.  Other Assessment Tools

19.  Selecting Published Instruments

18. Assembling Evidence of Student Learning into Portfolios

17. Writing Multiple-Choice and Other Objective Tests

16.  Creating Effective Assignments

Understanding Assessment

• Assessment is more effective and useful when it’s integrated across multiple settings. Assessments that are embedded into individual courses, for example, can often provide information on student achievement of program learning goals and general education learning goals as well as course learning goals (Chapter 12).

Using this book to assess student learning in your setting No matter the setting in which you’re assessing student learning, this book can help. Table 2.1 suggests chapters that may be especially relevant to each setting.

Time to think, discuss, and practice 1. Does your college (or, if applicable, the system it is part of) define what it considers to be an academic program? 2. Review the examples of co-curricular experiences listed in this chapter. Which experiences are offered in your college? 3. Brainstorm a learning goal that might be addressed in both academic and co-curricular programs at your college.

22

CHAPTER 3

What Are Effective Assessment Practices? Some valuable ideas you’ll find in this chapter The best assessments are those whose results are used to inform meaningful, important decisions. Effective assessments are clear to everyone: Students as well as faculty and staff. Effective assessments don’t sacrifice validity on the altar of reliability. Retention, graduation, and job placement rates alone don’t tell us much about student learning because so many other factors can affect them.

23

I

Understanding Assessment

t is impossible to determine whether any assessment tool is good quality by simply examining it. Its quality depends on how it is used. Consider two admittedly ridiculous examples. A test assessing computation skills cannot be a good-quality assessment for critical thinking skills, no matter how good it looks on face value. And an assessment using a rubric designed to assess critical thinking, no matter how well it is designed, cannot be considered good quality if the rubric is used to assess papers in which students are asked to provide only descriptions or summaries. An assessment can thus be considered good quality only if it is the right assessment for the learning goals you want to assess and the decisions you want the resulting evidence to inform. In other words, a good-quality assessment is effective in achieving its purpose. An assessment tool is also good quality only if it is part of a good-quality assessment process – including the assignment or directions given to students, the scoring procedure, and the use of the resulting evidence. So it’s more helpful to think of effective assessment practices than good-quality assessments. The traits of effective assessment practices discussed in this chapter reflect several published statements of principles of good assessment practice (List 3.1). Some people also use the principles of good assessment practice published by the now-defunct American Association for Higher Education’s Assessment Forum in 1992. But the statements in List 3.1 are more recent and reflect evolution in assessment research and practice since the AAHE statement was developed.

List 3.1  Statements of Principles of Good Assessment Practice ■■

■■

■■ ■■

■■

■■

■■

■■

24

Excellence in Assessment Evaluation Rubric 2017 of the National Institute for Learning Outcomes Assessment (www.learningoutcomesassessment.org/eiadesignation.html) Criteria of the CHEA Award for Outstanding Institutional Practice in Student Learning Outcomes of the Council for Higher Education Accreditation (www.chea.org/chea%20award/CHEA_ Awards_All.html) Guidelines for Judging the Effectiveness of Assessing Student Learning (Braskamp & Engberg, 2014) Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council for Measurement in Education, 2014) Code of Fair Testing Practices in Education of the Joint Committee on Testing Practices (www.apa.org/ science/programs/testing/fair-testing.pdf) Code of Professional Responsibilities in Educational Measurement of the National Council on Measurements in Education (www.ncme.org/ncme/NCME/NCME/Resource_Center/LibraryItem/ Code_of_Professional_Responsibilitie.aspx) Program Evaluation Standards of the Joint Committee on Standards for Educational Evaluation (Yarbrough, Shulha, Hopson, & Caruthers, 2010) ETS International Principles for the Fairness of Assessments of Educational Testing Service (www.ets.org/s/about/pdf/fairness_review_international.pdf)

meaningful ways

Effective Assessment Practices

The statements of good practice in List 3.1 collectively suggest one fundamental, overriding trait of effective assessment practices: They yield evidence of student learning that is used to inform meaningful, substantive changes to teaching and learning, including resource support for those changes. One of the great things about defining effective assessment practices this way is that you do not need a formal background in social science research methods to recognize whether your assessment practices are good quality. If you are comfortable using your evidence to inform meaningful decisions, your assessment practices are good enough, period. If you’re not comfortable using your evidence to inform meaningful decisions, use the steps in List 3.2 to figure out why and plan a more useful assessment process next time.

CHAPTER 3

Effective assessment practices yield evidence that is useful and used in

List 3.2  Steps to Ensure That Evidence of Student Learning Is Useful and Used ■■ ■■

■■

■■

■■

■■ ■■ ■■ ■■ ■■ ■■

■■

■■

■■

Identify and articulate clear statements of your most important learning goals (Chapter 4). Ensure that curricula and learning experiences are designed to make certain that students have ample opportunities to achieve those learning goals (Chapter 5). Understand why you are assessing student learning and the decisions you and others will use the resulting evidence to inform (Chapter 6). Use assessments that have been thoughtfully developed or chosen (Chapter 7) to address your key learning goals. Use assessments that are likely to yield reasonably accurate and truthful evidence of student learning, that are significant and varied, that are fair and unbiased, and that are conducted ethically (all discussed in this chapter). Look at enough evidence of student learning (Chapter 12). Offer flexible assessment options (Chapter 9). Clearly and defensibly define what satisfactory evidence of student learning looks like (Chapter 22). Summarize evidence meaningfully and appropriately (Chapter 23). Integrate evidence into an overall picture of student learning (Chapter 24). Share evidence of student learning clearly and readily with people who can use it to inform decisions (Chapter 25). Commit to and nurture a culture of using good-quality evidence to inform decisions large and small (Chapter 9). Commit to and nurture a culture of evidence-based betterment and innovation rather than maintaining the status quo (Chapter 14). Complete each of these steps collaboratively (Chapter 13).

25

Understanding Assessment

Effective assessments focus on what’s most important Keep the following ideas in mind. Effective assessments focus on important learning goals:  The key learning that students take with them when they successfully complete a learning experience. Effective assessments include direct ­evidence of student learning.  Imagine, however distasteful this may seem, that Direct and Indirect Evidence of Student Learning there’s a critic of the learning experiences Direct evidence of student learning is the kind that would convince a skeptic that students indeed you help provide, convinced that what you have learned what they need to persist, graduate, offer is a joke, with students learning little transfer, obtain jobs, and otherwise succeed. Indirect evidence is less compelling evidence of worthwhile. Direct evidence of student student learning. learning (see Jargon Alert) is the kind that even that critic would find hard to argue with. While indirect evidence can add useful insight, no assessment of knowledge, understanding, or thinking or performance skills should consist of indirect evidence alone (see Jargon Alert). List 3.3 gives examples of direct evidence of student learning, and List 3.4 gives examples of indirect evidence.

Jargon Alert!

List 3.3  Examples of Direct Evidence of Student Learning ■■ ■■ ■■ ■■

■■

■■

■■ ■■

■■ ■■ ■■

26

Capstone experiences (Chapter 5), assessed using a rubric (Chapter 15) Other written work, performances, and presentations, rated using a rubric Portfolios of student work (Chapter 18) Scores and pass rates on appropriate licensure/certification exams (such as Praxis or NCLEX) or other published tests (such as Major Field Tests) that assess key learning goals (Chapter 19) Scores on locally designed multiple-choice and/or essay tests, accompanied by test blueprints (Chapter 17) describing what the tests assess Score gains (value added) between entry and exit on published or local tests or samples of student work (Chapter 22) Assessments of student skills observed by their field experience supervisors (Chapter 13) Observations of student behavior (such as presentations and group discussions), undertaken systematically and with notes recorded systematically (Chapter 21) Summaries and assessments of electronic class discussion threads (Bauer, 2002) Classroom response systems (such as clickers) Student reflections on their values and attitudes (Chapter 20), if developing those are intended goals of the program

■■

Honors, awards, and scholarships earned by students and alumni

■■ ■■

■■ ■■

■■

■■

■■ ■■ ■■ ■■ ■■

■■

■■

Effective Assessment Practices

■■

Course grades (Chapter 1) and grade distributions Assignment grades, if not accompanied by a rubric or scoring criteria (Chapter 1) Test grades, if not accompanied by scores on subsets of test questions mapped to a test blueprint (Chapter 17) Retention and graduation rates For four-year programs, admission rates into graduate programs and graduation rates from those programs For two-year programs, admission rates into four-year colleges and graduation rates from those programs Scores on tests required for further study such as the Graduate Record Examinations (GRE) that evaluate skills learned over a lifetime rather than through a college learning experience Quality or reputation of graduate and four-year programs into which alumni are accepted Placement rates of graduates into appropriate career positions and starting salaries Alumni perceptions of their career responsibilities and satisfaction Student ratings of their knowledge and skills (Chapter 20) Those questions on end-of-course student evaluation forms that ask about the course rather than the instructor Student/alumni/employer satisfaction with learning, collected through surveys, exit interviews, or focus groups (Chapter 20) Voluntary gifts from alumni and employers Student participation rates in faculty research, publications, and conference presentations

■■

CHAPTER 3

List 3.4  Examples of Indirect Evidence of Student Learning

Effective assessments are significant.  Quizzes, homework, and class discussions can give us insight into whether students are on track to achieve key learning goals. But far more useful are significant learning activities that ask students to demonstrate or perform the skills they have learned, especially learning activities that mirror real-world experiences. Chapter 16 discusses how to create these kinds of learning activities, and Chapter 5 discusses capstones, which are particularly useful as both learning and assessment activities.

Effective assessment practices yield reasonably accurate and truthful evidence of student learning Assessments need to give us reasonably accurate and truthful information on what students have learned. This information doesn’t have to be perfect, but it should be good enough quality that we can use the resulting evidence with confidence to 27

Understanding Assessment

Jargon Alert! Validity

inform meaningful plans and decisions, as discussed in the previous section.

Validity refers to the usefulness of information on whatever an assessment is intended to assess. There are several forms of validity. The most important when assessing student learning are content validity, which is how well the assessment yields meaningful evidence of student achievement of the learning goals it is intended to assess, and consequential validity (Pike, 2012), which is how well the evidence can be used to inform meaningful, substantive decisions and solve problems. Chapter 24 briefly discusses how to analyze the validity of assessments.

Is it possible to assess completely accurately?  Unfortunately, it is not possible to determine with complete confidence exactly what students have and haven’t learned, because we can’t get inside their heads to find out. The best we can do is to look at samples of their behavior – what they write, ­produce, say, and perform – and from those samples try to estimate or infer what they have truly learned. Even under the best of circumstances, making an inference from these snapshots of behavior is bound to be at least somewhat inaccurate because of what psychometricians call measurement error – fluctuations in human performance that we can’t completely control. We cannot control the following, for example: • Whether a student is ill on the day she completes an assignment or takes a test • Memory fluctuations (we all periodically blank out on key names and facts) • Luck in whether a particular assignment or test question focuses on something a student knows well (we all learn some aspects of a subject better than others) • Mental set (sometimes we have flashes of insight; sometimes we seem in an inexplicable mental rut) A perfect assessment – one giving absolutely accurate information on what students have learned – thus does not exist. “One of the most dangerous and persistent myths in American education is that the challenges of assessing student learning will be met if only the right instrument can be found” (Schneider & Shulman, 2007, p. vii). We must instead simply strive to make assessments sufficiently truthful that we will have reasonable confidence in the resulting evidence and can use it with confidence to make decisions about learning goals, curricula, and teaching strategies. The following principles will help. Effective assessments are clear to everyone.  Learning goals, assignments, tests, rubrics, and other assessment tools are written in clear and observable terms, so they can be applied and interpreted consistently and equitably (Berrett, 2015; Brookhart, 2013; Lane, 2012; Selke, 2013). Effective assessments yield generalizable evidence of student learning  that lets us draw overall conclusions. Unfortunately, this is hard to do because, as already

28

CHAPTER 3 Effective Assessment Practices

noted, there is no perfect assessment measure. Any single assignment or test question provides an incomplete and possibly distorted picture of student learning (Huot, 1990) that is affected by what students are asked to do and their backgrounds. Consider these examples: • Maria is not a strong writer but great at visualizing concepts. She will better demonstrate her understanding of a complex concept if she can draw a diagram rather than write an explanation. • Robert’s culture values collaboration, and he learns more when he works with others than when he studies alone. He will better demonstrate his learning if he can work with others on a group presentation rather than make a solo presentation. • Omar is not a good test taker but very creative. He will better demonstrate his understanding if he can create a video explaining a complex concept rather than take a test. • Keisha attended a high school that stressed rote memorization and drill. She will better demonstrate her knowledge of American history on a fill-in-the-blank test than in a term paper that requires critical thinking skills. • In an assessment of general education skills, some students may have an unfair advantage or disadvantage because of their major (Banta, 2008). Latoya has studied marketing and thus has an advantage in a general education assessment that asks students to use critical thinking skills to design a public relations campaign. Using a variety of assessments acknowledges the variety of experiences that students bring, gives all students their best chance to shine (Grove, 2016), and lets us infer more confidently how well students have achieved key learning goals (Lane, 2010 & 2012; Rezaei & Lovorn, 2010). Chapter 16 offers suggestions for varying assignments and learning activities. Assessment variety is especially important for goals to instill attitudes and values, because direct evidence of these learning goals is difficult if not impossible to collect. Chapter 21 discusses strategies to assess these traits. Effective assessments have an approJargon Alert! priate range of outcome levels.  Their floor and ceiling (see Jargon Alert) are Floor and Ceiling appropriate to the students being assessed The floor of an assessment is the lowest possible score or rating; the ceiling is the highest possible (Brookhart, 2013; Lane, 2012). The ceiling score or rating. should be achievable by at least some of the students whose work is being assessed (Selke, 2013), while the floor should represent an unacceptable performance level, perhaps where students were before they began the learning experience.

29

Understanding Assessment

Jargon Alert! Reliability Reliability is the consistency or dependability of an assessment tool. There are several kinds of reliability. Interrater reliability is the consistency between two raters. Internal consistency is the consistency among items in a test or survey that are all supposed to assess the same thing. Reliability refers only to the consistency or dependability of an assessment tool such as a test, rubric, or survey. Reliability is not the term for consistency across a college’s learning venues, programs, and modalities, which is discussed later in this chapter and in Chapter 5.

Effective assessments are dependable and consistent,  yielding reliable e­ vidence. Consistency is affected by the clarity of the assessment tool, the training of those who use it, and the degree of detail provided to students in the assignment (Parkes, 2012). Chapter 24 discusses how to evaluate the consistency or reliability of student learning evidence (see Jargon Alert). Publishers of published instruments (Chapter 19) should provide statistical ­evidence of reliability.

Effective assessments maintain an appropriate balance among quality, ­dependability, and usefulness.  Barbara Wright expressed this beautifully when she asked, “Are we sacrificing validity on the altar of reliability?” (personal c­ ommunication, August 9, 2013). You can spend a lot of time and resources measuring the consistency of your assessment (calculating interrater reliabilities, for example), but your assessment will be nonetheless meaningless – and your time and resources wasted – if your assessment is not clearly tied to important learning goals. Maintaining the right balance is discussed further in Chapters 12 and 19.

Effective assessment practices are fair and unbiased No one wants to use an assessment tool with obvious stereotyping or offensive material, of course. But it’s easy to assess in ways that inadvertently favor some students over others. Effective assessment processes yield evidence and conclusions that are meaningful, appropriate, and fair to all relevant subgroups of students (Lane, 2012; Linn, Baker, & Dunbar, 1991). The following tips minimize the possibility of inequities. Chapter 21 offers additional suggestions on ways to look at attitudes, values, and other qualities that are particularly difficult to fairly and accurately assess. Don’t rush.  Assessments thrown together at the last minute invariably include flaws that greatly affect the fairness, accuracy, and usefulness of the resulting evidence. Plan your assessments carefully.  Aim not only to assess your key learning goals but to do so in a balanced, representative way. If your key learning goals are that 30

Effective Assessment Practices

Aim for assignments and questions that are crystal clear.  If students find a question difficult to understand, they may answer what they think is the spirit of the question rather than the question itself, which may not match your intent. Creating clear assignments, test questions, and survey questions is discussed in Chapters 16, 17, and 20, respectively.

CHAPTER 3

students should understand what happened during a certain historical period and evaluate the decisions made by key figures during that period, for example, your test should balance questions on basic conceptual understanding with questions ­assessing evaluation skills. Test blueprints (Chapter 17) are a great tool for ensuring this kind of balance. Similarly, plan an assignment (Chapter 16) by first writing a rubric (Chapter 15): A list of the key things you want students to demonstrate on the completed assignment.

Guard against unintended bias.  A fair and unbiased assessment uses contexts that are equally familiar to all and uses words that have common meanings to all. A test question on quantitative skills that asks students to analyze football statistics might not be fair to women, and using scenarios involving farming may be biased against students from urban areas, unless you are specifically assessing student learning in these contexts. Ask a variety of people with diverse perspectives to review assessment tools.  This helps ensure that the tools are clear, that they appear to assess what you want them to, and that they don’t favor students of a particular background. Chapter 13 further discusses the role of collaboration in the assessment process. Try out large-scale assessment tools.  If you are planning a large-scale assessment with potentially significant consequences, try out your assessment tool with a small group of students before launching the large-scale implementation. Consider asking some students to think out loud as they answer a test question; their thought processes should match up with the ones you intended. Read students’ responses to assignments and open-ended survey questions to make sure their answers make sense, and ask students if anything is unclear or confusing.

Effective assessment practices are conducted ethically The principles of good practice cited in List 3.1 include several ethical tenets. One of these – giving students ample opportunities to learn the skills needed for the assessment – is discussed in Chapter 5. Tenets for sharing and using evidence fairly, ethically, and responsibly are discussed in Chapters 25 and 26, respectively.

31

Understanding Assessment

Inform students of the nature, purpose, and results of each assessment.  There are several aspects of assessment that should be shared with students. Chapter 4 ­discusses sharing learning goals; Chapter 16 discusses sharing the purpose, learning goals, and rubrics for learning activities and assignments; and Chapter 17 d ­ iscusses sharing a test blueprint as a study guide. If students have add-on graduation or program completion requirements (Chapter 21), such as compiling a portfolio, completing a survey, or taking a comprehensive examination, inform them of this in writing and as early in their program as possible. Assessment activities can be valuable learning opportunities for students only if they receive prompt, concrete feedback on their performance (Butler & McMunn, 2006). To require students to participate in an assessment and not offer feedback on their performance diminishes the overall value of the assessment experience and is inconsiderate of their contributions. Protect the privacy and dignity of those who are assessed.  Take appropriate security precautions before, during, and after you conduct an assessment. Protect the confidentiality of individually identifiable information. Password-protect all electronic files that include identifiable information, and store paper records with identifiable information in locked file cabinets. If several people are reviewing samples of student work or accessing an electronic file, removing information that identifies individuals may be a wise precaution.

While it is important to protect student privacy, faculty and staff must also have sufficient information to do their jobs, and this can involve sharing identifiable information. Some faculty and staff, for example, periodically meet to discuss the progress of each of the students in their program. Faculty and staff also consult with their colleagues about their students in less formal settings; a faculty member concerned about a student’s slipping performance might consult with the student’s advisor for ideas on how to help the student get back on track. Faculty and staff are simply carrying out an important part of their responsibilities when they hold such conversations, and considering identifiable evidence of student learning can make the conversations more fruitful. Assess student learning fairly, equitably, and consistently.  No matter how carefully assessments are constructed, they are nonetheless subject to errors and biases such as those in List 3.5. See List 3.6 for strategies to minimize assessment errors and biases and achieve greater assessment consistency. Some of these strategies are easier said than done, of course! Chapter 12 offers suggestions for deciding if these strategies are worth the time and expense.

32

■■

■■ ■■ ■■

■■

■■

■■

■■

■■

Effective Assessment Practices

■■

Leniency errors occur when faculty and staff judge student work better than most of their colleagues would judge it. Generosity errors occur when faculty and staff tend to use only the high end of the rating scale. Severity errors occur when faculty and staff tend to use only the low end of the rating scale. Central tendency errors occur when faculty and staff tend to avoid both extremes of the rating scale. Halo effect bias occurs when faculty and staff let their general impression of a student influence their ratings, perhaps giving higher ratings to a student who seeks extra help or lower ratings to a student who is quiet in class. Contamination effect bias occurs when faculty and staff let irrelevant student characteristics (such as inadvertent typographical errors or ethnic background) influence their ratings. Similar-to-me effect bias occurs when faculty and staff give higher ratings to those students whom they see as similar to themselves, such as students who share their research interests. First-impression effect bias occurs when faculty and staff’s early opinions of a student’s work distort their overall judgment of that work. A student who presents her outstanding research in a sloppy poster display might suffer from first-impression effect bias, as might a student whose generally excellent essay opens with a poorly constructed sentence. Contrast effect bias occurs when faculty and staff compare a student against other students instead of established standards. Faculty might give an Unacceptable rating to the worst paper they read, even though the paper meets stated standards. Rater drift occurs when faculty and staff unintentionally redefine rating criteria over time. Some faculty and staff, as they tire from rating student work, get grumpy and more stringent, while others skim student work more quickly and rate more leniently.

CHAPTER 3

List 3.5  Examples of Assessment Errors and Biases

List 3.6  Strategies to Minimize Assessment Errors and Biases ■■

Use a well-crafted analytic rubric (Chapter 15) – one that clearly describes student achievement at each of the rubric’s performance levels.

■■

Remove or obscure identifying information from student work before it is assessed. (This is called blind scoring and is discussed in Chapter 12.) Practice consistency when multiple faculty and staff are assessing student work with a rubric. First, discuss and come to agreement on the meaning of each performance level. Then assess a few samples of student work, share your ratings, and discuss and resolve any differences. Once you’re collectively sure that you’re all reasonably consistent, you can begin assessing the full array of student work. Have each sample of student work assessed independently by at least two faculty or staff members. If those two disagree on any sample of student work, have that work assessed by a third person to break the tie. (This is called double scoring and is discussed in Chapter 12.) Reassess the first few samples when faculty and staff are assessing many samples of student work, to guard against rater drift. Periodically schedule a refresher practice session when faculty and staff are assessing large samples of student work, in which they all compare their ratings and discuss and resolve any emerging differences.

■■

■■

■■

■■

33

Understanding Assessment

34

The role of institutional review boards in assessment Title 45, Part 46 of the Code of Federal Regulations (www.hhs.gov/ohrp/regulationsand-policy/regulations/45-cfr-46/index.html) describes federal policy for the protection of human research subjects. The regulations stipulate that colleges must establish institutional review boards (IRBs) to ensure that research protects and poses no significant risk or threat to the rights and welfare of human subjects. There are three levels of review: full review (which requires appearing before the entire IRB), expedited review (in which at least one member of the IRB reviews the research plan), and exempted from review (under which the research plan must still be sent to the IRB). The regulations define research as “a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge.” Two kinds of research activities are exempt from the policy. One exemption is for research conducted in educational settings involving normal educational practices, such as research on instructional strategies or the effectiveness of teaching methods. The other exemption is for research involving the use of educational tests, surveys, interviews, and observations. The only exception to this second exemption is when the information obtained can be linked to the subjects and any disclosure of their responses places them at risk of liability or might damage their financial standing, employability, or reputation. Should an assessment effort come under IRB review? Some colleges take the position that assessment activities are action research (Chapter 1), designed only to inform local teaching and learning practices and not to develop or contribute to generalizable knowledge. Under this position, faculty and staff are not engaged in research as defined by this policy and there is no reason to involve the IRB. Some colleges take the position that assessments are conducted in established educational settings that involve only normal educational practices and/or educational tests, surveys, and interviews that do not place subjects at risk. Under this position, assessment efforts are exempt from this policy and there is no reason to involve the IRB. Some other colleges require that assessment plans be submitted to the IRB with a formal request for exemption. Still other colleges take the position that assessment activities should undergo review. Their rationale is that, even though most assessments are not designed to contribute to generalizable knowledge, faculty and staff may decide later to share what they have learned with professional colleagues in a conference or a journal. Some colleges that require IRB review of assessment activities take the position that assessment activities pose “minimal risk” to subjects – no more than that “ordinarily encountered in daily life” – and therefore qualify for expedited review. Obviously, interpretations of the federal regulations vary, and there is no clear consensus on the role of IRBs in assessment programs. Ask the chair of your college’s IRB for its interpretation.

2. One of the learning goals of Mackenzie College’s general education curriculum is for students to develop a tolerance for perspectives other than their own. Brainstorm three survey questions they might ask that you think would yield unbiased evidence. 3. An English literature faculty member wants to assess the writing and research skills of her department’s graduating seniors. Brainstorm possible examples of direct evidence of student learning that could be helpful in assessing students’ writing and research skills.

Effective Assessment Practices

1. History faculty members are assessing students’ writing skills by assessing senior theses for organization, focus, style, and mechanics. What might the faculty do to protect the privacy of the students and their professors as they conduct this assessment?

CHAPTER 3

Time to think, discuss, and practice

35

Part 2

Laying a Foundation for Assessment Success

CHAPTER 4

Learning Goals: ­Articulating What You Most Want ­Students to Learn Some valuable ideas you’ll find in this chapter Memorized knowledge is less important than finding and using information. Professionalism is growing in importance as a learning goal. While Bloom’s taxonomy is the best-known framework for articulating learning goals, other taxonomies fill in some voids. Consider abolishing the word demonstrate from learning goals.

39

Laying a Foundation for Assessment Success

T

o see why learning goals are vital to assessment, imagine that an English professor teaching nineteenth-century poetry asks his students to keep journals in which they reflect on what they’ve learned as the course progresses. With no clear learning goals for the journal, his assignment is simply “After every class, jot down your thoughts about what we’re learning.” With such a vague assignment, his students don’t really know what to do, and they probably won’t learn much from the experience. And it isn’t clear how the professor should assess the journals. He could assess them for any of the following: • Writing qualities such as organization, clarity, grammar, or accuracy in reflecting class lectures and discussions • Signs that students are developing an increased appreciation of nineteenthcentury poetry • How well students appraise scholarly interpretations of nineteenth-century poetry • How students relate nineteenth-century poetry to other literature, culture, and events of the times Which of these approaches is the appropriate way to assess the journals? There’s no way to tell because, without learning goals, we don’t know the purpose of the journal-keeping experience. As this example shows, without learning goals, students don’t know how to spend their learning time and energies, and we don’t know what or how to assess.

The vocabulary of goals Goals, objectives, competencies, outcomes, proficiencies . . . all of these terms can describe what students are supposed to learn, but we don’t use them consistently in higher education. The vocabulary we use is less important, however, than how goals are stated, which is discussed later in this chapter. Unless an agency or accreditor requires otherwise, use whatever terms work best with your college’s culture. That said, here is how these terms are often used. Goals  state what you, your colleagues, or your college aim to achieve. Goals can describe aims outside the teaching/learning process as well as within it. Within the teaching/learning process, an astronomy professor might have a goal that her students learn about our solar system, your colleagues may have a goal to offer a quality educational program, and your college may have a goal to encourage students to engage in community service. Outside the teaching/learning process, you may have a goal to complete some research this year, your department may have a goal to sponsor a regional scholarly conference, and your college may have a goal to raise $8 million in alumni gifts. 40

Articulating Student Learning Goals

Learning outcomes or learning goals  are goals that describe what students will be able to do as the result of a learning experience. More specifically, learning outcomes are the knowledge, skills, attitudes, and habits of mind that students take with them from a learning experience. While the term learning outcome has become prevalent in American higher education, it can be confusing because it can be used to describe two very different things: an intended or expected learning outcome, which is what is discussed in this chapter, and an actual learning outcome or result. This book therefore generally uses the term learning goals to describe intended learning outcomes, and it occasionally uses the term outcomes to describe actual results.

CHAPTER 4

Outcomes  are goals that refer to the end rather than the means, the result rather than the process. A goal that truly describes an outcome explains why we do what we do. A faculty member’s real goal is not that students write a term paper but that they write effectively in their future studies and work. A student activities director’s real goal is not to offer a student leadership development program but for students to lead student organizations effectively. A true college goal is not to raise $8 million but to increase educational opportunities for deserving students through additional scholarships.

Learning objectives  may be synonyms for learning goals; or they may describe detailed aspects of learning goals; or they may describe the tasks to be accomplished to achieve the learning goal – the means to the end, the process leading to the outcome. Within the broad learning goal of understanding basic concepts about our solar system, for example, an astronomy professor might have an objective that her students will recall basic facts about each planet. If one of her learning goals is for students to explain science concepts, her objectives might be for them to write essays and critique the drafts of their peers. Competencies and proficiencies  are terms sometimes used to describe learning goals or objectives. They are typically used to describe skills rather than knowledge or attitudes. Aims  describe aspirations that we value but that may not be achievable. We would like all students to become passionate lifelong learners, for example, but there’s no way we can ensure that every student will graduate with this disposition.

What do students need to learn? Learning goals fall into a few broad categories. 41

Laying a Foundation for Assessment Success

Knowledge and understanding While knowledge and understanding are important outcomes of many courses and programs, today they are less important than they were a generation or two ago for several reasons. The amount of knowledge available to us has exploded.  We know, for example, far more about the building blocks of matter than we did a generation ago. There is more history to understand than when our parents went to school, and the study of world history in the United States has broadened from a Eurocentric model to a global one. The number of scholarly journals in almost every field has grown exponentially. Today there are so many important concepts that we can’t expect students to remember them all. We have increasingly easy access to knowledge.  Is it so important to remember a formula, date, or vital statistic when it can be looked up effortlessly? Finding and using information is more important than remembering it.  Today we need to be able to find, analyze, and use facts and concepts as well as understand them. Our knowledge base will continue to expand and evolve.  Today’s students will someday need information that hasn’t yet been conceived and insight that hasn’t yet been drawn, rendering obsolete some of the information we now teach. Should we focus on having students remember material that may soon be outdated or irrelevant? Or should we focus on developing the thinking skills they’ll need to master new concepts on their own? Much of what students memorize is committed to short-term memory and quickly forgotten.  Imagine how your students would do if they retook your final exam – with no additional studying – just a few weeks or months later. How much would they remember from the first time they studied for it? Probably not much . . . and is it worth spending time teaching material that’s so quickly forgotten? Or should we focus instead on developing skills and attitudes that will last a lifetime, such as the ability to write well, analyze the difference between two theories, or appreciate American folk music? Does this mean that students shouldn’t memorize anything anymore?  Absolutely not! A certain amount of memorized knowledge is necessary and important. We wouldn’t want to fly in a plane whose pilot has to look up the meaning of that flashing light as the plane goes into a nosedive. We wouldn’t want to be operated on by a surgeon who has to pause to read up on how to stop that excessive bleeding. ­Students will always need to remember and understand certain fundamental ­concepts – just not as much as they once did.

42

Critical thinking  is a widely used term whose meaning lacks popular consensus. It’s often used as an umbrella term for many kinds of thinking skills that go beyond basic understanding, including analysis, synthesis, evaluation, problem-solving, information literacy, and some habits of mind. If critical thinking emerges as a potential learning goal, spell out the kinds of thinking skills it encompasses in your situation. Higher order thinking skills (HOTS)  are those beyond basic understanding. HOTS may thus be a synonym for critical thinking, depending on how the latter is defined.

Articulating Student Learning Goals

As knowledge and understanding have declined in importance, thinking skills have grown. Many kinds of thinking skills are emphasized in today’s college curricula.

CHAPTER 4

Thinking skills

Application  is the capacity to use knowledge and understanding in a new context. It includes the abilities to understand cause-and-effect relationships, interpret logical propositions, criticize literary works, and apply scientific principles to research problems if these relationships, propositions, works, and problems are new to the student. Many mathematics word problems require application skill. Analysis  is the capacity to break a complex concept apart to understand the interrelationships among its components. Students who can analyze can identify the elements, relationships, and underlying principles of a complex process. Analysis is not merely understanding the relationships of components of a process or concept explained in coursework; that would be simple understanding. Students who can analyze can understand the structure of things they haven’t seen before. They can think holistically, make a case, discover the underlying principles of a relationship, and understand organizational structure. Evaluation, problem-solving, and decision-making  skills have more in common than not. They are all skills in making informed judgments about the merits of something the student hasn’t seen before. They include skills in conducting research, making appropriate choices, solving problems with no single correct answer, and making and justifying persuasive arguments. They are not merely understanding and reflecting arguments that have been presented in coursework; that would be simple understanding. Information literacy  (Association of College & Research Libraries, 2016) is often erroneously thought of as library research skills. It is a much broader set of skills, 43

Laying a Foundation for Assessment Success

reflecting today’s reality that much research and inquiry takes place outside of the traditional college library. Information literacy can include the following skills: • Recognizing the need for information to answer a question or solve a problem • Determining what information is needed • Finding that information, whether through traditional library research, online sources, professional manuals, colleagues, original research, or other appropriate sources • Evaluating the information for credibility and relevance • Using the information to answer the question or solve the problem • Using the information legally and ethically, respecting intellectual property, and citing and acknowledging the work of others accurately Information literacy includes many of the (critical) thinking skills discussed in this chapter, and some might consider it synonymous with research or problemsolving skills. Synthesis  is the capacity to put what one has learned together in a new, original way. It includes the abilities to theorize, reflect, construct hypotheses, generate new ideas and new ways of viewing a situation, and invent. Creativity  is another concept whose meaning lacks popular consensus. Among creativity scholars, the most widely accepted definition is “a novelty that’s useful (or appropriate)” (Ryan, 2015, p. 8). Creativity might also be thought of as producing new and good ideas or products through imaginative skills, which might include the abilities to synthesize as discussed above, be flexible, take intellectual risks, and be open-minded to new ideas. Metacognition  is learning how to learn and how to manage one’s own learning by understanding how one learns, thereby preparing for a lifetime of learning. Metacognition skills may include: • Using efficient learning techniques • Forming efficient plans for completing work • Evaluating the effectiveness of one’s actions such as one’s problem-solving strategies • Critically examining and evaluating the bases for one’s arguments • Correcting or revising one’s reasoning or arguments when self-examination so warrants

Other kinds of learning goals Today’s students also need to develop performance skills, attitudes and values, and habits of mind. 44

Habits of mind  include the skills and dispositions in List 4.1 (Costa & Kallick, 2009). Richard Shavelson (2007) used the phrase personal and social responsibility (PSR) skills to characterize many of these traits.

List 4.1  Habits of Mind ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■

Articulating Student Learning Goals

Attitudes and values  include such things as an appreciation of the arts, a commitment to a healthy lifestyle, and a dedication to community service. Many attitudes and values are important because they contribute to quality of life and public good (Busteed, 2013). Chapters 20 and 21 offer suggestions for assessing attitudes and values.

CHAPTER 4

Performance skills  are physical skills such as the abilities to deliver a presentation, operate equipment, manipulate a tool, wield a paintbrush, hit a softball, or dance. Chapter 21 offers suggestions for assessing performance skills.

Persisting Thinking and communicating with clarity and precision Managing impulsivity Gathering data through all senses Listening with understanding and empathy Creating, imagining, innovating Thinking flexibly Responding with wonderment and awe Thinking about thinking (metacognition) Taking responsible risks Striving for accuracy Finding humor Questioning and posing problems Thinking interdependently Applying past knowledge to new situations Remaining open to continuous learning

Professionalism  is a habit of mind that includes coming to work (or class) on time, getting work done correctly and on schedule, and giving a task one’s best effort. Plenty of employers complain about employees’ lack of professionalism, a good reason to consider making professionalism a learning goal. Consider assessing ­professionalism through student behaviors such as submitting assignments on time, following assignment instructions correctly, and participating in group work and class discussions. 45

Laying a Foundation for Assessment Success

What do students most need to learn today? There have been numerous surveys of the knowledge, skills, and competencies that employers seek in new hires (for example, see Carnevale, Smith, & Strohl, 2013; Gallup, Inc., 2014; Hart Research Associates, 2013; and Wiseman, 2013, as well as the National Association of Colleges and Employers [www.naceweb.org/career-readiness/ competencies/career-readiness-defined/]). They suggest that, while many jobs require a combination of career-specific and transferrable skills, transferrable skills are rising in importance (Suskie, 2014). Most employers now believe that both careerspecific and transferrable skills are important to advancement and long-term career success, rather than just one or the other, and over 90 percent agree that a “capacity Jargon Alert! to think critically, communicate clearly, and Hard and Soft Skills solve complex problems is more important Hard skills are career-specific skills required for a than . . . undergraduate major” specific occupation. Soft skills are the generalizable, transferrable skills that are applicable to a wide (Hart Research Associates, 2013, p. 1). range of careers and are often required to advance List 4.2 lists transferrable skills that employers beyond a specific position. mention often.

List 4.2  Transferrable Skills Valued by Employers ■■ ■■

■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■

Teamwork and collaboration, including interpersonal skills such as listening and leadership Communication, including written, oral, and visual communication, especially articulating ideas clearly and effectively Real-world problem-solving Critical thinking and analysis, especially in evaluating information and conclusions Professionalism Working with people from diverse cultural backgrounds Flexibility and adaptability to change, including the capacity to continue learning Creativity and innovation Ethical judgment Quantitative and technology skills, especially understanding numbers and statistics

Characteristics of effective learning goals Writing well-stated, appropriate learning goals is one of the hardest steps of the assessment process. There’s no simple formula that will automatically generate effective learning goals, but the following principles will help.

46

Articulating Student Learning Goals

When we teach, we are taking our students on a journey, and our learning goal is the destination, not the route they take to get there – the outcome, not the learning process. Consider this general education learning goal: “Critically assess and express ideas about cultural diversity and individual roles in society.” This goal states what students will do in the courses they take to complete this general education requirement but not what they will be able to do after they complete the requirement. What will completing this task prepare them to do after they graduate? That’s the true learning goal. If you find it hard to make the leap from articulating learning processes to articulating learning outcomes, root cause analysis can be helpful. Ask why: Why do you ask students to write a research paper? Why do you require five lab reports? The answers, which are the true goals, can vary considerably. Faculty might ask students to write a research paper, for example, to improve their skill in a variety of areas: • Conducting original research in the discipline • Conducting library research in the discipline • Interpreting the published research of others • Writing in the language of the discipline • Analyzing evidence Understanding the true goal of a learning experience helps us create a more effective learning activity (Chapter 16) and assess it more fairly and appropriately. A faculty member who wants students to learn how to conduct library research, for example, might ask students to prepare an annotated bibliography rather than a full-blown research paper and assess it for library research skills rather than writing quality. A faculty member who wants students to learn how to write in the language of the discipline might ask students to write a research proposal rather than a full-blown research paper and assess it for writing skills rather than for research quality.

CHAPTER 4

Describe outcomes, not learning content, products, or activities

Clarify fuzzy terms Learning goals are sometimes phrased using broad, nebulous terms such as • Students will learn . . . • Students will understand . . . • Students will become aware of . . . • Students will appreciate . . . • Students will think critically . . . • Students will write proficiently . . .

47

Laying a Foundation for Assessment Success

Such vaguely stated learning goals can lead to confusion. Consider a goal mentioned at the beginning of this chapter, that students will understand basic concepts about our solar system. Astronomy professors may have differing opinions about what constitutes basic concepts. Do basic concepts include just basic facts about each planet or also theories about how the solar system was created? The faculty may also have differing opinions about what it means to understand those basic concepts. Does it mean memorizing basic facts? Or does it mean using information about our solar system to speculate about the characteristics of other solar systems? If a learning goal isn’t clear to you, it won’t be clear to your students. They’ll have difficulty figuring out what and how to learn, and you’ll have difficulty coming up with an appropriate assessment. Clarifying fuzzy terms by using vocabulary that even beginning students understand gives faculty, staff, and students a clearer common understanding of what to teach, what to learn, and what to assess. Restating “Think critically” as “Analyze and evaluate arguments,” for example, helps everyone understand what they are to do. Sometimes learning goals have a public relations purpose as well as a role in the teaching/learning process, and this can affect how they’re stated. “Think critically” and “communicate effectively” are the kinds of statements that parents, donors, and other public audiences want to see in mission statements, brochures, and websites, even if they are not terribly clear. If you need a fuzzy statement for these purposes, it’s fine to translate it into clearer terms for teaching, learning, and assessment purposes.

Use concrete, observable action verbs when possible Fuzzy, vague goals can be avoided by using concrete action words that describe, in explicit, observable terms, what students can do after they’ve learned what you want them to. A goal that students will explain or describe a concept is clearer and more straightforward to assess than a goal that students will understand a concept. Upon completing the astronomy professor’s solar system unit, for example, perhaps her students will be able to describe the key characteristics of each planet. Perhaps they will be able to create a visual model of the solar system. Perhaps they will be able to explain why each planet except Earth cannot support human life. Concrete action words help students understand what you want them to learn. They also make the assessment self-evident, as shown in these examples.

48

Assessment: Write short descriptions of the key characteristics of each planet. Action-word goal: Create a visual model of the solar system. Assessment: Create a scale model of the solar system. Assessment: Write a short explanation of why each planet except Earth cannot support human life.

If you’re struggling to come up with an observable action verb to clarify a fuzzy learning goal, imagine that you run into two alumni of your college or program. As you talk with them and learn about what they’ve been doing and what they’re thinking, it becomes clear to you that one has achieved your (fuzzy) goal and one has not. What might they tell you that would help you tell them apart? For example, graduates who demonstrate cultural sensitivity might continue to learn about other cultures and/or listen to and consider viewpoints different from their own – all observable goals.. Consider abolishing the term demonstrate from your learning goals.  Demonstrate is an action verb, but it can be fuzzy, depending on what is being demonstrated. A goal that nursing students demonstrate the proper technique for drawing a blood sample is fine, but a goal that students demonstrate critical thinking skills or an appreciation of the arts is fuzzy. Think about banning demonstrate from your assessment lexicon.

Articulating Student Learning Goals

Action-word goal: Explain why each planet except Earth cannot support human life.

CHAPTER 4

Action-word goal: Describe the key characteristics of each planet.

Be rigorous, yet realistic Learning goals should be appropriate for the setting (Chapter 2). Learning goals for degree programs should be more rigorous than those for co-curricular experiences, learning goals for seniors should be more rigorous than those for first-year students, and learning goals for graduate students should be more rigorous than those for undergraduates. The Degree Qualifications Profile (degreeprofile.org) articulates a variety of learning goals for associate, bachelor’s, and master’s degrees and may be helpful in making sure your learning goals are of appropriate rigor.

Be neither too broad nor too specific While fuzzy goals are problematic, so are goals that are too specific, because “the price of precision is the narrowness of scope” (Shulman, 2007, p. 24). Translating communicate effectively in writing into concrete action verbs would result in a long list of the many characteristics of effective writing – a list so long as to be unwieldy. With so much detail, the forest – the important learning goal – may be lost among the trees of specific action-word outcomes. The best learning goals thus fall between the two extremes of fuzzy and too specific. 49

Laying a Foundation for Assessment Success

Too vague: Students will demonstrate information literacy skills. Too specific: Students will be able to use the college’s online services to retrieve information. Better: Students will locate information and evaluate it critically for its validity and appropriateness.

Within these two extremes, there is a wide continuum of acceptable specificity. Sometimes it’s more useful to express a learning goal in relatively vague terms such as know, understand, become aware of, or appreciate rather than with observable action verbs that are too narrow. Relatively vague learning goals can be clarified by developing an accompanying rubric (Chapter 15) that lists, say, the key traits of effective writing.

Use compound statements judiciously Consider this learning goal: Describe, compare, and thoughtfully evaluate the values, beliefs, and traditions of a variety of cultures, and evaluate and apply a variety of disciplinary approaches to significant sociocultural problems and issues. There’s an enormous amount packed into this statement; it’s really over a dozen quite different learning goals strung together. It’s hard to think of how to assess this goal, let alone help students achieve it. But now consider this learning goal: Define the essential aspects of complex problems, investigate them, propose solutions, and evaluate the relative merits of alternative solutions. There are several learning goals packed into this statement too, but in this case the learning goals are all part of a larger learning goal on problem-solving – they all fit together, and it’s not hard to think of ways to teach and assess them collectively. So if a learning goal is a compound statement, ask yourself how students will achieve the goal and how you will assess it. If the goal can be assessed with one assessment, it’s probably fine, but if it needs multiple assessments, try breaking the statement apart into separate, assessable goals.

Limit your goals I have seen associate degree programs with 45 learning goals and general education curricula with over 100. There is no way that students can learn so many things well in the time we have with them or that faculty can assess them meaningfully. Consider focusing on just three to six key learning goals for each program and for each general education requirement.

50

Research Effective learning goals flow from systematic evidence of not only what is happening at your college but also what is happening in the world around it. List 4.3 gives examples of both internal and external resources that you may find helpful for identifying potential learning goals. Three resources are worth highlighting. List 4.3  Resources for Identifying Potential Learning Goals

Articulating Student Learning Goals

Because learning goals should be integrated as appropriate across settings (Chapter 2), they should be developed not in isolation but as the result of research, reflection, and informed, collaborative discussion.

CHAPTER 4

How to identify what you most want students to learn

Internal Resources ■■ ■■

■■

Your college’s mission statement, vision statement, and strategic goals Syllabi of current courses, especially capstone courses (Chapter 5), which can be analyzed for stated or implicit learning goals Transcripts, which can be analyzed to identify the courses that students choose to fulfill program and general education requirements

External Resources ■■ ■■ ■■ ■■ ■■ ■■

■■

Goals or standards espoused by relevant disciplinary associations and accreditors Surveys or interviews of prospective employers of graduates of your college or program Admissions criteria for academic programs that students pursue after program completion LEAP Essential Learning Outcomes (www.aacu.org/leap/essential-learning-outcomes) The Degree Qualifications Profile (degreeprofile.org) VALUE rubrics (www.aacu.org/value-rubrics), which offer good, concrete definitions and explications of some notoriously fuzzy learning goals Published taxonomies for learning goals

LEAP Essential Learning Outcomes  are part of an initiative of the Association of American Colleges and Universities (AAC&U) called Liberal Education and ­America’s Promise (LEAP). The outcomes, often referred to as the LEAP goals, aim to identify what all college students need to learn in order to be prepared for twenty-first-century challenges. The Degree Qualifications Profile (DQP),  sponsored by the Lumina Foundation, is intended to define “what students should be expected to know and be able to do once they earn their degrees” at the associate, bachelor’s, and master’s levels ­(Adelman, Ewell, Gaston, & Schneider, 2011, p. 1). 51

Laying a Foundation for Assessment Success

Taxonomies for learning goals.  While Bloom’s taxonomy (Anderson & Krathwohl, 2000) (see Jargon Alert) is the best-known taxonomy for learning goals, it does not highlight many of the skills and dispositions needed today, such as teamwork, ­flexibility, professionalism, and metacognition. And many interpret its cognitive hierarchy Jargon Alert! erroneously to mean that students must master one level before proceeding to Bloom’s Taxonomy the next (for example, that students must Bloom’s taxonomy (Anderson & Krathwohl, 2000) divides learning into three domains: cognitive, ­thoroughly understand a concept before affective (attitudes and values), and psychomotor they can begin to apply or analyze it). (athletic skills, laboratory skills, and so on). Cognitive skills are organized into a hierarchy of six Consider instead other taxonomies categories: remember, understand, apply, analyze, (List 4.4) that fill voids in Bloom’s taxonomy evaluate, and create. Many people still refer to the original names of these six categories: knowledge, and bring to the forefront habits of mind not comprehension, application, analysis, evaluation, emphasized in Bloom’s, such as persisting, and synthesis (Bloom, 1956). striving for accuracy, and metacognition, as well as skills that are a combination of thinking and performance skills such as leadership and teamwork. If you decide to use a taxonomy, use it as a guide, not a mandate. Some of your learning goals may not fit well into any of its categories, and don’t force them.

List 4.4  Taxonomies for Learning Goals Bloom’s Taxonomy (Anderson & Krathwohl, 2000)

• Cognitive • Remember • Understand • Apply • Analyze • Evaluate • Create • Affective • Psychomotor Taxonomy of Significant Learning (Fink, 2013)

• • • • • •

52

Foundational knowledge: Understanding and remembering information and ideas Application: Skills, thinking (critical, creative, practical), managing projects Integration: Connecting ideas, people, realms of life Human dimension: Learning about oneself and others Caring: Developing new feelings, interests, values Learning how to learn: Becoming a better student, inquiring about a subject, self-directed learning

Dimensions of Learning (Marzano, Pickering, & McTighe, 1993)

Articulating Student Learning Goals

• Attitudes and perceptions (e.g., experience a sense of comfort and order, perceive tasks as valuable and interesting) • Acquire and integrate knowledge • Declarative (construct meaning, organize, store) • Procedural (construct models, shape, internalize) • Extend and refine knowledge (compare, classify, inductive and deductive reasoning, analyze errors, analyze perspectives) • Use knowledge meaningfully (decision-making, problem-solving, invention, experimental inquiry, investigation, systems analysis) • Habits of mind • Critical thinking (seek accuracy, seek clarity, maintain an open mind) • Creative thinking (persevere, generate new ways of viewing situations) • Self-regulated thinking (plan appropriately, identify and use necessary resources, respond appropriately to feedback)

CHAPTER 4

List 4.4 Continued

Taxonomy of Educational Objectives (Marzano & Kendall, 2008)

• Knowledge • Information (details, organizing ideas, principles) • Mental procedures • Psychomotor procedures • Levels of processing • Retrieval • Comprehension (integrating, symbolizing) • Analysis (matching, classifying, analyzing errors, generalizing, specifying) • Knowledge utilization (decision-making, problem-solving, experimenting, investigating) • Metacognition (specifying goals, process monitoring, monitoring clarity, monitoring accuracy) • Self-system thinking (examining importance, efficacy, emotional response, and overall motivation)

Reflection While research is helpful, it’s rarely a good idea to adopt learning goals from external sources wholesale, even highly regarded statements such as the LEAP outcomes. Reflect on the mission, values, and culture of your college, program, course, or other learning experience. A bachelor’s degree program in biology that mostly prepares students for research careers through further graduate study may have somewhat different learning goals, for example, than one that mostly prepares students to be high school teachers.

53

Laying a Foundation for Assessment Success

Collaboration Students learn more effectively when their learning experiences are purposeful, coherent, and integrated (Chapter 5), making it important for faculty and staff to articulate key learning goals collaboratively (Chapter 13) rather than in isolation. Conversations on learning goals can be difficult, especially when participants are used to working independently. But these conversations can also be one of the most invigorating and rewarding aspects of an assessment effort, because they address the heart of faculty and staff work: teaching and learning. Focus these conversations on identifying common ground and emphasizing and respecting the many things that faculty and staff already do well (Chapter 14). Begin by compiling and sharing the results of any research or reflection on learning goals on a handout or large chart or slide for all to see. With this information in hand, faculty and staff can discuss topics regarding learning goals such as those in List 4.5.

List 4.5  Examples of Discussion Topics Regarding Learning Goals ■■

■■

■■

■■

■■

■■

■■

■■

54

Why do we offer or require this learning experience, be it a course, program, general education requirement, or co-curricular experience? Why is it important that students study or experience this? How do we want this experience to prepare them for or enrich whatever they do after graduation? What is this learning experience all about? What do we want all students to get out of this learning experience? Why are those things important? What do we want students who successfully complete this learning experience to know or be able to do five or ten years after graduating? What do our students do after they graduate? What are the most important things they need for success in those pursuits? What makes our graduates successful? What makes them attractive to potential employers, graduate programs, and the like? If we are preparing students for specific careers, what knowledge, skills, and attitudes do employers look for? What do we value most about our discipline? What does our disciplinary association (or major authorities in our discipline) think are the most important things that students should learn? Why are those things important? How does this learning experience relate to others that we offer? How does this course relate to other courses in this program or general education curriculum? How does this program or general education requirement relate to other disciplines that students may be studying? How does this cocurricular experience relate to students’ academic studies?

Learning goals for specific settings The principles discussed so far in this chapter apply to learning goals in any setting. Here are some additional suggestions for specific settings.

Learning goals for academic programs As discussed in Chapter 5, an academic program should be greater than the sum of its parts—more than a collection of courses. It should thus have program learning goals (see Jargon Alert) that are broader than those of its courses, as discussed later in this chapter. If you’re struggling to identify program learning goals that cross courses, start by Jargon Alert! looking at course syllabi for any common Program Learning Goals themes in course learning goals. Also think Program learning goals are so important that about why employers or faculty in more they are addressed throughout the curriculum. Accordingly, every student in the program must advanced programs want students to take take at least two courses that help him or her these courses. If you are developing learning achieve the program learning goal – even in associate degree and certificate programs. goals for an associate degree program in pre-engineering, for example, the program probably includes science and math courses that help students graduate with stronger scientific reasoning and quantitative skills than students in non-STEM (science, technology, engineering, and mathematics) programs. So scientific reasoning and quantitative skills would be two potential learning goals for the pre-engineering program.

Articulating Student Learning Goals

Because student learning and curricula are shared faculty responsibilities, faculty should collectively endorse institutional and general education learning goals through the college governance process. Program faculty should collectively endorse program learning goals. Strategies for achieving consensus are discussed in Chapter 13.

CHAPTER 4

Consensus

Learning goals for general education curricula Many general education curricula require students to take courses in disciplinary areas such as the natural sciences, social sciences, mathematics, and humanities, among others. It’s tempting to develop general education learning goals that align with these multidisciplinary requirements: analyze social science information, appreciate the fine and performing arts, and so on. 55

Laying a Foundation for Assessment Success

But one of the purposes of general education is helping students develop the transferrable skills, attitudes, and dispositions that prepare them to transition into a host of careers (Chapter 5). This suggests that general education learning goals should not be requirement-specific but should instead be achieved through multiple requirements, just as program learning goals are achieved through multiple course requirements. Instead of a learning goal that students analyze and evaluate information about social science issues, consider a learning goal that they simply analyze and evaluate information. This suggests that faculty should collaborate across disciplines to identify general education learning goals, rather than asking, say, the mathematics faculty to develop a quantitative skills goal and the English faculty to develop a writing goal.

Learning goals for transfer associate degrees Most community colleges offer an associate degree designed for students who plan to transfer to a four-year college to complete a bachelor’s degree. As discussed in Chapters 2 and 5, usually roughly 45 credits of these programs consist of general education requirements; the remaining 15 credits or so are typically either free electives or what is variously called a track, specialization, concentration, or emphasis. For the 45-credit portion of these degrees, the learning goals should be general education learning goals. The remaining 15 credits may have additional learning goals, or students may develop some general education goals at a deeper or broader level than other students do. Students with social science concentrations, for example, might develop stronger information literacy and analysis skills than students in some other concentrations, while students with visual arts concentrations might develop visual communication skills in addition to the competencies they learn in the general education core.

Institutional learning goals A number of regional accreditors require colleges to articulate institutional learning goals (see Jargon Alert). Begin the process by reviewing your college’s statements of mission, values, and vision, as well as its strategic goals. Often institutional learning goals are embedded in these documents. Some mission statements, for example, aver that graduates will be lifelong learners or committed to service to others. If institutional learning goals are stated or implied in these documents, accreditors may expect them to be assessed.

56

CHAPTER 4 Articulating Student Learning Goals

Focus and limit institutional learning Jargon Alert! goals. Two that especially resonate with faculty, administrators, employers, Institutional Learning Goals policy makers, and other audiences are Institutional learning goals are learning goals that every student at a college, regardless of program, is communication and information literacy. expected to achieve by graduation. Faith-based colleges often have institutional learning goals addressing spiritual development. Chapter 5 discusses several curricular models for ensuring that all students achieve institutional learning goals by the time they graduate. No matter which model you choose, institutional learning goals should be aligned with other learning goals; integrating learning goals across settings is discussed later in this chapter.

Learning goals for short learning experiences This chapter earlier suggested aiming for no more than three to six learning goals per learning experience, when appropriate and feasible. For short programs – certificates, concentrations, minors, co-curricular and other non-credit learning experiences – even three to six learning goals may be too many to achieve realistically in the time you have with students; one or two may be plenty. Learning goals for short learning experiences are also often narrower in scope; a learning goal to “form stress management skills” is more appropriate for a co-curricular learning experience than “communicate effectively.”

Learning goals for co-curricular experiences As with other relatively short learning experiences, co-curricular experiences should have a very limited number of learning goals – perhaps just one or two. Co-curricular experiences ideally focus on learning goals that are collegewide priorities (in other words, institutional learning goals). Three are often particularly apropos: • Interpersonal skills including teamwork, leadership, collaboration, and working with people from diverse backgrounds • Analysis and problem-solving, including creative thinking • Applying classroom learning to solving real-world problems Co-curricular staff are sometimes more comfortable describing what students will do during an experience rather than what students will be able to do after and as a result of the experience. A goal to participate in service activities benefitting others, 57

Laying a Foundation for Assessment Success

for example, describes what students will do during a service learning program but not what they will prepared to do as a result of it. Push everyone to articulate outcomes rather than experiences. Examples of more appropriate learning goals for a service learning program, for example, include enhance the quality of life in one’s community, interact effectively with people of varying backgrounds, and foster and support a constructive climate for accomplishing teamwork. Co-curricular experiences are discussed further in Chapter 8.

Examples of meaningful, effective learning goals Table 4.1 provides examples of learning goals that meet many of the characteristics discussed in this chapter, along with the type of skill or disposition each goal is an example of and the settings (Chapter 2) in which one might see each goal. Table 4.1:  Examples of Effectively Expressed Learning Goals Learning Goal

58

Type of Skill or Disposition

Examples of Possible Settings

Explain the impact of past events on Understanding U.S. international relations today.

Course

Use equipment safely, minimizing the risk of injury.

Performance

Course

Apply economic principles to decisions in everyday life.

Application

Course

Analyze and interpret data.

Analysis

Course, program, or general education

Systematically analyze and solve problems.

Analysis; problem-solving

Program

Advocate and defend one’s views and refute opposing views.

Analysis; evaluation; communication

Course, program, or general education

Judge the effectiveness of the use of color in a work of art.

Evaluation

Course

Research, identify, and justify potential careers.

Evaluation; decision-making; Course or co-curricular experience information literacy

Make appropriate inferences and deductions from information.

Evaluation; information literacy

Course or general education

Design community service projects.

Synthesis

Program, general education, co-curricular experience, or institutional

Synthesis

Program

Conceive of original, unorthodox but feasible solutions to a problem.

Creativity

Program or general education

Write with clarity, coherence, and correctness.

Communication

Program, general education, or institutional

Lead a group to a consensus on a plan of action.

Interpersonal

Course or co-curricular experience

Appraise the strengths and weaknesses of one’s completed work.

Metacognition

Course, program, co-curricular experience, or general education

Use effective time management skills.

Habits of mind

Course or co-curricular experience

Choose ethical courses of action.

Attitudes and values

Program, general education, or institutional

Articulating Student Learning Goals

Design and conduct research studies to test hypotheses or theories.

CHAPTER 4

Table 4.1 (continued)

Integrating learning goals throughout your college Chapter 5 noted that students learn more effectively when their learning is purposefully integrated. For this to happen, learning goals should be integrated across settings. Most course learning goals, for example, should help students achieve a program learning goal, a general education learning goal, or both, as appropriate. A few examples: • Several courses in a program may each help students develop a specific technological skill. Those course learning goals collectively help students achieve a program learning goal to use technologies in the field appropriately and effectively. • A course learning goal that students solve a specific kind of problem helps students prepare to achieve a program learning goal to design appropriate approaches to solving a variety of problems in the discipline. • An English course on Shakespeare might have a course learning goal to analyze scholarly views on character motivations. This learning goal, along with other course learning goals in other English literature courses, prepares students to achieve the English program learning goal to conduct research on issues in the study of literature. Just as some course learning goals should help students achieve program learning goals, some program learning goals should help students achieve institutional 59

Laying a Foundation for Assessment Success

learning goals. A program learning goal that students write effectively in the language of the discipline, for example, helps students achieve an institutional learning goal that students communicate effectively in writing. Similarly, co-curricular learning goals ideally help students achieve a general education or institutional learning goal, such as skills in communication, leadership, and problem-solving. Integrate learning goals only as appropriate.  While interrelated goals are an important aim, it is not practical to expect absolute lockstep relationships among goals in every setting. No one expects every course learning goal to contribute toward a program or general education learning goal, and no one expects every program learning goal to contribute toward an institutional learning goal. An institutional learning goal that students understand diverse cultures could be supported by learning goals in many academic programs but perhaps not by the mathematics program. A program goal to write effectively in the language of the discipline might be supported by goals in several but not all courses in the program. As discussed in Chapter 5, the idea is simply to give students plenty of diverse opportunities to achieve key learning goals.

Putting your learning goals to work Finalizing your learning goals is only the beginning, not the end! Share your learning goals.  Some students learn more effectively when they understand what they are to learn and why (List 26.1 in Chapter 26), so make your learning goals readily available to them. Institutional, general education, and program learning goals can all be included in the college catalog; course learning goals can be included in catalog course descriptions and course syllabi. Be prepared to revise your learning goals.  Writing effective learning goals is an art, not a science. It comes easily to some people and remains a struggle for others. Goals are also more easily crafted for some courses and programs than for others. As with all other aspects of assessment, consider learning goals a work in progress. Be prepared to refine them after you implement them and see how well they work.

Time to think, discuss, and practice 1. A professor has asked his students to write a paper in which they are to discuss a historical event. Under what circumstances would this assignment assess only simple understanding of the event? Under what circumstances would this assignment assess a thinking skill? What thinking skill(s) might it assess? 60

b. Prepare for transfer or graduate study. c. Develop aesthetic appreciation. 3. Write a learning goal for students in a program in which you teach or are enrolled that would focus on developing evaluation, problem-solving, or decision-making skills.

Articulating Student Learning Goals

a. Demonstrate analytic skills.

CHAPTER 4

2. The Landisville College faculty have agreed that they would like to include the following as goals of their general education curriculum. Restate these goals so that they meet the characteristics of effective learning goals discussed in this chapter.

61

CHAPTER 5

Designing Curricula to Help Students Learn What’s Important Some valuable ideas you’ll find in this chapter A program is more than a collection of courses, and a course is more than topics in a textbook. Teaching to the test may not be such a bad thing. It is unfair to place full responsibility for a key program learning goal on one faculty member or one course. Bloated curricula full of elective choices may hinder student success.

63

Laying a Foundation for Assessment Success

C

hapter 1 explained that assessment is part of a four-step process of helping students learn: Establishing clear learning goals, ensuring that students have sufficient opportunities to achieve those goals, gathering evidence of student learning, and using that evidence to understand and improve student learning. If you find yourself struggling to figure out how to assess something, the problem is likely not assessment per se, but the first two steps. After all, if you have clear learning goals and you’re giving students ample opportunity to achieve them, you should already be grading students on their achievement of those learning goals, and there’s your assessment evidence. So the root cause of assessment struggles is often poorly articulated learning goals, a poorly designed curriculum, or both. Learning goals were discussed in Chapter 4; this chapter discusses curriculum design.

What are the characteristics of effective curricula? Effective curricula, whether for a course, program, general education curriculum, or any other learning experience, share the following traits.

An effective curriculum treats a learning goal as a promise All students, no matter what curricular and co-curricular choices they make, deserve to have confidence that, if they complete their work successfully, they will be prepared for what lies next, be it the next course in a sequence, a subsequent program, or their life’s work. Learning goals are promises that we make to our students, their families, employers, and society that every student who passes this course, or completes this learning experience successfully, or graduates with this degree can do the things we promise in our learning goals. An effective curriculum is designed purposefully to fulfill this promise. In other words, it is aligned with its learning goals (see Jargon Alert). Everyone involved with the curriculum has a clear understanding of how and in what ways the curriculum will help students learn, grow, and develop. They can describe what Jargon Alert! students will be able to do upon completion Curriculum Alignment of the experience, and they do their part to Curriculum alignment is ensuring that your course, help students learn to do those things well. program, or general education curriculum is designed to give every student enough opportunity Catalog course descriptions focus not on to achieve its key learning goals. It is an important course content but on what students will be way to make sure that you keep your promises to your students that they will achieve those able to do upon successful completion of learning goals. the course. 64

Backwards Curriculum Design

Jargon Alert! Competency-Based Programs Competency-based programs, a recent arrival on the United States higher education scene, replace traditional courses with a set of learning activities designed to help students achieve and demonstrate the program’s learning goals. Students progress through the program not by completing courses but by completing the learning activities.

Isn’t this teaching to the test?  In a word, yes. As discussed in Chapter 1, the teaching-learning-assessment process begins by identifying what we want our students to learn, then providing them with good opportunities to learn those things, then assessing how well they have learned those things. In other words, effective assessment assesses “what matters most” (Angelo, 1999, p. 3). Teaching to the test gets a bad name when it refers to providing the answers in advance, thereby compromising the integrity of the assessment. Teaching to the test also gets a bad name when the test measures something other than what we value, either because someone else has told us what to assess or because our own tests measure relatively trivial learning. Keep in mind that “assessments must be designed so that the tests are worth teaching to” (Shulman, 2007, p. 24).

Designing Curricula for Student Learning

Backwards curriculum design develops a course, program, or other learning experience by first articulating its key learning goals, then designing the course or program so students achieve and demonstrate those learning goals. Some faculty have described it as first writing the final assignment or exam for a course, then designing the course so that students who complete all its learning activities successfully can earn an A on the final assignment or exam.

CHAPTER 5

Jargon Alert!

Effective curricula are thus designed around learning goals, not content areas. Two useful approaches for accomplishing this are backwards curriculum design (see Jargon Alert) and competencybased program design (see Jargon Alert). A true competency-based program is dramatically different from traditional college programs because it replaces course credits with competencies. But the underlying philosophy, in which all learning experiences are designed around and assessed according to key learning goals, can be applied to any course, program, or other learning experiences.

An effective curriculum is responsive to the needs of students, employers, and society An effective curriculum meets student needs and helps them prepare for success in whatever comes next in their lives. Because the vast majority of today’s college students are preparing for eventual employment, an effective curriculum meets the needs of employers as well as students. It also serves the public good by preparing students to be contributing citizens. 65

Laying a Foundation for Assessment Success

An effective curriculum also explains clearly to students why its learning goals are important and how achieving those goals will contribute to student success (Tinto, 2016) as well as to the public good. As Ken O’Donnell explained, “When they see . . . how college learning can be applied in life and the real world, then they don’t have those nagging questions, ‘Why am I taking this course?’ ‘Is this really the best use of my time?’ ‘Shouldn’t I be earning money?’” (Kuh & O’Donnell, 2013, p. 25). This is especially important for general education curricula and liberal arts programs that don’t lead directly to one specific career path. The information in List 5.1 can be used to help develop a responsive curriculum, be it a program, general education curriculum, course, or co-curricular experience. List 5.1  Information That Can Be Used to Help Ensure That a Curriculum Is Responsive ■■ ■■ ■■

■■ ■■

■■ ■■

Prospective students’ interest in the learning experience and its learning goals Needs and expectations of prospective employers and graduate programs Prospective employers’ demand for students who have completed and achieved the key learning goals of the learning experience Perceptions of the learning experience by employers and other public audiences Characteristics and comparative strengths and weaknesses of competing learning experiences, at this college and elsewhere Regional and/or national trends in the discipline The regional climate for higher education and this learning experience, including public and private support

An effective curriculum is greater than the sum of its parts Imagine a construction program that consists of courses on all aspects of construction – plumbing, electricity, carpentry, building codes, construction plans, and so on – but nothing more. A graduate would be able to do each of those things well, but would you want to hire her to build an addition onto your house? Of course not—to be a competent contractor, a graduate would need to know not just how to do each of these things, but how to connect those skills in order to build an addition in which all the pieces come together appropriately and effectively. A program is more than a collection of courses, and a course is more than topics in a textbook. A curriculum is holistic and coherent. The activities in a coherent course and the courses in a coherent program all integrate with one another (Cullen, Harris, & Hill, 2012) to fulfill their purpose: student achievement of key learning goals and preparation for post-college success. The curriculum is judged not by the program’s individual courses or the course’s activities, but by the overall result – student achievement of the curriculum’s key learning goals – which should be greater than the sum of its parts. 66

to achieve its key learning goals

Designing Curricula for Student Learning

“A college experience should approach learning as a process – one that is cumulative, iterative, multidimensional and, most importantly, self-sustaining long beyond graduation” (Salisbury, 2013, para. 5). Effective curricula are designed to ensure that students have multiple, iterative opportunities to develop and achieve key learning goals, through a variety of learning activities and settings. There’s a simple reason for this: We learn best through repeated practice in varied contexts. It’s simply unfair to place full responsibility for student achievement of a key course goal on just one assignment. Similarly, it’s unfair to place full responsibility for student achievement of a key program or general education learning goal on one faculty member or one required course. For a program goal that students write effectively in the language of the discipline, for example, the curriculum should ensure that all students in the program take multiple courses in which they learn how the discipline defines good writing and how to write in the discipline, and they receive constructive feedback on their writing.

CHAPTER 5

An effective curriculum gives students ample and diverse opportunities

An effective curriculum has appropriate, progressive rigor Consider how some of the U. S. regional accreditors’ standards define programs (emphases added): • “Appropriate breadth, depth, rigor, sequencing, time to completion, and synthesis of learning” (Accrediting Commission of Community and Junior Colleges, Western Association of Schools and Colleges, 2016, p. 14) • “Characterized by rigor [and] coherence . . . designed to foster a coherent student learning experience and to promote synthesis of learning” (Middle States Commission on Higher Education, 2014, p. 7) • “Rigor that [is] consistent with mission . . . A coherent design with appropriate breadth, depth, sequencing of courses, and synthesis of learning” (Northwest Commission of Colleges and Universities, 2010, p. 6) There’s a theme here: an effective program curriculum is appropriately rigorous, and students develop more advanced skills as they progress through the program. This means that some courses must be more advanced than others and build on what’s been learned in earlier courses. Degree programs with effective curricula thus include some meaningful study at an advanced level that requires a prerequisite or some kind of advanced standing. This in turn means that course numbering systems must have meaning. Courses numbered in the 200s should be more advanced than introductory 100-level courses. Courses numbered in the 300s should have junior-level depth 67

Laying a Foundation for Assessment Success

and rigor. Courses numbered in the 400s should be more advanced than 300level courses. And graduate courses should require more advanced study than undergraduate courses. The principle of appropriate, progressive rigor applies to courses and other learning experiences as well as to programs. Every learning experience should be appropriately rigorous. Even a co-curricular experience should not aim to develop high-school-level skills, and assignments at the end of a course should build on and be more rigorous than what was learned earlier in the course.

An effective curriculum concludes with an integrative, synthesizing capstone experience An effective curriculum, whether a course, program, general education curriculum, or other learning experience, expects students to pull the pieces together into a coherent whole. Indeed, several U.S. regional accreditors explicitly expect academic programs to require synthesis of learning. The best way to do this is with a capstone requirement (see Jargon Alert), especially one Jargon Alert! that incorporates research-informed learning Capstones strategies (List 26.1 in Chapter 26), such A capstone (Hauhart & Grahe, 2015) is a holistic as working collaboratively and addressing activity that students complete as they approach the end of a learning experience. Capstones give complex, real-life problems. Program or students an opportunity to see the big picture: To general education capstones are especially integrate, synthesize, apply, and reflect on what they have learned throughout their studies. helpful for students who have swirled through several colleges or programs because they can reinforce key skills and competencies such as writing, critical thinking, and information literacy. Examples of program capstone experiences include senior projects, such as research papers; performances and exhibitions; independent studies; field experiences such as internships, clinicals, practicums, and service learning experiences; and theses, dissertations, and oral defenses. Examples of course capstone assignments include portfolios (Chapter 18), independent and team projects, and reflective papers (Chapter 20).

An effective curriculum is focused and simple Once upon a time, college catalogs burst with thousands of course offerings, but those days are ending. Colleges are increasingly recognizing that focused, streamlined curricula have several advantages, not the least of which is that they free up time for other important work, including assessment (Chapter 12). 68

Designing Curricula for Student Learning

Simple, basic curricula facilitate transfer.  Students who attend multiple colleges in the United States before earning a degree face a variety of barriers, all flowing from one fact: While one of the great strengths of U.S. higher education is its diversity, this means that U.S. colleges and programs do not all offer the same curricula and do not have the same degree requirements (Suskie, 2014). Courses completed at one college or in one program may not count toward another college’s requirements. The more distinctive a college’s curricular requirements, the harder it is for transfer students to complete a degree in a timely fashion. If a college’s general education curriculum requires a course addressing social issues unique to the college’s region, for example, a student who has completed all the general education requirements at another college may still need to take this additional general education course.

CHAPTER 5

Focused curricula improve student success and completion.  Structured or guided curricular pathways in community colleges increase students’ retention and ­ graduation rates (Bailey, Jaggars, & Jenkins, 2015). Some community colleges now offer meta-majors: focused pathways that aim to increase student success by ­preparing students for a cluster of four-year college majors, such as health sciences, ­engineering, or the humanities, rather than just one specific major (Waugh, 2016).

An effective curriculum uses research-informed strategies to help ­students learn and succeed Jargon Alert! High Impact Practices High impact practices (HIPs) are educational experiences that make a significant difference in student learning, persistence, and success because they engage students actively in their learning (www.aacu.org/resources/high-impact-practices). They include: • First-year experiences • Learning communities • Writing-intensive courses • Collaborative learning experiences • Service learning • Undergraduate research

The past few decades have yielded a remarkable body of research on strategies that help students learn and succeed (List 26.1 in Chapter 26). Effective curricula incorporate these strategies, especially highimpact practices (see Jargon Alert). Effective curricula particularly recognize that students spend their learning time and energies on what they will be graded on. This means that tests, graded learning activities, and grading criteria all focus on the key learning goals of the curriculum. Chapter 16 discusses this further.

• Field experiences • Capstone courses and projects

69

Laying a Foundation for Assessment Success

An effective curriculum is consistent An effective curriculum provides consistent learning experiences, with consistently rigorous expectations for student learning, wherever and however it offers learning opportunities: online, face-to-face, on the college campus, or at a remote location. As a matter of integrity, every student who passes a course, no matter the section in which he or she enrolls, should have achieved the course’s key learning goals and be prepared for whatever comes next. This cannot happen without careful, consistent oversight and coordination, which takes time and resources. It’s a good idea for faculty teaching sections of a course to collaborate in order to develop a shared core package that provides the course’s key learning goals that all students must achieve and perhaps a common capstone assignment or some questions to be included on all final exams. Beyond that core, faculty can flesh out their own syllabi and class materials as they see fit. Collaboration and academic freedom are both discussed in Chapter 13.

Curriculum design in specific settings Here are some tips for curricula in some of the learning settings discussed in Chapter 2.

Courses Course learning goals aren’t course topics; they’re the skills and competencies that students learn throughout a course, through iterative, progressive learning activities. If a course learning goal is that students develop an appreciation of other cultures, for example, the syllabus should include a series of assignments and classwork designed to help students develop this appreciation. Course curriculum maps, discussed later in this chapter, are a great tool to help ensure that a course curriculum truly focuses on important learning goals.

General education curricula The following traits of effective curricula are especially important for general education. Effective general education curricula treat a learning goal as a promise.  Ensure that every undergraduate who completes your general education requirements, no matter which general education courses or sections he or she has chosen, is competent at every general education learning goal. 70

Designing Curricula for Student Learning

Effective general education curricula give students ample and diverse opportunities to achieve key learning goals.  This may sound impossible, because many general education curricula require students to complete just one course for each of a variety of requirements (say, one math course, one natural science course, one art course, and so on). But a good general education curriculum is greater than the sum of its parts, with overarching goals that are addressed in multiple requirements, as suggested in Chapter 4. Some community colleges, for example, require that every general education course address critical thinking, and some colleges require quantitative skills to be addressed in general education social science and natural science courses as well as in mathematics courses. This helps ensure that, no matter how long students enroll at your college or what courses they take, they’ll leave with stronger skills than when they arrived.

CHAPTER 5

Effective general education curricula are relevant and responsive to the needs of students and other stakeholders,  including employers and society (Maxwell, 2013). It’s not enough to tell students they must take certain courses or fulfill certain requirements “because it’s good for you” or “someday you’ll thank me for making you do this” (Suskie, 2014). Throughout your general education curricula, emphasize skills that employers are seeking (Chapter 4) such as teamwork and problem-solving. Look to the colleges participating in AAC&U’s “Give Students a Compass” initiative (www.aacu.org/compass) for models and ideas.

Effective general education curricula for bachelor’s degrees conclude with an ­integrative, synthesizing junior-level capstone experience.  As noted earlier, these capstones are especially helpful for students who have swirled through several ­colleges or programs, because they can reinforce key general education learning goals such as writing, critical thinking, and information literacy. Effective general education requirements and course offerings are limited, simple, and traditional.  A plain vanilla natural science requirement or art history survey course has the best chance of meeting general education requirements at other colleges, should a student decide to transfer. Limiting the number of offerings can also lead to significant savings in faculty time, as discussed in Chapter 12. Traditional doesn’t mean boring or irrelevant, of course. You can focus an art history survey course, for example, on artwork particularly likely to engage your students and meet their interests and needs.

Institutional learning goals As noted in Chapter 4, several regional accreditors require colleges to articulate and assess institutional learning goals. There are several models for designing curricula to address institutional learning goals. 71

Laying a Foundation for Assessment Success

In the general education curriculum.  Because all undergraduates, regardless of major, must achieve general education learning goals by the time of graduation, a college’s general education learning goals may be considered its institutional learning goals. In all degree programs.  Faculty may agree that certain learning goals are so important that they are addressed in every program; if so, these are institutional learning goals. Writing and critical thinking are two examples of potential institutional learning goals that might be addressed this way. In both the general education curriculum and all degree programs.  Faculty may agree that certain learning goals are so important that they are addressed in all ­programs and in the general education curriculum. If so, these are institutional learning goals. Hamilton College, for example, characterizes itself as “a national leader for teaching students to write effectively” (www.hamilton.edu/about), and it emphasizes this skill throughout all curricula. This lets students develop writing skills from both a liberal arts and a disciplinary perspective. A first-year general ­education writing course might, for example, address writing from a broad liberal arts ­perspective, while courses in each academic program address writing in the discipline. In co-curricular requirements.  Some college statements of mission and values articulate institutional learning goals that are addressed outside of general education and program curricula. A faith-based college may have goals related to spirituality that students develop through participation in co-curricular activities rather than – or in addition to – the general education curriculum and their major.

No matter which of these models your college uses to address institutional learning goals in its curricula, make sure that every student, regardless of major or co-curricular involvement, has ample and unavoidable opportunity to achieve every institutional learning goal before graduating. If an institutional learning goal is to commit to service to others, for example, simply offering elective service opportunities that students may or may not participate in does not suffice.

Pick-from-a-list curricula Some colleges offer what might be called pick-from-a-list curricula in which students are required to choose one or more courses from a list to fulfill a program or general education requirement. A general education curriculum, for example, may let students fulfill its science requirement by choosing from a list of science courses, while an English program may require students to choose two courses 72

Some colleges let students design their own majors or concentrations, sometimes by picking from a list, as described above, and sometimes by giving them freer rein. Self-designed programs can meet the needs of students for whom existing programs aren’t a good fit. They can also help students who have swirled through multiple colleges and programs complete a degree by packaging completed courses in a unique but meaningful way. But self-designed programs can also be an assortment of unrelated courses or experiences. Without sufficient curricular structure or guidance, students may be tempted to choose courses that fit their schedule or whose professor they like rather than courses that will help them move ahead purposefully. Effective selfdesigned programs have the characteristics of any other effective curriculum. In particular, they: • Have clearly articulated learning goals. • Are responsive to the student’s needs. • Are greater than the sum of their parts. • Give students ample and diverse opportunities to achieve key learning goals. • Have appropriate, progressive rigor. • Conclude with an integrative, synthesizing capstone experience. Self-designed programs can have these characteristics if students are required to do the following: • Articulate the key learning goals of their program: The key things they would like to be able to do by the time they graduate. • Choose courses that will help them achieve their learning goals iteratively, at appropriately and increasingly rigorous levels. (This may require having more information about potential courses than catalog descriptions.) • Have their learning goals and course plan approved by an academic advisor.

Designing Curricula for Student Learning

Self-designed programs

CHAPTER 5

in American literature from a list of eight. Some academic programs are almost entirely pick-from-a-list, giving students no shared curricular experience beyond, perhaps, a required introductory course. Pick-from-a-list curricula can be effective if the lists are built around program learning goals rather than or in addition to content. Faculty might decide, for example, that all courses fulfilling a general education social sciences requirement must help students develop information literacy skills. Faculty in a history program might decide that every course offered to fulfill the Asian history requirement must help students learn how to make analytical comparisons. This helps ensure that, no matter which courses students pick from the list, they’ll spend enough time on the curriculum’s key learning goals.

73

Laying a Foundation for Assessment Success

74

• Periodically review their goals and progress with an academic advisor, and be prepared to modify their goals and course plan as they move through the program and discover new interests or needs. • Complete an integrative, synthesizing capstone experience such as an independent study project or field experience.

Interdisciplinary programs Interdisciplinary programs often face two curricular challenges. First, they are often pick-from-a-list programs with the challenges discussed earlier. Second, their courses often largely serve students in other programs. Students in an International Studies interdisciplinary program might take a course on the economy of South Asia, for example, that is probably designed to achieve the learning goals of the Economics program. The Economics program learning goals may not align with the International Studies program learning goals, and faculty teaching the course on the economy of South Asia may not be able or willing to address International Studies program learning goals as well. To address these challenges: • Invite faculty teaching potential courses for the interdisciplinary program to help articulate the interdisciplinary program’s key learning goals. • Review the syllabi of potential courses for the interdisciplinary program. Use a curriculum map, discussed later in this chapter, to identify which courses address each program learning goal. • If a potential course does not appear to address any of the interdisciplinary program’s learning goals, see if the faculty teaching the course are willing to incorporate any of the learning goals into the course. If not, the course should be removed as a choice in the interdisciplinary program. • Make sure that the interdisciplinary program has enough structure to ensure that every student will achieve its learning goals before graduating. Often this means, at a minimum, requiring an entry-level course that introduces students to the program of study plus a capstone experience in which students integrate what they have learned throughout their studies. The program might have a pick-from-a-list design in which lists of potential courses are organized around program learning goals. Some interdisciplinary programs have so few students that it isn’t costeffective to offer a course or seminar as the capstone experience. In these situations, a field experience, independent study, or course of directed readings may be an appropriate capstone experience if it emphasizes and reinforces the program’s key learning goals.

Effective transfer associate degree programs are meaningfully defined.  This is discussed in Chapter 2. Effective transfer associate degree programs are coherent.  They are more than a collection of courses, greater than the sum of their parts. They have clearly articulated learning goals that go beyond general education learning goals (Chapter 4), and they are purposefully designed to help students achieve those goals. There are two basic ways to use the additional 15 to 18 credits beyond general education requirements to accomplish this. One is to make those additional credits into meaningful, coherent concentrations; the other is to make them into meaningful self-designed programs in which students identify their own learning goals and curricular choices under the guidance of an academic advisor, as suggested earlier. As already noted, there is growing evidence that guided curricular pathways increase student success and completion rates, suggesting that the first approach may be more effective.

Designing Curricula for Student Learning

As noted in Chapter 2, many community colleges offer associate degrees designed for transfer into bachelor’s degree programs. They often consist of about 42 to 45 credits of general education requirements plus an additional 15 to 18 credits that may be free electives or a specialization, concentration, track, or emphasis. Effective transfer associate degree programs have the same characteristics as any other effective curriculum. Three of these characteristics may be particularly challenging.

CHAPTER 5

Transfer associate degree programs

Effective transfer associate degree programs have appropriate, progressive rigor and conclude with an integrative, synthesizing capstone experience.  Simply put, students must complete at least one 200-level course that cannot be taken the first semester. A program that’s only a collection of 100-level courses does not constitute a meaningful associate degree program, and it does not prepare students for juniorlevel study once they transfer. Unfortunately, some community colleges feel pressure from the Lumina Foundation’s completion agenda (www.luminafoundation.org/goal_2025) to eliminate course prerequisites, thinking that prerequisites may represent barriers to completion. In some systems and states, some community college faculty say that their counterparts at local four-year colleges don’t want them to teach anything beyond introductory courses – the four-year faculty want to teach the rest themselves. (My reaction? What snobs.) Passaic County Community College (2016) has a creative solution. Each of its degree programs has designated one 200-level course as its program capstone. For its 75

Laying a Foundation for Assessment Success

English AA program, for example, the capstone course is “Topics in Literature,” and for the Psychology option in the Liberal Arts AA program, the capstone course is “Social Psychology.” While the program capstone courses have no course prerequisites, they are open only to students who have completed at least 45 credits and have taken at least one other course in the discipline. This forces students to take the course as they approach graduation, making the course an ersatz capstone experience.

Co-curricular experiences An excellent co-curricular experience is not offered in isolation but is integrated with other offerings both inside and outside of the classroom (Suskie, 2015), and the co-curricular team works with colleagues to help students see connections in their learning. Co-curricular experiences are discussed further in Chapter 8.

Curriculum maps Jargon Alert! Curriculum Map A curriculum map is a chart identifying the key learning goals addressed in each of the curriculum’s key elements or learning activities.

Curriculum maps (see Jargon Alert) are excellent tools to ensure that a course, program, general education curriculum, or other learning experience is designed to give students enough opportunity to achieve its key learning goals.

Course curriculum maps A curriculum map for a course can be a simple three-column chart (Bain, 2004; Fink, 2013) (Exhibit 5.1). The first column lists the course learning goals, the second column lists the learning activities that will help students achieve each course learning goal, and the third column lists the graded assessment that will demonstrate to the faculty member how well students have achieved the course learning goal. A general education art appreciation course, for example, might list in the first column “Analyze your own and others’ responses to a work of art” as a course learning goal. In the second column, students might learn how to do this through “Small group discussions of critical responses to artworks.” In the last column, students might demonstrate their achievement of this goal through a “Paper comparing your response to a work of art to those of critics.” Here are some tips on creating course curriculum maps.

76

This is how you’ll learn how to do it (homework, classwork, assignment, and so on).

This is how you’ll show me that you’ve learned how to do it (test, paper, project, presentation, and so on).

Start with the first column, then fill out the last column, then fill out the middle column  (Fink, 2013). This ensures that course learning activities help prepare students for the capstone projects and other key assignments that they’ll be graded on.

Designing Curricula for Student Learning

This is what you’ll learn how to do (course learning goal).

CHAPTER 5

Exhibit 5.1  Template for a Three-Column Curriculum Map for a Course Syllabus

Make the assessments in the last column meaningful.  Quizzes and homework problems are a good way to check if students are on track but not substantive enough to confirm that they have achieved significant course learning goals. Make the learning activities in the middle column active and meaningful.  Listening to a lecture, reading a textbook, and class discussions can be important parts of the learning experience but are generally insufficient because they don’t actively engage every student. If your curriculum includes these experiences, follow up on them with hands-on learning activities that get students thinking and doing, such as writing, answering questions, or creating or completing something. Chapter 16 offers suggestions. For some learning goals, the second and third columns may be the same.  For example, students may develop information literacy skills (second column) through a library research project, and their skills may be assessed (third column) by grading the completed library research project. Or the second and third columns may be different; students may develop a math skill by doing homework problems (second column), and their skills may be assessed (third column) through an exam. Sometimes multiple learning goals can be assessed through a course ­capstone.  The third column may list the same assessment for several learning goals. Learning goals for writing, critical thinking, information literacy, and ethics may all be assessed through a research paper, for example. 77

Laying a Foundation for Assessment Success

Consider making this a four- rather than three-column chart.  After the first column (“This is what you’ll learn how to do”), insert one more column, titled “And learning this will help you learn how to . . .” (Exhibit 5.2). In this additional column, list the program, general education, or institutional learning goal that the course learning goal helps students achieve. For the art appreciation course mentioned earlier, for example, the second column might say that achieving the course learning goal “Analyze your own and others’ responses to a work of art” will help students achieve the general education learning goal to “Think critically and analytically.” The additional column thus ensures alignment between course and program learning goals.

Exhibit 5.2  Template for a Four-Column Curriculum Map for a Course Syllabus This is what you’ll learn how to do (course learning goal)

And learning this will help you learn how to (program, general education, or institutional learning goal)

This is how you’ll learn how to do it (homework, classwork, assignment, and so on)

This is how you’ll show me that you’ve learned how to do it (test, paper, project, presentation, and so on)

Curriculum maps for programs, general education, and institutional learning goals A curriculum map for a program, a general education curriculum, or institutional learning goals is also a simple chart. It compares key learning goals against curricular requirements: Required courses and perhaps co-curricular requirements such as a service learning experience. Some learning management systems and assessment information management systems (Chapter 11) include modules for curriculum maps. Many program curriculum maps list the key program learning goals down the left column and program courses and co-curricular requirements across the top (Exhibit 5.3). It’s fine, however, to flip these two axes, listing the program learning goals across the top and courses and other curricular requirements down the left column, especially for programs with extensive course offerings. Then check off the learning goal(s) addressed by each curricular requirement. 78

Learning Goal

Collect, analyze, and evaluate evidence and information

✓

Capstone Requirement

✓

✓

✓

✓

Lead a team charged with solving problems Convey findings and solutions clearly in writing

Advanced Course

✓

✓

✓

✓

✓

Keep in mind the purpose of curriculum maps for programs, general education curricula, and institutional learning goals: They are intended to help us analyze whether the curriculum gives students enough opportunity to achieve the curriculum’s key learning goals. Meaningful curriculum maps thus have the following characteristics:

Designing Curricula for Student Learning

Choose and apply appropriate problem-solving methods and tools

Introductory Problem-Solving Course Methods Course

CHAPTER 5

Exhibit 5.3  Curriculum Map for a Hypothetical Certificate Program

Electives have no place in a program curriculum map.  (This simplifies things, doesn’t it?) Again, the purpose of the curriculum map is to make sure that every student, regardless of curricular choices, has ample opportunity to achieve key learning goals. By definition, electives aren’t taken by everyone. While they may help students who participate in them achieve certain learning goals, they’re not useful in a curriculum map. Group clusters of pick-from-a-list courses together in one column.  As discussed earlier in this chapter, some curricula offer clusters of courses and ask students to choose one, two, or more of those courses. If the courses in a cluster all address one or more shared learning goals, group them together in the curriculum map, in one column or one row, depending on how your map is designed. If the courses do not share any key learning goals, treat them as electives and omit them from the map. Check off a learning goal for a required course (or cluster of courses) only if a significant part of the final course grade is based on progress toward achieving the learning goal.  A quantitative skills program learning goal, for example, might be checked off only if at least, say, five percent of the final course grade is based on students’ quantitative skills. This is because the purpose of curriculum maps is to identify where students really work on a learning goal, and students focus their learning time and energies on what they’re graded on. Course and program curriculum maps should sync.  In other words, program learning goals should be checked off on a program curriculum map only if they show up on the course curriculum map discussed earlier. 79

Laying a Foundation for Assessment Success

Use codes instead of checkmarks only if they’re meaningful and helpful.  Some learning management systems and assessment information management systems ask faculty to use codes instead of checkmarks on curriculum maps, indicating whether students are introduced to, develop, or demonstrate satisfactory achievement of the learning goal. Such codes help demonstrate that the program has progressive rigor – one of the characteristics of effective curricula. They also help identify assessment opportunities; the courses in which students are supposed to demonstrate satisfactory achievement are often ideal places for assessment. But not all curricula work this way. The assumption of these codes is that students take at least three courses addressing each learning goal, one in which they are introduced to it, one in which they develop it, and one in which they demonstrate satisfactory achievement of it. In relatively short programs, such as certificates and general education curricula for career associate degrees, it’s not practical or appropriate to spend at least three courses addressing a particular learning goal at increasing levels of rigor. And sometimes students aren’t introduced to a learning goal; they just plunge into developing it in a variety of courses. So if codes aren’t meaningful or helpful, think twice about using systems that require this kind of coding.

Analyzing curriculum maps Step back, look at the completed curriculum map – whether for a course or program – and ask yourself what it tells you about how well the curriculum meets the traits of effective curricula. Is every learning goal addressed in the curriculum?  You may find that your curriculum addresses all its key learning goals well. Or you may find that a key learning goal is addressed nowhere or – for a program or general education curriculum – in only one or two elective or pick-from-a-list courses that relatively few students take. This makes it possible for students to graduate without having taken any courses addressing that key learning goal.

80

Is every learning goal addressed in multiple learning experiences?  You may find that, while all students have an opportunity to achieve a key learning goal, that opportunity is not enough for them to achieve it well. Perhaps a key course learning goal is addressed only through one brief homework assignment, for example. Or perhaps the public speaking skills expected in a program are addressed in only one course, giving students inadequate opportunity to practice and improve their public speaking skills. Note that, in the example in Exhibit 5.3, most learning goals are addressed in at least two units, and some especially critical goals are addressed in three. Most curricula are already packed to the gills, so how can faculty add more intensive study of a particular goal? List 5.2 offers some suggestions.

■■

■■ ■■

■■

■■

■■

Designing Curricula for Student Learning

■■

Make some tough choices regarding learning goal priorities. Reduce attention to some less important goals to make additional room for more important ones. Drop a less critical requirement or scale back coverage of less important concepts. Increase the credit value of a key course, or spread a curriculum over two courses. Students may be more successful in Calculus I, for example, if they can study it in two three-credit courses rather than in one four-credit course. Replace a program elective with a required capstone course that reinforces the learning goal. Require students in their last semester to complete an independent project that emphasizes this goal. If you want to help students learn how to make oral presentations, consider requiring graduating students to make oral presentations on their projects at a department research conference. Require students to take an appropriate course in another department as a general education requirement, if the goal is a transferrable skill such as conducting statistical analyses. (If you elect this approach, you still need to build attention to these skills within your own curriculum, so students learn the nuances of applying the skill to your discipline.) Give students more responsibility for learning on their own. Consider a flipped class (see Jargon Alert), in which students learn basic content knowledge outside of class, perhaps by reading the textbook or working with peers, then spending class time on key learning goals. If students need to strengthen their writing skills but you don’t have the time to read every draft, periodically ask students to read and comment on each other’s drafts (Chapter 13). Or have students strengthen their writing skills by composing summaries of concepts that classmates can use as supplemental reading or study guides. Look for ways to use class time more effectively. If students need to strengthen their oral presentation skills, but you don’t have time to hear individual oral presentations, have students teach key topics to the class. Or have students make group presentations rather than individual ones.

CHAPTER 5

List 5.2  Strategies to Add More Intensive Study of a Key Learning Goal

Does the curriculum conclude with an integrative, synthesizing capstone ­experience?  A capstone is not only a great learning opportunity for students but an opportunity to obtain evidence of how well students achieve key curricular Jargon Alert! learning goals.

Flipped Classrooms

For generations, many college courses have been designed so that students are first taught in the classroom and then they reinforce and deepen their learning through homework. In a flipped classroom, students first learn basic content outside of the classroom, and then they reinforce and deepen their learning in the (face-to-face or online) classroom. Class time is spent in interactive engagement with fellow students, under the guidance of the professor, applying learning to new situations, analyzing, evaluating, creating, and, above all, thinking.

Is the curriculum sufficiently focused and simple?  Remember that a complex ­curriculum with myriad options can be a barrier to degree completion; curricula with clear pathways are often more effective. Does the curriculum use research-informed strategies to help students learn and succeed?  Does a program curriculum include, for example, high impact practices 81

Laying a Foundation for Assessment Success

such as learning communities, writing-intensive courses, service learning, undergraduate research, or field experiences?

Other tools to understand curricula Transcript analysis, syllabus analysis, and enrollment patterns can all be helpful tools in understanding program and general education curricula. Transcript analysis.  For a pick-from-a-list curriculum, or one with some pickfrom-a-list components, consider reviewing a sample of transcripts of graduating students to learn which courses they chose and when they took them. This analysis can help answer questions such as the following: • Which courses do students usually take to fulfill college, general education, or program requirements? • When do they take them: In the intended sequence or, say, by postponing a fundamental course such as mathematics or public speaking until just before graduation? • Are students achieving a general education learning goal through courses in their majors? • Does the rigor of learning experiences build as students progress through the curriculum? Syllabus analysis.  Reviewing course syllabi can help answer questions such as the following: • Do students have enough learning activities to achieve course, program, general education, and/or institutional learning goals, as appropriate? • Do learning experiences connect and build in rigor as students progress through the curriculum? • Are standards reasonably consistent across sections of a course? • Do course curricula reflect research-informed practice? Enrollment patterns.  If you have a pick-from-a-list curriculum, looking at enrollments by course may help identify curriculum bloat. But if your curriculum includes courses that may be taken by students outside the program (for example, courses that may count toward either a major or a general education requirement, or courses that count toward more than one major), transcript analysis may be more helpful in identifying how many students in your program are enrolled.

82

List 5.3  Strategies to Improve a Program or General Education Curriculum ■■

■■

■■

■■ ■■ ■■

■■ ■■ ■■ ■■ ■■

Give new or increased attention to certain key learning goals in appropriate courses through additional learning activities. Drop some key learning goals that may have sounded like good ideas but aren’t really being addressed in the curriculum. Modify the curriculum so it better meets the needs of students, employers, and/or society by modifying some existing courses, adding one or more new courses, or both. Have certain courses more purposefully reinforce what students learn in other courses. Replace pick-from-a-list curricula or curricular components with more focused pathways. Either renumber some courses or modify their standards so course levels align better with expected rigor. Add or improve a capstone requirement. Reduce curriculum bloat by dropping low-enrollment courses from curriculum options. Increase the use of high-impact practices and other research-informed strategies. Develop a core syllabus for multisection courses. Improve communications among faculty teaching at multiple locations to enhance consistency.

Designing Curricula for Student Learning

Remember that the purpose of creating and analyzing a curriculum map is to make sure that the curriculum has the traits of an effective curriculum discussed at the beginning of this chapter. Your analysis of a program, general education, or institutional learning goal map and the other tools discussed here may lead you to conclude that the curriculum is fine, or it may lead to consideration of some potential changes in the curriculum such as those in List 5.3.

CHAPTER 5

Using curriculum maps and other tools to improve curricula

Curriculum review is not a one-and-done process Curricula are never static. As faculty, student, societal, and employer needs change, so may curricula. A course, program, or other learning experience that met the traits of effective curricula a few years ago may no longer do so today. So it’s a good idea to review curriculum maps and syllabi every few years to make sure they’re accurate and that the curricula continue to meet the traits of effective curricula. Program curriculum maps and syllabi can be reviewed as part of program review (Chapter 26). Curriculum maps and syllabi for general education and institutional learning goals can be reviewed by a general education committee or other appropriate group on a rolling multiyear schedule. For example, courses that meet the general education science requirement can be reviewed one year, while courses that meet the humanities requirement can be reviewed the next. 83

Laying a Foundation for Assessment Success

For More Information Three excellent resources on curriculum design are Designing and Assessing Courses and Curricula: A Practical Guide (Diamond, 2008), Creating Significant Learning Experiences: An Integrated Approach to Designing College Courses (Fink, 2013), and The Learner-Centered Curriculum: Design and Implementation (Cullen, Harris, & Hill, 2012). The Quality Matters Program (www.qmprogram .org) has a rubric of eight standards for online courses (Quality Matters Program, 2011), many of which are applicable to face-to-face courses as well. Two excellent resources on general education and liberal arts curricula are the Association of American Colleges & Universities (www.aacu.org) and the Association for General and Liberal Studies (www.agls.org).

Time to think, discuss, and practice 1. Think of a course, program, or other learning experience in which you’ve taught or been enrolled. How well did it meet the traits of effective curricula discussed in this chapter? Can you think of any adjustments that might be made to improve its effectiveness? 2. Try creating – and analyzing – a course curriculum map for a course that you’ve taught or in which you’ve been enrolled. 3. Choose a degree program offered by your college and review its curriculum as described in your college’s catalog or website. Does it seem appropriately lean and focused? If not, brainstorm some strategies to streamline it that the program’s faculty might take under consideration.

84

CHAPTER 6

How Will Your Evidence of ­Student Learning be Used? Some valuable ideas you’ll find in this chapter Assessment has three basic purposes: stewardship, accountability, and, most important, giving students a great education. Don’t undertake any assessments without a clear understanding of the decisions that the resulting evidence will inform. Assessment committees and coordinators don’t really use student learning evidence to inform decisions; they’re conduits, getting the evidence to the people who will use it.

85

Laying a Foundation for Assessment Success

T

oday too many faculty and staff are spending too much time and energy collecting evidence of student learning that ends up on a (perhaps virtual) shelf. Yes, they may disseminate the evidence and perhaps briefly discuss it in a meeting, but the evidence is never really used in any substantive way. Why? Invariably, faculty and staff plunged into assessment without thinking first about why they were conducting the assessment (other than to get through the next accreditation review), who needed the resulting evidence, and what decisions the evidence would inform. The principle of form following function applies here. Once you understand why you are assessing and the decisions that the resulting evidence of student learning is likely to inform, you can design assessment processes to yield evidence that will be truly useful and used rather than sitting on that shelf.

Why are you assessing? Assessment has three fundamental purposes: Ensuring and improving educational quality, stewardship, and accountability. This chapter introduces these three fundamental purposes. Using evidence of student learning to ensure and improve educational quality and for stewardship is discussed in Chapter 26; using evidence of student learning for accountability is discussed in Chapter 25.

Assessment’s fundamental purpose: Giving students the best ­possible education Today’s most effective faculty are dedicated teachers who want to do the best possible job helping their students learn – and want and need feedback in order to do so. They are part of a revolution in higher education that Robert Barr and John Tagg (1995) described as a new paradigm for teaching and learning. Barr and Tagg observed that, historically, many faculty viewed their responsibility as providing instruction rather than fostering student learning. Under this traditional or teaching-centered paradigm, faculty taught primarily by presenting lectures and assigning readings and homework. They believed that, if students didn’t learn the material and earned poor grades, the fault was with the students, Jargon Alert! not themselves. Such faculty felt no sense Learning-Centered of responsibility to reach out to students Learning-centered (Weimer, 2013) classrooms are proactively and help them learn. those in which students are actively engaged in Today faculty and staff increasingly their learning, and where faculty and students share responsibility for learning. A learningfollow a learning-centered paradigm centered college actively fosters these practices. (see Jargon Alert) that draws on significant 86

Using Evidence of Student Learning

Does assessment really improve learning?  In a word, no. “Assessment all by itself is an insufficient condition for powerful learning and improvement” (Schneider & Shulman, 2007, p. viii). It is not the assessment itself, but how faculty, staff, and college leaders use it that can lead to improvement in student learning. But does assessment have even this impact? Assessment has certainly led to changes in teaching, as evidenced by case studies compiled by the National Institute for Learning Outcomes Assessment (NILOA) (www.learningoutcomesassessment .org/TFComponentUSLE.htm) and others (Baker, Jankowski, Provezis, & Kinzie, 2012; Banta, Lund, Black, & Oblander, 1996; Bresciani, 2006; and Bresciani, 2007). But have these changes in teaching resulted in improved learning? Most chief academic officers agree that assessment has improved the quality of teaching and learning at their college (Jaschik & Lederman, n.d.; Peterson & Einarson, 2001). But systematic, rigorous evidence is scant except for one important study conducted several years ago (Lattuca, Terenzini, & Volkwein, 2006). Why this paucity of systematic evidence that assessment leads to better learning? One reason is that disappointing outcomes may be due to a variety of factors, including poorly articulated learning goals, a curriculum not designed to help students achieve those goals, ineffective teaching methods, and inadequate or inappropriate support services, among others. The specific causes of and solutions for disappointing outcomes are rarely obvious. So it may take some time and trial and error before faculty and staff can identify the best changes to make, implement those changes, and see the impact of those changes. Another reason for the paucity of systematic evidence that assessment leads to better learning is that assessment is action research (Chapter 1), not experimental research. Action research is context-specific, informal, and designed only to inform individual practice. It does not have the precision, rigor, or generalizability of experimental research. These defining characteristics make it difficult to substantiate the impact of assessment on learning improvement.

CHAPTER 6

research on how to help students learn and succeed that is summarized in List 26.1 in Chapter 26. While there will always be some unmotivated students who deserve to fail, learning-centered faculty and staff assume that, if a significant number of their students do not achieve their learning goals, their curricula and teaching methods share at least some of the responsibility. Under this new paradigm, the purposes of assessment have expanded. The major, if not sole, purpose of assessment under the teaching-centered paradigm was to give students feedback and grades. Under the learning-centered paradigm, another purpose of assessment is to give feedback to faculty and staff on what is and isn’t working and decide what changes are warranted, if any, to help students learn and succeed even more effectively.

87

Laying a Foundation for Assessment Success

Consider, for example, a group of faculty who assess their students’ analysis skills with a rubric (Chapter 15) that they’ve developed. They’re not satisfied with the resulting evidence of students’ analysis skills, so they decide to change their students’ learning activities. The students’ subsequent papers, assessed using the same rubric, are much better. The faculty did their work carefully enough that they have confidence in their documented evidence that assessment has led to improved student learning. But this is not experimental research. The faculty did not randomly assign students to classes with and without the new pedagogy; they did not adjust statistically for mitigating factors (perhaps the second group of students enrolled with better analytical skills); and they did not apply statistical tests to see if the growth was statistically significant. Indeed, the prospect of this kind of quantitative experimental design and analysis might scare some faculty off! So this is not the kind of work that the faculty will publish in a peer-reviewed journal or present at a professional conference. And there is no way that the hundreds, if not thousands, of these kinds of success stories can be aggregated as meaningful research. But, as many college assessment coordinators can attest, these kinds of assessment-informed improvements in student learning are happening every day.

Assessment’s second purpose: Stewardship Colleges run on other people’s money (Suskie, 2014). They have been entrusted with millions of dollars in resources from students and their families, taxpayers, donors, and others. Those people expect faculty, staff, and institutional leaders to be wise stewards of their resources (see Jargon Alert). And a critical stewardship responsibility is using good-quality evidence of student learning to help inform resource deployment decisions, including how everyone spends their time. Using assessment for stewardship is especially important for programs that may Jargon Alert! be perceived by some stakeholders as having Stewardship relatively low impact. Such programs may Stewardship is the prudent, effective, and judicious include co-curricular experiences, student care and use of resources entrusted by others. support services, programs that serve relatively small numbers of students, and programs whose goals appear disconnected from college priorities as stated in its mission, strategic goals, and institutional learning goals. As discussed in Chapter 26, assessment can be critical to determining whether the value of these kinds of programs is worth the college’s investment in them. 88

Accountability

Assessment’s third purpose: Accountability

What decisions will your evidence inform? Who will make them?

Using Evidence of Student Learning

The third major purpose of assessment is to demonstrate accountability (see Jargon Alert) to stakeholders, including current and prospective students and their families, government policy makers, accreditors, employers, alumni, governing board members, and taxpayers. Accountability includes sharing both successes and steps being taken to make appropriate, evidence-informed improvements. Chapter 25 offers suggestions on how to do this.

Accountability is assuring your stakeholders of the effectiveness of your college, program, service, or initiative.

CHAPTER 6

Jargon Alert!

We are (or should be) assessing to help people make better decisions than they would otherwise make about educational quality, stewardship, or accountability. Specifically, we should be assessing because someone or some group wants to be more confident that the decisions they’re making are the right decisions (Hubbard, 2014). Meaningful assessment thus requires a clear understanding of the following: • Who wants or needs to see this evidence of student learning? • Why do they want or need to see this evidence? • What decisions are they making that this evidence should inform? • How will this evidence help them make a better decision – one in which they will have more confidence? While in theory almost anyone might find almost any evidence of student learning of interest, there are just a few groups most likely to use your evidence to inform significant decisions. Faculty and staff  who teach or otherwise provide learning experiences in a course, program, general education requirement, or co-curricular experience can use evidence of student learning to inform decisions related to giving students the best possible education, including the following: • How well are students learning the most important things we want them to learn? • How well are students learning what they need to succeed in their future endeavors? • How successful are students in graduating, finding appropriate jobs, and transferring to other colleges? 89

Laying a Foundation for Assessment Success

• If students are not learning some important things well, what are the stumbling points? How might we change what we’re doing to help them learn more effectively? • Might different teaching methods or technologies lead to better student learning? • Would new or increased resources help students learn more effectively? Where and how would those resources have the greatest impact on student learning? • Are we getting better at helping students learn? Do recent innovations, such as moves to online learning or learning communities, help students learn more effectively? Faculty and staff can also use evidence of student learning to inform decisions regarding stewardship, including the following: • Is our program of appropriate quality and value? Is it effective in achieving its goals? • Is our program meeting regional or national needs? • Is our program operating efficiently as well as effectively? College leaders and board members  can use evidence of student learning to inform the same decisions as faculty and staff, but their focus is usually on collegewide rather than course- or program-specific decisions. They can also use evidence of student learning to inform decisions on whether to support proposed changes, especially broad, pervasive changes, as discussed in Chapter 26. Accreditors  use evidence of student learning to inform decisions on educational quality and stewardship including the following: • How well are students learning the most important things that the college or program wants them to learn? If students are not learning those things well, what is the college or program doing to improve learning? • How well are students learning what they need to succeed in their future endeavors? If students are not learning those things well, what is the college or program doing to improve learning? • How successful are students in graduating, finding appropriate jobs, and transferring to other colleges? If students are not very successful, what is the college or program doing to improve success? • Is the college or program of appropriate quality and value? How effective is it in achieving its goals? If not, what is it doing to improve its effectiveness?

90

Government policy makers  at federal, state, and (for community colleges) local levels can use evidence of student learning to inform decisions on educational quality and stewardship. Questions of interest to them include the following: • How successful are students in graduating, finding appropriate jobs, and transferring to other colleges? • How well is this college or program meeting regional or national needs?

Prospective students and their families  can use evidence of student learning to answer the following questions of interest to them: • How well are students learning what they need to succeed in their future endeavors? • How successful are students in graduating, finding appropriate jobs, and transferring to other colleges?

Using Evidence of Student Learning

Employers  can use evidence of student learning to answer the following questions of interest to them: • Are students learning what they need to succeed in my workplace? • Is this college or program meeting my employment needs?

CHAPTER 6

• Is this college or program operating efficiently as well as effectively? • Should we support proposed changes to this college or program? • Is this college or program worthy of our investment and support?

Alumni, foundations, and other donors  like to invest in successful enterprises, so they can use evidence of student learning to answer the following questions of interest to them: • How well are students learning what they need to succeed in their future endeavors? • How successful are students in graduating, finding appropriate jobs, and transferring to other colleges? • Is the college or program of appropriate quality and value? Is it effective in achieving its goals? • Is the college or program operating efficiently as well as effectively? • Is the college or program worthy of our investment and support? Assessment committees and coordinators,  as discussed in Chapter 9, aren’t typically empowered to use evidence of student learning to inform the kinds of decisions discussed in this chapter. But they have two important roles in ensuring that evidence of student learning is used to inform important decisions on educational quality stewardship, and accountability. First, they are cheerleaders, facilitators, and mentors, helping everyone learn how to conduct assessments that will be useful and used to inform significant decisions. To that end, they monitor and support assessment activities as discussed in Chapter 10. Second, assessment committees and coordinators are conduits, getting evidence of student learning on its way to the people who will use it, in formats that will help those people make appropriate decisions. This often means summarizing 91

Laying a Foundation for Assessment Success

evidence across programs, analyzing the summarized evidence for key findings, and sharing those key points. Chapters 23, 24, and 25 discuss summarizing, analyzing, and sharing student learning evidence, respectively.

Purposes and audiences vary by setting Whatever the setting for assessment (Chapter 2), its fundamental purpose is giving students the best possible education in that setting, and the primary audience for evidence of student learning in that setting is the faculty and staff who provide that education. In broader settings, such as large programs and general education curricula, stewardship and accountability become more important, and the range of audiences interested in the evidence becomes wider.

Time to think, discuss, and practice 1. Does each of these faculty members have a learning-centered or a teachingcentered approach? Why? a. Dr. Aziz wants students to discover principles on their own rather than absorb them through a lecture. b. Dr. Berger views her job teaching calculus as identifying which students should continue as math majors and which should not. c. Dr. Cowell views his job as designing a learning environment for his students. d. In Dr. Diaz’s classes, term papers are due on the last day of class. He grades them during the following week, and students can pick them up when the next term begins. 2. The teacher education faculty at Cape May College require students to compile portfolios documenting achievement of the program’s key learning goals. They assess the portfolios using a rubric. a. Identify three possible uses of the evidence resulting from this assessment. b. Try to identify decisions that the evidence might inform. Who would make those decisions?

92

CHAPTER 7

Planning Assessments in Academic Programs Some valuable ideas you’ll find in this chapter The best place to assess program learning goals is in a capstone requirement. Grades alone may not tell us much about student learning, but the grading process can yield a wealth of valuable information. Assessment plans should aim to assess all key learning goals within two years – three at most. If an assessment is no longer providing useful information, stop using it and do something else.

93

Laying a Foundation for Assessment Success

A

ssessment planning begins with three cornerstones: • Clearly articulated statements of what you most want students to learn (Chapter 4) • Curricula designed to achieve those goals (Chapter 5) • An understanding of why you are assessing student learning and the uses you may make of the resulting evidence (Chapter 6) This chapter focuses on building on those cornerstones to plan assessments of program learning goals. Many of the principles for assessing program learning goals can be applied to other settings, and Chapter 8 does exactly that, applying them to general education curricula, co-curricula, and other settings.

Choosing the best places in program curricula for assessment While the main purpose of a curriculum map (Chapter 5) is to analyze how well the curriculum is designed to help students achieve key learning goals, the map also can be used to identify the best places in the curriculum to assess student achievement of the curriculum’s key learning goals. Let’s clarify what we mean by assess here. If you’re teaching something, you’re probably grading students on it. If you’re grading students on it, you’ve got assessment information in hand. So, through grading processes, faculty should be routinely assessing students’ achievement of key learning goals in every course throughout the curriculum. If a program curriculum is properly designed to help students achieve its key learning goals, assessment of program learning goals should be happening everywhere, all the time, through the grading process. So what we mean by assess here is identifying key points in the curriculum where assessment evidence from the grading process is compiled, analyzed, discussed, and used to make significant decisions about how to improve teaching and learning. It’s not practical to complete those extra steps for every quiz and homework assignment – that’s too much work for too little insight. There are three points in program curricula that are especially well-suited for compiling, analyzing, and using evidence of student learning. Capstone requirements.  The best and easiest place to collect evidence of student achievement of key program learning goals is a capstone experience (Chapter 5): a significant assignment or project that students complete just before they graduate (Hauhart & Grahe, 2016). Capstones provide a wonderful venue for assessing program learning goals because, if they are well designed, they require students to demonstrate achievement of several of those goals, yielding a lot of bang for the buck. A senior research project, for example, might be assessed through a rubric (Chapter 15) for key program learning goals such as written communication,

94

CHAPTER 7 Planning Academic Program Assessments

critical thinking, information literacy, and research design. If students make presentations on their projects, the presentations can be assessed for oral communication skills as well. Consider another example: An associate degree program in culinary arts might have a capstone project in which students design, plan, prepare, and present a meal. The project might be assessed, again with a rubric, for skills in culinary techniques, menu planning and design, and restaurant management. As you can see from these examples, capstones can provide a holistic portrait of much of what students have learned throughout their program. If your program does not include a required capstone, use your curriculum map and transcript analysis (Chapter 5) to identify one or more ersatz capstones: Courses addressing key program learning goals that students typically take shortly before graduation. Perhaps students in a business associate degree program, for example, often take a course in entrepreneurship shortly before graduating, and the small business plan they create in the course might assess program goals in strategic planning, financial management, and marketing. If your program doesn’t have a capstone or ersatz capstones, well, it isn’t really a program, as discussed in Chapter 5. You’ll need to rethink the curriculum so that it has appropriate, progressive rigor. Foundational requirements.  Many programs – and some general education ­curricula – begin with one or more required introductory courses. Undergraduate sociology programs, for example, often begin with an “Introduction to Sociology” course. Capstones of these foundational requirements, such as final exams, are good places to assess whether students have achieved the foundational knowledge and skills needed to achieve key learning goals later in the program. Cornerstone requirements.  Some program curricula have a required course normally taken about midway through the program. Some bachelor’s degree majors, for example, have a required sophomore-level research methods course. These are great places to confirm that students are on track to achieve key program learning goals by the time they complete the program, especially for students who transfer elsewhere before graduating, as discussed in Chapter 8.

Choosing the best assessments for program learning goals There are potentially endless ways to assess student learning, and every assessment has potential value. Which ones are best in your situation depends primarily on the purpose of your assessment (Chapter 6) and the learning goals you are assessing (Chapter 4). Other factors such as resource availability (Chapter 11) and college culture (Chapter 14) can also affect your decision. That said, however, just a handful 95

Laying a Foundation for Assessment Success

Table 7.1:  Common Assessment Tools If You Want to. . .

Use These Sources of Information

And Assess Them Using

Assess knowledge and conceptual understanding

Multiple-choice tests (Chapter 17)

Item scores, mapped back to test blueprints (Chapter 17)

Assess thinking and performance skills

Paper, projects, performances, essays, exhibitions, field experiences, and other learning activities (Chapter 16)

Rubrics (Chapter 15)

Assess attitudes and values

Reflective writing (Chapter 20)

Qualitative analysis (Chapter 24)

Draw an overall picture of student learning, including thinking and performance skills as well as attitudes, values, and habits of mind

Portfolios (Chapter 18)

Rubrics (Chapter 15) and reflective writing (Chapter 20)

Compare your students against peers

Published instruments (Chapter 19)

Item scores and instrument sub-scores, mapped back to key learning goals (Chapter 23)

of assessment tools will do a fine job assessing most program learning goals; they are presented in Table 7.1. As Table 7.1 points out, most assessments have two components: sources of assessment information and tools to assess that information. Student papers, for example, are a source of assessment information that can be assessed with rubrics. What about grades?  As discussed in Chapter 1, grades alone are usually insufficient to achieve assessment’s main purposes of educational quality ­stewardship, and accountability, but they can be useful albeit indirect evidence of student learning (Chapter 3). What about student ratings?  As discussed in Chapter 20, student self-ratings and satisfaction with their learning experiences can be helpful, because dissatisfaction or a low self-rating is a clue that students may not have learned some important things. But self-ratings and satisfaction are indirect and therefore insufficient evidence of student learning. Students sometimes have misperceptions about their skills, and some might be satisfied with a learning experience simply because it didn’t require a lot of work.

96

Planning Academic Program Assessments

Developing written plans that work out the logistics

CHAPTER 7

What about retention, graduation, and job placement rates?  These are important outcomes, but they’re indirect and therefore insufficient evidence of student learning because they don’t tell us exactly what students have and haven’t learned. If we know, for example, that 95 percent of the graduates of a teacher education program find jobs as teachers, we can conclude that they have probably learned important things, because they’re attractive to employers, but we can’t tell from this statistic alone exactly what they have and haven’t learned well. An even greater concern with retention, graduation, and job placement rates is that they can be affected by factors beyond our control such as the economy and job market. A 95 percent teacher placement rate, for example, may be due as much to a regional shortage of teachers as to what students learned in a teacher preparation program. An economic downturn may mean that many well-prepared graduates can’t find jobs for reasons that have nothing to do with what they learned or how well they learned it.

Almost any undertaking benefits from some kind of plan. Even a shopping trip can be more successful if you start with some notes on what you want to buy and which stores you want to visit. The more complex and extensive your undertaking, the greater your need for detailed plans. It’s hard to imagine a wedding or a vacation tour coming off successfully without some organization. Assessment is no different than any other undertaking; it will be more effective and successful if you first plan your work. The longer and more complex the learning experience, the more important organization and plans become. Written plans for assessments of program learning goals are helpful in several ways. • They communicate to everyone in the program what is to be done and each person’s role and responsibilities (Chapter 9). • They help ensure that assessment efforts will meet relevant accreditation requirements. • If people in the program receive constructive feedback on their plans, the plans help ensure that assessment efforts will be effective (Chapter 3). Effective assessment plans are simple (Chapter 12), flexible (Chapter 9), and developed collaboratively (Chapter 13). They also have the following characteristics. Effective plans convey the assessments’ purpose.  They are designed to make sure students get the best possible education, not merely to assess what they’ve learned. They make clear why the assessments are being conducted, who will use the resulting evidence of student learning, what questions the evidence will help answer, and what decisions the evidence will inform (Chapter 6). 97

Laying a Foundation for Assessment Success

Effective plans are comprehensive.  They explain how every key program learning goal will be assessed. They answer questions such as those in List 7.1 with enough details to show that everything’s been thought through. Rather than simply say that student papers will be assessed using rubrics, for example, they say who will draft the rubric and by what deadline, and they make clear which student papers will be assessed and in which course(s) and class section(s).

List 7.1  Questions to Address in an Assessment Plan ■■ ■■

■■

■■

■■

■■

■■ ■■

■■

■■

■■

■■

■■

■■

How will each learning outcome be assessed? If student classwork (such as tests and capstone assignments) is part of the assessment plan, what will the teaching faculty do with that classwork? If you are looking at work from a sample of students rather than all students, who chooses the sample – a coordinator or the teaching faculty? How are they to choose the sample? What do the teaching faculty submit: Student work samples, completed rubrics, tallies of rubric scores, or something else? Where and how do teaching faculty submit this information? (Chapter 11 discusses assessment information management systems, which may be helpful here.) Who assesses the student work? In other words, who scores student work using a rubric or scores answers to multiple-choice tests: The faculty teaching the class, a faculty committee, or some other group? How will you decide what constitutes acceptable outcomes (Chapter 22)? Who will contact faculty to provide the above information plus assessment tools such as a rubric or set of test questions? When and how will this contact take place? If the plan includes assessments that are not part of classwork, such as a survey or published test, who is responsible for getting this done? Where and how will student work samples be housed, if at all (Chapter 9)? Who will have access to them? What steps will you take to ensure that identifiable information such as student names is kept confidential? Who will summarize the evidence (Chapter 23), analyze it (Chapter 24), and share it (Chapter 25)? Where and how will these summaries and analyses be housed? Who will discuss the evidence? Who will decide what changes are warranted and implement those changes (Chapters 6 and 26)? Where and how will these decisions be documented? How exactly will the evidence feed into division and college-wide planning and budget processes, as warranted (Chapter 26)? How will this all be communicated to the faculty and staff working on this assessment?

Discuss your plans for housing, summarizing, and analyzing your evidence of student learning with your college’s technical support staff before you finalize your tests, rubrics, and surveys and conduct your assessments. Otherwise, it may not be possible to record, summarize, or analyze your evidence as you hope. 98

CHAPTER 7 Planning Academic Program Assessments

Effective plans assess all key learning goals within two years – three at most.  Some programs choose not to assess every program learning goal every semester or every year. If a program staggers its assessments over several semesters or years (Chapter 12), the plan should cover the entire multiyear timeframe. That timeframe should run no more than two years, however – three at most. You wouldn’t use a bank statement from four years ago to decide if you have enough money to buy a car today! Similarly, faculty shouldn’t be using evidence of student learning from four years ago to decide if student learning today is adequate. Assessments conducted just once every several years also take more time in the long run, as chances are good that faculty won’t find or remember what they did several years earlier, and they’ll need to start from scratch. This means far more time is spent planning and designing a new assessment – in essence, reinventing the wheel. Imagine trying to balance your checking account once a year rather than every month – or your students cramming for a final rather than studying over an entire term – and you can see how difficult and frustrating infrequent assessments can be, compared to those conducted routinely. Effective plans are required in course and program proposals.  The work of thinking about assessment can bring to light important questions about learning goals and curriculum design that can lead to rethinking them. Requiring course and program proposals to include complete, feasible assessment plans ensures that these questions are asked right from the start. If the assessment plan is developed only in a later, separate process, there may be extra work in having to go back and revise the course or program learning goals and curricula.

Regularly reviewing and updating your plans Assessment is a perpetual work in progress. As the needs of your students evolve in a rapidly changing world, so will your goals, curricula, teaching methods, and assessment practices. As discussed in Chapter 12, there’s no point in repeating assessments that have become outdated, that no longer provide new insight, or that consume time and resources disproportionate to the value of the information they provide. If you have been working on assessment for a while, counter assessment fatigue by taking a breather for a semester, regrouping with your colleagues, and reflecting on your assessment efforts to date and their outcomes. If you are reflecting on assessment efforts throughout your college, inform these conversations by preparing a snapshot of where your college is with assessment and where it’s going (Exhibit 7.1). This kind of snapshot also gives college leaders, accreditors, state agencies, and other audiences an overview of assessment efforts. 99

Yes No – By 4/30/19

Biology B. S.

History B. A.

Yes Yes

Writing

Social sciences

General education requirements:

Yes

Learning Goals Articulated? If Not, by When?

Accounting B.S.

Academic programs:

Program or General Education Requirement

Yes

Yes

No – By 9/15/19

No – By 3/1/20

Yes

Assessment Strategies Identified and Developed? If Not, by When?

Yes – Every course once every three years, on a rotating schedule, starting Fall 2018

Yes – Every semester starting 2017

No – Every fall, starting 2019

No – Every year, starting 3/1/20

Yes – Every spring, starting 2016

Assessments Implemented? When? ­Frequency?

No – Every year, starting 4/1/20

No – Every year, starting 11/1/20

Yes – Every spring, starting 2017

Assessment Results Used for Planning, Budgeting, and ­Decision-Making? When? Frequency?

Yes – Even-numbered years, by 3/1

Yes – Even-numbered years, by 9/1

Yes – Every year, by 9/1 Yes – Every year, by 12/1

No – Every spring, starting 2/1/20

No – Every year, starting 9/1/20

Yes – Every fall, starting 2016

Assessment Results Compiled and Shared? When? ­Frequency?

Exhibit 7.1  An Example of a Completed Chart for Monitoring Assessment Progress Across a College

CHAPTER 7 Planning Academic Program Assessments

Your reflection might also be informed by some rubrics or rating scales evaluating the status of assessment at your college (Fulcher, Swain, & Orem, 2012; Penn, 2012). Exhibit 7.2 is one example; another is the evaluation rubric for the Excellence In Assessment (EIA) Designation of the National Institute for Learning Outcomes Assessment (www.learningoutcomesassessment.org/eiadesignation .html). Consider asking a sample of faculty, administrators, and board members to complete a rating scale, of their perceptions of what is happening throughout your college and tallying the results (Chapter 23). Also consider using performance indicators (Chapter 23) such as the following to track assessment progress: • Number of programs with implemented assessments and resulting evidence of student learning • Number of programs or courses whose curricula or pedagogies have been changed in response to evidence of student learning • Resources invested in supporting assessment activities (Chapter 11), including professional development (Chapter 10) Consider repeating your appraisal – whether through a snapshot, rubric, or performance indicators – once every year or so, to help faculty and staff see how far they’ve come. Once you’ve collected information on where you are with assessment, use the information to ask the kinds of questions listed in List 7.2, and use the results of your discussion to fine-tune assessment strategies. As you consider modifications, aim for a balance between necessary change and consistency. It can be frustrating when the rules of the game are constantly changing. So seek to fine-tune rather than dramatically overhaul your assessment processes, unless your college community acknowledges the need for wholesale change.

101

Laying a Foundation for Assessment Success

Exhibit 7.2  A Rating Scale Rubric for Evaluating College-Wide Student Learning Assessment Processes No plans = No documented evidence that we have plans to do this. No evidence = Our college appears to be aware that we should do this, but there is no documented evidence that this is happening. Nascent = We have documented evidence that this is happening in just a few areas (for example, only in programs with specialized accreditation). Some = We have documented evidence – not just assurances – that this is happening in some but not most areas (for example, in a number of academic programs but not yet in general education, or in on-campus programs but not yet online, or in undergraduate programs but not yet graduate) Most = We have documented evidence – not just assurances – that this is happening in most but not all areas. Pervasive = We have documented evidence – not just assurances – that this is happening everywhere, including in graduate as well as undergraduate programs, liberal arts as well as professional programs, general education as well as majors, student development as well as academic programs, off-campus as well as on-campus locations, online as well as face-to-face offerings, certificate as well as degree programs, programs developed and delivered by third parties as well as by our college. For academic programs, the general education

No

No evi- Nascent

Some Most Perva-

­curriculum, and institutional learning goals:

plans dence

1. Expected institutional, program, general education, and

□

□

□

□

□

□

sive

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

7. Evidence of student learning is used to inform meaningful □

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

□

course learning goals are clearly articulated and relevant to students and employers. 2. Standards and targets for determining how well learning goals have been achieved are clear, appropriate, and justifiable. 3. Collected evidence of student learning includes direct evidence and is of sufficient quality that it can be used with confidence to make appropriate decisions. 4. Evidence of student learning is clearly linked to learning goals. 5. Evidence of student learning is shared in useful, understandable, accessible forms with relevant audiences. 6. Evidence of student learning balances appropriately successes and areas for improvement. decisions, including teaching and learning betterment, resource deployment, and priorities. 8. Evidence of student learning is used to assure relevant public audiences of institutional and program effectiveness in meeting audience needs. 9. There is sufficient engagement, momentum, and simplicity in current assessment practices to ensure that assessment processes will remain sustained and pervasive.

102

■■

■■

■■

■■

■■

■■

■■

■■

■■

Planning Academic Program Assessments

■■

Goals and Outcomes: How well has assessment in practice matched our intentions? Is everything working the way we intended? Successes: What have been our major success stories? What has worked well? Where have we seen the most progress? Disappointments: What have been our biggest disappointments and struggles? What hasn’t been helpful? What was more difficult than we’d anticipated? Where has progress been slower than we had anticipated? Quality: Is our evidence of student learning generally of sufficient quality that we have confidence in it and use it for betterment? Value: Are our assessments giving us useful evidence of student learning? Which have led to meaningful changes in teaching and learning? Which haven’t? Cost and Benefits: What has been the cost of our assessment efforts in terms of time, dollars, and other resources? What have been the benefits of assessment activities, and how do the costs measure up against the benefits? What has taken too much time, effort, or money? Resources and Support: What resources and support would most help us develop and implement better ways to use evidence of student learning to improve teaching and learning? Changes in Learning Goals: Have our learning goals for our students changed? Are our assessment tools still aligned with our learning goals? Changes in Context and Culture: What new questions about student learning have emerged? How has college culture evolved since our assessments were first planned? How have students changed? How has current thinking on the nature and practice of assessment changed? How might these changes impact assessment policies and practices? Possible Modifications: What would we like to do over the next few years? Are our plans to assess student learning still appropriate? How might we improve our efforts to use evidence of student learning to give our students the best possible education?

CHAPTER 7

List 7.2  Questions to Ask About Assessment Efforts

Time to think, discuss, and practice 1. Complete the chart in Exhibit 7.2 for your college – or perhaps just your academic program or unit. What has been going well? What most needs attention at this point? 2. If anyone in your group has already been part of an effort to assess student learning in a program or general education requirement, discuss the questions in List 7.2. Do your answers suggest any changes in assessment approaches from this point forward?

103

CHAPTER 8

Planning Assessments in General Education, Co-curricula, and Other Settings Some valuable ideas you’ll find in this chapter Assessing general education learning goals is hard because, in part, there’s often no ownership of the curriculum. American colleges and universities often appear embarrassed of their general education curricula. Co-curricular experiences need assessments that are short, engaging, and fun!

105

Laying a Foundation for Assessment Success

A

ssessment planning for general education curricula, co-curricula, and other settings begins with the same cornerstones as for academic programs. • Clearly articulated statements of what you most want students to learn (Chapter 4) • Curricula designed to achieve those goals (Chapter 5) • An understanding of why you are assessing student learning and the uses you may make of the resulting evidence (Chapter 6) Start assessment planning in these settings with these cornerstones and the principles for planning assessments of program learning goals that were discussed in Chapter 7. This chapter offers suggestions for applying those principles to these additional settings.

Planning assessments in general education curricula Assessing general education learning goals is harder than assessing program learning goals for several reasons. American colleges and universities, frankly, often appear to be embarrassed of their general education curricula.  Information on general education curricula is often buried on a college’s website, in the midst of an enormous PDF document of the catalog, and not mentioned anywhere else. Faculty and academic advisors often describe general education curricula to their students as requirements rather than opportunities and encourage students to complete them quickly to “get them out of the way.” Very few present general education as the best and most important part of a college education. There’s often no ownership of the general education curriculum.  General ­education courses are increasingly taught by part-time adjuncts rather than full-time faculty. Those full-time faculty who do teach general education courses are often more interested in their own academic programs. And who’s in charge of the multidisciplinary general education requirements such as humanities or social sciences, making sure they deliver on their intentions? Who cares how well s­ tudents achieve the learning goals of the diversity requirement, which students can fulfill by taking any one of 20 courses offered by eight different academic departments? General education assessment requires collaboration,  and many colleges operate in a culture of silos. Faculty teaching courses that count toward a general education requirement are often not in the habit of collaborating across disciplines to articulate shared general education learning goals and potential assessment strategies. Some general education curricula are, frankly, ineffective  – outdated, politically driven, or both. General education learning goals are often fuzzy and not explicitly aligned with general education requirements (Chapter 5). Many general education

106

Many general education curricula lack a capstone requirement,  the easiest and best place to assess key learning goals (Chapter 7). Some colleges try to assess general education learning goals through one collegewide strategy,  such as a published test, locally developed rubrics, or the VALUE rubrics developed by the Association of American Colleges & Universities (www .aacu.org/value-rubrics). While some colleges find such approaches effective and helpful, others do not, for several reasons. First, it’s hard to come up with a single rubric or other assessment tool that’s meaningfully applicable to student work taken from many different courses and disciplines. The traits of effective writing in the humanities, for example, differ from those in the natural sciences. A top-down assessment may thus inappropriately push some disciplinary square pegs into the round hole of a single assessment strategy. The resulting evidence may not be meaningful to many faculty (“That’s not how I define critical thinking!”), making it hard to use the evidence to make substantive changes in teaching and learning. Second, college-wide assessment processes, usually coordinated by a committee, may prevent teaching faculty from participating meaningfully in the assessment process. The role of teaching faculty may be limited to submitting samples of student work that are assessed by a committee or other group. The teaching faculty submitting samples are largely disconnected from the assessment process, so they don’t feel ownership. Assessment is something done to them, rather than something in which they meaningfully participate. Furthermore, using a single college-wide assessment structure can encourage inappropriate comparisons. Should we really compare students’ critical thinking skills in literature courses with those in chemistry courses? Finally, some college-wide assessments are add-on assessments (Chapter 21), and motivating students not only to participate in them but to give them their best effort is hard.

Planning Assessments in Other Settings

At some colleges, introductory survey courses serve two masters:  General education and one or more programs. An introductory biology course, for example, may be required of biology, health science, and agriculture majors as well as fulfill a general education science requirement. Such a course must help students achieve relevant general education science learning goals and give other students the foundational knowledge and skills needed to achieve program learning goals. This can be too tall an order for a single course. Two or more separate courses would solve the problem, but smaller colleges often can’t afford to do this.

CHAPTER 8

courses continue to focus on content rather than the thinking skills that are the hallmark of a twenty-first-century liberal arts education (Chapter 4).

107

Laying a Foundation for Assessment Success

Some colleges, on the other hand, try to assess general education learning goals on a course-by-course basis,  letting faculty identify their own embedded assessments (Chapter 12) for the general education courses they teach. This approach gives faculty a strong sense of ownership and may therefore generate evidence of student learning that they find useful. But this approach can also generate hundreds of assessment reports, making it difficult to aggregate the evidence and get an overall picture of how well students throughout the college are achieving general education goals.

Addressing the challenges of assessing general education learning goals Each of these challenges can be successfully addressed, although the solution sometimes requires dramatic changes in college practices. Start by considering the ideas for building a pervasive, enduring culture of evidence and betterment in Part 3 of this book. Then, as suggested at the beginning of this chapter, start planning assessment in general education curricula by reviewing the principles for assessment in academic programs discussed in Chapter 7. Then consider the following: Help everyone understand the purpose and value of your general education ­curriculum.  Instill in faculty and staff a commitment to sell it to students. Help ­faculty and staff convey that, while a career-specific program prepares s­ tudents for one job (and that may be all that some students want or need), a liberal arts ­education with its greater emphasis on transferrable skills has the potential to ­prepare students to transition into a host of careers. As a result, liberal arts ­graduates earn more in the prime of their careers than career program graduates (Humphreys & Kelly, 2014). Those with a liberal arts education also lead richer, more fulfilling lives and are more likely to serve the public good (Association of American Colleges & Universities, 2002; Busteed, 2013). As Apple co-founder Steve Jobs said, ­“Technology alone is not enough. It’s technology married with liberal arts, ­married with humanities, that yields the results that make our hearts sing” (“Steve Jobs,” 2011). Clarify your general education learning goals and map them to your general ­education requirements,  as discussed in Chapters 4 and 5. This exercise may lead to the realization that your general education curriculum is broken and needs a thorough overhaul. This may seem like a dismaying conclusion, but keep in mind that a rethought curriculum will give students an even better education than they are getting now, and that’s what this is all about.

108

Planning Assessments in Other Settings

Otherwise use a bottom-up rather than top-down, one-size-fits-all general education assessment strategy.  Unless a consistent approach is mandated by an accreditor, state agency, or the like, there is usually no need for just one approach throughout the general education curriculum. Invite faculty teaching courses in related disciplines with shared general education learning goals to identify a common assessment strategy that reflects those goals through the lens of their curricula. The science faculty might agree to include a common set of test questions on their final exams, while faculty helping students develop an appreciation of diverse perspectives might ask students to write a reflective essay. Then have a faculty group review the reports of these assessments holistically and qualitatively for recurring themes, as discussed in Chapter 24. List 8.1 offers some additional suggestions, including some reiterated from Chapter 7, for assessing general education learning goals.

CHAPTER 8

If a college-wide general education assessment tool works for you, go for it.  If a standardized, centralized approach is effective and helpful, don’t mess with success. But reconsider if it seems cumbersome, time consuming, and not all that helpful in improving learning.

List 8.1  Suggestions for Assessing General Education Learning Goals ■■

■■ ■■

■■

■■

■■

Establish a standing general education committee charged with ensuring the quality and effectiveness of the general education curriculum, including how well students achieve its key learning outcomes. Also identify a coordinator for each general education requirement (Chapter 9). Build a culture of collaboration, as discussed in Chapter 13. Help faculty emphasize thinking skills over content in introductory survey courses. List 5.2 in Chapter 5 offers suggestions on how to do this. Consider adding a junior-level capstone to a baccalaureate general education curriculum (Chapter 5). It’s the best and easiest place to collect evidence of student achievement of many general education learning goals. Develop a written plan that: • is developed collaboratively, • explains the assessments’ purpose, • is comprehensive, • covers up to two years – three at most, • includes all logistical details and deadlines, and • is flexible and updated regularly. The assessment tools in Table 7.1 will probably be most useful, especially rubrics and reflective writing. Remember that grades alone do not provide sufficient evidence of achievement of general education learning goals (Chapter 1). (continued)

109

Laying a Foundation for Assessment Success

110

List 8.1  Continued ■■

■■

Foundational requirements, such as introductory survey courses, are good places to assess whether students have foundations needed to achieve your general education learning goals. You may decide to summarize and analyze a sample rather than all student work (Chapter 12).

Planning assessments in co-curricula Assessing student learning in co-curricular experiences can also be challenging (Henning & Roberts, 2016; Yousey-Elsener, Bentrim, & Henning, 2015); some of the reasons are discussed in Chapter 21. The following suggestions will help. Make sure every co-curricular experience is designed purposefully.  An excellent ­co-curricular experience is designed just like any other learning activity (Chapters 5 and 16): It has a clear purpose, with one or more clear learning goals; it is designed to help students achieve those goals; and it assesses how well students have achieved those goals. Chapter 4 talks more about articulating learning goals for co-curricular experiences. Recognize that many co-curricular ­experiences focus on student success Jargon Alert! as well as student learning – and assess Student Success both.  Many ­co-curricular experiThere’s no widely accepted definition of student ences, including ­orientation programs success, but it might be helpful to think of it as students’ success in achieving their goals, which and ­first-year ­experiences, are explicitly may include any of the following: intended to help students succeed in college: • Developing new skills that will eventually lead To earn passing grades, to progress on to a higher standard of living schedule, and to graduate. So it’s important • Earning a degree or certificate – and persisting in their studies until they do so to assess both student learning and student • Transferring successfully to another college to success (see Jargon Alert) in order to show continue their studies that the value of these programs is worth the Student success may be measured through metrics college’s investment in them. such as retention, graduation, transfer, and job Keep in mind, however, that it’s placement rates, grades, course repeat rates, and eventual earnings as well as through assessments often hard to determine definitively the of student learning. impact of one co-curricular experience on student success because there may be other mitigating factors. Students may successfully complete a first-year experience designed to prepare them to persist, for example, but then leave because they’ve decided to pursue a career that doesn’t require a college degree. They may successfully complete a tutoring program but still fail the course if a family emergency forces them to miss the last two weeks of classes.

Planning Assessments in Other Settings

Focus assessment efforts on co-curricular experiences where significant, ­meaningful learning is expected.  Student learning may be a very minor part of what some student affairs, student development, and student services units seek to accomplish. The registrar’s office, for example, may answer students’ ­questions about registration but not really offer a significant program to ­educate s­ tudents on registration procedures. And while some college security ­operations view educational programs on student safety as a major component of their mission, others do not. Focus assessment time and energy on those co-curricular e­ xperiences that are large or significant enough to make a real impact on student learning (Chapter 12).

CHAPTER 8

Recognize that some programs and services focus on student success but not student learning.  Some programs under a student affairs, student development, or student services umbrella are not co-curricular learning experiences. Giving commuting students information on available college services, for example, is not really providing a learning experience. Neither are student intervention programs that contact students at risk for poor academic performance to connect them with available services. Programs designed primarily to help students succeed rather than learn should be assessed with measures such as those noted in the Jargon Alert defining student success.

For voluntary co-curricular experiences, start but don’t end by tracking ­participation.  Obviously if few students participate, impact is minimal no matter how much student learning takes place. This makes participation an important measure. Set a rigorous but realistic target for participation, count the number of students who participate, and compare your count against your target. Voluntary co-curricular experiences call for fun, engaging assessments.  No one wants to take a test or write a paper to assess how well they’ve achieved a co-curricular experience’s learning goals. Group projects and presentations, role plays, team ­competitions, and Learning Assessment Techniques can be more fun and engaging (Chapters 20 and 21). Short co-curricular experiences call for short assessments.  Brief, simple assessments such as minute papers, rating scales, and Learning Assessment Techniques (Chapter 20) can all yield a great deal of insight. It’s hard to obtain compelling evidence of some co-curricular learning goals.  Some co-curricular experiences aim to instill attitudes or values such as spiritual development or a commitment to serving others. As discussed in Chapter 21, these kinds of learning goals can only be assessed with indirect evidence (Chapter 3), which can be less compelling than direct evidence. 111

Laying a Foundation for Assessment Success

Co-curricular experiences often teach processes or performances, which can be harder to assess than products.  Chapter 21 suggests strategies to assess teamwork and other processes and performances. Consider assessing student satisfaction, especially for voluntary experiences.  Student dissatisfaction is an obvious sign that there’s a problem! But student satisfaction levels alone are insufficient assessments because they don’t tell us how well students have learned what we value.

Planning assessments in other settings Many of the principles for assessing program, general education, and co-curricular learning goals apply to assessments in other settings as well. Here are additional suggestions for some specific settings. Assessments of student learning in field experiences are discussed in Chapter 13.

Institutional learning goals The principles for assessing general education learning goals apply to institutional learning goals. Keep in mind, for example, that one size may not fit all. There’s no law that says an institutional learning goal must be assessed the same way in all settings or that all institutional learning goals must use the same assessment tool such as a rubric. Consider a bottom-up approach rather than a top-down approach, as suggested earlier in this chapter. Communication and collaboration (Chapter 13) are just as important to assessing institutional learning goals as they are to assessing general education learning goals.If students achieve an institutional learning goal through both the general education curriculum and their major, for example, the goal can be assessed in a program capstone, but the resulting evidence should be reviewed and analyzed collaboratively by faculty in the program and faculty teaching relevant general education courses. Interpersonal skills are an example of an institutional learning goal that might be developed – and assessed – in the general education curriculum, in academic programs, and in co-curricular experiences, making communication and collaboration with co-curricular staff essential.

Graduate programs Assessing student learning in graduate programs is often easier and simpler than in undergraduate programs because most graduate programs have one or more 112

The lack of common curricular experiences (such as a required capstone course) in pick-from-a-list programs (Chapter 5) makes it difficult to find opportunities to collect assessment information at appropriate points in the program. In order to fix the assessment “problem,” faculty must first fix the bigger problem of program incoherence. See Chapter 5 for suggestions.

Planning Assessments in Other Settings

Pick-from-a-list programs

CHAPTER 8

required capstones: Comprehensive exams, field experiences such as clinicals, projects, theses, and dissertations. Assessment in some PhD programs may be a challenge because students may pursue highly individualized research in their dissertations, with little in the way of shared program learning outcomes. Treat these programs as self-designed programs, which are discussed below.

Interdisciplinary and self-designed programs When interdisciplinary and self-designed programs have clearly articulated learning goals and purposeful, coherent curricula, assessment can be accomplished through an independent study capstone project or experience and/or a portfolio (Chapter 18). Students in self-designed programs can plan how they will document achievement of their learning goals as part of their program planning process with their advisor (Chapter 5).

Programs with small numbers of students Some academic programs and co-curricular experiences have very small numbers of students. In these situations, traditional quantitative assessments such as tests, rubrics, and survey rating scales often aren’t very insightful, because just one unusual student can throw the numbers out of whack. Instead, take advantage of the small numbers to get in-depth, qualitative information on students’ learning experience. Reflective writing and interviews (Chapter 20) as well as portfolios (Chapter 18) can be helpful. Also consider collecting evidence from students in successive cohorts (up to two years) to get enough evidence to draw meaningful conclusions.

Transfer associate degree programs The best place to assess student achievement of the learning goals of a transfer associate degree program would be a capstone requirement or at least an ersatz

113

Laying a Foundation for Assessment Success

capstone, as discussed in Chapter 7. But if many students transfer before taking the capstone or ersatz capstone, an assessment at that point may not be very helpful. So it’s a good idea to identify a cornerstone course, also discussed in Chapter 4: A key program requirement that students often take before they transfer that addresses one or more key program learning goals. At that point you can assess whether students are on track to achieve the program learning goal, although of course they probably won’t yet be where you want them to be by graduation. Assessing program learning goals in entry-level foundational courses is often of limited value, unless many students transfer immediately after the first semester. At this point, assessments will tell you how well students have achieved some of the foundations needed to achieve the program learning goal, but not much more.

A curriculum in flux Colleges are rarely static. There’s always some curriculum that’s undergoing review and revision. If a different curriculum is going to be put in place in the next year or two, does it make any sense to start assessment now? This depends on the kinds of changes envisioned. Sometimes the learning goals of a curriculum aren’t changing; the only modifications are curricular ones to help students better achieve those goals. In this situation, it makes sense to start the assessment now to see if the modified curriculum does indeed lead to improved learning. In other cases, some learning goals of the curriculum are changing but others are remaining the same. It’s the rare general education curriculum, old or new, that doesn’t have a writing component, for example. In these cases, it usually makes sense to start assessing the learning goals that will continue from the old to the new curriculum, even if the learning goals may be restated. But if faculty will spend the next year completely rewriting a curriculum with entirely different learning goals, it may not make sense to begin now to assess a soon-to-be-obsolete curriculum, and a brief postponement may be reasonable.

Time to think, discuss, and practice 1. Examine your college’s general education curriculum (if it offers multiple general education curricula for students in different programs, choose one). Come up with a “sales pitch” explaining to students how the curriculum will benefit them, in terms they’ll appreciate. 2. Rose Hill College has a one-hour program during new student orientation to familiarize new students with the college’s array of programs and services to meet their academic, personal, social, financial, and health needs. Brainstorm the program’s learning goal and a way to assess how well participating students achieve this goal. 114

Part 3

Building a Pervasive, Enduring Culture of Evidence and Betterment

CHAPTER 9

Guiding and Coordinating Assessment Efforts Some valuable ideas you’ll find in this chapter Being flexible in requirements, expectations, and approaches is one of the keys to assessment success. The most important traits in an assessment coordinator are sensitivity, open-mindedness, flexibility, and a passion for teaching and learning. Design assessment infrastructure in ways that facilitate not the collection but the use of resulting evidence of student learning.

117

Creating a Culture of Evidence and Betterment

I

ndividuals and small groups of faculty and staff can and do assess learning goals for courses and co-curricular experiences on their own, without oversight. But assessments in broader settings benefit from guidance and coordination for several reasons (Kinzie & Jankowski, 2015) emphasized repeatedly throughout this book. • Collaboration. Learning goals and curricula for academic programs, general education curricula, and college-wide learning are collaborative responsibilities – and so is their assessment. • Integration. Just as learning experiences are more effective when they are integrated, so are assessments of those learning experiences. An integrated approach to assessment is likelier to have broader impact in ensuring and improving learning. • A culture of evidence. At its heart, assessment is all about supporting and strengthening a pervasive culture of using good-quality evidence to inform decisions large and small. That can’t happen unless systems are put in place to get evidence of student learning into the hands of decision-makers. And the key to that is “integrating assessment work into the college’s governance and organizational structures” (Kinzie & Jankowski, 2015, p. 78).

What guidance and coordination are needed? Broad assessment efforts can benefit from the following kinds of guidance and coordination (Kinzie & Jankowski, 2015).

A shared vision of a pervasive, enduring culture of evidence and betterment While a lot of people talk about building a culture of assessment, this book doesn’t, because the phrase suggests a focus on doing assessment rather than using assessment. And while a lot of people say the purpose of assessment is improvement, this book minimizes that term as well, because it can suggest enhancing what we’re already doing rather than rethinking what we’re doing and perhaps doing something quite different – which is sometimes the change that is needed. So this book focuses instead on building a culture of evidence and betterment, a phrase that describes a commitment to using all kinds of evidence – not just evidence of student learning – to inform changes, not just improvements, in curricula, teaching/ learning strategies, learning goals, co-curricular experiences, and support systems. At a college with a culture of evidence and betterment, assessment is not just a burst of effort right before an accreditation review, and it doesn’t happen in just a few pockets here and there. Assessment is a useful part of the fabric of everyday 118

■■

■■

■■

■■

Assessment helps students learn more effectively because the grading process motivates students to do their best; because assessment feedback helps students, faculty, and staff understand learning strengths and areas for improvement; and because the clear learning goals and standards that assessment requires help students understand where they should focus their learning time and energies. Assessment activities bring faculty and staff to discuss together important issues such as what they teach, why, and their standards and expectations. In other words, assessment encourages faculty and staff to undertake a collaborative approach to teaching (Chapter 13). Assessment activities help faculty and staff see how courses and other learning experiences link together to form coherent, integrated programs and how the courses they teach contribute to student success in subsequent pursuits (Chapter 5). Assessment activities bring neglected issues to the forefront. Some colleges have problems that have been swept under the carpet far too long: Outdated general education curricula, a dysfunctional governance system, or outmoded pedagogies. Launching an assessment effort often requires addressing issues that should have been tackled long ago, sometimes making the assessment process even more useful than the product.

Guiding and Coordinating Assessment Efforts

List 9.1  Examples of the Benefits of Assessment

CHAPTER 9

life, with everyone in the habit of using evidence in their everyday work to inform decisions large and small. One of the most important responsibilities of those guiding and coordinating assessment is sharing this vision (Kinzie & Jankowski, 2015) and serving as assessment advocates, articulating the many benefits of assessment efforts, including those in List 9.1.

A flexible approach Another key responsibility of those guiding and coordinating assessment is recognizing that one size does not fit all. An important way to build a pervasive culture of assessment is to respect the people who contribute (Chapter 14), and one way to do this is to empower faculty and staff to make appropriate choices and decisions about assessment. A bottom-up approach, in which faculty and staff establish their own, diverse assessment structures, processes, and tools, generally leads to more useful evidence of student learning than a top-down approach under which everyone must conform to a uniform structure. Faculty and staff soon see the value of greater consistency and then buy into more integrated structures as needed. Offer faculty and staff the following flexibilities: Flexible assessment methods.  Let faculty and staff vary assessment practices by discipline and program so the resulting evidence is of maximum value. While rubrics (Chapter 15) are among the most popular and useful assessment tools, they’re not the best choice for every situation, for example. Mathematics faculty may find traditional tests most useful, while faculty in the humanities may find reflective 119

Creating a Culture of Evidence and Betterment

writing more insightful. So don’t require everyone to use a rubric or any other single assessment tool, and certainly don’t require everyone to use the same rubric. Flexible timelines and schedules.  Adapt assessment schedules to meet evolving needs. Suppose your assessment of students’ writing skills finds that they are poor. Do you really want to wait two or three years to assess them again? Disappointing outcomes call for frequent reassessment to see if planned changes are having their desired effects. Assessments that have yielded satisfactory evidence of student learning are fine to move to a back burner, however. Put those reassessments on a staggered schedule (Chapter 12), and repeat them only once every two or three years to make sure student learning isn’t slipping. This frees up time to focus on more pressing matters. Flexible deadlines.  While you may expect all programs to submit annual assessment reports (Chapter 10), be flexible on the date they’re due. Give faculty and staff a selection of dates spread over the academic year and let them choose the date that works best for them. Flexible report formats.  As discussed in Chapter 10, unless your accreditor explicitly requires that evidence of student learning be presented in a consistent format, do not require everyone to contort what they’ve done to fit a common template. Offer report templates and assessment technologies (Chapter 11) as tools, not mandates.

Other elements of guidance and coordination Those providing guidance and coordination are also responsible for the following: Clear, consistent guidelines.  Appropriately consistent policies and guidelines (Chapter 10) help everyone understand what they are to do and what constitutes good assessment practice. Professional development.  Offer a variety of resources to those who need additional assistance understanding what they are to do and what constitutes good assessment practice (Chapter 10). If you have an assessment committee (discussed later in this chapter), its members may need professional development to enable them to carry out their responsibilities. Communication.  A communications plan can help keep faculty and staff informed of assessment expectations, resources, and so on. 120

Synthesis and sharing of student learning evidence.  As noted in Chapter 6, assessment coordinators and committees rarely use evidence of student learning to make decisions regarding teaching, learning, planning, and resource allocation. Instead, they serve as conduits of evidence of student learning. They periodically aggregate and synthesize evidence, identify any pervasive issues with student learning, and convey those broad concerns, perhaps with recommendations, to decision-makers who can address those issues through planning and budget decisions. A foundation of trust and empowerment.  Those guiding and coordinating assessment ensure that assessment is conducted and the resulting evidence is used ethically and responsibly (Chapters 3 and 26). They also ensure that the faculty, who are responsible for effective teaching and curriculum design, are likewise responsible for effective assessment.

Guiding and Coordinating Assessment Efforts

Quality assurance.  Periodic reviews of assessment efforts (Chapter 7) and resulting evidence of student learning (Part 5) help ensure that assessment is being done, that it is being done reasonably well, and that the resulting evidence is used to inform meaningful decisions.

CHAPTER 9

Collaboration.  Appropriate coordination can foster cross-campus collaboration (Chapter 13) on shared learning goals (Kinzie & Jankowski, 2015).

Who should provide guidance and coordination? The following traits are important in anyone involved in guiding and coordinating assessment efforts, including assessment committee members. Sensitivity and open-mindedness.  Effective assessment coordinators and committee members develop successful working relationships with faculty and staff with a broad range of disciplinary backgrounds, values, and perspectives. In order to do this, they listen to and learn from faculty and staff with different experiences, priorities, and points of view. Flexibility.  Effective assessment coordinators and committee members recognize the need for flexibility discussed earlier. They acknowledge that there are many legitimate ways to approach assessment. They understand that the primary purpose of assessment is to help faculty and staff give students the best possible education, and if bending the rules encourages faculty and staff to achieve that end, they are willing to do so. Rather than continue to enforce past assessment processes and 121

Creating a Culture of Evidence and Betterment

expectations rigidly, they adapt past processes to meet current needs. They recognize that assessment will not proceed across the college in lockstep fashion; some departments and programs will take longer than others, and some may require more support. A passion for teaching and learning.  The credibility of assessment coordinators and committee members is greatly enhanced if they have teaching experience, even if only as a part-time adjunct. This allows them to empathize with faculty workloads, student characteristics, and the effectiveness and frustrations of various assessment strategies. Assessment coordinators and committee members are even more effective when they have a passionate enthusiasm for teaching and for helping students learn. Credibility.  Effective assessment coordinators and committee members are respected by faculty, staff, and college leaders as reasonably familiar with current thinking on teaching, learning, and assessment. But even more important, they are willing to acknowledge that they don’t have all the answers, and they are willing to learn. They know how to find answers to emerging assessment questions, whether through networking with peers, online searches, or published literature. Technical, analytical, and social science research skills, while helpful, are not absolutely necessary. Indeed, some outstanding assessment experts and scholars come from the humanities and other disciplines that do not emphasize such skills. Colleges that invest in assessment technologies do need staff with the technical background to support those technologies, as discussed in Chapter 11, but this may be provided by the information technology unit rather than an assessment coordinator.

Do you need an assessment coordinator, committee, or both? While guidance and coordination can be provided by individuals, committees, or both, often at least one committee is needed to fulfill all the responsibilities described in this chapter. Who should serve on an assessment committee?  Aim for a selection process that will yield committee members with the four traits described above: Sensitivity and open-mindedness, flexibility, a passion for teaching and learning, and credibility. Sometimes a combination of volunteer, elected, and appointed members yields the best mix of skills and traits. Just as teaching faculty have lead responsibility for guiding the curriculum and student learning, teaching faculty should have lead responsibility for 122

CHAPTER 9 Guiding and Coordinating Assessment Efforts

guiding assessment activities. One of the best ways to empower faculty with this responsibility is to compose the assessment committee largely or wholly of teaching faculty, with a teaching faculty member as chair. Try to ensure that faculty are drawn from a representative cross-section of disciplines without making the committee so large as to be unwieldy. Depending on the assessment committee’s charge, its membership may also include the following: • Representatives of student development staff and student support staff, who can help coordinate and synthesize the assessment of student learning inside and outside of the classroom • Institutional researchers, who can contribute indirect evidence of student learning and help connect evidence of student learning to other measures of student success such as retention rates • Students, although there are many other, perhaps more meaningful ways for students to contribute, as discussed in Chapter 13 A college-wide assessment coordinator is an ex officio committee member but not the chair. This conveys tangibly that the assessment coordinator’s role is one of support, not leadership, which is a faculty responsibility. Identify a network of assessment coordinators.  Ask each academic program to designate a faculty member to serve as the program’s assessment coordinator. Also identify an assessment coordinator for co-curricular experiences and for each general education requirement. Those individuals can be the main contacts of the college-wide assessment coordinator and committee, oversee development of their program’s assessment plan, and make sure it is executed. They can also informally network with and support one another. The optimal number of assessment coordinators and committees varies by institutional size and complexity.  Small, focused colleges often do not need a full-time assessment coordinator; a faculty member or administrator can assume responsibility for coordinating college-wide assessment on a part-time basis. Very large, complex universities may be best served by having an (again often part-time) assessment coordinator within each college or school. While one college-wide assessment committee may be sufficient, other models include: • Separate committees responsible for student learning assessment and for other aspects of institutional effectiveness (Chapter 24) • A separate committee responsible for the assessment of the general education curriculum and/or institutional learning goals • At a large university, separate committees responsible for assessment in each school or college

123

Creating a Culture of Evidence and Betterment

Keep in mind that the more assessment committees you have, the more time and effort you will spend on coordinating what they’re doing rather than getting assessment done and used. “Having multiple and separate reporting lines for assessment activity . . . has the potential to create silos among units and limit the benefit of assessment efforts to improve student learning” (Kinzie & Jankowski, 2015, p. 87).

How should assessment coordinators and committees be positioned in the organizational and governance structures? No one organizational model is ideal, as what’s best depends on factors such as college mission, culture, size, and external factors such as state and accreditation mandates (Kinzie & Jankowski, 2015). But the following principles apply widely. Also see Chapter 12, which discusses linking assessment with other college processes, and Chapter 13, which shares ideas for fostering collaboration. Position the assessment coordinator and committee in ways that facilitate the use of evidence of student learning.  Design your assessment organizational and governance infrastructure so that it gets evidence of student learning into the hands of decision-makers who will use it in meaningful discussions and decisions about teaching, curriculum design, and support services. This means that assessment coordinators and committees should ideally report directly to college leaders or bodies that are respected by the college community, that need and expect student learning evidence, and that are ready to use it to inform key decisions. In other words, aim to “intentionally link assessment expertise with units that can take advantage of results” (Kinzie & Jankowski, 2015, p. 87). Connect the assessment coordinator and committee with a teaching-learning Jargon Alert! center  to help faculty and staff use eviTeaching-Learning Center dence of student learning to inform changes A teaching-learning center (TLC) is a faculty to teaching methods and curricula (Kinzie & professional development center charged with helping faculty improve their teaching. Jankowski, 2015; see Jargon Alert). The assessment coordinator and committee and the teaching-learning center can co-sponsor professional development events on topics such as teaching and assessing critical thinking or teaching and assessing in large-class settings. This collaboration can help faculty and staff see that assessment is simply part of the teaching-learning process.

124

Connect the assessment coordinator and committee with curriculum committees  if the committees are separate. The committees can collaborate to develop and use a consistent vocabulary that helps everyone see the connection between assessment and effective curriculum design. They can also partner in defining criteria for evaluating curricular and assessment plans (Chapter 7) – which, if they are separate, should share common elements. This can save faculty time, forestalling the need for them to prepare one report or proposal for the assessment committee and a separate one for the curriculum committee.

Guiding and Coordinating Assessment Efforts

Connect the assessment coordinator and committee with the institutional research office  to facilitate the integration of evidence of student learning with other measures of institutional effectiveness (Kinzie & Jankowski, 2015). This helps to create an overall picture of college-wide successes and shortcomings that can be used to set priorities, including college-wide and division planning and resource allocation decisions (Chapter 26).

CHAPTER 9

Embed assessment coordination within the responsibilities of existing committees.  Rather than create stand-alone assessment committees, consider charging an existing committee, such as a curriculum committee or general education committee, to include assessment guidance within its responsibilities. This sends a message that assessment is part of regular work to ensure curricular quality and effectiveness and not a separate add-on responsibility solely to placate accreditors.

Formalize the assessment infrastructure.  Formalize the reporting relationships of assessment coordinators and committees as well as their connections to other individuals, units, and groups. Incorporating them into existing organizational and governance infrastructure sends a powerful message that assessment is not a fad but a permanent part of the fabric of college life. Consider making the assessment committee part of the faculty governance structure, such as a standing committee of the faculty senate. This helps convey that the faculty are responsible for leading student learning assessment.

Time to think, discuss, and practice 1. What might be a suitable membership composition of an assessment committee at your college? Justify your choices, and reflect on what they tell you about your college’s values and culture. 2. If your college has a teaching-learning center, what is its relation to those who guide and coordinate assessment? How, if at all, might the relationship be strengthened? 125

Some valuable ideas you’ll find in this chapter

CHAPTER 10

Helping Everyone Learn What to Do Today there is a wealth of opportunities to learn about assessment. Templates for assessment reports can help educate everyone on good assessment practices. Use a rubric to offer constructive feedback on assessment reports. Gradually introduce flexibility in reporting expectations.

127

Creating a Culture of Evidence and Betterment

M

any faculty and staff have no formal training in and little opportunity to learn about instructional design, teaching techniques, and assessment techniques such as creating an assignment, writing a multiple-choice test, or grading an essay. Helping faculty and staff develop their assessment skills makes the assessment process far less formidable to them.

Provide opportunities to learn about assessment Because faculty and staff differ in their needs and preferences, offer a variety of opportunities for faculty and staff to learn how to assess. Chapter 12 discusses searching for and adapting assessment practices elsewhere; this section offers additional suggestions. Build a library.  List 10.1 suggests some good choices for any assessment practitioner’s bookshelf, and List 10.2 suggests some potentially relevant journals. There are also dozens if not hundreds of journals on teaching in specific disciplines that include articles on assessment; contact your college’s librarians for suggestions.

List 10.1  Seminal Books on Assessing Student Learning Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education by Trudy Banta and Catherine Palomba (2014) This is a soup-to-nuts primer on student learning assessment in higher education. The authors especially emphasize organizing and implementing assessment. Learning Assessment Techniques: A Handbook for College Faculty by Elizabeth Barkley and Claire Major (2016) This successor to the classic Classroom Assessment Techniques (Angelo & Cross, 1993) expands and reconceptualizes Classroom Assessment Techniques (CATs) into a fresh set of Learning Assessment Techniques (LATs) – simple tools for learning and assessment – that faculty will find invaluable. How to Create and Use Rubrics for Formative Assessment and Grading by Susan Brookhart (2013) This book offers persuasive arguments for approaches to rubric development that will lead to sounder, more useful rubrics. Creating Significant Learning Experiences: An Integrated Approach to Designing College Courses by L. Dee Fink (2013) This book advocates backwards curriculum design (Chapter 5) and presents an important context for assessment: its role in the teaching process. Using Evidence of Student Learning to Improve Higher Education by George Kuh, Stan Ikenberry, Natasha Jankowski, Timothy Cain, Peter Ewell, Pat Hutchings, and Jillian Kinzie (2015) This book makes a powerful argument for moving from a compliance approach to assessment to one that makes assessment meaningful and consequential. If you feel your college is simply going through assessment motions, this book will give you plenty of practical ideas to make it more useful. Five Dimensions of Quality: A Common Sense Guide to Accreditation and Accountability by Linda Suskie (2014)

128

List 10.2  Journals and Other Publications That Address Assessing Student Learning in Higher Education Academic Exchange Quarterly Active Learning in Higher Education American Educational Research Journal American Journal of Distance Education American Journal of Evaluation Applied Measurement in Education Assessment and Evaluation in Higher Education Assessment in Education: Principles, Policy and Practice Assessment Update

Journal of College Student Retention Journal of College Teaching & Learning Journal of Educational Measurement Journal of Effective Teaching Journal of Excellence in College Teaching Journal of General Education Journal of Graduate Teaching Assistant Development Journal of Higher Education

Change College Teaching Community College Journal of Research and Practice Computer Supported Cooperative Work: The Journal of Collaborative Computing and Work Practices Education Policy Analysis Archives Educational Action Research Educational Evaluation & Policy Analysis Educational Research e-Journal of Instructional Science and Technology Electronic Journal of e-Learning Innovations in Education and Teaching International International Journal for the Scholarship of Teaching & Learning International Journal of e-Portfolio Journal of Applied Learning in Higher Education

Journal of Interactive Learning Research Journal of Online Learning and Teaching Journal of Student Centered Learning Journal of the Scholarship of Teaching and Learning Journal on Excellence in College Teaching Liberal Education National Teaching and Learning Forum New Directions for Evaluation New Directions for Teaching and Learning New Directions for Higher Education Peer Review Practical Assessment, Research, & Evaluation Research and Practice in Assessment Research in Higher Education The Teaching Professor

Helping Everyone Learn What to Do

This book presents basic principles for ensuring quality in higher education, including principles for obtaining and using meaningful, useful assessment evidence. Assessment Clear and Simple: A Practical Guide for Institutions, Departments, and General Education by Barbara Walvoord (2010) This slim volume is a great introduction for anyone feeling overwhelmed by all he or she needs to learn about assessment. Effective Grading by Barbara Walvoord and Virginia Anderson (2010) With its simple language and its focus on the grading process, this book is a great way to help faculty develop or improve assessments in their courses. It introduces them to many important assessment ideas that apply to program and general education assessments as well.

CHAPTER 10

List 10.1  Continued

129

Creating a Culture of Evidence and Betterment

Join a membership organization.  AALHE, the Association for the Assessment of Learning in Higher Education (aalhe.org), focuses on student learning assessment. There are also some regional networks of assessment practitioners around the U.S. and in some other countries; check with colleagues at nearby colleges to see if one is in your area. If not, consider starting one! For information on assessment in general education curricula, contact AAC&U, the Association of American Colleges & Universities (www.aacu.org), and AGLS, the Association for General and Liberal Studies (www.agls.org). For information on assessment in student life programs, consider any of these organizations: • CAS: Council for the Advancement of Standards in Higher Education (www.cas.edu) • ACPA: American College Personnel Association (myacpa.org/) • NASPA: National Association of Student Personnel Administrators (www .naspa.org) Organizations that address responsibilities associated with assessment, such as teaching, institutional effectiveness, and planning, include: • AEA: American Evaluation Association (www.eval.org) • AHEE: Association for Higher Education Effectiveness (ahee.org) • AIR: Association for Institutional Research (www.airweb.org) • POD: Professional and Organizational Development Network in Higher Education (podnetwork.org) • SCUP: Society for College & University Planning (www.scup.org) Also consider contacting relevant disciplinary associations; some offer resources on assessing student learning in their discipline. Participate in conferences, workshops, and webinars.  Face-to-face conferences and workshops are among the best places to network with colleagues and learn about the latest and best assessment practices. Most of the organizations listed above offer annual conferences and sometimes other professional development opportunities such as webinars. Also consider any of these: • The Assessment Institute in Indianapolis (assessmentinstitute.iupui.edu), the largest and oldest conference in the United States devoted solely to assessment in higher education • The Lilly Conference Series on College and University Teaching and Learning (lillyconferences.com) • Accreditors’ annual conferences and other professional development opportunities • Smaller assessment conferences that may be found around the U.S. and in other countries

130

Subscribe to (free) news feeds.  These may not address assessment per se, but they have plenty of news on forces affecting it. Consider subscribing to: • Inside Higher Ed (www.insidehighered.com/content/sign-inside-higher -­eds-newsletters) • Lumina Foundation Daily Higher Education News (www.luminafoundation.org/ newsblasts) • Education Dive: Higher Ed (www.educationdive.com/signup) • Academe Today, the free daily newsletter of the Chronicle of Higher Education (www.chronicle.com/newsletter/Academe-Today/3)

Helping Everyone Learn What to Do

Join online discussion groups.  The most active online discussion group for assessment is the unmoderated ASSESS listserv sponsored by AALHE. To join or search its archives, visit lsv.uky.edu/archives/assess.html.

CHAPTER 10

If you or colleagues attend a conference or workshop, plan to share what you’ve learned with others at your college. Consider offering a summary on a college web page, a short article shared through email or social media, or a short presentation over a brown-bag lunch.

Offer local professional development opportunities.  It’s very likely that faculty and staff at your college share common questions about assessment (“How should I assess group work?” “How should we define critical thinking in our discipline?”). It’s equally likely that someone at your college is already using an effective assessment practice that others might learn from and emulate or adapt. So consider offering any of the following: • An assessment web site, newsletter, or column in the employee newsletter with brief readings, guides, examples of good practices (both at your college and elsewhere), and links to off-campus resources • Professional development programs such as workshops, speakers, and brown bag lunch discussions • A learning community: A group of faculty and staff who meet periodically to explore an assessment topic, discuss a book chapter or journal article, and so on (Parsons, Lovato, Hutchinson, & Wilson, 2016) • A cadre of assessment mentors: Faculty and staff available to answer questions on aspects of assessment that they have experience with, such as testing, surveys, focus groups, or statistics Use professional development funds wisely.  A great way to model evidenceinformed decision-making is to use systematic evidence to decide how best to use professional development funds for assessment. Use assessment reports, aggregated

131

Creating a Culture of Evidence and Betterment

student evaluations of teaching, faculty surveys, and other systematic feedback to consider questions like these: • What questions about assessment are most common? • What are the biggest misconceptions about assessment? • What are the biggest barriers that keep faculty and staff from moving ahead with assessment? • What are the biggest shortcomings that you see in program, general education, and co-curricular assessment reports? Then use your findings to design professional development opportunities to meet the most pressing needs that you identify. As you consider professional development resources, don’t forget about sabbatical and travel funds and policies. Consider giving funding priority to: • Sabbatical leave requests to engage in scholarship of teaching (Chapter 14) • Travel requests to attend conferences and events that will help faculty increase their knowledge and skills on how to teach, design curricula, and assess • Sabbatical and travel requests that include plans to disseminate what’s been learned to colleagues across your college

Provide guidelines on what everyone is to do Any task is less daunting if we understand exactly what we are to do, and assessment is no exception. We joke that students’ most common questions about an assignment are “How long should it be?” and “When is it due?,” but many of us find our work easier to accomplish if we have similar structure and guidelines. Give faculty and staff clear expectations and guidance on precisely what they are to do, but at the same time, offer flexibility and options (Chapter 9). Develop a local statement of principles of good assessment practice.  A statement that describes characteristics of good assessment and your college’s commitment to fostering successful assessment practices assures faculty and staff that there will be college support for their assessment efforts and that assessments will be conducted and used ethically and appropriately. Such a statement might proclaim, for example, that evidence of student learning will not be used punitively (Chapter 26). While existing principles of good practice (Chapter 3) are a good starting point for developing a local statement, it’s usually not a good idea to adopt an existing statement wholesale, as your college culture undoubtedly has unique perspectives, values, and concerns. If faculty and staff are unionized, for example, a local statement might relate assessment expectations to contractual rights and responsibilities.

132

Define key terms.  While oral and written examinations have been part of education for hundreds of years, only in the last century have the theory and science

As much as everyone hates preparing reports, they’re a necessary part of the assessment process for three reasons. First and most important, they are a way to share evidence of student learning with those who can use the evidence to inform decisions (Chapter 25). Second, they are a learning opportunity for faculty and staff working on assessment, providing an opportunity to reflect on, analyze, and receive feedback on their assessment efforts. And regular requests for reports are an impetus to keep working on assessment, helping to make assessment a sustained rather than sporadic effort. To ensure that assessment reports achieve all three of these purposes, design them so they follow the principles for sharing evidence discussed in Chapter 25. Then develop written guidelines for the reports that mirror the traits of great learning activities (Chapter 16).

Helping Everyone Learn What to Do

View assessment reports as a learning opportunity

CHAPTER 10

of educational measurement been studied systematically. Because the assessment of student learning in higher education is relatively new compared to many other fields of study, and because it has been undertaken by people from disciplines with widely differing orientations, the vocabulary of assessment is not yet standardized. Chapter 1, for example, discusses several ways that the term evaluation is used. As faculty and staff are introduced to assessment, many won’t be familiar with the term assessment or will have conflicting or misinformed ideas about what it means. Having a locally accepted definition of assessment, along with simple, jargon-free definitions of some other potentially fuzzy terms such as goal (Chapter 4), program (Chapter 5), standard (Chapter 22), and rubric (Chapter 15) may help prevent confusion or disagreement over exactly what everyone is to do.

Articulate clearly the purpose, audience, and use of the report and what is to be included.  Explain the decisions that the report is to inform and how each element of the report will be used. Provide not only what to include but when the reports are due and to whom they should be submitted. It’s usually helpful if program assessment reports include brief summaries of the following: • The program’s major learning goals • How, when, where, and how frequently each key learning goal is being assessed • What has been learned from each assessment • How that evidence of student learning has been used, as appropriate, to improve teaching and learning and for stewardship and accountability (Chapter 6) • Any additional information that helps the reader understand what’s happening, such as plans to modify assessment strategies Provide the rubric that will be used to offer constructive feedback.  Just as students deserve to know the grading criteria for their assignments and what kinds of work will 133

Creating a Culture of Evidence and Betterment

earn an A, faculty and staff deserve to know the kinds of assessment practices that are considered acceptable at your college. Use your local statement of good assessment practice to create a rubric delineating the characteristics of exceptional, adequate, and inadequate assessment practices at your college. Design this rubric as carefully as you would rubrics to assess student work (Chapter 15), focusing on the key traits of effective assessment practices rather than minutia. Exhibit 10.1 is an example of such a rubric. Exhibit 10.1  A Rubric for Providing Feedback on Assessment Plans and Reports Needs Attention

Meets Standard

Best Practice

Roles and

Does not

The program has identified who

In addition to meeting the described

Responsi­bilities

yet meet the

coordinates the assessment of program

standard, program faculty, including

described

learning goals.

adjuncts as appropriate, understand their

standard.

roles and responsibilities in assessing student achievement of program learning goals. Faculty periodically reflect on their assessment efforts and identify ways to keep it as simple and useful as possible.

Key

Learning

Learning goals describe in explicit,

In addition to meeting the described

Learning Goals

goals do not

observable terms, using action words,

standard, key program learning goals

yet meet the

how students will be able to use their

are clearly and actively communicated to

described

knowledge, what thinking skills and

students and faculty in the program.

standard.

disciplinary dispositions they will have, and/or what else they will be able to do upon completion of the program.

Curriculum

Does not

Every student has sufficient opportunity

In addition to meeting the described standard,

Design

yet meet the

to achieve each learning goal, taking

faculty collaborate to ensure curricular

described

at least two courses that address the

alignment with key program learning goals,

standard.

learning goal.

and every student in the major has ample opportunity to master the learning goal through more than two courses.

Assessment

Does not

Each assessment method clearly

In addition to meeting the described standard,

Methods

yet meet the

matches the learning goal being

evidence is provided that the assessment

described

assessed, and multiple assessments are

methods yield truthful, fair information

standard.

used systematically over time.

that can be used with confidence. Program faculty have defined and justified rigorous but achievable standards for satisfactory achievement of each program learning goal

Use of Results

Does not

Assessment results are summarized,

In addition to meeting the described

yet meet the

shared, and discussed with faculty

standard, assessment results are shared

described

teaching in the program and are used to

with other audiences and are used to

standard.

modify learning goals, teaching methods, support planning and resource decisions, as and/or curriculum as appropriate.

134

appropriate.

Helping Everyone Learn What to Do

Offer flexibility in assessment report formats.  While a single template for assessment reports adopted for college-wide use can simplify college-wide analysis of evidence, it will inevitably force some programs and units to put the square pegs of their stories into the round holes of the template. Unless your accreditor explicitly requires that evidence be presented in a consistent format, offer a report template as an optional tool, not a mandate. Allow faculty and staff to tell their stories in ways that best work for them, asking only that they provide the information needed to fulfill the report’s purposes and that they follow the guidelines for sharing evidence effectively that are discussed in Chapter 25. Allow those who must submit an assessment report to a state agency or specialized accreditor to simply share relevant portions of that report rather than rewrite it into your template’s format.

CHAPTER 10

Templates for reports can educate everyone on good assessment practices.  ­Faculty and staff may better understand assessment expectations if you offer a simple reporting template as part of your guidelines. Templates can also make the job of reporting easier because, in subsequent years, the report writer can simply update the previous year’s report. A brief, straightforward word processing document or spreadsheet may be all that’s needed, although some colleges use the templates embedded in assessment information management systems (Chapter 11). Exhibit 10.2 is an example of a simple template for an annual assessment report. Just as students sometimes better understand your expectations for an assignment if they see a sample of a completed assignment, faculty and staff may better understand your assessment expectations if they see a sample of a completed template, either for a real or a hypothetical program.

Provide constructive feedback.  Just as students learn best when they receive prompt, constructive feedback on their work, so do faculty and staff as they learn how best to assess student learning. As suggested earlier, use rubrics (Exhibit 10.1) to offer feedback. Point out good practices as well as areas for improvement, using gentle, supportive language that frames the feedback as collegial advice rather than approval or disapproval. Offer to follow up with a meeting or other personalized assistance to help faculty and staff figure out how to improve their assessment practices. To help keep assessment efforts moving along, keep the review process as fast and simple as possible. Just as when we teach students, brief feedback provided within a couple of weeks is far more helpful than extensive feedback provided months later. Who should provide feedback? Because faculty should have leadership responsibility for assessment (Chapter 9), it’s appropriate for faculty peers such as a faculty-led committee to provide feedback. If committee members initially don’t feel 135

Through what courses/ learning activities do you ensure that all students have enough opportunity to learn this?

How and where in the curriculum are you assessing how well your students are learning this?* How often do you conduct this assessment?

Summarize your assessment results: What have you learned about how well you are achieving this goal?

What is your standard for adequate student learning, and what is the basis for that standard? How do your results compare with your target?

*Please attach your assessment tools (assignments, grading criteria, scoring guidelines, surveys, and the like)

6

5

4

3

2

1

Key learning goal: What should students be able to do AFTER completing the program?

Exhibit 10.2  A Template for an Annual Program Assessment Report How have you used this information to help your students learn?

Optional comments (such as changes to goals, assessments, or schedule)

1. What are the three biggest things that faculty and staff at your college most need or want to learn about assessment? a. Do you feel you can answer this question reasonably accurately, or do you need to conduct a needs assessment of some kind?

Helping Everyone Learn What to Do

Time to think, discuss, and practice

CHAPTER 10

qualified to generate feedback, the assessment coordinator might draft responses that the committee can review and finalize. But make this an interim rather than a permanent approach. Who should receive feedback? Obviously a program’s assessment coordinator should receive feedback on its assessment report. Whether the feedback should be shared with all faculty and staff in the program or with the program’s dean or vice president depends on college culture. The aim of all communications should be to support assessment efforts, not to impede the process by creating problems or perceptions of problems.

b. Realistically, what might be done to help them learn those things? 2. Reflect on what you most need to learn about assessment, and draft a plan to try to learn those things.

137

Some valuable ideas you’ll find in this chapter Help everyone find time to talk!

CHAPTER 11

Supporting Assessment Efforts Choose technologies that support your assessment processes, not the other way around. Choose technologies that help many people, not just a few. Make technologies an optional tool, not a mandate.

139

Creating a Culture of Evidence and Betterment

T

he major resource needs of most assessment efforts are: • Professional development to help everyone understand what they are to do, as discussed in Chapter 10 • Individuals and groups to guide and coordinate assessment efforts, as discussed in Chapter 9 • Time for everyone to engage in assessment and use the resulting evidence In addition, some colleges and programs find that technologies such as assessment information management systems are worthwhile investments. And some colleges have other resource needs such as published tests and surveys. This chapter discusses how best to support assessment efforts with time, technologies, and other resource needs. As you consider how best to support assessment efforts, consider the costeffectiveness of time and resource investments, as discussed in Chapter 12. If an assessment is especially costly in either dollars or time, ask yourself how likely it is to lead to meaningful improvement in how, what, or why students learn. If significant benefits aren’t likely, the expenditure is probably not worthwhile.

Time One of the most common concerns expressed by faculty and staff about assessment is “But when I am supposed to find the time to do this?” The answers are to stop doing something else, consider released time, and help everyone find time to talk.

Stop doing something else There are only so many hours in a week, and we can’t create more time. So the main way to find time to work on assessing and improving student learning is to stop or scale back something else. If faculty and staff are told they must continue to do everything else while they tackle assessment, the implicit message is that everything else is more important than assessment – in other words, assessment really isn’t all that important. If stopping something altogether seems too draconian, try scaling it back or putting it on hiatus for a semester or year and see what happens. What would happen if, for example, a particular committee didn’t meet for the rest of the year? What might you stop or scale back? Try keeping logs of how you and your colleagues spend your time. Then analyze the logs to identify work that consumes considerable time with relatively little benefit to your college and its students. Also consider any of the following ways to find time (Suskie, 2014):

140

Review everyone’s work in light of your college’s mission.  Some work is essential to achieving the mission; other work may be tangential. Review the costs and benefits of those tangential activities and decide whether to continue them.

Look at student enrollment patterns, and remind yourself that lean curricula may increase student success rates (Chapter 5). Then ask yourself if a course, program, or other learning experience that attracts relatively few students is more important than the time freed up if it were no longer offered.

Supporting Assessment Efforts

Fight curriculum bloat.  It takes considerable time to offer a wide array of courses, programs, and other learning experiences. Each one needs to be reviewed, updated, aligned, planned, taught, and assessed. Syllabi for every course must be developed and maintained; learning activities must be planned and assessed. The fewer courses, programs, and other learning experiences you offer, the more time everyone has for assessment.

CHAPTER 11

Limit your learning goals.  The more learning goals you have, the more work you have to keep them updated, aligned, and assessed. As discussed in Chapter 4, it’s usually better to help students achieve a few important learning goals well than to rush through many. Simply put, quality is more important than quantity. Ask yourself and your colleagues, “What are the most important things we want students to learn?”

Look at how time is spent within courses.  Consider dropping or shortening the midterm examination to free up time to assess student projects. Consider flipped classes (Chapter 5): Moving some lectures online for students to view or read on their own, creating more class time for students to collaborate on assignments and for faculty to review assignments with individual students. Or move tests online, freeing up class time for other kinds of assessments such as group presentations. Look at committees.  Many colleges have too many, and committee work has a way of expanding to fill the time allotted for meetings! Look for ways to simplify college processes.  If your college has a process of ongoing academic program review (Chapter 26), for example, consider streamlining it to deemphasize those elements that are relatively tangential to student learning and student success. Synchronize accreditation and accountability processes.  Consider the following questions:

• Do assessments of program learning goals feed into program reviews (Chapter 26)? Do the evidence, analysis, and conclusions of program reviews feed into accreditation reviews? • Can the assessment section of a recent specialized accreditation report be cutand-pasted into a regional accreditation report? 141

Creating a Culture of Evidence and Betterment

• Are a specialized and regional accreditor willing to do a collaborative process in which you submit one report to both of them and then host a joint visit by both of them? • Is an accreditor willing to adjust the date of your next review by a semester or year, forward or backward, to help you synchronize your workload in preparation for multiple accreditation reviews?

Consider released time Think about giving released time to faculty and staff charged with launching particularly time-consuming assessments outside their normal purview, such as the assessment of a major institutional or general education learning goal. Or exempt them from other committee service or writing the usual annual report of department activities.

Help everyone find time to talk Sometimes the biggest problem with getting assessment done is simply finding time for faculty and staff to talk – about what they most want students to learn, about how to modify the curriculum, about the evidence of student learning that they’ve collected and analyzed, and about how to improve student learning. Consider carving out times in the academic calendar when faculty can have these important conversations. Some colleges don’t schedule any classes on, say, Wednesdays at noon, giving departments and committees time to meet. Some set aside professional development days or assessment days at the beginning, middle, and/or end of the semester. Think twice about filling these days with a program that everyone is expected to attend; today it’s the rare college where everyone has the same professional development needs and will benefit from the same program. Instead, consider asking each department or unit to design its own agenda for the day. Offer some professional development opportunities that faculty and staff may choose to attend depending on their interests and needs.

Technologies Today many colleges and universities are finding it worthwhile to invest in technologies to support assessment activities, along with technical and support staff to help with the mundane but essential tasks of collecting, processing, and analyzing evidence of student learning. Technologies can help with a number of tasks (List 11.1). 142

■■ ■■

■■ ■■ ■■ ■■ ■■

Today there are literally dozens of technologies that can support assessment, including the following: • Assessment information management systems • Learning management systems (see Jargon Alert) • Curriculum mapping software • Planning management systems • Accreditation reporting systems • Learning analytics systems • Cloud servers to retain documentation of assessment processes and resulting evidence • Statistical analysis software • Qualitative analysis software • Data visualization software • Rubric development software • Testing software • Portfolio management systems Choosing an appropriate technology can be an overwhelming process. Not only are there dozens of options, but some can Jargon Alert! be major investments and the choices are Learning Management Systems rapidly evolving – some of today’s key players Learning management systems are data didn’t exist a decade ago. Because choices are management software systems designed to deliver and manage educational courses and programs. changing so quickly, this book cannot cite or Examples include Blackboard, Canvas, Jenzabar, recommend any particular systems. But here and Moodle. are some steps to follow to make sure your technology investment is a wise one.

Supporting Assessment Efforts

■■

Entering, processing, and analyzing evidence of student learning Maintaining student portfolios online Sharing and presenting evidence of student learning (Chapter 25) Maintaining assessment plans, evidence of student learning, and other records Creating online tests and/or surveys Creating scannable bubble-sheet paper tests and/or surveys Creating scripts and response entry forms for computer-assisted telephone surveys Making audiovisual recordings of student presentations and performances

CHAPTER 11

List 11.1  Assessment Tasks That Technologies Can Help With

Clarify the requirements and expectations of your accreditor(s).  While a few accreditors expect assessment information to be compiled and presented in a 143

Creating a Culture of Evidence and Betterment

particular way (for example, in online student portfolios), most do not require or expect that colleges invest in or use particular technologies to meet accreditation requirements. Review the purposes of your assessment efforts  (Chapter 6) so you choose technologies that help you achieve those purposes. Keep in mind that the fundamental purpose of assessment is to give your students the best possible education, not to get through accreditation, and your technologies should support that fundamental purpose. Failing to do this is probably the biggest mistake colleges make when choosing assessment technologies.

Identify the key problem(s) that you want technologies to solve. For example, you may want to: • Ease the time and effort that faculty and staff spend compiling and analyzing evidence of student learning. • Archive assessment work, so everyone is not continually reinventing the wheel through inevitable faculty and staff transitions and can examine trends over time. • Help students synthesize their learning by reviewing their work in a portfolio (Chapter 18). • Help public audiences quickly grasp your college’s story (Chapter 25). Give a leadership role in the decision to the people who will use the product:  Faculty and staff on the assessment front lines. Make them, not the information technology staff, the drivers in identifying and choosing technologies. Draw on relevant faculty expertise. Faculty teaching visual communication, for example, can be helpful in identifying appropriate presentation software. Choose technologies that help many people, not just a few.  Don’t choose an assessment information management system primarily to help an assessment or accreditation committee aggregate and analyze evidence of student learning across your college. This choice can lead to a lot of people doing a lot of extra work just to ease the workload of a small group. Choose a system that makes everyone’s jobs easier. Identify technologies that support your assessment processes, not the other way around.  Many colleges choose an assessment information management system and then create assessment processes to conform to the system. Some colleges, for example, require everyone to use a rubric because the system they’ve adopted is designed around rubrics. This invariably leads to trying to push some square pegs

144

Don’t get what you don’t need.  There’s no point in investing in expensive technologies to generate and maintain assessment reports if reports generated through word processing are working fine for everyone. Similarly, there is no point in investing in statistical analysis software if no one wants to do anything more than tally results.

Supporting Assessment Efforts

To make sure you choose the right system, first design a low-tech process, perhaps having everyone document their evidence of student learning in simple spreadsheets and reporting on them in simple word processing documents. This initial experience will help faculty and staff understand their goals and needs, come to realize that additional technologies will make their lives easier and the process more useful, and make an appropriate technology choice. Once your low-tech process works reasonably well, look for an assessment information management system that mirrors it but reduces everyone’s workload or solves other problems.

CHAPTER 11

into round holes, forcing everyone to use a rubric when sometimes a qualitative assessment tool such as reflective writing (Chapter 20) or multiple-choice tests (Chapter 17) may be a better fit with a program’s learning goals and culture.

Make technologies an optional tool, not a mandate.  You want faculty and staff to welcome the technology as solving a pervasive problem; you don’t want them to view the need to learn how to use the technology as another assessment burden. If you encourage but don’t require faculty and staff to use new technologies, the early adopters can serve as advocates to their more reluctant colleagues. Once those who are not technologically savvy can see firsthand that the technology makes assessment work faster and easier, they’ll be more enthused about using it. Consider whether your college has adequate staff support  to implement the technology successfully. Many systems cannot be implemented without the ongoing support of staff with technological expertise. Identify potential vendors  through conference exhibits and presentations, published literature, colleagues at other colleges, and online searches and discussion lists. Ask vendors for references  – colleges currently using their technologies – and ask the assessment coordinators and/or assessment committee members at those colleges about their experiences (List 11.2).

145

Creating a Culture of Evidence and Betterment

List 11.2  Questions to Ask Vendor References About Assessment Technologies ■■ ■■ ■■ ■■

■■ ■■ ■■

■■

■■

■■

How much does the technology cost: Both initial cost and annual costs? How widely is it used at your college? Who is using it? Which features are being used? Does the technology generate summaries or analyses of student learning evidence? Are they visuals (Chapter 25)? How readable and useful are they? How user-friendly is the technology? How much and what kinds of training have faculty and staff needed to use the technology? How well does the technology interface with other technologies used at your college? For example, how well does an assessment information management system interface with your learning management system? Does the technology have any particular hardware or software requirements? Does it require, for example, that personal computers use a particular operating system, have a certain amount of memory, or use a particular internet browser? What kind of support is needed from information technology staff to use and maintain the technology? If you could make this decision over again, would you still invest in this technology? Why or why not?

Ask vendors for demonstrations  of the technology. These are often available online. Consider the balance of costs versus benefits.  Assessment technology investments can run into tens or even hundreds of thousands of dollars, not including local training and staff support. This is a major investment, so make sure you have a clear sense of the payoffs and whether they are worth the costs.

Other resource needs Beyond technologies, potential expenses related to assessment can include any of those in List 11.3. List 11.3  Potential Assessment Resource Needs ■■

■■ ■■

■■

146

Resources to address disappointing outcomes (Chapter 26) including curriculum development grants (Chapter 14) Faculty and/or staff to guide and coordinate assessment (Chapter 9) Professional development funds for faculty and staff to learn about teaching, curriculum design, and assessment (Chapter 10), including resources for books, periodicals, and conference attendance Budget supplements, stipends, or extra compensation for extraordinary assessment work, such as

■■ ■■ ■■ ■■ ■■ ■■ ■■

Time to think, discuss, and practice

Supporting Assessment Efforts

■■

coordinating assessment of a general education requirement or helping to read and score large numbers of student papers or projects Annual events such as an assessment fair or a celebratory luncheon Published tests and surveys Incentives for students to participate in add-on assessments (Chapter 21) Staffing for telephone surveys and focus groups (Chapter 20) Refreshments for assessment meetings and training sessions Audio or videotaping and transcription of focus groups Videotaping of student presentations Printing paper-and-pencil tests and surveys

CHAPTER 11

List 11.3  Continued

1. Can you think of any committees at your college whose work might be put on hiatus for a semester or a year? What would be the impact of doing so? 2. Imagine that your college’s co-curricular offerings have received a grant to invest in an assessment information management system. There are no constraints on what might be chosen; the system doesn’t need to conform to any college or system technology policy, and it doesn’t need to interface with other college information management systems. Identify up to three system characteristics that would be especially important to have – that would most help co-curricular staff and other decision-makers. 3. The service learning program at Rose Hill College is planning an assessment of its key learning goals. Ask one member of your group to play the role of the director of the program and answer your questions about the program’s learning goals, scope, assessment plan, and so on. What resources in List 11.3, if any, might the director need to carry out the assessment? Which of those would be essential versus nice-to-have?

147

Some valuable ideas you’ll find in this chapter

CHAPTER 12

Keeping Assessment Cost-Effective If assessment is to be sustained, it must be cost-effective, yielding benefits that justify the time and money put into it. Think twice about blind-scoring or double-scoring student work. Choose a source of assessment information that provides evidence of student achievement of multiple learning goals.

149

Creating a Culture of Evidence and Betterment

T

he more complex your assessment processes, the more precious time they consume. Faculty and staff may be willing to make an extra push on assessment to get through an accreditation review, but after that assessment fatigue can set in. Simple, practical, cost-effective processes are far more likely to be sustained, while overly complicated or ambitious processes quickly collapse under their own weight. This chapter shares a variety of strategies for keeping assessment simple and costeffective and thereby sustainable. Not all of them may be feasible or appropriate for your college and setting; choose the strategies that you think will be most likely to work.

Monitor your investment in assessment The business world’s concept of return on investment applies to assessment. Assessments should yield dividends – namely better decisions leading to more effective student learning experiences – sufficiently worthwhile to justify our investment of time, effort, and resources. They should not take so much time that they detract from the far more important work of teaching. As suggested in Chapter 7, periodically review assessment activities in terms of time and resources invested. Rethink any assessments that, while useful, have sapped too much time or resources.

Keep it useful Chapter 3 emphasizes the most important characteristic of effective assessment processes: They yield evidence of student learning that is used to inform important decisions on important goals. If you’ve found that an assessment you’ve been conducting hasn’t been useful, stop doing it and start doing something else! Conduct only assessments that will impact important decisions.  In fact, conduct only assessments that will help make better or different decisions than what you might make without them (Hubbard, 2014). Conduct only assessments that focus on areas of great interest and great need, so people see value in spending time on them. Focus on significant learning experiences.  Ask yourself, “How much do we really need to assess this learning goal in this setting? What are the consequences if we don’t assess it?” (Hubbard, 2014). Don’t require every administrative and student support unit to have and assess learning goals, for example, as discussed in Chapter 8. Focus assessment efforts where significant, meaningful learning is expected.

150

Focus on high-enrollment learning experiences.  When offered a choice of courses, students tend to gravitate toward a relatively small subset of what’s available. They’re

Faculty and staff at hundreds of colleges throughout the world have made tremendous strides in developing new and improved strategies to assess student learning. Capitalize on their work by researching current thinking on how each learning goal might be assessed and looking for models that you can adapt to your circumstances. List 12.1 suggests some places to look. Of course, if you use or adapt someone else’s rubric, survey, or other assessment tool, ask for permission and acknowledge the work of the original author.

Keeping Assessment Cost-Effective

Don’t reinvent the wheel

CHAPTER 12

following the Pareto principle or 80-20 rule: Eighty percent of students enroll in just twenty percent of the courses offered. No matter how many general education courses your college offers, for example, there are probably no more than 15 or 20 courses that the vast majority of students take to complete their general education requirements. Start your assessment of general education learning goals by assessing student learning in just those 15 or 20 high-enrollment courses. Your resulting evidence will have broad impact on the great majority of your students, with far less work than assessing learning in every general education course. You can then assess the other general education courses . . . but after you consider whether you should continue to offer them as general education courses when relatively few students take them for general education credit (Chapter 5).

List 12.1  Where to Look for Assessment Ideas ■■

■■ ■■

■■

■■

Online searches, because many colleges, programs, and faculty post rubrics, surveys, and other assessment tools online Online discussion lists or boards on teaching in your discipline or co-curricular area Professional and disciplinary associations, which may maintain online warehouses of rubrics and other assessment tools. Three websites that are particularly rich sources of information on how to assess student learning: • The National Institute for Learning Outcomes Assessment’s website (www .learningoutcomesassessment.org) offers a wealth of publications and resources. • The massive Assessment Commons website (AssessmentCommons.org) offers hundreds of (unvetted) links to assessment guidelines, institutional assessment websites, and information on assessment of specific disciplines and skills, among other resources. • The Association of American Colleges and Universities’ website (www.aacu.org), while focusing on assessment in general education and the liberal arts, offers resources that will be helpful to anyone assessing almost any kind of student learning goal in any setting. Reviews of published scholarly literature for examples of rubrics, surveys, and other assessment tools

151

Creating a Culture of Evidence and Betterment

Should you adopt an existing assessment tool wholesale?  Adopting a rubric, survey, or other assessment tool used by another college makes sense only if the assessment tool reflects your learning goals (Quinlan, 2012) and your learning activity – “conditions not easy to achieve” (Walvoord, 2010, pp. 13–14). A more useful option may be to adapt one or more existing rubrics or surveys to align better with your learning goals and circumstances (Morgaine, 2010). Or simply use existing rubrics to “spur faculty discussion about common criteria” (Walvoord, 2010, p. 13).

Start with what you have Every college has some assessment information on hand. Maximize the use of existing information before creating or purchasing new assessment tools. Ask your colleagues: • How do we know that our students are successful? How do we know that they are indeed learning what we value? • How confident are we in our conclusions about what students have learned? Why? Are our conclusions based on solid evidence or on anecdotes and impressions? • How would we convince board members, employers, and other audiences that graduating students have in fact learned these things? • If we’re not sure if our students have achieved our goals, do we at least have some clues? List 12.2 gives examples of direct and indirect evidence of student learning (Chapter 3) that may already be on hand. Lists 3.3 and 3.4 in Chapter 3 offer more ideas. Even if available assessment information is incomplete or isn’t rigorous, it may still yield hints about what’s working and what might be changed. List 12.2  Examples of Evidence of Student Learning That May Already Be on Hand Direct Evidence ■■ Class tests and assignments ■■ Scores on published tests, such as tests for placement, certification, or licensure ■■ Ratings of students by field experience supervisors ■■ Assessment information assembled to meet specialized accreditation requirements ■■ Grading criteria for class assignments and tests Indirect Evidence ■■ Grades ■■ Retention and graduation rates ■■ Placement rates of graduates into appropriate careers and/or subsequent academic programs ■■ Surveys of students and alumni ■■ End-of-course student evaluations of teaching ■■ Counts of students participating in co-curricular experiences

152

Focus on your most important learning goals.  Focus assessment efforts on a limited number of learning goals that you and your colleagues feel are most important – perhaps no more than three to six, as suggested in Chapter 4. Don’t collect more evidence than you can handle.  A 30-question alumni survey will yield reams of results that must be tabulated, analyzed, and discussed. A sixquestion survey, while not yielding the same breadth of information, will take far less time to implement, summarize, share, and use. Similarly, a one-page rubric may not address your learning goals as thoroughly as a two- or three-page rubric, but it will force you to focus on your most important learning goals and yield results that are far easier to summarize and analyze. Also consider limiting the volume of assessment information that you ask students to provide. As Chapter 16 suggests, consider carefully whether the time that students put into your assignment – and that you put into grading or otherwise assessing it – yields a proportionate payoff in terms of their learning. Perhaps a two-page abstract will help them hone their writing skills as much as a ten-page term paper – and give you just as much evidence of student learning and take far less time to assess. List 16.1 in Chapter 16 offers suggestions for alternatives to traditional papers and essays that may be faster – and more interesting – to assess.

Keeping Assessment Cost-Effective

It’s impossible to assess everything that we want students to learn, so don’t even try. Think of assessment as putting together a jigsaw puzzle when we don’t have enough time to assemble the entire puzzle. We want to put together just enough pieces to get a reasonably good sense of what the completed picture would look like.

CHAPTER 12

Aim for just enough evidence

Have reasonable expectations for quality While the higher education community needs more scholarly research on how to teach and assess, realistically most faculty and staff don’t have the time – or interest – to conduct assessments that consistently meet the standards for publication in peer-reviewed research journals. As discussed in Chapter 1, most assessment is a form of action research, a type of research that, while disciplined and systematic, is inherently imperfect. So don’t expect perfection – in fact, acknowledge the old saying that the perfect is the enemy of the good. Aim not for replicable, generalizable research but for evidence that is simply good enough and relevant enough to use with confidence to make decisions about educational quality, stewardship, and accountability (Chapter 6). 153

Creating a Culture of Evidence and Betterment

Weigh the consequences of making the wrong decision with imperfect evidence.  Consider this scenario: faculty teaching a general education statistics course assess students’ quantitative reasoning skills. The students’ skills aren’t very good, but there’s a problem: The faculty have evidence from just two of 15 course sections, so their sample might be off. Should they take steps now to try to improve students’ quantitative skills in this course, or should they wait another semester to collect more evidence? The faculty face five possible decision outcomes:

Outcome 1. Faculty decide to do nothing, and students in subsequent courses do just fine without any changes. (In other words, this was indeed an off sample.) Outcome 2. Faculty decide to do nothing, and students in subsequent courses continue to finish the course with inadequate quantitative skills. Outcome 3. Faculty change how they teach the course, and subsequent students do better because of the changes. Outcome 4. Faculty change how they teach the course but, despite the faculty’s best efforts, the changes don’t help improve students’ quantitative skills. Outcome 5. Faculty change how they teach the course, and subsequent students do better, but not because of their changes – the students are simply better prepared than this year’s students were. So the risk of doing nothing is getting Outcome 2 instead of Outcome 1: Future students doesn’t learn the quantitative skills they need to learn, and they consequently run into trouble in later courses or their jobs until faculty eventually decide to make some changes. The risk of changing things, meanwhile, is getting Outcome 4 or 5 instead of Outcome 3: The faculty make changes, but the changes don’t help. The consequence here is the faculty’s wasted time. If their college invests in resources such as an online tutoring module or some released time to work on this, there may be wasted money as well. The question then becomes, “Which is the worse consequence?” Normally the first consequence – continuing to pass students with inadequate quantitative skills – is worse than the second – wasting time and resources on changes that don’t help. If the first consequence is indeed the worse one, it makes sense to go ahead with changes even with possibly insufficient evidence. But if the second consequence involves a sizable investment of time or resources, then it may make sense to wait for corroborating evidence before making that major investment. Here’s another way to look at this: Faculty are always making informal decisions about changes to their teaching based on imperfect evidence. Assessment should simply help them make somewhat better-informed decisions, not infallible decisions (Hubbard, 2014).

154

Blind Scoring and Double Scoring In blind scoring, identifying information is removed from samples of student work before the work is assessed with a rubric by an assessment committee or other group. In double scoring, two faculty or staff members assess each sample of student work, the two ratings are compared, and if they differ sometimes a third person assesses the work.

Think twice about blind scoring and double scoring student work.  List 3.6 in Chapter 3 suggests strategies to minimize assessment errors and biases and achieve greater assessment consistency, including blind scoring and double scoring (see Jargon Alert). While blind scoring and double scoring can be good research practices, they have costs.

Keeping Assessment Cost-Effective

Jargon Alert!

CHAPTER 12

Streamline implementation.  Quick, useful results can help build enthusiasm for assessment, so “don’t let anxiety about what could go wrong” with assessment keep everyone from getting started (Hubbard, 2014, p. 196). For example, don’t ask faculty and staff to wait until they have a complete, approved plan in place to begin collecting assessment information. Encourage everyone to begin with small-scale assessment projects that they can expand later. If they want to try a portfolio (Chapter 18), for example, suggest that they start by including in it just one or two examples of student work that assess just one or two key learning goals. Once the faculty and staff have worked out logistical kinks, they can gradually expand the scope of their assessment efforts.

• Faculty and staff doing the scoring must sometimes be compensated for their extra work. • There may be a cost in morale, as faculty providing the student work may wonder why they aren’t trusted enough to assess their students’ work themselves. • Double scoring literally doubles the assessment workload. • Blind scoring disengages rank-and-file faculty from the assessment process, leading them to view themselves as providers of evidence of student learning, not consumers of it. To decide if your college should be deploying scarce resources of time and money on strategies to minimize assessment errors and biases and increase assessment consistency, consider the questions in List 12.3. If you’re not sure what to do, try a simple experiment: Ask some faculty to score their students’ work themselves and see if their aggregated scores differ significantly from the aggregated blind scores. If the aggregated scores do differ, try to improve consistency by using other strategies in List 3.6 in Chapter 3. Implement blind scoring or double scoring as a last resort.

155

Creating a Culture of Evidence and Betterment

List 12.3  Is It Worth Taking Extra Steps to Minimize Assessment Errors and Biases and Increase Consistency? ■■

■■

■■

■■

What decisions will this evidence of student learning inform? Minimizing assessment errors and biases becomes important when student learning evidence is used to inform major decisions, such as whether students graduate or whether a program continues to be funded. Do you have problems with assessment errors and bias? Look at the rubric ratings that faculty and staff are assigning to student work. Are they already reasonably consistent, or are some faculty and staff more lenient or more stringent than the majority? If there may be reason for concern, do a spot check: Ask a couple of faculty or staff members to blindly reassess a few student work samples to verify that there is indeed a problem with consistency. Are the assessments already yielding evidence of sufficient quality that you can use them with confidence for the decisions they’re intended to inform? Remember that quality (validity) is more important than consistency (reliability). See Chapter 24 for strategies to analyze the quality of your assessments. Will the added confidence from increasing assessment consistency be worth the added cost in time and money?

Use assessments that do double or triple duty Look for assessments that can provide helpful information on multiple learning goals. Course or program capstone assignments (Chapter 5) are great opportunities to do this. A research paper, for example, can yield information on students’ writing skills, research skills, information literacy skills, and analysis skills. If learning goals are connected across settings (Chapter 4), a single assessment may provide evidence of student learning in multiple settings. Suppose, for example, that Church Hill College has an institutional learning goal that students will use and analyze a variety of resources to make decisions and solve problems. Now suppose that the English BA program has a goal that students will conduct research on issues in the study of English literature – a program learning goal that helps students achieve the institutional learning goal. Both goals might be assessed for English majors by having seniors complete a required research paper on an issue in the study of English literature and assessing the papers using a rubric that addresses the program and institutional learning goals. Use embedded assessments.  Embedded assessments generally require less extra work than add-on assessments (see Jargon Alert). And because they are typically designed locally by faculty and staff, they often match up well with local learning goals and yield information that faculty and staff value (Hutchings, Jankowski, & Ewell, 2014). They are therefore particularly likely to be used to understand and improve teaching and learning. 156

CHAPTER 12 Keeping Assessment Cost-Effective

Start at the end and work backward.  Jargon Alert! Begin by looking at your students shortly before they complete their learning Embedded Assessments and Add-On Assessments ­experience: The end of your course or Embedded assessments are course assignments and learning activities that can provide evidence ­co-curricular experience or their last of student achievement of program, general semester before they graduate from your education, or institutional learning goals. A paper that students write in an advanced course, for program. Capstones (Chapter 5) are a great example, can show not only what they have place to conduct these summative assessments learned in that course but also the writing skills that they have developed throughout the entire (see Jargon Alert). If you are satisfied with program. Add-on assessments, discussed in ­student achievement of your learning goals at Chapter 21, are ungraded assessments that are not this point – say, they are writing ­beautifully – course or program requirements. there may be no need to drill down into their achievement in earlier courses. Drill down only for those learning goals for which you Jargon Alert! have disappointing outcomes. One exception to the principle of starting Summative and Formative Assessments at the end and working backward is when a Summative assessments are those completed at the end of a course, program, or other learning sizable proportion of students leave before experience. Formative assessments are those completing the learning experience. Students undertaken while student learning is taking place rather than at the end. Because formative in transfer associate degree programs, for assessments are done midstream, faculty and example, often transfer to a four-year college staff can use evidence from them to improve the learning of current students by making immediate before graduating. As discussed in Chapter 8, changes to classroom activities and assignments in these situations consider assessing student and by giving students prompt feedback on their strengths and weaknesses. Summative evidence learning in a cornerstone course that students of student learning can be used to improve the typically complete before leaving. This helps learning of future student cohorts. ensure that they’re on track to achieve key learning goals elsewhere. Another exception to the principle of starting at the end and working backward is when you’re launching a new program. In this situation, you don’t want to wait until the end of the program to find out that students haven’t learned what you want them to. Plan on some formative assessments (see Jargon Alert) in foundational and cornerstone courses (Chapter 5). Then you can adjust the remainder of the program curriculum as appropriate.

Consider samples Obviously, the more assessment evidence you collect and consider, the greater confidence you’ll have in your conclusions about student learning. Faculty and staff who look at 300 essays will have more confidence in their conclusions about student 157

Creating a Culture of Evidence and Betterment

writing skills than those who look at ten essays. But more evidence means more precious time spent collecting and examining it, so an important question is how much evidence is enough. Should you collect evidence from everyone or just a sample?  As discussed in Chapter 5, learning goals are promises. We are promising our students, their families, employers, and society that students who successfully complete a course, program, general education curriculum, or other learning experience can do the things we promise in our learning goals. As a matter of integrity, we should therefore make sure, through assessment, that every student who completes a learning experience has indeed achieved its learning goals. There are a few situations, however, where it is more practical to look at samples of students rather than everyone (List 12.4). In these situations, a representative sample can yield information that is almost as accurate as information from everyone. Consider that professional pollsters, trying to determine the opinions of millions of people, rarely survey more than 1,000 people.

List 12.4  Four Situations Where Samples May Make Sense ■■

■■

■■

■■

158

Published instruments. As discussed in Chapter 19, published tests and surveys get us out of the ivory tower, providing a reality check on how our college or program and our students stack up against peers. But published instruments can be expensive, and the cost of administering them to everyone may be prohibitive. Local surveys, interviews, and focus groups. While the dollar costs of these assessments are not as high as published instruments, the time and effort needed to conduct them with all students may be prohibitive. These are indirect assessments (Chapter 3), intended to add insight rather than provide conclusive evidence of student learning, so it’s usually not essential to have all students participate in them. Making evidence of student learning part of a broader analysis. For many faculty, the burden of assessment is not assessing students in classes – they do that through their grading processes. The burden is in the extra work of folding their assessment into a broader analysis of student learning across a program or general education requirement (Chapters 7 and 24). Sometimes faculty submit rubric or test scores to an office or committee; sometimes faculty submit actual student work; sometimes a committee assesses student work. These additional steps may be laborious and timeconsuming, and samples of student work may save considerable time. Learning goals that you don’t expect everyone to achieve. While many colleges aim to prepare students to engage in lifelong learning or in the civic lives of their communities, these are not the kinds of learning goals that the college can realistically promise that every student will achieve. For such aspirational learning goals, it may be fine to look at samples of students to estimate how many are indeed on the path to achieving them.

CHAPTER 12 Keeping Assessment Cost-Effective

How large of a sample should you examine?  Jargon Alert! Suppose that faculty and staff assess a sample of 200 essays and find that 84 percent of them Error Margin An error margin isn’t really an error; it’s a are at least satisfactory in terms of overall phenomenon that exists because even a good writing skill. They would like to be able to say random sample is unlikely to mirror all students that 84 percent of all 3,500 students at their precisely. It’s an estimate of the maximum difference between the percentage observed in the sample and college write satisfactorily. Assuming that the the true percentage of all students from whom the students in the sample mirror all students, sample was drawn. An outcome that “84 percent of student papers were satisfactory, with an error can they say this? No! It’s very unlikely that margin of plus or minus seven percent” means exactly 84 percent of all students write satisthat those conducting the assessment are very sure (95 percent sure) that between 77 percent and 91 factorily. The real overall percentage may be percent (84 percent plus and minus seven percentage 83 percent, 82 percent, or even 91 percent. points) of papers from all students are satisfactory. The sample’s 84 percent is an estimate of the true percentage, and the possible discrepancy between 84 percent and the true percentage is called an error margin (see Jargon Alert). The sample size you choose depends on how large an error margin you’re willing to tolerate. Table 12.1 lists the error margins of various sample sizes.

Table 12.1:  Error Margins of Various Sample Sizes Random Sample Size

Error Margin

9604

1%

2401

2%

1067

3%

600

4%

384

5%

264

6%

196

7%

The error margin for other sample sizes can be estimated with this formula: 1 100% where n your sample size n

So if, for example, faculty and staff assess 200 essays, the approximate error margin is: 1 100% 200

.005 100% .07 100% 7%

While professional pollsters often aim for samples of about 1,000 people, with an error margin of three percent, unless your assessments may lead to major (read expensive) changes, a sample of no more than 300 or 400 is probably sufficient. But what if 300 or 400 is still too large to be practical? What if faculty and staff have the 159

Creating a Culture of Evidence and Betterment

time to score only, say, 50 or 100 essays? Yes, you can use smaller sample sizes – whatever number you think is feasible – if you recognize that your error margin will be larger. If the assessment will likely lead only to minor changes, such as adjusting the curriculum in a course or two, a smaller sample may be fine. Another approach to determining an appropriate sample size is to look at sequential samples. Start by looking at a representative sample of 10 essays. Then look at another representative sample of 10 essays to see if they add any new insight. If they do, look at a third sample of 10 essays. Eventually you will look at an additional sample of 10 essays that adds no new insight and, at that point, you may conclude that you’ve looked at enough essays. What if you have a very small program or college? Obviously, you don’t need to examine 300 papers if you have only 250 students in your program. Table 12.2 lists the sample sizes needed for a five percent error margin from some relatively small groups of students. Table 12.2:  Sample Sizes Needed from Small Groups for a 5% Error Margin Number of Students You Are Sampling From

Random Sample Size

1000

278

500

217

350

184

200

132

100

80

50

44

The ultimate answer to how much evidence is enough is to use your common sense. Collect enough evidence to feel reasonably confident that you have a representative sample of what your students have learned and can do. The sample should simply be large enough and representative enough that you can use the resulting evidence with confidence to make the kinds of decisions discussed in Chapter 6. A good test of this is to imagine you’re making a money bet on the resulting evidence (Hubbard, 2014). How much money would you be willing to bet that the evidence from your sample is accurate enough to support whatever decision it will inform? If you wouldn’t be willing to risk much money and the decision is a major one, you need a better sample. How might you choose a representative sample?  Samples can be selected in a variety of ways. Here are three ways particularly appropriate for assessing learning goals:

Simple random samples – in which every student has an equal chance of being selected – are a straightforward way to ensure that your sample is representative 160

Keeping Assessment Cost-Effective

Simple random samples aren’t always practical. If you want to administer an in-class survey, for example, it wouldn’t be feasible to choose a random sample of the entire student body, go to every class, and ask just the sampled students in each class to complete the survey while the rest of the students sit idle. If a simple random sample is not realistic, other kinds of samples are possible.

CHAPTER 12

of all your students. A simple random sample might be drawn by writing every student’s name on a separate slip of paper, putting all the slips in a bag, shaking the bag, and drawing out as many names as you need. This can be done electronically by using software to generate a random sample. If such software isn’t available (check with your technical support staff), select students based on the last one or two digits of their student identification numbers, as these last digits are usually randomly distributed. For example, if you have 250 students and wish to examine writing samples from 50 (or 20 percent) of them, you could choose all students whose student identification numbers end in, say, 4 or 5, which would be 20 percent of all possible digits 0 through 9.

Cluster random samples involve taking a random sample of subgroups of students and then collecting information from everyone in those subgroups. You could choose a random sample of first-year writing classes, for example, and then assess essays written by everyone in those classes. Or you could choose a random sample of floors in the residence halls and interview everyone on those floors. Purposeful or judgment samples are carefully but not randomly chosen so that, in your judgment, they are representative of the students you are assessing. Suppose that you want to assess essays written by students in first-year writing classes and you would like to select a random sample of classes. Unfortunately, you know that while some faculty will gladly cooperate, others will decline to participate. You can still obtain a sample of essays by choosing, from those classes with cooperating faculty, a sample of classes that meet on various days and at various times and thus appear, in your judgment, to represent a good cross-section of all first-year writing classes. Or suppose that you want to assess student learning in general education science courses. While students may meet this requirement by taking any of 17 courses, 80 percent take one of just two courses: Introductory biology and introductory geology. (This is the 80-20 rule discussed earlier in this chapter.) Collecting assessment information from just these two courses will be far simpler and more cost-effective than collecting information from all 17 courses and will give you useful information on 80 percent of your students. If you must use a small or nonrandom sample or if you have a low participation rate, it’s especially important to collect information showing that your sample is representative of all relevant students. This is discussed in Chapter 24. 161

Creating a Culture of Evidence and Betterment

Consider staggering assessments . . . meaningfully List 12.4 described four situations in which it may not be practical to assess the learning of every student. In some of these situations, an alternative to sampling may be to stagger assessments, perhaps by assessing one learning goal one year, another the second year, and so on. But a fixed multiyear cycle of staggered assessments often doesn’t make sense for several reasons: • If you find that a sizable proportion of your students are failing to achieve a particular learning goal, do you really want to wait two, three, or more years to find out if they’re doing better? Such an approach is frankly irresponsible. • A cycle asking faculty and staff to assess a learning goal only once every several years conveys that the assessment is extra work (“Oh, darn, this year we have to assess X on top of everything else we’re doing”) rather than a normal part of teaching. • Cycles longer than two years can lead to assessment being conducted in fits and starts, with no one remembering what was done before and wheels therefore continually reinvented. • If you are using embedded capstone assessments, as suggested earlier in this chapter, you are assessing several learning goals at once. There’s no need to assess one goal one year, and another the next. What makes more sense is using evidence of student learning to continually refine a fluid schedule. If your students show unsatisfactory achievement of a particular learning goal, repeat that assessment frequently to make sure your intended changes are having the desired effect. But if your students show fine achievement of a particular learning goal, move that assessment to a back burner, repeating it perhaps every two or three years to make sure student achievement isn’t slipping.

Keep the grading burden manageable One of the reasons that traditional multiple-choice tests continue to be popular is that they can be scored very quickly. Finding time to assess other forms of student work can be a challenge, especially for faculty teaching courses with high enrollments. Here are a few suggestions: Use rubrics  (Chapter 15). If they’re well-crafted, they really speed up the process of assessing student papers and projects. Don’t waste your time scoring assignments with obviously inadequate effort.  Establish gateway criteria (Walvoord & Anderson, 2010) and give them to students, in writing, with the assignment. Then fail assignments that don’t meet those minimum standards, or return them for immediate revision. 162

Minimize paperwork Regular reports on how student learning is being assessed, the resulting evidence of student learning, and uses of that evidence are important for a variety of reasons (Chapters 10 and 25). But edicts for elaborate, lengthy, rigidly formatted, and seemingly pointless plans and reports not only suck up time but can alienate faculty and staff and stifle creativity and flexibility. If faculty and staff must spend considerable time preparing lengthy plans, they will loathe facing the seemingly monumental task of updating or amending them. Do all you can to keep reports to a bare bones minimum, and streamline and simplify the process of preparing them. Aim for plans and reports that are simple, supportive, flexible, sensitive to workload – and also meaningful. Ask initially for too little information rather than too much; you can always ask for additional information in subsequent reports. See Chapters 10 and 25 for more information on assessment reports.

Keeping Assessment Cost-Effective

Stagger assignment due dates  so each class section turns in its assignments in a few days or weeks apart and you’re not overwhelmed with papers or projects at any one point in the term.

CHAPTER 12

Use minimal marking  (Haswell, 1983). Instead of correcting student writing errors, simply circle or highlight the error and have the student figure out what’s wrong and what the correction should be.

Time to think, discuss, and practice 1. Brainstorm a list of information on student learning that your department or program might already have on hand, either in its own files or from another college office. 2. Of 300 Jameson College alumni responding to a survey, 74 percent said they were satisfied with the quality of their Jameson education. Calculate the error margin of this result and write a sentence explaining the result and the error margin. 3. The Business program at Calvert College requires students to compile a portfolio of their work. The program’s 10 faculty members would like to assess student learning by examining a sample of portfolios from its 200 graduating students. It takes about 30 minutes to review each portfolio. How many portfolios would you recommend that the faculty examine? Why? 4. Ask those in your group who teach to share how they now try to save time grading.

163

Some valuable ideas you’ll find in this chapter Learning goals, curricula, and assessments should be designed collaboratively, with participants drawn from all relevant sectors of the college community.

CHAPTER 13

Collaborating on Assessment

Academic freedom doesn’t relieve faculty of their responsibility to ensure that all students have sufficient opportunity to achieve those learning goals that faculty collectively agree are essential. Consensus does not mean unanimity.

165

Creating a Culture of Evidence and Betterment

A

s noted in Chapter 3, effective assessment practices involve the right people. Even if you are assessing only in a single class or co-curricular experience, collaboration with others is important for several reasons. • Collaboration helps ensure that students have purposeful, coherent, and integrated educational experiences (Chapter 5). Such experiences happen only when a community has developed a common understanding of key learning goals and a shared curriculum purposefully structured to help students achieve those goals. • Collaboration brings a variety of perspectives to the table. Asking people with varying backgrounds and viewpoints to review draft or potential assessment tools leads to better, fairer assessments. And crafting effective learning goals and assessment strategies requires two different outlooks: The ability to see the big, eventual destination of the learning experience and the ability to lay out specific, detailed steps to get there. If your strength is one of these, working with someone with the opposite perspective is helpful. • Collaboration contributes to a sense of ownership of the assessment process, and people with a sense of ownership are more likely to implement assessment strategies successfully and use the resulting evidence meaningfully. Anyone excluded from assessment planning may be quick to judge an assessment strategy as poor quality and inappropriate, simply because they had no role in its development. The ensuing debate can make it impossible to use the resulting evidence to improve student learning experiences. • Collaboration can improve all phases of the assessment process. Collaboration can strengthen decisions about learning goals (Chapter 4), assessment plans and implementation (Chapters 7 and 8), assessment information management systems (Chapter 11), and using evidence of student learning to inform decisions (Chapter 26). Collaborators should include anyone who has a stake in what is being assessed, how it is being assessed, and/or uses of the resulting evidence. Valuable collaborators may include faculty, co-curricular staff, students, institutional researchers, employers, field experience supervisors, and alumni. If students pursue further education, faculty at those colleges can be helpful collaborators.

Involving faculty Faculty share collective responsibility for ensuring that students leave courses, programs, and general education curricula consistently prepared for whatever comes next: Further study, a career, or other pursuits. Just as no one individual faculty member should be responsible for the achievement of a key program, 166

Collaborating on Assessment

Cast an invitation net widely.  If you are discussing course learning goals, invite faculty teaching courses taken before or after your course to discuss key learning goals, how they are assessed, and decisions based on the resulting evidence. If you are discussing program learning goals, invite faculty teaching any required courses offered by other departments. If you are assessing general education learning goals, invite all faculty teaching the general education courses addressing the learning goals under discussion – both full-time and adjunct. Throughout this book are many suggestions for questions and topics for collaborative discussion, such as List 4.5 in Chapter 4 and List 15.2 in Chapter 15.

CHAPTER 13

general education, or institutional learning goal (Chapter 5), no one individual faculty member should be responsible for assessing those goals. In these settings, assessment is a shared responsibility, not the responsibility of the faculty member who happens to be teaching the course in which the learning goal is being assessed.

Give teaching faculty a leadership role.  Chapter 9 notes that, just as faculty have a leadership role in curriculum design and student learning, so should they lead assessment of student learning. The greater their role, the more ownership they have of the assessment process and resulting evidence of student learning, and the more likely that they will take the evidence seriously and use it to identify and implement advancements in quality and effectiveness. The usual mechanism for giving faculty a leadership role is through committees charged with assessment leadership that are composed largely or wholly of teaching faculty, chaired by a teaching faculty member, and part of the faculty governance structure, as discussed in Chapter 9. Involve adjunct faculty.  Today many college courses are taught by part-time adjunct faculty. How can assessments be planned and implemented when adjuncts – whose time with and responsibilities to the college are limited – do much of the teaching? While answers depend on college culture, resources, adjunct contracts, and other factors, List 13.1 offers some suggestions. What about academic freedom?  While some might argue that establishing common learning goals or assessment strategies flies in the face of academic freedom (see Jargon Alert), there is no basis for such claims. The “Statement of Principles on Academic Freedom and Tenure” by the American Association of University Professors (AAUP) includes just one line on teaching: “Teachers are entitled to freedom in the classroom in discussing their subject” (1940, p. 1). The AAUP’s statement does not give faculty autonomy in choosing their subject or deciding how to assess student learning. The AAUP has recently clarified its position on assessment and academic freedom. 167

Creating a Culture of Evidence and Betterment

List 13.1  Strategies to Involve Part-Time Adjunct Faculty in Assessment ■■

■■

■■

■■

■■

Provide a core set of course materials for adjuncts, such as a core syllabus, core learning goals, and core assignments, rubrics, and test questions upon which they can build. Require participation in assessment activities within adjuncts’ contractual responsibilities. Adjuncts might be required to include certain questions in their tests, to use a common end-of-course assignment, to use a department rubric to assess student papers, or to participate in workshops or meetings related to assessment. Invite adjuncts to professional development opportunities, including those on teaching, learning, and assessment. Schedule these events at times when adjuncts are most likely able to attend. Assign full-time faculty as coordinators (with a stipend, reduced teaching load, or other compensation) for courses and programs that are especially adjunct-heavy. Assign full-time faculty as mentors (again with some form of compensation) to adjunct faculty in their departments.

• “Assessment of student learning and Jargon Alert!  reform of teaching and academic programs are core academic activities. Academic Freedom As such, the AAUP sees them as being Academic freedom (AAUP, 1940) is the right to engage in research, scholarship, inquiry, the primary responsibility of faculty  – and expression without fear of repercussion. individually and collectively” (Gold, Rhoades, Smith, & Kuh, 2011, p. 7). • “There is no reason that a faculty cannot collectively take on the task of identifying student learning outcomes, conducting those assessments, and revising curriculum accordingly” (p. 7). • “. . . the AAUP emphasizes the collective responsibility of the faculty as a whole for academic programs, suggesting that an academic department, for instance, can adopt pedagogical or curricular standards that colleagues teaching the course(s) need to adopt” (p. 7). This does not mean, of course, that all faculty must teach identical curricula and use identical pedagogies or identical assessments. “Excellence in student learning can be demanded without telling a faculty member how to achieve or how to evaluate it. Such a demand is not a violation of academic freedom” (Bresciani, 2006, p. 133). Once the faculty agree on key shared learning goals, individual faculty members can choose additional goals for their own classes and decide how best to help those students achieve all learning goals, both shared and unique.

Involving co-curricular staff As discussed in Chapter 5, co-curricular experiences are best integrated with other learning experiences, both in and out of the classroom, so students see connections 168

Institutional researchers are the repositories and analysts of facts and figures about a college and its students. They are often responsible for conducting surveys of students and other stakeholders such as faculty and alumni. They can thus help you connect evidence of student learning with other relevant data on student learning and student success, such as grades, retention and graduation rates, enrollment patterns, and student demographics (Chapter 24). Because they can be so helpful, Chapter 9 suggests including institutional researchers on assessment committees.

Collaborating on Assessment

Involving institutional researchers

CHAPTER 13

in their learning. This requires collaboration between faculty and co-curricular staff. Learning experiences “need to be integrated into a coherent whole for each and every student. And this integration needs to occur as a result of . . . closer collaboration between faculty and student affairs professionals – including academic advisors, career counselors, and other campus educators who work every day to help students make sense of their educational experiences” (Humphreys, 2013, para. 13). A good way to help foster collaboration is to include a representative of the co-curricular staff on committees charged with coordinating and supporting student learning assessment, as discussed in Chapter 9.

Involving students Including students in assessment efforts has two major benefits: It engages them more actively in their own learning, and it helps faculty and staff plan, implement, and use assessments more effectively. Students add a fresh and vital perspective. As the primary consumers of the teaching process, students keep us focused on fundamental questions such as why we do things the way we do. List 13.2 suggests ways to involve students.

Involving students’ peers Students can assess their peers’ presentations, drafts, groupwork contributions, and other assignments by completing rubrics or structured observation guides (Chapter 15). Exhibit 21.1 in Chapter 21 is an example of a rating scale for assessing peer group members. Peer assessments can be an important part of both learning and assessment. • Peer assessments help the students who are being assessed by adding the insight of people (fellow students) with very different perspectives from their professor. • Peer assessments of contributions to collaborative learning activities can motivate the assessed students to participate fully and effectively in those activities, especially if the assessments count in some way toward final grades. 169

Creating a Culture of Evidence and Betterment

List 13.2  Examples of How Students Can Engage in Assessment ■■

■■

■■

■■

■■

■■ ■■ ■■ ■■

■■ ■■

■■ ■■

Identify the purposes of assessment in which they’re participating, thereby helping to ensure that assessments are relevant and worthwhile to students. Articulate learning goals (Chapter 4) for a learning experience, helping to ensure that the learning goals are important and valuable to students. Discuss the characteristics of excellent work and the performance level to which they should strive, perhaps by helping to draft a rubric (Chapter 15) or offering feedback on a draft of one. Discuss appropriate assessment strategies for specific learning goals, perhaps by helping to draft an assignment (Chapter 16) or survey. Discuss how they should spend their study and learning time and where they should focus their efforts as they work on an assignment. Research and select a published survey (Chapter 19). Identify incentives for students to participate meaningfully in add-on assessments (Chapter 21). Choose what goes into a portfolio (Chapter 18). Review logistical plans for assessments to make sure they are feasible and not unduly burdensome to students (Chapters 7 and 8). Review and offer constructive feedback on the work of their peers. Review feedback on their own learning, such as completed rubrics and test item scores, and use that feedback to plan how to learn more effectively in the future. Review aggregated evidence of student learning and discuss its implications. Advise on how to share aggregated evidence of student achievement of program, general education, co-curricular, or institutional learning goals with students (Chapter 25).

• Peer assessments help the students who are doing the assessments, because the act of assessing peer work can strengthen critical thinking skills and help students understand faculty expectations. • Peer assessments help assess learning goals that faculty and staff may not be able to observe directly, such as leadership or collaboration skills, as discussed in Chapter 21. A significant shortcoming of peer assessments is that they can be subject to biases and errors (Chapter 3). Students may downrate a fellow student with good skills and traits simply because they don’t like him, for example. Because of the potential for inaccuracies, use peer assessments cautiously. If you are counting peer assessments toward course grades, weight them minimally, perhaps no more than five percent.

Involving employers Current and prospective employers of graduates can provide valuable information about their employee needs and how well a program or college prepares students for careers in their businesses and agencies. Employers can review draft rubrics to confirm that learning activities and rubrics address the skills they need and that 170

Many academic programs include a field experience such as an internship, practicum, clinical experience, co-curricular service learning experience, or student teaching assignment. If this experience gives students opportunities to practice applying the knowledge and skills they’ve learned in their academic program to real-life situations, a carefully constructed assessment form completed by field experience supervisors can provide direct and compelling evidence of how well students have achieved key program learning goals. To be useful, supervisors’ assessment forms should ask explicitly about student achievement of key program learning goals. The utility of rating scales may be limited because supervisors may not understand or share your standards for student performance and, without clear guidance, they tend to overrate students. A better choice is an analytic rubric (Chapter 15) that makes clear to supervisors what you consider to be exemplary and adequate performance. Consider asking some field experience supervisors to help design, review, or try out the rubric. Supervisors can also be asked for reflective comments on student performance (Chapter 20). Asking students to write reflectively on their field experiences and what they learned from them can yield additional, insightful evidence of student learning.

Collaborating on Assessment

Involving field experience supervisors

CHAPTER 13

your expectations are of appropriate rigor. Consider convening an advisory panel of local employers to offer this feedback. If you decide to survey employers of your graduates (Chapter 20), keep in mind that, because of liability and privacy issues, they will generally refuse to give information about individual employees. Ask them instead for general perceptions of your program or college and its graduates (“How well does Jamesville College prepare students in terms of leadership skills?”).

Involving alumni Alumni can contribute indirect but nonetheless useful evidence of student learning by describing their post-graduation experiences and their views of their education at your college. They can share, for example, how well they think their academic program prepared them for their current position or enriched their life. It’s not always practical or meaningful to contact alumni, however. It can be extraordinarily difficult to get a representative sample of alumni to participate in a survey or any other assessment. It can be equally difficult to separate the impact of a course, program, or college from the impact of postgraduate experiences. Alumni may develop a passion for lifelong learning not through your curriculum 171

Creating a Culture of Evidence and Betterment

but through later study or a mentor at work, for example. And by the time you assess alumni, especially if you want to see how they are living their lives a number of years after they graduate, you may be assessing obsolete learning goals or a curriculum no longer offered, a pointless task. But the aphorism “as the twig is bent, so grows the tree” can be helpful here. If we want alumni to be passionate, lifelong learners, it’s not unreasonable to expect students to start displaying relevant traits before they graduate – a point where assessment is more practical and useful. You can ask students who will soon graduate about their leisure reading choices, their internet surfing habits, the television shows they watch, and their plans for further education – all potential signs of their propensity to be passionate, lifelong learners. Student reflection and behaviors (Chapter 21) can thus serve as proxy measures that give some worthwhile clues on how those students may think and feel later in their lives. No, these strategies aren’t ideal or definitive, but they still give useful insight.

What about former students who leave before graduating? Students who leave before graduating are a tempting source of assessment information, especially on factors affecting student persistence. But the point of departure or afterward can be the worst time to try to collect information from them. Departing students who are disenfranchised from their college by the time they decide to leave will say they’re leaving because of financial difficulties or personal reasons – pat, socially acceptable answers that, if true, are often only symptoms of underlying issues. Furthermore, students who have already left can be difficult to track down, resulting in a poor response rate. You’ll get far better information if you survey currently enrolled students. Hold the survey results for a term or year, find out who is still enrolled and who has left, and compare the two groups for differences in responses.

Reaching consensus

172

What if faculty, staff, and others try to collaborate but simply cannot come to agreement? What if, for example, they can’t agree on key learning goals or they can’t agree on curricular modifications to improve student learning? There’s no law that says 100 percent unanimity is needed in order to move forward, and there are several techniques to help gauge consensus, make decisions, and proceed: • Vote, let the majority rule, and move on. • Break participants into small groups and ask each group to identify perhaps three goals or ideas that everyone in their group agrees are important. The groups then share their goals or ideas and identify common goals or ideas across groups.

1. Create a list of potential learning goals for a course or program. 2. Distribute the list to participating faculty and staff, and ask each to check off those goals that he or she thinks should be one of the key goals of the course or program. Sometimes it is agreed that everyone will vote for no more than, say, six or eight goals. 3. Collect the lists, tally the checkmarks, and share the results with faculty and staff. 4. Strike those goals with no votes. (The group may also agree to strike those goals with just one or two votes.) 5. Sometimes a few goals now clearly emerge as the top vote-getters, and the group agrees to focus on them, ending the process. 6. If consensus cannot be reached after the first round, redistribute the (possibly abbreviated) list with the initial results noted, and ask faculty and staff to vote again. Sometimes it is again agreed that everyone will vote for no more than, say, six or eight goals. 7. Again collect, tally, and share the lists. Human nature is such that few people persist in voting for a goal supported by hardly any others, so consensus on a few manageable goals is usually reached by this point. If not, the cycle is repeated until consensus is achieved.

Collaborating on Assessment

List 13.3  The Delphi Method for Achieving Consensus on Key Learning Goals

CHAPTER 13

• Use the Delphi method (Hsu & Sandford, 2007). The example in List 13.3 is for achieving consensus on key learning goals, but the process can be applied anytime a few ideas must be prioritized out of many.

Valuing collaboration If your college’s culture is one of silos more than one of collaboration, you may need some incentives to motivate faculty and staff to work collaboratively. In performance evaluations, consider valuing collaborative work. When making decisions about uses of professional development funds and travel funds, consider giving priority to collaborative projects.

Time to think, discuss, and practice 1. Ask a member of your group to play the role of the assessment director at Mount Harmon University, which has a strong culture of working in silos rather than collaboratively. Interview the director about the reasons for this culture, and brainstorm some feasible strategies to move toward a culture of collaboration. 2. The Communications program at Church Hill College is creating a business advisory group, which will meet twice a year over dinner and will be available for brief email contacts the rest of the year. Brainstorm some practical ways that the group might help the program faculty with student learning assessment. 173

Some valuable ideas you’ll find in this chapter

CHAPTER 14

Valuing Assessment and the People Who Contribute There’s no magic formula for creating a culture in which everyone habitually engages in assessment and uses the resulting evidence to inform decisions. What works at your college depends on its culture, history, and values. Pervasive, ongoing assessment happens only when college leaders are actively committed to it. Offer curriculum development grants to help faculty and staff find time to address disappointing evidence of student learning, thereby countering the fear that such evidence will be used punitively. Don’t expect to get everyone on board.

175

Creating a Culture of Evidence and Betterment

P

art 3 of this book has offered many strategies for getting colleagues on board with assessment, and an underlying theme throughout is valuing and respecting them. When we value and respect people: • We take the time to given them clear guidance and coordination (Chapter 9). • We invest in helping them learn what to do (Chapter 10). • We support their work with appropriate resources (Chapter 11). • We help them find time to work on assessment and keep assessment efforts costeffective (Chapter 12), because we respect how busy they are. • We help them work collaboratively rather than in silos (Chapter 13), because many hands make light work. This chapter offers additional strategies to value and respect assessment and the people who work on it. But we’ll begin by talking about why we need to spend so much time and effort getting people on board in the first place.

Why is this so hard? The movement to assess student learning in higher education has been around for roughly three decades. Why, then, is there still foot-dragging on assessment at many colleges? The root causes of foot-dragging are varied because college cultures, histories, values, and personalities differ. Consider these hypothetical – but realistic – examples: • Everyone at Rehoboth College feels so overworked that they resent the added burden of assessment. • At Northern University, research productivity is valued far more than good teaching, so few people are interested in assessing – or improving – student learning. • Manheim College has a long history of discord and disrespect between the faculty and administration, so there is a high level of suspicion that unsatisfactory evidence of student learning will be acted upon punitively, with promotions denied and programs terminated. • Mount Joy College is highly selective, largely because of its long-standing elite reputation, and there’s an (unspoken) worry that evidence of student learning may show that the college isn’t quite as special as it purports to be. • New Castle University’s assessment committee established university-wide learning goals without input from or approval of the faculty . . . and then wonders why the faculty aren’t helping to assess those goals. Because each of these colleges has a different root cause of assessment footdragging, each needs different strategies to address that foot-dragging. There’s thus no magic formula for getting everyone on board at every college. You’ll need to identify the root causes of foot-dragging at your college or in your program in order 176

Root Cause

Strategies to Address That Root Cause

Time: People don’t have time to work on assessment or on addressing any problems that assessment uncovers.

Stop or scale back something else, and help people find time to talk (Chapter 12).

Pointlessness: Assessment is a useless paperpushing exercise; the work isn’t helpful and reports vanish into a black hole.

Use evidence of student learning to inform important decisions on important goals (Chapters 6 and 26), and have flexible expectations for assessment (Chapter 9).

Understanding: People don’t understand what they are to do. Some have misunderstandings about assessment. Some don’t know how to change curricula and teaching methods in ways that address disappointing evidence of student learning.

Help everyone learn what they are to do (Chapter 10) and provide guidance and coordination (Chapter 9). Help faculty and co-curricular staff learn how to design effective curricula (Chapter 5) and use research-informed teaching methods (Chapter 26).

Valuing Assessment Contributors

Table 14.1:  Common Root Causes of Foot-Dragging on Assessment

CHAPTER 14

to choose the strategies that will be most helpful. Table 14.1 lists some common root causes and strategies to address them. Note that the theme of value and respect pervades the strategies, and the strategies convey value and respect in tangible ways, through actions and resources, not just through lip service.

Lack of support: People don’t have the resources Provide guidance, coordination, and technical and to do assessment well. other support (Chapters 9 and 11). Past experiences: People have been burned by past experiences with assessment; assessment rules are always changing; people have assessment fatigue.

Respect that history and address it forthrightly (discussed in this chapter).

Fear: People fear that evidence of unsatisfactory student learning will be used punitively, perhaps to close a program, lay off faculty or staff, or deny someone a promotion or tenure.

Provide planning and budget support to address disappointing evidence of student learning; establish policies on appropriate and inappropriate uses of evidence (all discussed in this chapter).

Reluctance to change: Change is hard! Some Value innovation (discussed in this chapter). people don’t want to change. And the tradition of scholarly research calls for researchers to conclude their work only with calls for further research or for others to make changes, not for researchers to act on findings themselves. So it’s not surprising that many assessment reports conclude with recommendations for modifications to assessment tools or methods rather than to teaching methods or curriculum design (Blaich & Wise, 2011). (continued)

177

Creating a Culture of Evidence and Betterment

Table 14.1 (Continued) Root Cause

Strategies to Address That Root Cause

Not valuing great teaching: As long as faculty get decent student evaluations, there’s no reason to change their teaching . . . and therefore no reason to assess. Sometimes teaching only gets lip service in promotion and tenure decisions; research is valued more.

Value great teaching and efforts to improve one’s teaching (discussed in this chapter).

Not valuing assessment efforts: Other than accreditation, there’s no real reward for doing assessment well.

Use evidence of student learning to inform important decisions on important goals (Chapters 6 and 26); respect and honor assessment efforts in other ways (discussed in this chapter).

A culture of isolation and silos

Value collaborating as a community (Chapter 13).

Pockets of mediocrity: A faculty member who, for example, teaches with entertaining but lightweight lectures and only tests students’ content knowledge might feel threatened by the prospect of teaching and assessing their critical thinking skills.

Help faculty and co-curricular staff learn how to apply research-informed teaching methods (Chapter 26), and recognize them when they do so (discussed in this chapter).

Make assessment useful and used We all like to know that our work is making an important difference. When assessment is used in meaningful, supportive ways, faculty and staff will be more inclined to participate. So, as discussed in Chapter 9, Part 3 of this book has focused on strategies not for building a culture of assessment per se but a culture of evidence and betterment: One in which evidence routinely informs decisions large and small, and in which evidence-informed betterment is valued. Chapters 6 and 26 discuss the many ways that evidence of student learning can be used to inform decisions.

Value great teaching At its heart, assessment is fundamentally a tool to bring about better learning (Chapter 6). If your college truly values great teaching and learning, and if you focus conversations not on assessment but on helping students learn, you will engage those faculty and staff who are enthusiastic teachers, and their enthusiasm may bring others on board. But truly valuing great teaching in the United States can be hard because many higher education resources are focused more on research than teaching. “For every 178

CHAPTER 14 Valuing Assessment Contributors

$100 the [U.S.] federal government spends on university-led research, it spends twenty-four cents on teaching innovation at universities” (Brown & Megan, 2015, para. 37). In other words, the ratio of U.S. federal investment in higher education research versus teaching is over 400 to 1. “Federal policy incentivizes research first, second, and third – and student instruction last . . . If one took its cues entirely on the federal government, the conclusion would be that colleges exist to conduct research and publish papers with student instruction as an afterthought” (para. 37). Valuing great teaching – and thereby assessment – thus requires deliberate, purposeful strategies to counter these external forces by cultivating a culture of teaching and learning as well as one of evidence and betterment. Focus conversations not on assessment but on topics such as what faculty and staff most want students to learn, curriculum design, and teaching methods. Then introduce the concept of assessment as a tool for facilitating these conversations (“How can we tell whether our curriculum is working?”). As suggested in Chapter 9, if your college has a teaching-learning center to help faculty and staff improve their teaching, consider affiliating the assessment coordinator with that center to convey visibly that assessment is simply a means to improve teaching and learning. Encourage the scholarship of teaching as a Jargon Alert! form of scholarship.  Scholarship, often considered strongly in decisions on faculty tenure, Scholarship of Teaching promotion, and merit pay, is traditionally The scholarship of teaching is the systematic development, through careful research, of new and defined as what Ernest Boyer (1997) called the better ways to help students learn. scholarship of discovery: Making an original, often research-based contribution to a discipline’s body of knowledge. Boyer proposed expanding this definition to encompass other forms of scholarship, including the scholarship of teaching (see Jargon Alert).

To be considered true scholarship, the scholarship of teaching requires faculty to document the effectiveness of their teaching through the assessment of student learning and to publish or present their findings in a peer-reviewed forum (Diamond, 2008). Valuing the scholarship of teaching as a form of scholarship can thus stimulate interest in assessment. Chapter 10 suggests journals and conferences for publishing or presenting scholarship on the assessment of student learning. Hire the right people.  Virtually every college is experiencing considerable faculty and staff turnover as those hired a generation or two ago retire. Faculty and staff vacancies are an excellent opportunity to move substantively toward a culture of great teaching and learning. Give hiring preference to faculty candidates with documented success in creating learning-centered environments for their students (Chapter 6) and in using assessment to strengthen teaching and learning. 179

Creating a Culture of Evidence and Betterment

Value innovation, especially in improving teaching If no one wants to improve or change anything, there’s no point in assessing. So it’s important to find ways to let faculty and staff know that their efforts at betterment and innovation are valued. Value, respect, and reward efforts to improve one’s teaching.  While most colleges say they prize teaching excellence, fewer tangibly value efforts to improve one’s teaching effectiveness. Faculty at some universities might put it this way: “I know I could be a better teacher. But my promotions are based on the research dollars I bring in and my publications, so that’s where I have to focus all my time. As long as my student evaluations are decent, there’s no incentive or reward for me to try to improve my teaching, and any time I spend on that is time taken away from my research, which is where the rewards are.”

Concrete, tangible incentives, recognition, and rewards can make a difference in bringing about teaching improvement (Kuh, Jankowski, Ikenberry, & Kinzie, 2014). List 14.1 suggests ways to value, respect, and reward efforts to understand and improve one’s teaching. List 14.1  Strategies to Value, Respect, and Reward Efforts to Improve Teaching ■■

■■

■■ ■■

■■

■■

180

Make the use of systematic evidence to change one’s teaching methods part of your college’s definition of teaching excellence, and give it strong consideration when making decisions about tenure, promotion, and merit pay. Include the advocacy and support of making evidence-informed changes to teaching in the performance evaluation criteria of deans and the provost. Support a teaching-learning center, as discussed in this chapter and Chapter 9. Offer a program of mini-grants that are available only to faculty who have assessed student learning and are disappointed with the resulting evidence. Faculty use the mini-grants – often known by euphemisms such as curriculum development grants – to research, plan, and implement changes that have been suggested by the evidence. Business and communication faculty at LaGuardia Community College, for example, used mini-grants to address business students’ underachievement in oral communication by incorporating new activities into introductory business courses (Provezis, 2012). What better way to counter the suspicion that unsatisfactory student learning evidence will be used punitively than to reward unsatisfactory evidence with grants to address the identified challenges? Faculty and staff are essentially paid for poor assessment results – if they act on them, of course. Give priority for sabbaticals and travel funds to faculty proposals to address disappointing evidence of student learning by investigating ways to change curricula and teaching methods. Design research doctorate programs, if you offer them, so that graduate students learn and practice research-informed teaching techniques that they can use when they receive faculty appointments.

CHAPTER 14 Valuing Assessment Contributors

Value, respect, and reward reasonable risk-taking.  Assessment is a process of trial-and-error experimentation, both in creating new assessment strategies and in using the resulting evidence to modify curricula, teaching methods, and support for the learning experience. Despite everyone’s best efforts, sometimes a new assessment strategy yields little useful information, and sometimes a change in curriculum or teaching methods is initially a failure. Rather than deny a faculty or staff member’s bid for promotion because of a lack of progress in student achievement or a drop in student evaluations of teaching, foster “a climate that rewards change and experimentation” (Smith, 2004, p. 33). Exhibit patience and wait for the kinks to be worked out. Some colleges, for example, do not evaluate faculty trying new teaching methods for the first semester or two, giving faculty time to refine what they’re doing.

Value and honor assessment efforts in tangible ways Why should a good and busy faculty or staff member pile assessment onto an already full plate of work? The best answers to this question have already been discussed in this chapter: • Use evidence of student learning to inform resource allocations and other important decisions (Chapter 26). • Offer curriculum development grants to help faculty and staff find time to rethink their curricula or learn about new teaching methods. • Value great teaching. • Value innovation, especially efforts to rethink and improve one’s teaching. It’s also helpful to recognize and honor those faculty and staff who are making strong, systematized assessment efforts by offering convincing, meaningful, and ongoing incentives and rewards, such as those in List 14.2 . Note that most of the strategies in List 14.2 have little direct cost, if any. List 14.2  Incentives and Rewards That Recognize and Honor Assessment Efforts ■■

■■

Move programs engaging in serious assessment efforts to the top of the funding list. Or drop to the bottom those without assessment activities under way. Or release annual operating budgets or portions thereof, such as travel funds, only to those programs that have submitted acceptable assessment reports. Allow faculty and staff who are fully engaged in assessment to submit less-frequent reports. If you ask for annual reports on assessment, for example, allow programs clearly engaged in ongoing, systematized assessment to submit biennial reports. (continued)

181

Creating a Culture of Evidence and Betterment

List 14.2  Continued ■■

■■

■■

■■

■■ ■■

Give faculty and staff written recognition that might be placed in their personnel files or submitted with applications for tenure, promotion, or merit pay. Recognition might be a certificate or a personalized letter of commendation signed by the president, chief academic officer, department chair, assessment committee chair, assessment officer, or a combination of these people. Make the criteria for such recognition clear, fair, consistent, and shared widely. Encourage faculty and staff to share what they’re doing with their peers through journal articles, conference presentations, and the like. Sponsor a celebratory event. Depending on your college’s culture, you might offer a special luncheon with a motivational speaker, a wine and cheese party, a barbecue, or an assessment fair akin to a conference poster session, with exhibits on assessment activities throughout your college. Encourage college leaders to commend publicly, in spoken and written remarks, those engaged in assessment. Honor faculty and staff who obtain grants that will strengthen assessment. Offer compensation for extraordinary assessment efforts. Most assessment activities should be embedded in normal teaching activities and thus shouldn’t warrant extra compensation. But sometimes special funds can be used to encourage participation in extraordinary assessment activities. Budget supplements, stipends, mini-grants, released time, or extra compensation can encourage faculty and staff to help read and assess large numbers of student papers or projects, spend other unusual amounts of time on assessment, or initiate significant assessment efforts on their own. “Next to disciplinary accreditation, funding from an external source may be the second most powerful incentive for turning faculty angst and even anger about assessment to acceptance, and even appreciation” (Banta, 2010, p. 3).

Respect everyone and what they have been doing Assessment is more likely to be successful when faculty and staff feel that they are respected as well as valued. Respect underlying values.  Every college community has collective values. Some are made explicit in a mission statement. One mission statement might promote specific religious values, for example, while another promotes civic responsibility. Other values are implicit. At some colleges, collaboration is a highly prized but unstated value, while at others individual autonomy is valued but again unstated. Clarifying your college’s underlying values and understanding how they might affect your assessment efforts can help foster success. At a college that values collaboration, involving a relatively large number of faculty and staff in the development of appropriate policies and procedures might be critical to assessment success. Research universities might find that faculty are especially interested in advancing – and assessing – the research skills of their students. 182

Valuing Assessment Contributors

Learn from past experiences, good or bad.  Past assessment experiences – good or bad – have a significant effect on people’s willingness to participate in gathering and using assessment information (Petrides & Nodine, 2005). Perhaps past assessment efforts consisted of a short burst of activity shortly before an accreditation review that died immediately thereafter, maybe because of a shifting focus to other priorities.

CHAPTER 14

Respect and build on what people have been doing well.  Some faculty and staff fear that assessment means that someone else will decide that much of what they have been doing their whole lives – what they teach, how they teach it, and how they assess it – is wrong. It’s thus important to emphasize the many things that faculty and staff do well. It’s especially important to emphasize the ways that many faculty and staff are already meeting at least some expectations for assessment. As Chapter 12 suggests, avoid reinventing the wheel by focusing initially on assessment information already in hand.

Talk with faculty and staff about those past experiences. What went well, what didn’t, and why? Which assessment activities are persisting today, which ones aren’t, and why? How can you build on past experiences and stop reinventing the wheel? Strategies to take stock of recent assessment efforts are discussed in Chapter 7. Today it’s also the rare college where everyone is at the same stage with assessment. Often programs with specialized accreditation are farther along than other programs. Talk to those faculty and staff for whom assessment is a natural part of life to see how their experiences might help those who are just getting started. It may also be helpful to think of another initiative that’s been launched successfully at your college, such as a first-year experience, a service learning initiative, or a reform of the general education curriculum. How was college-wide buy-in achieved with that initiative? You may be able to emulate or adapt what worked then. Focus on what people see as important.  Many people approach a new project with more enthusiasm if they find it inherently interesting, useful, and stimulating. So, as suggested in Chapter 12, help faculty and staff focus on assessment efforts that will answer questions in which they’re most interested and give them useful information for important decisions. Perhaps they are tired of their program’s image as mediocre and want to start aggressively countering this notion with solid evidence. Perhaps they are convinced that smaller classes would lead to more effective learning but need evidence to justify this to college leaders. Perhaps they are frustrated with their students’ inadequate writing skills and want to learn how best to improve those skills. 183

Creating a Culture of Evidence and Betterment

The critical role of college leaders While there are many reasons why some college communities are engaging in assessment and others are not, one factor predominates. If college leaders are truly and actively committed to assessment, assessment gets done and it gets done well. If college leaders – especially the chief academic officer (vice president for academic affairs, provost, or dean) and the chief executive officer (president or chancellor) – aren’t on board, there may be pockets of assessment efforts across the college but not the pervasive, enduring culture of evidence and betterment discussed in Chapter 9. Indeed, the vast majority of strategies offered in Chapters 9 through 13  and this chapter can be implemented only with the active and strong support of college leaders. List 14.3 lists some of the ways that engaged, committed leaders foster a culture of evidence and betterment. Most of these have already been discussed in this chapter. List 14.3  How College Leaders Can Foster a Culture of Assessment ■■

■■

■■

■■

184

Value college culture and history. • Promote a culture of respect for diversity in what people do, what they value, and their perspectives. • Focus resources and energies on the college’s fundamental mission: teaching and learning. Respect and empower faculty, staff, and students. • Recognize and reward those faculty and staff who collaborate to build a community that has a common understanding of what is important for students to learn and a curriculum with a purposeful structure. • Establish communication channels with those working on assessment, and use those channels to promote and facilitate assessment. • Charge faculty with assessment leadership (Chapter 13). • Encourage flexible rather than rigid approaches (Chapter 9). Value innovation and risk-taking, especially in improving teaching and learning. • Encourage and reward efforts to improve teaching and learning, including reasonable risk-taking. • Make support of innovation and teaching improvement a factor in evaluating deans and other appropriate administrators. • Encourage and reward faculty and staff who engage in the scholarship of teaching. • Bring the right people on board. Hire those with documented success in creating learning-centered environments and in using evidence of student learning to improve their teaching. Value assessment efforts. • Use evidence of student learning to inform important decisions on important goals, including resource allocations, and do so ethically and appropriately (Chapter 26). • Take the time to acquaint yourself with current thinking on the principles and practices of assessment.

• Recognize and honor assessment efforts in tangible ways.

Valuing Assessment Contributors

• Understand why assessment is important and have a personal commitment to it. • Inspire interest in assessment throughout the college through a clear vision of assessment that you share enthusiastically with others. • Establish a sense of urgency, making clear why assessment is a priority and why it must be taken seriously. • Talk up assessment in your formal remarks and informal conversations with faculty, students, and staff, explaining how assessment will benefit your college, individual programs, and individual faculty and staff members. • Provide incentives, resources, infrastructure, and support for assessment efforts (Chapters 9 and 11). Sponsor consultants, speakers, and forums on assessment and support these programs with your active presence. • Make support of assessment a factor in evaluating deans and other appropriate administrators.

CHAPTER 14

List 14.3  Continued

The role of governing boards.  Boards and accreditors have a lot in common (Suskie, 2014). Both are responsible for ensuring a college’s quality – including educational quality – and stewardship through oversight. Both generally rely on volunteers to provide this oversight. And both necessarily focus on the big picture rather than on specific details. Neither has time to pore through lengthy documents or sit through long presentations.

In order to fulfill their charge of ensuring educational quality, board members need, at least annually, “strategic-level information and evidence” that assures them “that assessment takes place and that results lead to action for improvement” (Association of Governing Boards, 2011, p. 6). They should see answers to the same basic questions as accreditors (Chapter 25) (Association of Governing Boards, 2011; Bok, 2006; Kuh, Jankowski, Ikenberry, & Kinzie, 2014). • What are the most important things our college wants students to learn? Why does our college think those things are important? • What evidence does our college have that students are learning those things? Is that evidence good enough and pervasive enough to draw meaningful conclusions about student learning? • Is our college satisfied with its evidence of student learning? Why or why not? • If not, what is our college doing about it? Chapter 25 offers suggestions on conveying information to board members and other audiences in ways that are quickly and easily understood.

185

Creating a Culture of Evidence and Betterment

Don’t expect to get everyone on board Cultural change is rarely rapid, and there are invariably pockets of stubborn resistance. It may be virtually impossible to engage some individuals in assessment. Accept them and don’t be discouraged by them. Plenty of initiatives are launched without unanimous faculty and staff buy-in, so you don’t need to get every single faculty and staff member on board for assessment to happen. Focus your energies on those faculty and staff who are skeptical but intrigued by assessment, as well as those who are enthused about assessment, and you’ll see good progress.

For More Information The National Institute for Learning Outcomes Assessment (NILOA) has published a number of reports on advancing cultures of evidence and betterment. Three important reports are Using Assessment Results: Promising Practices of Institutions That Do It Well (Baker, Jankowski, Provezis, & Kinzie, 2012), Valuing Assessment: Cost-Benefit Considerations (Swing & Coogan, 2010), and From Gathering to Using Assessment Results: Lessons from the Wabash National Study (Blaich & Wise, 2011).

Time to think, discuss, and practice 1. Share your experiences with assessment – as a faculty member, staff member, or student – with your group. a. Share both a positive and a negative experience, if you can. b. What did you learn from those experiences – good or bad – that you can use today? 2. Ask one member of your group to play the role of the academic vice president of a career college who wants faculty and staff to develop an assessment plan for the college’s general education curriculum. Interview the vice president and brainstorm three things that he or she might realistically do to help create a positive, supportive climate for this project. 3. Brainstorm three practical, realistic ways to celebrate and reward assessment efforts that would fit well with your college’s culture.

186

Part 4

The Assessment Toolbox

Some valuable ideas you’ll find in this chapter

CHAPTER 15

Designing Rubrics to Plan and Assess Assignments Rubrics can speed up the grading process. Use a rubric as a tool to plan your assignment: Write your rubric before you write the assignment that it will assess. Write rubrics using terms that students understand. It’s okay to include somewhat intangible traits in a rubric.

189

R

The Assessment Toolbox

ubrics – and the term rubric – pervade higher education (Kuh, Jankowski, Ikenberry, & Kinzie, 2014). A generation ago, few faculty were using rubrics to assess student learning. Today, many if not most faculty do (Zilvinskis, Nelson Laird, & Graham, 2016), and they often talk enthusiastically about them. It’s one of the great success stories of the assessment movement.

Jargon Alert! Rubric A rubric is a written guide for assessing student work. At a minimum, it lists the things you’re looking for when you assess student work.

Jargon Alert!

What is a rubric? “There is perhaps no word or phrase more confusing than the term ‘rubric’” (Hafner & Hafner, 2003, p. 1509) (see Jargon Alert). There are several formats for guides to assess student work but no consensus on which formats should be called rubrics (Suskie, 2017).

Checklists

Checklists (see Jargon Alert) can be used to verify whether students have followed A checklist is a list of traits that should be present a procedure, such as whether a dental in student work. It does not assess the frequency or student examining a patient has followed all quality of those traits. safety protocol steps. Students can also use checklists to self-assess their work before they turn it in (Have I proofread my paper? Does my bibliography use proper formatting conventions? Did I include at least eight references?). Exhibit 15.1 is an example of a checklist for safe culinary practices. Checklist

Exhibit 15.1  A Checklist for Safe Culinary Practices The student: ■■ Wears close-fitting rather than loose clothes. ■■ Wears gloves, hair restraints, and apron as appropriate. ■■ Washes hands before each shift and at appropriate points. ■■ Cleans and sanitizes food preparation and storage surfaces. ■■ Cooks food to safe temperatures. ■■ Cleans and sanitizes equipment and utensils, including knives, promptly and properly. ■■ Stores equipment and utensils, including knives, properly. ■■ Stores food in clean, labeled, and dated containers. ■■ Stores food at safe temperatures. ■■ Disposes of waste promptly and appropriately.

190

Rating scale rubrics

Jargon Alert!

Designing Rubrics

Rating scale rubrics (see Jargon Alert) are quick and easy to create and use, so they may have an A rating scale rubric is a checklist with a rating scale important place in many assessment programs, added to show the frequency or quality of each especially for relatively minor assignments and trait that should be present in student work. co-curricular experiences. They can also be a starting point for an eventual analytic rubric (Quinlan, 2012; Walvoord & Anderson, 2010), which is discussed next. Exhibit 21.1 in Chapter 21 and Exhibit 7.2 in Chapter 7 are examples of rating scale rubrics. The major advantage of rating scale rubrics is that they are fast and easy to create: Just list the things you’re looking for in student work, add a rating scale, and you’re done. This makes them a good choice for less significant assignments. The major shortcoming of rating scale rubrics is that performance levels are not clearly described. In Exhibit 21.1, for example, Often isn’t defined. The vague nature of rating scale rubrics can lead to several problems, all of which make them inappropriate for major, large-scale assessments that may lead to substantial decisions. • Faculty and staff may be inconsistent in how they rate performance. Because performance levels are not defined, one faculty member might rate a paper Outstanding in its articulation of information and ideas, while another faculty member might rate the same paper Very Good. • Students don’t receive detailed feedback. Yes, students can learn from a completed rating scale rubric that their paper’s organization was relatively weak and their grammar was relatively strong, but from the scored rating scale rubric alone they won’t learn exactly how their organization was weak or how it might be improved. • Rating scale rubrics can lack credibility with some audiences, because they can be susceptible to biases and errors (List 3.5 in Chapter 3). Rating Scale Rubric

CHAPTER 15

Jargon Alert!

Analytic rubrics

Analytic rubrics (see Jargon Alert) might be considered the gold standard of rubrics An analytic rubric replaces the checkboxes of a because they document standards for rating scale rubric with clear, brief definitions of student performance (what Chapter 22 calls each level of performance for each trait. local standards) explicitly. Students, faculty, accreditors, and other audiences are much more likely to understand exactly what an Outstanding or Very Good rating means than when a rating scale rubric is used, and faculty assessments of student work are much more consistent. Indeed, analytic rubrics are “the most reliable of all direct writing assessment procedures” (Huot, 1990, p. 238). Exhibit 18.2 in Chapter 18, Exhibit 23.1 in Chapter 23, and Exhibit 10.1 in Chapter 10 are examples of analytic rubrics. Analytic Rubric

191

The Assessment Toolbox

Coming up with succinct but explicit descriptions of every performance level for everything you’re looking for in completed assignments is not easy, however! It can require negotiation, tryouts, and revisions and can therefore be ­time-consuming. So don’t feel that you need to develop an analytic rubric for every assignment. Analytic rubrics are a good choice under the following circumstances: • You are undertaking important assessments whose resulting evidence may contribute to major decisions such as accreditation, funding, or program continuance. • Several faculty and staff are collectively assessing student work, because analytic rubrics’ clear descriptions make scores more consistent. • Students need clear, detailed feedback on their strengths and weaknesses. • Skeptical audiences will be examining the rubric results with a critical eye.

Jargon Alert!

In some massive assessment projects, that faculty and staff don’t need to complete A holistic scoring guide provides short narrative a rating scale or analytic rubric for every descriptions of the characteristics of work at each performance level, enabling faculty and staff to assignment that’s being assessed. Perhaps make an overall judgment about the quality of they must read and score 1,500 entering work (Allen, 2006; Brookhart, 2013). They yield students’ essays to decide who should enroll only global scores of overall student performance. in an introductory writing course. The major purpose of such placement assessments is not to give feedback to individual students or to identify strengths and weaknesses in student learning but to make decisions within a tight timeframe. In these situations, a holistic scoring guide (see Jargon Alert) may be a good choice. Examples of holistic scoring guides include those developed by the American Council for the Teaching of Foreign Languages (2012) for assessing reading, writing, speaking, and listening in a foreign language. Holistic scoring guides have two major shortcomings. First, few student works will meet any one description precisely. The works must be scored at the performance level for which there is a preponderance of evidence, and it can be difficult for faculty and staff to do this consistently (Selke, 2013). Second, holistic scoring guides do not yield feedback on students’ strengths and weaknesses and therefore cannot be used to identify and make changes to address weaknesses. Holistic scoring guides are therefore typically used for large-scale placement decisions (Jonsson & Svingby, 2007). Holistic scoring guides and analytic rubrics should address the same traits and the same performance levels; the difference between them is only in how the trait descriptions are organized and formatted. So you can create a holistic scoring guide by first creating an analytic rubric, then merging the descriptions of traits at each performance level into a single statement for that performance level. Alternately, you can first draft the holistic scoring guide and then convert it into an analytic rubric format to confirm that it addresses all traits across all performance levels (Selke, 2013). Holistic Scoring Guide

192

Holistic scoring guides

Designing Rubrics

Some faculty and staff find it difficult to come up with a rubric to assess student work. Some may be uncomfortable with the idea of quantifying their assessments and prefer a qualitative approach. Some may have never thought about what they expect Jargon Alert! and value in student work. Yes, these faculty Structured Observation Guide have been awarding grades (they “know an A structured observation guide is a rubric whose A when they see it”), but they have done so rating scale is replaced with space for comments. based more on gut instinct than anything else. These faculty may prefer a structured observation guide (see Jargon Alert): A subjective and qualitative (Chapter 20) – but nonetheless direct and valid – assessment of student learning. Exhibit 15.2 is a structured observation guide for a student production in a theater program.

CHAPTER 15

Structured observation guides

Exhibit 15.2  A Structured Observation Guide for a One-Act Play The effectiveness of each of the Appraisal/Feedback following in conveying the production’s meaning or theme

Evidence Supporting the Appraisal

Pace and rhythm

Characterizations

Stage presence and business

Stagecraft: Costume, lighting, set, and sound designs Creative vision and risk-taking

Sparkle and audience engagement

Total integrated production effect

193

The Assessment Toolbox

Structured observation guides can be helpful in articulating learning goals (Chapter 4), articulating criteria for rating scale or analytic rubrics, assessing fuzzy learning goals, and as tools for students to assess the work of their peers (Chapter 13). They may be a good choice in some co-curricular experiences. Students training to become residence hall assistants, for example, might be assessed through role plays in which one student plays the role of the assistant and others play the role of resident students. Peer students and staff observing the role plays could assess them with a structured observation guide.

So what is a rubric? As noted earlier, there is no consensus on which of the formats discussed here should be called a rubric. Opinions fall into three camps (Suskie, 2017). The first camp defines rubrics broadly and flexibly as guides for assessing student work; it would consider all the formats discussed here to be rubrics. The second camp defines rubrics as providing not just traits but also standards or levels of quality along a continuum; it would consider rating scale rubrics, analytic rubrics, and holistic scoring guides to be rubrics. The third camp defines rubrics narrowly as only those scoring guides that include traits, a continuum of performance levels, and descriptions of each trait at each performance level; it would consider only analytic rubrics and holistic scoring guides to be rubrics.

Why use a rubric? If you have never used a rubric to assess student assignments, you will find that using a well-crafted rubric makes your life easier and improves student learning in the ways shown in List 15.1. Rubrics are not a panacea, however. They’re great for assessing thinking and performance skills, but multiple-choice tests (Chapter 17) are a better choice for assessing knowledge and basic understanding, and reflective writing (Chapter 20) is a better choice for assessing many attitudes and values. Rubrics may be inappropriate with small numbers of students, because results may bounce up and down too much from one cohort to the next to be meaningful. And while assessing student work is far more consistent with a rubric than without one, it is still subjective, and you may not get it right the first time you use one.

194

■■

■■

■■

■■

■■

■■

■■

■■

■■

Designing Rubrics

■■

Rubrics can help clarify vague, fuzzy goals. A goal such as Demonstrate effective writing skills is vague – what are effective writing skills? – but difficult to clarify succinctly (Chapter 4). A rubric can provide this clarification. Rubrics can lead to better curriculum design and assignments through the careful, hard thinking that goes into their development (Walvoord & Anderson, 2010). Rubrics can help students understand your expectations. If you distribute your rubric with the assignment, students will understand better what you want them to do and where they should focus their energies. You’ll have fewer questions from students, and they may find the assignment a richer, more rewarding experience. Rubrics can inspire better student performance. Rubrics show students exactly what you value and what you’ll be looking for when you assess their assignments. Knowing what you expect will motivate some (not all!) to aim for the standards you’ve identified. Rubrics can help students self-improve. If you encourage students to use the rubric to selfassess their work before turning it in, in order to make sure the assignment is complete and up to acceptable standards, you are helping them develop the important lifelong skill of metacognition (Chapter 4): understanding how they learn by reflecting on how they learn. Rubrics can improve feedback to students. Marked rubrics give students a clear picture of their strengths and weaknesses – a more complete, holistic picture of their performance than comments alone. Rubrics make scoring more accurate, unbiased, and consistent (Jonsson & Svingby, 2007). They ensure that every assignment is assessed using the same criteria. Rubrics can make scoring easier and faster. While it may seem that using a rubric adds an extra burden to the grading process, rubrics actually make the grading process faster because they keep you focused on the big things you’re looking for in student work. You also won’t need to write as many comments on papers. Rubrics can reduce arguments with students. By making assessment criteria explicit, rubrics prevent arguments with students (like “Why did he get a B- when I got a C+?”). Conversations with students focus on how they can improve their performance rather than defending your assessment practices. Rubrics improve feedback to faculty and staff. If a number of students aren’t demonstrating adequate achievement of a particular skill, rubrics bring this to your attention. The consistency that rubrics add to the assessment process makes it possible to track changes in student performance as you refine your teaching. Rubrics can help determine, for example, whether introducing collaborative learning activities into classes helps improve students’ analysis skills.

CHAPTER 15

List 15.1  Benefits of Well-Crafted Rubrics

Creating effective analytic rubrics The rest of this chapter focuses on developing and using analytic rubrics, but most of the principles can be applied to other rubric formats. There are six fundamental steps in analytic rubric development (Suskie, 2017): 195

The Assessment Toolbox

1. Identify how and where the rubric will be used. 2. Articulate clear learning goals for the assignment being assessed, and explicate them into traits. 3. Identify and label performance levels for each trait. 4. Create descriptions of each trait at each performance level. 5. Develop or review the assignment. 6. Test and revise the rubric. If the work of creating an analytic rubric with your colleagues seems overwhelming, start by creating a rating scale rubric. Once faculty and staff use it, they may come to realize that they need a rubric with greater clarity and be ready to invest the time and effort to develop an analytic rubric.

Identify how and where the rubric will be used Evidence of student learning from rubrics can be used to inform two broad categories of decisions (Suskie, 2017): • Making summative determinations about learning, such as a student’s grade or placement or the overall efficacy of a course, program, or college in helping students achieve a particular learning goal. These decisions can be informed by rubrics of any format. • Improving learning through formative and diagnostic feedback to individual students, faculty, and staff on successes and areas for improvement in learning. These decisions are best informed by analytic rubrics, although rating scale rubrics, checklists, and structured observation guides can also be of value. Rubrics used to inform these decisions fall on a continuum from narrow to broad uses. Task-specific rubrics,  at the narrow end, are used to assess or grade one assignment, such as an essay exam question (Brookhart, 2013). They are so specific that they apply only to that one assignment. Because their specificity may give away the correct response, they cannot be shared with students in advance.

196

Primary trait scoring guides or primary trait analysis  are used to assess a family of tasks rather than one specific task (Lane, 2010; Lloyd-Jones, 1977). The premise of primary trait analysis is that the essential (primary) traits of a learning goal such as writing vary by type of assignment (Freedman, 1991). Primary trait scoring guides focus attention on only those traits of the learning goal that are relevant to the task

Designing Rubrics

General rubrics  are used with a broader variety of assignments (Brookhart, 2013). They list traits that are generic to a learning goal and are thus independent of topic, purpose, or audience (Freedman, 1991). Exhibit 23.1 in Chapter 23 is an example of a general rubric. While it was developed to assess research papers in speech language pathology/audiology, it could be used to assess research papers in many social science and health disciplines.

CHAPTER 15

at hand. Exhibit 18.2 in Chapter 18, for portfolios in a graduate course in student learning assessment, is an example of a primary trait scoring guide, as it could be used with a family of portfolio assignments on student learning assessment.

Developmental rubrics  or meta-rubrics are general rubrics whose performance levels cover a wide span of performance and are used to show growth or progress over time (Rhodes & Finley, 2013; Solomon, 2002). The 6+1 Trait Writing rubrics (Education Northwest, 2014) and VALUE rubrics (www.aacu.org/value-rubrics) are examples of developmental rubrics.

Rubrics toward the middle of this continuum may be more practical and useful than task-specific rubrics. Unlike task-specific rubrics, primary trait scoring guides and general rubrics do not need to be rewritten for every assignment (Brookhart, 2013; Lane, 2010), saving time. And students who see the same rubric for multiple assignments will focus on developing pervasive skills rather than just getting individual assignments done (Brookhart, 2013). Rubrics toward the middle of this continuum may also be more useful than broad developmental rubrics. The necessary lack of precision in developmental rubrics may make them difficult to interpret and act upon (Cooper, 1977). Developmental rubrics designed to assess student growth over several years are not appropriate to assess student growth in one course, and they are therefore not suitable for grading (Rhodes & Finley, 2013; Stevens & Levi, 2012).

Articulate clear learning goals and explicate them into traits Start creating a rubric by clearly articulating the learning goals that the assignment and rubric will assess (Chapter 4). Then explicate the learning goals into the traits that students will demonstrate in the completed assignment. (Here again there is no consensus on vocabulary; the traits listed in a rubric may be called criteria, dimensions, attributes, strands, or domains.) If the rubric is to assess critical thinking, for example, what are the key traits of effective critical thinking? Describe traits of the learning goals being assessed, not the assignment; this helps both faculty and students focus on learning skills rather than assignment completion (Brookhart, 2013). The questions in List 15.2 may be helpful. 197

The Assessment Toolbox

List 15.2  Questions to Help Identify What You’re Looking for in Student Work ■■ ■■ ■■ ■■

■■

Why are we giving students this assignment? What are its key learning goals? What do we want students to learn by completing it? What are the skills we want students to demonstrate in this assignment? What are the characteristics of good student work? What are the characteristics of good writing, a good presentation, a good lab report, or good student teaching? What specific characteristics do we want to see in completed assignments?

State each trait in explicit, concrete terms.  Writing quality tells students and colleagues little about what you’re looking for; organization and structure tells them far more. In analytic rubrics, traits are expressed as nouns or noun phrases (Walvoord  & Anderson, 2010) because the traits are further explicated with performance-level descriptions. For other forms of rubrics, express the traits as learning goals (Chapter 4). How many traits should be listed in a rubric?  Less is more. Rubrics can typically provide distinct information for only a small number of traits (Lane, 2010). Lengthy rubrics can make it hard for students to focus on the key traits of the assignment and are more time-consuming to score. What is perhaps more important than the number of traits, however, is to make sure that they are the right traits. List only traits that are essential or vital to the assignment. Include the intangibles and the unexpected.  Some faculty and staff have found that students who are given rubrics along with an assignment do exactly what the rubric tells them to do but no more, discouraging creativity and yielding solid but somewhat flat and uninspired products (Cooper & Gargan, 2009). To encourage originality, effort, and that unexpected but delightful something extra, simply build these qualities into the rubric. Omit extraneous traits  unrelated to the assignment’s intended learning goals (Messick, 1989 & 1994). For a rubric intended to assess oral presentation skills, for example, extraneous criteria might include whether the student attended presentations by peers. (Behaviors such as attendance and completing assignments on time can certainly be graded, but do so by assessing an appropriate learning goal such as professionalism, discussed in Chapter 4.)

Create the rating scale The next step in developing an analytic rubric is to identify and label its performance levels: The headings of the rubric’s columns. 198

Designing Rubrics

Beyond these three levels, you may want to add a level between exemplary and adequate, and you may wish to add an almost-adequate level between adequate and inadequate. Don’t use more than five levels, because faculty and staff have a hard time distinguishing consistently between ratings of, say, 6 and 7 on a 10-point scale.

CHAPTER 15

Create three to five performance levels.  Most rubrics have at least three levels indicating inadequate, adequate, and exemplary performance. A rubric with only two levels – adequate and inadequate – is a checklist, and adding the exemplary level motivates students to do better than merely adequate work. (Developmental rubrics have at least three levels indicating entry-level performance before education has begun; adequate performance at the midpoint of the course, program, or other learning experience; and adequate performance at the end.)

Label each level with names, not just numbers.  Don’t ask, for example, for a rating of 1 through 5, with 1 being best and 5 being worst. People will have different conceptions of 2, 3, and 4 unless you spell them out. There is no hard-and-fast rule on how to label each performance level, except to make clear which level represents minimally acceptable performance. Use descriptors that are clear and relevant to your students and colleagues. Examples of possible performance-level labels are:

• Exceeds Standard, Meets Standard, Approaching Standard, Below Standard • Excellent, Very Good, Adequate, Needs Attention • Letter grades (A, B, C, D, F) if you make clear which grade (C or D) represents minimally acceptable performance

Create descriptions of each trait at each performance level Rubric traits are typically listed down the left side of a rubric and performance levels across the top, creating a grid. The next step in developing an analytic rubric is to fill in the grid’s boxes. Specify observable behaviors.  Use descriptive, objective terms rather than subjective, judgmental, values-based terms such as Poor, Limited, Proficient, Advanced, or Above Average. The descriptions should be clear enough to give students concrete feedback on how to improve weaknesses in their performance (Andrade, 2000). The descriptions should also be clear enough that those who use the rubric simply match the student work to a description rather than make a subjective interpretation (Brookhart, 2013).

At the same time, however, rubrics require a certain amount of inference and professional judgment (Brookhart, 2013; Clauser, 2000). It’s impossible, for example, to have a completely objective description of poetic merit. Overly rigid, concrete descriptions can focus on the trivial rather than the essential – the trees instead of the forest – and force students to think narrowly rather than outside the box (Cooper & Gargan, 2009).

199

The Assessment Toolbox

Value-based terms are appropriate in a rubric if students have been taught what they mean (Walvoord & Anderson, 2010). If students have learned what adequate evidence is, for example, it’s fine to use the phrase adequate evidence in a rubric. Use brief, simple, clear, jargon-free terms that everyone understands,  including students, faculty, prospective employers, and administrators. Consider using language “written from the students’ point of view, using first person, in language the students can understand . . . expressed in the manner that students would think about their work” (Brookhart, 2013, p. 46). Articulate the minimally adequate level first.  The most important performance level, and thus the one that should be completed first, is the one that describes minimally acceptable performance – in other words, passing work, as discussed in Chapter 22. Use parallel language across performance levels.  All performance-level descriptions should address the same performance trait elements (Brookhart, 2013; Selke, 2013). For creativity, for example, the following performance-level descriptions would be inconsistent:

• • • •

Unacceptable: No creative component Developing: Limited creative thinking Competent: Thinks critically and synthesizes new approaches Exemplary: Significantly advances the state of the art

Clearly distinguish each performance level from the others.  Performance levels represent points along a continuum, possibly one of those in List 15.3 (Selke, 2013, p. 94). No matter the continuum you use, your descriptions at each performance level should represent points along the continuum that are clearly distinct from one another. You may find that some rubric traits may have only three distinct performance levels while others have five (Humphry & Heldsinger, 2014). If so, feel free to black out some boxes in the rubric grid.

List 15.3  Examples of Continuums for Rubric Performance Levels ■■ ■■ ■■ ■■ ■■ ■■

200

Number of components of the trait that have been mastered Number of sequential steps of the trait that have been taken Strength of the work on this trait Thoroughness of the work on this trait Complexity of the work on this trait How well the work on this trait meets established standards or requirements within this context

Trying out rubrics on actual student work leads to more useful rubrics with better performance descriptions (Morgaine, 2010). Use the rubric to assess some actual samples of student work, including some of your students’ best and worst work. Look at the rubric scores, and see how well they match what you think the works deserve (Quinlan, 2012). Then revise the rubric if necessary to improve its clarity and usefulness. Creating effective assignments is discussed in Chapter 16.

Designing Rubrics

Develop or review the assignment, and try out and revise the rubric

CHAPTER 15

Avoid counting things.  Of the continuums described in List 15.3, probably the least useful are those that count things (Brookhart, 2013). Usually what’s more important than the number of grammatical errors, for example, is whether they distract or impede meaning. What’s more important than the number of references cited is whether they’re the right references for the thesis under discussion.

Additional steps to consider Some additional steps, while not essential, can greatly improve the quality of a rubric. Two steps in addition to those discussed in this section are searching for and adapting assessment ideas, including rubric models, which is discussed in Chapter 12, and collaborating with others, which is discussed in Chapter 13.

Looking at samples of student work as you develop the rubric In addition to using samples of student work to try out a completed rubric, as suggested above, use some samples of student work – perhaps from the last set of student work you graded – to help identify traits for a rubric or structured observation guide and performance-level descriptions for an analytic rubric. Sort the samples into three groups representing what you consider to be high-, medium-, and low-quality work (Andrade, 2000; Cooper, 1977). Then review the samples, taking informal notes on questions such as these: • Why was some work considered high-quality? What are the defining traits of high-quality work? • Why was some work considered low-quality? What are the defining traits of lowquality work? • How do the high- and medium-quality samples differ? • How do the medium- and low-quality samples differ? • What kinds of student work represent minimally acceptable work? Why? 201

The Assessment Toolbox

Then review your notes for patterns or themes. You may observe that some traits are mentioned in your notes repeatedly. These represent your implicit expectations for your students and can become the rubric’s traits. They may also suggest performance-level descriptions for some analytic rubric traits. Do this exercise before you start looking for other rubrics that might be models, if you decide to do so. This will help ensure that your rubric will represent what you and your colleagues most value.

Developing families of rubrics Some colleges use a single general or developmental rubric to assess an institutional or general education learning goal across a host of courses and disciplines. Others insist that all rubrics must have uniformly named performance levels. The intent of these mandates is usually to enable faculty and staff to draw overall conclusions about student achievement of the learning goal and, sometimes, to make comparisons across learning goals and/or disciplines. But these strategies may be neither necessary nor valid nor useful. You don’t need broad general or developmental rubrics to satisfy accreditors or to get actionable information about college-wide student learning (Walvoord, 2014). For example, you don’t need a common rubric format to conclude that students in a variety of disciplines and settings are struggling to support assertions with evidence. And comparisons across disciplines and courses may not be valid because of the apples and oranges being compared; no one would expect history majors’ quantitative skills to stack up to those of engineering students, for example. And, as noted earlier in this chapter, broad general or developmental rubrics may be too imprecise to be truly useful. Instead, when assessing an institutional or general education learning goal, begin with a general or developmental rubric, but use it only as a framework to develop a family of more specific rubrics (Cooper, 1977; Lane, 2012; Lesmond, McCahan, & Beach, 2017; Messick, 1989) whose formats, traits, and descriptions vary in ways that make each rubric meaningful and useful in its setting. If you are assessing critical thinking, for example, start with a developmental rubric, then use it to build critical thinking rubrics for disciplines or families of disciplines such as the natural sciences, business, and the humanities. Then compare the resulting evidence qualitatively rather than quantitatively, looking for patterns in student achievement across learning goals and disciplines, as discussed in Chapter 24. Yes, this approach is a little messier than using just one rubric, but it’s much more meaningful.

202

Designing Rubrics

Technological tools for developing and using rubrics have grown in recent years. Several websites offer free templates and other simple software for creating and saving rubrics; one example is iRubric (www.rcampus.com/indexrubric.cfm). Assessment information management systems, which can facilitate the recording, analysis, and storage of student learning evidence from rubrics, are discussed in Chapter 11. Avoid systems that require everyone to use a rubric or to format a rubric in a particular way.

CHAPTER 15

Using technological tools

Using rubrics with advanced students Faculty teaching seniors or graduate students sometimes worry that rubrics provide too much guidance to their students, serving as a crutch when students should be figuring out expectations on their own. After all, once students graduate and are in the workplace, they will need to self-assess and self-correct their work without a professor’s rubric at hand. If this is a concern, consider making rubrics less explicit as students progress through their program. First-year students need a clear explanation of what you consider good organization of a paper; seniors and graduate students shouldn’t. Or have students collaborate to develop their own rubrics, subject of course to faculty review.

Time to think, discuss, and practice One of Belleville College’s general education learning goals is “Students will be able to write effectively.” The faculty has decided to assess this by asking all graduating students to write a one-page (400–500 word) review and analysis of arguments for and against making community service a college graduation requirement. 1. Follow the steps in this chapter to create an analytic rubric for this learning goal with a three-point scale: outstanding, acceptable but unexceptional, and unacceptable. 2. Exhibit 15.3 is a student submission for this assignment. Use your rubric to assess this paper. Compare your completed rubric with those of others on your team. Does the rubric appear to work the way you intended, or does it need refinement?

203

The Assessment Toolbox

204

Exhibit 15.3  A Student Essay on Making Community Service a Graduation Requirement Of all the requirements for graduation, community service is not usually one of them. However, some colleges are considering adding this as a prerequisite to receiving a diploma. This idea has caused disputes between some students, who do not wish to volunteer, and faculty, who feel that volunteering should not be required in order to graduate from an institute of higher learning. One opinion is that as a graduating college student, you should not only be well educated, but also well rounded in general, and community service is one aspect that will help you to become a more well-rounded person in general. This is the opinion of the people who advocate for community service. By requiring students to perform so many mandated hours of community service, they feel that the students will become enriched in ways that a classroom cannot provide. Another opinion of faculty is that students do not have to volunteer in order to get a good education, which is the primary function of a university, and therefore, required community service should not be necessary in order to receive a diploma. Some students share this opinion also. They feel that community service should be a personal opinion based on personal interests and reasons for wishing to volunteer. They believe that if students are forced to volunteer in order to receive the diploma they have worked so hard for, since the community service work is not coming from their hearts, they will not be giving their all, simply going through the motions to satisfy the requirement. If students are required to provide a certain number of community service hours, this may also detract from their attention to their school work, causing grades to suffer. Some faculty have taken this into consideration. They are not sure if creating mandatory community service hours is worth the possible decline in students’ GPA’s because they are so concerned with finding places to conduct community service and finding the time to perform their mandated hours. Another question that is concerning the faculty of universities is whether or not there are enough locations in which students could perform community service. For some colleges that are not located around a large city, the number of places that need volunteer work may not be sufficient enough to accommodate all the students that are attending the school. If there are not enough open spaces in volunteer organizations outside of the school, should the university be obligated to create situations in which volunteers are needed in the school so that students can perform their needed hours of community service? All of these questions and concerns need to be adequately addressed before a decision is made at any university or post-secondary school. They should be addressed not only with faculty and staff of the school, but also students, in order to hear their points of view.

Some valuable ideas you’ll find in this chapter Every assignment should help students achieve important learning goals.

CHAPTER 16

Creating Effective Assignments Aim for assignments that students find relevant and engaging. Ask yourself if students will learn significantly more from a 30-page assignment than a five-page assignment – enough to justify the additional time spent on it. Address plagiarism consistently and collaboratively throughout your college through education as well as ramifications.

205

The Assessment Toolbox

T

he world’s best critical thinking rubric is useless if it’s used to assess only students’ summaries of a concept. It’s also useless if the guidelines given to students are so vague that students don’t produce what faculty are hoping to see. Well-crafted assignments are thus an essential part of the assessment process. This chapter discusses the characteristics of a great assignment, how to construct assignment guidelines, and how to counter plagiarism.

What is a great assignment? Great assignments have traits that research has shown help college students learn (List 26.1 in Chapter 26). Great assignments consistently focus on important learning goals.  The assignment guidelines given to students and the assessment criteria for completed assignments both focus on the same important learning goals (Winkelmes, Bernacki, Butler, Zochowski, Golanics, & Weavil, 2016).

Jargon Alert! Performance Assessments and Traditional Assessments Performance assessments ask students to demonstrate their skills rather than describe or explain those skills through traditional tests. Performance assessments have two components: assignment guidelines that tell students what they are expected to do or produce and assessment criteria – usually a rubric (Chapter 15 – used to assess completed work. Performance assessments are sometimes called alternative assessments because they are alternatives to traditional tests. Traditional assessments are the kinds of tests that have been around for decades, if not centuries: multiple-choice and other objective tests, essay tests, and oral examinations. Students have historically completed traditional assessments in controlled, timed examination settings.

Great assignments are learning opportunities.  Exams – whether multiple-choice, short-answer, or essay – are opportunities to assess but not opportunities for students to learn. Yes, students learn when they study for an exam, but the time they spend taking the exam is not time spent learning. The time students spend completing great assignments is time spent in active learning through performance assessments (see Jargon Alert).

Great assignments are meaningful and worthwhile.  While quizzes, homework, and class discussions can give us insight on whether students are on track to achieve key learning goals, far more useful – both to us and to our students – are learning activities that require significant time and effort. Capstones (Chapter 5) are excellent examples. But make sure that the time students put into a major assignment yields an appropriate payoff in terms of their learning. Will students learn twice as much from an assignment that takes 10 hours of their time as from one that takes 5 hours?

206

Creating Effective Assignments

Great assignments are relevant and engaging.  They engage students by giving them hands-on practice rather than only listening to lectures or reading textbooks. Students find them interesting and worthwhile – something that responds to an issue they care about or that is relevant to their careers or lives. If you’re not sure how to make an assignment more interesting, ask your students! They often have great ideas.

CHAPTER 16

Great assignments mirror real-world Jargon Alert! ­experiences.  Authentic assessments (see Jargon Alert) ask students to work Authentic Assessments on real-life tasks such as analyzing case Authentic assessments ask students to complete messy real-world tasks with multiple acceptable studies with bona fide data, conductsolutions, rather than fabricated problems for ing realistic laboratory experiments, or which there is only one correct answer. completing field experiences. Try “You are there” scenarios: “You are an expert ­chemist (statistician, teacher, anthropologist, etc.) asked to help with the following situation . . .”

Great assignments have challenging but realistic, attainable expectations.  Often when students know exactly what they need to do to earn a high grade, they will rise to meet that standard, even if it means accomplishing things to which they never thought they could aspire. Ask students to demonstrate not just simple understanding but also thinking skills such as analysis, evaluation, and creativity. Focusing on these kinds of skills makes the assignment more challenging, worthwhile, and interesting, and it promotes deeper learning. Great assignments are varied.  As discussed in Chapter 3, the greater your variety of student learning evidence, the more confidently you can infer how well students have learned what you want them to. Instead of assessing students solely through multiple-choice tests or solely through writing assignments, for example, assess them using a combination of tests, writing assignments, and other projects. One assignment might be a panel presentation, another could be a chart or diagram, and a third a written critique. Students might convey the essence of a novel’s protagonist through a diagram, video, or oral presentation rather than through a traditional essay. List 16.1 offers examples of assignments beyond the usual term paper or essay. Many of these assignments generate visual products that are faster to grade than traditional papers. If you think your assignments are in a rut, or if you’re struggling to come up with learning activities for some learning goals, consider Learning Assessment Techniques (LATs) (see Jargon Alert). As you can see from the examples in 207

The Assessment Toolbox

List 16.1  Examples of Assignments Beyond Essays, Term Papers, and Research Reports ■■

■■

Abstract or executive summary Advertisement or commercial Annotated bibliography Biography or realistic fictional diary from a historical period Briefing paper Brochure or pamphlet Campaign speech Case study/analysis Client report Collaborative group activity Database Debate or discussion Debriefing interview preparation Dramatization of an event or scenario, in writing or a presentation Editing and revision of a poorly written paper Evaluation of opposing points of view or the pros and cons of alternative solutions to a problem Experiment or other laboratory experience Field notes Game invention Graph, chart, diagram, flowchart, or other visual aid Graphic organizer, taxonomy, or classification scheme Handbook or instructional manual

■■

Learning Assessment Techniques (see Jargon Alert) Letter to an editor or business Model, simulation, or illustration Narrative News report on a concept or from a historical period Oral history recording of an event Plan to research and solve a problem Plan to conduct a project or provide a service Portfolio (Chapter 18) Poster, display, or exhibit Presentation, demonstration, or slideshow Proposal for and justification of a solution to a problem Reflection on what and how one has learned (Chapter 20) Review and critique of one’s own work or that of a peer, a performance, an exhibit, a work of art, a writer’s arguments, or how something could have been done better Selected portions of research paper (for example, only the problem statement and the review of literature) Survey (Chapter 20), including an analysis of the results Teaching a concept to a peer or child Video recording

■■

Journal or log (Chapter 20)

■■

Website

■■ ■■ ■■ ■■

■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■

■■ ■■

■■ ■■ ■■ ■■

■■

Jargon Alert! Learning Assessment Techniques (LATs) Learning Assessment Techniques, or LATs (Barkley & Major, 2016), are learning activities organized around the six dimensions of L. Dee Fink’s (2013) taxonomy of learning goals (Chapter 4).

■■ ■■ ■■ ■■

■■ ■■ ■■ ■■ ■■ ■■ ■■

■■

■■

■■

■■

■■

Table 16.1, many LATs are relatively quick and engaging assignments. Learning Assessment Techniques: A Handbook for College Faculty (Barkley & Major, 2016) offers practical suggestions on how to assess each LAT.

Great assignments fit the scope of the learning setting.  Capstones (Chapter 5) are great learning and assessment opportunities in courses, programs, and significant co-curricular experiences such as service learning projects. Smaller, more focused 208

Dimension of Learning

Example

Foundational Knowledge Entry Ticket

Description

Application

What’s the Problem? Students look at examples of common problem types (such as types of fallacies in arguments) and name the type of problem each example represents.

Integration

Concept Map

Students draw a diagram that is a network of ideas or concepts studied, with lines connecting related concepts.

Human Dimension

Dramatic Dialogues

Students create an imagined discussion of a problem or issue between two real or imaginary characters.

Caring

Three-Minute Message

In three minutes, students deliver a compelling argument supported with convincing details and examples.

Learning How to Learn

Student Generated Rubrics

Students use an exemplary model of student work (essay, lab report, work of art, and so on) to create a rubric (Chapter 15) articulating the key traits of effective work.

Creating Effective Assignments

At the beginning of a class, students answer a brief question on an assigned reading on an index card or other medium.

CHAPTER 16

Table 16.1:  Examples of Learning Assessment Techniques

learning experiences call for simpler, shorter assignments and assessments such as the Learning Assessment Techniques discussed above. Great assignments are clear to students.  This is discussed in the next section.

Crafting effective assignments Carefully crafted assignments are critical parts of the teaching/learning process because, regardless of what we state in syllabi or say in class, assignments are the most powerful way we communicate our expectations to students. Thoughtfully written assignment guidelines inspire students to give the assignment their best effort and thereby achieve the assignment’s learning goals. With poorly written guidelines, students may complete an assignment without learning what we want them to learn. Suppose, for example, that history faculty want students to be able to analyze the impact of a noteworthy individual on the outcome of World War II. They ask students to write a term paper on “a person involved with World War II” with no further guidance or direction. Some students might complete the assignment by only summarizing the life history of an individual, doing nothing to develop – or 209

The Assessment Toolbox

demonstrate – their analysis skills. Faculty assessing the papers may find little evidence of analysis skill, not because students are poor at analysis but because the assignment never explicitly asked the students to analyze. Mary-Ann Winkelmes (Berrett, 2015) offers an elegant framework for crafting effective assignments: Give students transparent information on the assignment’s task, purpose, and assessment criteria. Use the principles of backwards curriculum design (Chapter 5) to approach this: First articulate the assignment’s purpose, then its assessment criteria, and then the task. Identify the assignment’s key learning goals.  Begin by identifying the assignment’s purpose, which is its key learning goals: What you want your students to learn from the assignment. The assignment should focus students on those learning goals that you consider most important. If you are giving a writing assignment, for example, identify the specific kinds of writing skills that you most want students to strengthen. Also take the time to articulate why those learning goals are important – how achieving them will help students, whether in their careers, in subsequent study, or in life. Some students learn more effectively when they see clear, practical benefit in what they are being asked to do. Draft the rubric to be used to assess completed assignments.  The rubric should reflect the assignment’s key learning goals and emphasize the most important traits of those goals. It may strike you as curious that the rubric should be drafted before creating the assignment itself. Shouldn’t we first create the assignment and then the rubric? But think of planning a road trip. When we use a map or GPS to plot a route, we first locate our destination and then chart the most appropriate route to get there. When we teach, we are taking our students on a similar journey. Our assignments are more effective if we first clarify what we want students to learn from the assignment (the destination) and then design an assignment that will help them achieve those ends (the route to get there). If this process differs from your experience and therefore seems daunting (“How can I possibly create grading criteria when I don’t know what I’m asking students to do?”), use an iterative process to create an assignment. First, list the assignment’s learning goals: The most important things you want students to learn by completing the assignment. Then draft rough ideas for the assignment itself. Next, use your ideas for the assignment to explicate your learning goals into rubric traits. Once you’ve spelled out the rubric traits and performance levels, clarify your assignment so it will elicit the work described in the rubric.

210

Craft the assignment guidelines.  Give your students clear, written instructions that point them in the right direction. The clearer your guidelines, the better some

CHAPTER 16 Creating Effective Assignments

students will do. An A assignment is a lot Jargon Alert! faster and easier to grade than a C or D assignment, so this is a win-win strategy: Prompt Your students work harder and learn more, A prompt is the guidelines for an assignment: The (usually written) statement or question that and you spend less time grading (Walvoord tells students what they are to do. Prompts are & Anderson, 2010)! used to communicate virtually everything we ask students to do. The title of an assignment is a powerful There are two basic kinds of prompts. Restrictedway to convey to students what you want response prompts ask everyone to provide pretty them to do (Walvoord & Anderson, 2010). much the same response, just in their own words. Mathematics problems, science laboratory Consider terms like argumentative essay, assignments, and essay test questions are often original research project, or sociological restricted-response prompts. analysis that make the assignment clearer Extended-response prompts give students latitude than the usual term paper. Begin the prompt in deciding how to complete the assignment. Their completed assignments may vary considerably in with an introductory sentence that’s an organization, style, and content. Suppose that overview of what you want students to do students are asked to speculate, with appropriate justification, on how our daily lives might be different and then answer the questions in List 16.2. today if the United States had never engaged in space Match the length of the prompt to the exploration. The visions and supporting evidence in equally outstanding papers might vary a great deal. scope of the assignment. Guidelines for major assignments such as portfolios or term projects can run a page or more. Brevity is important, however, when you are asking for very short responses such as minute papers (Chapter 20) or when you are giving timed in-class assignments such as an essay exam. In these situations, every minute counts and time spent reading your directions is time that can’t be spent thinking or responding. Consider asking students to provide a written reflection on their completed assignment as discussed in Chapter 20. Break large assignments into smaller pieces.  Rather than distribute a major assignment on the first day of a course and collect it on the last day, break the assignment into pieces that are submitted or checked at various points during the course. You might ask students to submit first their research paper topic, then an annotated bibliography for it, then an outline of it, then a draft of it for peer review. This helps students manage their time and, more importantly, gets those heading in a wrong direction back on track before it’s too late for them to salvage their project. Scaffolding an assignment (see Jargon Alert) also makes your job of assessing the comJargon Alert! pleted assignments easier because, as noted Scaffolding earlier, good work is faster and easier to Scaffolding is breaking an assignment into pieces or steps that are progressively challenging and giving assess than poor work. And, as discussed in students support as they work on each step of the next section of this chapter, scaffolding the process. can discourage plagiarism.

211

The Assessment Toolbox

List 16.2  Questions to Address in a Prompt for an Assignment ■■

■■

■■

■■

■■ ■■

■■

Why are you giving students this assignment? • What is its purpose? • What do you expect students to learn by completing it? • How will it help prepare them to succeed in later courses, in the workplace, and/or in their lives? What skills and knowledge you want students to demonstrate? • Explain terms that may be fuzzy to your students even if they are clear to you, such as compare, evaluate, and discuss. • Explicitly ask for the outcomes you’re seeking. For example, if the rubric assesses the quality of arguments, explicitly ask students to make arguments that meet the rubric’s traits. • Ask only for knowledge or skills that are listed on the rubric (Messick, 1994). What should the completed assignment look like? • What should be included in the completed assignment? • Who is the (perhaps hypothetical) audience for the assignment: academicians, people working in a particular setting, or the general public? • How should students format the completed assignment? • If the assignment is to write or present something, what is an optimal length? How are students to complete the assignment? • How much time do you expect them to spend on the assignment? • How do you expect them to focus their time and energy? • If this is a course assignment, how much will it count toward the final course grade? • What readings, reference materials, technologies, and other resources are they expected to use? • Can they collaborate with others? If so, to what extent? What are the deadlines for the assignment? What assistance can you provide while they are working on the assignment? • Are you willing to critique drafts, for example? How will you score or grade the assignment? • The best way to communicate this is to give students the assignment’s rubric (Howell, 2011).

Depending on your students’ needs, the assignment’s learning goals, and your time constraints, at these checkpoints you might: • Simply check off that this portion of the project is complete or in progress. • Review and comment on this portion of the project. • Have student peers assess this portion of the project using a rubric that you provide. • Give this portion of the project a tentative grade (pending subsequent revisions). Can assignments be purposefully vague?  While good prompts are often generous in the guidance they give to students, some faculty like to give purposefully vague assignments, because one of the things they want students to learn is how to figure out the assignment on their own. This practice can be fine, but only if: 212

The work of others is so readily available today that student plagiarism is a growing concern. While there is no way to eliminate plagiarism, the strategies to counter plagiarism in List 16.3 (Carroll, 2004) may help. List 16.3  Strategies to Counter Plagiarism ■■

■■

■■

■■

■■

■■

Creating Effective Assignments

Countering plagiarism

CHAPTER 16

• One of the learning goals of the assignment is to learn how to choose, define, or clarify a problem or issue; and • Students have opportunities to learn and practice these kinds of skills before tackling the assignment; and • This learning goal is reflected in the rubric used to assess the assignment.

Use detection judiciously. • After papers are turned in, ask students to summarize them. • Use online search engines to search for similar passages. • Interview students or ask them to write reflectively about the process they used to write the paper. Review papers for the following: • Out-of-character work • Abrupt changes in language, referencing systems, or vocabulary • Fully finished works with no evidence of research and writing processes • Anachronisms or only dated references Explicitly teach and model academic rules, values, and conventions. • Provide plenty of instruction, learning activities, and feedback that help students understand exactly what plagiarism and academic integrity are. Focus on what students should do rather than what they should not do. Test their understanding through realistic test questions and assignments on plagiarism. • Model academic integrity in your own examples, lectures, and discussions by citing the sources to which you refer. Provide opportunities for students to learn, practice, and get feedback on research and writing skills in your discipline. Use fair assessment practices (Chapter 3). • Give clear prompts that are plainly linked to key learning goals. • Vary the kinds of assignments you give, as discussed earlier in this chapter. • Give creative assignments that don’t lend themselves to plagiarism. Assign oral or visual presentations rather than written papers; scaffold large assignments, as discussed earlier in this chapter; or give assignments that ask students to relate concepts learned to personal or local experiences. Work with your colleagues to make a concerted and consistent effort to address plagiarism. • Develop and implement appropriate and consistent policies for all students and programs. • Be consistent in how plagiarism policies are explained, applied, and enforced. • Provide timely, transparent, and defensible penalties.

213

The Assessment Toolbox

For More Information The DQP Assignment Library (www.assignmentlibrary.org) is a searchable online library of peerreviewed course assignments addressing a broad range of learning goals in a variety of disciplines. It also offers additional readings on crafting assignments (www.assignmentlibrary.org/resources). Creating Significant Learning Experiences: An Integrated Approach to Designing College Courses (Fink, 2013) offers a model for aligning learning activities to key learning goals.

Time to think, discuss, and practice 1. Seaside College is designing a new first-year seminar to help its students develop the skills and dispositions they need to succeed in college. The faculty have decided that students need to learn how to evaluate information critically. Suggest an assignment that students in the seminar might complete to help them develop that skill. 2. Choose one of the following (poorly written!) prompts: • Compare the writing styles of F. Scott Fitzgerald and Ernest Hemingway. • Compare the Republican and Democratic parties. • Describe the operation of a microscope. • Research the demographics of various ethnic groups in the United States. • Compare the strengths, weaknesses, and uses of quantitative and qualitative assessment. a. Choose one person in your group to play the role of the faculty member who wrote the prompt. That person will answer your group’s questions about the course or program for which the prompt was written and the learning goal(s) that the prompt is intended to assess. b. Decide what makes the prompt ineffective. c. Rewrite the prompt so it has the traits of good prompts.

214

Some valuable ideas you’ll find in this chapter

CHAPTER 17

Writing Multiple-Choice and Other Objective Tests Test blueprints – lists of the learning goals covered by the test – are vital to planning effective multiple-choice tests. Well-designed multiple-choice questions give useful diagnostic information on where the student’s thinking went wrong. Multiple-choice and matching questions can assess some kinds of thinking skills, including the abilities to apply and to analyze.

215

The Assessment Toolbox

W

hile performance assessments (Chapter 16) are growing in popularity, multiple-choice and other objective tests may still have a place in some assessment toolboxes. This chapter discusses how to plan an objective test, how to write effective multiple-choice and other objective test items, and how objective tests can be used to assess some thinking skills.

Jargon Alert!

What is an objective assessment?

Subjective assessments (see Jargon Alert) require professional judgement to score. An objective assessment has only one correct Objective assessments (see Jargon Alert) answer for each question, while in a subjective can be scored accurately by a reasonably assessment a variety of answers may be correct. competent eight-year-old armed with an answer key, although interpretation of the scores requires professional judgment. Jargon Alert! Objective assessments are almost always Quantitative Assessments tests, while subjective assessments may Quantitative assessments use structured, be tests or performance assessments predetermined response options that can be (Chapter 16). summarized into meaningful numbers and analyzed statistically. Multiple-choice tests, rubrics Some people confuse quantitative (Chapter 15), and rating scales (Chapter 20) are assess­ments (see Jargon Alert) and examples of quantitative assessments. objective assessments, erroneously thinking that quantitative assessments are objective. Actually, it’s the other way around. Objective tests are quantitative, but so are subjective assessments using rubrics and rating scales. Objective and Subjective Assessments

Why use an objective test? Objective tests remain widely used for several reasons. Students can provide a great deal of information on a broad range of learning goals in a relatively short time.  Testing experts call this efficiency. If you want to assess student learning on a wide array of concepts and skills, a 45-minute multiple-choice test will give you more comprehensive information on their learning than a 45-minute essay test. Objective tests encourage broader – albeit shallower – learning than subjective assessments because of their efficiency.  Asking students to write a paper on a poem by Wordsworth is a good choice if the learning goal is to analyze that poem 216

What is 2 × .10? A. 20 B. 2.10 C. .2 D. .02

Each distracter gives a clue on where the student’s thinking goes wrong. Choosing B, for example, indicates that the student confuses multiplication and addition signs. Choosing A indicates that the student confuses multiplication with division, while choosing D indicates a problem with decimal place value. Faculty and staff can thus use the evidence of student learning from well-written multiple-choice tests to identify areas of difficulty and help their students accordingly.

Multiple-Choice and Other Objective Tests

Well-constructed multiple-choice tests can help diagnose problem areas.  ­Consider this simple example:

CHAPTER 17

thoroughly but a poor choice if the learning goal is to analyze Romantic literature. For the latter goal, an objective test asking students to react to a variety of examples of Romantic literature might be a better choice.

Objective tests are fast and easy to score,  although they are difficult and time-consuming to construct. If they are stored securely so they can be reused, the payback on the time spent writing them increases. Some people think that objective tests are more valid or accurate than subjective assessments. To the contrary, both objective and subjective assessments require professional judgment. In objective tests, professional judgment goes into developing or choosing test questions, which can vary considerably in their difficulty and in the learning goals they assess, and in setting standards for adequate performance (Chapter 22). So all assessments are subjective, in that they all require professional judgment, and objective assessments are not necessarily better than subjective ones.

Why not use an objective test? Just like any other assessment, objective tests have shortcomings and are not always appropriate. Subjective assessments such as performance assessments are increasingly popular for several reasons. Objective tests cannot assess many important thinking skills,  including organization, synthesis, and original thinking. 217

The Assessment Toolbox

Objective tests are less convincing evidence of real-world skills than authentic assessments  (Chapter 16). Writing samples, for example, are more convincing evidence of writing skill than answers to multiple-choice questions on how to write. Similarly, watching dental students treat a cavity provides more compelling evidence of their skill than answers to multiple-choice questions on how to do this procedure. (Assessing performances such as this procedure is discussed in Chapter 22.) Objective test scoring procedures usually don’t allow for nuances.  On a subjective math test, for example, students can receive partial credit for doing part of a multistep problem correctly. But on a multiple-choice math test, they receive no credit for an incorrect answer, even if they do much of their work correctly. Objective test items are hard to write.  Writing good, clear multiple-choice items with good distracters can be difficult and time-consuming. Objective tests sometimes require significant reading ability.  If so, they may not fairly assess students who have achieved the test’s learning goals but read slowly or for whom English is a second language. While reading is an essential skill, if we are assessing understanding of science concepts, we want an assessment tool that assesses just that.

Jargon Alert! Test-Wise Test-wise students are adept at discerning inadvertent clues to correct and incorrect multiplechoice options. This increases their chances of choosing the correct answer, even if they haven’t learned what the test is assessing.

Jargon Alert!

It’s possible to guess the correct answer to most objective items  through either plain luck or test-wise skills (see Jargon Alert). While a well-constructed test minimizes testwise students’ advantage, there is always the possibility that students who haven’t learned what they’re supposed to will do relatively well on a test through chance alone.

Planning an objective test

Chapter 16 explained that effective assignments are planned by developing a rubric: A list of the learning goals or traits that students are to demonstrate in the completed assignment. Effective tests are similarly planned by developing a test blueprint (see Jargon Alert). Exhibit 17.1 is an example of a test blueprint for an exam in an Educational Research Methods course.

Test Blueprint

A test blueprint or table of specifications is list of the learning goals addressed on the test. It may also list the number of test items assessing each learning goal.

218

(There is one test question on each topic unless otherwise indicated.)

Instrumentation and Survey Research ■■ Identify the relative merits and limitations of published and locally developed instruments. ■■ Recognize examples of each of the four frames of reference for interpreting scores. ■■ Recognize appropriate uses of each item format (such as multiple-choice and Likert scale). ■■ Understand the characteristics of a good instrument item, including how to avoid biased questions. Descriptive Statistics ■■ Select the most appropriate descriptive statistic for a given research situation. ■■ Use percentage guidelines to interpret standard deviations. ■■ Identify the direction and strength of r and/or a scatterplot. ■■ Identify the likely direction and strength of a correlation between two given variables.

Multiple-Choice and Other Objective Tests

Sampling ■■ Recognize the difference between a sample and a population. ■■ Understand how each type of sample is selected. ■■ Choose an appropriate sample size.

CHAPTER 17

Exhibit 17.1  A Test Blueprint for an Exam in an Educational Research Methods Course

Validity and Reliability ■■ Identify the type of reliability or validity evidence provided by given information on an instrument. (2 questions) ■■ Understand the meaning and implications of measurement error. ■■ Recognize examples of measurement error. ■■ Understand the general principles for ensuring validity. Inferential Statistics ■■ Select the most appropriate inferential statistic (t, F, or χ2) for a given research situation. (2 questions) ■■

■■

Identify the most common cutoff points that statisticians use in deciding whether two means differ statistically significantly from one another. Interpret the results of t-tests as presented in research articles.

Experimental Research ■■ Interpret the symbolic representations of experimental designs. ■■ Identify the appropriate research design for a given research situation. Correlational Research ■■ Explain what r2, R, R2, and partial correlations are and what they tell us. (2 questions) ■■ Explain what regression analysis is used for and what it tells us.

Why create a test blueprint? Chapters 5 and 16 note that students spend their learning time and energies focusing on what they’re graded on. No matter what your stated learning goals are, if your test focuses on memorized knowledge, for example, that’s what students will spend their time learning, and that’s what they’ll take away from the learning experiences you 219

The Assessment Toolbox

provide them. Test blueprints help ensure that students focus on learning what you think is most important. They accomplish this in several ways: Test blueprints help ensure that the test focuses on the learning goals you think are most vital.  Suppose that you are writing a test for Units 8, 9, and 10 of a course. While you consider Unit 10 the most important of the three, you may find that it’s much easier to think of test questions for Unit 8. If you write a test without a blueprint, you can easily end up with too many questions on Unit 8 and too few on Unit 10. Students taking such a test may be able to earn a decent score without having mastered important learning goals of Unit 10. Test blueprints help ensure that a test gives appropriate emphasis to thinking skills.  Faculty writing test questions without a blueprint often end up with questions asking for basic conceptual knowledge rather than questions asking students to interpret, infer, analyze, or think in other ways. In fact, tests written without blueprints sometimes become tests of trivia rather than tests of thinking skills. Students who do well on such tests may not have mastered important skills, while students who have truly learned those important thinking skills may nonetheless earn low scores. Test blueprints make writing test questions easier.  Armed with a test blueprint, you’ll know exactly what must be covered on the test (one question on Concept A, two on Skill B, and so on), and you’ll spend less time pondering what questions to write. Test blueprints help document that students have achieved key learning goals.  As discussed in Chapter 23, scores on each test item, accompanied by test blueprints that describe each item’s learning goal, are direct evidence (Chapter 3) of exactly what students have learned.

Identify the learning goals you’d like to address on the test Start creating a test blueprint by using syllabi, lesson plans, learning activities, and other curricular materials to list the learning goals that you’d like to assess. Phrase your learning goals using action verbs that describe what students should be able to think and do (Chapter 4). Instead of simply listing Hemingway in a test blueprint on twentieth-century American literature, for example, state the knowledge and skills you want students to demonstrate regarding Hemingway (perhaps Identify works written by Hemingway and Distinguish Hemingway’s writing style from those of his peers). Keep in mind that objective tests can assess more than basic knowledge and understanding – you’ll see how later in this chapter. 220

For a comprehensive exam, decide how many test items you’d like on each major unit A midterm exam, for example, might cover the first five units in the course syllabus. Decide how many test questions you want on each of those units, in proportion to the relative importance of each unit (perhaps 6 questions on Unit 1, 10 questions on Unit 2, and so on).

Finalize the test blueprint and provide it as a study guide

Multiple-Choice and Other Objective Tests

You may decide that some learning goals are so important that you want three, four, or more questions on each of them, to be sure that students have achieved the goal and can address it from multiple perspectives or contexts. Other learning goals may need only one question.

CHAPTER 17

Decide how many test items you’d like for each learning goal

Compare the number of items you’d like to have on each learning goal or objective (and, for a comprehensive exam, on each major unit) with the total number of items you can realistically expect students to answer in the testing period. You may find that you can’t address every learning goal on the test. Perhaps, for example, you’d need 40 items on the test to cover your learning goals, but students will be able to answer only 30 in the testing period. You’ll need to eliminate some of the less-important learning goals to make sure the test gives enough attention to the most important ones. Once you have finalized the test blueprint, provide it to your students to help them focus their studies on the learning goals that you think are most important.

Decide how difficult your test should be Objective tests have two different assessment purposes. One is to make sure that students have learned the essentials – matters so vital that the consequences of not having learned them are dire. In health and medical disciplines, for example, if students haven’t mastered every key learning goal, a future patient might conceivably die. In other disciplines such as laboratory science and culinary arts, if students haven’t mastered key learning goals on safety, someone might be gravely injured. And in many disciplines, certain basic concepts or skills must be mastered in order to succeed in later courses or in a career. In these situations, students who have studied and prepared well should be expected to answer virtually every item correctly. A score below 90 percent or even 95 percent for these kinds of test items might be considered failing. 221

The Assessment Toolbox

The other potential purpose of an objective test is to distinguish students with superb levels of learning from those with mediocre levels of learning – to identify the students who deserve an A and not just a passing grade. These tests have challenging questions that only the top students answer correctly. Many tests aim to serve both purposes. Some test items address fundamental essentials that virtually every student should answer correctly, while other test items are intentionally difficult to challenge the very best students and thereby separate the A, B, and C students. Review your test blueprint and identify which learning goals are essential and which are more challenging. Writing difficult multiple-choice questions to separate the A, B, and C students is, well, remarkably difficult. It’s very hard to come up with a test question that assesses an important learning goal but that a large proportion of students will get wrong. It’s much easier to write difficult items that are on minutiae, trick questions that require nuanced reading, or questions that assess logical reasoning skill rather than key learning goals. Follow the suggestions in List 17.1 to write challenging multiple-choice questions that assess important learning goals. List 17.1  Tips for Writing Challenging Rather Than Trick Questions ■■ ■■

■■

■■

■■

Use a test blueprint. It ensures that each item assesses an important learning goal. Make your tests open-book, open-note. Tell students they can use any resource they like except a friend or the means to communicate with one. Using open-book, open-note tests forces you to eliminate items assessing simple knowledge that students can look up. Your test will include only items that assess deeper comprehension and thinking skills. Build items around common misconceptions. Many people, for example, think that plants get nutrients only from soil and water, not air; this misconception can become the basis of an effective botany test question. Create interpretive exercises (discussed later in this chapter). They assess thinking skills such as application and analysis. Evaluate your test results using the tools in Chapter 24. Revise any unnecessarily difficult items before including them in another test.

Writing good multiple-choice items As with any other assessment, multiple-choice tests should yield fair and truthful information on what students have learned (Chapter 3). List 17.2 offers suggestions for writing fair and truthful multiple-choice items (Paniagua & Swygert, 2016; Parkes & Zimmaro, 2016; Haladyna, 2015) that follow two basic precepts. • Remove all the barriers that will keep a knowledgeable student from answering the item correctly. Students who have truly learned the concept or skill that an item assesses should choose the correct answer. 222

General Tips ■■ Keep each item as concise as possible. Short, straightforward items are usually easier for students to understand than complex statements. Avoid irrelevant material, digressions, and qualifying information unless you are specifically assessing the skill of identifying needed information. Don’t repeat the same words over and over in the options; put them in the stem (see Jargon Alert). ■■ Define all terms carefully. What do you mean by sometimes, usually, or regularly? If you ask Which bird is largest? make clear whether you mean largest in terms of wingspan or weight. ■■ Don’t make the vocabulary unnecessarily difficult. Except for terms you are specifically assessing, keep your vocabulary simple – perhaps high school level. Otherwise, you may unfairly penalize students who have achieved your learning goals but don’t have a strong general vocabulary. ■■ Watch out for interlocking items in which a student can discern the answer to one from the content of another. Review carefully all items that share similar options. Also don’t ask students to use their answer to one question to answer another. If they get the first question wrong, they will automatically get the other question wrong as well, even if they have learned the concept or skill assessed in the second question.

Multiple-Choice and Other Objective Tests

List 17.2  Tips for Writing Good Multiple-Choice Questions

CHAPTER 17

• Remove all clues that will help a less-than-knowledgeable student answer the item correctly. Students who truly haven’t learned the concept or skill that an item assesses should answer the item incorrectly.

Writing a Good Stem ■■ The stem should ask a complete question. The student shouldn’t have to read the options to discern the question. To check this, see if students would be able to answer the question posed in the stem correctly if no options were provided. ■■ Avoid “Which of the following” items. They require students to read every option and can penalize slow readers in a timed testing situation. ■■ Ask questions that can’t be answered from common knowledge. Someone who hasn’t studied the material shouldn’t be able to answer the questions correctly. ■■ Avoid negative items. In a stressful testing situation, students can miss the word not or no. If you must have negative items, underline, capitalize, and/or boldface words like NOT or EXCEPT. ■■ Avoid grammatical clues to the correct answer. Test-wise students know that grammatically incorrect options are wrong. Use expressions like a/an, is/are, or cause(s). Writing Good Options ■■ You needn’t have the same number of options for every question. Three options are fine (Rodriguez, 2005), and some questions may have only three plausible options (such as Increases, Decreases, Remains unchanged). Only rarely are more than four options needed. A good fifth option is often hard to come up with, takes extra reading time, and only reduces the chances of randomly guessing the correct answer from 25 percent to 20 percent. ■■ Order options logically. Order options numerically if they are numbers, and alphabetically if they are single words. This helps students who know the answer find it quickly. If the options have no intuitive order, insert the correct answer into the options randomly. (continued)

223

The Assessment Toolbox

List 17.2 Continued ■■

■■

Line up options vertically rather than horizontally. It’s much easier – and less confusing – to scan down a column than across a line to find the correct answer. If you are using a paper test and your options are so short that this seems to waste paper, arrange your test in two columns. Make all options roughly the same length. Test-wise students know that the longest option is often the properly qualified, correct one. ■■ Avoid repeating words between the stem and the correct answer, a clue that test-wise Jargon Alert! 0 will pick up. Stems, Options, Responses, Distracters, and Foils: ■■ Avoid using None of the above. A student The Elements of Multiple-Choice Test Items may correctly recognize wrong answers without The stem of a multiple-choice test item is the part knowing the right answer. So use this option that asks the question. It may be phrased either as a question or as an incomplete sentence. Responses only when it is important that the student know or options are the list of choices in a multiplewhat not to do. If you use None of the above, choice item from which the student chooses an use it in more than one question, both as a answer. The incorrect options of a multiple-choice item are called distracters or foils because their correct answer and as an incorrect option. purpose is to distract or foil students who don’t ■■ Avoid using All of the above. This option know the correct answer from choosing it. requires students to read every option, penalizing those in a timed testing situation who know the material but are slow readers or for whom English is a second language. Students who recognize Option A as correct and choose it without reading further are also penalized. All of the above also gives full credit for incomplete understanding; some students may recognize Options A and B as correct and therefore correctly choose All of the above even though they don’t recognize Option C as correct.

Writing Good Distracters ■■ The best distracters (see Jargon Alert) help diagnose where each student went wrong in his or her thinking, as discussed at the beginning of this chapter. Identify each mental task that students need to do to answer a question correctly, and create a distracter for the answer students would arrive at if they completed each step incorrectly. ■■ Use intrinsically true or at least plausible statements. Test-wise students recognize ridiculous statements as wrong. To see if your test has such statements, ask a friend who’s never studied the subject to take the test. His or her score should be roughly what would be earned from guessing randomly on every item (25 percent for a four-option multiple-choice test). ■■ Each distracter in difficult items should be attractive to at least a few students. Unless the test item assesses an essential skill that the vast majority of students should answer correctly, if no one chooses a distracter, it’s a waste of reading time. You can check this after the test is administered by counting the number of students choosing each option. ■■ Use distracters that will foil test-wise students who haven’t learned the concept or skill being assessed. Create a verbal association between a distracter and the stem, for example, or make one of the distracters a relatively long option.

224

(Correct answers are in italics.) 1. Which statement refers to measurement as opposed to evaluation? A. Emily got 90% correct on the math test. B. Chris’s test scores have increased satisfactorily this year. C. Justin’s score of 20 on this test indicates that his study habits are ineffective. D. Keesha got straight As in her history courses this year. 2. Alyssa took a test on Tuesday after a big fight with her parents Monday night. She scored a 72. Her professor let her retake the same test on Thursday when things cooled off. She scored 75. The difference in her scores may be attributed to: A. chance or luck. B. lack of discrimination. C. lack of validity. D. measurement error. 3. People who score high on the Meyers Musical Aptitude Scale usually score low on the Briggs Biologists Aptitude Test. People who score low on the Meyers usually score high on the Briggs. Which of the figures below most likely represents the correlation between the two tests? A. .80 B. .00 C. –.10 D. –.60 4. Choose the most likely correct answer to this nonsense question, based on what you know about informed guessing on tests. A drabble will coagulate under what circumstances? A. Only when pics increase B. Only when pics change color C. By drawing itself into a circle D. Usually when pics increase, but occasionally when pics decrease

Multiple-Choice and Other Objective Tests

Exhibit 17.2  Multiple-Choice Questions on Assessment Concepts

CHAPTER 17

Writing good multiple-choice items can be difficult; test publishers write, try out, and discard many, many items for each one that ends up in a published test. Exhibit 17.2 gives some examples of multiple-choice items that follow most of the suggestions in List 17.2 and assess thinking skills as well as basic understanding. But even these examples don’t completely follow all the suggestions in List 17.2. Don’t expect to be able to follow every suggestion all the time, and don’t expect your test questions to work perfectly the first time you use them. Analyze the test results (Chapter 24), revise the test accordingly, and within just a few cycles you’ll have a really good test.

225

The Assessment Toolbox

Jargon Alert! Interpretive Exercises Interpretive exercises, sometimes referred to as integrative item sets, context-dependent or enhanced multiple-choice items, or scenario testing, consist of a stimulus, such as a reading passage, vignette, diagram, or chart, that students haven’t seen before, followed by a set of objective items on that stimulus.

Interpretive exercises Interpretive exercises (see Jargon Alert) have three defining characteristics. The stimulus material that students respond to must be new to the students;  they must never have seen it before. This requires students to apply what they have learned to a new situation, making interpretive exercises a good way to assess application skills.

Students must read or examine the stimulus material  in order to answer the objective items that follow. They should not be able to answer any of the items simply from their general understanding of what they’ve learned. This makes interpretive exercises an assessment of application skill rather than simple conceptual understanding. The items must be objective,  with one and only one correct answer for each item. If you ask students to write or otherwise create something in response to the stimulus, you have a performance assessment, not an objective test. Chapter 16 discusses how to create effective performance assessments. Exhibit 17.3 is an example of an interpretive exercise. Interpretive exercises can assess skills in generalizing, inferring, concluding, problem-solving, and analysis as well as applying knowledge and skills to new situations. Performance assessments can assess these skills as well, but interpretive exercises, like all objective item formats, are more efficient, as discussed earlier in this chapter. Interpretive exercises are not always appropriate, however. If the stimuli are reading passages, interpretive exercises may unfairly penalize students who have achieved your learning goals but are slow readers or for whom English is a second language. And, while interpretive exercises are very good for assessing some thinking skills, they cannot assess other thinking skills such as organizing, defining problems, and creating. The key to writing good interpretive exercises is to keep in mind their three defining characteristics. List 17.3 offers additional suggestions.

K-type items K-type multiple choice items (see Jargon Alert) have been largely discredited (Paniagua & Swygert, 2016; Parkes & Zimmaro, 2016). One reason is that they require students to read every option and study all the possible permutations. 226

Item 1

A

Top third Bottom third Item 2 Top third

B*

C

D

10 1

4

3

2

A*

B

C

D

8

Bottom third

2 7

3

B

C*

D

Item 3

A

Top third

5

1

4

Bottom third

2

4

4

C

D

Item 4

A*

Top third

10

Bottom third

9

B 1

Multiple-Choice and Other Objective Tests

(Item analysis – the concept assessed by these – is discussed in Chapter 24.) Items 1–5 refer to the item analysis information given below. The correct options are marked with a *.

CHAPTER 17

Exhibit 17.3  An Example of an Interpretive Exercise

Write the item number (1, 2, 3, or 4) in the space provided. 1. 4 Which item is easiest? 2. 3 Which item shows negative (very bad) discrimination? 3. 2 Which item discriminates best between high and low scores? For the remaining items, write the option letter (A, B, C, or D) in the space provided. 1. B In Item 2, which distracter is most effective? 2. A In Item 3, which distracter must be changed?

List 17.3  Tips for Writing Good Interpretive Exercises ■■

■■ ■■ ■■

Keep the size of the stimulus in proportion to the questions asked. Having students read a full page of text in order to answer only three questions is hardly an efficient use of their time. Generally, aim to ask at least three questions about any stimulus, and ask more about longer stimuli. Be on the lookout for interlocking items. They seem to crop up more often in this format. Give students realistic scenarios. Be creative! The stimulus need not be a reading passage; it can be any of the following: • A chart, diagram, map, or drawing with real or hypothetical information (Exhibit 17.3) • A brief statement written by a scholar, researcher, or other significant individual • A description of a real or imaginary scenario, such as a scientific experiment or a business situation • For foreign language courses and programs, any of the above written in a foreign language

227

The Assessment Toolbox

This takes a lot of time, which limits the Jargon Alert! number of questions – and learning goals – that can be included in a timed testing K-Type Items period. The format also penalizes students K-type or complex multiple-choice items consist of a stem followed by options, more than one of who are slow readers or for whom English is which may be correct. Those options are followed a second language. by additional options that are combinations of the initial options (A and B, A and C, All of the above, Another reason that K-type items and so on). are largely discredited is that they give test-wise students clues from the options and combinations presented. A test-wise student who recognizes that Option A is wrong, for example, will eliminate all the other options that include A. Similarly, a test-wise student who recognizes that Option B is correct will eliminate all options that don’t include B. The only legitimate reason to give students K-type items is to prepare them for a licensure or certification exam that uses this format. In this situation, your course or program’s learning goals should include the analytical reasoning skills needed to tackle these questions successfully, and your students should have plenty of opportunity to learn and practice those skills before taking your tests.

Matching items If, as you write a multiple-choice test, you find yourself writing several items with similar options, consider converting them into matching items (see Jargon Alert), which are especially efficient multiple-choice questions. Because students need to read only one set of options to answer several items, students can often answer five well-written matching items more quickly than five multiple-choice items, giving you more assessment information in the same amount of testing time. Matching items are also faster to write than multiple-choice items because you don’t have to come up with a fresh set of distracters for each item. Matching items are a good way to assess certain kinds of basic knowledge. Students Jargon Alert! can match terms and definitions, causes Matching Items and effects, people and their achievements, Matching items (also known as extended matching foreign words and their English translations, or R-type items) are a set of multiple-choice items or tools and their uses. with identical options. Matching items can also assess some thinking skills, especially the ability to apply what students have learned to new situations and the ability to analyze interrelationships. Students can match concepts with examples, causes with likely effects, symptoms with likely diagnoses, diseases with potential treatments, 228

■■

■■

■■

■■

■■

A matching set should consist of homogenous items. Every option in the answer key should be a plausible answer for every item in the set. Otherwise, test-wise students will quickly eliminate implausible answers, while students who are less test-wise will read and consider the full set of responses over and over. Allow students to use each option more than once or not at all. A perfect match (in which each option is the answer for exactly one item) gives an unfair advantage to test-wise students, who will cross out each option as it is chosen and then guess among the options that are left. A perfect match also gives an unfair disadvantage to students who misunderstand one item but truly know all the other answers; if they choose one incorrect answer, they must, by process of elimination, choose a second incorrect answer, because it will be the only option left. Make it easy for students who have learned the concept or skill to find the correct answer. Make the longer statements the stems, and limit the answer key to single words or short phrases. For example, list definitions as the stems and the terms they define as the options. Otherwise, students will need to continually scan through a list of lengthy options to find the correct answers, which penalizes those who have learned the material but are slow readers or for whom English is a second language. Limit the number of matching items in a set to no more than 10 or so, and keep the entire exercise on one page or screen. Arrange the options in a logical order (usually alphabetically). Give clear directions. In an introductory sentence, explain how the stems and options are related (for example, ask students to match each theory with the person who conceived it). Point out that options may be used more than once or not at all; your students may have not seen this kind of matching set before. Consider giving the set of stems and the set of options explanatory titles (for example, Theory and Author). Be inventive! Matching sets need not be lists of words or phrases. Students can match concepts with their symbols, pictures of objects with their names, or lettered parts of a diagram, drawing (say, a microscope or cell), map, or chart with their functions, uses, or meaning.

Jargon Alert! True-False Items True-false or binary items are multiple-choice items with only two options.

Multiple-Choice and Other Objective Tests

List 17.4  Tips for Writing Good Matching Items

CHAPTER 17

and hypothetical problems with the tools, concepts, or approaches needed to solve them. List 17.4 offers suggestions to help create good sets of matching items . . . possibly quite different from those you may have used or seen in the past. Exhibit 17.4 is an example of a matching set that follows these suggestions and assesses application skills as well as conceptual understanding.

True-false items The most common use of true-false items (see Jargon Alert) is to assess basic knowledge: Is a given statement correct 229

The Assessment Toolbox

Exhibit 17.4  Matching Items from a Nursing Research Methods Test In this set of matching items, some options may be used more than once or not at all. Correct answers are in boldface. Match each measurement (1–5) with its level of measurement. (A–D)

A. Interval B. Ordinal C. Nominal D. Ratio D 1. Fluid intake, in ounces, of a post-surgical patient C

2. Religious affiliation

D 3. Medication dosage C

4. Type of adjuvant therapy (chemotherapy, hormonal therapy, or radiation therapy)

B

5. Level of patient advocate support for a patient (very supportive, moderately supportive, somewhat supportive, not supportive)

Adapted with permission from test questions written by Dr. Christina Barrick, Associate Professor of Nursing, Towson University

or not? But they can also be used in other situations that have just two possible answers: Is a statement fact or opinion? Is this an example of direct or indirect evidence of student learning? True-false items may be an appropriate alternative to the K-type item format, discussed earlier, when students must know that more than one option may be correct. For example, students may need to know that a symptom may be a sign of several diseases but not others. You can assess their knowledge of this by asking them a series of true-false items, each asking whether the symptom is a sign of a particular disease. But otherwise, true-false items have such serious shortcomings that they should be used rarely. • Students who haven’t learned the material have a high probability (50 percent) of guessing the correct answer. • Unlike multiple-choice and matching items, true-false items give no diagnostic information; they give no clues about where students who answer incorrectly went wrong in their thinking. • It’s difficult to write true-false items assessing thinking skills, although they can be used in interpretive exercises. • For classic true-false items – those giving true or false statements – it can be very difficult to write unambiguous, unqualified statements that are either always definitely true or always definitely false. • Students may correctly recognize a false statement without knowing its true counterpart. 230

List 17.5  Tips for Writing Good True-False Items

■■ ■■ ■■

Keep them simple. Avoid lengthy qualifiers and broad generalizations, which can be confusing and hard to make plausible as true or false statements. Use them only to assess important learning goals. It’s easy for true-false items to descend into trivia. Avoid negative and double-negative statements. These are especially confusing in a true-false format. Keep the proportion of true statements close to but not exactly 50 percent. Test-wise students will scan the number of true statements they’ve marked and use that to decide how to guess on the items they don’t know.

Jargon Alert! Completion or Fill-in-the-blank Items

Completion and fill-in-theblank items

To be truly objective items, completion items (see Jargon Alert) should have only one correct answer. Recall the definition of an objective test given at the beginning of this chapter: A test is objective if a reasonably competent eight-year-old armed with an answer key can score it accurately. True completion items can be scored in this fashion. Many short-answer items are really subjective, with a number of acceptable answers that require professional judgment to score. While such subjective items may be an appropriate part of an assessment program, they take considerable time to score and provide limited information. Performance assessments, discussed in Chapter 16, may be a better choice. Completion items are a good choice to assess those essential facts that must be memorized and should not be guessed from multiple-choice items. They are also appropriate when the correct answer would be easy for students to recognize in a multiple-choice format. Completion items are widely used in mathematics, for example, when a test-wise student might deduce the correct multiple-choice answer by working backward from each option. They can be a good way to develop multiple-choice distracters for future tests: Simply choose the most frequently chosen incorrect answers as distracters. Truly objective completion items rarely assess thinking skills except in mathematics. Because scoring is difficult to automate, this format is not a good choice for large-scale assessment programs. List 17.6 offers suggestions for writing truly objective completion or fill-in-theblank items.

Completion items are multiple-choice items with no options. They pose questions that students answer with a word, number, or symbol. Fill-in-the-blank items are completion items posed as a sentence with a missing word, number, or symbol.

Multiple-Choice and Other Objective Tests

■■

CHAPTER 17

If true-false items appear to be appropriate for your situation, the suggestions in List 17.5 will help make the best of them.

231

The Assessment Toolbox

List 17.6  Tips for Writing Good Completion or Fill-in-the-Blank Items ■■ ■■

■■

■■

Design them so one specific word, number, or symbol is the only correct answer. Keep all blanks or spaces for recording answers of uniform length. Blanks or spaces of varying length give test-wise students clues. To facilitate scoring, have students record all their answers in a column on one side of the page. If you are using fill-in-the-blank items, make the blanks in the sentences very short placeholders and have students write their answers in a column of longer blanks. If you are using fill-in-the-blank items, structure sentences so the blanks are toward the ends of the sentences. The sentences will be easier for students to understand than if the blanks are at the beginning. Avoid lifting sentences out of a textbook. Too often the resulting items are ambiguous or focus on trivia.

Pulling an objective test together Before assembling items into a test, review them in terms of the following: • Do the items follow the test blueprint? Do they each assess an important learning goal, or are any of them trick questions that ask about trivia? • Are the items at an appropriate reading level? Other than vocabulary terms that you are specifically assessing, are the items simple, clear, and straightforward? Are they free of excessive verbiage? • Would experts agree on the answers? • Do the items appear to be of appropriate difficulty? • Are there any interlocking items or items with any other clues for testwise students? Next, order your items. The first items should be the easiest ones, to reassure testanxious students, and quickly answered, to help those who aren’t test-wise. The last items should be the most difficult and the most complex (requiring the most thinking time). Interpretive exercises often go toward the end. Write directions that provide the information in List 17.7. Finally, let the test sit for 24 hours, and then proofread it one last time. Prepare the scoring key before the test is duplicated or posted online, as the process of preparing the key can identify typos and unclear items missed in earlier readings.

232

■■ ■■

■■ ■■

Should students explain their answers to objective items? As noted earlier, a major concern with objective items, especially true-false items, is the possibility that students who haven’t learned the material can still guess the correct answer. One way to solve this problem is to ask students to write brief explanations of why they chose their answer. For true-false items, students can be asked to correct any statements they mark as false. While this does eliminate the possibility of students guessing the correct answer, it also removes one of the fundamental advantages of objective items: Their efficiency. Students won’t be able to answer as many questions in a given amount of testing time, so the test will assess fewer learning goals. And, rather than having a scanner or a competent eight-year-old score the tests, you must read every answer and use your professional judgment to decide which are correct and which are not, which takes far more of your time. If you want students to explain their answers, consider instead giving them an assignment or essay question for which they must compose a more complete written response. This will elicit deeper thought, give you richer assessment information, and give your students a better learning experience. See Chapter 16 for suggestions.

Multiple-Choice and Other Objective Tests

■■

The purpose of the test How the answers will be scored How to answer (Can they choose more than one answer? Should they choose the one best answer? Is guessing encouraged?) How to record answers Any time limits (If the test is lengthy and timed – a two-hour final exam, for example – you may want to suggest time limits for each section.)

CHAPTER 17

List 17.7  Information to Include in Test Directions

Should students be encouraged to guess? In a word, yes. Test-wise students readily guess on items of which they’re unsure. They know that, if they can eliminate even one option as implausible, they raise their odds of guessing correctly beyond random chance. If students who aren’t test-wise aren’t encouraged to guess, they’re being unfairly penalized for not being sufficiently test-savvy.

233

The Assessment Toolbox

Time to think, discuss, and practice Write each of the following for a unit or concept that you teach or have studied, following as many of the guidelines in this chapter as you can. Share your drafts with group members for feedback and suggestions. 1. Six multiple-choice items 2. A set of matching items 3. An interpretive exercise

234

CHAPTER 18

Assembling Evidence of ­Student Learning into Portfolios Some valuable ideas you’ll find in this chapter Portfolios should have a clear educational purpose as well as a clear assessment purpose. Student reflection is an essential component of a portfolio. Portfolios are an especially good assessment choice for self-designed majors and programs with small numbers of students.

235

P

The Assessment Toolbox

ortfolios may be the most exciting and the most challenging assessment tool that we have. They can be valuable learning opportunities and assessment tools, but they can also take a great deal of time to manage and assess. This chapter explains what a portfolio is and discusses when and how to use portfolios most effectively.

Jargon Alert! Portfolio Portfolios (Eynon & Gambino, 2017; Light, Chen, & Ittelson, 2011) are collections rather than single examples of student learning. Their defining features include a clear educational purpose, student participation in selecting the contents, assessment criteria usually specified in a rubric, and student reflection.

What is a portfolio? You may remember keeping a folder of your work in one or more of your grade school classes. What are the differences between that folder and portfolios of student work (see Jargon Alert) as they are used today in higher education?

A clear educational purpose.  While folders are simply repositories for student work, portfolios are designed to help students learn in some way. If the sole purpose of your portfolio is to assess student learning, there are simpler, easier, and faster tools discussed in other chapters. Student participation in selecting contents, using faculty selection criteria.  While the contents of a folder are specified by a teacher and may include everything students create in a class, students participate in choosing what goes in their portfolios, using selection criteria provided by faculty and staff. Students might be asked, for example, to include four assignments from a course or program: One that best shows their research skills, one that best shows their writing skills, one that best shows their ability to use a particular concept to solve a problem, and one from which they feel they learned the most. Making these kinds of choices can help students develop skills in synthesis and metacognition (Chapter 4). Student reflection.  Portfolios also help students develop skills in synthesis and metacognition by including written student reflections on the significance and contribution of each item in the portfolio (Zubizarreta, 2004). Illustration of student growth and development.  While folders are often composed only of final products, portfolios can illustrate growth, either by including early examples of student work or by including documentation of the processes students went through in producing their work, such as notes, drafts, or work logs. 236

When are portfolios appropriate and feasible? Portfolios can be a rich and compelling source of evidence of what students have learned – indeed, so rich that they might be considered a gold standard of assessment. List 18.1 summarizes the main reasons to use portfolios. The primary shortcoming of portfolios is the considerable time they can consume, both for students to compile and for faculty and staff to monitor and assess. Strategies to keep portfolios manageable are discussed later in this chapter.

Assembling Student Learning Portfolios

Assessment criteria.  While folders may not be assessed systematically, portfolios are assessed using clear, consistent criteria, usually in the form of a rubric (Chapter 15).

CHAPTER 18

Continual updating.  While students add to folders but don’t refine them, students can continually update portfolios, substituting one assignment for another. As their writing skills evolve, for example, students may change the portfolio item that they think best shows their writing skills.

List 18.1  Why Use Portfolios? ■■ ■■

■■

■■

■■

■■ ■■ ■■

Portfolios assemble in one place evidence of many different kinds of learning and skills. Portfolios encourage faculty and staff to examine student learning holistically – seeing how learning comes together – rather than through compartmentalized skills and knowledge. Portfolios can show faculty and staff not only the outcome of a course, program, or learning experience but also how students have grown as learners. They’re thus a richer record than test scores, rubrics, and grades alone. Portfolios encourage students to become actively involved in their learning by thinking about and choosing how to document achievement of learning goals. Portfolios encourage student reflection (Chapter 20), which develops skill in synthesis and metacognition. Portfolios encourage diversity rather than conformity in learning activities. Portfolios provide fodder for rich conversations between students and the faculty and staff they work with. By providing information on not only what students have learned but also how they have learned, portfolios help faculty and staff refine what and how they teach.

Because of the time they can consume, portfolios may not be feasible in largescale learning experiences and programs. They are particularly appropriate in courses and programs with the following characteristics: Self-designed programs.  Students pursuing self-designed programs of study (Chapter 5) typically set their own learning goals. Portfolios are an ideal way to document student learning of one-of-a-kind sets of learning goals. 237

The Assessment Toolbox

Courses and programs with small numbers of students.  Assessment tools such as tests, surveys, papers, and capstone projects may not be very useful with small numbers of students, because differences in evidence of student learning may be due more to fluctuations in student characteristics than to differences in teaching/ learning processes. Portfolios are a better choice in these situations because they give a more thorough picture of student growth and development. Courses and programs that focus on developing thinking skills.  The papers, ­projects, and other student work typically included in portfolios can provide strong evidence of thinking skills such as writing, analysis, creativity, and evaluation, but they typically do not paint a comprehensive picture of students’ knowledge. If your goals include broad, comprehensive conceptual understanding (say, understanding the anatomy and physiology of the human body), traditional multiple-choice tests (Chapter 17) will help you assess those goals more effectively than portfolios. Courses and programs that focus on developing synthesis and metacognition skills.  As noted earlier, the self-reflection element of portfolios makes them ideal for developing these skills. A nursing program might, for example, use portfolios to help students learn to reflect on their development as caring, compassionate health care providers.

Planning a portfolio assessment If not carefully planned, portfolios can be a huge amount of work to assess; someone needs to sift through everything in them and make sense of it. A great deal of thought, time, and effort thus precedes successful portfolio implementation. Consider the questions in List 18.2 as you plan your portfolio assessment. List 18.2  Questions to Consider as You Plan a Portfolio Assessment ■■

■■

■■

■■

238

What are the learning goals of the portfolio? Students should learn something important from the process of compiling and reflecting on their portfolio. Decide why you are having students create portfolios and the learning goals that the portfolio should document (Chapter 4). Who are the portfolio’s audiences: Students, faculty, administrators, employers, accreditors, or some other group? Clarify the questions that each audience wants answered by the portfolios and the decisions that the student learning evidence from the portfolio assessment should inform (Chapter 6). How will portfolios be stored and accessed securely? A decade ago, portfolios were typically stored in cumbersome paper files, but today a variety of learning management systems and assessment information management systems (Chapter 11) can store them electronically. Who owns each portfolio? What happens to each portfolio when the student completes the course, graduates, or leaves the program before graduating?

■■ ■■

■■

■■

What might go into a portfolio?  List 18.3 suggests items that might be included in portfolios. While students should choose at least some items for their portfolios, faculty and staff may specify some items, and some may be chosen collaboratively by the student and a faculty or staff member.

List 18.3  Examples of Items That Might Be Included in a Portfolio ■■ ■■

■■

■■

■■

■■

Assembling Student Learning Portfolios

■■

How and when will students choose what to put in the portfolio? How and when will students reflect on their portfolios? If the portfolios will not be graded, what incentives will be offered (Chapter 21) to ensure that students put good effort into assembling and reflecting on their portfolios? How will the portfolios be assessed? What will be the key traits of successful portfolios articulated in the rubric used to assess them? Who will assess the portfolios? When?

CHAPTER 18

List 18.2  Continued

A table of contents, perhaps a checklist of potential items An introductory statement that will help those assessing the portfolio understand it better (Students might state, for example, why they enrolled in this course or program, their learning goals, their background before enrolling, or their career or life goals.) Examples of student work, which might include, depending on the portfolio’s learning goals: • Papers, projects, and other assignments (List 16.1 in Chapter 16) • Peer assessments • Attitude and interest surveys • Tests or logs of test scores • Statements of students’ learning goals and personal development goals • Notes from faculty, staff, and supervisors from observations, conferences, and interviews • Field experience supervisor assessments Evidence of learning processes, growth, and improvement such as: • Work completed early in the course or program (though these may not be available for students who transfer into a program after completing early coursework elsewhere) • Drafts • Work logs A reflection page in which students reflect on the overall contents of the completed portfolio (List 18.6), perhaps accompanied by reflections on each item in the portfolio (Exhibit 18.1) Faculty or staff member’s evaluative summary of the portfolio, perhaps in the form of a completed rubric, perhaps accompanied by brief faculty comments on or assessment of each item in the portfolio

Keep things manageable.  Imagine a class or program with 25 students, each of whom submits a portfolio with 10 examples of their work (plus reflections on each). That’s 250 items (plus reflections) to be reviewed – a daunting task. How can you 239

The Assessment Toolbox

keep the work of creating, managing, and assessing portfolios from ­overwhelming both students and faculty? Chapter 12 offers many suggestions for keeping assessments cost-effective. List 18.4 offers additional ideas for portfolios. List 18.4  Suggestions to Keep Portfolio Assessments Manageable ■■ ■■

■■

■■

Limit the number of items in each portfolio. Keep portfolio items short (no more than a few pages). Consider including only excerpts from lengthier items. Start small. Perhaps ask for only one or two items the first time you use portfolios, and then gradually increase the contents for subsequent cohorts of students. For course and program portfolios, use a score-as-you-go approach. As faculty grade each item that will go in the portfolio, ask them to attach their completed rubric. This saves the labor of a second review.

Provide guidelines to students.  Because preparing a portfolio may be a new ­experience for some students, it is important to give them clear written guidelines that explain the portfolio assignment. Chapter 16 offers suggestions on creating clear guidelines for any learning activity; List 18.5 offers some additional suggestions specific to portfolios.

List 18.5  Questions to Address in Portfolio Guidelines to Students ■■

■■

■■ ■■

■■

What are the learning goals of the portfolio? What will students learn by compiling it, beyond what they will learn by completing each individual item? What will be included in the portfolio? Which items are mandatory, and which do students choose? When will the choices be made? What are key deadlines? When do students review and update their portfolios? How will the portfolio be stored? When and how can students access it? What will happen to it at the end of the course or program or if students leave before the end? How and when will faculty and staff assess each item and the overall portfolio?

If students are developing a program portfolio, give them written guidelines for it as soon as they enter the program. Review the guidelines with them to emphasize the importance of the portfolio and answer any questions they might have. The guidelines should include periodic points at which students update their portfolios and review them with a faculty member. Because portfolios may identify the need for or interest in additional study on a particular topic or the need to further develop a particular skill, these reviews can help students plan their studies. Students with inadequate portfolios should have time to make them acceptable without delaying their graduation. 240

Answer each of the following questions with no more than one sentence each. 1. What type of assessment tool is this?

Assembling Student Learning Portfolios

Exhibit 18.1  A Reflection Sheet for Individual Portfolio Items from a Graduate Course on Assessment Methods

CHAPTER 18

Encourage thoughtful student reflection.  As noted earlier, one of the defining ­characteristics of portfolios is the opportunity for students to learn by reflecting ­holistically on their work. Students can be asked for brief reflections on each item in their portfolio (Exhibit 18.1), a longer reflective essay on the portfolio as a whole, or both. List 18.6 suggests prompts for self-reflection that might be particularly appropriate for portfolios; Chapter 20 discusses self-reflection further. To keep things manageable and encourage students to refine their thinking, limit the number of prompts you pose, and stipulate a maximum length for each response. Many reflection questions can be answered effectively in a single sentence.

2. In what course or unit that you teach will you use this assessment tool?

3. What learning goals does this tool assess?

4. Why did you choose this item for your portfolio? What does this item show me about you as a teacher?

5. What did you learn by creating this item?

6. Do you have any questions about this item?

My comments to you:

List 18.6  Examples of Prompts for Student Reflection on a Portfolio ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■

Which item is your best work? Why? Which item is your most important work? Why? Which item is your most satisfying work? Why? Which item is your most unsatisfying work? Why? In which item did you stretch yourself the most, taking the greatest risk in what you tried to do? List three things you learned by completing this portfolio. What does this portfolio say about you as an emerging professional or scholar in this discipline? What are your goals for continuing to learn about this discipline?

241

Needs improvement

Some items are inappropriate for the assessment setting (discipline, course level, etc.).

Some items are fuzzy.

Some items do not reflect an understanding of good practices.

Some items appear to have limited usefulness in understanding and improving learning.

Some items do not appear to be practical or transferable to other courses or programs.

The reflection provides a justification for selecting each portfolio item. But some justifications may not be based on principles of good practice or selfanalysis of areas for self-improvement.

Trait

Choice

Clarity

Understanding of good practices

Usefulness

Practicality and transferability

Reflection

assessment setting, and some are novel as well as effective approaches to assessment in the setting.

assessment setting.

The reflection provides a justification for selecting each portfolio item that is largely based on principles of good practice and self-analysis of areas for self-improvement.

All items appear to be practical but may not all be transferable to other courses or programs.

All items reflect at least an adequate understanding of good practices.

The reflection provides a justification for selecting each portfolio item that is based on principles of good practice and selfanalysis of areas for self-improvement. The reflection also synthesizes the justifications into an overall reflection of the student as an evolving assessment practitioner.

All items appear to be practical and transferable to other courses and programs.

All portfolio items appear to be useful in understanding and improving learning.

All items reflect a thorough understanding of good practices.

All items are clear and consistent.

All items are appropriate for the

All items are appropriate for the

All items are reasonably clear and consistent.

Outstanding

Satisfactory

Exhibit 18.2  A Rubric for Assessing Portfolios from a Graduate Course on Assessment Methods

Assembling Student Learning Portfolios

The purposes of your portfolio assignment should determine how you will assess the completed portfolios. Consider the following questions. • Are you assessing student progress in achieving major course or program learning goals? If so, use a rubric (Chapter 15) that lists those goals and provides criteria for acceptable performance. Exhibit 18.2 is an example of a rubric for a course portfolio. • Are you encouraging metacognition (Chapter 4)? If so, assess the portfolios in terms of the effort students put into self-reflection (Exhibit 18.2). • Are you encouraging students to improve their performance? If so, have students include drafts in their portfolios, and assess the portfolios in terms of improvement. • Are you encouraging risk-taking and creativity? If so, assess the portfolios in terms of the risks students took in their work, perhaps as evidenced in their written reflection on the portfolio.

CHAPTER 18

Assessing portfolios

Time to think, discuss, and practice 1. Do you have any experience with portfolios, either as a teacher or as a student? a. If you do, share what you learned from the experience. What was the best part of the experience? What was the least satisfying aspect of the experience? If you had the opportunity, what would you improve about the experience? b. If you haven’t had any experience with portfolios, share what excites you about portfolios and what makes you skeptical about them. 2. Identify someone in your group who is teaching a course or other learning experience that doesn’t use a portfolio, but for which a portfolio might be a worthwhile learning experience. Help that group member answer the following questions: a. What might be the learning goals of the portfolio? What might students learn through the process of assembling and reflecting on their portfolios? b. What should the portfolio include? (Remember to keep the portfolios small enough to be manageable!) c. What might be the criteria for assessing each completed portfolio?

243

Some valuable ideas you’ll find in this chapter

CHAPTER 19

Selecting Published Instruments Use a published instrument only when: Your learning goals match those covered by the instrument. The resulting evidence of student learning can be used to improve teaching and learning and inform planning and budgeting decisions. Tests that require significant reading may not be fair to students who are slow readers or for whom English is a second language.

245

The Assessment Toolbox

Jargon Alert! Published and Standardized Instruments

P

Published instruments are published by an organization and used by multiple colleges. Standardized instruments are published instruments that are administered and scored under comparable (standardized) conditions to ensure that scores are comparable across colleges and across time. All students receive the same instructions on completing a standardized instrument and, if there is a time limit for completing the instrument, it is enforced at all administration sites. If a standardized instrument asks students for writing samples that are scored using rubrics, the scorers are trained to ensure scoring consistency. Many published instruments are not standardized.

ublished instruments (see Jargon Alert), including tests, surveys, and rubrics, play a role in many assessment programs. This chapter discusses why published instruments might be useful, how to find potential instruments, and how to evaluate them to determine if they are appropriate for your situation. (Specific instruments are mentioned as examples, not endorsements.) Most published instruments relevant to student learning assessment in higher education fall into four basic categories. List 19.1 gives examples – not a comprehensive or endorsed list – of published instruments in each category.

Why use a published instrument? Published instruments can add important dimensions to an assessment effort. Published instruments get us out of the ivory tower.  Published instruments let us compare our students against those at other colleges, a potentially valuable perspective. Without peer benchmarking (Chapter 22), we may think our students are learning a great deal when in fact they are learning less than their peers. Or we may be initially disappointed by our evidence of student learning, only to learn that it is better than that at many peer colleges. The availability of peer information (or norms, discussed later in this chapter) is probably the strongest argument for using a published instrument. On the other hand, as mentioned in Chapter 22, some published instruments do not have good-quality peer information. Published instruments can have greater perceived legitimacy  than locally designed assessments. Board members, legislators, donors, and other external audiences are sometimes more impressed by student performance on a published examination than they are by the evidence of student learning from locally designed assessments. Published instruments can have good-quality questions.  Some publishers design questions in consultation with experts in the field and test them extensively before including them in the published instrument. Because faculty and staff are typically not assessment experts, local tests (Chapter 17) may be poorly designed and written. 246

Tests That Aim to Assess the Transferable Intellectual Skills and Competencies Typically Developed in General Education Curricula or Throughout Undergraduate Studies ■■ ETS Proficiency Profile (PP) (www.ets.org/proficiencyprofile) ■■ ACT Collegiate Assessment of Academic Proficiency (CAAP) (www.act.org/content/act/en/productsand-services/act-collegiate-assessment-of-academic-proficiency.html) ■■ Praxis Core Academic Skills for Educators Tests (www.ets.org/praxis/about/core/) ■■ CLA+ (cae.org/flagship-assessments-cla-cwra/cla/)

Selecting Published Instruments

Tests Designed to Assess the Subject-Specific Knowledge and Skills Acquired in Major Fields of Study ■■ Major Field Tests (MFTs) (www.ets.org/mft) ■■ GRE Subject Tests (www.ets.org/gre/subject) ■■ Praxis Subject Assessments (www.ets.org/praxis/about/subject) and Praxis Content Knowledge for Teaching Assessments (www.ets.org/praxis/about/ckt/) ■■ NCLEX for nurses (www.ncsbn.org/nclex.htm)

CHAPTER 19

List 19.1  Examples of Published Instruments for Student Learning Assessment in Higher Education

Rubrics ■■ VALUE rubrics (www.aacu.org/value-rubrics) ■■ American Council of Teacher of Foreign Languages (ACTFL) Proficiency Guidelines (actflproficiencyguidelines2012.org) Surveys of Student Experiences, Perceptions, and Attitudes ■■ National Survey of Student Engagement (NSSE) (nsse.indiana.edu) ■■ Community College Survey of Student Engagement (CCSSE) (www.ccsse.org) ■■ Student Satisfaction Inventory (www.ruffalonl.com/complete-enrollment-management/studentsuccess/student-satisfaction-assessment/student-satisfaction-inventory) ■■ CIRP Freshman Survey (heri.ucla.edu/cirp-freshman-survey/)

Published instruments sometimes have evidence of their quality.  Some publishers have made serious efforts to evaluate and document instrument quality. Because publishers may be able to invest more resources in time-consuming and expensive validation studies than an individual college can, their research may be more rigorous and extensive. Published instruments take less of our time.  It can take months of hard work by many people to draft, refine, and pre-test a locally designed test or survey before it is good enough to be usable. Published instruments can be researched, adopted, and implemented more quickly and with less work. Publishers often take care of ­summarizing and analyzing results.

247

The Assessment Toolbox

When is a locally designed assessment a better choice? In a survey by the National Institute for Learning Outcomes Assessment (NILOA), roughly 80 percent of colleges reported that “[published] test results were not usable for campus improvement efforts” (Jankowski, Ikenberry, Kinzie, Kuh, Shenoy, & Baker, 2012, p. 13). While published instruments have value under some circumstances, a locally designed assessment may be a more appropriate choice. Your college’s or program’s key learning goals may not match those assessed by available published instruments.  If your college defines critical thinking as the ability to think creatively and originally, for example, most published critical thinking tests will not give you the evidence of student learning that you need. If your chemistry program focuses on preparing students for entry-level work in medical laboratories, a chemistry test designed to assess preparation for graduate study isn’t appropriate. The degree of congruence between the learning goals of a college or program and those addressed by a published instrument is the most important consideration in deciding whether to use it. Local instruments are more likely to be relevant and therefore useful  because they’ve been custom-designed by faculty and staff. If your students are atypical, comparing them against peers is not helpful,  as discussed in Chapter 22. Suppose that Roslyn College has many students for whom English is a second language and who therefore do relatively poorly on typical writing assessments. Or suppose that many students at Woodwyn College never studied algebra in high school, leaving them far behind typical college students in quantitative skills. In these cases, a locally designed assessment may be more useful. A published instrument may not be practical.  Many published tests and surveys are add-on assessments (Chapter 21), and it can be extraordinarily difficult to motivate students not only to participate in such assessments but to give them serious thought and effort. Published instruments may also not be practical to administer in terms of time and logistical requirements. The cost of a published instrument may outweigh the benefits . . . or simply be unaffordable.  A single administration of some published instruments can cost thousands of dollars. Is the information gained from the instrument worth this investment? Some colleges simply don’t have this kind of hard cash available.

248

Selecting Published Instruments

If you think that a published instrument might be a useful component of your assessment program, collaborate with faculty, staff, and students in the review and decision process (Chapter 13). Begin by clarifying the decisions that the instrument should help inform (Chapter 6). There’s no point in surveying students about experiences with high-impact practices (Chapter 5), for example, if college leaders aren’t interested in improving or changing those practices. Questions to consider include the following: • What learning goals are you trying to assess? • What information do you need? • How will you use the resulting evidence of student learning? • How might the evidence be used to improve student learning or success? Once purpose is clarified, use the resources in List 19.2 to identify potential published instruments. Then try to obtain additional information on the instruments you’ve identified, ideally the information in List 19.3.

CHAPTER 19

Identifying potential published instruments

List 19.2  Resources for Identifying Potential Published Instruments ■■

■■

■■

■■

■■ ■■

The Mental Measurements Yearbook (MMY) (Carlson, Geisinger, & Jonson, 2017), available in many college libraries, is an excellent source of information on instruments assessing all kinds of mental traits. It includes instrument information, contact information, research citations, and critical reviews by scholars. Tests in Print (TIP) (Anderson, Schlueter, Carlson, & Geisinger, 2016), also available in many college libraries, is another excellent source for identifying potential instruments. It provides information on more instruments than the MMY and is published more frequently, but it lacks MMY’s critical reviews and citations. The Test Collection at ETS (www.ets.org/test_link/about) is a database of more than 25,000 published instruments. Measuring Quality in Higher Education: An Inventory of Instruments, Tools and Resources (apps.airweb.org/surveys/) is maintained by the Association for Institutional Research. Internet search engines such as Google can also help identify potential instruments. Professional journals, conferences, and colleagues can be valuable sources of information on potential instruments. But keep in mind that just because a published instrument is used at other colleges, doesn’t mean that it’s right for yours.

Evaluating potential published instruments After you have obtained information on each potential instrument, review the information in terms of the following questions. 249

The Assessment Toolbox

List 19.3  Useful Information on Potential Published Instruments ■■

■■

■■

■■

■■

■■

A copy of the instrument itself, or at least examples of items. Examining individual items helps determine if they are clear and measure what they purport to measure. Because of test security or copyright concerns, some publishers will not let you view the full instrument before purchasing it. Those publishers should let you see enough examples of items – perhaps on an outdated edition of the instrument – to make an informed decision about whether to adopt the instrument. If the publisher is unwilling to do this, move on to another instrument (Banta, 2008). Instructions for administering the instrument, which help determine if the instrument can be feasibly administered in your situation. Ordering information and prices, which help determine if the instrument is financially feasible. There may be separate charges for purchasing copies of the instrument, getting results processed, and receiving the resulting evidence of student learning in the format you need. How completed instruments are scored. Some instruments can be self-scored; others must be sent to the publisher for processing, tabulation, and analysis; and still others give you the option of either approach. How the results are reported to you and prices for each option. Some publishers offer a variety of report options and charge higher fees for more detailed reports. Some offer full data sets, allowing you to analyze the evidence yourself so you can, for example, compare residential and commuter students or look only at business majors. Technical information. This includes information on how the instrument was designed, how it was normed, and evidence of its validity and reliability. This information may be available on the publisher’s website, in a manual, or as a series of reports.

Does the instrument’s stated philosophy match yours? The publisher should have a statement on the instrument’s philosophy and purpose, and that philosophy and purpose should match those of your college or program. Some faculty believe, for example, that general education courses in mathematics should emphasize the development of analytical reasoning skills through the study of pure mathematics, while others believe that such courses should emphasize the development of skills for solving real-life problems. The quantitative reasoning test you choose should match your faculty’s philosophy.

Do the instrument’s learning goals and content match what you emphasize? Test publishers should provide information on the specific kinds of knowledge and skills assessed by their test. Some publishers provide a test blueprint or table 250

Ask the following questions to determine if the evidence of student learning provided by the instrument will be useful to you and your colleagues. If the evidence won’t be useful, there’s little point in using the instrument. How does the publisher recommend using the results?  The publisher should offer suggested uses and inappropriate uses of results. A quantitative reasoning test designed to assess students’ everyday problem-solving skills would be inappropriate, for example, for deciding whether to place students into calculus. Make sure that the publisher considers your planned uses of the results to be appropriate.

Selecting Published Instruments

Will the resulting evidence of student learning be useful?

CHAPTER 19

of specifications (Chapter 17) that lists the skills and content covered by the instrument. Review this information carefully and see how well the instrument’s learning goals and content correspond to your learning goals and curriculum. A writing test that focuses on grammar, for example, would be inappropriate if your writing courses focus on developing well-reasoned arguments. If possible, examine some actual items on the instrument to verify its focus.

Does the publisher provide just one or two global scores or more detailed evidence of student learning?  While external audiences interested in accountability want to see a quick snapshot of student learning, it’s prudent to invest in an instrument that also provides the detailed feedback needed to understand and improve student learning. A critical thinking test that provides only a single critical thinking score for each student, for example, won’t help you identify strengths and weaknesses in students’ thinking skills nor how to improve them. A useful published instrument yields sub-scores for specific skills and competencies aligned with your key learning goals, with sufficiently detailed feedback to tell you what students are learning relatively well and any learning shortcomings that should be addressed. How long is the instrument?  Some publishers, recognizing that students do not want to spend a great deal of time on tests and surveys, design relatively short instruments. While these instruments can give good information on how students are doing in general, short instruments are often too imprecise to make sound decisions about individual students or even individual programs. A brief writing test may provide useful information on the overall effectiveness of a writing program but too little information to determine which students need remediation. Is the instrument designed to be used with all students or just a small sample?  Some published instruments are very expensive and cumbersome to administer – so much so that the publishers recommend administering them to 251

The Assessment Toolbox

samples of students (Chapter 12) rather than to all students. Information from a sample of students can yield information on how students are doing in general. But small samples do not yield enough information to let us examine subgroups of students, such as those in a particular program. Looking at these kinds of breakouts is often essential to understanding how we might improve student learning.

Does the instrument have convincing evidence of its quality? A good-quality published instrument meets the traits of effective assessment practices discussed in Chapter 3, especially the following: Alignment with your learning goals.  As discussed earlier, the most important characteristic of a published instrument is how well it aligns with your key learning goals. An appropriate balance among evidence of validity, reliability, and usefulness  as discussed and in Chapter 3. It’s a lot easier to measure reliability than validity. Some publishers tout reliability evidence as evidence of their instrument’s “quality,” glossing over a lack of substantive evidence of content validity or consequential validity. Ask some tough questions. What is this test or survey supposed to be measuring, in concrete terms? What is the evidence that this test or survey really provides meaningful, useful information on those things? For example, how do the publishers of a critical thinking test define the kinds of critical thinking skills it purports to assess? What evidence shows that it truly assesses those kinds of critical thinking? What evidence shows that it assesses skills developed in college and not, say, aptitude or skills developed over a lifetime? Some publishers may provide research evidence of the instrument’s validity. This information can be quite technical, and you may wish to ask a psychology or education faculty member to help interpret it. Examples of validity evidence that a test publisher might provide include the following. Look at the quality of the research studies; some may be conducted on very small or unrepresentative groups of students. • Correlations of instrument scores with scores on other similar instruments • Correlations of instrument scores with grades in appropriate courses • Correlations among sub-scores (Scores on related sub-tests, such as reading and writing, should have some degree of correlation, while scores on dissimilar sub-tests, such as writing and quantitative skills, should be less strongly correlated.) • Increases in scores after participating in an appropriate program (Graduating engineering students, for example, should score higher on a test of engineering knowledge and skills than entering students.)

252

Reading skills are essential for success in college and beyond, so it may be appropriate that a test include considerable reading material to which students must respond. A test on literature, for example, should ask students to read and respond to literature passages. If a test’s reading level is too difficult, however, it becomes more a general reading comprehension test than a test of its purported subject. If students do poorly on a sociology test with a difficult reading level, for example, we can’t tell if they did poorly because they don’t understand sociology concepts or if they understand sociology concepts but are simply poor readers. Highly verbal tests are particularly likely to underrepresent the achievements of students for whom English is a second language and those from disadvantaged backgrounds. Tests that minimize text and maximize the use of charts, diagrams, pictures, numbers, and other nonverbal material may be fairer to these students but may give a less complete picture of their performance. If you are looking for a published survey rather than a test, it is equally important to consider the reading skills required by the instrument. If the survey’s reading level is too difficult, some students may misinterpret questions and give erroneous responses.

Jargon Alert! Norms Published instrument norms are the results – averages, percentile distributions, and the like – for groups against which your results may be compared. An instrument publisher may provide norms, for example, for all U.S. college students, for students at community colleges, and for Latino students.

Selecting Published Instruments

How much does the instrument emphasize reading skills?

CHAPTER 19

Other evidence of quality.  A good-quality published instrument also has evidence of the following traits of effective assessment practices discussed in Chapter 3. • Content and instructions that are clear to faculty, staff, and students • Generalizability • Appropriate range of outcome levels • Guards against unintended bias A good-quality instrument is also reasonably current. The publisher should periodically review and update it to ensure that it reflects current research, theories, and practices and does not have obsolete material.

Does the instrument have adequate and appropriate norms? If you plan to use a published instrument to compare students against those in peer programs or colleges (Chapter 22), its value depends on the quality and completeness of the groups used to establish the instrument’s norms (see Jargon Alert). An instrument 253

The Assessment Toolbox

normed largely on students at selective liberal arts colleges would be inappropriate, for example, for assessing the performance of students at an open-admission community college. Norms for any published instrument are based only on students from those colleges that have agreed to use the instrument . . . and only on those students who have agreed to participate. So norms are not based on a true random sample of – and may not be representative of – students enrolled in all peer colleges and universities (Baglin, 1981). An instrument’s technical information should include information on how its norms were developed. Find out the following: How many colleges and students are included in the norms?  Obviously you can have greater confidence in norms developed from thousands of students at dozens of colleges than in norms developed from a few hundred students at a handful of colleges. Do the norms represent your college?  Do the norms include an adequate number of students from colleges with missions and characteristics similar to yours? Do the norms represent your students?  Do the norms include an adequate number of students similar to those at your college? If your college attracts commuters, students of color, or students in particular majors, are such students well represented in the norms? When did the norming take place?  Students today are different from students 10 years ago, so norms created then and not updated since may not be useful.

Is the instrument practical to administer? Consider the following: Incentives.  Chapter 21 discusses the significant challenges of motivating students to participate meaningfully in add-on assessments such as published tests that are not required for certification or licensure. As that chapter emphasizes, the major challenge with most add-on assessments – indeed, their major drawback – is convincing students not only to participate in them but also to give the assessment tasks serious thought and effort. Chapter 21 offers suggestions on motivating students to participate in add-on assessments.

254

What if there is little or no information on a potential instrument? Because interest in assessing student learning in higher education is relatively new, many published instruments aimed at the higher education market have been developed only recently, and their publishers may not yet have answers to all the questions discussed in this chapter. Validity studies may be “sparse to non-existent” and norms may be “based only on the small samples of institutions” (Banta, 2008, p. 4). Lack of information on an instrument does not automatically mean that it is poor quality and should be removed from consideration. It simply means that the instrument is unproven, and an unproven instrument may nonetheless be a useful addition to an assessment program. Some published surveys, for example, have few or no published validation studies, but faculty and staff have still found them helpful. Locally developed instruments usually have little systematic evidence of their quality and can be nonetheless very useful. But if you are considering an instrument whose quality is largely undocumented, let the buyer beware. You and your colleagues must rely on your own appraisals rather than the work of others to determine if the instrument is of sufficient quality to be useful. Review the instrument itself and whatever information you do have about it, and ask the questions in List 19.4.

Selecting Published Instruments

Administration requirements.  Instruments administered online may be inappropriate, for example, if a significant number of your students do not have easy access to a computer or lack confidence in using one.

CHAPTER 19

Time required to administer the instrument.  Instruments that take a long time to finish may be difficult for students to complete.

List 19.4  Questions to Ask About Instruments with Little Published Information ■■

■■

■■ ■■

■■

Overall, does the instrument make sense? Does it appear to measure what it purports to measure? Does the instrument look like it would give us useful information? Does its content appear to match our learning goals? Are the individual items clearly written? Will students have any difficulty answering them? Does the instrument appear to be unbiased and fair? Does it have any stereotyping or offensive material, for example? What is the potential for harm if it turns out that the instrument is of poor quality and does not give accurate information? Any assessment instrument – validated or not – should never be the sole basis for any important decision, as discussed in Chapter 26. It should simply add to the picture of student learning and growth that you are drawing from multiple sources.

255

The Assessment Toolbox

You can also try out the instrument on a small scale, essentially conducting your own validation study. If you are considering a published writing test, for example, you could give the test to some students completing a writing course and compare test scores against course grades or rubric scores on essays they’ve written in class. If the scores correlate reasonably well, you may decide that the test is a worthwhile addition to your assessment program.

A note on the VALUE rubrics The VALUE rubrics developed by the Association of American Colleges & Universities (www.aacu.org/value-rubrics) differ from other published instruments in that they are “meant to be adapted in order to reflect the individual mission, program mix, and student demographics of the institutions where they are used” (Rhodes & Finley, 2013). So don’t feel obliged to use them as written. Use them as a kickoff for discussion. If you decide to use them, feel free to adapt them so they align with your learning goals. There’s a good bit of overlap among the rubrics; you may find it helpful to merge some traits from several of the rubrics into a new rubric. See Chapter 15 for more information on developing rubrics.

Is a published instrument right for you? Chapter 3 emphasizes that any assessment effort should include multiple assessments of student learning because all assessment tools and strategies – including published instruments – are inherently imperfect. A combination of locally designed and published assessments generally provides a more complete picture of student learning than either type of assessment alone. Just as homebuyers quickly learn that the perfect house does not exist (at least not within their budget), neither does the perfect published instrument. No publisher will probably ever be able to answer all the questions posed in this chapter to the complete satisfaction of you and your colleagues. List 19.5 summarizes fundamental questions to ask yourself about any potential published instrument. If your answers to those questions are a resounding No, the instrument that you are considering does not meet your needs and should not be considered further. If you answer No to these questions for all the instruments you’ve identified and reviewed, turn your attention to designing and implementing a local assessment strategy.

256

■■ ■■

■■ ■■

■■

For More Information For a more thorough discussion of published tests, see Educational Testing and Measurement: Classroom Application and Practice (Kubiszyn & Borich, 2013). For more information on the VALUE rubrics, see On Solid Ground: VALUE Report 2017 (Association of American Colleges & Universities, n.d.).

Selecting Published Instruments

■■

Does this instrument assess what we think is important? Are we reasonably convinced that the instrument indeed measures what it purports to measure? Will students have compelling reasons to participate in this assessment and to take it seriously? Will the resulting evidence help us understand what and/or why students are learning? Is the instrument part of a multiple-measures approach to assessing student learning? Will we be collecting other types of evidence to corroborate what this evidence tells us? Will the evidence help us decide how to improve students’ learning experiences?

CHAPTER 19

List 19.5  Deciding If a Published Instrument Is Right for You

Time to think, discuss, and practice 1. One of Kingsway College’s learning goals is that graduating students write effectively. Brainstorm three arguments for assessing this goal using a locally designed assessment strategy and three arguments for assessing it using a published instrument. 2. Brandywine Community College aims for all its students to graduate with “quantitative problem-solving skills.” a. Ask a member of your group to play the role of the college dean and answer any questions your group has about this learning goal or how the college’s curricula are designed to help students achieve it. b. Use the resources described in this chapter to identify up to three possible published instruments for assessing quantitative problem-solving skills. c. Find whatever information you can on each instrument and evaluate it. Which instrument would you recommend to the college, if any? Why?

257

Some valuable ideas you’ll find in this chapter Qualitative assessments are underused and underappreciated but help uncover problems – and solutions – that can’t be found through quantitative assessments alone.

CHAPTER 20

Other Assessment Tools

Reflective writing is valuable as a learning strategy as well as an assessment strategy . . . but think twice about grading it. Ecosystem rating scales can provide helpful insight on student achievement of some learning goals.

259

The Assessment Toolbox

R

ubrics, prompts, tests, and portfolios constitute the vast majority of assessment tools used in higher education today. But there are a number of other assessment tools that can be helpful, including reflective writing, rating scales, surveys, interviews, and focus groups. Many of these are particularly good choices for co-curricular experiences. Faculty have always used informal, sporadic, anecdotal observations and input as feedback on student learning. The tools in this chapter differ in that they are structured, systematic, and consistent. Hearing a few students exclaim about the insight they gained from a class is exciting and rewarding, but it is not truly useful, meaningful assessment unless you ask all your students, or a representative sample of them, about insight on a regular basis.

Reflective writing An intriguing development in education has been the increasing value placed on reflection: Encouraging students to reflect on what, how, and why they have learned. There are several reasons for this development: Reflective writing helps students learn.  The best assessments are learning experiences for students as well as opportunities for us to see what they have learned, and this is where reflection shines as an assessment strategy (Ash & Clayton, 2009). Reflection helps students learn by encouraging two increasingly important skills: Synthesis and metacognition (Chapter 4). Asking students to reflect at the end of a course, program, or other learning experience on the key lessons they’ve learned, the learning strategies they used, and how the pieces fit together can help them develop these skills. Reflective writing is a great tool for assessing many attitudes and values  that students can fake on tests, surveys, and graded papers, as discussed in Chapter 21. Reflective writing balances quantitative assessments with qualitative insight.  Quantitative assessments (Chapter 17) tell us what students have learned; qualitative assessments (see Jargon Alert) such as reflective writing Jargon Alert! can tell us why. They give us fresh insight, Qualitative Assessments allow us to explore possibilities that we Qualitative assessments use flexible, naturalistic haven’t considered, and help us uncover methods and are usually analyzed by looking for recurring patterns and themes. Reflective problems – and solutions – that we writing, online discussion threads, and structured wouldn’t find through quantitative assessobservation guides (Chapter 15) are examples of qualitative assessments. ments alone. 260

The easiest, fastest way to encourage reflection is to give students just one or two thought-provoking questions and ask them to write no more than a sentence in response. The minute paper (Weaver & Cotrell, 1985) is one of the best ways to do this. It’s called a minute paper because students take no more than one minute to complete it. Minute papers usually ask students to answer just one or two questions along the following lines: • What was the most important thing you learned during this learning experience? • What important question remains unanswered? To encourage honest feedback, students are often asked to complete minute papers anonymously. But consider giving students the option of adding their name and email address if they would like a response. This lets faculty and staff provide individual assistance to students and makes minute papers a stronger learning opportunity.

Other Assessment Tools

Minute papers

CHAPTER 20

Reflective writing can add a human dimension to assessment reports,  enhancing dry tables and graphs with living voices.

Other short questions and prompts While minute paper questions are intended to elicit very brief replies, sometimes it’s helpful to ask students to write a bit more – perhaps a couple of sentences or a paragraph. One of the great advantages of brief reflective writing is that it can yield useful information quickly and easily. You can use minute papers and other brief reflections literally in your very next class or co-curricular experience. They are a great choice for very short learning experiences such as tutoring sessions. Analyzing responses can also be very easy: Simply read through the responses (or, if you have hundreds, a sample of them) for a quick sense of the most common themes. You can then address any areas of concern in the next class or meeting. (Chapter 24 discusses more formal ways to analyze qualitative evidence of student learning.) List 20.1 presents questions and prompts (Chapter 16) that you might use to ask students to reflect. It’s not an exhaustive list; use these prompts as inspiration for developing your own. To save your students time in responding (and your own time in reviewing their responses), ask no more than three questions at one time. 261

The Assessment Toolbox

List 20.1  Examples of Prompts for Reflection on a Learning Experience ■■ ■■ ■■ ■■ ■■

■■ ■■

■■

■■

■■

■■ ■■

■■

■■

■■

■■ ■■

■■

■■ ■■

■■

■■ ■■ ■■ ■■

262

How do you feel about writing/teaching/biology/sociology? What makes a person a good writer/teacher/biologist/sociologist? How do you feel about yourself as a writer/teacher/biologist/sociologist? What are your strengths as a writer/teacher/biologist/sociologist? Your weaknesses? Describe something major that you’ve learned about yourself from this assignment/course/program/ learning experience. What was your favorite aspect of this course/program/learning experience? Why? What might you say if you have a chance to speak to your friends and family about this course/ program/learning experience? What suggestions would you give other students on ways to get the most out of this assignment/ course/program/learning experience? What was the one most useful or meaningful thing you learned in this assignment/course/program/ learning experience? What did you learn about writing/research/leadership/other skill from this assignment/course/ program/learning experience? What was your biggest achievement in this course/program? What one assignment for this course/program was your best work? What makes it your best work? What did you learn by creating it? What does it say about you as a writer/teacher/biologist/ sociologist? If you could change any one of the assignments you did for this course/program, which one would it be? What would you change about it? What goals did you set for yourself in this assignment/course/program/learning experience? How well did you accomplish them? What strategies did you use to learn the material in this assignment/course/program? Which were most effective? Why? What risks did you take in this assignment/course/program/learning experience? If you were to start this assignment/course/program/learning experience over, what would you do differently next time? What problems did you encounter in this assignment/course/program/learning experience? How did you solve them? In what ways have you improved as a writer/teacher/biologist/sociologist? List three ways you think you have grown or developed because of this assignment/course/program/ learning experience. What have you learned in this assignment/course/program that will help you continue to grow as a writer/teacher/biologist/sociologist? What did you learn from this assignment/course/program that is not reflected in your work? What would you like to learn further about this subject/discipline? Why? In what area would you like to continue to strengthen your knowledge or skills? Write one goal for next semester/year and tell how you plan to reach it.

Table 20.1:  Student Definitions of Leadership Before and After Participating in a Leadership Development Program Initial Definition of Leadership

Later Definition of Leadership

The ability to give and take orders and being able to take charge of a large group of people.

I have learned that leadership is not a one-person show. To be a good leader, you must have the respect from your committee and you must be able to communicate.

Other Assessment Tools

Sometimes it can be helpful to ask students to reflect at both the beginning and end of a course, program, or co-curricular experience and then compare their responses to give you a sense of their growth and development. You might ask students to self-assess their skills and attitudes, to define a key concept (“What is poetry?”), or to explain why a subject or discipline is important. Table 20.1 gives examples of student definitions of leadership before and after participating in a leadership development co-curricular program. The pairs of responses powerfully convey what students learned in the program.

CHAPTER 20

Before-and-after reflection

The presence of a strong, task-oriented and social- Leadership isn’t as easy as it looks! Leadership oriented yet compromising force or person. takes a lot of hard work and there are ups and downs to the position of a leader. My definition has changed to include the importance of diverse people. Leadership is a responsibility or a skill/trait one possesses that makes a person stand out above everyone else.

Leadership is a collective process. You need leaders and followers to make an event/ organization successful.

Leadership involves taking control in an organizational way. A leader must know how to dictate responsibility as well as work with others to achieve a goal.

Leadership has a lot to do with confidence. Most of the confidence lies within yourself but you also have to have confidence in the people you’re working with. My definition of leadership has changed in the sense that I feel like it is more delegating and following up with your delegations than actually taking a lot of work upon yourself.

Leadership is an important element of life that can only be fulfilled by individuals possessing the motivation, insight, and communication skills to fulfill the mission, goals, and objectives of an organization.

Leadership is ever changing. Now I think leadership is defined by the people being led.

Adapted by permission from responses to prompts by Tess Shier, Coordinator for Campus Programs, Office of Student Activities, Towson University

263

The Assessment Toolbox

Longer self-reflection assignments While short prompts for reflective writing are excellent ways to gain quick and easy insight into student learning, longer assignments reflecting on learning experiences throughout a course or program can help students synthesize what they’ve learned and clarify their attitudes, values, and learning strategies. Exhibit 20.1 is an example of such a prompt. Exhibit 20.1  A Prompt for a Reflective Paper on an Internship Write a reflective paper on your internship experience. The purpose of this assignment is for you to develop and demonstrate the ability to: • • • • • •

Set explicit goals for your own work; Monitor your progress toward meeting goals; Seek out and use feedback from others; Evaluate your learning and performance; Assess personal strengths and weaknesses; and Communicate clearly and professionally through writing.

Your reflective paper should be five pages long, double-spaced, 10- or 12-point font with oneinch margins. Use the objectives listed above as a basis for your reflection and answer the following questions: • • • •

How valuable was the internship experience to you? What specific experiences do you think made you stretch and grow as a professional? What could you have done differently during your internship to improve the learning experience? Would you recommend this placement to others? Why or why not?

Adapted with permission from a prompt created by Drs. Sharon B. Buchbinder and Donna M. Cox, Health Care Management Program, Towson University

Writing meaningful reflective writing prompts Prompts for reflective writing should follow the guidelines for crafting prompts in Chapter 16 as well as the suggestions for assessing attitudes and values in Chapter 21. In particular, prompts for reflective writing should be phrased to elicit honest answers, with no obviously right or wrong answer.

Can and should student reflections be graded? The answer to this question isn’t straightforward. On one hand, one of the purposes of reflective writing is to elicit honest, truthful thoughts from students. Grading student reflections may stifle that honesty, encouraging students to write only what they think we want to hear instead of what they truly think and feel. Indeed, in 264

CHAPTER 20 Other Assessment Tools

some circumstances simply asking students to put their names on their reflections – even if the reflections aren’t graded – may dissuade them from providing trustworthy information. On the other hand, grades can also be important incentives for students to give careful thought to their reflections. Depending on the learning goals of the assignment, reflective writing can be graded in terms of the underlying elements necessary for valid reflection such as: • Evidence of thinking skills such as analysis and synthesis • Effort in completing the assignment thoughtfully • Meaningfulness of personal responses and opinions • Connecting one’s opinions to what has been learned in the course, program, or co-curricular experience Both of these positions have merit, and the best approach will depend on your students and what you are asking them to do. Generally short reflections such as minute papers may be submitted anonymously. And some beginning students may be more comfortable reflecting anonymously until they develop increased confidence in their ability to reflect. But some students may need the incentive of credit toward a grade in order to complete a longer written reflection thoughtfully. Even if you don’t grade the assignment, students deserve feedback on how well they are achieving its learning goals. A simple rubric (Chapter 15) can provide such feedback.

Rating scales Rating scales (Table 20.2) can use any descriptors that span a spectrum. Participants can be asked to make ratings on scales ranging, for example, from Excellent to Poor, Frequently to Never, Strongly Approve to Strongly Disapprove, or Table 20.2:  Examples of Rating Scales Type of Rating Scale

What It Assesses

Rating scale rubrics (Chapter 15)

Student achievement of various traits of key Faculty, staff, field learning goals as demonstrated in a student work experience supervisors, product or performance and student peers

Self-ratings of achievement

Student achievement of various traits of key learning goals

Students

Ratings of satisfaction

Satisfaction with various aspects of a learning experience

Students

Ratings of frequency

How frequently a student has engaged in an activity or used a service

Students

Who Completes It

265

The Assessment Toolbox

Much Better to Much Worse. Letter grades (A, B, C, D, and E or F) may be a good choice when asking about perceived quality or satisfaction because they are so widely understood. Rating scales are popular assessment tools because they are efficient (a great deal of information can be obtained quickly and compactly), the results can be easily tallied and summarized, and they permit comparisons among answers within the rating scale. Rating scales have two major shortcomings, however. First, as mentioned in Chapter 15, rating scale levels may be interpreted inconsistently. Second, rating scales are transparent: “right” and “wrong” answers are self-evident. When students use rating scales, they may be tempted to give you the answers they think you want to hear rather than their honest appraisals. Because of these shortcomings, rating scales – other than rating scale rubrics used by faculty, staff, and field experience supervisors – are indirect and therefore insufficient evidence of student learning (Chapter 3). Use rating scales to assess student learning only when they add useful insight beyond direct evidence of student learning (Chapter 3) or when direct evidence isn’t possible (Chapter 21). Make sure the rating scale will yield information related to key learning goals. For more tips on constructing an effective rating scale, follow the suggestions for rubrics in Chapter 15.

Likert rating scales Likert (pronounced Lick-ert) rating scales are probably the best-known rating scales, characterized by the descriptors Strongly Agree, Agree, Disagree, and Strongly Disagree. Exhibit 20.2 is an example of a Likert scale for graduating students. If you use a Likert scale, include some statements presenting a negative or opposing view. This will help prevent the yeasayer/naysayer effect, in which some people with generally positive feelings toward your topic may check all the Strongly Agree responses without reading each item and those with generally negative feelings may do the opposite. Should a Likert scale include a Neutral or Neither Agree Nor Disagree column in the center? It depends on whether it’s appropriate to force your participants to agree or disagree. If you are asking students whether “The instructor spoke clearly,” it might be argued that the instructor either spoke clearly or didn’t, and every student should be able to either agree or disagree with this statement.

266

Please tell us how well each CIS program learning goal was met for you personally. For each objective, please check the one response that best reflects your feelings, opinions, and/or experiences.

A = Agree U = Unsure D = Disagree SD = Strongly Disagree SA A □ □

U D SD □ □ □

a. I feel proficient with IS tools, IS analysis and design, and the role of IS in organizations.

□

□

□ □ □

b. I feel proficient in relevant aspects of quantitative analysis.

□

□

□

□

□ □ □ □ □ □

a. I understand calculus and statistics.

□

□

b. I understand basic principles of accounting and economics.

□

□

3. I am prepared to be an informed and involved member of my community and □ a responsible IS professional. □ a. I am familiar with basic concepts and contemporary issues in the social

□

1. I have a thorough grounding in key IS principles and practices and the quantitative analysis principles that underpin them.

2. I understand the quantitative and business principles that underlie IS principles and practices.

Other Assessment Tools

SA = Strongly Agree

CHAPTER 20

Exhibit 20.2  An Exit Survey for Students Completing a BS in Computer Information Systems

□ □ □ □ □ □ □ □ □

□

□ □ □

□

□

□ □ □

□

□

a. I can work effectively in teams.

□

□

b. I can communicate effectively.

□

□

□ □ □ □ □ □

□

□

sciences and humanities. b. I understand global, social, professional, and ethical considerations related to IS. 4. I have appropriate social and organizational skills.

5. I can acquire new knowledge in IS and engage in lifelong learning.

□ □ □ □ □ □

Adapted with permission from a survey developed by the faculty of the Department of Computer and Information Sciences, Towson University

Ecosystem rating scales Ecosystem rating scales (see Jargon Alert) can ask for information such as the following: • Student satisfaction with various aspects of a program and the relative importance of each aspect

267

The Assessment Toolbox

Jargon Alert! Ecosystem Rating Scales Ecosystem rating scales (also known as goal attainment scaling or gap analysis) ask for two ratings, with the second rating giving information on the context or environment in which the first rating is made and thus helping to interpret the first rating.

• S tudent satisfaction with various aspects of a program and how much the student used or participated in each aspect • Student self-ratings of various learning goals or traits and how much the learning experience helped the student achieve each goal (Exhibit 20.3)

Exhibit 20.3  A Self-Assessment of Library Skills, with a (Fictitious) Student’s Responses For each of the following skills, please make TWO ratings: 1. How strong are your skills in each of the following areas? 2. How much have the library session(s) in this course helped you to develop each of the following skills? Your skill level

Low 1. Identify potential sources of information related to this course. 2. Find information that’s appropriate for and relevant to this field of study.

How much library session(s) have helped

High Low

High

1 2 3 4 5 1 2 3 4 5 □ □ □ □ ☒ □ □ □ □ ☒ □ □ □ □ ☒ ☒ □ □ □ □

3. Critically evaluate information that you find, including its ☒ □ □ □ □ □ □ □ □ ☒ accuracy, authority, coverage, objectivity, and currency. ☒ □ □ □ □ ☒ □ □ □ □ 4. Cite the work of others accurately and ethically. Adapted with permission from a rubric developed by the Library faculty of Towson University

268

Ecosystem rating scales can add useful insight, as their second rating greatly improves interpretation of the first. Consider the responses in Exhibit 20.3. • The student rated her skill in identifying potential sources of information high and gave the library session a high rating in helping her develop this skill, suggesting that the student learned a great deal in the session. • The student rated her skill in finding information high but gave the library session a low rating in helping her develop this skill, suggesting that she may have learned how to do this elsewhere. • The student rated her skill in evaluating information low but gave the library session a high rating in helping her develop this skill. This suggests that the student recognizes she needs more time and work beyond a single library session to develop this challenging skill.

Surveys

Surveys (see Jargon Alert) can be useful Surveys are systematic efforts to collect information assessment tools when you want to learn about people by asking them to respond to specific about the experiences, views, or plans of questions about their backgrounds, experiences, plans, opinions, and attitudes. Surveys can be rating a large number of students. Surveys are scales, requests for factual information, prompts for increasingly conducted online, although reflection and opinion, or a combination of these. they can also be administered on paper (in settings such as a class or meeting) or via telephone interviews. Published surveys are discussed in Chapter 19, and technologies supporting surveys and other assessments are discussed in Chapter 11. Surveys have two major shortcomings. First, because information is selfreported rather than observed, surveys are indirect and therefore insufficient evidence of student learning (Chapter 3). The other problem is survey fatigue (Jankowski, Ikenberry, Kinzie, Kuh, Shenoy, & Baker, 2012; McCormick, Gonyea, & Kinzie, 2013): People today can feel deluged with surveys and may therefore be disinclined to respond. Chapter 21 discusses these issues further. Surveys

Jargon Alert!

Other Assessment Tools

Jargon Alert!

CHAPTER 20

• The student rated her skill in citing the work of others low and gave the library session a low rating in helping her develop this skill. This suggests that the library session might be improved in this regard.

Interviews and focus groups

Interviews and focus groups (see Jargon Alert) can be used to ask participants to reflect on themselves and their experiences and also to collect some information on their behaviors. Because interview and focus group responses can be wide-ranging and often do not yield quantitative evidence, qualitative summaries and analyses (Chapters 23 and 24) are typically used to look for response patterns and themes. A major drawback of interviews and focus groups is motivating people to participate and to take the interview or focus group seriously, providing thoughtful responses. Chapter 21 discusses this further. Another drawback is that interviews and focus groups are typically time-consuming and are therefore conducted with relatively small numbers of students. They are consequently not the best choice for making inferences about large groups of students.

Interviews and Focus Groups

Interviews consist of open-ended questions and sometimes simple rating scales asked by an interviewer via telephone or in person. Focus groups are in-person interviews of small groups of people.

269

The Assessment Toolbox

Interviews and focus groups can nonetheless have an important role in an assessment program, as they can help with the following: • Planning an assessment by identifying goals, issues, and questions that your assessment efforts should address • Corroborating and perhaps strengthening the credibility of evidence of student learning from quantitative assessments • Understanding evidence of student learning from quantitative assessments (If your students’ academic performance is disappointing, for example, focus groups can help you understand why.) List 20.2 (Suskie, 1996) offers suggestions for conducting effective focus groups and interviews. List 20.2  Tips for Focus Groups and Interviews ■■

■■

■■ ■■ ■■

■■

■■

Plan focus group sessions and interviews as carefully as you would any other assessment. Have a clear purpose for the sessions and know how you will use the resulting evidence. Don’t let them turn into unfocused gripe sessions. Plan on conducting several focus groups, not just one or two, because of the possibility that one opinion driver may dominate participants’ comments in any one group. Professionals continue conducting focus groups until they begin to hear the same themes expressed repeatedly across groups. Select a site in a conducive environment that participants find comfortable. Make sure participants are a representative sample of the group whose thoughts you seek. Offer an incentive to attend. Only rarely will the topic alone be a sufficient enticement to participate. You may need to offer a meal or a gift certificate, reimburse participants for travel expenses, and/or pay them outright. Select and train interviewers and focus group moderators carefully. Focusing questions is an art. Focus group moderators must be trained to elicit responses from all participants and keep the talkative few from dominating the session. Write out the moderators’ introduction and questions beforehand so sessions stay on track, and arrange to have participants’ comments recorded in some way. Plan questions carefully. They should be broad enough to elicit discussion, yet focused enough to make replies useful. Some examples: • What did you think was the best part of this program? • Tell me one way you would improve this program. • How has Bellefonte College helped you to be successful here? • What do you think will be the biggest barrier to finishing your degree?

Journals Journals (see Jargon Alert) may be designed to help students develop any of the following. • A skill through repeated practice. One of the most effective ways to learn is through plenty of study and practice (List 26.1 in Chapter 26), and journals 270

CHAPTER 20 Other Assessment Tools

can help ensure that students routinely Jargon Alert! practice an important skill such as writing, critical thinking, or problemJournals solving. Students in a first-year writing A journal is a document into which students make repeated entries during a course, program, or other course might be asked to write a learning experience. paragraph each day in a journal, while students in a social work course might be asked to journal short critical analyses of each reading assignment. • Study skills. Students might be asked to make weekly journal entries regarding their work on a major project such as a term paper to ensure that they are staying on track toward its timely completion. • Conceptual understanding. Faculty can periodically review journals that briefly summarize lessons or readings to ensure that students are grasping key points correctly. • Metacognition and synthesis skills. Students can be asked to journal what they learned from each learning activity, how they solved each problem, their reactions to each reading, or how each learning activity or reading affected their attitude toward the subject. After making routine entries on a regular basis, students might be asked to look back through their journals and reflect on how their skill or attitude has developed. Journals can be very time-consuming both for students to complete and for faculty and staff to assess, so use them only if what students learn from the experience makes the time worthwhile. As with any other learning activity, journals are worthwhile only if they are carefully planned, with clear learning goals, clear guidelines on exactly what students should write in the journal and how often, and clear assessment criteria aligned with the journal’s learning goals (Chapter 16). If you don’t have time to read and comment on every journal entry, try asking students to read and comment on each other’s journals, following your guidelines. Or you can read and offer feedback on random excerpts. Example 11 in Effective Grading (Walvoord & Anderson, 2010) is a rating scale rubric for assessing journal entries. For More Information An excellent article on reflective writing is “Generating, Deepening, and Documenting Learning: The Power of Critical Reflection in Applied Learning” (Ash & Clayton, 2009). A classic resource on surveys is Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method (Dillman, Smyth, & Christian, 2014). For more information on designing interviews, see Qualitative Research: A Guide to Design and Implementation (Merriam & Tisdell, 2016).

271

The Assessment Toolbox

Time to think, discuss, and practice 1. Ask a member of your group to play the role of the student life director at Harmony Community College. He or she will answer any questions you have about the college, its students, the co-curricular learning experiences it offers, and their learning goals. The director would like to survey students to assess their satisfaction with key co-curricular experiences and identify possible changes to improve student learning and success. Draft an ecosystem rating scale that could be used in the survey. 2. Think of a course you have taught or taken. Develop a short question or prompt for reflective writing related to the key learning goals of the course.

272

Some valuable ideas you’ll find in this chapter Phrase questions about attitudes and values so that there are no obviously correct or socially acceptable answers.

CHAPTER 21

Assessing the Hard-to-Assess

Motivating students to participate in add-on assessments and to give those assessments serious thought and effort is a significant challenge. Don’t try to assess what cannot be taught.

273

The Assessment Toolbox

T

here are several reasons why some learning goals are especially hard to assess effectively. • Some learning goals are promises we can’t keep. • We can’t see into the soul. • Performances are harder to assess than products. • Some learning goals can’t be assessed with graded assignments. This chapter shares strategies to address each of these challenges.

Some learning goals are promises we can’t keep As discussed in Chapter 5, learning goals are promises. We promise that, by the time students finish a learning experience or graduate, they will be able to do these things. But there are some things we hope to instill in students that we can’t guarantee, such as being a passionate lifelong learner, appreciating artistic expressions, or making ethical decisions. The following questions will help you figure out if and how these kinds of aims can be assessed. Is the learning goal stated clearly, using observable action verbs?  Try to articulate the goal following the suggestions in Chapter 4. Appreciating diversity may seem impossible to assess, for example, but restating it as Communicating effectively with people from diverse backgrounds turns it into an assessable learning goal. How have others assessed this learning goal?  No matter what you’re trying to assess, someone somewhere has already been trying to assess it. Don’t reinvent the wheel; learn what others have done to assess this goal (Chapter 12). How would you recognize people who have achieved this learning goal?  Chapter 4 suggested a strategy to clarify fuzzy goals that can be used here. Suppose, for example, that one of your college’s learning goals is for students to be passionate ­lifelong learners. Imagine that you run into two graduates of your college. As you talk with them, it becomes clear that one is a passionate lifelong learner and the other is not. How might you tell this? You might learn that, compared to the other graduate, the passionate lifelong learner: • Reads about a broader array of topics from a broader array of sources. • Watches more educational programming and visits more educational websites. • Attends a variety of cultural events. • Furthers his or her education through credit-bearing or non-credit classes. • Expresses curiosity about things outside his or her experience and areas of knowledge. 274

Assessing the Hard-to-Assess

Is the learning goal teachable?  Some learning goals are not assessable because, while they’re commendable intentions, they’re not teachable. Consider the goal that students will “have a lasting curiosity about other places and people.” While f­ aculty and staff may certainly hope that they will instill this trait in their students by ­modeling this behavior, encouraging expressions of curiosity, and so on, it may be hard to think of learning activities that will actively help students develop this trait and ensure that all students will graduate with it. One way to figure out if a learning goal is assessable is thus to ask faculty and staff to share how they help students achieve it. What learning activities do they give students to help them achieve the goal? If the faculty and staff can name specific learning activities, the goal is assessable – indeed, the completed assignment can probably serve as the assessment. But if your question draws blank stares and comments such as “Well, I think they pick it up by osmosis,” the goal may not be teachable – or assessable. Don’t try to assess what cannot be taught. Acknowledge these kinds of goals as important aims, but “don’t hold people accountable for things they cannot do” (Ewell, 2002, p. 16).

CHAPTER 21

Of course, people could not do any of these particular things and still be passionate lifelong learners. Similarly, people could do some of these things and still not be truly passionate lifelong learners. But the more evidence we have of these kinds of behaviors, and the greater the variety of evidence, the more confidently we can infer that graduates have indeed developed this trait. It’s notoriously difficult to get assessment information from alumni but, as discussed in Chapter 13, information from students who are about to graduate can serve as proxy measures that give worthwhile insight into how alumni may think and feel later in their lives.

What knowledge and skills are part of this learning goal?  Many hard-to-assess learning goals require knowledge and skills that are assessable. We can’t guarantee, for example, that all graduates will make ethical decisions, but we can make sure that they understand the characteristics of ethical decisions and can evaluate the ethics of various decisions. Those learning goals can be assessed with the assessment tools discussed in this book, such as tests and assignments scored with a rubric. How important is this learning goal?  We all have many things we’d like our ­students to learn – far more than we’re realistically able to help students achieve in the time we have with them. Don’t try to assess everything. Instead, as s­ uggested in Chapter 12, assess what really matters, focusing on truly important goals and ­generating results that inform truly important decisions. Ask, “Can our s­ tudents lead happy and fulfilling lives if they graduate without having achieved this particular learning goal?” and “What will happen if we don’t assess this learning goal?”

275

The Assessment Toolbox

At the same time, however, there are some who argue that we focus too much on assessable learning goals at the expense of more important aims that are harder to assess. They worry that a push for career readiness, for example, will be at the expense of other aims such as thoughtful appreciation of works of art or compassion for others. A world where such traits are a rarity would be a dismal place. So continue to acknowledge and value difficult-to-assess goals even if you’re not assessing them.

We can’t see into the soul We can assess students’ knowledge and skills because we have a fairly good window into their brains. An assessment can tell us if students can’t analyze data well, for example. It’s much harder to assess attitudes and values because we don’t have a good window into the soul. If students haven’t developed an attitude or value, they can fake it on an assessment, telling us what they think we want to hear rather than what they really feel or believe. Students might say on an assessment that they have grown spiritually, for example, even if they haven’t truly done so. In short, knowledge and skills can be assessed with convincing direct evidence (Chapter 3), but attitudes and values can be assessed only with indirect evidence, which is less convincing. The key to assessing attitudes and values meaningfully is to aim for evidence that, while indirect, is as honest and accurate as possible. The two basic strategies to do this – what might be called windows into the soul – are reflection (Chapter 20) and behaviors. If we collect information on reflection and behaviors carefully, and if we corroborate that information with a variety of measures (Chapter 3), we can infer how well students have achieved learning goals for the soul. Information on behaviors can be obtained through many of the assessment tools discussed in Chapter 20, including ratings scales, surveys, interviews, and focus groups. If you are assessing whether students are becoming passionate lifelong learners, for example – a scenario from the beginning of this chapter – you could ask students in a survey or interview about the behaviors mentioned in that scenario, such as what they read or what websites they visit. Because indirect evidence calls for a greater inferential leap than direct evidence, it’s especially important that assessments asking for reflection and behaviors follow the suggestions in Chapter 3 for ensuring good-quality, fair, and unbiased assessments. Also consider the following suggestions: Phrase questions so there’s no obviously right or wrong answer.  List 20.1 in Chapter 20 offers examples of such prompts. 276

Assessing the Hard-to-Assess

Avoid making significant memory demands.  Asking “How many hours did you study yesterday?” will give you even more accurate information than asking “How many hours per day do you usually study?” But also avoid asking for very precise responses. Questions asking for grade point average to two decimal places will probably not be answered accurately, for example.

CHAPTER 21

Ask for concrete behaviors rather than generalizations.  It’s easier for people to describe concrete behaviors accurately than to make accurate generalizations. Questions such as “Do you consider yourself a hard-working student?” tempt students to tell you what they think you want to hear, even if you collect their responses anonymously. Asking “How many hours per day do you usually study?” will elicit more honest answers. In general, think twice about asking students to rate themselves on attitudes and values, whether on locally designed or published rating scales.

Consider your questions from the opposite point of view.  One way to detect some kinds of potential bias, especially in surveys and interviews, is to ask, “If someone were hoping to see the exact opposite of the results that I’m hoping for, would he or she conduct the same assessment in the same way?” If you’re hoping to show that students respect diverse viewpoints, for example, imagine that you’re trying to show that they’re narrow-minded (difficult though this may be for you!). Would you still ask the same questions and phrase them the same way? Ask about both the pros and cons of an issue.  Don’t ask for criticisms without giving a chance to provide praise as well. Don’t ask students if they favor extending the hours that a service is available (who wouldn’t?) without noting the costs and other drawbacks. Avoid questions that people are uncomfortable answering honestly.  People are generally comfortable agreeing with the status quo and providing socially a­ cceptable answers. They are usually uncomfortable admitting inferiority or ­wrongdoing and may deceive themselves about their inclinations in sensitive areas, even if you are asking for information anonymously. People may not answer ­honestly, for example, when asked if they go to art museums or religious services or if they voted in the last election. Many will also inaccurately report their salary, job, or whether they failed a particular course. Phrase questions to elicit thoughtful replies.  If your learning goal is that ­students appreciate modern art, asking, “What is your attitude toward modern art?” may elicit responses such as “I like it” or “I’ve learned a lot about it.” Questions such 277

The Assessment Toolbox

as “Which work of art studied in this course or program would you most like to display in your home, if you could? Why?” are far likelier to elicit thoughtful responses and tell you more about your students’ true attitudes toward modern art. If appropriate, let people admit that they don’t know or can’t remember.  Forcing an opinion from people who don’t have one or a memory from people who can’t remember one is tantamount to forcing them to lie. Keep things simple.  Because attitudes and values are harder to assess, corroborating evidence is important . . . but we can easily drown in it. Aim for just enough evidence that you can feel confident in the conclusions you’re drawing, following the suggestions in Chapter 12.

Performances are harder to assess than products Students demonstrate their achievement of some learning goals through a performance rather than a product. Teamwork and leadership skills, for example, are demonstrated through group work. Lab safety skills are demonstrated while working in a lab. Culinary techniques are demonstrated while students are preparing a dish. There are four basic ways to assess learning goals demonstrated through such performances; none of them are always the best choice: Direct observation.  Faculty and staff can sometimes observe each student performance, such as preparing a pastry, and assess the performance using a rubric (Chapter 15). But sometimes this is difficult or impossible. A science professor may not have enough time, for example, to observe every student in a lab. If a co-curricular staff member sits in on a group meeting, his presence may alter the team’s discussions and outcomes. Student reflection.  Students can be asked to reflect on their performance (Chapter 20) by answering questions such as “How did you help the group complete its assignment? How did you hinder it?” Logs of their contributions to performances such as class discussions may also add insight. Peer assessments.  When groups of students are working collaboratively, students can assess their peers’ interpersonal skills by completing rubrics or structured observation guides (Chapter 15). Exhibit 21.1 is an example of a rating scale for assessment of peer contributions to a group project. Students can be asked to assess their entire group as well as individual members (Barkley, Major, & Cross, 2014). Chapter 13 discusses peer assessments further. 278

CHAPTER 21

Exhibit 21.1  A Rating Scale Rubric for Assessing Fellow Group Members Your name: Name of the group member you’re evaluating:

1. Did his or her fair share of the work.

Almost always □

Often Sometimes Rarely □

□

2. Participated actively in the group’s activities.

□

□

□

3. Contributed useful ideas, suggestions, and comments.

□

□

□

4. Listened carefully.

□

□

□

5. Appreciated others’ ideas.

□

□

□

6. Asked others to clarify their ideas, if necessary.

□

□

□

7. Expressed disagreements respectfully.

□

□

□

8. Did not dominate the conversation or interrupt others.

□

□

□

9. Tried to help the group reach consensus.

□

□

□

10. Helped the group stay on the topic.

□

□

□

11. Helped the group not waste time.

□

□

□

12. Helped me learn more than if I had worked alone.

□

□

□

□ □ □ □ □ □ □

Assessing the Hard-to-Assess

This group member . . .

□ □ □ □ □

Tests.  Students can be given a test of interpretive exercises (Chapter 17) in which they read or watch a video of a scenario, then respond to questions about the processes they would use in the scenario. Students in a culinary program might, for example, be asked questions about how they would plan and prepare a dish. Students’ interpersonal skills might be assessed by having them watch a video of a group interaction and answering questions on the actions they would take in response to what they see. While interpretive exercises can yield useful evidence of student learning, they’re not as authentic as having students demonstrate the skills being assessed.

Some learning goals can’t be assessed with graded assignments Grades are a powerful incentive for students to participate in an assessment and to give it serious thought and effort (Hutchings, Jankowski, & Ewell, 2014). Motivating students to participate in an ungraded add-on assessment can be a challenge. There is no magic solution or foolproof incentive; what entices some students will not entice others. Because of this, except for licensure or certification exams, add-on assessments should never be the centerpiece of an assessment program, although if steps are taken to motivate students, they can add useful insight. 279

The Assessment Toolbox

How can you maximize participation in add-on assessments? Four factors probably have the most effect on participation rates (List 21.1) (Suskie, 1996). Usually you cannot change the nature of the assessment or the students who must be assessed. But you can make the assessment appear important and be considerate of your participants – and thereby maximize participation. Brainstorm all possible reasons for students not to participate and do all you can to overcome those obstacles. Here are some specific suggestions. List 21.1  Factors Affecting Participation in Add-On Assessments ■■

■■

■■

■■

The nature of your assessment. A brief survey asking for simple, nonthreatening opinions will generally get greater participation than a test that requires three hours on a Saturday morning and a good deal of careful thought and effort. The people you are assessing. Students who have been dismissed from a college will be less likely to participate in a survey, for example, than students who are currently enrolled and in good standing. How important the assessment appears to be. If the assessment appears likely to inform important decisions, participants’ contribution will seem worthwhile and they’ll be more likely to participate. How considerate you are of your participants. Students are doing us a great favor when they participate in add-on assessments. If we show our appreciation by doing all we can to minimize their trouble and make their job as easy as possible, they will be much more likely to participate and provide sound, useful information.

Convince participants of the importance of the assessment activity.  Explain how participation will make an impact on something significant that students understand and appreciate. Include questions in the assessment that participants will find ­interesting and important. Cultivate a strong college culture of assessment (Part 3) in which students continually hear that this add-on assessment is integral and valued, not a superfluous extra. Appeal to participants’ self-interest.  Answer their unspoken question, “What’s in this for me?” Explain how the results will benefit them directly or something about which they are concerned. Offer to send them a summary of the results that will let them see the impact of their efforts. If possible, give students individualized feedback on their strengths and weaknesses or how they compare to their peers. Also guarantee unconditional confidentiality. If you are using assessments with code numbers or other identifying information, explain why. Stress that you will look only at aggregated responses.

280

Be sensitive to survey fatigue.  Ask yourself if you really need to conduct an add-on assessment. If you are considering a survey, interview, or focus group, check with others at your college who might be planning something similar. Sometimes

Assessing the Hard-to-Assess

Minimize the inconvenience of the assessment.  If the assessment cannot be ­completed online at the participant’s convenience, schedule it at a convenient time and place – not before a holiday or during finals week – and give participants plenty of advance notice. If it’s not possible to conduct the assessment during regularly ­scheduled class time in appropriate courses, consider conducting the assessment on several days, at several times, so students can find an assessment period that they can attend regardless of their other obligations. Some colleges schedule an assessment day once each term or year, during which no regular classes are held and students instead participate in add-on assessment activities.

CHAPTER 21

you can piggyback on another survey, adding a few questions to it rather than ­creating a separate survey. Or you can each plan to survey different samples of ­students so that no one receives more than one survey.

Keep the assessment short.  The shorter the assessment, survey, or interview, the more considerate you are of your participants’ time and the higher the participation rate. Try to keep any paper survey to no more than one piece of paper, a telephone interview to no more than 5 minutes, and a focus group or in-person interview to no more than 30 minutes. Review the questions posed in Chapter 6 about the purpose of your assessment and make sure that the assessment focuses only on key learning goals and issues related to them. Open-ended survey questions are usually not very popular with survey recipients because they lengthen the time required to complete the survey and require more mental energy to respond. Very often survey respondents simply leave them blank! So use them sparingly – only when your question is interesting enough that participants will want to answer it, or when a rating scale might bias responses by steering participants in a particular direction, and when your question can be answered in no more than one sentence. Keep the assessment clear.  Participants shouldn’t have to spend time trying to figure out what the assessment is really looking for or how to use necessary ­technologies such as navigating through a website. If possible, try out the assessment with a small group of students to be sure the guidelines and questions are truly clear and any technologies are easy to use. Make the assessment engaging and fun.  Several Learning Assessment Techniques (Barkley & Major, 2016) call for students to work collaboratively rather than ­individually on a test or quiz, perhaps by participating in a quiz bowl. If you’re not sure how to make an add-on assessment more fun, ask your students for ideas!

281

The Assessment Toolbox

Provide a material incentive to encourage students to participate.  Possibilities include the following: • A token incentive enclosed with every invitation (perhaps a coupon for a free ice cream cone), because it can create a sense of obligation • A material incentive to participating students such as cash, a complimentary meal, or gift certificate • Perks that are highly prized but have little or no direct cost, such as registration or housing preference, a parking space in a prime lot, or extra graduation tickets • Entering the names of those participating in random drawings for significant prizes such as laptops or tablets The effectiveness of material incentives can vary dramatically depending on college culture, student values, and the assessment itself. Free pizza might work beautifully with some students and be a dismal failure with others. The key is to find an incentive that is particularly appealing to your students. Sometime a collegespecific item, such as a T-shirt with logo or tickets to a popular campus event, works best. With other students, a more generic incentive such as a complimentary meal or a gift card is more effective. If the time and thought contributed are significant, cash compensation may be necessary. Ask a few students to suggest incentives that would convince them to participate, and consider trying out incentives with small groups of students before launching a full-scale assessment. You may want to use some special incentives only with those subgroups of students whose participation rates are historically low. Allow students to include assessment results in their credentials at their discretion.  This is especially effective, of course, if prospective employers or graduate programs value the assessment. Some students may find that they can strengthen their job or graduate school applications by including items from their portfolios or by having their academic record note that they scored at, say, the 87th ­percentile on a nationally recognized exam. Include scores in student credentials only if the student so chooses, or students who think they will do poorly will be unlikely to participate. Give recognition to top scorers or the first students to return a survey.  Students earning exceptional scores on important assessments might receive a notation on their transcripts. This incentive will not work, of course, with students who think they have no chance of earning a top score. Make participation in the assessment a requirement of a program or course.  While this can be an effective participation incentive (Ekman & Pelletier, 2008), it will not necessarily compel students to give the assessment their best effort. Sometimes an

282

Assessing the Hard-to-Assess

Avoid setting a minimum add-on assessment score as a graduation or progression requirement.  While it may be tempting to ensure student participation in an add-on assessment by establishing a minimum score as a graduation or progression requirement, single scores should never be the sole basis of any major decision such as retention or graduation. Requiring a single minimum score as a graduation or ­progression requirement is an unethical use of assessment results (Chapter 26). Minimum scores may be used as graduation or progression requirements only if students have multiple opportunities to complete the assessment successfully. ­Ideally they should also have an alternative means of demonstrating competence, such as submitting a portfolio of their work for assessment by a faculty panel.

CHAPTER 21

add-on test or portfolio can be scored quickly enough that the results might be factored into a course grade. While this would be a powerful incentive for students to give the assessment their best effort, do this only when the add-on assessment is designed to assess key course learning goals. Otherwise this is an inappropriate use of results (Chapter 26). If you decide to make participation in an add-on assessment a requirement, put the requirement in writing and draw it to the attention of students entering the course or program.

What is an acceptable participation rate in an add-on assessment? This question has both simple and complex answers. The simple, traditional answer has been to aim to have 70 percent to 80 percent of those contacted participate in a survey or interview and to consider a 50 percent participation rate minimally adequate. But too often these aims are just not realistic. The complex and better answer is that the quality of participants is more important than the quantity. In other words, having participants who are truly representative of the group from which you’re sampling can be more important evidence of your assessment’s credibility than its participation rate. Imagine that faculty and staff at two very large universities, each graduating about 8,000 students annually, want to learn about their seniors’ self-perceptions of their thinking skills. Faculty and staff at Eastern State University send a survey to all 8,000 seniors, of which 400 are returned. Faculty and staff at Western State University send a survey to a random sample (Chapter 12) of 600 seniors. They make strong efforts to convince those students to complete and return the survey and, as a result, 360 are returned. Which is the better approach? While more students completed Eastern State’s survey, its response rate is only five percent. This makes it unlikely that the respondents are representative of all seniors. Some cohorts – perhaps students in 283

The Assessment Toolbox

certain majors – may be underrepresented, which would call the value of the survey into question. Western State’s survey, on the other hand, yields a more respectable 60 percent response rate, which gives us more confidence that the respondents are a good cross-section of all seniors, even though the number of returned surveys is smaller. Furthermore, Western State’s approach may be more cost-effective; it may be less expensive to survey 600 students intensively than 8,000 scattershot. Relatively small-scale assessments with high participation rates may thus yield more credible results than larger assessments with low participation rates. No matter what your participation rate is, collect some basic demographic information on your participants such as major, age, gender identity, or employment status. When you share your results, report your participation rate and describe how representative your participants are of the group you’re surveying (Chapter 24), so audience members can judge for themselves how credible the survey results are.

Time to think, discuss, and practice 1. One of Hacienda College’s goals is to prepare students to “succeed in a diverse, global society.” Brainstorm some ways that the college could assess student achievement of this goal. 2. The International Studies faculty wish to interview graduating seniors on their perceptions of the program. Brainstorm three approaches that the faculty might use to convince seniors to participate in an out-of-class interview. 3. Faculty in the Biology program would like to survey recent alumni to assess their satisfaction with the program and identify areas for improvement. Assume the survey will be sent to all alumni graduating within the last five years. Brainstorm three feasible strategies to maximize the participation rate.

284

Part 5

Understanding and Using ­Evidence of Student Learning

Some valuable ideas you’ll find in this chapter

CHAPTER 22

Setting Meaningful Standards and Targets There are many ways to set standards or benchmarks, and any one alone may give an incomplete picture of student learning. Set a standard for adequate performance by thinking about what would not embarrass you. For essential goals, aim for almost all students to achieve your standard. What is often more important than value-added is whether students earning a degree have the competencies expected of a college graduate.

287

Understanding and Using Student Learning Evidence

288

T

oday many colleges have worked hard to collect considerable evidence of student learning, only to see conversations then come to a halt. Why is it so hard to use evidence of student learning to identify and implement changes? Chapter 14 lists plenty of reasons, but there’s one more: You cannot share your successes or identify what might need improvement until you have a clear definition of what constitutes successful outcomes. Consider this scenario: A group of faculty has assessed student writing using a rubric, and the resulting evidence is shown in Exhibit 22.1. Now what? Often faculty don’t know what to do with this kind of evidence, because they’ve never had a conversation about which performance level represents – in this scenario – satisfactory writing skills and what proportion of students the faculty should reasonably expect to reach at least that level. The faculty might agree that they want students reach Level 3 on every trait. But this might place the faculty in a no-win situation. If less than 100 percent of students earn a 3, the faculty would have to conclude that they must make changes to try to improve student learning. But if every student earns a 3 on every trait, they might feel the outcomes are suspect – perhaps that their standard was too low. The outcomes are either too poor or too good – never such that the faculty can conclude, “We’re not perfect, but we’ve done our job, and we’ve done it pretty darned well, and all we need to do from this point forward is stay the course. Let’s focus on other aspects of student learning that need our attention more urgently.” Now let’s repeat the scenario but with a twist: The faculty have assessed student writing using this rubric, but this time they have agreed that a score of 2 is the minimum they consider acceptable for each trait. They have also agreed that they want no more than five percent of students to score a 1. Now they see from Exhibit 22.1 that these targets have been met for every trait except two: Those for content/reasoning and grammar/mechanics. This gives the faculty a clear path to using their evidence. They can celebrate that their students met their targets on most traits, and they can identify and implement strategies to help students with reasoning and grammar. There are two lessons here: • Defining successful outcomes means having a clear sense of what is good enough. In the parlance of the road trip analogy in Chapter 16, this means articulating clearly how you will know that your students have arrived at your intended destination for them. In the second scenario, the faculty set a standard or benchmark (see Jargon Alert) that students should score at least a 2 on a 3-point scale in order to be deemed acceptable writers. • Numbers have meaning only when they are compared against other numbers. In the second scenario, the faculty compared the 15 percent of students earning a 1 for grammar/mechanics against their target (see Jargon Alert) that no more than five percent earn a 1.

Paper has a clear purpose & shows awareness

Audience

is somewhat flat and dull. 40% of students: Paper has several wordy sentences/phrases.

the paper interesting. 55% of students:

Paper is appropriately concise.

Choice

Conciseness

the reader.

Multiple errors impede meaning or distract

sentences/phrases. 15% of students:

Paper has numerous wordy

5% of students:

paper is unengaging.

0% of students:

Multiple sentences are awkward or unclear.

0% of students:

with evidence.

Ideas & positions are often not supported

This rubric uses ideas in the “Standards for a ‘C’ Grade in English Composition” approved by Maryland chief academic officers (http://mdcao.usmd .edu/engl.html).

nor overly distract the reader.

mechanics, etc.; errors do not impede meaning

purpose but some are inappropriate, or the paper

for audience & purpose, are varied, and make

Word

Substantially free of errors in grammar,

Style options (tone, word choice) are appropriate Style options are largely reasonable for audience & Style options are largely inappropriate, or the

Tone &

mechanics

but some sentences are awkward or unclear. 65% of students:

in pattern. 35% of students:

Impeccable grammar, spelling, punctuation, &

Sentences are generally clear & well structured,

All sentences are clear, well structured, & varied

Structure

Mechanics

always well developed. 70% of students:

evidence & relevant facts, examples, details, etc. 30% of students:

Sentence

55% of students:

supported with some evidence but are not

are well developed & supported with convincing

30% of students:

Reasoning is sound; ideas & positions are

Grammar/

45% of students:

transitions. 10% of students:

or use effective organizational devices &

Multiple paragraphs fail to make clear points

Exceptionally sound reasoning; ideas & positions

Most but not all paragraphs have clear points,

All paragraphs have clear points with effective

Structure

5% of students:

Reasoning

ineffective. 40% of students:

transitions) are always effective & smooth. 55% of students:

Paragraph

Content/

devices may lack smoothness, be missing, or be

thesis statement, opening/closing paragraphs,

to follow.

organizational devices, & transitions.

is good enough to be understandable, but

organization. Organizational devices (title,

transitions). 45% of students:

throughout the paper. Overall organization

throughout the paper. Well-planned

Central idea is unclear. Paper is difficult

audience. 5% of students:

Paper does not show awareness of purpose &

Level 1 5% of students:

organizational devices (e.g., topic sentences &

Paper vaguely presents a central idea supported

audience. 65% of students:

Paper shows limited awareness of purpose &

Level 2 30% of students:

Paper clearly presents a central idea supported

of audience. Organization 30% of students:

Level 3 65% of students:

Purpose &

Exhibit 22.1  Rubric Results for a Hypothetical Assessment of Written Communication Skills

Understanding and Using Student Learning Evidence

Jargon Alert! Benchmarks, Standards, and Targets There is no popular consensus on what these terms mean. In this book, a benchmark or standard for student learning is a minimally acceptable student achievement or performance level. A target is the proportion of students that should achieve at least that minimally acceptable level.

Implementing these two principles is hard, for two reasons. First, there are many kinds of numbers – many perspectives – against which evidence of student learning might be compared. Second, deciding what is good enough is a subjective process requiring informed judgment. This chapter addresses these challenges by taking you through three steps:

1. Choosing the perspective(s) that you will use to compare your evidence 2. Setting an appropriately rigorous benchmark or standard for student achievement 3. Setting an appropriately rigorous target for the proportion of students achieving that standard There’s a third lesson from these two scenarios: These steps should be discussed when you’re planning an assessment, not when you’re facing the resulting evidence. Otherwise, you may end up with an assessment that doesn’t help you answer questions that you and others care about.

Choosing an appropriate perspective for comparison Imagine that Michael, one of your students, scored a 55 on a test. Did he do well or not? Answering this question is critical to figuring out how to use this score to inform decisions about his learning. If we decide he did well, we can celebrate, but if we decide he didn’t do well, we have work to do. Michael’s score of 55 alone tells us nothing about how well he did. In order for his score to have meaning, we must compare it against something else. There are several “something elses” against which his score might be compared, and each perspective has pros and cons.

Local standard A local standard (or competency-based or criterion-referenced) perspective compares Michael’s 55 against a standard established by those teaching in the course, program, learning activity, or college. Suppose that your department colleagues agree that students should earn at least a 35 in order to earn a passing grade on the test. According to this standard, Michael did well. An advantage of local standards is faculty ownership (Chapter 13). “Faculty pay much closer attention to assessment results when they have played a role in establishing performance expectations” (Pieper, Fulcher, Sundre, & Erwin, 2008, p. 7). 290

An external standard perspective compares Michael’s 55 against an externally established standard. Suppose that your disciplinary association has decreed that students should earn a 45 on the test Michael took in order to be considered employable in a career in your discipline. According to this standard, Michael did well. External standards most often come from tests required for certification or licensure, such as the Praxis (www.ets.org/praxis) for teachers and the NCLEX (www.ncsbn.org/nclex.htm) for nurses. An advantage of external standards is that, for better or worse, someone else has set your standard for you; you don’t need to spend time and effort deciding what an appropriate standard should be. Some public audiences, including some employers and government policy makers find external standards more compelling than local standards. The main disadvantage of external standards is that they may be based on published tests whose learning goals don’t align well with yours, which would make them of limited value in understanding and improving student learning (Chapter 19).

Setting Meaningful Standards and Targets

External standard

CHAPTER 22

Local standards are inappropriate for self-reported information such as rating scales because, as discussed in Chapter 21, self-reported information can be unreliable. Comparing self-reports against peers or against past students is more meaningful because, while people do sometimes purposely or inadvertently misrepresent themselves, there is usually no reason to believe that your students are any more likely to misrepresent themselves than other students, including students from a few years ago or students at other colleges.

Peers A peer (or comparative or norm-referenced) perspective compares Michael’s 55 against the scores of peers. Suppose that the national average for the test Michael took is 70. Under this benchmark, Michael didn’t do very well with his score of 55. Peers may be internal or external to your college. Internal peers might be the students at your college, the students in Michael’s program at your college, or students with traits similar to Michael’s, such as part-time or online students. Internal peers can also be students who did not have the same experience. Suppose, for example, that Michael is in an online course. Comparing his class’s scores against those of students who took the same course in a traditional setting would help us understand the effect of online instruction on student learning. Keep in mind, however, that sometimes comparisons among students with different experiences are pointless. It would be inappropriate to compare the scientific reasoning skills of history and biology majors, for example.

291

Understanding and Using Student Learning Evidence

External peers might be students in colleges similar to yours, in programs similar to Michael’s, or with traits similar to Michael’s. To be meaningful, external peers need to be appropriate; there’s no point in comparing students at your ­ open-admission two-year college against those at highly selective four-year colleges. List 22.1 offers resources for identifying peer colleges in the United States. List 22.1  Resources for Identifying Potential Peer Colleges ■■ ■■

■■

■■

■■

Carnegie classifications (carnegieclassifications.iu.edu) Higher education organizations and consortia, such as the American Association of Universities (AAU), the American Association of State Colleges and Universities (AASCU), the National Association of Independent Colleges and Universities (NAICU), or the New American Colleges & Universities (NAC&U) Online higher education databases such as College Results Online (www.collegeresults.org) that can be searched for colleges with traits such as a particular size or retention rate Publishers of assessment instruments (Chapter 19), who may share the names of institutions using their instrument For public colleges, fellow members of a state or regional system

Some colleges find it helpful to identify two sets of peers: current peers and aspirational peers. Others find it useful to identify several sets of peers, perhaps one set for the engineering school and another for the fine arts programs. The main advantage of the peer perspective is that it adds a fresh viewpoint and gets us out of the ivory tower, as discussed in Chapter 19. But the peer perspective also has a number of limitations: Sometimes colleges and programs have no peers.  Some programs and colleges, such as those with specialized missions or serving unique student populations, have no peers, making the peer perspective inappropriate. Sometimes peers are not really peers.  Differences between your students’ learning and that of their peers may be due to factors other than the quality of their learning experiences. The quantitative skills of students at a peer college might be stronger than those of your students, for example, simply because more students at the peer college are enrolled in science and engineering programs. Sometimes peers don’t use a comparable assessment.  The peer perspective works only if your college or program and its peers use the same rubric, test, or survey. People focus on the peer average even when it is inappropriate.  Unless everyone in the peer group is performing identically, it’s mathematically impossible to expect every college, program, or student cohort to perform at or above the average. Given 292

Brightline

Comparisons to external peers may be less meaningful and useful than ­comparisons to internal peers,  because there is often more within-college ­variability than between-college variability. In other words, knowing that the writing skills of your business students are weaker than those of your graphic ­communications students can be more useful than knowing that the writing skills of all your students are a bit better than those of students nationally.

Value-added

Setting Meaningful Standards and Targets

A brightline is a line in a table that shows either an average of the numbers in the table or a target such as a minimum pass score on an exam. The brightline is highlighted with shading, boldface, or lines. The other lines in the table are sorted from highest to lowest and the brightline is inserted at an appropriate point.

CHAPTER 22

Jargon Alert!

colleges’ varying missions and student backgrounds and needs, below-average performance will sometimes be ­completely appropriate. But the ­temptation remains strong to use the peer average as a b ­ rightline (see Jargon Alert) and to look ­negatively on below-average outcomes and positively on above-average outcomes.

A value-added (or growth, change, longitudinal, or improvement) perspective compares Michael’s 55 against his performance when he began his studies. Suppose that, a year ago, Michael scored a 25 on the same test. Under this benchmark, Michael shows considerable improvement. The value-added perspective may be a good choice if it is important to document that a course, program, or other learning experience yields significant gains in student learning. You might be asked to justify, for example, that the $50,000 your college invests in its service learning program significantly improves students’ skills in communicating with people of diverse backgrounds, and a ­ value-added approach may provide the most convincing evidence of this. The value-added perspective can also be helpful if your students are so different from typical students that the peer perspective is inappropriate. Suppose that your college enrolls many students who never studied algebra in high school, leaving them far behind typical college students in quantitative skills. With an external peer perspective, you might conclude that your mathematics program is deficient. But with a value-added perspective, you may learn your students’ skills improve significantly and your programs are actually quite effective. The value-added perspective can also be helpful if you have too few students to compare with confidence against local standards, external standards or peers. For example, if your program typically graduates only a handful of students each year, their scores on a particular assessment could fluctuate considerably from one year to the next, making it difficult to decide how successful the program is. In this 293

Understanding and Using Student Learning Evidence

case, it may be more useful to examine each student’s individual growth. This is called an ipsative assessment. The value-added perspective nonetheless has some serious shortcomings: The value-added perspective may not be relevant or useful.  Employers care less about how much students learn in college than whether they graduate with the competencies employers need. “Imagine an open-admission college whose mission is to educate a very underprepared student population. Students enter reading at the third grade level, and after Herculean work by the faculty and staff, they graduate four years later reading at the eleventh grade level. This is tremendous value-added, but would employers want to hire graduates who are not reading at the college level?” (Suskie, 2007, p. 12). We can’t be sure that improvement is the result of our efforts.  The value-added perspective is often confused with the pre-post experimental research design used in the social sciences. Unlike a pre-post design, the value-added perspective does not compare the change we see against that of a randomly assigned ­control group of students who have not attended our college. So we can’t be sure that any growth is due to our curricula and pedagogies. It may instead be due to concurrent ­influences such as part-time jobs or even normal maturation. Growth in a ­student’s oral communication skills, for example, may be due more to her part-time job as a restaurant server than to her speech communication course, while the refinement of her life goals may be due as much to normal maturation as to a first-year seminar. Changes between entry and exit scores can be so imprecise that they mask the true amount of change.  The error margin (Chapter 12) of value-added scores is generally about twice as large as that of the scores themselves. Suppose that student essays are scored using a 5-point scale. The average score for entering freshmen is 3.2 and the average for rising sophomores is 3.4 – a discouragingly small gain. Part of the reason for this apparently small change may be that the 5-point scale hides the growth of a student who scores, say, a very low 3 – almost a 2 – at entry, and a very high 3 – but not quite a 4 – at exit. With the 5-point scale, the student would earn a 3 at both entry and exit and appear to have no growth at all. Value-added assessments are generally not possible for students who transfer into or out of a college or program.  If your college or program serves large numbers of such students, value-added evidence may be from the relatively small proportion of students who persist through the end of a program, who “might be systematically stronger than those choosing to depart or delay completion” (Pieper, Fulcher, Sundre, & Erwin, 2008, p. 6).

294

The value-added approach doubles workload,  because the assessment must be ­conducted twice instead of once. Ask yourself if the additional information gained from a value-added approach is worth the extra time and labor. Value-added ­benchmarks aren’t needed in those disciplines about which most students know little or nothing when they enter, such as occupational therapy or aeronautical engineering.

Setting Meaningful Standards and Targets

It can be difficult to motivate students to do their best on assessments given at the beginning  of a course, program, or other learning experience. Indeed, there is a subliminal disincentive for students to do their best – after all, the worse they do at entry, the bigger the potential growth by exit.

CHAPTER 22

Value-added evidence may be affected by the assessment’s floor and ceiling  (Chapter 3). Students who earn close to the highest possible score when they enter may not show much growth simply because they are so close to the assessment’s ceiling. Students who earn the lowest possible score when they enter may not show much growth if their true entry-level performance is actually much lower than the assessment’s floor.

Historical trends A historical trends perspective compares current students against peers in prior classes. Suppose that, on Michael’s test, 95 percent of this year’s students earned at least a 35 (the local standard mentioned earlier), compared to 85 percent last year, and 80 percent the year before. According to this perspective, this year’s class did quite well. The historical trends perspective differs from the value-added perspective because it looks at changes in successive groups of students rather than change within one group or one student. When people say they’re collecting baseline data, they’re probably planning to use the historical trends perspective. The historical trends perspective is helpful in evaluating the effectiveness of changes made in a course or program. It can also be useful if your students, courses, or programs are very different from their peers. With an external standard or peer perspective, relatively low scores might lead you to conclude that your learning experiences are inadequate, but with a historical trends perspective, you may learn this year’s learning experience is significantly more effective than last year’s and you are making good progress. The shortcoming of the historical trends perspective is that meaningful historical data are not always available. Students change, their needs change, curricula change, and pedagogies change. A survey conducted a few years ago may not have asked the questions that concern us today, for example. In such cases, we might want to use a different assessment today than what was used a few years ago. 295

Understanding and Using Student Learning Evidence

296

If you are considering a historical trends benchmark, balance your need for historical information with your need for information that’s relevant today. Review your assessment tools on a regular basis and revise any items or sections that are now outdated or irrelevant, even if that means losing some historical information.

So which perspective is best? Table 22.1 summarizes the five perspectives that have been discussed, giving an example of a benchmark or standard for each perspective as well as an example of a target, which is discussed later in this chapter. Consider the following as you decide which perspective to use. Table 22.1:  Perspectives for Comparing Evidence of Student Learning Perspective Questions Each ­Perspective Can Answer

Hypothetical Example of a Benchmark or Standard

Hypothetical Example of a Target

Students should earn at least a 2 on each trait of a rubric with a 4-point scale.

At least 90 percent of students should meet the standard.

Local standards

Are our students meeting our own standards?

External standards

Are our students Teacher education students must At least 85 percent of meeting standards set earn at least 150 to pass the state students should meet the by someone else? teaching certification exam. standard the first time they take the test.

Peers

How do students compare to peers at other colleges?

Students should score above the national average on the XYZ Critical Thinking Test.

At least 75 percent of students should score above the national average.

How do students compare to peers at our college?

Psychology majors should score above the college-wide average on a rubric assessing quantitative skills.

At least 80 percent of Psychology majors should score above the collegewide average.

Value-added Are our students improving?

Between the beginning of their first semester and the point at which they earn 60 credits, students should improve at least 1 point on a 5-point rubric.

At least 80 percent of students should improve at least 1 point.

Historical trends

Students this semester should score higher on an accounting exam than students five years ago.

At least 75 percent of students should score at least 81, which was the average score five years ago.

Is our program improving?

Setting Meaningful Standards and Targets

The local standard is generally the most important one . . . but also the hardest to defend.  The local standard is the foundation from which other standards can be ­identified. In order to identify suitable peers, decide whether value-added is sufficient, or decide if this student cohort is achieving sufficiently more than previous cohorts, you need to start with a sense of what you consider minimally acceptable work – and why.

CHAPTER 22

No one perspective is perfect.  Each perspective has strengths and weaknesses and paints an incomplete picture of success. Conclusions about Michael’s performance have varied considerably in the examples throughout this chapter, depending on the kind of benchmark or standard against which his score has been compared. “In essence, the perspectives that have been described here can be thought of as a set of lenses through which an object – assessment results – can be viewed. While each perspective or lens can give us a view of the object – student learning – the view through each lens is somewhat incomplete, because each looks at the object from only one angle, and the view through each lens is somewhat distorted, because no assessment tool or strategy is completely accurate. Viewing assessment results using only one perspective thus gives us an incomplete and somewhat distorted perception of student learning” (Suskie, 2007, p. 13).

Multiple perspectives give a more balanced picture of student learning.  ­“Examining student learning through multiple lenses, from multiple angles, or at least having the latitude to choose the lenses that provide the most meaningful view, gives us the best chance of answering the complex question of what our students have learned and whether they have achieved our goals for them” (Suskie, 2007, p. 13). Choose the perspective that is most meaningful and useful for you.  While, as already suggested, it’s often best to start by discussing a local standard, try to look at your evidence from at least one other perspective if you can.

How to set an appropriately rigorous benchmark or standard If your evidence of student learning will be viewed by skeptical audiences, you will need to convince those skeptics that your local standard is truly appropriately rigorous. This section of this chapter provides steps to do this (Cizek, 2012). Focus on what would not embarrass you.  As noted at the beginning of this chapter, a benchmark or standard for student learning defines minimally acceptable student achievement or performance. If you are struggling to conceive what this level of student achievement would look like, ask the following questions: • What achievement level would earn the lowest possible passing grade at this point in the student’s studies? 297

Understanding and Using Student Learning Evidence

298

• What is the lowest achievement level that would nonetheless adequately prepare students for success in what comes next in their lives: The next course in a sequence, more advanced study, or employment? • What is the lowest achievement level that wouldn’t embarrass you, if people learned that a student passed this course or graduated performing at that level? You’d be embarrassed, for example, by a graduate of your program who couldn’t put together a coherent paragraph. How will the evidence be used?  Setting standards is basically a judgment call, and you’re probably not going to get it exactly right. Whether you aim for a standard a little on the high side or a little on the low side depends on how the evidence will be used. As discussed in Chapter 6, the fundamental purposes of assessment are ensuring and improving educational quality, stewardship, and accountability. These purposes suggest the following principles: • If you want to identify areas for improvement, set a relatively high bar so you increase your chances of identifying what needs to be improved. • If you want to ensure that college resources are used to fund areas needing improvement, set a relatively high bar to help ensure that the areas that need funding receive it. • If you want to ensure that students are competent when they complete a course, program, degree, or other learning experience, set a relatively high bar to be sure that they are indeed competent. • If you want to maintain the status quo and avoid extra work, set a relatively low bar so your program or college looks successful and doesn’t need any changes. What is the potential harm of setting the standard too high or too low?  If you set standards too low, you risk passing or graduating students who are unprepared for whatever comes next in their lives. If you set standards too high, you risk wasting time and resources working to improve student learning when it really doesn’t need to be improved. The temptation to set standards so low that all students appear to succeed is strong; if they’re all successful, we don’t have to change anything. But generally there’s more harm in setting a standard too low than too high, because a standard that’s too low can adversely affect students’ futures. A low standard can let subpar students slip through the cracks, completing the learning experience without having learned what they need to succeed in whatever comes next in their lives. Some faculty and staff think that they should set a relatively low bar so their program or college appears successful to accreditors. But, while all accreditors want to see evidence that your students are meeting your standards, what’s of greater concern to them is that colleges and programs have appropriate rigor and that they use evidence of student learning to improve what they’re doing and to inform

List 22.2  Sources of External Insight on Potential Standards ■■

■■

■■

■■

■■

■■

■■

Convene an advisory panel of employers and ask them about the performance levels they expect in new hires. Ask them to review drafts of rubrics to make sure the rubrics address the skills that employers seek at appropriate levels of rigor. Consult with your disciplinary association. Some have issued statements on expectations for graduates of programs in their discipline. Consult with faculty colleagues teaching in peer programs at peer colleges. They may have set standards for their own assessments that you can adapt. If your students continue on to further study, consult with faculty in more advanced programs about what they expect in an entering student. Review the Degree Qualifications Profile (degreeprofile.org/read-the-dqp/dqp-cover), which aims to articulate the proficiencies students should achieve by the time they earn an associate, bachelor’s, or master’s degree. Review the VALUE rubrics (www.aacu.org/value-rubrics), which define various performance levels for a number of important learning goals. Look online. A simple web search may turn up examples of rubrics and benchmarks, particularly for common learning goals such as writing, science laboratory skills, and information literacy.

Setting Meaningful Standards and Targets

Externally informed standards are more defensible.  While this chapter offers many suggestions on setting standards, there is only one really hard-and-fast rule: Standards must be justifiable. You must be able to convince your stakeholders that your standards are appropriately rigorous. You must be able to explain not only what minimally acceptable student achievement is but why you consider that level minimally acceptable. One of the best ways to justify the rigor of your standards is to bring external perspectives and information into the standard-setting discussion. List 22.2 suggests ways to do this. Also consider the following ideas.

CHAPTER 22

planning and budget decisions. This argues for setting a standard a bit too high rather than a bit too low.

Define your learning goals, traits, and performance levels clearly.  Appropriate ­standards are much easier to establish if your learning goals (Chapter 4), rating scale traits (Chapter 20), and rubric traits (Chapter 15) are clear to students and your ­colleagues. If you’re using an analytic rubric, the column that describes minimally a­ dequate performance defines your standard. As explained in Chapter 15, when you create a rubric, complete this column before completing the other performance-level columns. Look at the assignment being assessed.  If you are asking students to complete an assignment under a time limit (such as an in-class essay exam) or without revisions, 299

Understanding and Using Student Learning Evidence

you’re not going to get their best work. Ask faculty to complete the assignment themselves, under the constraints that students have, and see if your standard is reasonable under the circumstances. Use samples of student work to inform your thinking.  Setting standards without looking at actual examples of student work may lead to arbitrary and perhaps unrealistic standards. Collect just a few examples of student work – good, bad, and mediocre – and discuss them with your colleagues, using the following questions as a springboard for conversation: • Which examples represent exemplary work? Why? Would it be realistic to establish these as the targets we aim for in all students? • Which examples are unacceptably inadequate? Which would embarrass us if they were from graduates of this program or college? Why? • How do exemplary, acceptable, and inadequate examples differ? • What kinds of student performance represent minimally acceptable work for a graduate of this program or college? Ground benchmarks with data . . . but after the first vote.  At this point you and your colleagues should be ready to take an initial vote on the benchmark or standard that will constitute minimally adequate performance. Once you’ve agreed on a tentative standard, test it. If you’re using a rubric, try it and see how the results compare to your standard. Then adjust your standard as you think appropriate. Set consistent standards.  As a matter of integrity, colleges should aim to provide consistently high-quality learning experiences, with consistent levels of rigor, to all students, no matter where or how they learn: On or off campus, face-to-face or online.

How to set an appropriately rigorous target for the proportion of students achieving your standard Once you have established your benchmark or standard, the next part of the standard-setting process is deciding how many students you’d like to achieve that standard. Express targets as percentages, not averages.  As discussed in Chapter 23, ­percentages are usually more meaningful and useful than averages. If you plan to use a rubric, for example, a target that 90 percent of your students will earn at least a minimally adequate rating is more understandable and useful than a target that students will earn an average of 3.1 on a 5-point rubric scale. Similarly, if you’re using 300

Setting Meaningful Standards and Targets

For basic, essential learning goals, aim for almost all student work to meet your standard.  Some learning goals are so basic, essential, or vital to health and safety that we want absolute assurance that every student graduates with minimally adequate achievement of them. No one wants tax returns prepared by an accountant who can prepare them accurately only 70 percent of the time, for example. But no matter how hard you try, it’s highly unlikely that every student will achieve your standard on every assessment. Every assessment is an imperfect, incomplete sample of what students have learned (Chapter 3). It’s thus entirely possible that a few students who have truly mastered important learning goals will do poorly on one particular assessment because they misunderstand it, are ill, or are facing a distracting personal issue. So for basic, essential learning goals – those with grave consequences for students who don’t achieve them at a minimally adequate level – a target close to, but not quite at, 100 percent is often appropriate.

CHAPTER 22

a test, a target that 95 percent of your students will earn at least a 65 is more understandable and useful than a target that they earn an average of 73.

For aspirational goals, your targets may vary.  Some learning goals and traits are aspirational; we hope students make progress with them, but we don’t expect all students to achieve them. No one expects every college graduate to be an inspirational leader or a compelling public speaker, for example. Weigh the costs and benefits of setting a high target for aspirational learning goals and traits. Will the occasional economics or biology student who slips through with, say, less-than-desired public speaking skills nonetheless have a productive and satisfying career? What would be the cost of ensuring that every student graduates with adequate public speaking skills? (Costs could include smaller classes, video equipment, faculty mentoring time outside of class, and time in the curriculum taken away from other learning goals.) Consider the 50 percent rule.  For aspirational learning goals, your target should be no less than 50 percent. This 50 percent rule is based on two premises: Most students want to study and learn enough to get a good grade, and it’s very difficult to write hard test questions or assignments (Chapter 19). Under these premises, more than half of students getting a particular test question wrong or failing a particular part of an assignment suggests that something’s wrong. The problem probably lies not with the students but with teaching methods, curriculum design, or the test or assignment itself. Chapter 26 discusses how to address these issues. Tests and rubrics often include both essential and aspirational learning goals.  Many faculty set local standards for tests by using the traditional practice of setting 90 to 100 percent correct as an A, 80 to 89 percent as a B, and so on. A score 301

Understanding and Using Student Learning Evidence

of 65 percent correct is often set as passing or minimally acceptable performance. But this practice can be arbitrary and difficult to justify, because scores depend on how difficult the test items are. A more defensible practice is to set varying standards and targets for each test item or group of items, based on the intended difficulty and importance of the learning goal being assessed (Chapter 19). Do this by first setting targets for each item or group of items. Test items that assess essential learning goals should have a high target: Close to 100 percent of students should answer correctly. Those that assess aspirational learning goals will have a lower target – but no lower than 50 percent answering correctly, as discussed above. The same process can be used with rubrics, which often include a mix of essential and aspirational traits. For essential traits, you may set a target that almost everyone – say, 90 percent – should achieve at least the minimally acceptable performance level. For aspirational traits, you may be satisfied with a lower target. As discussed in Chapter 23, it’s not always necessary or useful to calculate an overall score for a test or rubric. If you don’t need to calculate an overall score, you don’t need to set a standard or target for the overall score. If you do need to set a standard for an overall test score, use the targets you’ve set for individual test items or clusters of items to arrive at a standard. Let’s say, for example, that you’re giving students a test. Half the items are on essential learning goals that almost everyone – let’s say 95 percent – should answer correctly. The other half are on aspirational learning goals that you expect perhaps 75 percent to answer correctly. This would suggest a standard for the test: An overall score of 85 percent correct represents minimally acceptable performance. You might set a target that at least 90 percent of students earn a score of at least 85 percent. (And, yes, what makes this discussion confusing is that we’re talking about percentages in two different ways: The percentage of items that students answer correctly – the standard – and the percentage of students whom we want to reach that standard – the targets.) Setting a standard for an overall rubric score is a bit more complicated than setting one for a test, because there are three ways to calculate the overall score. They are discussed in Chapter 23. Consider setting targets for exemplary as well as minimally adequate performance.  There’s probably no faculty member alive who would be satisfied if every student performed at a minimally adequate level and none did any better. So consider setting a second target for the proportion of students for whom you’d like to see exemplary performance. You might set a target that 90 percent of your students meet your minimally adequate standard and a second target that at least 30 percent meet your exemplary standard.

302

Once you have set initial standards and targets, see how a full group of students does and whether your standards and targets appear to be appropriate. After you’ve scored tests, for example, look through a sample of them to confirm that your standard makes sense. If you set a standard that a test score of 85 percent represents minimally adequate work, for example, look at some of the tests scoring near 85 percent correct. Do the tests scoring 85 percent or 86 percent indeed represent what you consider to be minimally adequate learning? Are those students adequately prepared for what comes next? If your standards appear appropriate but student achievement falls considerably short of your targets, consider revising the targets to set both destination and milestone targets. Suppose, for example, that your target is that 85 percent of your students meet your standard, but only 65 percent do. Make 85 percent your long-term destination target – say three years from now – and perhaps set 75 percent as a milestone target that you aim to achieve next year. As with every other aspect of assessment, view standard- and target-setting as a perpetual work in progress, something to be adjusted as you learn more about your goals, the assessment tools you’re using, and your students’ performance.

Setting Meaningful Standards and Targets

Standard- and target-setting as iterative processes

CHAPTER 22

Consider setting targets as a range rather than an arbitrary number.  A single target, perhaps that 85 percent of your first-year students will achieve your minimal standard for writing, can come across as arbitrary. After all, your college will ­probably not close if only 84 percent achieve that standard, and there will probably not be fireworks if 86 percent do. So consider setting a range of targets. The lowest point in the range might be called the critical level – the point that raises a red flag signaling unacceptable student learning outcomes that must be addressed. The top end is the aspirational target – the point at which you say, “This is, realistically, very good for us, and it’s not worth investing more to improve it any further” (Jose Jaime Rivera, personal communication, August 14, 2013).

Time to think, discuss, and practice 1. For each of the following scenarios, discuss whether the department should conclude that it is achieving its goal. Are you able to decide this? If not, what more do you need to know in order to make a decision? Be specific! a. One of the Foreign Language Department’s goals is for its seniors majoring in French to surpass seniors nationally on the Major Field Test in French. This year 55 percent of seniors scored above the national average on the test. 303

Understanding and Using Student Learning Evidence

304

b. One of the Social Work Department’s goals is for its graduates to succeed in graduate study. A survey of alumni who graduated five years ago shows that 40 percent have earned an MSW. 2. For each of the following learning goals, faculty have not yet decided how they will be assessed. They want to first decide which kind(s) of benchmark or standard would be most appropriate – and realistically feasible – for interpreting the evidence. Then they will choose an appropriate assessment. What kind(s) of benchmark or standard would you recommend? Why? a. One of the Foreign Language Department’s goals is for seniors in its Arabic program to be fluent in reading, writing, speaking, and listening in Arabic. b. One general education goal is for students to think critically and analytically. c. One institutional learning goal is for students to develop a stronger sense of self-identity, including a greater understanding of their personal values.

CHAPTER 23

Summarizing and Storing Evidence of Student Learning Some valuable ideas you’ll find in this chapter Most evidence of student learning can be summarized with simple tallies and percentages. Word clouds are easy, fast tools for informal summaries of qualitative evidence. Keep a small sample of good, bad, and mediocre student work on file to provide firsthand evidence of your standards and rigor.

305

Understanding and Using Student Learning Evidence

A

fter weeks or months of careful planning, your assessments have finally been conducted. Now what? Before you can analyze your evidence of student learning (Chapter 24), share it with others (Chapter 25), and use it to launch conversations about its implications (Chapter 26), you’ll need to summarize your evidence. This chapter discusses several ways to do so, while Chapter 25 discusses how to summarize evidence in tables and graphs. This chapter also discusses documenting and storing evidence of student learning.

Planning your summary How you summarize evidence depends on the kinds of standards or benchmarks you are using and the kinds of evidence you have.

What kinds of standards and targets are you using? In order to be meaningful and useful, your evidence will need to be compared against standards and targets, as discussed in Chapter 22. You’ll need to summarize your evidence in a way that mirrors those standards and targets. For example, if your target is that 75 percent of your students surpass a minimum performance level that you have established, your evidence should be summarized as the percentage of students who surpass that level.

What kinds of evidence do you have? Evidence of student learning falls into five categories that affect how it can be summarized and analyzed: Qualitative  evidence is open-ended, text-based evidence, such as reflective writing (Chapter 20), notes from focus groups and interviews, and responses to open-ended survey questions. Categorical  (or nominal) evidence breaks students into discrete categories. Students’ majors, hometowns, and responses to multiple-choice items are examples of categorical information. These kinds of information can be tallied, but means or medians cannot be calculated. Ordered  (or ranked or ordinal) evidence can be put in a meaningful order. Many rubrics (Chapter 15) and survey rating scales (Chapter 20) yield ordered evidence. Medians can be calculated, and evidence can be analyzed using non-parametric statistics. 306

Summarizing and Storing Student Learning Evidence

Dichotomous  evidence has only two possible values; pass/fail grades and full-time/ part-time status are examples. Although dichotomous evidence is categorical, some researchers assert that it can be treated as scaled evidence because there are only two values with only one interval between them.

CHAPTER 23

Scaled  (interval and ratio) evidence is numerical and the difference between, say, a 1 and a 2 is the same as the difference between a 4 and a 5. Grade point averages, earned credits, salaries, and retention, graduation, and job placement rates are examples of scaled evidence. Means can be calculated, and evidence can be analyzed using a wide variety of powerful statistical techniques. A topic of hot debate among social science researchers is whether evidence from rating scales and rubrics can be treated as scaled rather than ordered evidence, as doing so makes possible a far broader range of statistical analyses. Some researchers assert that we can’t be sure that the degree of difference between, for example, Strongly Agree and Somewhat Agree on a Likert scale (Chapter 20) is the same as the difference between Somewhat Agree and Slightly Agree, so evidence from rating scales should be considered ordered, not scaled (Association of American Colleges & Universities, n.d.). But other researchers argue that statistical analyses designed for scaled evidence are sufficiently robust that they can be used to analyze ordered evidence. These researchers point to the multitude of published journal articles using such analyses with ordered evidence as substantiation that this approach is widely accepted.

Summarizing quantitative evidence Quantitative evidence of student learning can be summarized with tallies, percentages, aggregates, averages, sub-scores, and performance indicators.

Tallies Tallies are straightforward counts of how many students earned each rating or chose each option. Table 23.1 is an example of a tally for assessments of 20 students by their peers using the rating scale in Exhibit 21.1 in Chapter 21. If you’re using a test, you can tally how many students answered each test question correctly. If the test is multiple-choice (Chapter 17), you can tally how many students chose each option of each test item – information that can be used to evaluate the quality of each test item, as discussed in Chapter 24.

307

Understanding and Using Student Learning Evidence

Table 23.1:  A Tally of Assessments of Students by Their Peers Almost always

Often

Sometimes

Rarely

1. Did his or her fair share of the work.

16

2

1

1

2. Participated actively in the group’s activities.

10

4

3

3

3. Contributed useful ideas, suggestions, and comments.

13

4

0

3

8

4

4

4

5. Appreciated others’ ideas.

14

1

3

2

6. Asked others to clarify their ideas, if necessary.

15

3

1

1

7. Expressed disagreements respectfully.

13

2

2

3

8. Did not dominate the conversation or interrupt others.

17

1

1

1

9. Tried to help the group reach consensus.

12

6

1

1

10. Helped the group stay on the topic.

13

3

3

1

11. Helped the group not waste time.

18

1

1

0

12. Helped me learn more than if I had worked alone.

12

3

2

3

This group member . . .

4. Listened carefully.

Percentages Percentages are easier to understand and more meaningful than tallies. Few people will understand the implications of 125 students passing a test; more will care that only 43 percent did. Table 23.2 provides percentages for the tallies in Table 23.1. Percentages also make it easier to see differences between groups of different sizes, such as your current class compared to a class four years ago or your students compared to peers at other colleges (Exhibit 25.2 in Chapter 25 is an example).

Aggregates: Summing evidence into overall scores Quantitative evidence can usually be aggregated into a single overall score. The main reasons to do so are to give individual students feedback on their overall performance on a learning activity and to help faculty determine grades. Otherwise, think twice about calculating overall test, rubric, or survey scores; these calculations are extra work, and overall scores are generally not helpful in understanding and 308

Often

Sometimes

1. Did his or her fair share of the work.

80%

10%

5%

5%

2. Participated actively in the group’s activities.

50%

20%

15%

15%

3. Contributed useful ideas, suggestions, and comments.

65%

20%

0%

15%

4. Listened carefully.

40%

20%

20%

20%

5. Appreciated others’ ideas.

70%

5%

15%

10%

6. Asked others to clarify their ideas, if necessary.

75%

15%

5%

5%

7. Expressed disagreements respectfully.

65%

10%

10%

15%

8. Did not dominate the conversation or interrupt others.

85%

5%

5%

5%

9. Tried to help the group reach consensus.

60%

30%

5%

5%

10. Helped the group stay on the topic.

65%

15%

15%

5%

11. Helped the group not waste time.

90%

5%

5%

0%

12. Helped me learn more than if I had worked alone.

60%

15%

10%

15%

Rarely

Summarizing and Storing Student Learning Evidence

Almost always

This group member . . .

CHAPTER 23

Table 23.2:  A Tally of Assessments of Students by Their Peers Presented with Percentages

improving student learning. If you’re using an analytic rubric (Chapter 15), for example, the percentage of students at each performance level of each trait is typically far more useful than a total rubric score. Three models for calculating an overall rubric score (Coombs, 1964; Steiner, 1966) are summarized in List 23.1. Of course, when rubrics are used for grading, tell students in advance which model will be used. List 23.1  Models for Calculating Overall Rubric Scores ■■

■■

■■

Compensatory: Scores on each rubric trait are summed or averaged into an overall score or grade, so a low score on one trait can be offset by a high score on another. Consider calculating a median rather than mean overall score (both discussed in the next section of this chapter), as it is more stable and less likely to be pulled by one extreme score (Brookhart, 2013). Conjunctive: Students must earn a passing or satisfactory rating on every rubric trait in order to earn a passing or satisfactory score or grade on the assignment. If the student passes this hurdle, use the compensatory model to arrive at an overall grade. Disjunctive: Students must earn a passing or satisfactory rating on certain rubric traits in order to earn a passing or satisfactory score or grade on the assignment. For example, you may decide that grammar is an essential trait, and students must earn a satisfactory rating on the grammar trait of a rubric in order to earn a passing grade on the assignment. If the student passes this hurdle, use the compensatory model to arrive at an overall grade.

309

Understanding and Using Student Learning Evidence

To see how the models differ, consider the rubric in Exhibit 23.1, scored for a paper by Melissa, a hypothetical student. Under the compensatory model, the paper’s scores for each trait (the boxes in boldface) sum to a total score of 80 or a grade of B−. Under the conjunctive model, however, the Unsatisfactory score for Writing Use/Mechanics means that the paper earns a failing grade. If Melissa’s professor deems Writing Use/Mechanics an essential trait under the disjunctive model, the paper earns a failing grade. But if her professor deems APA Rules the only essential trait of the paper, under the disjunctive model her paper earns a grade of 80 or a B−. Give each rubric performance level a single numerical value, not a range.  To facilitate calculating an overall score or grade, faculty and staff often give each performance level (the heading at the top of each rubric column) a numerical value. A fair, equitable rubric has a single numerical value, not a range of values, for each trait’s performance level, because a score in a range can be chosen in an inconsistent, unfair, or biased fashion (Selke, 2013). Furthermore, the rubric doesn’t give students clear feedback on why they earned a score at the top or bottom of the range. If you think you need to give some students at a particular performance level more points than others, split the performance level into two columns, one with a higher point value, one with a lower. Weight rubric traits if you like.  Feel free to weight traits that you view as more important with higher numerical values. The rubric in Exhibit 23.1 does this. But see if weighting rubric traits impacts students’ overall scores or grades on the assignment. Weighting is extra work and, if it doesn’t really affect grades, it’s not worth it.

Averages Averages are numbers that summarize the central tendency of the results of an assessment. The best-known average is the mean: The arithmetic average we learned in grade school. It’s an appropriate statistic for scaled evidence. If your evidence is ordered, a more appropriate statistic is the median – the midpoint of all results when they are listed from highest to lowest – although, as noted earlier, some researchers are comfortable using means with ordered evidence. Medians are a good choice with ordered or scaled evidence when a few very high or very low results distort the mean. Suppose, for example, that 20 students take a national certification exam whose passing score is 80. Eighteen students score 85 and pass comfortably, but two students fail miserably, scoring 10. The mean of these 20 students is 77.5, which gives the incorrect impression that the “average” 310

All APA rules are followed for citations, headers, numbers, series, quotes, references, and so on. (10 points)

APA Rules (10 points)

11 or more violations of APA rules and/or 6 or more missing or incorrect citations and references. (3 points)

More than 16 errors across the paper make it difficult to follow. (3 points)

Writing style makes more than 4 sections of the paper difficult to read and understand. (3 points)

Fewer than 3 original ideas related to the topic are presented OR all ideas are not well explained. (12 points)

Organization is unclear in 4 or more sections. (12 points)

Adapted with permission from a rubric adapted by Sharon Glennen and Celia Bassich-Zeren, Department of Communication Sciences and Disorders, Towson University

4–10 violations of APA rules and/ or 3–5 missing or incorrect citations and references. (5 points)

The paper is free of spelling, syntax, formatting, and punctuation errors. (10 points)

Writing Use/ Mechanics (10 points)

Fewer than 3 violations of APA rules, or 1–2 missing or incorrect citations and references. (7 points)

Syntax or vocabulary is complex, awkward, or filled with jargon in 3–4 sections of the paper OR words are used incorrectly in 3–4 sections of the paper. (5 points)

Syntax or vocabulary is complex, awkward, or filled with jargon in 1–2 sections of the paper OR words are used incorrectly in 1–2 sections of the paper. (7 points)

Tone is professional, vocabulary and syntax are mature, and easy-to-understand terms are used throughout the paper (10 points)

Writing Style (10 points)

The paper has 6–15 spelling, punctuation, formatting, syntax errors. (5 points)

Ideas are presented but in a vague, generic format OR rationales for 2 or more ideas are weak. (16 points)

Specific ideas are presented but the rationales for 1 idea may be weak. (18 points)

Specific ideas for improving research or other ideas are presented in an organized manner with logical rationales. (20 points)

Conclusion/ Original Thought (20 points)

The paper has fewer than 5 spelling, punctuation, formatting, and syntax errors. (7 points)

Organization is unclear in 2–3 sections OR headers and preview paragraphs or sentences are missing. (16 points)

Organization is unclear in 1 section (unfocused paragraphs, poor topic sentences, poor transitions). All other sections are logically organized. (18 points)

Organization is clear; good framework. Headers, preview paragraphs, topic sentences, and transitions aid in understanding main points. Information is presented logically. (20 points)

Organization (20 points)

The paper is unclear and difficult to understand across 4 or more sections. (12 points)

Information is unclear and difficult to understand in 2–3 sections. (16 points)

Information is unclear and difficult to understand in 1 section. (18 points)

Information is presented clearly, completely, and accurately across all sections. At least 3 major sections; at least 1 major section has 2–3 subsections. (20 points)

Content (20 points)

Inadequate (F) The introduction is disorganized and difficult to follow. The main argument and the author’s views are not introduced. (5 points)

Barely Adequate (C) The introduction presents the main argument and the author’s views but is disorganized and does not flow smoothly. (7 points)

Good (B) The introduction is organized but does not adequately present the main argument or does not state the author’s views. (8 points)

The introduction smoothly pulls the reader into the topic, is organized, presents the main argument clearly, and states the author’s views. (10 points)

Introduction (10 points)

Exemplary (A)

Boldface statements and numbers are earned by Melissa, a hypothetical student.

Exhibit 23.1  A Scored Rubric for a Research Report in Speech-Language Pathology/Audiology

Understanding and Using Student Learning Evidence

student failed the test. The median of 85 better reflects the performance of the typical student in this group. If your evidence is categorical, the appropriate statistic is the mode: The most frequent result.

Sub-scores for each learning goal or trait Sometimes several items on a test or survey address a shared learning goal or trait. Perhaps three survey items all address students’ attitudes toward diversity or five test items assess students’ ability to apply their knowledge to solving problems. Aggregating test or survey results into sub-scores for each learning goal or each trait of a learning goal can help us understand student learning, as discussed in Chapter 24. Test sub-scores can be determined by mapping item results back to the test blueprint you used to design the test (Chapter 17). Suppose that the biology faculty have written a 15-item test for a general education biology course. The test has five items each on three learning goals: Understanding the interrelationship of components of the scientific method, using key vocabulary appropriately, and quantitative reasoning. Table 23.3 Jargon Alert! summarizes the results mapped back to these Performance Indicators, Metrics, Performance three learning goals. Measures, Key Performance Indicators, and Dashboard Indicators

Performance indicators, metrics, and performance measures are quantitative measures of student learning or other aspects of college performance that are distilled down to single numbers such as a percentage or average. Performance indicators that are particularly critical to monitoring quality and effectiveness are called key performance indicators (KPIs) or dashboard indicators. When referring to performance indicators, this book generally uses the term measure.

Performance indicators Busy college leaders, board members, and government policy makers like performance indicators (see Jargon Alert) because they are quickly and easily digested. But performance indicators are rarely used to

Table 23.3:  Biology Test Outcomes Mapped Back to the Test Blueprint Learning Goal

Test Items Addressing This Learning Goal

Scientific method

4, 7, 11, 12, and 15

80%

Key vocabulary

1, 3, 6, 8, and 9

84%

Quantitative reasoning

2, 5, 10, 13, and 14

48%

Average Proportion of Students Answering These Questions Correctly

Adapted with permission from an example developed by Virginia Anderson, Professor of Biology, Towson University

312

Unless you have just a handful of evidence of student learning, you will probably want to enter your evidence into a database to make it easier to summarize. Today there are many technologies that facilitate data entry, including spreadsheet and database software, statistical analysis packages, software designed to support tests and surveys, learning management systems, and assessment information management systems. A scanner or optical mark reader can be used to enter results from paper bubble-sheet tests and forms. If you are considering such technologies, see Chapter 11 for more information.

Summarizing and Storing Student Learning Evidence

Technological support for summarizing quantitative evidence

CHAPTER 23

share evidence of student learning because such evidence often cannot be distilled down to a single meaningful number. Most assessment tools, including rubrics (Chapter 15) and surveys (Chapter 20), yield evidence of student learning for multiple traits or criteria. But we cannot ignore increasingly vocal calls for simple, clear summaries of evidence of student learning that public audiences such as employers and policy makers can easily absorb and understand. Chapter 25 offers suggestions for doing this.

Summarizing qualitative evidence Qualitative evidence of student learning such as reflective writing (Chapter 20), open-ended survey questions, and focus group records can be summarized through quick read-throughs, grouped listings, and thematic analysis (Merriam & Tisdell, 2016). Quick read-throughs.  The fastest way to get a sense of responses to brief qualitative assessments is to read through them quickly for general impressions. Scanning minute paper responses (Chapter 20), for example, may give you a good sense of what your students are finding especially difficult. If you have too many responses to read them all, look at a random sample. The problem with quick read-throughs, however, is that your impressions may not match reality. Three students out of fifty, for example, may make comments that make a deep impression, perhaps because their comments are unexpected or confirm a prior hunch of yours. You may be more likely to remember and act on their comments than on something mentioned by eight or ten students that didn’t leave as much of an impression.

313

Understanding and Using Student Learning Evidence

Grouped listings.  If you have brief responses that fall into reasonably discrete categories, you may wish to list the responses in a word processing document, and then use the program’s cut-and-paste function to group them into categories. List 23.2 provides qualitative feedback on an assessment workshop that’s been grouped according to the workshop’s main topics. This simple summary makes clear that participants most often mentioned rubrics as the most useful thing they learned about. Word clouds.  Word clouds (see Jargon Jargon Alert! Alert) can be quickly generated using free online word cloud generators; find one by Word Clouds doing an online search for word cloud. Word clouds are visual images of words, with the most frequently used words in the largest fonts. The shortcoming of word clouds is that they analyze the frequency of individual words, not phrases, so they may not convey the meaning or intent of what has been written. A word cloud of the responses in List 23.2, for example, generates rubrics in the largest font, followed by writing, multiple-choice, self-reflection, and will in the second-largest font. The word cloud thus accurately captures participants’ three biggest takeaways – rubrics, multiple-choice, and self-reflection – but it includes extraneous words as well. Word clouds are therefore probably best used when you want only a quick, informal qualitative analysis. Use them in place of or to confirm a quick readthrough, or to help you decide if it’s worthwhile to spend time preparing a grouped listing or a thematic analysis. Thematic analysis.  More extensive qualitative evidence, such as reflective essays and transcriptions of focus group interviews, don’t lend themselves to simple groupings. Such evidence may be analyzed using thematic analysis (Merriam & Tisdell, 2016), which synthesizes qualitative evidence into a holistic description by organizing text into categories or groups and looking for common themes, patterns, links, and relationships. For more information on thematic analysis, talk to social sciences faculty at your college who are interested in qualitative research. Ensuring consistent, fair qualitative summaries.  Summarizing qualitative evidence is more subjective than summarizing evidence from quantitative assessments. Careful, consistent, informed judgment is essential. Suppose that one of the c­ omments in List 23.2 was “Using rubrics for student self-ratings.” Would this be categorized under rubrics, self-reflection, both, or in a separate category? These kinds of decisions must be made thoughtfully and consistently.

314

What Was the One Most Useful Thing You Learned in This Workshop?

Multiple choice (9 comments): • Tips for writing multiple-choice questions • Creating multiple-choice questions • Multiple-choice tests – I haven’t used in a long while and will rethink. • The criteria for writing good MC tests • How to avoid pitfalls in the writing of multiple-choice exams • Options for multiple-choice questions • Interpretive exercises – I think these will be most useful. • I learned how to use scenarios effectively! • Interpretive exercises may work well in my comparative literature course, where I usually emphasize essay writing.

Summarizing and Storing Student Learning Evidence

Rubrics (13 comments): • Characteristics and advantages of different types of rubrics • Descriptive rubrics seemed useful • Examples of rubrics • Reinforcing understanding and validity of rubrics • Using rubrics • Rubrics are a very good thing when instructors buy into using them. • Developing rubrics • Holistic scoring • The three different kinds of rubrics and how to begin writing them (I’m going to begin soon!) • How to construct a rubric – from basic to complex • Creating rubrics may be an excellent collaborative exercise by which department colleagues establish common goals and compare their grading definitions (scales). • The potential range and flexibility of rubrics • Examples of different types of rubrics and examining purpose of rubric to select one

CHAPTER 23

List 23.2  A Summary of Qualitative Participant Feedback on an Assessment Workshop

Self-reflection (5 comments): • The self-reflection info will really work for my students. • Reflective writing – I think these will be most useful. • The role of self-reflection and metacognition • Role of self-reflection in assessment as a strategy • Examples of self-reflection questions General/miscellaneous (3 comments): • How to process and assess the assessment tools we use on a daily basis • Great tips and tools for assessing student learning • That assessment encompasses test design and grading

315

Understanding and Using Student Learning Evidence

Taking steps to ensure consistent, appropriate categorizations of qualitative evidence is particularly important if the evidence is part of a major, important assessment effort. In this situation, create some written rules or examples for categorizing ambiguous responses. After all the evidence has been categorized, review your decisions to make sure they have been consistent. Better yet, consider having two people independently read and categorize responses and compare. It’s also a good idea to corroborate conclusions from qualitative assessments by comparing them against evidence from other assessments, as discussed in Chapter 24. Technological support for summarizing qualitative evidence.  If your qualitative evidence is voluminous, consider using qualitative research software to summarize it. After responses are keyed in, qualitative research software typically highlights words and phrases and counts their frequency. Some software also groups all responses on each theme together to facilitate review and analysis. For information on such software, speak to a social science faculty member interested in qualitative research, or do an online search for “qualitative research” or “qualitative analysis.” If qualitative research software isn’t available to help analyze voluminous responses, choose a sample of responses (Chapter 12) and develop a simple coding scheme to examine them. You might, for example, code reflections mentioning writing as the most important thing learned as 1, code those mentioning oral presentation skills as 2, and so on. Or use an assortment of highlighting colors to code each theme.

Documenting and storing evidence If evidence of student learning is not recorded, it cannot be shared; and if it is not shared, it cannot be discussed and used (Bresciani, 2006). If records of evidence are not maintained, progress cannot be tracked and memory is lost. There are plenty of stories of lost evidence of student learning, forcing everyone to start from scratch. So a certain amount of recording is unavoidable. Make sure your evidence is securely saved, with backups! Assessment information management systems, which can ease the burdens of documentation and storage, are discussed in Chapter 11. In addition to saving summaries of evidence, you may also wish to consider saving the following: Raw data  such as each response to each test question or survey item and each rating on each rubric trait. Delete identifying information, such as faculty or student names or identification numbers, to ensure confidentiality. As discussed in Chapter 3, if you want to preserve identifying information, store raw data very securely in a password-protected file. 316

Summarizing and Storing Student Learning Evidence

Samples of student work.  Sometimes an accreditor expects or requires that student work be kept on file. If these documents have identifying information, such as faculty name or student identification number, remove such information or store these documents very securely. If you are not required to keep all student work on file and you don’t have a learning management system to do so, think twice, as maintaining these files can add significantly to the burden of assessment. Consider instead keeping just a few samples of student work. They can help you make sure that your standards aren’t inadvertently slipping over time, and they can also provide firsthand evidence of your standards to accreditors and other external audiences (“This is an example of a paper we consider minimally acceptable.”). If you decide to keep samples of student work, be sure to keep a representative sample, not just your students’ best work. Samples of student work that you judge minimally acceptable and inadequate are powerful evidence of your standards and rigor.

CHAPTER 23

Notes on coding.  If any information is coded (say, ratings of Excellent are coded as 5), keep careful notes explaining every piece of information in your files and the meaning of each code. The notes will minimize confusion and will be invaluable should anyone decide to repeat the assessment later.

Time to think, discuss, and practice 1. Discuss whether evidence from the following assessment tools is qualitative, categorical, ordered, scaled, or dichotomous: a. Exhibit 7.2 in Chapter 7: A Rating Scale Rubric for Evaluating College-Wide Student Learning Assessment Processes b. Exhibit 15.1 in Chapter 15: A Checklist for Safe Culinary Practices c. Exhibit 15.2 in Chapter 15: A Structured Observation Guide for a One-Act Play d. Exhibit 18.2 in Chapter 18: A Rubric for Assessing Portfolios from a Graduate Course on Assessment Methods 2. Think of an assignment that you’ve either assigned students or that you’ve completed as a student. Which do you think would be the most appropriate model for grading it: compensatory, conjunctive, or disjunctive? Why?

317

Some valuable ideas you’ll find in this chapter

CHAPTER 24

Analyzing Evidence of Student Learning To understand why students haven’t achieved your learning goals as well as you had hoped, look at additional information. When in doubt, simply ask your students why they didn’t do better on an assessment. They’ll be happy to tell you what you did wrong! If your top students do worse on a multiple-choice item than your bottom students, the item should be thrown out.

319

Understanding and Using Student Learning Evidence

O

nce your evidence of student learning is summarized, you’ll need to analyze it in order to understand its implications. Then you can share your summary and analysis with others (Chapter 25) and use them to launch conversations about next steps (Chapter 26). The most important question you’ll want to answer as you analyze evidence is how well your students have learned what you want them to learn. To understand why they have or haven’t achieved your learning goals, you may need to fold additional information into your analysis. You may also want to analyze evidence to answer questions about stewardship (Chapter 6). And you may want to evaluate the quality of your assessment strategies to see if they’re yielding reasonably accurate information. This chapter addresses all these topics.

Answering the most important question: How well did students learn what you wanted them to learn? The most important question to be answered by evidence of student learning is how well students achieved the learning goals that were assessed. Analyzing evidence from objective tests, rubrics, and rating scales.  Remember from Chapter 23 that total scores from these kinds of quantitative assessments are often not very useful in understanding and improving student learning; what’s more useful is looking at evidence from rubric traits, objective test or rating scale items, or sub-scores for groups of items assessing a shared learning goal. There are two basic ways to analyze these kinds of evidence. One is simply to compare them against one another to identify relative strengths and weaknesses. The biology test results in Table 23.3 in Chapter 23, for example, suggest that student achievement regarding the scientific method and key vocabulary is relatively strong, but quantitative reasoning skills are relatively weak. The shortcoming of this approach is that there will almost always be some item, trait, or sub-score with comparatively weak results, putting you into an endless and perhaps needless cycle of improvement. The other (and better) approach is to compare evidence for each rubric trait, test or rating scale item, or sub-score against targets you’ve established and justified for them (Chapter 22). For example, you may have set a target that 90 percent of senior theses should score at least adequately in terms of organization, but your assessment finds that only 80 percent scored at least adequately in terms of this. With either approach, recognize that some differences may not be large enough to be meaningful. Don’t focus, for example, on comparing the 15 percent of student presentations with inadequate organization to the 17 percent with

320

Analyzing Evidence of Student Learning

Analyzing evidence from qualitative assessments.  See if qualitative evidence (Chapter 20) falls into the patterns you were expecting. You may have hoped, for example, that at least a third of students would report that one of the main things they gained from their service learning experience was an increased sense of compassion for the underserved, but far more – almost half – mentioned this. If you ask students, “What was the most important thing you learned in this learning experience?,” compare their answers against the learning goals of the experience. If students’ answers match the learning goals, you’ve probably delivered the curriculum with appropriate balance. But if a number of students mention ideas that you consider relatively trivial and relatively few mention your biggest themes, you have a clue that the learning experience may not have effectively helped students achieve key learning goals. If you ask students, “What question remains uppermost in your mind as we end this learning experience?” and students’ responses are all over the map, you may be able to conclude that they have no major sticking point. But if a number of students raise questions about a point related to a particular learning goal, you have a clue that the curriculum or teaching methods may need to be modified to help students better achieve that learning goal.

CHAPTER 24

inadequate supporting visuals. Statistical tests can help you decide if differences are large enough to be significant. Keep in mind, however, that a statistically significant difference may not be large enough to have any real practical, meaningful significance. In a survey of 1,500 students, for example, a 4 percent difference between answers to two items may be statistically significant but not large enough to merit special attention. Pointing out only differences of, say, 10 percent or more will focus everyone on the major differences that need discussion and decisions.

Integrating assessment pieces into an overall picture Solitary assessments, all inherently imperfect samples of student learning (Chapter 3), may tell us relatively little about how well students have achieved important learning goals. It’s when we integrate evidence from those assessments with each other that we get information that we can use with greater confidence. This integration need not be quantitative. In fact, quantitative aggregations of evidence of student learning can lead to comparing apples and oranges, such as writing skills of students in the humanities and in engineering. Or it can lead to pushing square pegs into round holes, such as forcing everyone to use a common rubric that’s not uniformly relevant or appropriate, as discussed in Chapter 9. Instead, integrate evidence through a humanistic, qualitative approach. A committee reviewing program and general education assessment reports, for 321

Understanding and Using Student Learning Evidence

example, might notice that several reports mention that students struggle with data analysis. Such findings, gleaned from qualitative rather than quantitative review, are nonetheless clear and actionable – they can lead to college-wide discussions and decisions on strategies to improve students’ data analysis skills.

If students performed poorly, why? Now take a close look at any evidence of student learning that you’ve found disappointing, and try to figure out why student learning fell short. Sometimes the reason is obvious, sometimes not. Chapter 26 discusses possible reasons for disappointing outcomes, including curriculum design, teaching methods, learning goals, support systems, and co-curricula. Sometimes the design and wording of the assessment itself is the culprit. Sometimes a prompt (Chapter 16) is unclear; sometimes a multiple-choice question (Chapter 17) assesses a relatively trivial concept that many students missed; and sometimes a rubric (Chapter 15) is misinterpreted by the faculty using it to assess student work. Part 4 of this book offers suggestions and tips on how to design effective assessment tools.

Additional information may help you understand your evidence Assessments of student learning tell what students have and haven’t learned, but most of them don’t tell why. Additional information on students’ learning experiences and factors affecting them can help answer questions such as the following: • Why did students learn X but not Y? • Why did some students master Z but others didn’t? • Which courses best prepare students to succeed in the program capstone? • What are the experiences of students most likely to graduate? • What instructional strategies most help students learn to think critically? • Students in which academic programs, if any, have trouble developing quantitative analysis skills? • Which high-impact practices (Chapter 5) most help students develop information literacy skills? Look at learning goals.  They can be analyzed to see how well they meet the traits of effective learning goals (Chapter 4) and how well they align with your assessment tools.

322

Analyzing Evidence of Student Learning

Look at the use of research-informed teaching methods and high-impact practices.  With your colleagues, examine the prevalence of each of the practices in List 26.1 in Chapter 26 and each of the high-impact practices in Chapter 5 in your course, program, general education requirement, or co-curricular experience, perhaps using a coding scheme such as the following: 1. Most students regularly experience this practice. 2. Most students experience this practice, but we might be able to make this even more pervasive. 3. Some students experience this practice, but we should try to make this far more pervasive. 4. Realistically, it would be hard to make this a more pervasive practice. Another helpful source of information is surveys of student engagement such as the National Survey of Student Engagement (nsse.indiana.edu) and the Community College Survey of Student Engagement (www.ccsse.org). Or you might ask students to keep journals or logs documenting the extent of their engagement in student learning, such as time spent on coursework, time spent in active hands-on learning, or interactions with faculty, staff, and other students. Look at other information on student experiences such as the following: • Student majors • Student participation in service learning activities or field experiences • Student participation in relevant co-curricular experiences (for example, biology majors’ participation in the Environmental Club) • Student attendance at co-curricular events such as disciplinary seminars and conferences or cultural events • Student participation in tutoring and other forms of academic support • Student satisfaction with learning experiences and support services

CHAPTER 24

Look at curricular materials.  Curriculum maps (Chapter 5), course catalog descriptions, and syllabi can help you figure out whether students are getting enough opportunities to develop and achieve the learning goals being assessed.

Look at information on student progress, persistence, and success  such as grades and retention, transfer, graduation, and job placement rates. Compare the transcripts of students who performed well and poorly on your assessment. Are there any patterns in the order in which they took required courses? In the grades they earned on those courses? In the elective or pick-from-a-list (Chapter 5) courses they chose?

323

Understanding and Using Student Learning Evidence

Jargon Alert! Student Achievement The 1998 Higher Education Act requires accreditors recognized by the U.S. federal government to require the colleges they accredit to demonstrate “success with respect to student achievement in relation to the institution’s mission, including, as appropriate, consideration of course completion, state licensing examinations, and job placement rates” (1998 Amendments to the Higher Education Act of 1965, Title IV, Part H, Sect. 492(b)(4)(E)). This language has remained largely in place through subsequent amendments and acts. The examples in this statement imply that the federal government defines student achievement as a combination of student learning, course and degree completion, and job placement.

Look at what students bring to the learning experience.  Student traits and prior learning experiences can yield insight into why some students have or haven’t achieved key learning goals. Examples of helpful information include: • Demographic data such as full-time/ part-time status, geographic origin, employment status, gender identity, and racial/ethnic status • Transfer status, including the college(s) the student transferred from, coursework completed there, and grades earned there • Placement test scores • High school records, including curriculum and grades

Ask your students!  If you’re still not sure why students performed poorly on a particular assessment, ask them! In an appropriate class, focus group, or other setting, tell them honestly, “A lot of you got that test question wrong or did poorly on that aspect of the assignment. We didn’t think it would be that hard. Why it was so difficult?” ­Students will give perceptive and useful replies! If you’re looking at multiple-choice test outcomes, well-designed distracters can diagnose where students went wrong with Jargon Alert! their thinking, as discussed in Chapter 17. New Data Analysis Tools: Learning Analytics, Big Data, Data Mining, and Predictive Analytics These concepts are so new that they don’t yet have widely accepted definitions, but here are some stabs at what they mean in higher education contexts. Big data refers to large amounts of data on students and student learning that can be analyzed to reveal patterns, trends, and relationships. Learning analytics and data mining use information science and other techniques to extract, analyze, and report information from big data that can be used to improve student learning and success, individually and collectively. Predictive analytics use data mining techniques to make predictions, such as on the traits of students who will learn best in an online setting or on the traits of students whose quantitative skills will improve most with face-to-face tutoring.

324

If you can, use learning analytics to connect the pieces.  Thanks in part to the growth of online education and the growing sophistication of many learning management systems (Chapter 11), increasing numbers of colleges have enormous arrays of data on their students, curricula, and support services as well as evidence of student learning. Learning analytics (see Jargon Alert) can facilitate the integration of evidence of student learning with other student data into an overall picture that explains why students have or haven’t achieved key learning goals.

Institutional Effectiveness and Institutional Assessment

Answering another key question: Stewardship

As discussed in Chapter 6, assessment has a second purpose beyond helping students get the best possible education: • Meeting stakeholder needs, especially student needs stewardship, or ensuring that good-quality • Serving the public good evidence, including evidence of student • Stewardship learning, is used to ensure the prudent, • Accountability effective, and judicious care and use of resources. Accountability, a third purpose Institutional assessment gauges institutional effectiveness. Because student learning is the heart of assessment, is discussed in Chapter 25. of most college missions, the assessment of student Stewardship and accountability are both learning is a major component of institutional assessment. facets of institutional effectiveness (see Jargon Alert). You may recall Michael, who earned a test score of 55 in a scenario described in Chapter 22. Suppose that it cost Michael’s college $400 (in salaries, benefits, facilities, equipment, overhead, and so on) to provide him with the education that led to his score of 55. Emily, enrolled in another section of the same course, also earned a 55 on the same exam, but at a cost of $300. A prudent college would ask why Emily was able to achieve the same outcome at lower cost and whether her experience might be replicated or adapted. Information that might be used in concert with evidence of student learning to analyze stewardship includes the following: • Opportunities, expenditures, and participation rates for faculty and staff professional development on teaching and learning • Number and dollar value of grants awarded to improve student learning (Chapter 14) • Funding for academic programs, support services, and co-curricular experiences Institutional effectiveness refers to the effectiveness of a college in achieving its mission, goals, and the following responsibilities (Suskie, 2014):

Analyzing Evidence of Student Learning

Jargon Alert!

CHAPTER 24

If your college does not yet have a learning analytics capability, you can still analyze and integrate information in less formal ways. Don’t aim to assemble all the information suggested here – you’ll be overwhelmed. Develop some hypotheses on why your outcomes are disappointing, and collect whatever information seems most likely to help you check your ideas. If your data are scaled or at least ordered (Chapter 23), inferential statistics can help you identify relationships or make predictions. If you have other kinds of data, or if you don’t have the capacity to conduct inferential statistical analyses, don’t worry – simple reviews of relevant information can often yield helpful insight.

325

Understanding and Using Student Learning Evidence

• Instructional facilities, technologies, and other resources • Class sizes, student/faculty ratios, and/or ratios of students to full-time faculty

Documenting and analyzing the effectiveness of your assessment strategies Whether you need to spend time documenting and analyzing the effectiveness of your assessment tools and processes depends on their setting (Chapter 2) and how the resulting evidence will be used (Chapter 6). Assessments used to inform large-scale, expensive, or life-changing decisions or whose resulting evidence is likely to be challenged need compelling evidence of their effectiveness. Examples include assessments used to inform decisions on who graduates, whether elaborate or expensive changes should be implemented, or whether a program should be terminated. It may not be worthwhile to invest time and resources in documenting and analyzing the effectiveness of assessments used to inform relatively minor decisions, such as changes in teaching strategies or in the curricular design of a single course or co-curricular activity. Table 24.1:  How to Document Evidence of Key Traits of Effective Assessment Practices

326

Traits of Effective Assessment Practices (Chapter 3)

How They Might Be Documented

Yielding evidence of student learning that is useful and used in meaningful ways

Decisions informed by evidence of student learning

Focusing on clear and important goals

The learning goals that are being assessed; the process used to develop them; comparisons of the assessment tool (test, rubric, prompt, survey, and so on) against those learning goals

Clarity

Agreement among faculty and staff on how assessment tools are to be used and how the resulting evidence of student learning is to be interpreted

Significance and variety

Copies of assessment tools; reviews by others

Fairness and lack of bias

Reviews by people of varying backgrounds

Ethical conduct

Prompts; announcements to students; evidence regarding scoring errors or bias (Chapter 3)

Appropriate range of outcome levels

Assessment scores; interviews with those doing the scoring

As noted earlier in this chapter, any single assessment may not be a representative, generalizable sample of what students have learned. If you decide that you need to analyze how well your evidence of student learning can be generalized into valid conclusions, consider the following questions: Do other assessments corroborate your evidence?  Students whose writing samples receive high scores, for example, should also receive relatively high scores on a published writing test, if you use one, and relatively high grades in writing-intensive courses. Comparing evidence from various assessments against one another is called corroboration or triangulation.

Analyzing Evidence of Student Learning

Analyzing the generalizability of your evidence

CHAPTER 24

One way to document the effectiveness of assessment practices is simply to keep records showing that they meet the traits discussed in Chapter 3. Table 24.1 offers other suggestions. Strategies to document and analyze the generalizability, dependability, and consistency of student learning evidence are discussed next.

Does evidence fall in appropriate patterns?  Students at the end of a program should generally do better on an assessment, for example, than students at the beginning. Some evidence should predict current or future performance; scores on a pre-calculus test, for example, should predict calculus grades at least somewhat accurately. And sometimes students should perform differently by major. Physics majors, for example, may appropriately score higher on a quantitative reasoning assessment than English majors. How well does your sample reflect all your students?  If you are assessing a sample of student work (Chapter 12), you may want to compare the traits of students in your sample with the traits of the group your sample was drawn from. If 55 percent of all your college’s students are women, for example, roughly 55 percent of the ­students being assessed should be women.

Analyzing the dependability and consistency of your evidence Dependability and consistency (Chapter 3) can be analyzed by considering the following questions. But while dependability and consistency can be important, keep in mind that the other traits of effective assessments discussed earlier are even more crucial (Chapter 3).

327

Understanding and Using Student Learning Evidence

328

Do faculty and staff assess student work consistently?  This can be determined by having two people score each student’s work sample, comparing their scores, and perhaps calculating a statistic representing their interrater reliability. As discussed in Chapter 12, double-scoring student work can be a time-consuming and costly undertaking, so consider whether the time and money spent is a good investment. How precise is your estimate from your sample?  When reporting evidence from samples, you’ll be much more credible if you mention its error margin. Chapter 12 explains how to calculate error margins. How well does each multiple-choice test item discriminate between high and low scorers?  Item discrimination in multiple-choice tests refers to the item’s capacity to discriminate between students who have generally learned what the test covers and those who haven’t. It makes the following assumptions: • The purpose of the test is to discriminate between students who have learned the test material well and those who haven’t. This premise works well when the purpose of the test is to identify a limited number of students who will earn an A, or receive a special honor or recognition, or be allowed to move on to further study. It doesn’t hold for assessments of essential learning goals that virtually all students are expected to achieve. • Test items that don’t help make this discrimination are a waste of testing time. This premise doesn’t hold for easy test items or tests of essential learning on which most students are expected to earn similar, high scores, because if everyone answers an item correctly, there can’t be any discrimination. • Students who do well on the overall test should be more likely to answer any one test item correctly than students who do poorly on the overall test. This premise holds for tests assessing a few closely related learning goals, but it doesn’t hold as well for tests assessing disparate learning goals. Test items on an exam that’s half on vocabulary and half on data interpretation will not discriminate as well as test items on an exam that only assesses vocabulary. • If students who do well on the overall test are less likely to answer a test item correctly than students who do poorly, the item is not working correctly; the top students read too much into the item. It should be thrown out and the tests rescored, as it unfairly penalizes the top students.

Analyzing Evidence of Student Learning

List 24.1  How to Determine the Discrimination of Test Items

CHAPTER 24

It isn’t hard to analyze test items (see Jargon Alert! Jargon Alert) for discrimination. Just follow the steps in List 24.1. Item Analysis Table 24.2 presents item discrimination Item analysis is examining the difficulty and discrimination of each multiple-choice test item. results for a six-item objective test. Item 5 It can also include examining the difficulty and is an example of a question with very good discrimination of each option within each item. discrimination. Only two students in the top group got this question wrong, while eight students in the bottom group got it wrong. This item clearly distinguishes the top students (as defined by this particular test) from the bottom ones. Item 2 is an example of a question with no discrimination; four students in each group got it wrong. Item 6 is an example of a question with negative discrimination; more top students got it wrong than bottom students. As noted earlier, Item 6 should be thrown out and the test rescored.

1. Sort the scored tests from highest to lowest total score. 2. Choose a manageable number of tests with the highest and lowest scores for further analysis. Depending on how many tests you have, you might choose the top and bottom thirds, the top and bottom fifths, or the top and bottom 10 tests. Make sure you have exactly the same number of tests in the top and bottom groups. 3. For each group, count the number of tests answering each item incorrectly (which is usually faster than counting the number answering each item correctly). 4. For each item, subtract the number of incorrect answers in the top group from the number of incorrect answers in the bottom group. This is a simple measure of each item’s discrimination.

Table 24.2:  Examples of Item Discrimination Results Item Number Number Incorrect in Top Group (10 Students)

Number Incorrect in Bottom Group (10 Students)

Difference (Bottom Minus Top)

1

0

0

0

2

4

4

0

3

0

1

1

4

2

5

3

5

2

8

6

6

6

2

−4

329

Understanding and Using Student Learning Evidence

Keep in mind that, in order to interpret item discrimination information correctly, the difficulty of each item must be taken into consideration. In Table 24.2, Item 1 is very easy (everyone in both groups got it correct), so of course there’s no discrimination. Everyone in the top group got Item 3 correct and only one person in the bottom group got it wrong, so while the discrimination of 1 may not seem high, the item is so easy that this is the highest possible discrimination. If you are stumped by why a multiple-choice test item has low or negative discrimination, look at the number of top and bottom students who chose each distracter. Exhibit 17.3 in Chapter 17 includes an example of this kind of information. Sometimes top students misinterpret and therefore choose a particular distracter. Test questions with one or two poor distracters can often be revised so they can be reused on future tests. How well does each rubric trait discriminate between high and low scorers?  You can compare student work with the highest overall rubric scores against those with the lowest overall scores in terms of the ratings they earned on each rubric trait. For example, if bottom-scoring essays generally score higher than top-scoring essays in terms of organization, you have an important clue that something isn’t right with the assignment or with how faculty and staff scored the essays.

For More Information This chapter is a very brief introduction to the complex concepts of inferential statistics, analytics, reliability, and validity. If you’re not knowledgeable about these concepts, there may be people at your college who can help you. Check with information technology staff, institutional research staff, and faculty with backgrounds in statistics or educational or social science research methods. Your college’s institutional research office should be able to advise you on the learning analytics capabilities of your college’s information management systems.

Time to think, discuss, and practice 1. Table 24.3 presents selected outcomes from the National Survey of Student Engagement (nsse.indiana.edu) for a sample of 250 seniors at Rodney College. For which items do you think there is a meaningful difference between outcomes for Rodney seniors and their peers? Why? 2. Ask one member of your group to play the role of the director of the first-year seminar at Bridgeview College, which is designed to prepare students to succeed in subsequent coursework by developing their information literacy skills. It is a somewhat costly offering, so faculty must demonstrate that students are 330

Percent of Seniors Reporting That They Never Did This During Their Senior Year At All Participating Rodney College Peers

Asked questions or contributed to course discussions in other ways

4%

2%

Prepared two or more drafts of a paper or assignment before turning it in

26%

20%

Asked another student to help you understand course material

6%

19%

Explained course material to one or more students

2%

8%

Prepared for exams by discussing or working through course material with other students

9%

23%

Worked with other students on course projects or assignments

2%

11%

12%

13%

2%

3%

Examined the strengths and weaknesses of your own views on a topic or issue

10%

2%

Tried to better understand someone else’s views by imagining how an issue looks from their perspective

8%

1%

Learned something that changed the way you understand an issue or concept

4%

1%

Connected ideas from your courses to your prior experiences and knowledge

2%

1%

Discussed course topics, ideas, or concepts with a faculty member outside of class

22%

28%

Gave a course presentation Combined ideas from different courses when completing assignments

Analyzing Evidence of Student Learning

At Rodney College

CHAPTER 24

Table 24.3:  Selected Outcomes from the National Survey of Student Engagement for Rodney College Seniors

achieving these skills at an appropriate level or the seminar risks termination. Faculty are using a rubric to assess students’ library research projects. What strategies in this chapter would you recommend that the director use to document the effectiveness of their assessment strategy? Why?

331

Some valuable ideas you’ll find in this chapter

CHAPTER 25

Sharing Evidence of Student Learning Share student learning evidence in ways that will lead to decisions. The briefer your summary, the more likely people will absorb it. The best way to convey key points is with well-organized, eyecatching visuals. Accreditation reports are cousins of reports to your board.

333

Understanding and Using Student Learning Evidence

T

his book has repeatedly noted that many colleges are sitting on piles of assessment information but not yet using it in meaningful ways. Many of the reasons for this have already been discussed in this book, but there’s one more: Student learning evidence is often not shared in ways that promote its use. Evidence can be put to good use only after careful consideration and discussion, and that can happen only if summaries and analyses of evidence are communicated usefully, clearly, and accurately. How you share summaries and analyses of student learning evidence should therefore be planned as carefully as any other part of the assessment process.

Share evidence in ways that will lead to decisions How can you avoid having assessment reports end up sitting on (often virtual) shelves, unread and unused? The key is to intentionally design reporting structures, processes, and formats to promote discussion and use of student learning evidence (Kuh, Ikenberry, Jankowski, Cain, Ewell, Hutchings, & Kinzie, 2015).

Identify the audiences and decisions that your evidence will inform Many years ago, I gave my college president what I thought was a very nice presentation of the results of a student survey I had conducted. His reaction? “Linda, this is what I call ‘gee whiz’ data. I think, ‘Gee whiz, this is interesting,’ but I don’t know what to do with it.” What I learned from that experience was to know what decisions your audience is facing and share only outcomes that will help inform those decisions. Remember that assessment has three fundamental purposes (Chapter 6): • Making sure students get the best possible education. This is – or should be – the fundamental concern of faculty, staff, and college leaders. • Stewardship. College leaders, board members, and government policy makers are especially interested in using college resources prudently, effectively, and judiciously. • Accountability. Board members and external audiences such as accreditors and government policy makers want to be assured of the effectiveness of your college, program, or other learning experience – in other words, that students are getting the best possible education with appropriate stewardship of resources. If you’re not sure what decisions a particular audience is facing, try to find out before you share evidence with them. Chapter 6 offers many possibilities. Don’t forget that one of the purposes of reports on assessment plans and processes is to help everyone involved in assessment learn what constitutes good practice, as discussed in Chapter 10. 334

different decisions

Sharing Evidence of Student Learning

Good assessment is “driven by genuine inquiry” (Hawthorne & Kelsch, 2012, p. 1) into real questions of interest, which may vary considerably from one audience to the next. Governing boards and government policy makers typically want to see the 30,000-foot picture of student learning at your college, for example, while faculty and staff more likely want to see detailed information on the effects of teaching methods and support on student learning in specific settings. Because different audiences want and need different evidence in different formats and in different levels of detail, no one report, database, or website will meet everyone’s needs (Diamond, 2008; Jankowski, Ikenberry, Kinzie, Kuh, Shenoy, & Baker, 2012). If you have surveyed graduating seniors, for example, perhaps only co-curricular staff need to see seniors’ opinions of co-curricular experiences.

CHAPTER 25

Different people need different information to inform

Share a story with a clear point Share student learning evidence in ways that tell important, coherent, interesting stories with clear points that plainly connect to the decisions your audiences are making. Consider this paragraph from a report on one hypothetical college’s participation in the Cooperative Institutional Research Program’s annual Freshman Survey (heri.ucla.edu/cirp-freshman-survey). Between 78% and 80% of Lawndale College first-year students rely on family financial support. Lawndale first-year students are more likely to have student loans (48%) than first-year students nationally. Only 57% were dependent on grants, compared to 62% nationally.

What’s the point of this paragraph? I don’t know, and if I don’t, the report’s readers probably don’t either, and they aren’t going to spend time puzzling it out. On the other hand, consider these paragraphs on different results from the same survey at another hypothetical college: First-year women at Kensington College have a stronger preparation for college than men. First-year women have, on average, earned higher grades in high school and have spent more time volunteering, studying, and participating in student clubs and groups. They are more likely attending college to “gain a general education,” “learn more about things that interest me,” “become a more cultured person,” and “prepare for graduate study.” First-year men, on the other hand, are less likely than women to have completed high school homework on time and to have come to class on time. They have spent more time in high school watching television and working. 335

Understanding and Using Student Learning Evidence

Men are more likely attending to “make more money” and it is more important to men to “be very well off financially.” Despite their weaker preparation for college, men rate themselves higher in intellectual self-confidence, mathematics ability, popularity, social self-confidence, physical health, and emotional health. First-year men and women at our college both need help adjusting to college, but of vastly different kinds. Women need more self-confidence, while men need more help building academic skills and an appreciation of the broader benefits of a college education.

Jargon Alert! Data and Information Data are a set of numbers (the singular is datum). Information makes clear the story that the numbers are telling.

Because the Kensington report has a clearer point, people will probably pay more attention to it than to the Lawndale one, even though it’s longer. So before you share student learning evidence, use your summary and analysis to figure out the big messages you want to convey to each audience. In other words, plan to share not data but information (see Jargon Alert).

Focus on sharing what’s most important with your most important audiences Yes, preparing multiple reports of student learning evidence takes a while, but it’s better to prepare several short useful reports than one lengthy report that no one looks at and is therefore a waste of time. Make your efforts worthwhile by identifying your most important audiences and understanding what they most want and need to know. Then share only evidence that each audience can act upon. Don’t bother telling your audiences that science majors graduate with stronger math skills than literature majors, for example. But do tell faculty that students at your college spend less time writing than students at peer colleges. If you have some evidence of student learning that no one seems interested in, ask yourself why you collected that evidence in the first place.

View sharing evidence as a teaching/learning process When you share student learning evidence, you are in essence educating your audiences about something important: How well students are learning what you all view as important. In other words, you are a teacher, and sharing student learning evidence is, at its heart, a teaching process; you are helping your audiences learn 336

You can share student learning evidence in ways that people ignore, that leave them confused, or that put them to sleep. Or you can share evidence in ways that immediately engage them in discussing and acting on your evidence. Here are suggestions for sharing student learning evidence in ways that will get it discussed and used.

Sharing Evidence of Student Learning

Create great visuals and reports

CHAPTER 25

what they need to learn about your evidence. Sharing student learning evidence effectively thus means following good teaching practices (List 26.1 in Chapter 26), including the following: • Understand your audiences and their needs. • Help your audiences see a synthesizing big picture: The big point of your assessment story. • Present your evidence in a variety of ways, as discussed in the next section. • Be balanced and fair, as discussed later in this chapter. • Actively involve your audiences, as discussed at the end of this chapter.

Choose the right format for sharing your evidence Today few people have the time to wade through a lengthy written report, and technologies have made possible a host of alternatives. List 25.1 summarizes some of the many ways to share summaries and analyses of student learning evidence.

List 25.1  Formats for Sharing Summaries and Analyses of Student Learning Evidence ■■

■■ ■■

■■

■■

Stacked bar graphs are the best way to share evidence from most rubrics and rating scales. They easily show the proportions of students that earned each rubric rating on each criterion. Exhibit 25.3, which is shown later in this chapter, is an example of a stacked bar graph. Line graphs are best for showing historical trends. Pie graphs are best for showing proportions when you have categorical or qualitative evidence (Chapter 20), such as the proportions of students choosing each response to a multiple-choice question. Exhibit 25.4, which is shown later in this chapter, is an example of a pie graph. Tables are appropriate when your audience needs to focus on specific figures rather than the overall picture of evidence that would be conveyed by a graph. They’re also fine when you lack the skill or technological support to create graphs and other visuals. Tables 23.2 and 23.3 in Chapter 23 and Table 24.3 in Chapter 24 are examples of tables summarizing student learning evidence. Bulleted lists can highlight key points. (continued)

337

Understanding and Using Student Learning Evidence

List 25.1 Continued ■■

■■

■■

■■

■■

■■

Infographics combine visuals and text to convey the key points of complex information. An infographic might, for example, include a graph summarizing evidence of student writing skills from a rubric, a pie chart of student feedback on their writing courses, and a bulleted list of key conclusions drawn from both sets of evidence. Infographics can be created with the help of a growing number of software applications such as Canva, Piktochart, and Venngage (which are offered as examples, not recommendations or endorsements). Hundreds of examples (good and not so good) are available online; click, for example, on the Education or Business links of dailyinfographic.com. Dashboards are sets of performance indicators (Chapter 23) that give a visual snapshot of progress toward a goal. If faculty want to improve general education skills, for example, they might use a dashboard of simple graphs summarizing student learning evidence from rubrics assessing general education learning goals. Effective dashboards are designed so their point can be grasped at a glance, and every performance indicator should be there with a specific decision in mind (Hubbard, 2014). Balanced scorecards are dashboards that aim to provide balanced information on the overall effectiveness of a college, program, or unit. Because the fundamental mission of every college and academic program is to help students learn, balanced scorecards for them include key student learning evidence and not just, say, financial and productivity indicators. Other visuals include scatterplot, bubble, heat map, radar, tree map, sunburst, box-and-whisker, waterfall, funnel, and Gantt visuals. The options for data visualization (see Jargon Alert) can seem overwhelming! But the visuals already mentioned are generally the best choices for sharing student learning evidence, so if you’re not familiar with these kinds of visuals, don’t worry about them. Oral presentations can be the best way to engage your audience in discussing and using student learning evidence; they’re discussed at the end of this chapter. Written reports are sometimes required by external agencies such as accreditors or government agencies. Suggestions for sharing student learning evidence with accreditors are offered later in this chapter.

Which format is best? It depends on what your audiences need to see and how they prefer to receive information. Today Data Visualization people increasingly expect and appreciate Data visualization is presenting data visually to help people understand its meaning and significance. visual information, but a few may prefer a traditional written report. The best approach may be a combination of modes, such as brief oral remarks accompanied by slides, interactive discussion, and a written executive summary of key evidence. Jargon Alert!

Keep it short Today few people have time to wade through a lengthy report, pore over complicated charts, sit through an extensive presentation, or click through endless web pages. 338

CHAPTER 25 Sharing Evidence of Student Learning

Most people want very brief summaries highlighting only key evidence of student learning. The shorter your communication, the more likely people will read, listen to, or view it, and the more likely your evidence will be put to good use. Avoid the temptation, therefore, to report responses to every survey item and every statistical analysis. While you may have found each detail of your assessment project fascinating, your audiences probably won’t. “Most information is useless. Give yourself permission to dismiss it” (Harris & Muchin, 2002, para. 4). Instead, aim to give your audience what they most need . . . and no more. Provide a quick overview of the summaries, analyses, and implications that will be of greatest interest to them – no more than one or two pages, if you’re communicating in writing. Pull out just the outcomes that clearly relate to key learning goals of greatest interest to your audience. Feel free to offer additional information on request. You might note, for example, that your summary is based on a review of a sample of 50 portfolios and that anyone interested in how the portfolios were chosen can contact you for more information. Don’t be disappointed, however, when no one takes you up on your offer!

Keep things simple and clear Simple, uncluttered, and readable summaries and analyses of evidence are more likely to be understood quickly and easily and are therefore more likely to be used. Limit the amount of information  you provide, as already suggested. Limit the number of wedges in a pie graph to no more than six, for example. If you find that you must insert vertical lines into a table to make it clear, you’re trying to cram in too much information. Use bulleted lists  rather than a traditional narrative of paragraphs, because people can scan bulleted lists more quickly. Limit the number of bullets in a list to five or six at most, or people will lose focus. Make every visual self-explanatory,  because some people may share the visual out of context or won’t read an accompanying narrative. Use headings or footnotes in the visual to provide definitions, assumptions, and notes to interpret the visual properly. Label every part of a table or graph clearly, including each table column and each graph axis, avoiding abbreviations. Avoid jargon.  Use consumer-friendly language that everyone will easily understand (Jankowski, Ikenberry, Kinzie, Kuh, Shenoy, & Baker, 2012). Avoid technical 339

Understanding and Using Student Learning Evidence

terms, formulas, statistical symbols, and research jargon such as aggregate, variable, subject, or population. Spell out every abbreviation the first time it is used. Explain statistical analyses in lay terms. The following paragraph, for example, describes a discriminant analysis without using that term: The analysis split the students responding to the survey into three groups: 462 students who returned or graduated, 50 students who were dismissed from the college for poor academic performance, and 92 students who voluntarily left without graduating. The three groups were then compared to identify distinguishing characteristics of each group.

Use numbers sparingly,  only when they’re necessary or inherently interesting. Readers will glaze over a paragraph like the following: Cape Anne College students are more likely to be first generation college students than college students nationally. Exactly 40% of fathers and 48% of mothers have never attended college, compared to 37% of fathers and 40% of mothers nationally. Over 36% of fathers of Cape Anne freshmen are college graduates, while nearly 27% of mothers have college degrees.

Now consider this rewrite which, with no figures, communicates the same point far more clearly and succinctly: The parents of Cape Anne College students are not quite as well educated as those of students nationally; they are less likely to have attended college or earned a college degree.

Round figures  to the nearest whole number. Stating that 10.3 percent of students said X and 10.5 percent said Y encourages audiences to focus on trivial differences and unnecessarily increases the number of digits that they must try to absorb. Have a friend review your draft.  Even if he or she knows nothing about assessment, a friend should be able to understand the key points you’re trying to convey. Don’t assume a software-generated visual is readable.  Visuals can be created with spreadsheet software such as Excel, presentation software such as PowerPoint and Prezi, data visualization software such as Tableau, and some assessment information management systems (Chapter 11). But some default settings generate visuals that won’t convey your key points clearly; you may need to tweak the visuals to make them readable and distortion-free.

340

Use engaging, meaningful, self-explanatory titles, headings, and subheadings  that help convince your audiences to read, view, or listen to your summary and analysis and help them quickly see what is useful to them. Try writing your titles and headings like newspaper headlines. “Students Successfully Integrate Ideas” says a lot more than “Critical Thinking Assessment Results.” Or consider a question-and-answer format; questions like “Why Do Students Fail General Education Courses?” can pique audience members’ curiosity.

Sharing Evidence of Student Learning

Your major points should pop out at readers, so they can easily see the connection between those points and the decisions they’re facing. While this chapter has talked about reports and has provided examples of excerpts of them, traditional narrative reports may be one of the least effective ways to share student learning evidence, because no one has time to study them. The best way to make points pop is through well-organized, eye-catching visuals.

CHAPTER 25

Make your key points pop

Use visuals and formatting to draw attention to key points.  Use boldface, italics, different fonts and colors, borders, brightlines, pull quotes, and sidebars to emphasize key points so they won’t be missed. If you are comparing your college against peers, for example, put your college’s information in boldface and use a brightline (Chapter 22) to show the peers’ average. Use traffic-light color coding:  Green for good news, such as meeting or exceeding targets, yellow for potential areas of concern, and red for disappointing outcomes. (For an example, see Brock University’s presentation of results from the National Survey of Student Engagement at brocku.ca/institutional-analysis/external-surveys/ nsse/NsseResults.) If your summary or analysis must be printed in only black and white, or if an audience member has a color vision deficiency, add textures to the colors so they still stand out; Exhibit 25.3 later in this chapter is an example. Cascade from major points to details.  Begin with a broad summary of your student learning evidence. A simple table may be more effective than text. If the report is long, provide a one-page executive summary . . . and don’t be upset if that’s all anyone reads! Sort your evidence in an order that helps convey your point.  Listing items in a table in their order on the original rubric or survey is neither interesting nor enlightening. Sort them from highest to lowest outcomes or frequency, so readers can quickly identify relative successes and areas of concern. 341

Understanding and Using Student Learning Evidence

Make it easy for readers to see differences and trends.  If a table is comparing male and female students, for example, add a third column showing the differences between them. Don’t rely on the standard framework for a traditional research report  that starts with a statement of the problem and then provides a review of the literature, methodology, results, and discussion. Most of your readers will be interested only in your results and discussion and won’t have the time or interest to wade through everything else. Sometimes, of course, your readers will need a very brief context for your summary and analysis, such as why you conducted this assessment and what it was designed to find out. And sometimes it’s helpful to provide information on other assessments, either at your college or elsewhere, or a brief summary of research on what helps students like yours to learn and succeed.

Three examples This section offers three examples that demonstrate many of the principles that have been discussed in this chapter for sharing student learning evidence. A table showing survey results.  Exhibit 25.1 shows an example of a poorly designed table of one (fictitious) school’s results for a portion of the Cooperative Institutional Research Program’s Freshman Survey (heri.ucla.edu/cirp-freshman -survey). The table tells no story and has no apparent point; few readers will derive anything meaningful from it. Compare the table in Exhibit 25.1 with the revision in Exhibit 25.2, which has several improvements. • The revised table presents only the most important information, which simplifies the table by bringing the number of columns down from 10 to 4. All the gender results and one of the two sets of national averages have been eliminated. (If the author needs to make a point of gender differences, those results can be put in a separate table.) • The first column has been right justified, so it’s easier for the reader to read the correct figures for each item. • All figures have been rounded to the nearest whole percent, simplifying the table and encouraging readers to focus on only relatively sizable differences. • Differences between local results and national norms have been added in a new column.

342

Men

Women

All

St. Stephen’s University

32.5% 48.3%

Get a job to help pay for college expenses.

Make at least a “B” average.

72.5% 61.2%

Get a bachelor’s degree.

Be satisfied with their college.

7.9%

10.9%

Change career choice.

Need extra time to complete their degree requirements.

13.4%

Change major field.

75.2%

81.1%

8.3%

53.5%

42.1%

11.9%

13.2%

Men

8.3%

69.7% 44.2%

77.7% 63.7%

8.1%

51.4% 42.3%

38.3% 35.2%

11.5% 10.8%

55.6%

69.0%

9.7%

44.3%

41.9%

12.8%

12.7%

Women

Men

9.6%

50.4% 43.1%

66.6% 70.5%

9.0%

43.3% 40.8%

38.8% 33.0%

11.9% 11.4%

53.2%

76.1%

10.3%

43.3%

41.1%

12.6%

13.4%

Women

48.8%

73.6%

10.0%

42.2%

37.5%

12.1%

13.3%

All

All Private Universities

12.2% 13.0%

All

All Universities

13.3% 11.7%

Percentage of first-year students saying chances are very good that they will:

Exhibit 25.1  A Poorly Designed Table

Understanding and Using Student Learning Evidence

Exhibit 25.2  An Improved Version of the Table in Exhibit 25.1 Percentage of first-year students saying chances are very good that they will:

St. Stephen’s University

All Private ­Universities

Difference

Get a bachelor’s degree

78%

74%

+4%

Be satisfied with their college

70%

49%

+21%

Make at least a “B” average

51%

42%

+9%

Get a job to help pay for college expenses

38%

38%

—

Change major field

13%

13%

—

Change career choice

12%

12%

—

Need extra time to complete their degree requirements

8%

10%

−2%

• The items have been ordered from greatest agreement to least agreement among St. Stephen’s students. • Differences have been calculated between St. Stephens students and the national averages, making clear which differences are big and which aren’t. All these changes make it easier for readers to discern the key point the table is trying to tell: St. Stephen’s first-year students are generally more optimistic about college than their peers at other private universities. Adding a meaningful title, and using a graph instead of a table to convey these results, would make this point even clearer. A stacked bar graph showing rubric results.  Exhibit 25.3 presents hypothetical evidence of student writing skills from a rubric. Note how the following are relatively easy to discern from this visual. • Students did relatively well on mechanics and their introduction and conclusions. This is relatively easy to see because outcomes are sorted from highest to lowest proportions of students who meet or exceed the standard (Chapter 22). • Student writing failed to meet the targets for source material and for body paragraphs. This is relatively easy to see because a good deal of the black part of the bars for source material and body paragraphs are to the left of the line showing the target. • Many student work examples did not provide evidence of the thesis trait. This is relatively easy to see because of the textures used in the bar graph. • The header explains that the faculty set a higher standard for mechanics than for the rubric’s other traits: That 85 percent of student work meets or exceeds the standard.

344

Source material Body paragraphs Organization Thesis Intro/conclusion Mechanics 0% Exceeds standard

10%

20%

Meets standard

30%

40%

50%

Approaches standard

60%

70%

80%

Doesn't meet standard

90%

Sharing Evidence of Student Learning

Note: The target for most traits, shown with the black line, is that at least 85% of students approach, meet, or exceed the standard. The target for Mechanics, also shown with the black line, is that at least 85% of students meet or exceed the standard.

CHAPTER 25

Exhibit 25.3  Results of a Rubric Assessing Writing

100%

Doesn't attempt

This visual thus helps faculty see the following fairly readily: • Faculty can take satisfaction that students are meeting their expectations for introduction and conclusions. • Students need the most help with source material, body paragraphs, and mechanics. • Faculty may need help developing assignments in which students develop and articulate theses. This visual would make even more of an impact if it used traffic-light color coding, with red for failing to meet the standard, yellow for approaching the standard, green for meeting the standard, and perhaps blue for exceeding the standard. It would also be improved by labeling the target line to show in the graph, rather than in the header, that Mechanics has a different target than the other traits. By adding a meaningful title and a bulleted list of the points above, this graph could become a good infographic. A pie graph showing qualitative evidence.  Exhibit 25.4 summarizes the qualitative feedback presented in List 23.2 in Chapter 23. This pie graph lets the assessment workshop facilitator quickly align participants’ key takeaways with the workshop’s key learning goals. (In this case, the takeaways matched the workshop’s key learning goals in proportion to the emphasis given to each, so no changes were needed.) 345

Understanding and Using Student Learning Evidence

Exhibit 25.4  A Pie Graph of the Qualitative Feedback in List 23.2 General/ miscellaneous 10% Self-reflection 17% Rubrics 43% Multiple choice 30%

Make evidence easy to find and access If people don’t have ready access to your summaries and analyses of student learning evidence, they can’t use it. But getting evidence in front of people is hard for several reasons: • Communication modalities have grown. Today people can receive information via emails, websites, social media, face-to-face or virtual presentations, and oldfashioned snail-mailed paper reports. • There’s no one best modality. Plenty of people don’t check their email regularly, for example. • Making evidence available isn’t the same as sharing it. Putting your evidence on a website doesn’t mean that anyone will see it, for example – you’ll need strategies to draw your audience to the website (Jankowski et al., 2012). Decide where and how to make your summaries and analyses available . . .  and how to let your audiences know where and how to access them. Think about how each of your audiences prefers to receive information. Observe which communication formats your audiences use most heavily and take advantage of them. List 25.2 offers ideas for announcement and dissemination venues. Consider embedding summaries and analyses of student learning evidence with other relevant information.  For example, include summaries of positive outcomes in materials for prospective students and employees, who will be attracted to your program or college when they see substantive evidence of its quality.

346

■■

■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■ ■■

Advertisements in the student newspaper Brochures Handouts Posters or banners Snail-mailed summaries

Sharing Evidence of Student Learning

■■

Face-to-face or online presentations at standing or special meetings of relevant groups Websites Emails with embedded or attached summaries or hyperlinks to summaries Social media with hyperlinks to summaries Newsletters Alumni magazines Department memos Press releases to the student newspaper and/or regional media

CHAPTER 25

List 25.2  Venues for Announcing and Sharing Summaries and Analyses of Student Learning Evidence

Plan multiple communication strategies.  Because there is no one best modality for sharing evidence or letting people know about it, you may need to plan a multipronged strategy, such as announcements through multiple social media channels. Should you share your evidence publicly?  Accreditors and government agencies require certain information to be made available to the public, usually on your college’s website or in its catalog. Beyond those requirements, share summaries and analyses of student learning evidence publicly only after answering the three fundamental questions in Chapter 6: • Who wants or needs to see this? • Why do they want or need to see this? • What decisions are they making that this should inform?

Be open, honest, balanced, and fair A number of professional organizations engaged in the assessment of human performance have developed statements of ethical standards for assessment (Chapter 3). They suggest the following strategies for sharing evidence fairly and ethically; Chapter 26 discusses using evidence fairly, ethically, and responsibly.

347

Understanding and Using Student Learning Evidence

Respect students’ privacy and dignity while concurrently giving faculty, administrators, and other decision-makers sufficient evidence to make meaningful decisions.  All the suggestions in this chapter relate to sharing outcomes for groups of students participating in an assessment. Individual outcomes should be shared only with the student and the faculty and staff directly involved in his or her education. Present fair and objective summaries and analyses of evidence. This includes making the following available on request: • The exact wording of survey and interview questions given to students • How the participating students were selected and evidence that they are a representative, unbiased sample of the students you wanted to assess (Chapter 12) • The number of students invited to participate, the number actually participating, and the participation rate (For example, “A random sample of 50 seniors was invited to participate in exit interviews. Twenty students, or 40 percent of those invited, participated.”) • Information that helps your audiences distinguish meaningful from trivial differences, including error margins for samples (Chapter 12) • Reasonable alternative explanations for results (For example, improvements in student learning may be due to higher admissions requirements rather than curricular changes.) Give appropriate attribution to the work and ideas of others.  When you adapt someone else’s rubric or survey, acknowledge the source. If anyone provided financial support for your assessment, helped with the mechanics, assisted with data entry and analysis, or helped in any other way, it’s only courteous to acknowledge and express appreciation for that assistance . . . especially if you’d ever like help again! Document dates and sources.  It’s amazing how useless a report is if no one can remember who did it or when. For the sake of those looking at your summary or analysis a few years from now, note the source and date on everything you share, including any visuals that might be shared separately. Someone will be grateful!

Share disappointing outcomes honestly and with context Sharing disappointing outcomes honestly and fairly is always hard. No one likes to hear bad news. People who are directly affected by disappointing outcomes – or who might be perceived as having played a role in them – may naturally feel threatened. 348

Start with the positives.  Start your story by pointing out relatively positive evidence, which puts the disappointing evidence in context.

Sharing Evidence of Student Learning

Actively involve those who may feel threatened.  People are less inclined to criticize something that they have helped to create, so develop assessments collaboratively (Chapter 13). Discuss disappointing outcomes first with those with a stake in them. If students’ general education quantitative skills are disappointing, for example, talk first to faculty teaching those skills in general education courses before issuing a report to the college community. Your conversations may provide insight that explains the outcomes and modifies your conclusions, or you may be able to help the faculty identify possible solutions. Your reports can then include their plans to address the disappointing outcomes.

CHAPTER 25

Use student learning evidence ethically and appropriately  as discussed in Chapter 26.

Focus on steps being taken to improve disappointing outcomes.  The story of disappointing outcomes is not the outcomes themselves but the steps being taken to improve them. So, after noting disappointing outcomes, focus on what’s being done about them, giving your report a very positive and even exciting problembeing-solved spin. Announcing that students have unsatisfactory writing skills is one thing; announcing a major initiative to improve student writing skills, based on recent evidence of their writing skills, sends a very different message. The more concrete the plans you provide, the greater the assurance you provide that shortcomings will be addressed as soon as possible. By starting with good news, briefly summarizing the disappointing news, then discussing steps being taken to address those disappointing outcomes, you are sandwiching bad news between two pieces of good news – a classic public relations tactic for turning lemons into lemonade. The disappointing outcomes become not a threat but an opportunity – to tell the world how good your college or program is and what it’s doing to become even better. Be gentle and sensitive.  Your audiences may bristle if you announce “bad news,” a “serious problem,” or a “failure” or if you pin the blame on individuals. They may be more receptive if you present bad news as an “area of concern” or a “suggestion for improvement.” Document the quality of your assessment strategy.  When faced with bad news, the instinct of many is to kill the messenger by blaming the bad news on flaws in the assessment (Petrides & Nodine, 2005). Chapter 24 suggests ways to document the quality of your assessment, including providing corroborating information. 349

Understanding and Using Student Learning Evidence

But be forewarned that, no matter how extensive your efforts to document the quality of your assessment strategies, it’s impossible to provide indisputable evidence of that quality to skeptics. You can never prove that your assessments are accurate and truthful; you can only collect evidence that your assessments appear to be accurate and truthful. In other words, no matter what you do, someone who wants to dispute your evidence will always be able to poke a hole in your assessment strategy. Acknowledge possible flaws in your assessment strategy.  All assessment methods are inherently imperfect and, despite your best efforts, they may not yield accurate information. Perhaps, through plain bad luck, you happened to sample the 50 worst student writing samples of the year. Assessments only provide indications of a problem, not proof of it. As noted in Chapter 12, it may be a good idea to repeat an assessment before concluding that changes are warranted if the changes would be expensive or time consuming.

Share your assessment story with accreditors While every accreditor’s requirements and expectations are unique, the following suggestions should help you prepare a successful accreditation report. Understand what your accreditor is looking for and why.  Read its guidelines and requirements carefully and thoroughly. Then, rather than comply blindly with the requirements, reflect on why the accreditor has those requirements. At their heart, accreditation requirements are intended to be principles of good practice (Suskie, 2014). The better you understand the principle of good practice underlying each requirement, the better your chances of submitting a successful accreditation report. Focus on four basic questions  that most accreditors ask about student learning assessment (Douglas Eder, personal communication, February 8, 2013): • What are the most important things we want our students to learn? Why do we think they are important? • What evidence do we have that students are learning those things? Is that evidence good enough and pervasive enough to draw meaningful conclusions about student learning?

350

Sharing Evidence of Student Learning

View an accreditation report as a cousin of reports to your board.  Accreditation reviewers and board members have a lot in common. • They are busy people whose time is precious. So keep your accreditation reports as simple and concise as possible, with plenty of visuals including dashboards or balanced scorecards. • They need to focus on the big picture more than the details. So start accreditation reports with a concise snapshot of where you are with assessment. (Exhibits 7.1 and 7.2 in Chapter 7 are examples of templates that might be used to provide such a snapshot.) Summarize supporting evidence in tables or graphs, and provide detailed supporting evidence in appendices or a virtual resource room. • They are (or should be) concerned with your story of using student learning evidence. Accreditors don’t want to pore over dozens or hundreds of program assessment reports to figure out the story themselves. They want you and your colleagues to do that analysis and share that story. • They need assurance that the evidence they’re reviewing is of sufficient quality to inform their decisions about the quality of student learning at your college. This means that accreditors need some information on how you assessed student learning as well as your resulting evidence. Accreditors typically want more information on this than board members.

CHAPTER 25

• Are we satisfied with student achievement of these learning goals? Why or why not? • If not, what are we doing about it?

Follow the suggestions in this chapter.  In particular, keep your report and supporting information succinct, simple, and clear; use visuals to make your key points pop; and be open, honest, balanced, and fair, sharing disappointing evidence honestly but with appropriate context.

Move from sharing to action If your evidence is going to be used, you’ll need not only to share it but to make sure your audiences are actively engaged in discussing it. A face-to-face session is one of the best ways to do this. List 25.3 offers suggestions for an absorbing, productive session.

351

Understanding and Using Student Learning Evidence

List 25.3  Tips to Engage Audiences in Face-to-Face Discussions of Student Learning Evidence ■■

■■

■■

■■

■■

■■

Consider asking a respected, engaging individual to sponsor or facilitate the session. This may help your evidence to be taken more seriously. If you are uncomfortable with public speaking, having a dynamic speaker help with a presentation may make it more effective. If you are presenting your summary and/or analysis of your evidence, keep your remarks short and informal. Plan to speak for no more than half of the allotted time, allowing the rest of the time for discussion. Rather than read from a prepared text, keep your presentation fresh by using a bulleted list to remind you what to say. • Design slides to support, not compete with, an oral presentation. Freestanding slides shared online need to be self-explanatory; those accompanying an oral presentation should not be. If a slide accompanying an oral presentation has more than a few words, your audience will focus on reading it rather than listening to you. Keep the text to just a few key words that emphasize your major points rather than longer phrases that compete with or duplicate your remarks. • Use a large font. For slides accompanying an oral presentation, use at least a 36-point font for major headings and 24-point for text. Make sure that those in the farthest seats can easily read your projected slides. If your material is too detailed to present on a slide using a large font, share it through a handout instead. • Use illustrations and animation effects sparingly, only when they emphasize your main points or help the audience transition from one point to the next. Superfluous illustrations and dazzling animation effects distract your audience, and your points are lost among them. Remain in charge. While it’s fine to allow questions and comments during the presentation portion of your allotted time, don’t let such remarks derail your presentation and eat up discussion time. Don’t be shy about saying, “I’m sorry to cut this off, but I want to make sure we have plenty of time for discussion once I finish sharing this.” Launch a structured conversation when you conclude your remarks by listing a few discussion questions on your last slide, such as: • What was the most important thing you learned today? • What one thing most surprised you? • How can we use this information to help our students? If your group is too large for you to call on everyone, use minute papers (Chapter 20) to get feedback or break your audience into small groups to discuss your questions. Each small group can then share its chief responses with the entire group. Record the answers to discussion questions on a flipchart, whiteboard, projected slide, or other medium visible to everyone in the audience. Transcribe the answers and send them to the participants, who can use them as the starting point for an action plan.

Perhaps the biggest challenge here is simply finding time for people to get together and have these discussions. Chapter 12 offers some suggestions on ways to carve out time. 352

Sharing Evidence of Student Learning

How data and information are shared is changing rapidly, so your best sources of additional information may be online. Do an online search for terms such as “infographics” or “data visualization,” and plenty of websites with good ideas will pop up. Another excellent source may be people within your college. If your college has a marketing office, or if it offers an academic program in marketing or visual communications, tap the expertise of those faculty and staff. Faculty may be able to create a class project to design cutting-edge visuals to share your evidence. A good resource on sharing student learning evidence is Making Student Learning Evidence Transparent: The State of the Art (Jankowski & Provezis, 2011). For examples of assessment reports that incorporate much of the advice in this chapter, check out the Transparency Framework of the National Institute for Learning Outcomes Assessment (NILOA) (learningoutcomesassessment.org/ TFComponentESL.htm). Capella University’s website summarizes its student learning evidence in simple dashboards (www.capellaresults.com/outcomes.asp). And Chapter 19 of Five Dimensions of Quality: A Common Sense Guide to Accreditation and Accountability (Suskie, 2014) offers many suggestions for preparing reports to accreditors.

CHAPTER 25

For More Information

Time to think, discuss, and practice 1. Imagine that the writing skills of students completing the first-year writing course at your college have been assessed using the ABC Writing Test. The results have come back and they’re very disappointing; on average, students score well below the national mean on every characteristic of effective writing assessed by the test. Discuss how you might constructively communicate this information to the faculty who teach first-year writing. 2. You have been asked to prepare a short summary of the survey results in Table 24.3 in Chapter 24 for Rodney College’s leadership team and board of trustees. a. What are the main points that you would want them to understand about the evidence from this assessment? b. Develop a summary (table, graph, slide, and/or text) that conveys those main points clearly, succinctly, and understandably. c. You have also been asked to prepare a short summary of the same survey results for Rodney College’s faculty. What are the main points you would want them to understand? How do those points differ, if at all, from the points you want to make to the leadership team and board?

353

CHAPTER 26

Using Evidence of S­ tudent Learning to Inform ­Important Decisions Some valuable ideas you’ll find in this chapter Hands-on practice is a far more effective way for students to learn than listening to lectures. Consider giving funding priority to resource requests supported by assessment evidence. Using a single assessment score as a gatekeeper graduation or progression requirement is an unethical use of evidence of student learning.

355

Understanding and Using Student Learning Evidence

C

hapter 6 explained that assessment has three main purposes: Giving students a great education, stewardship, and accountability, and it gave examples of the decisions that assessment can inform in each of those contexts. This chapter delves into the specifics of using evidence of student learning to inform those decisions and concludes with guidelines for using evidence ethically and responsibly.

Using student learning evidence to recognize and celebrate successes Because a fundamental purpose of assessment is betterment­– making changes to help students learn and succeed even more effectively – it’s easy to focus on problems and not see – let alone celebrate – successful outcomes that need no changes. Consider the scenario at the beginning of Chapter 22, in which faculty might find that any outcome, no matter how good, warrants change. We in higher education are so self-critical! No matter what evidence we have of student learning, we can find something wrong, something that needs to be changed. The way around this is to set clear, defensible standards and targets for student learning, as discussed in Chapter 22. Then, when those standards and targets are met, celebrate! Host a party for students, faculty, and staff. Recognize students for their collectively outstanding work, and offer them opportunities to share what they’ve learned well, such as by presenting at a student research conference. If you find that students repeatedly, consistently surpass your standards, feel free to discuss whether your standards might be ratcheted up a bit in order to stretch and challenge students even further. Recognize and celebrate rather than reward.  While good outcomes should be celebrated, think twice about rewarding them with, say, budget supplements or merit increases for the following reasons: • Rewards can tempt faculty and staff to twist or distort their evidence to look as good as possible, instead of focusing on evidence that identifies what might need to be changed. • Rewards may force faculty and staff to compete against one another for a limited pool of merit funds, which can destroy the culture of collaboration that is an essential to successful assessment (Chapter 13). • Rewards may simply recognize the status quo. If one program attracts especially well-prepared students, for example, its students may graduate with particularly strong writing or thinking skills because of what they brought into the program and not because of any special effort by the faculty and staff.

356

Disappointing evidence of student learning can lead to decisions to try to improve learning by rethinking curricula, teaching methods, learning goals, support systems, and/or co-curricula.

Rethink curricula Take a hard look at how much attention the curriculum gives to a learning goal with particularly disappointing outcomes. If by graduation students’ quantitative skills aren’t what you’d like to see, for example, perhaps quantitative skills need to be addressed in additional required courses. Or if students leave orientation unfamiliar with the college services available to them, rethink where, when, and how they learn about available services during orientation. Follow the suggestions in Chapter 5, perhaps by feeding student learning evidence into established curriculum review procedures.

Informing Decisions with Student Learning Evidence

Use student learning evidence to give students the best ­possible education

CHAPTER 26

Recognize and honor assessment efforts.  While good outcomes should be celebrated, even more recognition and reward should go to commendable assessment efforts, as discussed in Chapter 14.

Rethink teaching methods Disappointing outcomes sometimes force us to own up to a cruel fact: Despite our best efforts, we simply didn’t teach something well. Fortunately, research has identified a number of practices to help students learn effectively that we can draw upon (Ambrose, Bridges, DiPietro, Lovett, & Norman, 2010; Association of American Colleges & Universities, 2011; Bain, 2004; Connor, 2011; Greenfield, Keup, & Gardner, 2013; Jankowski, 2017; John N. Gardner Institute, 2005a and 2005b; Kuh, Kinzie, Schuh, Whitt, & Associates, 2010; Kuh & O’Donnell, 2013; McClenney, Marti, & Adkins, n.d.; McCormick, Gonyea, & Kinzie, 2013; Nilson, L. B., 2016; Pascarella & Blaich, 2013; Pascarella & Terenzini, 2005; Seymour & Lopez, 2015; Tinto, 2012; Weimer, 2013). Those practices are summarized in List 26.1, and some are discussed further in Chapter 16. Improving teaching methods can be difficult because few faculty and staff have had formal opportunities to learn about research-informed teaching methods. Chapter 10 offers suggestions for addressing this. 357

Understanding and Using Student Learning Evidence

List 26.1  Strategies That Help College Students Learn A growing body of research evidence indicates that students learn most effectively when: ■■ Students see clear relevance and value in their learning activities. ■■ Students are instilled with a can-do attitude. ■■ Students are academically challenged and given high but attainable expectations, such as through assignments with scaffolding (Chapter 16). ■■ Learning activities and grades focus on important learning goals. Faculty organize curricula, learning activities, and assessments to help students achieve important learning goals. Students spend their time and energy learning what they will be graded on. ■■ Students understand course and program learning goals and the characteristics of excellent work, often through a rubric. ■■ Students spend significant time and effort studying and practicing. ■■ Students interact meaningfully with faculty – face-to-face and/or online. ■■ Students collaborate with other students – face-to-face and/or online – including those unlike themselves. ■■ New learning is related to students’ prior experiences and what they already know, through both concrete, relevant examples and challenges to their existing paradigms. ■■ Students learn by doing, through hands-on practice engaging in multidimensional real-world tasks, rather than by listening to lectures. ■■ Students use their learning to explore, apply, analyze, justify, and evaluate, because facts memorized in isolation are quickly forgotten. ■■ Students participate in out-of-class activities that build on what they are learning in the classroom. ■■ Students can get support when they need it: academic, social, personal, and financial. ■■ Students receive frequent, prompt, concrete feedback on their work, followed by opportunities to revise their work. ■■ Students integrate and see coherence in their learning by reflecting on what and how they have learned, by constructing their own learning into meaningful frameworks, and through synthesizing capstones (Chapter 5). ■■ Their college and its faculty and staff truly focus on helping students learn and succeed and on improving student learning and success.

But isn’t poor performance sometimes the student’s fault rather than ours?  Of course. As the saying goes, you can lead a horse to water, but you can’t make it drink. Despite our best efforts, some students will not make an adequate effort to learn, and they deserve to fail. But these students are usually in the minority. Most students generally want to do whatever is necessary to pass their courses and graduate. The 50 percent rule discussed in Chapter 22 may be a useful tool in identifying where responsibility for poor performance lies.

358

Your learning goals may have a role in disappointing outcomes.

Are your goals appropriate?  It’s unrealistic, for example, to expect to turn an incompetent writer into one capable of writing a senior thesis in just one semester. Student life staff can’t expect students to become outstanding leaders in a oneweek leadership program. Librarians can’t expect students to master a full array of information literacy skills in a one-hour bibliographic instruction class. Consider whether learning goals need to be restated at levels that remain rigorous but are realistic.

Informing Decisions with Student Learning Evidence

Do you have too many goals?  Often the reason for outcomes falling short is not that your learning goals are too ambitious but that you have too many of them, causing faculty, staff, and students to lose focus and diffuse energies. Can a typical student truly be expected to achieve all of the learning goals that have been identified for your course, program, or other learning experience? Students will learn and remember more if you focus on just a few key learning goals than if you address many learning goals superficially. If you have too many learning goals, you and your colleagues must decide which ones to emphasize and which to scale back on. For example, if your students are not writing as well as you would like, look at all your other learning goals for them. Can time spent on a less-critical learning goal be reduced, freeing faculty and student time to work on writing? Chapter 4 offers suggestions for collaboratively refining learning goals.

CHAPTER 26

Rethink learning goals

Rethink support systems and co-curricula Disappointing evidence of student learning can inform modifications to admissions and placement criteria, support services such as academic advisement and tutoring, and co-curricular experiences such as first-year experiences and service learning experiences.

Use student learning evidence to ensure stewardship As discussed in Chapter 6, assessment has a second purpose beyond helping students get the best possible education: Stewardship, or ensuring the prudent,

359

Understanding and Using Student Learning Evidence

effective, and judicious care and use of resources. (Accountability, a third purpose of assessment discussed in Chapter 6, is addressed in Chapter 25.) Evidence of student learning can be used to inform stewardship decisions through resource allocations, program reviews, and, in limited circumstances, performance evaluations.

Informing resource allocation decisions Evidence of student learning can be used to inform important decisions regarding resources such as the following: • What are the most significant shortcomings in student learning – the ones where an investment of scarce resources can make the most difference? • What is the impact of our resource investments? Do class size or online tutoring modules impact student learning, for example? What is the impact of our investments in faculty professional development? • What changes in resource investments are most likely to have a significant impact on student learning? How, for example, might smart-classroom technologies be used more effectively to help improve student learning? Would deploying service learning resources differently better help students develop compassion for others? Two strategies can really help student learning evidence lead to better resource allocation decisions: Give funding priority to resource requests supported by disappointing evidence of student learning.  A request for new lab equipment that is based on unsatisfactory student lab skills might receive priority, for example, over a request without such evidence. This can be a powerful strategy to promote a culture of assessment (Chapter 14), because faculty and staff will quickly learn that student learning evidence literally pays off. Give funding priority to pervasive rather than isolated problems.  This can foster collaboration rather than competition over scarce funds (Chapter 13). If, for example, there’s pervasive evidence that students’ information literacy skills are inadequate, funding priority might be given to strategies that will help students, faculty, and staff with this across several programs.

Informing program reviews Program reviews (see Jargon Alert) can be useful tools for improvement or they can be meaningless paper-pushing exercises, depending on how they are designed 360

Many colleges base faculty tenure, promotion, and merit pay decisions at least in part on teaching effectiveness. While student evaluations of teaching may be helpful indications of teaching effectiveness if they are well designed and ask the right questions, an even better gauge of teaching effectiveness is how well students have learned what they’re supposed to. This is a strong argument to allow faculty to include student learning evidence as evidence of teaching effectiveness for tenure and promotion decisions. On the other hand, as discussed later in this chapter, student learning evidence should not be used punitively, and major decisions – such as denying tenure or promotion – should not be based on evidence from a single assessment. Indeed, such decisions can quickly kill the integrity of an assessment effort. Student learning evidence might be used to help inform faculty and staff performance evaluation decisions only under the following circumstances: • Student learning evidence is part of a teaching portfolio (Seldin, Miller, & Seldin, 2010) documenting teaching effectiveness from a variety of perspectives. • Faculty and staff are evaluated not on student learning evidence per se, but on how they are using that evidence to improve teaching and learning. • Individual faculty and staff are evaluated only on student learning in their own classes and learning experiences, not in broader settings such as an entire program or general education requirement.

Informing Decisions with Student Learning Evidence

Informing performance evaluations

CHAPTER 26

and used. Meaningful program reviews Jargon Alert! focus on program quality, along with need, cost, and cost-effectiveness, and they are Program Review used to develop evidence-informed goals A program review is a comprehensive evaluation of an academic program that is designed both to and plans for advancing the program and foster betterment and to demonstrate stewardship allocating resources to support achievement and accountability. Program reviews typically include a self-study conducted by the program’s of those goals and plans. faculty and staff, a visit by one or more external But what is program quality? Today it reviewers, and recommendations for changes based on the conclusions of the self-study and is increasingly defined as how effectively a the reviewer. program is achieving its goals, suggesting that a program review should focus on collecting and examining evidence of goal achievement (Suskie, 2014). Because student learning is a fundamental goal of any academic program, student learning evidence should thus be a focus of program review. Program reviews are a good time to synthesize and reflect on student learning evidence collected over the previous few years and identify strategies to address any persistently unsatisfactory outcomes.

361

Understanding and Using Student Learning Evidence

Consider making inclusion of student learning evidence voluntary. Faculty can be encouraged to include student learning evidence in tenure and promotion applications along with other evidence of teaching effectiveness without being compelled to include it.

Changes to assessments don’t count Today the most frequent “uses” of student learning evidence are refining the assessment tool or process. Rubrics are tweaked for clarity; survey administration procedures are revised to achieve a better response rate; multiple-choice test questions are rewritten. This makes sense the first time a new, untested assessment is used. People in higher education are very good at research, and good research often includes refining the methodology after a pilot study. If your analysis of student learning evidence (Chapter 24) identifies anything that isn’t working appropriately, of course you should revise your assessments before using them again. But research protocols do not call for endless pilot studies, and neither should assessments. It’s a lot easier to change rubric criteria than to use the student learning evidence from that rubric to make substantive changes to curricula and teaching methods. Repeated refinements of assessment tools and processes are not really uses of student learning evidence. They’re stalling tactics, putting off the day when the evidence is used to make the truly meaningful changes discussed in this chapter (Blaich & Wise, 2011).

Use student learning evidence for broad impact Today student learning evidence is often used to make relatively minor tweaks to classes and courses. Faculty who are dissatisfied with their students’ skills in citing research literature, for example, might agree to spend more time explaining how to do this in their classes. This kind of fine-tuning is low-cost and requires the consensus of a relatively small number of faculty, but it does not lead to broad, pervasive advancements in quality and effectiveness. Using evidence of student learning to identify and address broader issues is not only prudent but a requirement of some accreditors and of the Excellence in Assessment designation of the National Institute for Learning Outcomes Assessment (www.learningoutcomesassessment.org/ eiadesignation.html).

362

CHAPTER 26 Informing Decisions with Student Learning Evidence

Using evidence to make broader, more significant changes is harder than making minor tweaks, but it can be done. Here are some examples: • After assessing student learning in its writing-intensive, capstone, and servicelearning courses, Daemen College hired a writing coordinator and writingin-the-disciplines specialist, added an information literacy component to its first-year writing course, increased the proportion of first-year writing courses taught by full-time faculty from 35 percent to 90 percent, and offered workshops for faculty teaching writing-intensive courses (Morace & Hibschweiler, n.d.). • After results of the National Survey of Student Engagement and other assessments suggested the need to improve students’ digital literacy, Carlow University implemented an extensive faculty training program (Indiana University School of Education, Center for Postsecondary Research, n.d.). • After assessing students’ writing skills, Juniata College held workshops for faculty on teaching and assessing writing and established new goals and standards for its writing-across-the-curriculum initiative (Jankowski, 2011). • After assessing first-year students’ writing and finding disappointing outcomes for critical thinking and information competence, Norco College appointed course mentors to improve the consistency of teaching the first-year writing course, wrote a handbook for faculty teaching the course, and provided models for assignments that address these two skills (Flick, 2014). • After assessing first-year students’ writing skills, faculty at Stockton University revised its core learning outcomes for writing and initiated new teaching methods to support the new learning outcomes (Isabella & McGovern, in press). • Disappointing evidence of student writing skills led Stetson University to implement writing-enhanced courses throughout the curriculum, help faculty develop teaching methods to support effective writing, and implement new assessments giving students feedback on the quality of their writing (O’Neill, Slater, & Sapp, in press). There’s a common theme to these examples: Meaningful advancements in quality and effectiveness require people to knit together assessment pieces in order to identify broad, pervasive issues (Chapter 24) and to support plans to address those issues with resources.

Use student learning evidence fairly, ethically, and responsibly List 26.2 offers principles for using student learning evidence. These are in addition to the principles for conducting assessment discussed in Chapter 3, as well as those for sharing evidence discussed in Chapter 25.

363

Understanding and Using Student Learning Evidence

List 26.2  Using Student Learning Evidence Fairly, Ethically, and Responsibly ■■

■■

■■ ■■ ■■

■■

■■

■■

■■

■■

■■

■■

Make assessments planned and purposeful with a cogent understanding of how the resulting evidence will be used (Chapter 6). Actively involve those with a stake in decisions stemming from student learning evidence (Chapter 13). Focus assessments on important learning goals (Chapter 4). Communicate evidence of student learning widely and transparently (Chapter 25). Don’t brush disappointing evidence under the carpet (Chapter 25). Remember that the primary purpose of assessment is to give students the best possible education, and addressing disappointing outcomes can help achieve this aim. Don’t let student learning evidence dictate decisions. Evidence should only help you and others use professional judgment to make appropriate decisions. Don’t base a major decision only on the outcome of a single assessment. Instead, promote the use of multiple sources of evidence when making any major decision. Don’t hold people accountable for things they can’t do (Chapter 21) such as failing to meet an inappropriately high writing standard or to instill a love of African art in every student. Don’t use student learning evidence punitively. Don’t react to disappointing evidence by immediately eliminating a program or denying promotion or tenure to the faculty involved. First provide an opportunity to address the concerns identified through the evidence. Don’t penalize students for our mistakes. If you are using assessments to assign grades and part of the assessment didn’t work as it should have, throw out that part of the assessment, give students credit for it, or give them an opportunity to revise and resubmit their work. Discourage others from making inappropriate interpretations or otherwise false or misleading statements about student learning evidence. Make available any qualifiers and caveats regarding your conclusions, such as a low participation rate, possible student misinterpretations of a test question, or the underrepresentation of men in your sample. Keep faculty, students, and staff informed on how student learning evidence supports major decisions.

For More Information A terrific resource on making assessment useful and used is Using Evidence of Student Learning to Improve Higher Education (Kuh, Ikenberry, Jankowski, Cain, Ewell, Hutchings, & Kinzie, 2015). Two excellent books on teaching are Teaching at Its Best: A Research-Based Resource for College Instructors (Nilson, 2016) and Learner-Centered Teaching: Five Key Changes to Practice (Weimer, 2013). The Lilly Conference Series on College & University Teaching and Learning (lillyconferences.com) are another great resource on good teaching practices, as are the Teaching Professor newsletter and conference sponsored by Magna Publications (www.magnapubs.com).

364

2. Are the academic programs at your college now required to undergo periodic program reviews? a. What are the guidelines for those reviews? b. Is evidence of student learning included in the review? How?

Informing Decisions with Student Learning Evidence

1. One of the goals of the International Business program is for students to be able to “write clearly and effectively.” Although International Business majors are asked to write term papers in at least four courses, their writing quality is nonetheless generally still inadequate by the time they’re seniors. Faculty are quick to point to the woefully poor writing skills of entering first-year students and equally quick to blame the English department for not bringing writing skills up to par in first-year writing classes. Brainstorm what might be done to improve students’ writing skills by the time they graduate.

CHAPTER 26

Time to think, discuss, and practice

365

References Accrediting Commission for Community and Junior Colleges, Western Association of Schools and Colleges (2016, July). Accreditation reference handbook. Novato, CA: Author. Adelman, C., Ewell, P., Gaston, P., & Schneider, C. G. (2011, January). The degree qualifications profile. Indianapolis, IN: Lumina Foundation. Allen, M. J. (2006). Assessing general education programs. Bolton, MA: Anker. Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works: 7 research-based principles for smart teaching. San Francisco, CA: Jossey-Bass. American Association of University Professors (AAUP). (1940). Statement of principles on academic freedom and tenure. Retrieved from www.aaup.org/AAUP/pubsres/policydocs/ contents/1940statement.htm American Council for the Teaching of Foreign Languages. (2012). ACTFL Proficiency Guidelines 2012. Retrieved from www.actfl.org/publications/guidelines-and-manuals/actfl-proficiencyguidelines-2012 American Educational Research Association, American Psychological Association, & National Council for Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: Authors. Anderson, L. W., & Krathwohl, D. R. (Eds.). (2000). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. London, England: Pearson. Anderson, N., Schlueter, J. E., Carlson, J. F., & Geisinger, K. F. (2016). Tests in print IX. Lincoln, NE: Buros Center for Testing. Andrade, H. G. (2000, February). Using rubrics to promote thinking and learning. Educational Leadership, 57(5), 13–18. Angelo, T. A. (1999). Doing assessment as if learning matters most. AAHE Bulletin, 51(9), 3–6. Angelo, T. A., & Cross, K. P. (1993). Classroom assessment techniques: A handbook for college teachers. San Francisco, CA: Jossey-Bass. Ash, S. L., & Clayton, P. H. (2009). Generating, deepening, and documenting learning: The power of critical reflection in applied learning. Journal of Applied Learning in Higher Education, 1, 25–48. Association of American Colleges & Universities. (2002). Greater expectations: A new vision for learning as a nation goes to college. Retrieved from www.greaterexpectations.org Association of American Colleges & Universities. (2011). The LEAP vision for learning: Outcomes, practices, impact, and employers’ views. Washington, DC: Author. Association of American Colleges & Universities. (n.d.) On solid ground: VALUE report 2017. ­Washington, DC: Author. Association of College & Research Libraries. (2016). Framework for information literacy for higher education. Retrieved from www.ala.org/acrl/standards/ilframework Association of Governing Boards. (2011). Board responsibility for oversight of educational quality. Washington, DC: Author.

367

References

368

Baglin, R. F. (1981). Does “nationally” normed really mean nationally? Journal of Educational ­Measurement, 18, 97–107. Bailey, T., Jaggars, S. S., & Jenkins, D. (2015). What we know about guided pathways. New York, NY: Columbia University, Teachers College, Community College Research Center. Bain, K. (2004). What the best college teachers do. Cambridge, MA: Harvard University Press. Baker, G. R., Jankowski, N., Provezis, S., & Kinzie, J. (2012). Using assessment results: Promising practices of institutions that do it well. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Banta, T. W. (2008). Trying to clothe the emperor. Assessment Update, 20(2), 3–4, 15–16. Banta, T. W. (2010, September-October). Editor’s notes: A little extra funding goes a long way. Assessment Update, 22(5), 3–4. doi:10.1002/au.225 Banta, T. W., Lund, J. P., Black, K. E., & Oblander, F. W. (1996). Assessment in practice: Putting principles to work on college campuses. San Francisco, CA: Jossey-Bass. Banta, T. W., & Palomba, C. A. (2014). Assessment essentials: Planning, implementing, and improving assessment in higher education (2nd ed.). San Francisco, CA: Jossey-Bass. Barkley, E. F., & Major, C. H. (2016). Learning assessment techniques: A handbook for college faculty. San Francisco, CA: Jossey-Bass. Barkley, E. F., Major, C. H., & Cross, K. P. (2014). Collaborative learning techniques: A handbook for college faculty. San Francisco, CA: Jossey-Bass. Barr, R. B., & Tagg, J. (1995). From teaching to learning: A new paradigm for undergraduate education. Change, 27(6), 12–25. Bauer, J. F. (2002). Assessing student work from chatrooms and bulletin boards. In R. S. Anderson, J. F. Bauer, and B. W. Speck (Eds.), Assessment strategies for the on-line class: From theory to practice (New Directions for Teaching and Learning No. 91). San Francisco, CA: Jossey-Bass. Berrett, D. (2015, September 21). The unwritten rules of college. Chronicle of Higher Education. Retrieved from chronicle.com/article/The-Unwritten-Rules-of/233245/?cid=cc&utm_ source=cc&utm_medium=en Blaich, C. F., & Wise, K. S. (2011, January). From gathering to using assessment results: Lessons from the Wabash National Study (NILOA Occasional Paper No. 8). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Bloom, B. S. (Ed.) (1956). Taxonomy of educational objectives, Handbook 1: Cognitive domain. New York, NY: Longman. Bok, D. (2006, March/April). Seizing the initiative for quality education. Trusteeship, 9–13. Boyer, E. L. (1997). Scholarship reconsidered: Priorities of the professoriate. Stanford, CA: Carnegie Foundation for the Advancement of Teaching. Braskamp, L. A., & Engberg, M. E. (2014). Guidelines for judging the effectiveness of assessing student learning. Chicago, IL: Loyola University Chicago. Retrieved from www.learningoutcomesassessment .org/documents/BraskampGuidelines.pdf Bresciani, M. J. (2006). Outcomes-based academic and co-curricular program review: A compilation of institutional good practices. Sterling, VA: Stylus. Bresciani, M. J. (Ed.). (2007). Assessing student learning in general education: Good practice case studies. San Francisco, CA: Jossey-Bass Anker Series. Brookhart, S. M. (2013, January). How to create and use rubrics for formative assessment and grading. Alexandria, VA: Association for Supervision & Curriculum Development.

References

Brown, D., & Megan, K. (2015, February 17). A new college compact: Addressing the cost & quality ­crisis in higher ed. Washington, DC: Third Way. Retrieved from www.thirdway.org/report/a -new-college-compact-addressing-the-cost-and-quality-crisis-in-higher-ed Busteed, B. (2013, August 27). Is college worth it? Gallup Business Journal. Retrieved from businessjournal.gallup.com/content/164108/college-worth.aspx#2 Butler, S. M., & McMunn, N. D. (2006). A teacher’s guide to classroom assessment: Understanding and using assessment to improve student learning. San Francisco: Jossey-Bass. Carlson, J. F., Geisinger, K. F., & Jonson, J. L. (2017). The twentieth mental measurements yearbook. Lincoln, NE: Buros Center for Testing. Carnevale, A. P., Smith, N., & Strohl, J. (2013, June). Recovery: Job growth and education requirements through 2020. Washington, DC: Georgetown University, Georgetown Public Policy Institute, Center on Education and the Workforce. Carroll, J. (2004, May 7). Fair assessment. Presentation at the University of Stirling, United Kingdom. Cizek, G. J. (Ed.) (2012). Setting performance standards: Foundations, methods, and innovations (2nd ed.). New York, NY: Routledge. Clauser, B. E. (2000). Recurrent issues and recent advances in scoring performance assessments. Applied Psychological Measurement, 24(4), 310–324. doi:10.1177/01466210022031778 Connor, R. (2011, June). Navigating a perfect storm. NILOA Newsletter. Urbana, IL: National Institute for Learning Outcomes Assessment. Retrieved from illinois.edu/blog/view/915/54008?display Order=desc&displayType=none&displayColumn=created&displayCount=1 Coombs, C. H. (1964). A theory of data. New York: Wiley. Cooper, B. S., & Gargan, A. (2009, Sept.) Rubrics in education: Old term, new meanings. Phi Delta Kappan, 91(1), 54–55. doi:10.1177/003172170909100109 Cooper, C. R. (1977). Holistic evaluation of writing. In C. R. Cooper & L. Odell (Eds.), Evaluating writing: Describing, measuring, judging (pp. 3–32). Urbana, IL: National Council of Teachers of English. Retrieved from ERIC database. (ED143020) Costa, A. L., & Kallick, B. (2009). Learning and leading with habits of mind: 16 essential characteristics for success. Alexandria, VA: Association for Supervision & Curriculum Development. Cullen, R., Harris, M., & Hill, R. R. (2012). The learner-centered curriculum: Design and implementation. San Francisco, CA: Jossey-Bass. Deming, W. E. (2000). Out of the crisis. Cambridge, MA: MIT Press. Diamond, R. M. (2008). Designing and assessing courses and curricula: A practical guide (3rd ed.). San Francisco, CA: Jossey-Bass. Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Hoboken, NJ: Wiley. Education Northwest. (2014). Traits rubric for grades 3–12. Retrieved from educationnorthwest.org/ traits/traits-rubrics Ekman, R., & Pelletier, S. (2008). Assessing student learning: A work in progress. Change, 40(4), 14–19. Ewell, P. (2002, July). Perpetual movement: Assessment after twenty years. Keynote address at the American Association for Higher Education Assessment Conference, Boston, MA. Eynon, B., & Gambino, L. M. (2017). High-impact eportfolio practice: A catalyst for student, faculty, and institutional learning. Sterling, VA: Stylus.

369

References

370

Fink, L. D. (2013). Creating significant learning experiences: An integrated approach to designing college courses. San Francisco, CA: Jossey-Bass. Flick, A. (2014, July-August). Can assessment loops be closed? Assessment Update, 26(4), 1–2, 14–16. doi: 10.1002/au.20006 Freedman, S. W. (1991, May). Evaluating writing: Linking large-scale testing and classroom assessment. Occasional Paper No. 27. Center for the Study of Writing, University of California Berkeley, Berkeley, CA, and Center for the Study of Writing, Carnegie Mellon University, Pittsburgh, PA. Retrieved from ERIC database. (ED335681) Fulcher, K. H., Swain, M. S., & Orem, C. D. (2012, January-February). Expectations for assessment reports: A descriptive analysis. Assessment Update, 24(1), 1–2, 14–16. Gallup, Inc. (2014, February 25). The 2013 Lumina study of the American public’s opinion on higher education and U.S. business leaders poll on higher education: What America needs to know about higher education redesign. Omaha, NE: Author. Gold, L., Rhoades, G., Smith, M., & Kuh, G. (2011, May). What faculty unions say about student learning outcomes assessment (NILOA Occasional Paper No. 9). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Greenfield, G. M., Keup, J. R., & Gardner, J. N. (2013). Developing and sustaining successful first-year programs: A guide for practitioners. San Francisco, CA: Jossey-Bass. Grove, J. (2016, September 29). Mature students ‘do better with non-written assessment.’ Times Higher Education. Retrieved from www.timeshighereducation.com/news/mature-students-dobetter-non-written-assessment Hafner, J. C., & Hafner, P. M. (2003). Quantitative analysis of the rubric as an assessment tool: An empirical study of student peer-group rating. International Journal of Science Education, 25(12), 1509–1528. doi: 10.1080/0950069022000038268 Haladyna, T. M. (2015). Developing and validating multiple-choice test items (3rd ed.). London, UK: Routledge. Harris, E., & Muchin, S. (2002). Using information architecture to improve communication. Evaluation Exchange, 8(3). Retrieved from www.hfrp.org/evaluation/the-evaluation-exchange/ issue-archive/public-communications-campaigns-and-evaluation/using-information-architectureto-improve-communication Hart Research Associates. (2013, April 10). It takes more than a major: Employer priorities for college learning and student success. Retrieved from www.aacu.org/leap/documents/2013_­ EmployerSurvey.pdf Haswell, R. (1983). Minimal marking. College English, 45(6), 600–604. Hauhart, R. C., & Grahe, J. E. (2015). Designing and teaching undergraduate capstone courses. San Francisco, CA: Jossey-Bass. Hawthorne, J., & Kelsch, A. (2012, July-August). Closing the loop: How an assessment project paved the way for GE reform. Assessment Update, 24(4), 1–2, 15–16. doi:10.1002/au.244 Henning, G. W., & Roberts, D. (2016). Student affairs assessment: Theory to practice. Sterling, VA: Stylus. Howell, R. J. (2011). Exploring the impact of grading rubrics on academic performance: Findings from a quasi-experimental, pre-post evaluation. Journal on Excellence in College Teaching, 22(2), 31–49. Hsu, C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10). Retrieved from PAREonline.net/pdf/v12n10.pdf Hubbard, D. W. (2014). How to measure anything: Finding the value of “intangibles” in business (3rd ed.). Hoboken, NJ: Wiley.

References

Humphries, D. (2013, Spring). Success after college: What students, parents, and educators need to know and do. Liberal Education, 99(2). Retrieved from www.aacu.org/liberaleducation/le-sp13/ humphreys.cfm Humphries, D., & Kelly, P. (2014). How liberal arts and sciences majors fare in employment. ­Washington, DC: Association of American Colleges & Universities and National Center for Higher Education Management Systems. Humphry, S. M., & Heldsinger, S. A. (2014). Common structural design features of rubrics may represent a threat to validity. Educational Researcher, 43(5), 253–263. doi: 10.3102/0013189X14542154 Huot, B. (1990, Summer). The literature on direct writing assessment: Major concerns and prevailing trends. Review of Educational Research, 60(2), 237–263. doi: 10.3102/00346543060002237 Hutchings, P., Jankowski, N. A., & Ewell, P. T. (2014). Catalyzing assignment design activity on your campus: Lessons from NILOA’s assignment library initiative. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Indiana University School of Education, Center for Postsecondary Research. (n.d.). Engagement insights: Survey findings on the quality of undergraduate education: National Survey of Student Engagement annual results 2016. Bloomington, IN: Author. Isabella, M., & McGovern, H. (In press). Standing still is moving backwards: Looking back on a successful assessment effort informs a shift forward. In C. Wehlberg & C. J. Stanny (Eds.), Assessment in Action: Evidence-Based Discussions About Teaching, Learning, and Curriculum (New Directions for Teaching and Learning). San Francisco, CA: Jossey-Bass. Jankowski, N. (2011, July). Juniata College: Faculty led assessment (NILOA Examples of Good Assessment Practice). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Jankowski, N. (2017). Unpacking relationships: Instruction and student outcomes. Washington, DC: American Council on Education. Jankowski, N. A., Ikenberry, S. O., Kinzie, J., Kuh, G. D., Shenoy, G. F., & Baker, G. R. (2012, March). Transparency & accountability: An evaluation of the VSA College Portrait pilot. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Jankowski, N., & Provezis, S. (2011). Making student learning evidence transparent: The state of the art. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Jaschik, S., & Lederman, D. (n.d.). The 2017 Inside Higher Ed survey of college and university chief academic officers: A study by Gallup and Inside Higher Ed. Washington, DC: Inside Higher Ed and Gallup. John N. Gardner Institute for Excellence in Undergraduate Education. (2005a). First year focus – foundational dimensions (Four-year college version). Retrieved from www.jngi.org/foe-program/ foundational-dimensions/four-year-first-year-focus John N. Gardner Institute for Excellence in Undergraduate Education. (2005b). First year focus – foundational dimensions (Two-year college version). Retrieved from www.jngi.org/foe-program/ foundational-dimensions/first-year-focus-two-year Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144. doi: 10/1016/jedurev.207.05.002 Kinzie, J., & Jankowski, N. A. (2015). Making assessment consequential: Organizing to yield results. In G. D. Kuh, S. O. Ikenberry, N. A. Jankowski, T. R. Cain, P. T. Ewell, P. Hutchings, & J. Kinzie, Using evidence of student learning to improve higher education. San Francisco, CA: Jossey-Bass.

371

References

372

Kubiszyn, T., & Borich, G. D. (2013). Educational testing and measurement: Classroom application and practice (10th ed.). Hoboken, NJ: Wiley. Kuh, G. D., Ikenberry, S. O., Jankowski, N. A., Cain, T. R., Ewell, P. T., Hutchings, P., & Kinzie, J. (2015). Using evidence of student learning to improve higher education. San Francisco, CA: Jossey-Bass. Kuh, G. D., Jankowski, N., Ikenberry, S. O., & Kinzie, J. (2014, January). Knowing what students know and can do: The current state of student learning outcomes assessment in U.S. colleges and universities. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Kuh, G., Kinzie, J., Schuh, J. H., Whitt, E. J., & Associates. (2010, June). Student success in college: Creating conditions that matter. San Francisco, CA: Jossey-Bass. Kuh, G. D., & O’Donnell, K., with Reed, S. (2013). Ensuring quality & taking high-impact practices to scale. Washington, DC: Association of American Colleges & Universities. Lane, S. (2010). Performance assessment: The state of the art. (SCOPE Student Performance Assessment Series). Stanford, CA: Stanford University, Stanford Center for Opportunity Policy in Education. Lane, S. (2012). Performance assessment. In J. H. McMillan (Ed.), SAGE Handbook of Research on Classroom Assessment (pp. 313–330). Thousand Oaks, CA: Sage. Lattuca, L. R., Terenzini, P. T., & Volkwein, J. F. (2006). Engineering change: A study of the impact of EC 2000. Baltimore, MD: Accreditation Board for Engineering and Technology. Lesmond, G., McCahan, S., & Beach, D. (2017, June 6). Development of analytic rubrics for competency assessment. Toronto, Canada: Higher Education Quality Council of Ontario. Retrieved from http://www.heqco.ca/en-ca/Research/ResPub/Pages/Development-of-Analytic-Rubrics -for-­Competency-Assessment.aspx Light, T. P., Chen, H. L., & Ittelson, J. C. (2011). Documenting learning with eportfolios: A guide for college instructors. San Francisco, CA: Jossey-Bass. Linn, R. L., Baker, E. L., & Dunbar, S. B. (1991, November). Complex, performance-based assessment: Expectations and validation criteria. Educational Researcher, 20(8), 15–21. doi: 10.3102/0013189020008015 Lloyd-Jones, R. (1977). Primary trait scoring. In C. R. Cooper & L. Odell (Eds.), Evaluating writing: Describing, measuring, judging (pp. 33–68). Urbana, IL: National Council of Teachers of English. Retrieved from ERIC database. (ED143020) Marzano, R. J., & Kendall, J. S. (2008). Designing and assessing educational objectives: Applying the new taxonomy. Thousand Oaks, CA: Corwin Press. Marzano, R., Pickering, D., & McTighe, J. (1993). Assessing student outcomes: Performance assessment using the dimensions of learning model. Alexandria, VA: Association for Supervision and Curriculum Development. Maxwell, D. (2013, Feb. 4). Time to play offense. Inside Higher Ed. Retrieved from www.insidehighered.com/views/2013/02/04/college-leaders-need-reframe-discussion-value-essay McClenney, K., Marti, C. N., & Adkins, C. (n.d.). Student engagement and student outcomes: Key findings from CCSSE validation research. Austin, TX: The University of Texas at Austin Community College Leadership Program. McCormick, A. C., Gonyea, R. M., & Kinzie, J. (2013, May-June). Refreshing engagement: NSSE at 13. Change, 45(3), 6–15. Merriam, S. B., & Tisdell, E. J. (2016). Qualitative research: A guide to design and implementation (4th ed.). San Francisco, CA: Jossey-Bass.

References

Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5–11. doi: 10/3102/0013189X018002005 Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13–23. doi: 10/3102/0013189X023002013 Middle States Commission on Higher Education. (2014). Standards for accreditation and requirements of affiliation (13th ed.). Philadelphia, PA: Author. Morace, R., & Hibschweiler, I. (n.d.). Daemen College and the use of VALUE rubrics. Retrieved from www.aacu.org/sites/default/files/files/VALUE/daemen.pdf Morgaine, W. (2010). Developing rubrics: Lessons learned. In T. L. Rhodes (Ed.), Assessing outcomes and improving achievement: Tips and tools for using rubrics (pp. 11–13). Washington, DC: Association of American Colleges & Universities. Nilson, L. B. (2016). Teaching at its best: A research-based resource for college instructors (4th ed.). San Francisco, CA: Jossey-Bass. Northwest Commission of Colleges and Universities. (2010). Standards for accreditation. Redmond, WA: Author. Nunley, C., Bers, T., & Manning, T. (2011, July). Learning outcomes assessment in community colleges (NILOA Occasional Paper #10). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment. O’Neill, M., Slater, A., & Sapp, G. (In press). Writing and the undergraduate curriculum: Using assessment evidence to create a model for institutional change. In C. Wehlberg & C. J. Stanny (Eds.), Assessment in action: Evidence-based discussions about teaching, learning, and curriculum (New Directions for Teaching and Learning). San Francisco, CA: Jossey-Bass. Paniagua, M. A., & Swygert, K. A. (Eds.) (2016). Constructing written test questions for the basic and clinical sciences. Washington, DC: National Board of Medical Examiners. Parkes, J. (2012). Reliability in classroom assessment. In J. H. McMillan (Ed.), SAGE handbook of research on classroom assessment. Thousand Oaks, CA: Sage Publications. Parkes, J., & Zimmaro, D. (2016). Learning and assessing with multiple-choice questions in college classrooms. London, UK: Routledge. Parsons, B. L., Lovato, C. Y., Hutchinson, K., & Wilson, D. (2016, Fall). Building evaluation capacity through CLIPs: Communities of learning, inquiry, and practice. In W. H. Rickhards & M. StittBergh (Eds.), Evaluating student learning in higher education: Beyond the public rhetoric (New Directions for Evaluation No. 151) (pp. 21–36). San Francisco, CA: Jossey-Bass. Pascarella, E. T., & Blaich, C. (2013, March-April). Lessons from the Wabash National Study of Liberal Arts Education. Change, 45(2), 6–15. Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: A third decade of research. San Francisco, CA: Jossey-Bass. Passaic County Community College. (2016, July 1). College catalog 2016–2017. Paterson, NJ: Author. Retrieved from prod.campuscruiser.com/cruiser/pccc/fatmir/College-Catalog.pdf Penn, J. D. (2012, November-December). Assessing assessment: Strategies for providing feedback on assessment practices. Assessment Update, 24(6), 8–9, 13. doi: 10.1002/au.246 Peterson, M. W., & Einarson, M. K. (2001). What are colleges doing about student assessment? Does it make a difference? Journal of Higher Education, 72(6), 629–669. Petrides, L., & Nodine, T. (2005). Accountability and information practices in the California community colleges: Toward effective use of information in decision-making. iJournal: Insight Into Student Services, 10. Retrieved from www.ijournal.us/issue_10/ij_issue10_07.html

373

References

374

Pieper, S. L., Fulcher, K. H., Sundre, D. L., & Erwin, T. D. (2008). “What do I do with the data now?” Analyzing assessment information for accountability and improvement. Research & Practice in ­Assessment, 2(1). Retrieved from www.rpajournal.com/dev/wp-content/uploads/2012/05/A11.pdf Pike, G. (2012, January/February). Assessment measures: Criteria for evaluating campus surveys. Assessment Update, 24(1), 10–12. doi: 10.1002/au.246 Provezis, S. (2012, June). LaGuardia Community College: Weaving assessment into the institutional fabric (NILOA Examples of Good Assessment Practice). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Quality Matters Program. (2011). Quality Matters rubric workbook for higher education. Annapolis, MD: MarylandOnline, Inc. Quinlan, A. M. (2012). A complete guide to rubrics: Assessment made easy for teachers of K-college (2nd ed.). Lanham, MD: Rowman & Littlefield. Rezaei, A. R., & Lovorn, M. (2010). Reliability and validity of rubrics for assessment through writing. Assessing Writing, 15(1), 18–30. doi: 10.1016/j.asw.2010.01.003 Rhodes, T. L., & Finley, A. (2013). Using the VALUE rubrics for improvement of learning and authentic assessment. Washington, DC: Association of American Colleges & Universities. Rodriguez, M. C. (2005, June). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practices, 24(2), 3–13. Ryan, A. (2015). Creativity: The ultimate teen guide (It Happened to Me No. 43). Lanham, MD: ­Rowman & Littlefield. Salisbury, M. (2013, December 5). Future-focused assessment. Inside Higher Ed. Retrieved from www.insidehighered.com/views/2013/12/05/risks-assessing-only-what-students-know-and -can-do-essay Schneider, C. G., & Shulman, L. S. (2007). Foreword. In R. J. Shavelson, A brief history of student learning assessment: How we got where we are and a proposal for where to go next. Washington, DC: Association of American Colleges & Universities. Secolsky, C., & Denison, D. B. (Eds.) (2017). Handbook on measurement, assessment, & evaluation in higher education (2nd ed.). Philadelphia, PA: Taylor and Francis. Seldin, P., Miller, J. E., & Seldin, C. (2010). The teaching portfolio: A practical guide to improved performance and promotion/tenure decisions. San Francisco, CA: Jossey-Bass. Selke, M. J. G. (2013). Rubric assessment goes to college: Objective, comprehensive evaluation of student work. Plymouth, United Kingdom: Rowman & Littlefield Education. Seymour, S., & Lopez, S. (2015, April 8). “Big six” college experiences linked to life preparedness. Washington, DC: Gallup. Retrieved from www.gallup.com/poll/182306/big-six-college -experiences-linked-life-preparedness.aspx Shavelson, R. J. (2007). A brief history of student learning assessment: How we got where we are and a proposal for where to go next. Washington, DC: Association of American Colleges & Universities. Shulman, L. S. (2007). Counting and recounting: Assessment and the quest for accountability. Change, 39(1), 20–25. Smith, P. (2004). Curricular transformation: Why we need it . . . how to support it.” Change, 36(1), 28–35. Solomon, P. G. (2002). The assessment bridge: Positive ways to link tests to learning, standards, and curriculum improvement. Newbury Park, CA: Corwin Press. Steiner, I. D. (1966). Models for inferring relationships between group size and potential group productivity. Behavioral Science, 11(4), 273–283. doi: 10.1002/bs.3830110404 “Steve Jobs.” (2011, Oct. 6). The Economist. Retrieved from www.economist.com/blogs /babbage/2011/10/obituary

References

Stevens, D. D., & Levi, A. J. (2012). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning (2nd ed.) Sterling, VA: Stylus. Suskie, L. A. (1996). Questionnaire Survey Research: What Works, 2nd ed. (Resources for Institutional Research, No. 6.) Tallahassee, FL: Association for Institutional Research. Suskie, L. (2007). Answering the complex question of “How good is good enough?” Assessment Update, 19(4), 1–2, 12–13. Suskie, L. (2014). Five dimensions of quality: A common sense guide to accreditation and accountability. San Francisco, CA: Jossey-Bass. Suskie, L. (2015). Introduction to measuring co-curricular learning. In L. Kennedy-Philllips, A. Baldasare, & M. Christakis (Eds.), Measuring co-curricular learning (New Directions for Institutional Research No. 164) (pp. 5–13). San Francisco: Jossey-Bass. doi:10.1002/ir Suskie, L. (2017). Rubric development. In C. Secolsky & D. B. Denison (Eds.), Handbook on measurement, assessment, & evaluation in higher education (2nd ed.). Philadelphia, PA: Taylor and Francis. Swing, R. L., & Coogan, C. S. (2010, May). Valuing assessment: Cost-benefit considerations (NILOA Occasional Paper #5). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Tinto, V. (2012). Completing college: Rethinking institutional action. Chicago, IL: University of ­Chicago Press. Tinto, V. (2016, September 26). From retention to persistence. Inside Higher Ed. Retrieved from www .insidehighered.com/views/2016/09/26/how-improve-student-persistence-and-completion-essay Upcraft, M. L., & Schuh, J. H. (2002). Assessment vs. research: Why we should care about the difference. About Campus, 7(1), 16–20. Walvoord, B. E. (2010). Assessment clear and simple: A practical guide for institutions, departments, and general education (2nd ed.). San Francisco, CA: Jossey-Bass. Walvoord, B. E. (2014). Assessing and improving student writing in college: A guide for institutions, general education, departments, and classrooms. San Francisco, CA: Jossey-Bass. Walvoord, B. E., & Anderson, V. J. (2010). Effective grading: A tool for learning and assessment in college (2nd ed.). San Francisco, CA: Jossey-Bass. Waugh, A. (2016, July). Meta-majors: An essential first step on the path to college completion. Boston, MA: Jobs for the Future. Weaver, R. L., & Cotrell, H. W. (1985). Mental aerobics: The half-sheet response. Innovative Higher Education, 10, 23–31. Weimer, M. (2013). Learner-centered teaching: Five key changes to practice. San Francisco, CA: Jossey-Bass. Winkelmes, M., Bernacki, M., Butler, J., Zochowski, M., Golanics, J., & Weavil, K. H. (2016, Winter/ Spring). A teaching intervention that increases underserved college students’ success. Peer Review, 18(1/2). Retrieved from www.aacu.org/peerreview/2016/winter-spring/Winkelmes Wiseman, P. (2013, June 24). Seeking soft skills: Employers want graduates who can communicate, think fast, work in teams. Star Tribune. Retrieved from www.startribune.com/business/212770791.html Yarbrough, D. B., Shulha, L. M., Hopson, R. K. , & Caruthers, F. A. (2010). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage. Yousey-Elsener, K., Bentrim, E., & Henning, G. W. (Eds.) (2015). Coordinating student affairs divisional assessment: A practical guide. Sterling, VA: Stylus. Zilvinskis, J., Nelson Laird, T. F., & Graham, P. A. (2016, October). Faculty use of rubrics: An examination across multiple institutions. Presentation at the Assessment Institute, Indianapolis, IN. Zubizarreta, J. (2004). The learning portfolio: Reflective practice for improving student learning. San Francisco, CA: Jossey-Bass Anker Series.

375

Index Page numbers in boldface refer to Jargon Alerts that define the indexed term. Page numbers in italics refer to examples. A academic freedom, 167, 167–168 accountability, 89, 141–142. See also audiences for assessment accreditation and accreditors, 90, 141–142, 350–351 achievement, student, 324 action research, 11 aggregates of student learning evidence, 308–310 alumni, 91, 171–172 analyzing curriculum maps, 80–81 effectiveness of assessment strategies, 326–330 enrollment patterns, 82 qualitative evidence, 321 quantitative evidence, 320–321 student learning evidence, 320–326 assessment(s), 8 add-on, 157, 279–284 adopting existing, 152 of attitudes and values, 276–278 audiences for, 89–92, 334–335, 336 authentic, 207 benefits, 119 bias, 30–31, 33, 156 ceiling, 29, 295 class-level, 16 co-curricular, 20–21, 110–112 staff roles, 89–90, 168–169 collaboration on, 106, 118, 121, 166–173 committee, 91, 121–125 conferences, 130–131 consistency, 30, 156, 314, 327–330 coordinating and coordinators, 91, 118–125 cost-effectiveness and cost-benefit, 140, 150–163 course-level, 16–17, 20–21 culture, 118, 176–186

effective, 24–33, 132, 153–156. See also effectiveness of assessment strategies under analyzing embedded, 156, 157 errors, 33, 156 ethical, 31–32, 363–364 faculty roles, 89–90, 166–168 fairness in, 30–31, 314 feedback, 133–137, 134–135 floor, 29, 295 formative, 157 general education, 18, 20–21, 106–110 generalizability, 28–29, 327 guidelines, 120, 132–133 institutional, 325 of institutional learning goals, 19, 112 integrating, 118 library of readings, 128–129 objective, 216 organization, 130 peer, 169–170, 278, 279 performance, 206 of performances rather than products, 278–279 planning and plans, 30–31, 94–103, 106–114 portfolio, 243 principles of good practice, 24, 132 program-level, 17–18, 20–21, 94–103 published, 246, 246–257 purposes, 86–89 qualitative, 260 quality, 153–156 quantitative, 216 resources and support, 140–146 student roles, 32, 169–170 subjective, 216 summative, 157 technologies, 142–146

377

Index

assessment(s) (continued) tools, 96, 189–287 traditional, 206 useful, 25, 178. See also using student learning evidence valuing assessment, 176–186 See also analyzing documenting, saving and storing student learning evidence evidence of student learning reports and reporting reviewing assessment activities summarizing student learning evidence assignments and learning activities, 201, 206–214, 208 characteristics of, 206–209 creating, 209–213 learning goals of, 206, 210 See also Learning Assessment Techniques prompt(s) attitudes and values, 45 assessing, 276–278 audiences for assessment, 89–92, 334–335, 336 See also accountability sharing student learning evidence averages, 310–312 B backwards curriculum design, 65 balanced scorecard, 338 behaviors as student learning evidence, 276–277 benchmarks and standards, 290, 290–300 benefits of assessment, 119. See also purposes of assessment betterment, 118 bias in assessment, 30–31, 33, 156. See also assessment under fairness big data, 324 blind scoring, 155 Bloom’s taxonomy, 52 blueprint, test, 218, 218–221, 219 boards, governing, and board members, 90, 185 brightline, 293

378

C capstone(s), 68, 79, 157 course, 77 general education, 71, 107 program, 75–76, 94–95 ceiling, assessment, 29, 295 charts See graphs of student learning evidence tables of student learning evidence tallies of student learning evidence checklist, 190, 190. See also rubric(s) climate for assessment. See culture closing the loop, 8. See also using student learning evidence co-curricula(r), 19, 72, 76 assessment, 20–21, 110–112 learning goals, 57–58 using student learning evidence to inform, 359 collaboration on assessment, 106, 118, 121, 166–173 on curriculum, 70 on learning experiences, 19 on learning goals, 54 valuing, 173 committee(s), 141 assessment, 91, 121–125 curriculum, 125 communicating student learning evidence. See sharing student learning evidence competencies, 41. See also learning goals competency-based program, 65 conferences, assessment, 130–131 consistency in assessment, 30, 156, 314, 327–330. See also reliability in curricula, 70 coordinating and coordinators of assessment, 91, 118–125 See also assessment under resources and support planning and plans for assessment cost-effectiveness and cost-benefit of assessment, 140, 150–163 course(s)

D dashboard, 338 indicators, 312 data, 336 mining, 324 visualization, 338 See also analyzing Degree Qualifications Profile (DQP), 51 degree

associate, 17–18 career associate, 17–18 graduate, 112–113 transfer associate, 17–18, 75–76, 113–114 learning goals for, 56 See also program Delphi method, 173 dependability in assessment. See assessment under consistency designing assessment programs. See planning and plans for assessment difficulty of tests and test items, 221–222 direct evidence, 26, 26, 152 directing assessment. See coordinating and coordinators of assessment directions assessment activity, 120, 132–133 assignment. See prompt portfolio, 240–241 test, 233 discrimination of test items and rubric traits, 328–330 dispositions. See attitudes and values disseminating student learning evidence, 121, 334–353 distracter, test item, 224 documenting, saving and storing student learning evidence, 316–317 double scoring, 155 drop-outs, student, 172

Index

curriculum, 70 curriculum maps, 76–78, 76, 78 learning goals, 70 syllabus analysis, 82 See also embedded under assessments criteria in rubrics, 197–198 criterion-referenced, 290–291 critical thinking, 43 culture of assessment, 118 of evidence and betterment, 118–119 curriculum (curricula), 64–84 alignment, 64 backwards design, 65 bloated, 141 coherent, 66, 75 collaboration on, 70 committee, 125 competency-based, 65 consistent, 70 course, 70 development grants, 180 effective, 64–70 in flux, 114 general education, 70–71, 72 graduate, 112–113 improving, 83 for institutional learning goals, 71–72 interdisciplinary, 74, 113 maps and mapping, 76, 76–81, 323 pick-from-a-list, 72–73, 79, 82, 113 responsive, 65–66 rigor in, 67–68, 75–76 self-designed, 73–74, 113, 237 transfer associate, 17–18, 75–76, 113–114 using student learning evidence to inform, 357 See also program(s)

E efficiency, assessment. See cost-effectiveness and cost-benefit of assessment employers, 65–66, 91, 171 engagement in assessment, 118, 176–186 e-portfolios. See portfolios error margin, 159, 159–160, 294, 328 errors in assessment, 33, 156 evaluation, 12 of assessment quality, 326–330 performance, 361–362 of published tests, 249–257 evidence of student learning. See also culture of evidence and betterment analyzing, 326–330 dichotomous, 307

379

Index

evidence of student learning (continued) direct, 26, 26, 152 documenting, saving and storing, 316–317 generalizability, 28–29, 327 indirect, 26, 27, 152 interval, 307 nominal, 306 ordinal, 306 qualitative, 306 ratio, 307 sharing, 121, 334–353 summarizing, 306–316 using, 86–92, 178, 356–364 See also assessment culture of evidence and betterment F faculty adjunct, 167–168 motivating to engage in assessment, 176–186 to improve teaching, 180–181 performance evaluations, 361–362 professional development, 120, 124, 128–137 role in assessment, 89–90, 166–168 See also committee coordinating and coordinators of assessment fairness in assessment, 30–31, 314 in sharing student learning evidence, 347–348 in using student learning evidence, 363–364 feedback on assessment efforts, 133–137, 134–135 field experience supervisors, 171 50 percent rule, 301 flipped classrooms, 81 floor, assessment, 29, 295 focus groups, 269, 269–270 foil, test item, 224 frequency distributions. See tallies G general education, 18 assessment, 18, 20–21, 106–110 curricula, 70–71, 72

380

curriculum maps, 78–80, 108 learning goals, 55–56, 108 generalizability of student learning evidence, 28–29, 327 goals, 40. See also learning goals good practice in assessment, 24–33, 132, 153–156 governance structure for assessment, 124–125 grading and grades, 10–11, 96, 162–163, 264–265, 323 graduate(s), 91, 171–172 programs, 112–113 graduation rates, 97 grants, curriculum development, 180 graph(s) of student learning evidence, 337 from a qualitative assessment, 346 from a rubric, 345 See also visuals of student learning evidence guessing on objective tests, 233 guidelines assessment activity, 120, 132–133 assignment. See prompt(s) portfolio, 240–241 scoring. See rubric(s) guiding assessment, 91, 118–125 H habits of mind, 45 higher order thinking skills (HOTS), 43 high impact practices (HIPs), 69, 323 holistic scoring guides, 192, 192 I improving, 118 co–curricula, 359 curricula, 83, 357 learning, 87–88, 357–358 learning goals, 359 resource allocations, 359–360 teaching, 357–358 motivating faculty to, 180–181 valuing, 180–181 See also betterment reviewing assessment activities incentives. See motivating indirect evidence, 26, 27, 152 infographics, 338

J journals, 270–271, 271 K key performance indicators (KPIs), 312 K-type test items, 227–228, 228 L leaders, college/institutional, 90, 184–185. See also boards, governing, and board members leading assessment. See coordinating assessment LEAP (Liberal Education and America’s Promise) Essential Learning Outcomes, 51 learning activities, 201, 206–214, 208 analytics, 324 -centered, 86 experiences, collaborating on, 70 goals. See learning goals improving, 87–88, 357–358 management systems, 143. See also technologies objectives, 41 outcomes, 41

See also Learning Assessment Techniques learning goals professional development Learning Assessment Techniques, 208, 208, 281 learning goal(s), 40–60, 58–59, 141, 153, 274–276, 322 analysis as a, 43 application as a, 43 aspirational, 301–302 of assignments, 206, 210 co-curricular, 57–58 collaboration on, 54 course, 70 creativity as a, 44 critical thinking as a, 43 effective, 46–50 essential, 301–302 evaluation as a, 43 general education, 55–56, 108 improving, 359 information literacy as a, 43–44 institutional, 19, 56–57, 57 assessing, 19, 112 curricula for, 71–72 integrating, 59–60 knowledge as a, 42 metacognition as a, 44, 238, 271 of objective tests, 220 problem solving as a, 43 professionalism as a, 45 program, 55 resources for identifying, 51 in rubrics, 197–198 synthesis as a, 44, 240, 271 taxonomies, 52–53 thinking skills as, 43, 238 transfer associate degree, 56 understanding as a, 42 using student learning evidence to inform, 359 See also habits of mind skills liberal arts, 18 library of assessment readings, 128–129 logistics, assessment. See planning and plans for assessment

Index

information, 336 information literacy, 43–44 institution, 17 institutional effectiveness, 325. See also accountability institutional learning goals, 19, 56–57, 57 assessment, 19, 112 curricula for, 71–72 institutional research, 125, 169 institutional review boards (IRBs), 34 instruments, published, 246, 246–257 evaluating, 249–257 identifying potential, 249 quality including validity and reliability, 252–253 standardized, 246 See also surveys integrating assessment, 118 learning goals, 59–60 student learning evidence, 321–322 interdisciplinary programs, 74, 113 interpretive exercises, 226, 226–227, 279 interviews, 269, 269–270 item analysis, 329

381

Index

M maps and mapping, curriculum, 76, 76–81, 323 analyzing, 80–81 course, 76–78, 76, 78 program or general education, 78–80, 78, 108 measurement, 13 measurement error, 28 See also error margin reliability metacognition, 44, 238, 271 metrics, 312 minimal marking, 163 minute papers, 261. See also reflection and reflective writing motivating faculty and staff to engage in assessment, 176–186 to improve teaching, 180–181 students to participate in assessments, 254, 280–284 N norm-referenced, 291–293 norms, 253, 253–254 O objective(s) learning, 41 test, 216, 216–233 learning goals for, 220 observation, 278 opinions, assessment of. See attitudes and values option, test item, 224 organization, assessment, 130 See also coordinating and coordinators of assessment planning and plans for assessment outcomes, 41 P Pareto principle, 151 participation and participation rates for add-on assessments, 280–284

382

for voluntary learning experiences and activities, 111 pedagogy. See teaching peers, 291–293. See also norms; peer under assessment percentages, 308, 309 performance(s) assessing student rather than products, 278–279 evaluation(s), 361–362 indicators, 312, 312–313 levels in rubrics, 198–200 measures, 312 skills, 45 See also standards persistence, student, 323 pilot testing, 31 placement rates, 97, 323 plagiarism, 213 planning and plans for assessment, 30–31, 94–103, 106–114 general education, 106–110 institutional learning goals, 112 program-level, 94–103 portfolio(s), 236, 236–243 assessing, 243 guidelines for, 240–241 reflection and reflective writing in, 236, 241 predictive analytics, 324 presentations of student learning evidence, 338, 352 primary trait scoring guide or primary trait analysis, 196–197 principles of good assessment practice, 24, 132. See also effective under assessment professional development, 120, 124, 128–137. See also -learning center under teaching proficiencies, 41. See also learning goals program(s) associate degree, 17–18 career associate, 17–18 competency-based, 65 curriculum maps, 78–80, 78, 108 graduate, 112–113 interdisciplinary, 74, 113 learning goals, 55

Q qualitative assessment and evidence, 260, 306 analyzing, 321 graph from, 346 summarizing, 313–316, 315 technologies for, 316 quality, assessment, 153–156 analyzing, 326–330 reviewing, 99–102, 100, 101–102, 121 quantitative assessment and evidence, 216 analyzing, 320–321 summarizing, 307–313 technologies for, 313 See also tools for assessment R ratings and rating scales, 96, 265–269 ecosystem, 267–269, 268 Likert, 266–267 See also rating scale under rubrics surveys readings on assessment, 128–129 reflection and reflective writing, 260–265, 262, 263, 264, 278

grading, 264–265 in portfolios, 236, 241 reliability, 30 of published instruments, 252–253 See also consistency error margin measurement error reports and reporting, 163 assessment plans and activities, 133–137 student learning evidence, 337–346 See also sharing student learning evidence research on teaching and learning, 357–358 See also action research institutional review boards scholarship of teaching resource allocation, improving, 359–360 resources and support for assessment, 140–146 for identifying learning goals, 51 See also curriculum development under grants response rates for add-on assessments, 280–284 test item, 224 results, assessment. See evidence of student learning retention, 97, 323 reviewing assessment activities, 99–102, 100, 101–102, 121 See also effectiveness of assessment strategies under analyzing feedback on assessment efforts rewarding assessment work. See assessment under valuing root cause analysis, 47 rubric(s), 162, 190, 190–203, 210 analytic, 191, 191–192, 134–135, 243, 289, 311 benefits of, 195 checklist, 190, 190 developmental, 197 general, 197 holistic scoring guide, 192, 192

Index

-level assessment, 17–18, 20–21, 94–103 pick-from-a-list, 72–73, 79, 82, 113 review, 360–361, 361 self-designed, 73–74, 113, 237 small (low-enrollment), 113, 237 transfer associate, 17–18, 75–76, 113–114 learning goals for, 56 See also curriculum prompt(s), 211, 212, 241, 262, 264. See also assignments and learning activities published, instruments, 246, 246–257 evaluating, 249–257 identifying potential, 249 quality including validity and reliability, 252–253 standardized, 246 See also surveys purposes of assessment, 86–89. See also benefits of assessment

383

Index

rubric(s) (continued) learning goals for, 197–198 performance levels in, 198–200 primary trait scoring guide, 196–197 rating scale, 102, 101–102, 279 structured observation guide, 193, 193–194 task-specific, 196 technologies for, 203 traits in, 197–198 S samples and sampling, 157–161, 251–252 cluster random, 161 judgment, 161 error margin, 159, 159–160, 294, 328 purposeful, 161 sequential, 160 simple random, 160–161 size, 159–160 See also generalizability of student learning evidence satisfaction, student, 112 saving student learning evidence, 316–317 scaffolding, 211 scholarship of teaching, 179 scoring blind, 155 double, 155 guidelines. See rubric(s) See also aggregates of student learning evidence averages self-assessment. See ratings and rating scales reflection and reflective writing self-designed programs, 73–74, 113, 237 sharing student learning evidence, 121, 334–353 disappointing evidence, 348–350 fairness in, 347–348 reports and visuals for, 337–346 See also audiences for assessment skills hard, 46 higher order thinking (HOTS), 43 performance, 45

384

soft, 46 transferrable, 46 See also learning goals staggering assessments, 162 stakeholders, 89. See also audiences for assessment standardized tests, 246 standards and benchmarks, 290, 290–300, 306 stem, test item, 224 stewardship, 88, 325–326, 359–362 storing student learning evidence, 316–317 strategies, assessment See tools, assessment planning and plans for assessment structured observation guides, 193, 193–194 student(s) achievement, 324 former, 172 learning evidence. See evidence of student learning motivating to participate in assessments, 254, 280–284 peer assessment, 169–170, 278, 279 prospective, 91 ratings and rating scales, 96, 265–269 role in assessment, 32, 169–170 satisfaction, 112 success, 110, 110–111, 323 See also participation rates performance evaluations under teaching sub-scores, 312 success, student, 110, 110–111, 323 summarizing student learning evidence, 306–316 qualitative, 313–316, 314 quantitative, 307–313 supporting assessment, 140–146 See also coordinating and coordinators of assessment survey(s), 269 fatigue, 280–281 See also instruments, published ratings and rating scales syllabus analysis, 82

stems, 224 trick questions, 222 true-false, 229, 229–231 -wise, 218 See also assessment instruments, published time for assessment, 140–142 tools, assessment, 96, 189–287 adopting existing, 152 assignments and learning activities, 201, 206–214, 208 focus groups, 269, 269–270 journals, 270–271, 271 portfolios, 236, 236–243 prompts, 211, 211, 241, 262, 264 published instruments, 246, 246–257 ratings and rating scales, 96, 265–269 reflection and reflective writing rubric(s), 162, 190, 190–203, 210 survey(s), 269 See also test(s) and test items traits in rubrics, 197–198 transcript analysis, 82 transfer and transfer rates, 69, 294, 323 trends, historical, 295–296 triangulation, 327 trick questions, 222 trying out assessments, 31

Index

T tables of student learning evidence, 331, 337, 344 tallies of student learning evidence, 307, 308 targets, 290, 300–303, 306 taxonomies for learning goals, 52–53 teaching -centered, 86 improving, 357–358 -learning center, 124 methods, 323, 358 performance evaluations, 361–362 research on, 357–358 to the test, 65 valuing great, 178–179 valuing improving, 180 See also curriculum teaching and learning under research technologies for assessment, 142–146 learning management systems, 143 for qualitative evidence, 316 for quantitative evidence, 313 for rubrics, 203 templates for assessment plans and reports, 135–137, 136. See also reports and reporting test(s) and test items blueprint, 218, 218–221, 219 completion, 231, 231–232 difficulty, 221–222 directions for 233 discrimination, 328–330 distracters, 224 fill-in-the-blank, 231, 231–232 foils, 224 guessing on objective, 233 interpretive exercise, 226, 226–227, 279 K-type, 227–228, 228 learning goals, 220 matching, 228, 228–229, 230 multiple-choice, 222–225, 225 objective, 216, 216–233 options, 224 published, 246, 246–257 R-type, 228

U using student learning evidence, 86–92, 178, 356–364 fairness in, 363–364 to inform co-curricula, 359 curricula, 357 learning goals, 359 teaching, 357 resource allocations, 359–360 See also accountability useful under assessment using existing assessment tools. See adopting existing tools under assessment

385

Index

386

V validity, 28 analyzing, 326–330 consequential, 28 content, 28 of published instruments, 252–253 See also effective under assessment quality, assessment value-added, 293–295 value of assessment. See benefits of assessment purposes of assessment

VALUE rubrics, 107, 256 values, assessing. See attitudes and values valuing assessment, 176–186 collaboration, 173 efforts to improve teaching, 180 great teaching, 178–179 visuals of student learning evidence, 337–346 W word clouds, 314 workload, assessment. See cost-effectiveness and cost-benefit of assessment

WILEY END USER LICENSE AGREEMENT Go to www.wiley.com/go/eula to access Wiley’s ebook EULA.