Evidence-Based Policymaking: Envisioning a New Era of Theory, Research, and Practice [2 ed.] 9780367523831, 9780367523855, 9781003057666

Citation preview

EVIDENCE-BASED POLICYMAKING

New thinking is needed on the age-old conundrum of how to connect research and policymaking. Why does a disconnect exist between the research community, which is producing thousands of studies relevant to public policy, and the policy community, which is making thousands of decisions that would benefit from research evidence? The second edition updates community dissonance theory and provides an even stronger, more substantiated story of why research is underutilized in policymaking, and what it will take to connect researchers and policymakers. This book offers a fresh look into what policymakers and the policy process are like, as told by policymakers themselves and the researchers who study and work with them. New to the second edition: • The point of view of policymakers is infused throughout this book based on a remarkable new study of 225 state legislators with an extraordinarily high response rate in this hard-to-access population. • A new theory holds promise for guiding the study and practice of evidence-based policy by building on how policymakers say research contributes to policymaking. • A new chapter features pioneering researchers who have effectively influenced public policy by engaging policymakers in ways rewarding to both. • A new chapter  proposes how an engaged university could provide culturally competent training to create a new type of scholar and scholarship. This review of state-of-the-art research on evidence-based policy is a benefit to readers who find it hard to keep abreast of a field that spans the disciplines of business, economics, education, family sciences, health services, political science, psychology, public administration, social work, sociology, and so forth. For those who study evidence-based policy, the book provides the basics of producing policy-relevant research by introducing researchers to policymakers and the policy process. Strategies are provided for identifying research questions that are relevant to the societal problems that confront and confound policymakers. Researchers will have at their fingertips a breathtaking overview of classic and cutting-edge studies on the multidisciplinary field of evidence-based policy. For instructors, the book is written in a language and style that students find engaging. A topic that many students find mundane becomes germane when they read stories of what policymakers are like, and examples of researcher’s tribulations and triumphs as they work to build evidence-based policy. To point students to the most important ideas, key concepts are highlighted in text boxes.

For those who want to engage policymakers, a new chapter summarizes the breakthroughs of several researchers who have been successful at driving policy change. The book provides 12 innovative best practices drawn from the science and practice of engaging policymakers, including insights from some of the best and brightest researchers and science writers. It also takes on the daunting task of evaluating the effectiveness of efforts to engage policymakers around research. A theory of change identifies seven key elements that are fundamental to increasing policymaker’s use of research, complete with evaluation protocols and preliminary evidence on each element. Karen Bogenschneider is a Rothermel-Bascom professor emeritus of Human Ecology at the University of Wisconsin-Madison. Her expertise is the study, teaching, and practice of evidence-based family policy. She founded and directed the Wisconsin Family Impact Seminars for 25  years and provided leadership for the Family Impact Institute for 15 years. Thomas J. Corbett is a senior scientist emeritus and retired Associate Director of the Institute for Research on Poverty at the University of Wisconsin-Madison. He taught social policy and consulted at all levels of government throughout his academic career.

EVIDENCE-BASED POLICYMAKING Envisioning a New Era of Theory, Research, and Practice 2nd Edition

Karen Bogenschneider and Thomas J. Corbett

Second edition published 2021 by Routledge 52 Vanderbilt Avenue, New York, NY 10017 and by Routledge 2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN Routledge is an imprint of the Taylor & Francis Group, an informa business © 2021 Taylor & Francis The right of Karen Bogenschneider and Thomas J. Corbett to be identified as authors of this work has been asserted by them in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988. All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. Trademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. First edition published by Routledge 2010 Library of Congress Cataloging-in-Publication Data A catalog record for this book has been requested ISBN: 978-0-367-52383-1 (hbk) ISBN: 978-0-367-52385-5 (pbk) ISBN: 978-1-003-05766-6 (ebk) Typeset in Sabon by Apex CoVantage, LLC

TO OUR SPOUSES. THEY HAVE PUT UP WITH MUCH AND RECEIVED SO LITTLE IN RETURN.

CONTENTS

Acknowledgmentsix Preface: Why We Care and What We Learned in the Last Decadexii Forewordxx SECTION 1

The Problem, the Players, and the Policy Process

1

  1 Why There Is a Disconnect Between Research and Policy, and What We Can Do

3

KAREN BOGENSCHNEIDER AND THOMAS J. CORBETT

  2 The History of Evidence-Based Policy: A Long-Held Dream

24

KAREN BOGENSCHNEIDER AND THOMAS J. CORBETT

  3 Understanding Policymakers: Insights From Science

34

KAREN BOGENSCHNEIDER AND THOMAS J. CORBETT

  4 Understanding the Policy Process: Insights From Insiders

60

KAREN BOGENSCHNEIDER AND THOMAS J. CORBETT

SECTION 2

Envisioning a New Era of Theory, Research, and Evaluation

91

  5 How Policymakers Say They Use Research: A Fresh Theoretical Framework

93

KAREN BOGENSCHNEIDER AND THOMAS J. CORBETT

  6 Configuring the Research and Policy Communities: Creating an Archipelago

129

KAREN BOGENSCHNEIDER AND THOMAS J. CORBETT

  7 Why Research Is Underutilized in Policymaking: Community Dissonance Theory KAREN BOGENSCHNEIDER AND THOMAS J. CORBETT

vii

155

C ontents

  8 Evaluating Efforts to Communicate Research to Policymakers: A Theory of Change in Action

195

KAREN BOGENSCHNEIDER, HEIDI NORMANDIN, ESTHER ONAGA, SALLY BOWMAN, SHELLEY MACDERMID WADSWORTH, AND RICHARD A. SETTERSTEN, JR.

SECTION 3

Envisioning a New Era of Practice

233

  9 When Researchers Successfully Engaged Policymakers: Breakthroughs Do Happen!

235

KAREN BOGENSCHNEIDER AND THOMAS J. CORBETT

10 Engaging Policymakers: Best Practices From Those Who Study It and Do It!

254

KAREN BOGENSCHNEIDER AND THOMAS J. CORBETT

11 Approaching Policymakers: The Critical Choice of Advocacy or Education

293

KAREN BOGENSCHNEIDER AND THOMAS J. CORBETT

12 Creating Culturally Competent Scholars Now and in the Next Generation

322

KAREN BOGENSCHNEIDER AND THOMAS J. CORBETT



Afterword: Turning From Analysis to Action

345

KAREN BOGENSCHNEIDER AND THOMAS J. CORBETT

References364 Appendix384 Index396

viii

ACKNOWLEDGMENTS

Where should we begin and end? So many institutions and individuals were instrumental in bringing this book to fruition. For policy research and outreach, it truly does take a village. First, we want to thank both the William T. Grant Foundation and the Spencer Foundation for their support of the first edition of this book. In particular, we wish to acknowledge Robert Granger, former president, and Vivian Tseng, vice president, of the William T. Grant Foundation, whose vision and enthusiasm are propelling research on research utilization to the forefront. This vision continues to be carried forward by Vice President Vivian Tseng and the new president, Adam Gamoran. We appreciate their support and that of Senior Program Officer Kim Dumont for the recent policymaker study described in this book. We are most pleased, however, that they have made the issue of advancing evidence-informed social policymaking a higher priority within the philanthropic and research communities. We also appreciate the enthusiasm, support, and patience of our editor Georgette Enriquez, who made this second edition possible. As in policy work, it is the people and the relationships that matter the most. No single group of people has provided Karen with more innovative ideas and unwavering support than the board of directors for the Family Impact Institute, who served without reimbursement because of their commitment to the cause of evidence-based policymaking. Their passion is inspiring. Their rewards are vicarious. Hats off to the board members who have served over the years: Richard Barrows, Maria Cancian, Thomas J. Corbett, Diane Cushman, Governor Tony Earl, Mary Fairchild, Rep. Curtis Gielow, Jane Grinde, Bill Kraus, Ambassador/Governor Madeleine Kunin, Robert Lang, Mark Lederer, Ruth Massinga, Marygold Melli, Kristin Anderson Moore, Douglas Nelson, Theodora Ooms, Robert Pietrykowski, William Schambra, Denise Skinner, Tim Smeeding, Laurence Steinberg, and Rep. Rebecca Young. Of course, this work would not have been possible without the contributions and input of a large number of policymakers over the years. Unfortunately, those who make and execute public policies often are maligned and ridiculed. We have watched their work at close quarters, however, and generally find their efforts to be of the highest quality and their dedication to the public good to be unquestioned. The difficulty of doing public policy work should never be underestimated. In addition, Karen owes a special debt of gratitude to several current and former state legislators who served as advisors and patiently tried to teach her to think like a policymaker: Rep. Joan Ballweg, Rep. Peter Bock, Sen. Alice Clausing, Sen. Alberta Darling, Rep. Curt Gielow, (the late) Rep. Tamara Grigsby, Rep. Stan Gruszynski, Sen. Sheila Harsdorf, Sen. David Helbach, Rep. Gordon Hintz, Sen. Joanne Huelsman, Rep. Jean Hundertmark, Sen. Julie Lassa, Rep. Amy Loudenbeck, Sen. Mark Miller, Sen. Gwendolyn Moore, Rep. David Murphy, ix

A cknowledgments

Sen. Luther Olsen, Sen. Mary Panzer, Rep. Sandy Pasch, Sen. Judy Robson, Sen. Carol Roessler, Rep. Melissa Sargent, Rep. Penny Bernard Schaber, Rep. Donna Seidel, Rep. Gary Sherman, Rep. Pat Strachota, Rep. Susan Vergeront, Rep. Daniel Vrakas, Rep. Joan Wade, Rep. Mandy Wright, and Rep. Rebecca Young. We also appreciate the many philanthropists who have encouraged our work, pushed our thinking, and provided the financial resources that policy work requires. Karen will be forever indebted to an Extension colleague, who believed enough in evidencebased policy to establish a $1/2 million endowment that continues the Wisconsin Family Impact Seminars to this day. This generous gift from Phyllis Northway and her husband, Don, was given with the support of their niece Eileen Teska. There are so many other donors to thank. The Family Impact Institute, which expanded the Family Impact Seminars to other states, has been supported over the years by the W. K. Kellogg Foundation (Vice President Gail Christopher and program officers Winnie Hernandez-Gallegos, Alice Warner-Mehlhorn, and Valorie Johnson), the David and Lucile Packard Foundation (program officers Kathy Reich, Lisa Deal, and Mary Larner), the William T. Grant Foundation (Vice President Vivian Tseng, President Adam Gamoran, former President Bob Granger, and Program Officer Kim Dumont), and the Annie E. Casey Foundation (program officers Michael Laracy, Robert Geen, and Don Crary). The Wisconsin Family Impact Seminars have been supported by the University of Wisconsin-Madison Chancellor’s Office (Vice Chancellor for University Relations Charles Hoslet), Division of Continuing Studies at the University of Wisconsin-Madison (Dean Marvin Van Kekerix); the Helen Bader Foundation, Inc. (program officers Robin Bieger Mayrl, Bob Pietrykowski, and Helen Ramon); the Ira and Ineva Reilly Baldwin Wisconsin Idea Endowment at the University of Wisconsin-Madison (Assistant Vice Chancellor Peyton Smith); the Lynde and Harry Bradley Foundation (Program Director William Schambra); the A. L. Mailman Family Foundation (program officer Joelle Fontaine); and Wisconsin Cooperative Extension Family Living Programs (Program Leader Laurie Boyce). The Seminars also were supported by the Board on Human Sciences Engagement Award of the Association of Public and Land-Grant Universities; the Robert G. F. and Hazel T. Spitze Land Grant Faculty Award (Robert Spitze); and private gifts from Thomas J. Corbett, Elizabeth C. Davies, Linda Davis, Bobette Heller, and Phyllis M. Northway. Several units on the University of Wisconsin-Madison/Extension campuses have supported this work, including the College of Letters and Science (Dean Gary Sandefur), Early Childhood Excellence in Education Evaluation Project, the Institute for Research on Poverty, the Population Health Center, the LaFollette Institute of Public Affairs, the School of Education, the School of Human Ecology (Dean Robin Douthitt), the School of Social Work (Director Kristy Slack), and the Sonderegger Research Center for Social and Administrative Pharmacy. The Midwest Welfare Peer Assistance Network (WELPAN) received generous support from The Joyce Foundation for over a decade. Tom especially acknowledges the first Project Officer Unmi Song for recognizing the potential of peer assistance forums as well as the role played by Jennifer Phillips, who continued support for this vision. Karen has benefited from excellent staff over the years, including Stephanie Eddy, Deb Hewko, Olivia Little, Jessica Mills, Heidi Normandin, Emily Parrott, Jennifer Seubert, and Meg Wall-Wild. One benefit of working at a university is the opportunity to work with bright, enthusiastic, and hard-working students and colleagues, who pitch in often out of the goodness of their hearts: Nicole Anunson, Karla Balling, Mary Ellen Bell, Libby Bestul, Bethany Brewster, Ross Collin, Mary Beth Collins, Lynn Entine, Lauren Fahey, William Michael Fleming, Bettina Friese, Leah Gjertson, Danielle Greenberg, x

A cknowledgments

Ashley Grendziak, Elizabeth Gross, Beverly Hartberg, Mark Japinga, Carol Johnson, Yae Bin Kim, Woon Kyung Lee, Lauren Lewis, Kirsten Linney, Kari Morgan, Jonathan Olson, Lisa Pugh, Elizabeth Ragsdale, Jennifer Reiner, Bonnie Rieder, Aimee Ray, Hilary Shager, Rebecca Shlafer, Beth Swedeen, Amy Taub, Darci Trine, Nicole Wolfe, and Olivia Zabel. Karen also thanks those involved in the publications for our most recent study of state policymakers, especially Elizabeth Day, who was involved in all aspects of the study; Emily Parrott, who managed the data set; and Christian Schmieder, who consulted on data management and analysis. Appreciation is also extended to those involved in data collection and analysis, including Heidi Normandin, Lindsay Fuzzell, Meghan Loeser, Shelley MacDermid Wadsworth, Rob Asen, Whitney Gent, and Bret Bogenschneider. Finally, Karen feels blessed by the encouragement of her best cheerleader, Reverend Jeanette Strandjord. And we both want to thank our spouses. They put up with much and get little in return.

xi

PREFACE: WHY WE CARE AND WHAT WE LEARNED IN THE LAST DECADE

It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, and comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows the great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who know neither victory nor defeat. Theodore Roosevelt, 1910

We have witnessed considerable hand-wringing in recent days about how the public’s business is being conducted. Approval ratings of our national elected officials have eroded in the last decade with only 21 percent of the electorate showing confidence in Congress in August 2020 (Enten, 2019; Pew Research Center, 2018; Statista, 2020). Accusations of ideological rigidity and partisan polarization dominate assessments by expert observers as well as the citizenry (Binder, 2003; Contandriopoulos, Lemire, Denis, & Tremblay, 2010; Henig, 2008; Peterson, 2018; Pew Research Center, 2019). Veteran legislators retire from office, expressing frustration with the environment in which the rules that govern our civil life are formulated (Fandlund, 2018; Schulz, 2014). Simultaneously, hundreds of scientists have been leaving government posts (Plumer & Davenport, 2019; Winter, 2020), and the morale among public officials and civil servants has declined (Stratford, 2019). As 2020 dawned, medical experts accused many policymakers of ignoring science by failing to heed warnings about the threat of a possible pandemic and then reacting too slowly or inappropriately once it had emerged (MacKenzie, 2020). What once had been a noble calling, lawmaking now seems more like an endless search for campaign funds to secure political advantage. Collaborative efforts with colleagues across the aisle are threatened amidst ideological posturing (Fandlund, 2018; Schulz, 2014). As observed by the late Senator John McCain (2017) in a speech on the Senate floor: Our deliberations today . . . are more partisan, more tribal, more of the time than any other time I remember. Our deliberations can still be important and useful, but I think we’d all agree they haven’t been overburdened by greatness lately. xii

P reface : W hy W e C are and W hat W e L earned

What was most surprising to Michelle Obama when she became First Lady was not the partisanship per se but policymakers’ acquiescence to it: “This is ‘just politics’ was always the mantra, as if we could do nothing about it, as if we’d all moved to a new city on a new planet called Politics, where none of the normal rules applied” (M. Obama, 2018, p. 265–266). Moments of despair occasionally reach the highest levels. In such an environment, our public officials find scant opportunity for reflective thought and informed dialogue as they go about their governance responsibilities (Peterson, 2018). The scholarly community has been particularly scathing in its assessment of the policy world by attacking its excess partisanship, the corrosive influence of money, and overly rigid ideological decisions (Mann  & Ornstein, 2016; Mayhew, 2011). Of utmost concern, researchers lament that the public’s business is done with meager, if any, reference to the potential contributions available from the scientific community (Gamoran, 2018; Peterson, 2018). At least that is how it looks from the research side of things these days. In the eyes of academics, the cynical and partisan nature of politics cries out for rigorous research and dispassionate analysis (Peterson, 2018). And yet, to members of the academy, policymakers appear to be approaching decisions as if none of this expertise existed and none of the vast reservoir of rigorous analysis were available (Gamoran, 2018). It seems as if we leave the making of society’s rules to power, passion, and the purse. When science is used, researchers sometimes complain their work is employed not to enlighten, but as debating points for positions reached for self-serving or partisan reasons. Without question, the quantity of research has expanded dramatically in recent decades, as has the sophistication of research methods (Gamoran, 2018; Haskins, 2018; Jinha, 2010; Peterson, 2018). Yet its role in shaping political debates and informing policy decisions does not reflect American’s high level of trust in scientists and scientific knowledge (Krause, Brossard, Scheufele, Xenos,  & Franke, 2019) nor the magnitude of the investment in science by government and the philanthropic communities (Campbell & Shirley, 2018; Grant, 2017). If correct, this is a perplexing conundrum demanding thoughtful attention. History will not look back on us with kindness if we permit this gap between science and policymaking to remain unaddressed. As with most conundrums, the explanations for our failings in policymaking are not reducible to simple answers. However, at the end of the day, we focus on a disconnect between the scholarly community, whom we label as research producers, and the policymaking community, whom we label as research consumers. Our so-called “producers” encompass a host of professional actors from university-based academics through highly trained investigators and analysts to be found in think tanks and government offices. What ultimately defines them is their curiosity about figuring out how the world works, while carrying out their investigative agenda guided by the critical scrutiny and objective standards of science. Our “consumers” obviously include lawmakers at all levels of government from our nation’s capital to the local level where many elected officials vote on matters affecting a bewildering array of local jurisdictions, from municipal and county entities to school boards and environmental oversight panels to name just a few. Our conception of these consumers obviously is broader than just lawmakers. In our view of policymakers, we include executive agency officials, program managers, and a host of front-line staff who attempt to shape policy decisions, oversee these decisions, implement them, and manage their operation. The decision to enact a new rule or law or program is a salient point in the policy process, but there is much that goes on before and after that decision point that is crucial to the quality and effectiveness of governing (Lester, 2018). Thus, we take xiii

P reface : W hy W e C are and W hat W e L earned

an expansive view of both policies and the set of institutional actors that develop, enact, and manage these rules and programs. Doing policy, in our view, is more of a marathon than a sprint with multiple races taking place simultaneously. What appears to be the finish line may merely be the starting point for further deliberation and action. Doing the public’s business is not easy and, moreover, has become big business. We witness endless campaigns and the expenditure of vast treasuries to elect our representatives. The days when William Proxmire, who served several terms as U.S. senator from Wisconsin during our own lifetimes, could run for reelection on less than $10,000 now seems fantastical (Loftus, 1994). Even routine races can cost tens of millions of dollars, with much support coming from out-of-state. In such a competitive arena, we need a specific focus lest we lose ourselves amidst the conceptual and practical complexities of this labyrinthian policy world. Thus, we concentrate largely, though not exclusively, on state-level policymakers. Why state policymakers? The national level is out of reach for most academics, so state-level policy dialogues are a more accessible arena. Moreover, given the purposes of this book, state policymakers have exhibited an appetite for evidence-based policy and, by necessity, most of the transfer of research into programs and policies takes place at the state level (Kendall-Taylor & Levitt, 2017; Lester, 2018). States have wide authority for addressing social problems, such as environmental protection, immigration, and voting rights, and have often been the first governmental body to respond, especially as Congress has become increasingly paralyzed (Merriman, 2019; Moncrief & Squire, 2017). Besides, the insights and lessons applicable to state capitals often can be extrapolated to the national and local levels as well (Nutley, Walter, & Davies, 2007). Our basic thesis in this second edition builds on the original 2010 publication, though seminal refinements to our thinking are introduced throughout. By way of reminder, our essential narrative goes as follows. The traditional story of how we do social policy in this country reveals an unfortunate and counterproductive disconnect between the research community that generates research-based evidence and the policy community that ought to apply this rigorous information to their decisions. In an era where research production has risen dramatically, and where so many public needs go unmet or inadequately addressed, the unfortunate failure to connect research with policy demands careful exploration and, if possible, some corrective remediation.

The Time Has Come for a Second Edition We agreed to a second edition because we are now positioned to tell an even stronger, more substantiated story of what it will take to connect researchers and policymakers into a more productive relationship. A second edition also enables us to incorporate the thoughtful scholarship that has emerged over the last decade and the findings of our own recent studies. We have benefited from many constructive suggestions from the reviewers of the first edition, as well as input from many readers who shared their views with us. Integrating all these inputs, we have come to what we think may be some new thinking on this age-old conundrum. The rationale for the book has not changed. Like most academics, it disappoints us that scientific output mainly circulates among those who do science as a profession and too seldom touches those outside the research community (McMurtrie, 2013). The numbers of research articles have grown exponentially, reaching a record 50 million in 2009 (Gamoran, 2018; Haskins, 2018; Jinha, 2010; Peterson, 2018). Each year, the federal government alone allocates $600  billion of grant funding for programs xiv

P reface : W hy W e C are and W hat W e L earned

and services in education, health, and the social sectors (Dumont, 2019). As individuals interested in public policy, we often are bitterly disappointed that so much of the output of our scholarly peers goes unappreciated in the policy world. Government decision-making, as portrayed in the media at least, increasingly appears to be dominated by factors other than evidence-based discourse. This gap between science and policy decisions remains a tragedy, given all the superb science that might be brought to bear on public issues. Paralleling the first edition, the second edition of the book is written for those who believe that public policy and rigorous research ought to go together and that good government should rest more on hard evidence drawn from dispassionate analysis. Unfortunately, no such bond exists. Rather, we see a substantial gap, some may say chasm, between the production of research and its utilization. Yet we believe that, despite much contrary evidence, there is a way of doing public policy in a more reflective manner, even if it is outside the public’s purview. There are policymakers who hunger for evidence and objectivity, though the magnitude of their appetite may depend on the topic under consideration (Bogenschneider  & Bogenschneider, 2020). Policymakers are constantly exposed to those with fixed positions and political agendas. Public officials can get all kinds of data but little real information, at least not in ways they can trust and embrace with confidence (Bogenschneider  & Bogenschneider, 2020). Thus, decisions appear to be made on inputs other than substantiated data, and the public continues to lose confidence in the bodies that create the policies by which we govern ourselves. These are not fortuitous trends for sustaining a robust democracy. At the same time, there are many researchers who would like to apply their work to the real world. They want to make a difference by improving the way society works or at least diminishing the suffering among the society’s most vulnerable (Bogenschneider, 2014; Goodvin  & Lee, 2017; Grisso  & Steinberg, 2005; Lipsey, 2009; Olds, 2016; Yoshikawa, 2011). However, they are not sure how to go about doing that. They vaguely sense that generating better public policies requires more than conducting research studies and publishing them in prestigious peer-reviewed journals, many that are not easily accessible to policymakers and surely not the public (Bogenschneider, Corbett, & Parrott, 2019). Unfortunately, that is what they have been trained to do and, thus, know how to do best. As with most of us, the easiest path is the one most often trod. Being effective in the alien policy arena demands more. But what? Clearly, reframing research for consumption by policymakers is an admirable goal that addresses one of the greatest challenges of our times. Warnings abound about the danger of global conflicts, melting ice caps, hungry children, and rising income inequality. Yet a less obvious problem underlies the others and may prove to be a more fundamental threat. The lack of dialogue between researchers and policymakers, along with a growing pessimism among academics that engaging policymakers is worth their time, means that research is less apt to become part of the policy conversation (Bogenschneider, Corbett et  al., 2019; Rorty, 1999; Tseng, 2019b; Ward, 2008). Many researchers simply do not understand the policy world well enough to know where to begin. At the least, this unfamiliarity jeopardizes their ability to communicate across the distinct cultures that separate them. At worst, this unfamiliarity morphs into disdain for the political process and disparagement of those involved in it. The same negative perceptions that sour the public on policymakers and the policy process also serve to dissuade research producers from engaging policymakers and the policy process itself. We hope to take a small step toward remedying this unfortunate breakdown. xv

P reface : W hy W e C are and W hat W e L earned

The Contributions of the Second Edition We believe this second edition makes four main contributions: First, it explores whether it is possible to do public policy in a more reflective manner. Second, it examines why science has failed to inform social policy with any consistency. Third, it investigates how theory can build a greater understanding of why there is mistrust and miscommunication among researchers and policymakers, and why we have been unable to evaluate efforts to communicate across conflicting cultures. Finally, it suggests pragmatic processes and procedures that might increase the utilization of research in policymaking. First, we assert that research can play a larger role in the making of key policy decisions. We examine the usual suspects for why it does not. We believe that there are many policymakers out there who really desire information and data with which to inform their decisions. But most of the input they receive comes with a partisan or normative direction. They get all kinds of slanted data and copious amounts of opinion, but little objective information (Bogenschneider & Bogenschneider, 2020; Bogenschneider, Corbett et al., 2019). At the same time, there are many researchers who would like to apply their work to the real world, in the hope of making a difference (Grisso & Steinberg, 2005). However, conducting policy-relevant research and reframing it for consumption by policymakers are important skills, yet are seldom part of academic training or socialization. In the end, many academics simply do not know where to begin. Second, we examine the usual suspects for why research producers and research consumers do not easily interact and, when they do, why the character of that communication is often strained. We suggest that understanding the communities in which policymakers and researchers exist is key. We conceptualize those who conduct studies and those who apply them as distinct communities that live on different islands. Put simply, it is as if the inhabitants of each island speak different languages, march to different drummers, and see the world through different lenses. Until we better appreciate the forces that separate and divide researchers and policymakers, we cannot construct effective strategies for bringing them together. When members of communities better understand how others see the world, greater communication and collaboration is possible. Each side can act on their mutual self-interest in fostering evidence-based policymaking either by reaching out to take advantage of research that is available or by ensuring that more research that is done responds to the needs of the public. This will only happen if both sides in this dyadic relationship better understand one another and how each other’s world functions. Otherwise, they will remain isolated on their separate islands. Third, we expand on our earlier theoretical thinking in a number of ways. We more deeply embed the notion of community within the pervasive context of culture. We argue that each member of the research and policy communities brings a distinct set of cultural baggage with them to their professional roles. In this edition, we elaborate on professional culture—the attitudes and behaviors of inhabitants that grow out of their personal background, education, training, and lived experience. These influences are drawn from the more permanent norms and values inculcated during the disciplinary socialization process associated with one’s preparation for a profession. We also look at the related phenomenon of the institutional culture in which researchers and policymakers operate. We specifically explore the history, traditions, norms, and rules of lawmaking bodies to uncover clues as to the way policy decisions are made, how information is screened and communicated, and what the dominant patterns are for interacting with and influencing each other. Without an appreciation of these cultural forces, miscommunication and mistrust can occur even during interactions entered into with the best of intentions. Unfamiliarity xvi

P reface : W hy W e C are and W hat W e L earned

with these foreign cultures can infuse cross-cultural communication with suspicion and discord. We also look more deeply, and with greater insight, into how policymakers view and use research, drawing lessons from their lived experience in their own words. Finally, we extrapolate from our theoretical framework to build a base of pragmatic knowledge about how to turn good research into policy. Based on our systematic studies of researchers and policymakers, a review of the literature, and our own extensive experiences in the policy world, we offer a fresh perspective on this persistent c­ onundrum— why is research ignored when we make public policy? We attempt to link theory to practice by proposing several pragmatic processes, practices, and approaches drawn from researchers who have been successful in bridging the research/policy divide and from policymakers about why and how they use research. This volume spends more time than the first edition breaking through the stereotypes about what policymakers are like and how the policy process works, building on the perspectives of policy insiders including policymakers themselves. Finally, we offer advice and guidelines on how to engage policymakers, and what knowledge, skills, and attitudes are needed to take research findings from the laboratory to legislative bodies.

Our Priors and Our Motivations Much of our professional work has been guided by a belief that public policy and rigorous research ought to go together. This has led us to focus on one salient factor that, most importantly, is amenable to change—the cultural disconnect between the communities of research and policy. Our theoretical approach, community dissonance theory, addresses this disconnect that unnecessarily widens the gap between researchers, who investigate how the world works, and policymakers, who attempt to manage that world for the greater good. One main premise of our theory is that better understanding among researchers of the inhabitants, institutions, and culture in which lawmaking occurs can facilitate communication with policymakers and enhance research use. The real question is how our complex model plays out in the ways that real people think, act, and communicate. The book cites several concrete examples of when researchers and policymakers effectively engaged in ways rewarding to both. For instance, in one of our studies, researchers told us that their initial impressions of policymakers often were that they had “short attention spans”, “were driven by money”, and were “looking for easy solutions to complex problems, unwilling to take risks, and arrogant” (Friese & Bogenschneider, 2009). However, these same researchers pointed out that prolonged contact with policymakers familiarized them with the cultural constraints of the policy context and helped dispel these first impressions; as relationships deepened, they came to be characterized by increased mutual understanding of each other that evolved into a sense of partnership. We also observed the same effect for policymakers; with more contact and exposure, their initial negative impressions of researchers changed in a positive direction. In the end, we are not naïve. We realize the limitations of this book. More systematic interactions are needed with research producers, research consumers, and knowledge brokers. More rigorous evaluations of how to effectively communicate research evidence for policy consumption are warranted. More study is necessary to assess whether our observations generalize to other settings and across policy issues. Those caveats acknowledged, we need to start somewhere. The need is there. The interest is there. The scientific community must act. We write this book in the spirit of stimulating dialogue and perhaps even some constructive controversy with the hope that it will spawn further theory and research on this xvii

P reface : W hy W e C are and W hat W e L earned

important topic. In fact, we see this edition as a call for an expanded research agenda, which we explicitly suggest in Section 3. Moreover, we need to prepare the next generation of scholars to explore not only the mysteries of the world about them, but the challenges of bringing their work to those who govern society. This book is unusual in that it includes many vignettes drawn from the authors’ extensive involvement with policymakers and their immersion in the policy world. Without revealing our ages, we can count close to eight decades in the public arena between us. We have worked with policymakers at the local, state, and national levels across the country and beyond. As part of our policy work, we have devoted good portions of our careers explicitly to bridging the gap between academia and public policymaking. In particular, we rely heavily on our experience with two specific models designed to bring research to policymakers that we helped design and manage. Karen has experience with the Family Impact Seminars, a series of presentations, briefing reports, and discussion sessions that bring relevant research to state policymakers. Tom was involved with the Welfare Peer Assistance Network or WELPAN, a group of senior welfare administrators from contiguous states convened to dialogue over time about the future of public assistance programs. These initiatives, coupled with too many other hands-on experiences to note here, give us a wealth of experiences with which to guide our inquiry and thinking. Karen is the former director of the Wisconsin Family Impact Seminars and the Family Impact Institute which, under her leadership, provided training and technical assistance to a couple dozen states across the country that were convening Seminars in their state capitals. To date, the 234 Seminars conducted across the country and the 38 Seminars held in Wisconsin have been evaluated systematically to determine what research and analysis policymakers responded to, and how the information was used. Tom developed the “peer assistance model” commonly referred to as the Welfare Peer Assistance Network or WELPAN. Assessments of WELPAN have also been done, particularly of the Midwest group, that completed some dozen years of collaborative work before dissolving. We also draw on insights forthcoming from shorter-term networks that thrived for a time on the West Coast and among several Southern states. As noted, we go well beyond the input provided by these sources, however. Over the years, we have tapped the insights and expertise of many informants drawn from several of our populations of interest. On the research-producing side of the equation, we interviewed basic researchers, applied researchers, and those who serve as intermediaries between the two communities. On the research-consuming side, Tom has worked and consulted with numerous policymakers, policy implementers, and policy managers as he helped design and redesign human service and welfare program around the country and in Canada. Karen has conducted policy trainings for members of national professional societies such as the American Association of Family and Consumer Sciences, the American Psychological Association, the National Science Foundation, the National Council on Family Relations, the Society for Research on Child Development, and so forth. Most recently, Karen and colleagues conducted 225 face-to-face interviews of state legislators and key informants about their research use, relationships with colleagues, and how partisan polarization affects policymaking. Karen’s findings inspire some confidence because she was able to leverage her long-term relationships with policymakers to secure participation by an extraordinarily high percentage of this hard-to-access population. We describe the response rates and the methods of these studies in detail in the Appendix. The book cites several recent publications based on our research and experience, which have appeared in journals such as American Psychologist (Bogenschneider, Day, & Parrott, 2019; Bogenschneider, Day, & Bogenschneider, 2020), Family Relations xviii

P reface : W hy W e C are and W hat W e L earned

(Bogenschneider, 2020a), the Journal of Family Theory and Review (Bogenschneider, Corbett et al., 2019; Day, MacDermid Wadsworth, Bogenschneider, & Thomas-Miller, 2019), and Psychology, Public Policy, and Law (Bogenschneider  & Bogenschneider, 2020). It also draws upon Tom’s two recent works written for popular consumption, Confessions of an Accidental Scholar (Corbett, 2018a) and Confessions of a Wayward Academic (Corbett, 2018c). Our work has brought us into contact with policy players who served in a breadth of roles in local, federal, and international jurisdictions. We generally tapped some of the best and brightest from both the research consuming and producing communities, so that we could elicit insights and incisive analysis about how to advance the evidencebased policymaking agenda. We owe a huge debt to the researchers cited throughout this book, especially Jeffrey Henig and Vivian Tseng, who have studied policymakers, knowledge brokers, and the connections of research to policymaking. As our interest in bridging the gap between the scientific and policy communities matured, our intellectual antennae were tuned to lessons that might be learned and solutions to be considered. Finally, we have reviewed the applicable literature and thought hard about how existing theory might help us understand what we have observed in our work with policymakers and researchers. Our work at the interface between research and policy, between the worlds of analysis and action, has provided a critical context for designing research questions, for interpreting results, and for applying those results to the policy world. We have been fortunate to enjoy enviable careers working directly with policymakers on public policy issues ranging from childcare to elder care, from welfare reform to education reform, and from health-care innovation to workforce preparation. We have worked with think tanks, advocacy organizations, and the philanthropic community. We also collaborated with intermediary organizations such as Child Trends, the National Conference of State Legislatures, the National Governors Association, and state nonpartisan legislative service agencies. In short, we have repeatedly gone where few members of the academy dare to venture—into the world of political action. Most remarkably, despite our long history of working closely with policymakers, Karen’s recent study of state policymakers has generated many new insights—ideas that extended our thinking in ways where previously we just had not connected the dots. Ah, just one more example of the value of research! Importantly, we have done all this public work from scholarly positions within the academy. Thus, we have been forced to work across the divide that separates the world of study from the world of action. That is a divide not easily crossed, at least not successfully. Now, we want to share what we and others have learned in Evidence-Based Policymaking: Envisioning a New Era of Theory, Research, and Practice. Karen Bogenschneider Thomas J. Corbett

xix

FOREWORD TO EVIDENCE-BASED POLICYMAKING: ENVISIONING A NEW ERA OF THEORY, RESEARCH, AND PRACTICE Vivian Tseng, Senior Vice President, William T. Grant Foundation In 2008, my colleagues and I at the William T. Grant Foundation launched an ­initiative to study the use of research evidence in policy and practice. That initiative stemmed from an organizational need and a personal passion. The Foundation’s mission is to support research to improve the lives of young people, and fulfilling that mission requires that the research be used in the policies, programs, and practices that affect youth. Yet, we had little understanding of whether the research we funded had any meaningful influence beyond the academy. Our staff could look backwards with 20/20 hindsight to tell a few stories about when research was used in the past, but we couldn’t predict when, how, and under what conditions research would influence policy or practice in the future. Lacking those insights, we couldn’t fulfill our mission. On a personal level, I had long struggled to reconcile my interest in research with my desire to advance social change. As a student, I had trained in Community Psychology and Asian American Studies because both fields were committed to research and action, but the reality is that the two fields entailed a lot more research than action. I went to work at the Foundation because Ed Seidman and Bob Granger, then Senior Vice President and President, promised an opportunity to figure out how research could influence policy and practice. Building a body of research on research use thus became part of my personal quest to make research more meaningful and impactful in bringing about a better society for young people. In the intervening years, the William T. Grant Foundation has invested over $25 ­million in 65 studies of the use of research evidence. A bright light amongst those projects is the study led by Karen Bogenschneider and discussed in the second edition of Evidencebased Policymaking. This book integrates cutting-edge theory and research with the wisdom that can only come from professional experience. Karen and Tom ground the book in the need to understand the worlds of policymaking: the policy process, its institutional and professional cultures, and the people who inhabit those worlds. The research community has little hope of influencing public policy if we are ignorant of how policymaking transpires or if we subscribe to stereotypes of policymakers. With deep respect for state legislators and a genuine curiosity about their work, Karen and Tom take us inside state capitols to see how legislators make decisions, how they interact with each other and stakeholders, and how those forces shape their judgments of the credibility of research and their use of it. We come away with

xx

FOREWORD

a greater appreciation for policymaking as a process of deliberation, negotiation, and c­ ompromise—so different from the linear and methodical process of conducting research. This book also advances our theoretical understanding of the uses of research evidence. Prior taxonomies of research use emphasize the end goals for using research: to frame the issues, inform decisions, or justify positions. Karen and Tom situate research use within its relational context. Legislative work is inherently a social process, and thus the use of research in legislatures is about using research to persuade others, to educate constituents, to build trust with colleagues, and to improve the quality of policy dialogue. In conceptualizing the use of research, researchers need to move beyond thinking of research solely in terms of its informational value and consider its relational value. Relationships are the cornerstone of research use in policy, as well as in practice. We see this in study after study about the use of research evidence—whether in state legislatures or Congress, across child welfare agencies or school districts, and among technical assistance providers or advocacy groups. In the last section of the book, Karen and Tom ground these theoretical and empirical insights with real-life examples of researchers working at the intersection of academia and policy, leaving us not only inspired but equipped with practical advice on how to make a difference. While some of the exemplar researchers forged relationships with policymakers and advocates to bring their own research to the policy table, others such as Karen and Tom created the platforms that enabled a broader group of researchers to connect with policymakers. My colleagues and I have drawn on these insights to enhance our pursuit of the Foundation’s mission. As a research funder, our main constituents are found in the research community, but to bridge research and policy, we needed to strengthen our relationships with policymakers and policy intermediaries and develop a rich, textured understanding of their work. Today, we not only fund research, we play a bridging role in connecting the research and policy worlds. We support our research grantees to develop relationships with policymakers, implementers, and advocates. Our hope is that, over time, researchers will adapt their research agendas to be more policy relevant and become go-to resources for policymakers when they need research answers. Our staff also build our relationships with policy professionals, inviting them to provide input on our work, comment on commissioned papers, write blog posts, and serve as peer reviewers to help us decide which studies are worth funding—activities that strengthen our support for research that is useful and used in policy. Building policy relationships and knowledge has also been critical for my own professional development and shaped the way I lead our grantmaking work. About half of the Foundation’s funding is in education, and while I have expertise in education research, I started my career at the foundation lacking an understanding of whether the research we funded was useful to school districts and how to improve its use. In 2012-2013, I fashioned my own professional development program by job shadowing district leaders and administrators across the country. My goal was to better understand their day-to-day work, so that I could identify ways to improve ours at the foundation. That experiential learning, and others since then, have been invaluable. The insights allow me to review grant proposals with a keener sense of how district leaders would view them, to socialize our staff in understanding policymakers’ perspectives, and to design more impactful funding programs. More recently, the foundation has channeled our lessons learned into a Rapid Response Research Grants Program. We were concerned about rising inequality after the 2016

xxi

FOREWORD

presidential election and the 2020 pandemic, and wanted to quickly mobilize research to address pressing issues. Because we knew that relationships were the cornerstone of research use, we required research-policy collaborations. Researchers partnered with policymakers and advocates to identify pressing questions, synthesize available research on them, and use that research to influence policy. For example, a partnership between Professor Alicia Modestino and the Boston Mayor’s office saved and then expanded a summer jobs program for low-income youth amidst the biggest pandemic in a century. Their success required aligning Modestino’s work with the policy process. Rather than produce a traditional report, she developed research products that fit the policy context. Just-in-time research syntheses in the form of memos and presentations informed policy deliberations that evolved by the day and week. And rather than playing the role of a detached expert who disseminates “what research says,” Modestino rolled up her sleeves to spend countless hours consulting with her policy partner on the nights and weekends leading up to budgeting and implementation decisions. Looking ahead, the lessons from this book are ever more important as we face distrust in science and scientists, whether due to deliberate misinformation campaigns or the historical legacy of racism in the research enterprise. In these times, we will need to harness what we have learned—that using research evidence is a social process, facilitated by trusting relationships. We will not advance evidence-informed policy by demanding that people focus on “facts.” Instead, we must undertake the hard work of building relationships to ensure that research is trusted, useful, and used. Vivian Tseng, Senior Vice President, William T. Grant Foundation

xxii

Section 1 THE PROBLEM, THE PLAYERS, AND THE POLICY PROCESS

In Section 1, we lay out background material essential to understanding the disconnect between research and policymaking before we begin erecting our revised theory in Section 2 and before we lay out the elements of a change agenda in Section 3. In this first section, we focus on four key aspects of the conundrum we intend to tackle—understanding and responding to the gap between science and policy. In Chapter 1, we explore what we mean by evidence-based policymaking, explain why it is important, and define key terms and concepts. In Chapter 2, we overview the emergence of contemporary science and some of the historical difficulties with inserting evidence into the policy process. In Chapter 3, we introduce the reader to policymakers, specifically those engaged in legislating laws. In Chapter 4, we introduce the reader to the policy process, focusing specifically on cultural differences in the worlds of research and policy that intensify the complexity of communicating across these conflicting cultures.

1 WHY THERE IS A DISCONNECT BETWEEN RESEARCH AND POLICY, AND WHAT WE CAN DO Karen Bogenschneider and Thomas J. Corbett

Why is it that an advanced society with access to top universities and scientists, which boasts of more Nobel Prize winners than any other country in the world, fails to use research to inform policy decisions in any substantive fashion? This chapter rounds up the usual suspects for why it is so hard to bring rigorous research and analysis to bear on some of our most vexing and perplexing social challenges. As a foundation for the chapters to come, the book begins with definitions of key terms and concepts. This is the greatest intellectual challenge of our time. If we want research to make a difference in the world, if we want to justify all the dollars going to research funding, we need to figure out what it takes for research to influence policy and practice in a way that actually improves child and youth outcomes. Vivian Tseng (2019b) Success or failure in the application of social science [to the policy process] depends on the mesh between scientific skills and political interests of the social scientist on the one side, and the political skills and the scientific interests of the policy maker on the other. Suzanne Berger (1980, p. vii) Ron Haskins, a respected former Republican committee staffer in Congress and now a Brookings Institution scholar, was asked several years ago about the role research played in what was, at the time, a contentious congressional debate about welfare reform. Without missing a beat, he responded that, based on his personal experience, the best research might exert 5 percent of the total influence on the policy debate, with an upside potential of 10 percent. Personal values and political power, Haskins went on to say to his now silent and disappointed audience, were what really mattered in Congress. His assessment might be more pessimistic today. Since 2017, when the Trump administration began, over 1,600 scientists have left their positions in the federal government, most from despair that their advice would now be ignored (Plumer & Davenport, 2019; Winter, 2020) and future input simply not sought or at least not heeded. Steps also have been taken to eliminate science units in federal agencies and disband at least one-third of expert advisory committees (Gamoran, 2019). Robert Brulle, an environmental sociologist at Drexel University, complained that some political officials had abandoned science

3

P roblem , P layers , and P olicy P rocess

during the 2020 COVID-19 pandemic: “I am alarmed that clear scientific facts such as basic epidemiology, can be negated by ideological rhetoric. It is as if we are in the Middle Ages, with superstition and irrationality ruling public behavior” (cited in Terkel, 2020). As troublesome as these contemporary events are, tensions between those who seek to understand the world and those who seek to govern it have always been there in one form or another. A distinguished former state welfare official, Don Winstead, who also has held a top research-oriented position in the U.S. Department of Health and Human Services, tells the following story. Upon convincing a key Florida state legislator to support a reform initiative, he asked for additional resources to evaluate whether the proposed changes might be effective. The legislative leader paused and then commented with incredulity: “If you don’t know whether or not the program is going to work, why are you asking me to sponsor it?” This was a classic failure to communicate between the worlds of analysis and of action. Winstead, an innovator and evidence-focused administrator at heart, was looking for new and promising policy opportunities to be empirically vetted, whereas the practical political actor was looking for the certitude to justify the investment of public dollars (Lennon & Corbett, 2003). This was a classic clash between the research culture and the policy culture. Another stark example occurred at the time when welfare reform garnered national attention. When it looked as if the federally mandated evaluation of the program being done in Wisconsin might not be producing supportive results, a political kerfuffle broke out between state officials and the researchers doing the evaluation. The researchers accused the state of meddling in their science, and state officials accused the academics of having an ideological bias. When a federal intervention failed to resolve the differences, the evaluation team was fired by the state and the expected evaluation was never completed. This was more than a failure to communicate. It was a dispute about the roles each side expected from the other in applying science to assess a politicized public issue (Corbett, 2018c). Here, the connection between science and policy had been ruptured completely. Vignettes, of course, are not convincing evidence, at least as we employ that term in this book. Still—and to borrow a well-worn phrase from former Vice President Al Gore—these stories touch upon an inconvenient truth. Those who believe that public policy and rigorous research ought to inform each other confront a bitter reality. We see a substantial gap between the production of research and its utilization. It is a divide that has plagued us since the techniques of modern science emerged at the dawn of the 17th century. Take for instance, when Galileo Galilei’s espousal of the Copernican view of a helio-centered solar system brought him into a contentious dispute with Vatican officials. The policymakers of that era were not impressed by the scientific breakthroughs of the man now considered by some as the father of modern physics. Misunderstanding and miscommunication continue to this day between those we have labeled research producers and those we call research consumers. Research producers include academic scholars and researchers, program evaluators, and a variety of policy analysts who interpret data and research (Goodvin & Lee, 2017). They belong to the community of those trying to better understand the world. Some do original research and others interpret and synthesize scholarly work for practical applications, often with considerable originality. As a body, they continue to generate voluminous reports, papers, and research articles, sometimes for the sake of science and sometimes with the explicit purpose of shaping public policy. Research consumers are those who interpret, implement, and manage policy (KendallTaylor & Levitt, 2017). They belong to the community of those trying to manage the

4

W hy a D isconnect B etween R esearch and P olicy

world for the greater good. Several scholars with extensive policy experience contend that research is used more than is commonly thought (Bogenschneider, Corbett et al., 2019; Gormley, 2011; Grisso & Steinberg, 2005; Henig, 2008). Few, however, would dispute that too many policy decisions are made with scant, or at least very selective, attention to the vast stores of available scientific knowledge (Gamoran, 2018; Peterson, 2018). Why is this so? After all, we think of our nation as an advanced, rational society with access to top universities and scientists that boasts of more Nobel Prize winners than any other country in the world (Greshko, 2018). Moreover, we continue to produce ever-growing volumes of increasingly sophisticated research and policy-oriented analysis. “Contemporary poverty research is very much an American invention, with a degree of specialization and an institutional apparatus that is unmatched in other parts of the world” (O’Connor, 2002, p. 3). Arguably, however, our national comparative advantage in science all too often fails to inform our public decisions in any direct or substantive fashion, whereas some of our peer nations aggressively push evidence-based policy agendas, even though their scientific capabilities fall short of ours (Dunworth, Hannaway, Holahan,  & Turner, 2008). A  U.K. white paper issued back in 1999 committed that nation to using evidence and research so that policy could better deliver on its long-term goals (Cabinet Office, 1999). This initiative stipulates that evidence will henceforth play a critical role in policymaking. The U.K. funding of research and policymaker partnerships has decreased the divide between them and increased the use of research evidence in policy and practice (Gamoran, 2019; Rubin, 2019). One prominent example is the audacious pledge of Prime Minister Tony Blair to end child poverty back when he was Prime Minister. He implemented a three-pronged set of evidence-based reforms, which was continued and expanded upon by the conservative government of Prime Minister David Cameron (Waldfogel, 2010). This gap between research production and policy application in the United States is not a new challenge. Some members of the academic community as well as the policy elite have been trying to figure out how to bring research to bear on policymaking for well over a hundred years (Smith, 1991). Academics traditionally have operated from the premise that policy decisions would be more effective and efficient if they were based on rigorous research and dispassionate analysis. Though one may argue that such a perspective is driven by the self-serving motives of an academic worldview (Oliver, Lorenc, & Innvaer, 2014), it is difficult not to warm to such an opinion. With ever-increasing amounts of data and analysis emerging from our universities and research/evaluation firms, it is unlikely that even the most studious of public officials can sort through and make sense of the science available to them (Gamoran, 2018; Peterson, 2018). And to make matters worse, the political payoff for using science often competes directly with input from the constituents or interest groups that are key to reelection bids. The image of a hungry mule situated among several large bales of hay comes to mind. If there were just one bale, the ravenous mule would settle down to enjoy a fine repast. But with several to choose from, the poor beast cannot decide and soon starves.

Defining Key Terms What exactly is meant by this notion of making evidence-based policy? What does it mean to use research to improve policy and practice?

5

P roblem , P layers , and P olicy P rocess

Policy Let us start with the concept of policy. We define it as follows: “Policy is the development, enactment, and implementation of a plan or course of action carried out through a law, rule, code, or other mechanism in the public or private sector” (Bogenschneider, 2014, p. 43). Sometimes policies are specific and obvious. A law is passed governing who will get publicly funded health-care coverage, under what circumstances, and at what price. But in a larger sense, policies can be less transparent and more diffuse. Think of the unwritten standards that shape behaviors in bureaucracies. Even though never formalized, they govern interactions between people and, thus, have the force of policy. How often have we heard the phrase “but we have always done it this way”. For us, however, policy encompasses what a rule is, how it was developed, and how it is carried out. Policies are found in numerous domains and can be economic, environmental, legal, or international in character. In that sense, our concept of policy is broad yet still explicit or formalized in some legal fashion. We are concerned about any formulation that governs what is done, for whom, how, and under what circumstances. We embrace these stipulations whether issued from a legislative body or legitimate executive officials at all levels of government (Lester, 2018). Social Policy Yet, in another sense, our scope is much more limited. We focus primarily on social policies that directly attempt to improve the quality of life of individuals and families (Bogenschneider, 2014, 2020a; Bogenschneider et al., 2020; Jacobs & Davies, 1994); in particular, our interests lie in how policies affect vulnerable families and children. For many questions, we look to research as a final arbiter when it is unsuited to perform that role. Technical questions are one kind of challenge. They can be exceedingly complex (think of President Kennedy’s charge to get a man to the moon), but eminently doable since the complexity is subject to scientific solutions. Given the willingness to spend enough money, science was able to provide the necessary input to realize that goal. The moon shot had a clear and unambiguous goal. The trick was only in how to do it and how to pay for it. Social policies, however, can be horses of a decidedly different color that go well beyond technological challenges. Social policies encompass deeply held values and preconceived ideas about how the world should work. High-profile and polarized issues that are driven by passion, such as abortion and reproductive rights, may well exceed the authority attached to scientific evidence. However, other social policy issues, such as those involving youth and families, are viewed as a valued population that policymakers of vastly different political perspectives deem to be deserving of support. Because youth and families embody a widely shared value premise, they are a pocket of policies where polarization is perceived as less important and science is sometimes seen as more important (Bogenschneider et al., 2020). Yet no matter the degree of relevance to policymakers and receptivity to research, social policies are somewhat unique in that they involve a high degree of uncertainty about government’s proper role in such matters (Bogenschneider et al., 2020). Evidence-Based Policy Policymakers use a range of evidence for their decisions, including constituent views, political calculation, gut instinct, and personal values. Our focus here is on research 6

W hy a D isconnect B etween R esearch and P olicy

evidence. We talk of evidence-based policy as if it were a tangible phenomenon, as if we possess a scholarly consensus that might be applied directly to difficult societal issues. The real world, however, is much more complicated. Our science is not that definitive, nor is science the only legitimate and proper basis for making policy decisions. In fact, it may be far more accurate to label our vision as advancing research-shaped decision-making or evidence-informed policymaking, as some have argued (Boaz, Davies, Fraser, & Nutley, 2019b). That said, we continue to use the term evidence-based policymaking because it is still the common parlance used in research and policy circles, though we expect that will change over time. In our definition, we think of evidence-based policy as a three-pronged process. First, evidence-based policy encompasses the application of rigorous research methods to better understand the problems that confront and confound policymakers, and also to build credible evidence about which policies, programs, and practices are effective (Baron, 2018). Second, our definition encompasses the actions of elected officials, such as governors, members of Congress, and state legislators, who use credible research evidence to enact laws that establish policies, programs, administrative activities, and so forth. Third, our definition embraces the implementation of laws in research-based ways by program administrators, managers, and front-line staff (Lester, 2018). Research Evidence Clearly, we can tangle ourselves in very convoluted semantic labyrinths here. To simplify matters, we approach the concept of evidence in a conscribed manner by defining it, as the William T. Grant Foundation does, as “empirical findings derived from systematic data collection and analyses” often verified by peer review and derived according to accepted scientific protocols. We begin with an initial premise that there is a set of scientific methods that serious scientists and scholars agree constitutes a proper way for testing ideas to distinguish fact from mere belief or casual supposition (Pinker, 2018). This definition includes quantitative and qualitative studies and encompasses numerical and conceptual insights (Rickinson & McKenzie, 2020). In our definition, we think of research evidence more broadly than a specific product and more as an open-ended process of knowledge creation and testing that is “independent”, “collective”, and “cumulative” (Henig, 2008, p. 152). However daunting the challenge, the scientific community works toward building a body of sound knowledge to help inform even the most contentious of policy issues, even if it cannot solve or resolve them. Social Research Evidence In particular, we focus on the intersection between social policy and social research and what can be done to improve the connection between the two. Davies and Nutley (2008) define social research as “any systematic process of critical investigation and evaluation, theory building, data collection, analysis and codification aimed at understanding the social world, as well as the interactions between this world and public policy/public service” (p. 7). Social research encompasses the same rigorous methods, the same scientific protocols, as do all other research arenas. The practitioners of social research are bound by the same canons of behavior as their peers in the hard sciences and exhibit similar attitudinal dispositions to their craft. Truth, however, seems more elusive in the social science arena than it does in the hard sciences. Social issues are associated with a universe of research and science subject to 7

P roblem , P layers , and P olicy P rocess

conflicting interpretation and application. The intersection of social science and social policy is fraught with a particularly high level of uncertainty.

What Went Wrong: Rounding Up the “Usual Suspects” What happened to the promise of science and dispassionate analysis? Without question, there are multiple explanations for the conundrum of why science fails to inform social policy with any consistency. To borrow Claude Rains’s famous line from the movie Casablanca, let us “round up some of the usual suspects”. These suspects, which are found in prominent theories purporting to explain why research is underutilized in policymaking, tap into various quirks inherent in the character of the policymaking process itself, the inherent limitations of scientific inquiry, and the societal structure of democratic institutions. We also believe that the underutilization of research stems from the very nature of the structural environments in which research producers and consumers operate. Each of these theoretical suspects undoubtedly contributes something to the illumination of this gap between these two communities of research producers and consumers. We review each of these categories of theories next. Theoretical Consideration #1: The Character of the Policymaking Process This cluster of theories offers several reasons why the very nature of the policy process may diminish the utility of science as a basis for making decisions. Policymaking is a political process that falls outside the bounds of scientific decision-making. Because the methods of social science are linear, time intensive, and governed by the canons of science, it is challenging for researchers to respond to the fast-breaking, influence-driven policy process (Gamoran, 2018). Our aim is not to be exhaustive here, but rather to introduce a few factors that threaten the utility of science in light of how the policy process works. First, the expectations around science may simply have been unrealistic because the policy process is far more complex than those outside the policy arena typically imagine. If the policy process were driven by a single criterion, it would be a much more straightforward task. In the past, research use was premised on the simplistic notion that it can be more or less injected directly into the policy process, which Pettigrew (1985) coined as the “hypodermic model”. This model falls short because no one factor typically determines an outcome nor can the policy process be reduced to a single action. Moreover, the social sciences seldom produce the kind of smoking-gun evidence or “killer studies” (Henig, 2008) that can single-handedly settle disputes on complex, multiply determined policy problems. Finding statistically significant effects on outcomes of interest may prove convincing to scientists, but unpersuasive to policymakers, who must weigh the magnitude of the effects with so many other factors of which scholars are only dimly aware. What would a new policy direction cost? What about its logistical feasibility? How would key supporters view a new direction if it increased the role of government or challenged the prevailing political consensus? And worse, what if the new evidence endorsed policies and programs that upset key constituencies normally allied with the policymaker? Statistical significance merely confirms that an impact is not likely attributable to chance. Substantive significance means that the impacts are still worth pursuing after taking into account a whole range of other critical considerations that arise throughout the policy process (Goodvin & Lee, 2017). 8

W hy a D isconnect B etween R esearch and P olicy

Key Concepts 1.1.  The Failure of Science to Shape Policy: Rounding Up the Usual Suspects Theoretical Consideration 1: The Character of the Policymaking Process • • • •

Expectations of what research can contribute are often unrealistic; Policy decisions are not driven solely by numbers or data; The real world changes faster than science can accommodate; and Politics is becoming more polarized.

Theoretical Consideration 2: The Limitations of Scientific Inquiry • Social problems are complex and the capacity of scientific tools and ways of framing research questions fail to capture this complexity; • Randomized controlled trials can discover what works, but not why it works, under what conditions, or for which populations; and • The application of even the most rigorous findings is often overlooked.

Theoretical Consideration 3: The Societal Structure of Democratic Institutions • The complexity of democratic decision-making constrains the role of science; • In the United States, the fragmentation of power complicates decision-making; • The U.S. version of democracy presumes competition of ideas and diminishes the prospects that decisions will be influenced by any single factor; and • In other corporatist countries, experts are situated in strong political parties, more institutionalized labor and employer federations, and moderately centralized governments.

Theoretical Consideration 4: The Structural Environment in Which Researchers and Policymakers Operate • They view the world through the lens of their education, training, and lived experience; • They speak different languages and use different communication styles; • They respond to diverse signals and incentives; and • The norms of their workplace demand different behaviors.

Second, policy decisions are not driven solely by numbers or data, nor should they be. As Kingdon (2011) points out, the doing of policy involves a complicated, often subjective, calculus that weighs numerous competing factors, including values, career aspirations, media attention, and constituent views, among many others. Evidence can and does play a role, but often a subordinate one to factors that are more salient at the time than data. 9

P roblem , P layers , and P olicy P rocess

Third, the policy world sometimes changes faster than science can accommodate. Policymakers confront the world on a real-time basis. They often receive immediate reports on emerging problems and soon become aware of new possibilities for policy and program responses. Policymakers are pressured to take prompt action. As aptly put by Vivian Tseng, Vice President of the William T. Grant Foundation, policymakers often need “answers yesterday for the decisions they need to make today” (Tseng, 2020b). The pace of science is notably slower, irrespective of how hard scholars labor. Academics, especially from the traditional disciplines, spend little time in the so-called “real world”, so research and evaluations often are not timely, and cutting-edge issues may not filter through the literature for some time. Finally, we live in times when “ideology divides the country and politics polarizes the population” (Kendall-Taylor  & Levitt, 2017, p.  709). Veteran political observers note that the level of partisanship has increased in the political class in recent decades (Haidt & Hetherington, 2012; Henig, 2008; Mayhew, 2011; Mann & Ornstein, 2016; Peterson, 2018). Several suspects are blamed for this polarizing trend. Money has become a dominant factor in running for office and the financial contributions flow to candidates who will deliver on the agendas of special-interest groups (Mayer, 2017; Rivlin, 2006). The Internet, 24-hour instant news, and other recent communication innovations make the political process transparent—a good thing in many respects. But such ready access to information makes it riskier to do business with political adversaries because one’s political base can easily become aware of and perhaps angered by any compromises or concessions that are made (Pinker, 2018). The whole process of selecting candidates, with the growing importance of primaries, can weed out candidates representing the middle of the road; these early stages of campaigning involve fewer active participants and give an edge to the more extreme elements in each party (Rosenthal, 2009). Theoretical Consideration #2: The Limitations of Scientific Inquiry The potential of science to bring reason and dispassionate analysis to policymaking is quickly tarnished by the inherent limitations of social science inquiry, which are exacerbated by the unavoidable ambiguity of policy issues. Social problems are notoriously difficult to define and solve, yet science requires clear definitions of problems and unambiguous criteria for solutions. An integral strength of science is its methodology, yet those very methods can sometimes result in findings so abstract, narrow, and circumstantial that their utility in guiding policy decisions is diminished. The result is a disconnect between what policymakers want to know and what science can provide. The utility of research for policy development purposes is also limited by the ways in which researchers tend to frame research questions. Too often, the questions are narrow, often driven by the methodologies in which researchers are proficient (Gamoran, 2018). According to Weiss (1978), researchers “do not pick the research method to suit the problem but almost unwittingly see that aspect of the problem to which their methodology applies” (p. 45). Even when they get the policy question right, research producers by the nature of their training tend to focus on what specific program or policy caused a certain outcome. The identification of an increasing number of effective programs was made possible by the advent of the randomized controlled trial, which for policy purposes, is one of the most important and useful scientific advances (Baron, 2018; Haskins, 2018). The randomized controlled trial became the gold standard of evaluation because of its ability to control for confounding factors. Yet the results still are valid only in the population involved in 10

W hy a D isconnect B etween R esearch and P olicy

the trial, as well as under its specific conditions, such as dosage and duration. Because a program works somewhere does not mean that it will work everywhere or under all conditions (Gamoran, 2018). To what other populations can the results be applied (Cartwright  & Hardie, 2012)? What kind of demographic, social, economic, political, or contextual differences might make the application of a new research-driven insight risky or even result in unanticipated side effects? What works in Situation A might not work in Situation B. A welfare reform initiative that works in a state with a booming economy might be doomed in an area with slack labor demand. Slight differences in a successful program, evaluated in a hot-house environment where resources are not a question, can fail to replicate when introduced into the real world under ordinary circumstances with resource constraints. The Achilles heel of the evidence-based movement, according to Haskins (2018), is the failure to replicate. Even if the intervention can be replicated for the target population, does the size of the benefit exceed the magnitude of the costs, and, given the gravitational pull of the status quo, what are the chances that the intervention will continue after the demonstration ends? Deaton and Cartwright (2018) raise these critical questions that point out limitations of the very methodology considered the “gold standard” in evaluation. Randomized controlled trials can build scientific knowledge by discovering “what works” but without knowing “why” they work, “we run the risk of worthless casual ‘fairy story’ theorizing . . . which leaves us uninformed as to whether the policy should be implemented” in the real world (p. 21). When research producers do examine why policies or programs are effective, they like to isolate the explanatory variable responsible for shifts in the outcome of interest. Yet researchers define efficacy in such a narrow way that “credibility in estimation can lead to incredibility in use” (Deaton & Cartwright, 2018, p. 3). Real life and real policy contexts are complex and involve numerous interdependent influences. Both our analytical tools and our way of framing research questions too often fail to incorporate this complexity (Maynard, 2006). Even with our most rigorous forms of scientific evidence, huge gaps in knowledge can remain when the findings are applied to the real world. Applying findings is an important element of effectiveness in the policy world, which too often has been overlooked in scientific inquiry in the research world. Theoretical Consideration #3: The Societal Structure of Democratic Institutions This theoretical cluster focuses on the societal structures that link research and policy. All rich democracies produce a cadre of academicians and researchers. Unlike most democracies, however, the U.S. version tends to fragment power in very deliberative ways. We celebrate the pull and tug of the democratic process, all of which serve to diminish the prospects of any policy actually seeing the light of day or being decided by a single factor. In some important ways, our policy process presumes competition and contention, a kind of Darwinian process where only the sturdiest ideas survive. This, however, is hardly an atmosphere that embraces elegant reasoning and dispassionate dialogue. In the United States, the pathways for integrating the institutions of research and power are, to borrow a phrase from Robert Frost, roads less often taken. In other democracies, however, trails have been blazed between the seats of power and the ivory tower that allow for a more regular and rational interchange of ideas and insights. Traffic is heavier, according to Wilensky (1997), because of where the experts 11

P roblem , P layers , and P olicy P rocess

are located. In contrast to the United States, experts in more corporatist countries like Austria, Germany, Japan, Norway, and Sweden, and to a lesser extent, Belgium and the Netherlands, are situated in strong political parties, more institutionalized labor and employer federations, and moderately centralized governments with the capacity to implement policy. In these democracies, the argument goes that problems identified in the seats of power more readily bubble up to the ivory tower, and knowledge gained in the ivory tower more systematically percolates down to the ground. Theoretical Consideration #4: The Structural Environments in Which Researchers and Policymakers Operate This set of theories focuses less on societal structures and more on the actors and modes of decision-making in the research and policy communities. These theories posit that research is underused in policymaking because researchers and policymakers come from disparate communities and operate in distinct structural environments that engage in discrete core technologies. The very institutions through which research is generated and public policy is enacted typically operate in ways that conflict with one another and that can be counterproductive to evidence-based policymaking. To bridge these conflicting communities, Nathan Caplan (1979) introduced the original two-communities theory that proposed that research was underutilized in policymaking because researchers and policymakers come from two different communities. Our latest reincarnation of this theory has extended beyond its conceptual appeal to deconstructing the “cultural” impediments to optimal communication. Culture is defined as a whole set of norms, rewards, and operating procedures that profoundly shape the way inhabitants think, act, and perceive the world. Researchers and policymakers are conceptualized to bring disparate professional backgrounds to their work and to operate in distinct institutional settings that substantively shape how they view their role in the world and act on a day-to-day basis (Bogenschneider & Corbett, 2010a; Bogenschneider, Corbett et al., 2019). They speak different languages, use different communication styles, and respond to diverse signals and incentives. The foundational premise of these theories is that a better understanding of a community’s inhabitants, institutions, and cultures could improve communication between researchers and policymakers, and enhance research utilization in policymaking. Understanding the cultural forces that frustrate the production and consumption of research is fundamental to developing strategies for effectively facilitating its use.

What We Need to Know to Enhance Evidence-Based Policy: Initial Factors to Consider We must confront an age-old conundrum. As our capacity to increase the supply of research has increased exponentially, we still struggle with bringing objective, dispassionate analysis to bear upon some of our most vexing and perplexing social challenges (Gamoran, 2018; Shonkoff  & Bales, 2011). Our operating premise, perhaps overly pessimistic, is that we can do little about many of the explanatory “suspects” that we have reviewed here, which suggests the futility of using social science research to change the way the policymaking process works. Indeed, it is unlikely that policymaking will develop into a rational reflective process, that researchers will develop methodologies that completely overcome the limitations or biases inherent in their studies, or that more centralized societal structures will emerge to integrate research and power. 12

W hy a D isconnect B etween R esearch and P olicy

Thus, we focus instead on the broader structural environments in which research producers and consumers operate. In this work, we introduce an extended model, which we call the community dissonance theory (see Chapter 7 for additional details). This theoretical framework portrays research producers and research consumers as functioning within a discrete number of disparate communities that find it difficult to communicate with each other. This communication breakdown occurs because each community operates within distinct professional and institutional cultures with different decision-making processes, interactional styles, epistemological frameworks, influence loops, focal interests, and salient stakeholders. Thus, the underutilization of research in policymaking is because of a communication gap that is attributable, at least in part, to behavioral factors. If the underlying forces are indeed behavioral, then they may well be less immutable and more malleable than many of the other factors at fault. If we acknowledge and pay careful attention to several pragmatic processes and procedures, we can increase communication and trust across the two cultures and, in so doing, enhance the utilization of research in policy decisions. In keeping with this theoretical argument, we adhere to the more moderate view of many policymakers (Kenley, 2019; Miller, 2019; Olsen, 2019) and scholars (Gormley, 2011; Grisso  & Steinberg, 2005; Henig, 2008)—that research can have at least some incremental impact on policy some of the time. Surely, it can have a broader impact than it currently enjoys. At the same time, we are fully aware of the complexities of using evidence to inform policy, the topic we turn to next. In truth, there is no simple, linear relation. Sometimes science is used, sometimes not. Sometimes it is used correctly, other times it is misused and misconstrued for political or partisan purposes. Sometimes researchers communicate well with policymakers, and other times they appear to represent quite different species. Efforts to connect research and policy can be fruitful, frustrating, futile, and even fun. To get us started down this road, let us capsulize some of the challenges of getting research producers and consumers on the same page. We delve into these in more depth and sophistication as the book unfolds. Let us start with a few factors that may impede our understanding of the science-policy connection and that make quick fixes unlikely to be of much use. The basics that researchers need to know include getting a handle on differences in the communities of research consumers and producers, the distinctive ways that they interact with each other, and the nature of research and policy questions.

Key Concepts 1.2.  Why Connections Between the Research and Policy Communities Are So Challenging • • • • •

Not all research consumers are alike; Not all research producers are alike; Not all interactions between research producers and consumers are alike; Not all research studies lend themselves to policy questions; and Not all policy questions lend themselves to research studies.

Not All Research Consumers Are Alike We must be far more appreciative of the complex character of the policy world. On the research-consumption side, we have those who make policy. Lawmakers in statehouses 13

P roblem , P layers , and P olicy P rocess

and Congress may get the most public attention, but they are not the only actors involved in doing the public’s business. Many higher-level officials in executive agencies also make policy as they interpret legislation, write regulations, and make the myriad of implementation decisions that transform policy ideas into practice (Lester, 2018). At the same time, we have a host of public officials who manage policies and programs on a daily basis. Though it is tempting to dismiss this population as those who merely execute decisions made by others, this view would be naïve indeed (Eggers & O’Leary, 2009; Lipsky, 1980). Discretion at the operational level is a powerful tool for doing a lot more than merely carrying out what others have determined. Also, public functions increasingly are being outsourced to nonprofit and for-profit agencies. Finally, there are all those who work at the margins of the policy process in hopes of shaping policy decisions and how they are carried out. All these policy actors look to research to inform their craft. What each group needs from research producers, however, may be quite different. Tom and Karen have compared notes on the research consumers they have worked with in the Welfare Peer Assistance Network (WELPAN) and the Family Impact Seminars (Seminars) initiatives. WELPAN was a network of senior welfare officials who met to dialogue with their peers about cutting-edge ideas and directions that included, from time to time, top political appointees as well as bureaucrats with a more operational focus. The Seminars, on the other hand, are a model for bringing quality research to state legislators as well as other high-level policy officials from the governor’s office and executive branch agencies. Across these two initiatives, we have worked to bring research and sound analysis to a variety of policymakers. Based on our combined experiences, two overriding lessons emerged. First, where you sit shapes where you stand. Legislators typically need general knowledge about what works and whether the policies being floated are good ideas. They need to be pointed in the direction of identifying which policies will be effective and efficient. Similarly, top political appointees in executive agencies need input on a more general level. As you move down the state agency bureaucracies, the information needs change somewhat. Agency bureaucrats have specialized training and experience on a narrower scope of expertise. For them, salient pieces of the puzzle are information about why things work as they do, whether they apply to different populations, and what best practices can make things work better. Despite these differences, there are many similarities. All policymakers are busy. They need input in a way that they can understand. They need a basis for trusting the messenger. Not All Research Producers Are Alike Similarly, we cannot assume that all research producers are alike. They differ in so many ways. First, they differ by disciplinary preparation. Economists do not see the world in the same way that sociologists do, or political scientists, or any of the other disciplines that engage in social policy research. Second, they differ in terms of methodological preferences. Some feel that social knowledge can only be advanced through formal experiments where subjects are randomly assigned to treatment and control groups. Others are comfortable with quantitative methods using econometric and other advanced estimation techniques to deal, for example, with observed and unobserved heterogeneity across key groups. Still others feel that contributions to the knowledge base can come from qualitative methods and case studies, and that these less quantitative methods add richly textured information to the understanding of a complex world. 14

W hy a D isconnect B etween R esearch and P olicy

Finally, research producers operate in vastly different institutional settings. There are academics doing basic research and academics doing applied work. Some are situated in research-oriented universities, others in think tanks and evaluation firms, others in advocacy organizations, and still others within government itself (Goodvin  & Lee, 2017). Each of these groups has its own operating climate and professional culture. Even today, much of the best applied research and program evaluation in the social sciences takes place outside the ivory tower (Tseng & Gamoran, 2017). Again, where one sits, as the old saying goes, shapes where one stands in critically important ways, such as how one thinks about research and what its contributions to the policy process might be. As an example, the Institute for Research on Poverty is a university-based interdisciplinary research organization focused on a common interest—the understanding of poverty and its amelioration. And some of that collaboration does take place. Yet achieving the ideal of interdisciplinary research remained a challenge until representatives from disparate disciplines came to appreciate what each can contribute. Over his four decades at the institute, Tom noticed that when economists gave a talk at a sponsored, brown-bag seminar, the audience often was dominated by economists. When a sociologist delivered the talk, the composition of the audience morphed in the expected direction—more sociologists. Tom occasionally noted that those in the audience from other disciplines (than the presenter) could be seen glancing at their watches as if to say, “When will this drivel come to an end?” Not All Interactions Between Research Producers and Consumers Are Alike Not all settings and strategies where research producers and consumers intersect are the same. How do you get information into the hands (and minds) of policymakers in ways that will be useful to them? Both worlds struggle with this issue. Think about it first from the policymakers’ perspective. What if they confront what we call a wicked social problem where desired outcomes are conflicted, theory is unsettled, and tactics for making progress are disputed (Sutherland et al., 2012)? Where would the decision-makers go for informative research? We know they don’t read scholarly journals (McMurtrie, 2013), but would they read a lengthy report, seek out a technical briefing, or attend an academic conference? What about the researcher who might have an answer to the question, or a new way of looking at it? What do they do with their findings—mail an article to a legislator or the governor, call and ask for a meeting, or work with intermediary organizations and advocacy groups? The point is that research producers and consumers come from different worlds. They operate differently in terms of preferred relational styles and respond to distinct environmental or social cues. The primary objective of researchers is the discovery of evidence, whereas the primary objective of policymakers is the instrumental use of evidence, broadly conceived as designing good legislation, stopping bad legislation, persuading colleagues to their point of view, earning the respect of colleagues, and so forth (Bimber, 1996; Bogenschneider, Day, & Parrott, 2019). Influence in the policy world is based on trust and common sense. Influence involves who is delivering the message, how information is communicated, and whether the message comports with underlying values (Bogenschneider & Bogenschneider, 2020). Influence in the research world is based largely on using accepted methods and rigorous procedures, and convincing others that scientific protocols were followed in the pursuit of truth. Research is based on precise adherence to well-established standards, whereas politics is based on negotiation and compromise and even pure power. Blending these worlds is not easy. Two of many illustrative examples are given here. 15

P roblem , P layers , and P olicy P rocess

The trade organization for welfare and related human service organizations held a meeting many years ago that brought together the leaders of welfare reform organizations and researchers from universities and top evaluation firms. The purpose of the meeting was to summit on how we should evaluate welfare reform as it unfolded. Over the first two days, researchers lectured to the audience about what they were doing and why it was important to the welfare leaders. On day three, several welfare leaders were scheduled to talk about their management and theoretical issues for which they needed help. But only three researchers remained for the third day. Finding out what was on the mind of their purported audience apparently was not a big priority for many members of the research-producing community. That struck Tom as extremely odd. Why wouldn’t researchers be falling all over themselves to find out what the consumers of their research were concerned about? However, this disconnect goes both ways. One WELPAN meeting focused on charting a possible future direction for the network. Members were asked a forced-choice question about whether future meetings should focus on innovative ideas drawn from research OR from other states. All the states (save one) chose their peers as the preferred source of new ideas, a finding similar to that in a 2001 independent review of WELPAN (SAL, 2001, p. 14). This experience raised Tom’s curiosity as to why and how the currency of research had been so marginalized. It was not that this group necessarily disparaged science; they did not. It was a case of valuing more the input of those with whom they shared a common experience and a sense of trust. This ethnocentric focus on the interests of one’s own community is one critical component of research underutilization. Not All Research Studies Lend Themselves to Policy Questions Too often, researchers focus on questions of interest to them. That is, what will advance their careers, bring scholarly distinction, or be of interest to their disciplinary colleagues (Gamoran, 2020)? Worse, they investigate questions simply because they can—the data and methods are readily available or financial support can easily be obtained. Perhaps the most overlooked aspect of increasing the policy utility of research involves getting the question right (Bednarek et al., 2018; Tseng & Gamoran, 2017). That involves ramping up the level of communication between research producers and consumers before research initiatives are set in stone. It involves generating new levels of communication and cooperation to ensure that research taps issues salient to the public in ways that can contribute to their resolution. Most important, this perspective suggests that the ideal points of interaction should not begin when the research is completed, but rather when the questions of interest are initially formulated (Tseng, 2012). Not long after national welfare reform passed, Tom was asked to participate on an expert panel convened under the auspices of the National Academy of Sciences to address how best to evaluate the success of welfare reform and help the reform movement advance in the right direction. As Tom listened to fellow top researchers who composed the panel, he began to wonder if the study group was headed in the wrong direction. As technically sophisticated as the group was, being replete with members from the academy and top-shelf research organizations, they were not particularly attuned to what was happening on the front lines of welfare reform. Welfare was becoming less and less about getting a check out the door and more and more about changing individual and family behaviors. Research questions and methods appropriate to understanding an

16

W hy a D isconnect B etween R esearch and P olicy

income-transfer program were not necessarily well suited to assessing a set of complex initiatives focused on behavioral change. In short, the group of experts had to spend time and energy understanding the dynamics of how welfare programs had changed to get the research questions right. If you do not do that well enough, everything else will fail. Feeling somewhat frustrated that the mostly academic-based committee members were disconnected from the rapid changes taking place in the culture of welfare offices across the country, Tom persuaded them to allow him to show a set of video clips he had taken at local welfare sites in several states. Despite many good-humored jokes about who would bring the popcorn, the videos, which had mostly local officials talking about how they were radically transforming welfare, was an eye opener to the researchers. They realized that they risked developing recommendations for evaluation methods based on programmatic understandings that quickly were becoming outdated. Yet the complexity does not stop here. Not all policy questions are amenable to scientific inquiry. Not All Policy Questions Lend Themselves to Research Studies Yet another macro factor must be considered—the very nature of the policy question itself. Arguably, some questions are more amenable to inputs from science than others. In recent studies, the odds that research would be applied to policy decisions were less likely on issues that were driven by morality, ideology, or passion. For instance, research was less likely to sway people on lightning-rod issues, such as abortion and reproductive rights, that divide people on religious or moral grounds (Bogenschneider & Bogenschneider, 2020). Yet many researchers tend to overgeneralize the impenetrability of research on moral issues to all value-based issues. Looking at the history of social policy, some of the nation’s most successful major legislation has been driven by shared values—women’s rights, drunk driving, civil rights, and smoking (Orren, 1988). In Karen’s recent study, we uncovered another pocket of issues that comprised a shared value premise where partisan polarization was less prevalent and where science played a more substantive role— youth and family issues. According to the legislators who championed them through the policy process, youth and family issues, unlike issues such as climate change, were widely seen as a “common interest”, though there was not always agreement about a “common answer” (Bogenschneider et al., 2020). When it came to identifying policy answers, one “very big split” between the parties was whether government was perceived as a force that can cause problems or as a recourse that can resolve them (Bogenschneider et al., 2020; Crowley, Supplee, Scott, & Brooks-Gunn, 2019). Many of the most perplexing policy questions revolve around what we call wicked problems, where normative, theoretical, and partisan contention exists, and the problems themselves seem to exceed the authority of science (Sutherland et  al., 2012). As we noted earlier, technical questions are one kind of challenge. They can be exceedingly complex. But given enough resources, science could provide answers to an array of what are essentially technical challenges. So-called wicked social problems typically involve emotional and normative questions where ends are in dispute, theoretical understanding is uncertain, and no policy alternative enjoys widespread support (Sutherland et al., 2012). For a long time, welfare reform had proven an intractable challenge because it touched on core, often contentious, societal issues—equality, fairness, family, personal

17

P roblem , P layers , and P olicy P rocess

responsibility, sex, and work to name a few. Commenting on ways to evaluate controversial welfare reform initiatives, Robert Lovell, a former head evaluator of social ­programs in Michigan, highlighted . . . the range of difficulties we face in applying the social science tools developed in the last 50 years to the problems faced by states in the next 5 years. This is not rocket science, it is much more difficult. The rocket engineer chooses among cost, weight, and reliability, has a very successful theory of physics to predict results, knows the goals exactly, and can test each component before assembling the product. Our tradeoffs among cost, reliability, timeliness, protection of subjects, and threats to validity are more complex, our theories have only weak, predictive power, we have as many goals as the number of programs we study, and we seldom have the luxury of studying each of our components separately. (as cited in Boehnen, Corbett, & Ooms, 1997, p. 9) Given all these complexities of getting research producers and research consumers on the same page, bringing research to bear on policymaking will take a policy mindset. Next, we turn to the principles that can guide you in the right directions and the caveats that can steer you away from unproductive deviations.

What We Need to Do to Build Evidence-Based Policy: Guiding Principles and Caveats The hard truth is that we live in a complex world. We touch on all these complexities, some more than others, in the coming chapters. We do so with trepidation and hope. The trepidation emerges from our awareness of just how demanding is the task of doing both good science and good public policy. For example, consider the fact that universitybased academics alone are generating research and analysis at a rapidly increasing rate, far more than can be absorbed by even their academic peers, let alone busy decisionmakers (Jinha, 2010; Peterson, 2018). This increasing volume of output leads to more differentiation and specialization even within disciplines, thus complicating the task of cross-disciplinary communication and collaboration. The political world, at the same time, is seen as drifting further and further into partisan, ideological divides that threaten to shove science aside to a position of irrelevance (Binder, 2003; Contandriopoulos et al., 2010; Henig, 2008: Kendall-Taylor & Levitt, 2017; Peterson, 2018). We are in an era where many argue that we must do more to promote the marriage of research and policy. In a sense, we argue for moving in the direction of evidence-based policymaking as a marriage between those who generate research, our research producers, and those who use research as they go about society’s business, our research consumers. Like all marital bonds, it is easier said than done. These diverse communities must begin to talk with one another, often and openly. Self-imposed isolation leads to mutual suspicion which, we argue, is counterproductive. The communication patterns necessary to make any professional marriage work between our increasingly complex set of research producers and consumers MUST go both ways, even multiple ways. This advice is not much different from what one would receive from a decent marital therapist. Ultimately, each will have to look honestly at the cultures within which they function and be sensitive to the culture of the other. Each may have to question basic assumptions and ways of doing business. Our challenge is to think through the ways in which we can 18

W hy a D isconnect B etween R esearch and P olicy

mitigate the impediments and difficulties that make communications between actors situated in these different cultures so difficult. As we consider our overarching conundrum regarding whether science can play a more central role in the doing of policy, we might profit by remaining cognizant of four key factors. We argue that one needs to begin with a policy mindset—knowing what time frames to expect, which types of intercultural awareness are needed for entering and operating in a foreign land, where to start, and even how to define success.

Key Concepts 1.3.  Building Evidence-Based Policy: Guiding Principles and Caveats 1. 2. 3. 4.

Doing policy is a marathon, not a sprint; The notions of inhabitants, institutions, and cultures are foundational concepts; Personal relationships are key; and Common notions of success are overly simplistic.

1.  Doing Policy Is a Marathon, Not a Sprint We all tend to look at the present and assume it represents an enduring reality. Or perhaps we look back a decade and swoon over the good old days. In short, we all fall into thinking about the world in a linear fashion where things seem to be getting better or worse along some form of monotonic trend line. Reality, of course, is more complicated. The progress of science does not fast-forward based on a single killer study, but is rather an “independent, collective, cumulative, open-ended enterprise of knowledge creation and testing” (Henig, 2008, p. 152). Similarly, in policymaking, we often anticipate that a lightbulb will go off, an insight will sweep through the body politic, and suddenly a new era of understanding will be embraced for all time. That is not how wicked problems are resolved in the real world (Sutherland et al., 2012). Progress is in fits and starts, trend lines more curvilinear than straight, and success not always manifest to the casual glance or at a single point in time. Moving to a policy world where rigor plays a central role would represent a major transformation. Thus, doing public policy is a marathon, not a sprint (Tseng, 2020a). 2.  The Notions of Inhabitants, Institutions, and Cultures Are Foundational Concepts Progress toward an evidence-based policy world, at least to the extent that such a thing is feasible, is intimately rooted in the culture of the world in which each of us functions. This notion of culture is all-embracing. We will spend a great deal of time exploring the notion of culture throughout this work, so we will only provide the barest outline of the concept here. We use the word culture to embrace the languages, perspectives, prevalent behaviors, relationships, rewards and punishments, rules, and understandings in our individual worlds. It is the soup in which each of us functions, often without understanding its character or impact. It literally shapes how we see the world around us. For the moment, let us start with three focal points. First, the inhabitants of every cultural milieu bring with them a set of priors that shape their interactions in life. Second, each 19

P roblem , P layers , and P olicy P rocess

inhabitant functions in an institutional setting that shapes behavior through its norms, languages, practices, and purposes. And third, there are distinct sets of cultural understandings that one embraces, almost unconsciously, which shapes relationships with colleagues and others as they go about their daily business. We must understand how culture is learned, so we can better understand its role in our professional lives and the professional lives of those we hope to engage. Nominally, our own culture is shaped by many things, but foremost is our training, our professional position, and the community groups with which we choose to engage (or which are thrust upon us). Remarkably, however, we remain unconscious of the cultural specifics that dominate our daily lives. We think of things that are specific to our world as universals. So, a researcher or academic approaching a policymaker is likely to assume that this other individual sees the world as he or she does (Kendall-Taylor & Levitt, 2017). Researchers may resort to using evidence in an aggressively assertive or advocacy-oriented manner. However, if they had knowledge of the ways researchers are stereotyped in the policy world, they would be persuaded that this is precisely the wrong tack to take. Excessive hubris feeds into the stereotype that academics are elitist eggheads. But why should researchers think otherwise and adopt a more sensitive approach unless, and this is the big factor, they take the time to look and understand this other person’s worldview? Nothing creates more misunderstanding and breakdowns in communication than lack of intercultural awareness. Again, this is a theme we will revisit throughout. 3.  Personal Relationships Are Key To build evidence-based policy, where should we start? We do not have a shortage of policy issues that could benefit from science. We have never had more science and evidence than we have at present (Jinha, 2010). We have instant access to information through the Internet, social media, 24-hour news cycles, and so forth. These are not the problems in moving toward an evidence-based policy world. Science does not have a communication problem, according to Yanovitsky (2019); it has a relationship problem. The fundamental importance of relationships is one of the main findings in a decade of research utilization studies being funded by the William T. Grant Foundation (Bogenschneider et al., 2020; Dumont, 2019; Friese & Bogenschneider, 2009; Gamoran, 2018; Mayne et al., 2018; Nutley et al., 2007; Oliver & Cairney, 2019; Oliver, Lorenc et al., 2014; Tseng, 2012, 2020a; Tseng & Gamoran, 2017; Tseng, Fleischman, & Quintero, 2018). In accord with this body of evidence, our work in the policy world over many years has suggested to us that how and who brings evidence to policymakers is as important as the quality of the research itself. Because the policy world is so inundated with input, and the incentive for obfuscation and even subterfuge is so strong, trust in the messenger is key (Asen & Gurke, 2014; Bogenschneider & Bogenschneider, 2020; Dumont, 2019; Gamoran, 2018; Honig, Venkateswaran, McNeil, & Twitchell, 2014). Trust requires several factors. You must take time to get to know your audience. You must seek out what they need to know, not what is easiest for you to provide. You must determine the most effective ways to communicate what they need and, even more importantly, want to know. You must respect your audience, understand their world, and realize that they do not necessarily see things as you do. Everyone is busy. This is particularly true of research producers and research consumers. For busy policymakers, the credibility of research is determined by the reliability of the source. Trust in the source serves as a cognitive shortcut for validating or vilifying 20

W hy a D isconnect B etween R esearch and P olicy

the credibility of research (Bogenschneider  & Bogenschneider, 2020). For researchers, publishing a study, particularly in a traditional venue that will earn the kudos of their peers, is not always the easiest way to generate the biggest returns in the policy world. Expanding one’s reach takes making the time to build relationships. 4.  Common Notions of Success Are Overly Simplistic What we look at as evidence of success and what we ought to be looking at are often two different things. We are all impatient, perhaps overly optimistic on occasion. Advocates of evidence-based policymaking want to look at the end points of the process and see results. Can we trace new laws and regulations and program designs directly back to research and rigorous analysis? Granted, there are outlier examples where such a case might be made. But these are likely the exceptions rather than the rule. Science is not a monolithic and unifying input. Even in the physical sciences where laws appear more universal, controversy and competing interpretations abound. Is light a particle or a wave? In the social sciences, the world is even more uncertain. So-called evidence is often weak or contradictory or simply hard to apply directly to the real world. More to the point, the policy process is long and complex with the final decisions being made at the conclusion of a lengthy process. There are many ways in which science can influence the policy process along the convoluted path from problem awareness to ultimate solution (Bogenschneider, Corbett et  al., 2019; Gamoran, 2018; Prewitt, Schwandt, & Straf, 2012; Weiss, 1978). For the policy novice, there is a tendency to look simplistically at causal links that are unidimensional in character. Does X cause Y? So, researchers and evaluators are trained to disentangle complexity and to seek out simple explanations. Will X reduce poverty, or get less-skilled people jobs, or result in happy families, or lead to productive young adults? It is a natural enough tendency to seek a singular solution, but likely misplaced. Just as there are few silver bullets that might resolve social maladies, there are not likely to be defined points of influence on policy decisions. A law or new program design seldom comes directly from a single “killer” study advanced at a given point in the policymaking process. Some time ago, an innovative “make work pay” initiative was tried in Milwaukee and later a similar effort was launched in Canada. It was called the New Hope project. The pilot programs were carefully designed and evaluated by the nation’s top social policy evaluation firm employing the highest standards of science (Miller, Huston, Duncan, McLoyd, & Weisner, 2008). The Annie E. Casey Foundation then supported an intensive effort to communicate the positive findings on labor supply and even child educational outcomes to policymakers across the land. Yet no one adopted the program in its entirety. Was it a failure? Hard to say since aspects of the model program were widely implemented in various forms. This pilot initiative, partly because the results were unassailable from the scientific point of view, did inform the public conversation and influence thinking about various aspects of assistance to poor families.

Conclusion We tried to develop a book of broad applicability even though we largely illustrate our ideas using selected policy domains, namely, the social policy arenas of poverty and family policy. We acknowledge that there is scant evidence that policymaking can be transformed magically into a dispassionate, evidence-based enterprise. We are fully aware that there is much to support a pessimistic perspective. Yet we remain optimistic. We like 21

P roblem , P layers , and P olicy P rocess

to think we are optimists who choose optimism with both eyes wide open to the potential perils. The average citizen wants good government, wants solutions as opposed to endless division and debate, and wants competence over scoring political points (Smith, 1991). We have come to know many researchers who are committed to applying their work in the real world, even when the culture of the academy does not always appreciate or reward their endeavors. We admire their passion. We admire their intellect. We admire the research skills they bring to tackle some of the toughest issues of our times. We also have come to know many policymakers out there who seek information on which to make hard choices about what to do for whom. They have always been there, even when it is not obvious at first glance. Recently, Karen personally sat across the desk from 128 sitting legislators in conversations that sometimes lasted over two hours. We respect them. We admire their passion, sometimes working on a bill for a decade before it passes. We admire the political skills they bring to tackle some of the toughest issues of our times. Getting the research producers and consumers together, however, has proven a lot more difficult than one might have imagined. We know this gap between research and policy is quite unyielding to easy remedies. Yet examples exist that demonstrate it can be done, though not easily or without effort. Nonpartisan research institutes like the Washington State Institute for Public Policy have used rigorous meta-analyses to respond to policymaker requests with “Consumer-Reports” style reports that consistently have driven state policy decisions (Aos, 2007; Goodvin & Lee, 2017). Their approach, known as the Results First Initiative, has increased spending on effective programs and cut spending on ineffective programs in 27 states (Lester, 2018). A cadre of pioneering researchers, as described in Chapter 9, have stepped out of their comfort zone and worked closely with policymakers to drive policy changes (K. Bogenschneider, 2020a; Boehnen et al., 1997; Day, MacDermid Wadsworth et  al., 2019; Grisso  & Steinberg, 2005; Lipsey, 2009, 2012; Olds, 2010, 2016; Yoshikawa  & Ariel, 2011). The Manpower Research and Demonstration Corporation, perhaps the preeminent evaluation firm doing welfare and workforce development studies, takes pride in doing a good deal of testifying before public officials and in developing easy-to-read reports that summarize their findings. Selected evaluation firms, policy think tanks, and representatives of the philanthropic community have taken great pains to prepare reports and documents that are written with a policy audience in mind. Furthermore, on the policy side of our classic divide, we have seen politics become increasingly vitriolic and partisan. Yet, even amidst a resurgence of bitter partisanship, hints of positive change still occur. For the past decade or two, state and federal policymakers have been experiencing an “evidence-based uprising” (Haskins, 2018, p. 8). The passage of six “tiered-evidence” social programs enacted between 2009 and 2011 and the Foundation for Evidence-Based Policymaking Act in 2018 is pushing federal agencies to make better use of government data (Baron, 2018; Gamoran, 2019). Recently, federal policymakers have come together around criminal justice reform (bop.gov/ inmates/fsa/overview.jsp), the Families First Coronavirus Response Act (www.congress. gov/bill/116th-congress/house-bill/6201), and the Family First Prevention Services Act (https://familyfirstact.org/). If we don’t find ways to help researchers talk with policymakers, and vice versa, what might the future look like? Can we envision more partisanship and interest-driven policies? Can we envision even greater influence of money over science (Rivlin, 2006)? Will we see growing despair around the competence of government and further erosion of 22

W hy a D isconnect B etween R esearch and P olicy

public support for social science research (Tseng, 2019b)? When asked about what was important for academic researchers to know in working with policymakers, a state senator who held a leadership position replied as follows (response is paraphrased): The University seems to be isolated and does not encourage and initiate contact with policymakers. Academics need to be much better connected to politics; otherwise, their budgets will get cut. In the face of significant cuts to the University and vindictive cuts to Extension, the University seems to think “this is not real.” They seem to be living in a different world as their budgets get sliced and brutalized. The University doesn’t seem to care, and academics don’t want to get their hands dirty in the political process. When Karen is recruiting faculty and Extension staff to lead the state Family Impact Seminars, she often hears a common refrain—“This is not a good time. Our budgets are too tight.” Karen’s resolute reply is “If not now, when?” For some in academia, the time may be now. It may now be close to a matter of survival (see Tseng, 2019b). We are not naïve. Doing public policy will never become a fully data-driven process, nor should we expect it to be. Any democracy must live with a give-and-take that is governed, in part, by values and positions that will never be fully informed by research evidence. Differences of opinion will always be vetted in an environment where a marketplace of ideas plays out in the public arena. In an important way, this is a strength inherent in our democratic process. At the same time, there is much that research evidence can do to reduce the level of conflict that tends to paralyze the policy process. Science will never fully replace values and power in the development of policy or in the ways that public matters are managed. In the end, though, we remain convinced that science can play a much bigger role than it now does.

23

2 THE HISTORY OF EVIDENCE-BASED POLICY A Long-Held Dream Karen Bogenschneider and Thomas J. Corbett

The role of research in informing public policy is a conundrum that has confounded researchers since the origins of scientific inquiry. This chapter overviews the long and labored history of the emergence of contemporary science. Embedded in the complicated evolution of science is the experience of prior generations who dreamed of an evidencebased policy world. The chapter concludes by fast-forwarding to the present and considering whether it is a dark time for science, as some contend, or whether the headlines in the 24-hour news cycle belie what is happening below the surface. Experience keeps a dear school, but fools will learn at no other. Benjamin Franklin (1757, p. 18) I hope that our level of greatness rises to the level of our opportunity. Theodore Roosevelt (1893) Doing public policy is not easy. Doing public policy has never been easy. The hope of applying research evidence to public policymaking has a long and complicated history. Yet the tortured path taken by science enthusiasts has not dulled the devotion of those working diligently to refashion efforts to improve society. Many seek to have the social sciences play a policy role that more closely reflects the position enjoyed by the hard sciences in industry, health, and space exploration. Hope for an evidence-based society persists in the face of sobering reality. The fact that younger scholars stumble across old questions and solutions as something new and exciting must say something about the intractable nature of some issues. Historical awareness of prior hopes and efforts is a bitter medicine at times, but an important prophylactic against overconfidence in the enthusiastic reformer. The disappointments of experience have a way of dulling our enthusiasm, yet never fully extinguishing it. We cannot abandon easily the long-held dream that the public’s business might be conducted in a more thoughtful way that we trace back to the origins of science itself.

This Long-Held Dream The belief that reason should complement power in governance has arisen throughout the ages. In the early Chinese dynastic period, Confucius placed great faith in an educated 24

T he H istory of E vidence - B ased P olicy

elite who would bring discipline, right thinking, and selflessness to the administrative tasks of running a complex society. Plato argued that power and authority ought to be concentrated in a class of philosopher-kings who might have the wisdom and judgment to exercise their roles with judiciousness and expertise. Auguste Comte, a 19th century thinker, regarded as the first philosopher of science, argued that policymaking should be left to what he termed the positive priesthood (Bogenschneider, Corbett et al., 2019).

Key Concepts 2.1.  The Evolution of Science and Scientific Thinking in Public Policy • Francis Bacon, Sir Isaac Newton, and John Locke provided the origins of the contemporary scientific approach in the 17th century. • James Lind used crude experimental methods to find a cure for scurvy in the 18th century. • David Hume popularized the notion of empiricism or direct observation by the end of the 18th century. • Edward Jenner used the scientific approach to discover the smallpox vaccination at the end of the 18th century. • Abraham Lincoln convened scientists in the Civil War period in what eventually became the National Academy of Sciences. • Scientists and policy reformers gathered in a Boston conclave that launched the American Association for the Promotion of Social Science in 1865. • The modern concept of the university emerged with the creation of Johns Hopkins University (1876) and Clark University (1887). • The use of science for public issues expanded greatly during the Progressive Era, the Great Depression, and World War II. • Another burst of optimism that science could solve social problems emerged during the 1960s and 1970s. • Ideology and partisanship have sobered earlier feelings of optimism in recent years.

The seeds of a scientific revolution, however, emerged as early as 1605. In that year, Francis Bacon in Novum Organum attacked deductive reasoning, arguing that “the logic now in use serves rather to fix . . . commonly received notions than to help search after truth.” (Barry, 2005). Later in that century, Sir Isaac Newton revolutionized physics and mathematics by developing inductive reasoning skills and mathematical tools, including calculus; both these advances were tentative steps toward our contemporary understanding of science. His contemporary, John Locke, emphasized the pursuit of knowledge through direct experience as opposed to revealed truth (Kuhn, 2012). These were all incremental movements away from a fixed understanding of the world and toward one in which the workings of the world about them were learned through systemic investigation. These were constructs that pushed us away from pre-ordained givens toward a world where we could unravel mysteries and create something better. In 1753, James Lind conducted a pioneering empirical study among British sailors and demonstrated that scurvy could be prevented by the consumption of limes. His comparison of those treated with an intervention against those not so exposed was a major 25

P roblem , P layers , and P olicy P rocess

methodological leap forward toward establishing causality in a more convincing fashion. The result was a healthier British navy that would substantially extend the Empire and a create a nickname that would stick until modern times: the British would henceforth be known as Limeys (Barry, 2005). David Hume soon followed by popularizing a concept that he termed empiricism (Kuhn, 2012). He and a growing number of his contemporaries realized that they might improve the human condition by systemically observing and investigating the world about them. The pace of technical change accelerated after a bright tinkerer figured out a way to pump water out of deep mines using a primitive steam engine. The industrial age was not long in following as more applications of this technological breakthrough followed (Rosen, 2010). But it was medical challenges that brought about the methodological advances later applied to solving so many societal problems. One prominent example is smallpox that had been a scourge for time immemorial. At the very end of the 18th century, Edward Jenner heard a milkmaid say, “I cannot take the smallpox since I have had the cowpox” (Barry, 2005). This comment intrigued him. Average folk around the world had already stumbled on the insight that had escaped the attention of the more learned among them. Milder forms of a disease could prevent deadlier versions of that or related ailments. Such insights, however, had never risen above the status of “old wives tales”. Medicine remained stuck with the technologies developed by the Greek doctor Hippocrates around 400 BCE and the Roman sage Galen at the dawn of the modern era. Jenner’s contribution was to rigorously examine whether inoculating patients with milder forms of a disease might have the hypothesized salutary effect. His systematic investigations proved his hunches and, along with several European scientists such as Koch in the 19th century, paved the way for modern medicine (Barry, 2005). Americans did not escape this elemental urge to use science to guide policymaking (Smith, 1991), though they lagged after their European counterparts in the beginning. For example, the prestigious National Academy of Sciences traces its origins back to the Civil War period. The emergence of ironclad warships played havoc with the operation of on-board compasses, given that the new lead siding attached to these crafts influenced the electromagnetic field. It proved embarrassing, not to say dangerous, when ship captains were ignorant of the direction they were heading during a naval battle. In response to this crisis, President Lincoln called upon the best scientists of the day. The professors he gathered solved the problem, a success that led to future leaders seeking out the help of scholars and the creation of what eventually became the National Academy we know today. Buoyed by this early positive experience, the federal government has been returning to the academy for help ever since, but with uneven success. Lincoln could not turn to the academy to resolve the central crisis of the day—slavery and the distinct cultural and political divides that separated North and South. Some issues then, and today, were not solvable with technical insight, numerical calculations, or simple brainpower and, thus, were beyond the purview of science. Another early example of a connection between American researchers and policymakers can be found in a conclave held in Boston in 1865. That gathering attracted 100 policy elite of the day—government officials, educators from the nation’s oldest colleges, writers, journalists, advocates, and charity workers interested in policy reform (Smith, 1991). Members of the conclave had witnessed the social and political transformations brought about by scientific advances in disease and hygiene, and technological developments such as railroads, steam power, and telegraphy. Confident that science also could successfully advance the social and economic problems of the day, they audaciously called themselves the American Association for the Promotion of Social Science. 26

T he H istory of E vidence - B ased P olicy

This name was not as pretentious as it sounds, given that modern scientific organizations such as the American Economic Association and the American Political Science Association date their roots back to this assembly. At the time, training in the social sciences was in its infancy. Still, members of the conclave considered themselves scientists. This scientific orientation set in motion the reflection required of each era regarding why and how research can guide the making of public policy. By the latter half of the 20th century, social policy and charitable philanthropy began to evolve from what had been mostly a religious and ethical responsibility to a public challenge addressable in a more sober and dispassionate manner. What came to be known as the scientific charity movement emerged in the decades after the Civil War (Trattner, 1974/1999). Help was to be provided to distressed communities through strategies based on careful data gathering and rigorous investigation of vulnerable families and the environments in which they lived. Simply doing good works was not enough to remedy society’s obvious failings. Rudimentary forms of what would evolve into social work practice was needed. Within a few years, the academy began to reflect these changes. Most notably, the first modern university was developed with the establishment of Johns Hopkins University in 1876. Until this time, the great seats of learning were dominated by the teaching of the ancient classics and the study of theological and philosophical questions. Knowledge was embraced and absorbed by looking to the past, not the future. It involved memorizing given truths and deducing reality from established principles, rather than unraveling the mysteries about us through systemic investigation. The Hopkins’s educational innovation, in effect, argued that knowledge was beginning to accrete at an increasingly faster rate. Four years was not enough to create a truly educated man or woman. Advanced, or graduate education, was now viewed as essential along with specialization in a specific discipline. “Renaissance men”, those capable of mastering all knowledge, had long gone extinct though that reality escaped complete acceptance at the time. A second graduate school was launched in 1887. Clark University, in Worcester, Massachusetts, was developed by Jonas Clark who designed the first building in a way that permitted conversion to a shoe factory if his educational experiment failed. However, it thrived, and the American Psychological Association was created at this institution in the late 19th century. At the time, academic interest was emerging about why individuals behaved as they did, particularly when they operated outside the norms of conventional society. Other specializations followed suit. The discipline of sociology emerged as scientists sought to understand how people, as a collective, functioned. In 1898, the first school of social work was established at Columbia University to systematically apply the accumulating knowledge of social and individual functioning to solve issues affecting vulnerable populations. Alas, progress was uneven across disciplines and specialties. As late as 1901, a student could be admitted to Harvard Medical School without any science or even a bachelor’s degree (Barry, 2005). The faculty of most of these medical schools were simply local practitioners who might be hired on the cheap. After all, there was not much knowledge to impart. All that was soon to change in the wake of a global pandemic that provided the impetus to transform medicine from an art to a science. In the 1918–20 period, the Spanish Flu killed from 50 to 100 million people worldwide, a bitter end to the carnage unleashed by World War I (Spinney, 2017). Once gaining a foothold, this trend toward systematic investigation and the development of science as a solution to physical and then social problems continued unabated. Settlement houses, such as Hull House in Chicago, began to collect data on the communities they served and to use those data to push for policies that would systematically 27

P roblem , P layers , and P olicy P rocess

address neighborhood issues (Trattner, 1974/1999). Help would no longer be provided to only one distressed family at a time. Rather, early social investigators such as Charles Booth and Beatrice Webb began to think about how information could be used to illuminate broader social challenges and to help shape their remediation through systemic policy and collective action (Nathan, 2000). The Progressive Era proved to be a rich period within which social dysfunction at the community level was systemically examined and documented. Reformers and enlightened government workers took basic steps to transition from initiatives based merely on religious or normative feelings to something founded on empirical evidence. Their data were crude, mostly based on collections of “investigations” that used primitive survey and observational techniques to minimize bias in sampling and data recording. Still, these community-wide efforts were an advancement over the investigatory efforts of friendly visitors. These well-meaning women attempted to reverse individual family suffering through the application of moral lessons. This so-called scientific charity movement took marginal steps toward more sophisticated data collection methods, but would hardly be considered scientific by contemporary standards. Tentatively, the Progressive Era saw the basic social questions being reframed from a focus on individual failure toward systemic or social breakdowns, a paradigmatic shift demanding new analytical methods (Katz, 1996). By the time the Great Depression reached its zenith in the 1930s, opposition to federal intervention in social and economic matters, which had weakened during the Progressive Era, further waned. Franklin Delano Roosevelt (FDR) had little time to worry about science as banks were closing by the hundreds, about a quarter of the workforce was unemployed, and poverty estimates from later years put the rate at something approaching 60 percent (Trattner, 1974/1999). He tried intervention after intervention, discarding those that seemed not to be working. Still, FDR often looked to universities for ideas and technical assistance. He brought Edwin Witte, an economist from the University of Wisconsin, to Washington, DC, to develop the foundations of his social welfare concept— the Social Security Act. The Wisconsin professor brought several colleagues with him and relied upon expertise from the academy for ideas and inspiration (Corbett, 2018c). Like Abraham Lincoln turning to the academy for solutions to wartime problems, the partnership between the federal government and the academy had been revitalized, but this time for assistance with domestic policy issues. President Franklin Roosevelt attracted hordes of intellectuals to Washington by establishing various advisory and planning agencies, including his legendary Brain Trust that played an active role in the design of the New Deal. He also reorganized the Executive Office in the late 1930s in ways that ensured his successors would have access to intellectual resources. By 1939, 7,800 social scientists were working in the federal government (Carlson, 2002), and between 1940 and 1945, research expenditures increased by some 1,500 percent (Featherman & Vinovskis, 2001). Science was moving into the arenas of human behavior and social organization in a big way. If the forces of nature could be subject to analytical scrutiny and understanding, why not individual and societal behaviors? Rigorous investigation and the application of analytical reasoning would crowd out moral or ethical arguments in the public deliberation of social policy matters. World War II served to stimulate an expansion in the use of science for public purposes, usually defense-related. Curiously, the “historical apex in the public image of science” was the dropping of the atomic bomb (Bimber, 1996, p. 13). For many U.S. citizens, and many others in government, the bomb and many other remarkable scientific advances during the wartime years, from radar to primitive computers, were concrete examples of 28

T he H istory of E vidence - B ased P olicy

how technology could address some of society’s most perplexing problems (Wallace & Weiss, 2019). Perhaps those eggheads could solve other problems that plagued ordinary citizens. Following the war, leaders in Washington fully expected, as did the citizenry, that scientists would turn their expertise to peacetime pursuits including advances in health, jobs, and the standard of living. Truman created the Council of Economic Advisers to keep the economy and labor market running smoothly and tried to push a national health plan to improve medical services as war-ravaged Britain had just managed to do (Skocpol, 1992). The first randomized controlled trials, which are the gold standard of evaluation, were published in the 1940s and 1950s to treat polio and tuberculosis. By 1962, the Food and Drug Administration began to require randomized trials for any new pharmaceutical drug (Baron, 2018). In the 1960s, many began to assume that, at long last, we had arrived at a new era of policymaking, one where data and informed deliberation were supplanting ideology and partisanship. Like getting a man to the moon, we could bring intellectual resources to bear on society’s most vexing social challenges. We seemed poised at the precipice of a new era in which an analytical, dispassionate mode of decision-making would prevail. Economist Robert Haveman captured the optimism of the 1960s when he noted that “logic, data, and systematic thinking were to compete with, if not dominate, ‘politics’ in the making of public decisions” (Haveman, 1987, p. 33). After all, increasingly sophisticated computers permitted us to employ larger data sets and to subject those data to astonishingly impressive analytical techniques. The number of policy schools, think tanks, and evaluation firms exploded (Smith, 1991). Resources from the philanthropic community expanded dramatically. Scholars and policy wonks gave a great deal of thought to the way policymaking was done, and how research and analysis fit into the doing and managing of public affairs. The 1970s and 1980s were known as the golden age of study and theoretical thought on research utilization (Weiss, 1983; Caplan, 1979; Lindblom  & Cohen, 1979). The two-communities conceptual framework of disconnected communities (i.e., researchers and policymakers) emerged and gained currency (Caplan, 1979). And governments at all levels invested increasing amounts in research focused on public policy issues (Haveman, 1987). Federal support for research and related development investments grew dramatically, more than doubling from less than 1  percent of gross domestic product in 1953 to almost 2  percent in 1965 (Congressional Budget Office, 2007). Moreover, faith in science was bipartisan. An abiding confidence in science was stated clearly by President Ronald Reagan early in his administration (1983, para. 16): I call upon the scientific community in our country, those who gave us nuclear weapons, to turn their great talents now to the cause of mankind and world peace; to give us the means of rendering these nuclear weapons impotent and obsolete. The post-war period did witness great advances in the social sciences. By the 1960s, economics was being transformed into a quantitative science, with the other social sciences soon to follow. Positivism would sweep the social sciences as research into all aspects of life became a growth industry. Yet with few exceptions, the social sciences have never quite fulfilled the heroic dream that breakthroughs in science could fundamentally resolve the big questions facing society (Henig, 2008; Smith, 1991; Weiss, 1999). One of those heroic dreams was that research could render the means to resolve poverty. 29

P roblem , P layers , and P olicy P rocess

Illustrating the Evidence-Based Movement— The Poverty Story When President Lyndon Johnson launched his famous War on Poverty in the mid-1960s, there was scant research on which to initiate this crusade. The architects of the War on Poverty in Washington realized they needed good science and scientists to advance their agenda. One example of this growing confidence in the benefits of scholarship was the creation of the Institute for Research on Poverty (IRP) at the University of WisconsinMadison in 1966. The institute was the first federally sponsored academic think tank devoted specifically to poverty-related investigations and was created at the beginning of an era associated with an explosion of large-scale, policy-oriented studies. In the early 1960s, one would have needed fewer than two pages to complete an exhaustive bibliography of studies related to poverty (Yarmolinsky, 1969). After the first 15 years of IRP’s existence, the center “had published 35 books, 650 discussion papers, and 18 special reports, and had spent about $20 million dollars” (about $100 million in today’s dollars; Haveman, 1987, p. 5). The Office of the Assistant Secretary for Planning and Evaluation (ASPE), the primary research office of what is now known as the U.S. Department of Health and Human Services, grew dramatically through the 1970s, reaching some 300 staff at its peak. This represents a tenfold increase in a little more than a decade. In FY 1977, total federal expenditures on poverty research approached $90 million (National Research Council, 1979). One of the most consequential advances in scientific methods, the randomized controlled trials, emerged more slowly in social policy than it did in other fields, such as medicine. Yet during the 1980s and 1990s, the number of experiments conducted on U.S. welfare and employment programs grew rapidly. Perhaps the most important outcome of this experimentation was the growing belief that science had a role to play in social policy (Baron, 2018). In his review of the use of research in welfare reform, Danziger (2001) concludes that the Family Support Act of 1988 is the only time in a half century that policy reflected the findings of social science research. Another instructive example is the passage of the seminal 1996 Personal Responsibility and Work Opportunity Reconciliation Act, widely considered the most significant welfare reform law since the Social Security Act of 1935. Huge investments were made in research and evaluation with the philanthropic community alone investing many millions in welfare-related research during this period. Yet numerous insiders felt that the ultimate passage of welfare reform was driven mostly by personal perceptions, value preferences, and political calculus (Haskins, 2006). O’Connor (2002) reached a similar conclusion, characterizing its passage as “a triumph of politics over scientific knowledge” and, as such, “a devastating blow to the poor” (p. 286). In the end, the law that emerged was a political and partisan compromise between congressional Republicans and President Bill Clinton that transformed cash assistance for poor families. Science played a rather small and disappointing role in this prolonged food fight, which ultimately was determined by policymakers who had the political muscle to push their vision of reform. In the memoir of his policy career, Tom (Corbett, 2018c) describes how research may have been deleterious to Clinton’s efforts to reform welfare, at least as the president envisioned reform. The process of planning his vision in detail was stalled, partly because every effort was made to accommodate the vast amount of research and program evaluations that were being released daily. Unfortunately, the research did not always lend itself to crisp, unambiguous answers to a host of questions, which further delayed the finalization of the reform proposal. When the president’s proposal was released in early 30

T he H istory of E vidence - B ased P olicy

summer of 1994, there was not enough time to vet the complicated bill before the congressional elections of that year. The Democrats lost control of the House, along with any opportunity to pass their vision of reform. The halcyon days when science would shape policy, where ideology would no longer dictate public discourse, and where partisan identity would be muted, were short-lived if, in fact, they existed at all. Faith in a proactive government buttressed by a sophisticated research and analytical apparatus increasingly was considered misplaced at best (Hayek, 1989) and perverse at worst (Murray, 1984). The abrasive influence of ideology and partisanship did not diminish. If anything, their roles increased to the dismay of those seeking an evidence-based policy process. Not surprisingly, federal support for research and development declined as a proportion of GDP until it leveled off at an investment rate similar to what existed in the early 1950s (Corbett, 2018a).

Fast-Forwarding to the Future—a Fading Dream Dismissing past failures, when President Barack Obama took office in 2009, he and his administration made another push for evidence-based policymaking across a host of policy arenas. Between 2009 and 2011, six “tiered evidence” programs were passed requiring rigorous evidence to qualify for federal funding in areas such as K–12 education, early childhood, home visiting, and teenage pregnancy prevention (Baron, 2018; Haskins & Margolis, 2015). This push seemed more aggressive than previous manifestations. Ron Haskins, a longtime Republican staffer in Congress where he served as a scholar of poverty and family issues, characterized the new groundswell of support for applying science to policymaking as “revolutionary” (Haskins & Margolis, 2015). Jon Baron, who founded the nonprofit Coalition for Evidence-Based Policy, asserted that it “marked a major milestone in the history of evidence-based policy” (2018, p. 47). Of course, debates about specifics remained and as political pundits often note, “the devil is in the details.” Then in January 2019, President Donald Trump signed the Foundations for EvidenceBased Policymaking Act, which pushed federal agencies to develop evidence-based policy complete with evaluation plans and designated staff to implement them (Gamoran, 2019). The foundational tenet that rigor and research ought to play a larger role in policy debates appeared to be on the ascendency. Unfortunately, it is easy to be seduced by supportive intentions within our own bubble. The positive words and actions that emerged a decade ago about a renewed age of science was not to endure. Again, as had happened often in the past, a stubborn dose of reality seemed to dash expectations. The polarization of Congress had never been clearer (Haidt & Hetherington, 2012; Shor & McCarty, 2011). A number of factors helped balkanize the political landscape—the elimination of the Fairness Doctrine, huge amounts of money being infused into politics after the Citizens United ruling, and the creation of cable TV outlets that focused on singular messages and narrow target audiences. The emergence of a social networking world permitted members of isolated tribes to speak mostly to one another, aiding and abetting a siloed mentality organized around narrow prejudices (Mayer, 2017). At the same time, the worlds of policy and science, in fact, would become ever more complex with a host of think tanks and evaluation firms joining university-based researchers in supplying the policy world with evidence. Trade groups, lobbyists, and advocacy organizations would grow in numbers and power to compete for the attention of those who make society’s rules. To cite one example, the American Legislative 31

P roblem , P layers , and P olicy P rocess

Exchange Council (ALEC) had been created in 1973 by a small band of conservative activists. Though touted as nonpartisan, it clearly is an agenda-driven organization, which for years had labored in obscurity. In the past couple of decades, however, the entity had emerged as a leading source of state-level legislation. Its model “Bills” were introduced across the country in state legislatures often with little adaptation to local circumstances (Jackman, 2013; Mayer, 2017). Observers felt as if they had traveled back in time to the late 1800s when corporate lawyers drafted legislation that would be enacted by friendly elected officials (Stark, 1995; Mayer, 2017). More recently, science itself has come under siege. In December of 2019, a New York Times article summarized what they called the Trump administration’s “war on science” (Plumer & Davenport, 2019). Though the piece nominally centered on climate issues, the charges were broad. Researchers, along with the many practitioners of rigorous analytics, were being silenced, fired, or merely ignored across the federal government. Scientific advisory panels had been disbanded and counsels of topical experts no longer summoned. This new anti-science sentiment seemed more severe and uncompromising than in previous retrenchments. Science and the role of evidence took another battering during the debate about how to respond to the COVID-19 pandemic (MacKenzie, 2020). The initial optimism of researchers merely a decade ago proved remarkably myopic in the end. What had transpired was a continuing drift in the currency of policy discourse away from evidence and science. On the policy side of things, the political arena evolved in the direction of ideological purity and partisanship. On the science side of things, the academy’s norms of scholarly distinction seldom involved “tech transfer”—applying the findings of science to policy and practice (Gamoran, 2020). This drift away from science has led to a rupturing of the connection between those who generate research (mostly but not exclusively research types) and those who consume research (mostly but not exclusively policymakers and administrators). The cultures that dominated these two worlds were to become more distinct, increasingly separated by perspective and language and even purpose. The corrosive influence of cultural disconnects run deep and were running deeper with each passing year.

Conclusion The early promise that intractable, or at least persistent, social challenges might be resolved by science never was achieved to the satisfaction of those laboring to inform policymaking. Most of us starting our careers in the early 1970s during the golden age of the study of research utilization would have predicted a bright future for the role of research in the doing of public policy (Weiss, 1983; Caplan, 1979; Lindblom & Cohen, 1979). That future would be guided less by passion and prejudice, and more by evidence and analytics. How could it be otherwise? Research methods were getting better, data were being collected and stored in accessible digitalized forms at ever-increasing rates, and newly minted PhDs spilled out of our universities armed with evermore sophisticated tools. Reality, however, never seemed to match the hopes of those advocating for science in the public arena. Fortunately, old dreams never die. They wait for new generations to appear and then are revived. As the third decade of the 21st century dawned, the role of science in America’s halls of power seemed as dim as it had ever been (Gamoran, 2019; Mann & Ornstein, 2016; Peterson, 2018). But a question remains: Are these really “dark times for science and public policy”, as Gamoran claimed in a recent article in Science (2019, p. 843)? What we see above the 32

T he H istory of E vidence - B ased P olicy

water line, in the headlines that dominate the 24-hour news cycle, may obscure more fundamental activities and crosscurrents happening out of sight, just below the surface. This often has been the case (Boaz et al., 2019b). Day-to-day we see what occurs on the national stage and what garners the media headlines like the budget reductions at the Centers for Disease Control and Prevention or the firing of scientific experts. But policymaking is a much broader net than that, with many more actors. Sometimes, we must look a bit harder, deeper, to see what is really going on. Throughout this book, we present data drawn from policymakers at the state level. Their responses to interview questions posed by Karen and her colleagues suggest a quite different reality from what we observe. Research is still both valued and desired. Yet research is not uniformly injected into policymaking across peoples and places, as famously proposed long ago by the “hypodermic model” (Pettigrew, 1985). Rather research appears to be used by a cross section of policymakers, on a slice of issues, and at particular points of the policy process (Bogenschneider & Bogenschneider, 2020). What we need, it would appear, is a more nuanced appreciation of policymakers and the policy process, as well as more effective strategies for bringing science from those that produce it to those who want and need it. We continue to hold onto the dream of an evidence-based policy world because we have seen glimpses of how policymaking has worked as it could and should. We hold dear the purpose of our dream of evidence-based policy: Liberal democracy matters, not as a means to something else, to any particular set of values, but as an end. It is for this reason that a liberal democratic politics is also a romantic politics, a politics of dreamers who decline to be bound by the dreams of previous generations. . . . At the end of the day, all that matters is that those with the power to do so resolve to reduce the suffering of those without. (Ward, 2008, p. 302) We continue to work toward our dream of evidence-based policy in the hope expressed by Theodore Roosevelt well over a century ago—that our level of greatness will rise to the level of our opportunity (1893).

33

3 UNDERSTANDING POLICYMAKERS Insights From Science Karen Bogenschneider and Thomas J. Corbett

This chapter provides “the basics” for those interested in studying or engaging policymakers. Because policymakers are easy to stereotype, the reader is introduced to what they are really like as described by those who study them and work with them. Effective research and practice depend on understanding the inhabitants of a world in which researchers do not live. With a deeper understanding comes the appreciation that not everyone who operates in the policy world is alike. Some policymakers are more effective than others, and some become a champion for an issue they care deeply about. Some could care less about research evidence and some are deeply committed to its use. Classic studies reveal the unique legislative styles that policymakers bring to their trade. What ordinary men are directly aware of . . . are bounded by the private orbits in which they live: people have direct awareness only of the world in which they operate. C.W. Mills (1959, p. 3) I doubt if it would be possible to overestimate the tremendous burden felt today by persons in public office when they are conscientious and sincere. So many perils to be averted, so many different historical exigencies to be planned for, and so far in advance, and so many forces at work against concerted and wellmeaning human efforts to ensure the triumph of the good! John Hall (1998, p. 95) Throughout this book, we explore differences between the research and policy cultures. Oddly enough, there are a couple respects in which these two communities are strikingly similar. First, almost as predictable as night follows day, each community will resort to stereotyping whatever they do not understand about the inhabitants or inner workings of an unknown culture. For example, who has not heard policymakers stereotyped as opportunistic, self-serving egotists? Who among us is not aware of instances where researchers are referred to as eggheads, nerds, and absent-minded professors? What academic wouldn’t bristle at being referred to in the way that Karen’s young son used to introduce her to his friends: “My mom’s a doctor, but not the kind that does you any good at all!” Policymakers and researchers also share the dubious distinction of being easy to stereotype because the public knows so little about what either of them are really like or what it takes to be effective in their jobs. Policymakers become known to the public 34

U nderstanding P olicymakers

through nonstop vilification by a full-time opposition poised to pounce on any mistake or error in judgment, and by a press constantly on the lookout for attention-grabbing headlines (Levin, 2005). Researchers also remain an unknown species ensconced as they are behind the walls of the ivory tower where they engage in highly technical work that is hidden from public view and that can take years before anything is ready for public consumption. If these stereotypes remain unchallenged, they can morph into misperceptions that can strain attempts at communication and collaboration across these conflicting cultures. A key element of our community dissonance theory is understanding the inhabitants of orbits in which we do not live and a world in which we do not operate (Mills, 1959). Almost instinctively, inhabitants filter their perceptions of other worlds through the lens of the orbit in which they live and operate, obscuring how deep and divisive the cultural differences can be, oftentimes unnecessarily so (Kendall-Taylor & Levitt, 2017). In this chapter, we delve into the mysterious orbit of the policymaker. Of course, the world of the researcher also deserves dedicated attention to break down the stereotypes that go both ways across the research-policy divide. However, for a book directed mostly to researchers, we turn our attention to policymakers in keeping with one of the main premises of our theory. If researchers better understood that alien species who inhabit the policy world, this might be a step forward in fostering better connections and communications. Why do our perceptions of policymakers matter? Vice President of the William T. Grant Foundation Vivian Tseng may have put it best: “You can’t understand the use of research evidence in decision-making if you don’t understand the would-be-users of research evidence and the context of their decision-making” (2019b). The importance of one’s perceptions of policymakers became abundantly clear during one of Karen’s trainings of new state directors of the Family Impact Seminars. After the opening discussion of how research can influence policymaking, a question was posed by a director with a degree in public affairs from an Ivy League school. She blurted out that she could envision how research evidence might be used in states like Massachusetts, Oregon, and Wisconsin, but the policymakers in her state were just not interested in research. Why should she even bother to spend her time doing something so hard with such uncertain outcomes? She immediately assumed disinterest absent any real evidence. The temptation to paint all policymakers with a broad brush is hard to resist. The stereotypes that policymakers and researchers hold of each other permeate a plethora of settings where policies are made. For example, Tseng and colleagues describe a similar experience in educational settings where mutual misperceptions can quickly sabotage cross-cultural communication. School district administrators and classroom teachers often feel “like second-class citizens, spoken down to by research experts”. At the same time, well-meaning researchers feel “frustrated that they are distrusted”. Absent respect for what each brings to the table, “the parties are at impasse and forward movement is all but impossible” (Tseng et al., 2018, p. 7). To break through prevailing stereotypes of what policymakers are like, we turn to science, drawing on classic and cutting-edge studies. We begin with our recent study of state legislators and how they describe what it takes to be an effective policymaker (see a description of our methods in the Appendix). Policymakers describe a job in which effectiveness entails more than making laws and where success depends on a requisite set of knowledge, attitudes, and skills. In addition, we review three studies, two qualitative and one quantitative, that provide fresh insights by looking beyond the generic label of “policymaker” to a more granular view of differences within the policy community. One common theme across these studies is that not all policymakers are alike. 35

P roblem , P layers , and P olicy P rocess

We also draw upon the observations of researchers who are one step removed from the policy orbit, but whose interactions with policymakers provide a valuable window into their motivations and dispositions. Researchers’ initial impressions of policymakers were stereotypically negative, but changed in several respects after they had logged in more time working with them. We wrap up this journey by discussing how a deeper and more discerning understanding of the inhabitants of the policy world can inform our efforts to be more policy-relevant researchers and better brokers of research evidence.

Understanding Effective Policymakers On-site studies of policymakers are rare (Fox, 2010; Mayhew, 2011; Merriman, 2019), so it is not surprising that little is known about what “policymakers actually do when they are doing their job” (Freeman, Griggs, & Boaz, 2011, p. 128). To address this shortcoming, Karen and her colleagues walked right inside the statehouse in two Midwestern states and interviewed 212 state legislators, enjoying exceptionally high response rates (see the description of our methods in the Appendix). Granted, this is only one window into the policy community. Yet speaking directly with policymakers seemed like a good place to start to expand our understanding of a population that has been largely overlooked in scientific inquiry. One of the main lines of questioning in our study dealt with how policymakers would define an effective state legislator. Legislators endorsed the fundamental importance of understanding the job of a policymaker, explaining that it is “different than any other line of work”. Legislators explained that unfamiliarity with the parameters of the lawmaking profession makes it “tough for people outside to always get a full grasp on why we do some of the things we do” (Bogenschneider, Corbett et al., 2019, p. 134). We agree that without this insider’s knowledge, it is hard to communicate with policymakers and be heard by them. One clear message from our legislators is how difficult it is to come up with a single definition of effectiveness because there are differences in what it takes to do their job inside and outside the statehouse (Day, Bogenschneider, & Parrott, 2019). If constituents were asked, their definition of effectiveness might differ from that of colleagues. One legislator put it this way: The constituents only really get a snippet of what you do—whatever appears in the media or Facebook. Your colleagues are inside the baseball stadium. So, they see the pitches as they’re thrown and swung at. The constituency sees much more of the strikeouts and the home runs. The results reviewed here are based on interviews of state legislators, so it privileges the views of colleagues who are inside the legislative bubble versus the views of constituents. We conducted line-by-line analysis of all 212 interviews and coded any mention of legislator effectiveness. Of the resulting 306 coding segments, the four most frequent themes revealed different dimensions of what it means to be an effective legislator (Day, Bogenschneider et al., 2019). Effectiveness was defined as what legislators do—work effectively on legislation and respond to constituent needs—but also how they are effective in these pursuits. Effectiveness encompassed the ability to build trusting relationships and garner a reputation for being committed to a larger purpose and the greater good. A snapshot of what we heard on each element of effectiveness is overviewed next.

36

U nderstanding P olicymakers

Key Concepts 3.1.  How Legislators Themselves Describe What It Means to Be Effective • • • •

Works effectively on legislation Responds to constituent needs Builds trusting relationships Demonstrates commitment to a larger purpose and the greater good

Works Effectively on Legislation The most frequent definition of effectiveness was passing good legislation, but also stopping bad legislation. One Republican prided himself on being “more of a stopper than a starter”. Yet if the standard for effectiveness is a simple dichotomy of either passing or stopping legislation, one legislator explained that you will fail much more than you succeed. For her, success included all the interim steps involved in advancing legislation toward a longer-term goal. “Is it success if you move the needle? I think absolutely it is.” In a similar vein, another legislator used a sport analogy to define effectiveness. Success is measured, not just by throwing the “Hail Mary”, but also by moving the ball down the field and getting the first downs. Legislators also cautioned about defining effectiveness in the easiest ways possible, such as looking at who authored a bill. This obscures those legislators who manage to get language into legislation that accomplishes their objectives, without ever having their name appear in the record. For example, legislators who were members of the minority party were described as effective if they were able to shape (not necessarily pass) legislation, and if they were able to take more incremental approaches to advancing legislation. One Democrat described what effectiveness means for minority members by using a bus analogy he heard once from former Indiana Governor and Senator Evan Bayh. He explained that when you’re in the majority, it’s like you’ve got your hands on the wheel of the bus and the seats behind the driver are screaming to turn this way or do it that way: Well, now I’m in the minority of a house of 99, so I’m way back in the bus. . . . The people that are most effective  .  .  . don’t sit in the back of the bus and scream. . . . They’re the ones who work their way up the bus, work well with others, and are thoughtful in the way they approach ideas and potential bills. Legislators in the majority party, however, are held to a higher standard, which was stipulated to include making big policy changes and working with the minority. In a state legislature in which Republicans are the majority party, a Democrat described what it takes for her to consider a Republican effective: In a situation like we’re in in Indiana with super-majorities, any idiot can get a stupid bill heard. Just because you get it heard and passed, it doesn’t mean that I  think you’re effective. So, an effective legislator from the majority is a person who can sort of act in a more bipartisan manner, even when they aren’t forced to.

37

P roblem , P layers , and P olicy P rocess

Even passing a lot of bills was no guarantee of effectiveness in the eyes of most legislators, but not all. In the interviews, some boasted of the number of bills they had passed in total or in comparison to their colleagues. For most legislators, however, effectiveness depended, not on the number of bills, but rather on how meaningful they were, particularly “in making this state better for everyone who lives here”: I want to know how much you moved policy. How much did you make a difference?  .  .  . Does the difference matter? Does it change quality of life? Does it improve health? . . . Not just does it change anything; does it improve anything?  .  .  . Does it start an engine? Does it build the fire? Does it put out a fire? . . . That’s why we’re here. We’re here to move things that benefit people, and not just fill a seat. Responds to Constituent Needs Another central element of being an effective legislator was responding to the needs of constituents and communicating their interests into the legislative process. It is not surprising in a representative democracy that constituents were central to how policymakers do their job, because they are integrally tied to whether policymakers keep their job (Fenno, 1978; Mayhew, 2011). The importance of constituents was a common refrain in the interviews, exemplified by these words: “A good legislator talks all the time about their constituents. . . . They love the people they get to represent.” Legislators explained how politics is different from other professions in a couple ways. First, legislators have two core sets of stakeholders—their constituents and their colleagues. Many professions have multiple stakeholders, but seldom two sets that hold such legitimate access to their time and attention, or impose such a significant influence on how effective they are in their jobs. For instance, one state legislator described the oversized importance of responding to constituents on a timely basis. He illustrated his point with one of the most effective policymakers he had ever known, who had been reelected to his seat in Congress several times. Yet, if there were nominations for policymaker of the year, this congressman would seem an unlikely candidate: He has this kind of gruff, almost unpolished personality. . . . You almost wince sometimes [because] he doesn’t even realize that he’s being gruff. . . . He’s just kind of awkward. . . . He’s not going to smile and nod and make you feel comfortable. The congressman was reliably reelected, according to this state legislator, because he made it a high priority to get back in touch with his constituents on a timely basis. When someone contacted his office, they would get a response in three days—just like clockwork. People appreciated his prompt response so much that they reelected him, even though they didn’t like him very much. This legislator explained to Karen that understanding effectiveness depends on appreciating how politics is like a whole different industry: “In how many industries could you be effective, just because you are prompt in your response to people?” Another way that policymaking differs from other professions is that effectiveness depends on more than figuring out people’s needs. To be effective, a policymaker must find that “sweet spot” of what are important needs in the eyes of one’s constituents and also make sense to one’s colleagues in the capital. Some policymakers intentionally make both stakeholders a priority and others prioritized one over the other. One legislator 38

U nderstanding P olicymakers

explained that he leaned more toward serving the needs of the constituents, who had elected him to office: The number one thing for me is . . . the wave of calls that come in every day, emails from people and constituents that need help. . . . My home phone number is on my business card. . . . I’ll answer the call . . . and I’ll say, “What can I help you with?” . . . It’s always the same response, “Well, I didn’t think I would get you.” And their voice always changes. The anxiety in that call changes. . . . I can tell that it really means a lot to people that they actually got the guy . . . they didn’t think they’d ever get. . . . It’s just nice to be in a position to open some doors for folks. . . . That’s what I think makes a legislator. . . . When you run for an office . . . you have to understand that wherever you may have imagined yourself to be in the social line-up, you now have dropped to the very bottom, because you’re the servant. You are the employee—that’s where you’re at. . . . You answer to about six and a half million employers. I understand the concept. That’s what the Founders had in mind when they set this thing up. . . . At the gas station, the guy at the other side of the pump will say, “Hey you’re Senator [name], aren’t you? Thank you for what you did.” There’s a reward to that. It might sound selfish, but it does make you feel nice when someone says, “I thank you for what you did for my family”. Because it makes me feel like I’m doing what I’m supposed to be doing. Builds Trusting Relationships Another characteristic of effective legislators is building relationships with colleagues who come to know you as being trustworthy, factual, and knowledgeable. Relationships are not an end in the policymaking process, but a means to those ends that policymakers are trying to achieve, such as influencing legislation: I think an effective legislator is somebody that people trust their judgment, not what [bills] they pass. . . . When they talk, do people listen and trust their judgment? You may never pass a bill, but . . . you have a lot of say in what does and what doesn’t [pass]. Legislators explained that it is hard to be effective without being able to build trusting relationships in a culture than runs on relationships. One legislator explained, “I know some ‘Lone Rangers’ in this building, and you can’t get anything done.” Another legislator described what happens when their colleague’s behavior runs counter to the dominant culture: “They came in and said, ‘This is my way or the highway.’ They’re going to have a rough go of it because, ‘Well, I’m sorry, Bucko, but that’s not how we operate here.’ ” Another legislator explained that consensus-building is critical to effectiveness, because anytime that you get 100 legislators together, you are guaranteed to get 100 different opinions. Working through these inevitable differences depends on being a colleague who is trusted and does not come across as being abusive or abrasive or cynical. Demonstrates Commitment to a Larger Purpose and the Greater Good Enacting legislation or working effectively with constituents or colleagues depends to a large extent on a policymaker’s underlying motivation. Legislators mentioned frequently 39

P roblem , P layers , and P olicy P rocess

in interviews that they always have their antennae up to see if their colleagues are making the “right choice for the right reasons”. The right reasons were when colleagues were able to step back and see the big picture—the effect on the entire state, not just one’s local constituency, and not just one’s current constituents, but also those in the future. The right reasons were taking tough votes that went against the pressure of party politics from one’s caucus or leadership, knowing full well that “you’re going to get beat up for it”. One Democrat described how her commitment to a greater good compelled her to act against her personal self-interest: There’s a number of initiatives. . . . I’m doing research and I’m working hard on. But if there’s someone . . . [in the majority] that cares as much as I do about this, and I need to bow out because me being involved is going to actually make it hard to get across the finish line, I’ll bow out. Because it’s not about me putting my name first on the bill. It’s not about me getting the press . . . and being in the spotlight. Ultimately, I’m here because I want to be working in the best interest of the people in my state. I love my community. I love my people.

Implications for Researchers and Knowledge Brokers These research findings provide a window into the policy orbit, which may be the next best thing to sitting across the desk from dozens of policymakers yourself. Getting a firsthand view unveils what it takes to be an effective legislator and, in so doing, calls into question prevailing stereotypes that policymakers are opportunistic, self-serving egotists. This analysis also reveals how policymakers might use research to pass or stop a bill or help advance a bill along the way. Effective policymakers use research to push back against prevailing party politics and to help them bring into view the bigger picture of how policy can best serve their constituents and the state. In addition, policymakers use research to connect with constituents and respond to the questions they ask and the issues they raise. Our analysis raises the intriguing prospects that bringing research to policymakers might take two routes. Research could be communicated to policymakers at the state capitol, either to the lawmaker directly or their key staff. Some legislators will leverage this input and become known as a factual and knowledgeable information source who can be a trusted arbiter for building consensus across conflicting views (Bogenschneider, Corbett et al., 2019). Getting research into the hands of a lawmaker who is trusted by colleagues will undoubtedly multiply its influence. Another plausible route for bringing research to policymakers is indirect. Research can be communicated to constituents in their districts, who have the ear of policymakers and are a potent influence on their policy decisions. When policymakers pledge to their strongest supporters back home that they will take a certain action, this is the “most durable kind of legislative commitment” (Fenno, 1978, p. 231). This analysis gives us a generic view of how policymakers operate, but it glosses over the more granular perspective of how individual policymakers differ in their motivations and predispositions—an important topic we turn to next.

Understanding Effective Champions of Policy Issues One difficulty in debunking stereotypes of policymakers is that they sometimes contain a kernel of truth. Without personal experience in the policy community, it is easy to fixate 40

U nderstanding P olicymakers

on a negative stereotype and deploy that to malign all policymakers. Take, for instance, Mayhew (2011) who described members of Congress as “nasty”, “mediocre”, “pork barreling” “lapdogs”. According to researchers who study cognitive bias, reality is not so “tidy, simple, predictable, and coherent” (Kahneman, 2011, p. 204). Karen learned firsthand in her study of state legislators that the real world is much more complex and much less predictable than stereotypes allow. Policymakers do not fit so neatly into the simple tidy box that we create in our minds. The shortcomings of simplistic stereotypes were readily apparent to Karen in her interviews of 212 legislators. She found that legislators were a diverse species—women and men, young and old, newbies and long timers, and legislators who self-identified as black, white, and Hispanic. Karen interviewed legislators from all walks of life such as lawyers and accountants, doctors and veterinarians, pastors and police officers. Several owned businesses ranging from restaurants to bowling alleys to farm supply stores. Some worked in low-end jobs as food service workers and others had worked in executive positions in Fortune 500 companies. The differences in backgrounds were sometimes palpable even before the interview began. In some offices, every available space was plastered with sports memorabilia, whereas others were lined wall-to-wall with books. The bookshelves in one office were stacked with neat rows of the state statutes punctuated with colored sticky notes for easy reference. The differences in dispositions were also obvious. One legislator explained that she was very introverted and had to force herself to speak up, whereas another legislator was so loquacious that he apologized at the end of the interview for taking up too much of Karen’s time. The differences in dogma were sometimes stark. One senator explained that everyone in his caucus believed no one chose to be on welfare, whereas another senator told a different story. He recounted a survey that he had commissioned in which over 75 percent of welfare recipients in his state said that they could have gotten a job, but receiving benefits was easier. A representative, who grew up in poverty, had a different take. When asked about the most frustrating part of championing youth and family issues, her immediate response was “losing”. She became emotional and explained through her tears that “there’s people out there that are really hurting”. Policymakers are an understudied population (Fox, 2010; Mayhew, 2011; Merriman, 2019) and, when they are studied, they tend to be grouped together as a whole rather than split out into subgroups (Oliver, Innvaer, Lorenc, Woodman, & Thomas, 2014). This has contributed to a scientific understanding of policymakers and their research use that is “unduly simplistic” (Weiss & Bucuvalas, 1980, p. 248). One study that moves beyond some of these simplistic assumptions that all policymakers are alike is this same study of state legislators described earlier. In this analysis, we examined how legislators, who were nominated by their colleagues as exemplar Champions of Youth and Family Issues, differed from other legislators in the study. Champions, who are conventionally referred to as policy entrepreneurs, are important players in shaping policy discourse on a specific set of issues (Mintrom, 2000). Without a Champion to advance an idea through the policy process, it can get lost in the avalanche of issues that compete for policymakers’ attention (Goodvin & Lee, 2017). How Champions of Youth and Family Issues Differed From Their Colleagues Here we profile 24 state legislators, who were nominated by colleagues as being exemplar Champions of Youth and Family Issues. We identify how they differed from their 41

P roblem , P layers , and P olicy P rocess

colleagues by comparing them to other legislators serving in the same lawmaking bodies (86 percent response rate). Drawing on all 212 legislator interviews, we coded any skill or attribute that a state legislator mentioned as contributing to effectiveness (see Bogenschneider et al., 2020). Of the 69 that organically emerged from the data, seven were mentioned by at least 50 percent of the Youth and Family Champions and were at least 20 percentage points higher than mentions by other legislators in the study: one attitude (commitment to a higher purpose or greater good), four skills (earns trust, listens, builds relationships with colleagues, builds relationships with external stakeholders), and two types of knowledge (knows how to maneuver the political process, knows policy issues). These findings were corroborated with 13 high-level, experienced key informants—­ former governors and gubernatorial staff, legislative analysts, lobbyists, and former legislators with an average of 31 years of state policy experience.

Key Concepts 3.2.  How Champions of Youth and Family Issues Differed From Their Colleagues Compared to interviews of other legislators, Youth and Family Champions made more mentions of: • • • • • • •

Being committed to a higher purpose or greater good Earning the trust of colleagues Listening to people including colleagues, constituents, and the public Building relationships with colleagues Building relationships with external stakeholders Being knowledgeable about how to maneuver the political process, which entails mastering lawmaking rules, procedures, and norms Being knowledgeable about policy issues

Commitment to a Higher Purpose or a Greater Good Of the Champions, 83 percent discussed commitment to a higher purpose or a greater good such as benefits to constituents or to their state, an attribute that was mentioned by only one-third of the other legislators. The Champions mentioned commitment to a people or a cause, such as a racial/ethnic group or the disadvantaged. A Democrat was “always trying to be the messenger for . . . the voiceless”. Another Democrat expressed a deep commitment to children, explaining her job as a mother is to care about her own child, but her “job as a representative is to care about your child”: I could send [names daughter] to the best schools. I  could move to the best neighborhoods, do everything picture perfect for her but she still has to, at some point, go out into the world and function with the same people who may not have had a good upbringing, who may not know right from wrong. It wasn’t then enough for me to be concerned about her. I needed to be concerned about my neighbors’ kids and the kids 20 blocks from me.

42

U nderstanding P olicymakers

Relationship Skills Youth and Family Champions discussed four relationship skills that differentiated them from their colleagues. First, 83 percent of Champions mentioned earning the trust of colleagues, compared to only 50 percent of other legislators. A Democrat elaborated on the importance of trust in the policy culture: “I don’t think you can get much done if there isn’t trust and respect”, especially in this building where “trust is thin. . . . If [colleagues] don’t trust you, you don’t get anything done—nothing, zero, zip! You are an outcast.” A second relationship skill, listening to people including colleagues, constituents, and the public, was mentioned over twice as often by Youth and Family Champions (58 percent) as by other legislators (28 percent). One Champion described how she listens and to whom: “I try to be more attentive in my listening with my colleagues that are across the aisle.” Another emphasized why listening is important: I always try to be extremely open-minded, really on any subject, because . . . listening to the other side in depth and asking questions can also make an argument stronger for what I believe in, or it may give me new information to change my, shift my perspective. A third skill, building relationships with colleagues, was mentioned by 71  percent of Champions compared to 50 percent of other legislators. A Democrat explained the political value of relationships with colleagues: “Without a relationship, there’s no way to find common ground. A relationship has to be there first.” These relationships may be established in committee assignments with other colleagues interested in the issue; of the 24 Champions, 89 percent were positioned on committees in the legislature where youth and family issues were discussed, which included children, youth, and families; education, colleges, and universities; corrections, criminal justice, and judiciary; health and mental health; and human services. A fourth skill, building relationships with external stakeholders, was mentioned by 67  percent of Champions compared to 20  percent of other legislators. Champions connected with constituents at community events such as funerals, graduations, or listening sessions. They brought together allies by building coalitions and convening summits. They invited the involvement of stakeholders, such as lobbyists and representatives of special interest groups, to vet drafts of bills and testify at hearings. Knowledge of Policy Issues and Processes The Youth and Family Champions frequently mentioned two kinds of knowledge important to effectiveness, which a key informant claimed moved a step beyond a good relationship: “It’s knowing how to activate it . . . or to use it.” First, one-half of Champions compared to 29 percent of the other legislators discussed the importance of knowing how to maneuver the political process, which entailed mastering lawmaking rules, procedures, and norms. Champions touched on political tactics like moving an issue to a supportive committee, threatening to alert the media to politically explosive issues, gaining supporters for your bill by agreeing to back a colleague’s bill, seeking buy-in of committee chairs, etc. A Democrat emphasized that for

43

P roblem , P layers , and P olicy P rocess

the minority party to be effective, it is paramount to actively engage with colleagues across the aisle: In this climate, if my name is first on a bill, it’s pretty much not going to go anywhere. . . . Figuring out how it is that we can work with each other to bring different members of our parties along and who is going to be talking to who. . . . It’s almost like playing a chess game, I hate to say—being pretty pragmatic, but also really authentic. Knowing policy issues (mentioned by 58 percent of Champions and 30 percent of others) meant becoming a “student” of the policies in your portfolio. A Democrat noted: “A  lot of the process of legislating is the journey that it takes to educate yourself on issues.” Implications for Researchers and Knowledge Brokers These results suggest how misleading it would be to conceptualize policymakers as a homogenous group because they are elected to the same job and work in the same setting (Levin, 2005). This study of effective legislators provides empirical evidence that not all policymakers are the same. Legislators who championed youth and family issues through the legislative process differed from their colleagues serving in the same body in several ways. For example, contrary to the negative stereotypes held by some, the Champions in our study were committed to purposes larger than themselves, took the time to learn about policy issues and processes, and were conscientious about listening and earning the trust of others inside and outside the statehouse. This suggests that researchers may want to hit the “pause” button before making blanket statements about policymakers regarding the attitudes they hold, the knowledge they acquire, and the skills they possess. Our data suggest considerable variability on all these dimensions within the policymaker population. When one embraces negative views, it is easy to find examples, even though they may be, in fact, outliers. Granted, these findings have several limitations that are fully detailed in a recent publication (see Bogenschneider et al., 2020). For example, it is unclear whether the seven skills and attributes that distinguished our Youth and Family Champions from their colleagues would extend to champions of other issues and policymakers in other states or those serving in local or national settings. The findings do inspire some confidence, however, given the high response rate in our study, their corroboration from key informants, and their consistency with the characteristics of policy entrepreneurs in previous studies (Kingdon, 2011; Mintrom, 2000; Nelson, 1984). In addition, the results resonate with evidence about how individuals can trigger a critical mass of support, which depends on both qualities of the individual and the network in which they operate (Watts, 2011). Compared to colleagues in the legislature, being recognized as a Champion depended on individual attitudes (commitment to a higher purpose), but also on network positioning (committee membership, relationships with external stakeholders), skills for operating in the network (earns trust, listens, builds relationships with colleagues), and network knowledge (knows political maneuvers, knows policy issues). The findings suggest that researchers could generate more policy-relevant questions and knowledge brokers could increase the uptake of research evidence by personally partnering with policymakers who champion issues related to their topics of interest. 44

U nderstanding P olicymakers

Researchers can locate Champions by looking to the committees that these topics pass through, or by partnering with intermediary organizations that have knowledge of and relationships with policymakers. The findings suggest that another best practice for advancing policy interests is to follow the example of these Champions by partnering with external stakeholders involved in the issue.

Understanding Legislators Who Use Research and Those Who Don’t From our recent trek inside the statehouses in two Midwestern states, we learned that policymakers were not all the same when it comes to their research utilization. In our interviews of 212 legislators, we found that research use ran the gamut with some policymakers who cared little about using research evidence, some who were seriously committed to its use, and others who fell between these two extremes. At one extreme was a legislator, who was so animated about the value of research that she spoke with Karen for over two hours. One theme of her interview was how hard it was to access nonpartisan, agenda-free research. As evidence for her claim, she plopped on her desk a big red crate that was filled to overflowing with brochures, pamphlets, and reports that she said could only be described as “partisan propaganda”. Another legislator enthusiastically greeted Karen and immediately pushed a pen in her hand with a request to autograph Karen’s family policy textbook that the legislator had purchased in preparation for the interview. Another thoughtful legislator was so intrigued with the interview questions and so taken with the interaction around them that he commented, in all sincerity, that he and Karen should write a book together. However, not every legislator was so enthusiastic about spending time with a professor. One legislator as he was escorting Karen to his office, yelled over his shoulder to his staff person, “Just how long do I have to meet with her?” Another balked at the questions on research use, explaining that he does not use a lot of academic research because the legislature is just not an academic place. Yet two interviews stand out as the bookends of legislators’ research use. Both interviews were with long-standing Republican legislators who were well respected in their states. Karen had made repeated attempts to schedule one of these interviews but received no acknowledgment of her requests. To her great surprise, the legislator finally returned her call and agreed to meet in a busy coffee shop several miles from the state capitol. Karen began, as she did every interview, by reviewing each provision of the informed consent procedure required for any university study. After agreeing to have the interview recorded and with the tape recorders whirring, the legislator paused and asked if he could be blunt. Karen’s emphatic response was “absolutely”, explaining that the study would be valid only if he were brutally frank. After the warm-up questions regarding areas of specialization, Karen turned to a series of questions exploring his use of research on the job. He immediately directed his attention to university research. He looked Karen squarely in the eye, so he could gauge her response to his frank reply. He asked why on earth he ever would use research produced by “liberal”, “leftist”, “eggheads”. Besides, why would a policymaker turn to research when most policy decisions could be made by relying on common sense? He wondered aloud why policymakers would even need research evidence when he could tell you at the beginning of the session how every single member would vote on every major issue that was scheduled to come before the body. Another Republican in the same state agreed to speak with Karen in his office in the state capitol, explaining that he preferred not to go over 50 minutes, facetiously noting 45

P roblem , P layers , and P olicy P rocess

that was the typical duration of most college classes. After the preliminary questions, Karen started the line of inquiry about his use of research. He interrupted and pointed to the credenza behind his desk that was lined with row upon row of manila file folders. He asked Karen if she would like to see the contents of his folders, which were used to store the data and research that he collected on every issue he was currently working on. He explained how he used research to check the accuracy of his own thinking to “make sure that this premise that you built your entire house of cards on is actually accurate”. He went on to explain in detail how he used research to winnow down differences of opinion to move dialogue away from a “philosophical or political” posturing and toward a “problem-solving exercise”. And, for the next 1¾ hours, he enthusiastically described when and how he used research in his job. Given the methods of her study, Karen was unable to determine exactly where each legislator fell on this continuum between the allegorical bookends of the least and most frequent users of research. We do know, however, that the legislator with the researchstuffed credenza received, by far, the most nominations from his colleagues as the most effective legislator in his state. We know from another of our studies that it is possible to quantify individual differences in research use that are hinted at by our qualitative interviews (see Bogenschneider, Little, & Johnson, 2013). In this study, the sampling frame consisted of (a) the entire state legislature (N = 129) in Wisconsin at the time, and (b) a randomly selected subsample (N = 67) of the 212-member New York State Legislature. These two states are similar in that neither has term limits and both are full-time legislatures with a high number of professional staff. They are different, however, in terms of geographic and political character. Wisconsin is a medium-sized, Midwestern, moralistic state with a substantial rural population, whereas New York is a large, eastern, individualistic state with a large urban metropolis (Elazar, 1984). We interviewed 109 state legislators, of which 74 were in Wisconsin (57 percent response rate) and 35 in New York (52 percent response rate). (Full details on the sample and methodology can be found in the Appendix.) A cluster analysis was used to classify policymakers into groups using three variables— their attitudes toward research use, accessing of research, and use of it in policy decisions (see Bogenschneider et al., 2013). These three variables were used in an analysis to identify four clusters that tap into policymakers’ general dispositions toward research use: Cluster 1, Enthusiastic Users; Cluster 2, Skeptical Users; Cluster 3, Enthusiastic Nonusers; and Cluster 4, Skeptical Nonusers. These labels provide a convenient way to discuss the clusters, but with the caveat that policymakers labeled nonusers do access and use research to some extent, though not nearly as extensively as those labeled users. The scores of each cluster on the valuing of research, research access, and research use are displayed in Figure 3.1. Cluster 1—Enthusiastic Users of Research In statistical tests, Enthusiastic Users of Research had significantly higher scores than any other cluster on the valuing of research and research use, and significantly higher scores than Clusters 3 and 4 on the accessing of research (see Figure 3.1). Policymakers in this group were enthusiastic about the value of social science research in policymaking and reported actively accessing research about policy/program evaluations and what other states were doing by reading reports, asking legislative service agency staff, attending seminars/presentations, and calling researchers/experts for advice. They also reported

46

U nderstanding P olicymakers

6 5.5 5 4.5

Cluster 1: Enthusiastic Users

4

Cluster 2: Skeptical Users

3.5

Cluster 3: Enthusiastic Nonusers

3 Cluster 4: Skeptical Nonusers

2.5 2 1.5 1

Valuing of Research

Research Access

Research Use

Figure 3.1 How Policymakers Vary in Their Valuing, Accessing, and Use of Research Note: Cluster analysis based on 109 state legislators, 74 from Wisconsin and 35 from New York

significantly higher levels of use than any other group, as indexed by talking with their colleagues about research and taking the results into account when making decisions or justifying decisions they had made. Overall, 25 respondents or about 1 in 4 legislators (23 percent) were classified as Enthusiastic Users. Cluster 2—Skeptical Users of Research Skeptical Users placed significantly lower value on research than policymakers in either Cluster 1 (Enthusiastic Users) or 3 (Enthusiastic Nonusers), and significantly higher than policymakers in Cluster 4 (Skeptical Nonusers). That is, they were more skeptical when asked about the value of social science research and its benefits to policy decisions. Yet they reported significantly higher levels of research access and use than Clusters 3 and 4. Thus, despite their skepticism about the value of research, these policymakers still were relatively active in seeking out research and using it in policy decisions. About 1 in 4 legislators in the sample (24 percent) fell into this cluster. Cluster 3—Enthusiastic Nonusers of Research Enthusiastic Nonusers had relatively high scores on valuing of research, significantly higher than the skeptics (Cluster 2 and Cluster 4), but significantly lower than the Enthusiastic Users (Cluster 1). Even though Cluster 3 rated the usefulness of research in policy decisions significantly higher than Cluster 2 (Skeptical Users), they still reported lower levels of both research access and use than their legislative counterparts in Cluster 2. Enthusiastic Nonusers also accessed and used significantly less research than Cluster 1 (Enthusiastic Users), but they reported more use and similar levels of access compared to Cluster 4 (Skeptical Nonusers). This cluster was the largest of the four with over onethird (36 percent) of legislators falling into this cluster.

47

P roblem , P layers , and P olicy P rocess

Cluster 4—Skeptical Nonusers of Research Skeptical Nonusers had low scores on all three variables. In statistical tests, these policymakers were the least enthusiastic about the value of research in policymaking and were those significantly less likely to use research. Like Cluster 3, the Skeptical Nonusers were less apt to access research than Clusters 1 or 2, differences that were statistically significant. It may surprise some that these skeptical nonusers of research represented a comparatively small proportion of the legislature overall. About 1 in 6 (16 percent) legislators fell into this cluster. Implications for Researchers and Knowledge Brokers The results provide several lessons for advancing evidence-based policymaking, two that are specified here. First, the findings confirm that there is considerable heterogeneity in how state legislators view the utility of research in policymaking. Some, but not all, policymakers clearly are interested in using research to guide policy decisions and others reject its utility in their legislative role. The findings shed light on why it is easy to stereotype policymaker’s research use. Each prevailing presumption is based on a kernel of truth. Those who question the idea that research is used in policymaking likely focus on the segment of the sample that reports infrequent use of research. In this study, just over half of the sample (53 percent) was composed of those classified as research nonusers (both the Enthusiastic and Skeptical Nonusers of Research) who were well below the sample mean (0.50 SD to 1.25 SD) for research use. To the contrary, those who claim that policymakers are receptive to using research in policymaking probably focus on those who report that they value, seek out, and use research in policy decisions. Almost half of our sample (47 percent) was composed of research users (both the Enthusiastic and Skeptical Users of Research) who were well above the mean for research use (0.5 SD to 1.5 SD) of their legislative peers. The conceptual error of zeroing in on one segment of the population assuming it represents the whole provides false guidance to those who would translate research for political consumption. Second, a provocative cluster that warrants further investigation is what we call the Enthusiastic Nonusers. They are enthusiastic about the value of research, but seldom use it in their decisions. Are these policymakers low-hanging fruit ripe for picking by brokers of research evidence? Given their low ratings on access, are they unfamiliar with the research that is available and, therefore, unable to use it in their decisions? Or does the nonuse occur because the research that comes to their attention fails to deal with important policy questions or is so outmoded or futuristic that its relevance is unclear? This raises the intriguing question of whether Enthusiastic Nonusers could be converted into Enthusiastic Users by providing the kinds of information policymakers seek, at the time it is needed, and in the formats they prefer. The research community would do well to seek ways of reaching out to this target population. Above all, the results indicate that there exist considerably higher levels of interest in research than the more cynical views of many outsiders suggest. That is a finding to keep in mind.

Understanding Legislative Styles in Their Home District In this section, we shift our attention to the constituents of policymakers, one of the primary stakeholders in their world. We present evidence that emerged from a classic study

48

U nderstanding P olicymakers

of policymaker’s perceptions of their constituents and the legislative style they developed to court constituents back home in their district. Previously in this chapter, we discussed the importance of constituents to a legislator’s effectiveness. Chief among them is that it is constituents who put policymakers in office and keep them there (Fenno, 1978; Mayhew, 2011). Conceivably, responding to constituents may require a different disposition and skill set than responding to one’s colleagues. A glimmer of the differences in policymaker behavior across these two settings was pointed out by a state legislator in our recent study: People don’t always recognize . . . that you work hard when you’re out on the campaign trail. It’s all about you. “I will do this” . . . “We” versus “Them”. . . . It’s important to stop when that’s done and make that turn. . . . Now I’m part of the team. I’m still the representative for my district, but I’m part of a team in my caucus. . . . It’s not just about me. In the 1970s, Richard F. Fenno Jr. carried out an ambitious qualitative study of 15 sitting members of the U.S. House of Representatives, two representatives-to-be, and one representative-elect (ten Democrats and eight Republicans). We stress this one study because it gives us rare insights into how policymakers organize their worlds outside the legislative institution in which they serve, and because it is unique in its depth and quality. Surprisingly, this world that exists outside of the legislative bubble in which they work often remains invisible to researchers and knowledge brokers, despite it being so critical to legislator’s political fortunes and professional effectiveness. It is remarkable, and instructive, that Fenno’s work has never been replicated or updated. Fenno, of the University of Rochester, spent eight years travelling with his sample of elected representatives in their respective districts located across the country. His resulting book, Home Style: House Members in Their Districts, was published in 1978 and describes how representatives view their constituencies and how these perceptions affect their behavior as lawmakers. Among other findings, Fenno describes four home styles, that is, the ways that policymakers present themselves in their district. He places policymakers along a spectrum that ranges from styles that focus on developing personal relationships to styles that stress the details of policy issues: person to person, the popular local boy, issue independence plus one-on-one, and articulating the issues. Each style also includes the element of service to the district, which will not be covered here. Policymakers whose primary style is servicing the district—interceding for constituents with federal bureaucracies or bringing federal funding into the district—seem less relevant to our interest in research-based policymaking. Clearly, this study is dated, and the details of their districts may have changed, but its implications for researchers and knowledge brokers remain. It is likely that some variation in styles along this spectrum of being based on relationships or issues would exist if the study were replicated today. These differences in style are not merely of academic interest. As you read Fenno’s depictions, think about the idiosyncratic differences in disposition and behaviors evidenced by these personal styles. Each is distinct. Each represents something of the character of the person holding the office and how they relate to key stakeholders in their world. Establishing a working relationship with the intent of bringing research into the policy arena probably means both learning about and accommodating these styles in some fashion. Connecting with a policymaker means knowing enough about that person to render the possibility of a connection likely.

49

P roblem , P layers , and P olicy P rocess

The Person-to-Person Style Congressman A views his district as homogenous. This southern, rural, and conservative district is primarily composed of one-party conservatives whose livelihood depends, in large part, on one industry. Congressman A is the third generation of his family who has lived and held office in this district, so he feels a deep sense of identification with his constituents. In fact, he does not think of his district in geographic terms, but rather in terms of personal contacts: “I represent a district in which my constituents and I have total mutual confidence, respect, and trust—95 percent, nearly 100 percent” (Fenno, 1978, p. 62). He presents his connections to his constituents in almost every conversation that he has, drawing on an encyclopedic memory of names and faces; family events including births, marriages, illnesses, and deaths; and his deep knowledge of the district’s agriculture and architecture, its flora and fauna, its industry and history. Following a Rotary Club speech, he named all 40 people present and what they did for a living, along with other assorted information about each one. Congressman A’s style of campaigning is based on cultivating personal, face-to-face relationships with people, a task that is difficult to delegate to others. His style involves little attention to or articulation of issues, and he was least comfortable in a format that included a lecture followed by questions and answers. Fenno (1978) observed that Congressman A’s issue-less style may depend, in a curious way, on an underlying consensus on issues such as race, foreign aid, government spending, and social conservatism. His style is one of being a stabilizer and maintainer. Thus, there are stylistic and strategic reasons for avoiding a focus on issues because they could be controversial and end up alienating some supporters. Even when faced with electoral difficulty, he does not speak about issues, describing instead an American heritage built on family and community connections. This theme serves to reinforce and legitimize his legislative style of maintaining personal relationships with people in his district. The Popular Local Boy Style Congressman B represents a heterogeneous district with rural, suburban, and urban constituents who work in a highly diversified economy. He describes his district as ideologically conservative, though it is not a one-party district like that of Congressman A; about 21  percent are Democratic and 21  percent Republican, with the rest being politically independent. Congressman B talks more about policy issues than Congressman A, but primarily on the single issue of a strong national defense, which he believes has support from the majority of his constituents. However, when asked about his political success, he attributes it, not to his national defense posture, even though it is an important issue in his district, but rather to being popular: Contrary to what people say—they say voters are issue-oriented and all that. . . . For most of the people I know it’s more a matter of personality. They pick out a personality whom they like, who they think is like them, whom they have confidence in, and they trust him to do what’s best for them. (Fenno, 1978, p. 73) In a district that is suspicious of outsiders and fearful of the unknown, Congressman B’s local credentials are golden. He was born and raised in the district and spent his whole life there. In high school and college, he was a local hero—a team captain, star 50

U nderstanding P olicymakers

athlete, and president of his high school class. He views an election campaign much like a popularity contest. In Fenno’s (1978, p. 74) words: “He is unfailingly nice to everyone, is invariably good-humored, and gives off an immense likability as a person.” Because of his attraction to popularity and his animus to controversy, he avoids taking positions on contentious issues. Unlike Congressman A, when he comes home to his district, he has fewer face-to-face encounters and accepts as many speaking engagements as possible. His presentation style is more impersonal and distant than that of Congressman A, making it appear that his remarks are directed more to demographic groupings than to individual voters. For example, Congressmen B works to court the women’s vote with his characteristic friendliness, masculine charm, and good humor, yet he maintains the same distance that he does in his relationships with other constituents. Issue Independence Plus One-on-One Style Congressman C represents a typical suburban district bordering one of America’s largest cities. He describes his district as mature first-, second-, or third-ring suburbs populated by well-educated, affluent, white residents. Congressman C thinks of his district as having two characteristics that define his presentation style: a high degree of interest in issues and a tradition of independence in politics. As a Republican in a district that typically votes 25 percent Republican and 35 percent Democratic, Congressman C presents himself as an issue-oriented independent to court the 40 percent in his district who are independent or of no known political affiliation. Unlike Congressman A or B, Congressman C thinks of his strongest supporters, not in terms of community ties or family and personal reputation, but rather as people interested in specific issues. He is not a single-issue candidate like Congressman B, although he does not believe that campaigns are won or lost on issues alone: The best way to win a vote is to shake hands with someone. You don’t win votes by the thousands with a speech. You win votes by looking individuals in the eye, one at a time, and asking them. Very rarely will anyone ask you about how you stand on something. (Fenno, 1978, p. 85) To be elected in his district, Congressman C believes that you must be thoughtful and informed on the issues, but there still is an element of the personal relationships that Congressmen A and B have with their constituents. For Congressman C, the clincher is not how to present oneself, but rather how to find constituents to present oneself to in a mobile district where 30 percent of the population turns over every two years. Congressman C’s constituents lead an active frenzied lifestyle, working long work hours in one or more jobs, and volunteering in civic and recreational activities. Meeting people at their workplace or organizational settings are his favorite ways to make contacts, especially if they allow one-on-one time, so he can present himself as concerned about their interests. Media is one way of gaining visibility, but its costs end up preoccupying Congressman C (and others like him in heavily urbansuburban districts) with the problem of raising enough money to purchase airtime. The Articulating-the-Issues Style Congressman D’s district was so heterogeneous that it could best be described as three different worlds—a city, its suburbs, and the rural countryside. The three worlds each 51

P roblem , P layers , and P olicy P rocess

seemed unaware that the other two worlds were even a part of the district. The fastestgrowing portion of the district, the suburbs, included some whose population was primarily white, Anglo-Saxon, and Protestant, and others that had primarily blue collar or ethnic residents. Thus, the district was not only socially and psychologically diverse, but segmented as well. Moreover, it could only be described as a district in transition. Congressman D’s ties to the district were not deep or strong; in fact, he lost the suburb in which he grew up by 1,000 votes in the last election. He was propelled into politics by his strong opposition to the Vietnam War, which forged a political style that has been issue-based ever since. Thus, those who are his strongest supporters do not back him because they know him or have a personal relationship with him, but because they agree on the issues and on the importance of emphasizing the issues. He has developed a home style that relied on his own personal verbal agility in debating, discussing, and explaining issues. In each campaign, he has pressed for debates with his opponents because this played into his personal strength of being able to articulate and respond to questions on the issues. In his first two campaigns, he presented himself mainly through coffees in constituents’ homes, often as many as eight or ten a day. Once in office, he scheduled open meetings, a setting in which he excels, in every city and town in his district. During each session of Congress, he held nearly 200 such meetings, during which he reminded constituents of the importance of his personal strength—being able to discuss any issue, explain his votes, and debate differences of opinion: “No congressman can represent his people unless he’s quick on his feet, because you have to deal with 434 other people—each of whom got there by being quick on his feet” (Fenno, 1978, p. 95). Yet he was keenly aware that his style at these town meetings involved more than his openness to discussing the issues and his agile and articulate presentation of self: People don’t make up their minds on the basis of reading all our position papers. We have twenty-six of them, because some people are interested. But most people get a gut feeling about the kind of human being they want to represent them. (Fenno, 1978, p. 95) Even though Congressman D is portrayed as the most issue-oriented style of the four policymaker types, he still recognized that his constituents desired access, communication, and relationships with their elected officials—qualities that he believed were important to reelection and to good representation. In the open meetings, he reached out for constituent trust by relaying the sense that this kind of give-and-take format required a special kind of congressman, a special kind of constituent, and, therefore, a special kind of relationship: “He massages the egos of his constituents by indicating how intelligent, aware, and concerned they are to engage with him in this new, open, rational style of politics” (Fenno, 1978, p. 96). Yet his attempt to connect with his constituents does not include connecting with their personal problems, which were delegated to his staff who accompanied him to the open meetings. A related theme in his bid for reelection was that he was a new kind of politician. He worked hard to disavow himself from the old-style politician that inhabited an “outrageous and outmoded” Congress (Fenno, 1978, p. 97). Thus, for a person who was not comfortable in person-to-person campaigning, presenting himself as an accessible, issueoriented, nonpolitician was a style that could win support in each of the distinct worlds of his district without alienating any of them.

52

U nderstanding P olicymakers

Implications for Researchers and Knowledge Brokers The inescapable lesson from this unique case study is that there may be no stereotypic policymaker and no “one-size-fits-all” strategy for reaching lawmakers with research. They each have a style that reflects their personality, their preferred mode of operating in the political world, and the constraints imposed by their constituents. Despite idiosyncratic differences across individuals, one domain dominates the policymaker’s world and shapes their exchanges with constituents and colleagues. It is the critical importance of relationships. These relationships may take a different form and serve a distinct purpose in the district and in a lawmaking body, but nonetheless they seem to be a central component of being an effective policymaker. Delivering research effectively requires an appreciation of the heterogeneity of policymakers. Those seeking to bring research into the policy arena must realize that not every policymaker will show interest in every issue and not every policymaker will respond to the same dissemination strategy. The importance of communicating research in different formats, and persistently doing so, was articulated by a long-serving, research-minded Wisconsin senator: One of the things that the Family Impact Seminars does is provide information in a variety of formats. I know it’s a lot of work . . . but it relates to the fact that we bring in information in different ways. And our staffs bring in information in different ways. Knowing that there is a booklet available online as well as a paper that has a summary of the presenters, oftentimes as an expansion of what the presenter presented, along with the bibliography. So, there are additional resources that we can access, or people who we can contact. We usually contact the researcher, or the person directly as opposed to reading the research. It is available online. It is available as a paper document. I’m old enough where I’m reluctantly moving from paper to the electronic format. . . . The seminars, not only the seminars, but also the break-out sessions or the brown-bag exchanges are very important. . . . Just knowing that those Family Impact Seminar documents are out there is useful. I’ll remember, “Didn’t they do something about health savings accounts a year and a half ago?” and I’ll go and look it up. There’s continuity with the organization, so I can go back and ask “Who was that guy? Who was the one that talked about, such and such?” [The Seminars are] creating a very valuable resource for legislators. Fenno’s (1978) results do suggest some testable hypotheses. Is one avenue for communicating research to policymakers going through a source they value—their constituents? Or can research dissemination be tailored to the top two or three legislative styles likely found in a lawmaking body, assuming we could reliably identify the styles currently in vogue? That might be a thought worth pursuing. Could it be that policymakers are more likely to respond to research when it is communicated by using familiar approaches, particularly those consistent with the home style with which they approach their districts? Could it be that connecting with policymakers like Congressman A may require a heavier focus on relationship building, whereas it may be a focus on issues that drive attempts to reach policymakers like Congressman D? We have noticed in our experience with the Family Impact Seminars that some legislators attend our convenings regardless of the topic, whereas others attend only when the issue overlaps with their policy portfolio.

53

P roblem , P layers , and P olicy P rocess

Understanding Policymakers Through the Eyes of Researchers and Intermediaries Who Work With Them We shift now to another perspective on policymakers by looking through the eyes of researchers who have traveled across cultural barriers to work with those doing policy. We conduct an abbreviated environmental scan of experienced researchers and knowledge brokers from a range of institutional settings. They are situated at both the state and federal levels and, thus, can provide us with a broad perspective of policymakers. We sought the input of those who worked in trade associations, in nonprofit organizations, in Congress, and in universities. By no means do we claim that this listing is exhaustive, but nonetheless we present it in the spirit of stimulating future study of whether the positive perceptions of policymakers that we report here extend across informants and policy settings. We then turn to a formal study of researchers who effectively communicated research to policymakers, whom we refer to as exemplar communicators, to examine how the positive perceptions that are reported here develop. Are researchers who wade into the policy arena motivated by their positive impressions of policymakers? Or do researchers begin the journey with few prior perceptions or with more negative impressions of policymakers that are dispelled over time with more contact and exposure? We delve into these questions using researchers who gave presentations at Family Impact Seminars and who were selected into the study because their presentations were the most well-received by policymakers. Most had extensive policy experience, which allows us to tap into their initial impressions of policymakers and how these evolved over time. Finally, we conclude with implications for those who are considering engaging with the policy world to suggest things they might anticipate when contacting inhabitants of this foreign culture. Perceptions of the Policymaker Species Across Institutional Settings We begin with the experiences of a longtime senior fellow at a national trade association, the National Conference of State Legislatures. Based on over a decade of experience, Mary Fairchild provided a perspective on what state policymakers are like that was counterintuitive to accepted convention. She expressed her surprise that the media and people she meets at family reunions and social gatherings are so “cynical” about policymakers, which flies in the face of her assessment that legislators are “top quality” and “ethical”. She explained that there is a “huge premium on honesty” in the policy process. Policymakers might be able to achieve some short-term goals by being “Machiavellian” in their approach, but these are not the sort of public officials who continue to contribute to the policy process over the long run. In her words: I don’t want to pretend like it’s just this wonderful, beautiful group of people that get along. I mean, there is supposed to be controversy. But . . . I think we can be really proud of our elected officials as human beings and they represent all the different aspects of society pretty realistically.  .  .  . I’m always disappointed when most of the attention focuses on the scandal, the negative. A researcher housed at the distinguished policy-oriented research organization, Child Trends, provided a similarly upbeat portrayal of her experience with policymakers. 54

U nderstanding P olicymakers

In the words of Kristin Anderson Moore, former president of the organization and now a senior scholar: People in the policy community are really pretty committed, very hardworking, smart. They are trying to do a good job. If you understand their starting point and their assumptions, then what they are doing really makes much more sense. So, I have maybe a more nuanced understanding of them. Edward Zigler, a developmental psychologist of Head Start fame, worked closely with Congress for over four decades. He found policymakers to be “dedicated public workers who really wanted to hear what research could tell them, and to use this information to form better policies” (Zigler & Styfco, 2002, p. 13). Reflecting on his experience at the state level, Dave Riley, an endowed chair at the University of Wisconsin-Madison, came to think of policymakers as being “a true representation of democracy”. He said that he found himself surprised, though in retrospect he should not have been, by the individual variation in how well-informed policymakers are: But there’s another way in which the Wisconsin legislators did surprise me. . . . Some of them are remarkably well-read. It has surprised me when I’m waiting in their offices and I look at the books on their shelves and then they pull them off and start talking about stuff they’ve read. Some of them really do their homework and really know things well and, of course, some of them are exactly the opposite too. They are an incredibly varied bunch. Changing Perceptions of Exemplar Research Communicators Positive portrayals of policymakers emerged across a range of informants who worked in a variety of settings. This raises the intriguing question of whether these positive perceptions were what motivated researchers to enter the policy world in the first place or whether these perceptions developed over time. We next shift to a study that provides preliminary evidence on this question. We interviewed 14 exemplar researcher communicators (78 percent response rate), who were selected because they passed both an “expert” and “policymaker” screen of their effectiveness in bridging the gulf between the research and policy cultures (see Friese & Bogenschneider, 2009). First, these exemplar communicators were one of 49 identified by “experts” serving on a seminar planning committee to be a panelist at a Family Impact Seminar. Of this list, 14 were selected for the study based on receiving the most positive reviews from “policymakers” in end-ofsession evaluations. The sample spanned the social sciences in fields such as developmental psychology, economics, family studies, health care, and sociology. These exemplars hailed from institutions such as Harvard University, the Oregon Social Learning Center, the University of Pennsylvania, and the Urban Institute. Each researcher had an average of 20 years of policy experience and, when combined, a total of 274 years of experience working with policymakers (see the Appendix for the methods of the study). The experience, affiliations, and policy credentials of these exemplar research communicators are displayed in Table 10.2 in Chapter 10. These exemplar research communicators reported that when they started working with policymakers, they viewed them through the lens of their own culture, which tainted their initial impressions. For example, one researcher’s reaction to his first 55

P roblem , P layers , and P olicy P rocess

experience testifying before Congress was how little he knew about the policymakers’ world: “I think my first impression was that I could never talk to those people because their world was so different than mine, and I didn’t want to think the way that they had to think” (Friese & Bogenschneider, 2009, p. 233). Initially, 11 of 14 of these exemplar communicators admitted that they held negative stereotypes about policymakers. However, these negative impressions were not sustained over time as researchers came to know policymakers as people and came to appreciate the skill set that it takes to do their job well. These perceptual shifts are predictable, according to one exemplar communicator, because it is always the case that as you get to know people, they become more like themselves and less like your preconceptions of them. Coming to Know Policymakers as People With increased exposure to policymakers, negative stereotypes tended to fade away. For example, the exemplar communicators reported that they initially thought that policymakers had “short attention spans”, “were driven by money”, and were “looking for easy solutions to complex problems, unwilling to take risks, and arrogant”. One researcher explained how these stereotypes had changed for her when she was able to “put a human face on policymakers”. One endowed chair at a major research university found himself pleasantly surprised by how “normal” and “everyday” policymakers were: “They play on hockey teams and . . . they’re just regular normal sort of people in many ways and you don’t expect that somehow when you see them in the paper or on the press.” One academic said that, based on ten years of experience, his initial impressions were simplistic and unsubstantiated, but changed as he came to understand policymakers in more complex ways: I have more of an appreciation of the contributions some of these folks make. They don’t make a lot of money, they get a lot of grief, they get called at home, and they have a few areas in which they want to make a difference, on which they have to work year after year before they make a difference. (Friese & Bogenschneider, 2009, p. 237) The more contact that researchers had with policymakers, they more they came to view them as “caring”, “committed”, “idealistic”, “intelligent”, and “rational” (Friese  & Bogenschneider, 2009, p. 235). Several researchers said that, based on their interactions with policymakers, they had grown to deeply respect them for their commitment and intelligence. For example, one researcher explained that he has come to appreciate that policymakers have “wisdom, knowledge, and experience to bring to the table, and that is no less than my knowledge and experience”. A sociologist and later a dean at his university, he gradually developed a level of respect for policymakers as he came to realize that their decisions to get involved in public life were often motivated by idealism and the desire to make a difference. Coming to Appreciate the Skill Set It Takes to Do the Policy Job Well The researchers’ initial forays into policymaking were met with surprise at how little they knew about the policymakers’ world. With increased exposure, one researcher said 56

U nderstanding P olicymakers

his negative stereotypes gradually changed as he became more aware of what the job of policymaking entailed, and the skill set that it took to do the job well: I came to appreciate the best part of a politician and public official  .  .  . the pragmatics, the way they have to make alliances, the way they have to be swayed by public opinion, have to take the pulse of their constituents. . . . They remember people’s names, they are gracious, they thank people, and they build relationships. To explain their reactions when entering an alien culture, some researchers drew comparisons between the core technology of the policy community and that of the research community. One researcher, who is a member of an international team analyzing family policy in the United States, the United Kingdom, Canada, and Australia, concluded that the job of a policymaker is harder than his job as a researcher because policymakers have to make decisions: “A  researcher can say ‘on one hand and on the other hand,’ [but] a policymaker has to decide.” One fellow at a nonpartisan policy research firm also compared the constraints that policymakers face to the ones that she encounters: “Policymakers have constituents that they have to answer to, have constraints around them—what they focus on and what their option set is. As a researcher, I don’t face those constraints.” Chief among these constraints were the budgetary and political considerations that “academics can largely ignore”. Researchers also described instances where there was a clash of cultures. Mutual misunderstanding on both sides of the research-policy divide sometimes led to mutual disappointment—policymakers trying to fit research into the realities of the political world and researchers worrying about the adjustments it would take if their research was to be used. Despite these challenges, researchers also reported coming to see policymakers as collaborators and how they were “able to establish a common mission in many respects” with them. One prescription drug researcher with 25 years of experience concurred that he encountered more politics than he expected, but also observed more successes than he anticipated: “parties working together, coming together for the common good.” These exemplar communicators found that open communication and prolonged contact with policymakers helped dial down stereotypes and break down cultural barriers. Perhaps most important for the purposes of this chapter is that none of their negative stereotypes persisted and their perceptions became more nuanced as researchers gained more awareness of, exposure to, and experience with policymakers. With more familiarity came the realization that they could work together toward the pursuit of a common goal, with each contributing their own wisdom and skills to the task at hand. Implications for Researchers and Knowledge Brokers Researchers who over time became exemplar policy communicators demonstrated how easy it is for researchers to view policymakers through the lens of their own culture. When they first entered the policy arena, two-thirds of these researchers admitted holding negative stereotypes about policymakers, none of which persisted. Once they came to know policymakers better, they came to see them in more nuanced and less negative ways, a pattern that appeared in both state and federal settings. Over time, researchers and policymakers were more apt to appreciate the wisdom and skills each contributed to a common agenda of improving the quality of policy decisions. 57

P roblem , P layers , and P olicy P rocess

Conclusion Getting started on advancing evidence-based policymaking can be daunting. Many researchers have about as much desire to engage policymakers as a “gorged boa constrictor might have to swallow a porcupine wrong-end-to” (Roosevelt, 1904). Yet the best preparatory work for advancing evidence-based policymaking is understanding the nature of the policymaker audience (Tseng, 2019b). This idea seems quite novel in this context, yet it builds on a long-standing scholarly tradition that calls for a clear articulation of the attitudes, behaviors, and motivations of the client or learner. Social workers claim ownership of a traditional adage to “begin where the client is”, just as educational psychologists take credit for establishing a similar principle as a key to effective teaching. The Cooperative Extension Service attributes its effectiveness in working with communities to its long history of placing the needs of local citizens at the center of its program planning process. Karen often jokes that one of her long-standing quests is to learn to think like a policymaker. Legislators find it quite amusing that a professor wants to learn to think like they do. In reality, her private quest forms the foundation of any effort to bring research to the attention of policymakers in ways that are useful to them in their jobs. The first lesson that Karen teaches any new staff member or graduate student is to form an image of a policymaker in their mind—what they do, which types of information they need, how they are likely to use it, and the ways in which they prefer to receive it. She tells them to imagine the kinds of information policymakers use when they give a stump speech on the Fourth of July, make brief comments at a Kiwanis meeting, write a newsletter to constituents, or try to persuade colleagues to their point of view. Fixing on these images of a policymaker is critical irrespective of whether the task at hand is planning a seminar, organizing a discussion session, or writing a policy brief (unless one cares little whether the product will be delightful for one’s colleagues but dreadful for policymakers). Of course, any advice has the intended benefit only if the image of the policymaker is an accurate one. One Republican in our study lamented that if you believed the television commercials about him, he hates “children and teachers and old people and probably disdains rainbows and puppies and cotton candy too”. The daily media coverage of policymakers portrays them through typically artificial partisan and ideological lens that obscure the heterogeneity revealed by science. Policymaking is a profession that attracts individuals from widely different backgrounds and perspectives. Some use research to design policies, others essentially to stop legislation, and others to build a reputation among colleagues as a trustworthy and knowledgeable source of information. Some legislators will turn to research to respond to constituent questions, and others will use it to develop relationships with external partners. We also have policymakers who are enthusiastic about seeking and using research in their decisions, alongside those who are enthusiastic about research but have not yet translated this enthusiasm into concrete action. You get the point. This quest to better understand the policymaker audience is made even more urgent by one of the most important lessons from our interviews of exemplar communicators— how easy it is to stereotype policymakers and how wrong these initial impressions often are. Almost four of five researchers, those who later became exceptionally skilled at navigating the research-policy divide, relayed to us that their initial impressions were dead wrong. Such wrong-headed thinking can sabotage any efforts at bicultural boundary spanning. The good news is that these stereotypes were not immutable, but proved 58

U nderstanding P olicymakers

malleable as researchers gained more exposure to policymakers and became more experience interacting with them. For many years, Tom taught the yearlong practicum course for second year graduate students pursuing a master’s degree in social work. This involved an internship in a site that would expose them to some aspect of doing policy work. He would do his best to expose his annual band of eager students to the harsh realities of what policymaking was all about, and the difficulties and frustrations that it entailed. Every year, he thought that some would transfer out of his course into one of the more traditional social work paths such as mental health or child welfare or geriatrics. And yet each year they would surprise him. Not one ever expressed later regret, at least not to him personally. Familiarity with the policy process did not breed contempt. We are reminded of the advice of one researcher who recommended approaching policymakers as an anthropologist and a change agent. Become “fascinated” by learning more about policymakers and become a “student” of who they are, how they think, and what it takes to be successful in their world. “Learn from these folks. Many will be happy to teach you.” We couldn’t agree more. Some of the content of this article was adapted from previous publications with permission. Copyright © 2020, American Psychological Association Bogenschneider, K., Day, E., & Bogenschneider, B. N. (2020). A window into youth and family policy: State policymaker views on polarization and research utilization. American Psychologist, https://doi.org/10.1037/amp0000681 Copyright © 2013, National Council on Family Relations Bogenschneider, K., Little, O. M., & Johnson, K. (2013). Policymakers’ use of social science research: Looking within and across policy actors. Journal of Marriage and Family 75, 263–275. https://doi.org/10.1111/jomf.12009

59

4 UNDERSTANDING THE POLICY PROCESS Insights From Insiders Karen Bogenschneider and Thomas J. Corbett

For researchers to connect with the world of policymakers, they must understand the processes involved in the making of public policy. Oftentimes, researchers misunderstand the policy process by inadvertently projecting the scientific processes they are familiar with onto the way policymakers make decisions. This chapter brings an unconventional point of view by looking at the policy process through the lens of culture. Conflicts can occur when incompatibilities between the cultures of the research and policy worlds remain unrecognized and unappreciated. It seems important to me that men of good will should make an effort to understand how the world ticks; it is the only way to make it tick better. Charles Percy Snow (1961, p. 66) Knowledge from research is itself the product of the context of its creation and will acquire new meaning within the fresh context of its use. Nutley et al. (2007, p. 303) In 2005, the National Research Council began an inquiry into how to improve the use of research evidence in the making of public policy. The endeavor was a serious one. The council convened workshops, engaged in discussions with researchers who study the issue, and consulted with policymakers themselves about their research use. The preliminary findings were met with such enthusiasm that a formal committee was appointed to develop a framework for how research could improve public policymaking. The final report was published in 2012 with a surprising conclusion—“The social science that is needed to understand the use of science is not research about the consequences of those decisions: it is research about the decision process itself” (Prewitt et al., 2012, p. 58). The report reflected a consensus among the committee that research is not useful in policymaking when it fails to reflect the realities of the decision-making process. The report specified the kinds of research that ignore the way policy decisions are made (p. 42): The research [that] should be ruled out [is]: (1) that the science/policy nexus can be uniformly understood in terms of rational decision-making models; (2) the assumption of a specified single actor with freedom to achieve goals formulated through a careful process of rational analysis characterized by a complete, 60

U nderstanding the P olicy P rocess

objective study of all relevant information and options; and (3) the definition of use as problem solving in the sense of a direct application of evidence from a specific set of studies to a pending decision. After decades of study of research utilization in policymaking, how could it be that we have come to the point where we know only what is not needed, but still struggle to identify what is needed (Oliver, Lorenc et al., 2014)? Tom points a finger at the research enterprise itself (Corbett, 2018a, p. 568): Many of us would decry the state of the art, what we call “black box” research. There is an interest in the policy front end and the results-focused back end, but all that juicy institutional stuff in the middle is brushed aside. Never open the box to look inside. Who knows what might pop out? Why is that? What does explain this conundrum? Why have researchers shown so little interest in formal policymaking processes, which have remained relatively static at the federal level since first established in 1789 (Mayhew, 2011)? If this conundrum is considered through the lens of community dissonance theory, as more fully explored in Chapter 7, the culprit could be the cultural blinders that obscure the fundamental differences in how the worlds of research and policy function. Researchers may be blindly interposing their own reading of the policy institution and its practices into their lines of inquiry, without fully grasping how starkly different the processes are for making decisions in the research and policy cultures (Friese & Bogenschneider, 2009; Weiss, 1999). As explained by Henig (2008, p. 221), “There are some built-in tensions between, on the one hand, the enterprise of research and the norms and institutions that support it and, on the other hand, the enterprise of democratic governance and its supportive institutions and norms.” These tensions can be “endemic” but need not be “debilitating” (Henig, 2008, p. 221). As a starting point, the policy process must be perceived, not as a “necessary evil”, but also as a “necessary good” (Henig, 2017). For those ensconced in the research culture, it is all too easy to lose sight that democratic norms and institutions are a “precious achievement” (Pinker, 2018, p. 343). One of our country’s founding documents, the Declaration of Independence, is said to be the most famous product of the Enlightenment era—a human invention that enhances the welfare of citizens by encouraging cooperative actions and discouraging selfish ones (Pinker, 2018). For those who live in a democracy, it is easy to take for granted how the democratic process prevents a single individual or group from exerting a disproportionate degree of influence over public policymaking (Albaek, 1995). When conflicts occur and people do not see eye to eye, there is a “huge difference between a political process in which people honestly try to see the world from different vantage points, and one in which they claim from the start that their vantage point is the right one” (Stone, 2012, p.  385). Decisions from such deliberative processes emerge with a stamp of authority: “Nothing—not the Will of God, not the Intrinsic Nature of Reality, not the Moral Law—can take precedence over the result of agreement freely reached by members of a democratic community” (Rorty, 1999, p. 237). Granted, democratic institutions do not always work perfectly, plagued as they are by inherent inefficiencies and conflicts. Yet cognitive psychologists liken them to a “factory that manufactures judgments and decisions” using formal processes that tend to work better than those used by individuals (Kahneman, 2011, p. 418). Institutions have built-in mechanisms for avoiding errors in judgment, such as proceeding more slowly 61

P roblem , P layers , and P olicy P rocess

and imposing orderly procedures that encourage more prudence in coming to decisions (Kahneman, 2011). The founders included “checks and balances” by intent, so that a new form of autocracy would not replace the old one. Without an appreciation for the process of democratic decision-making, researchers can shy away from engaging in policymaking believing the process to be byzantine and inscrutable. Without an understanding of how governments grind out policy decisions, researchers can feel lost in an obtuse process, leading to frustration and disappointment. Left unchecked, this disappointment can devolve into disdain for the political process itself and diminish any desire to connect with the policy community. We begin here with two vignettes on how the policy process works. We first tap into the perspectives of one population of policy insiders—state legislators—who describe an open, transparent, rule-guided process designed to produce better policy decisions. After that, we turn to another policy insider, a knowledge broker who worked closely with policymakers for over four decades. Tom tells the story of welfare reform that exemplifies how diffuse, uncertain, and complex policymaking can be. We also draw on our study of 14 researchers who effectively engaged with policymakers (Friese & Bogenschneider, 2009), and our recent study of 225 state policymakers who described research use in the policy culture (see the methods in the Appendix). We employ these studies and draw on our experiences to illustrate ten observations about how the policy process really works—observations that may seem obvious to policy insiders, but perhaps surprising to those viewing the process from the outside. Countless books and monographs have been written across the decades that describe the processes and players through which a bill becomes a law (Bogenschneider, 2014; Levin, 2005; Lindblom, 1968; Ross & Staines, 1972) and the administrative and political constraints imposed on policymakers by the institutions in which they operate (Caplan, Morrison,  & Stambaugh, 1975; Mooney, 1992; Webber, 1986). We will not attempt to replicate these treatises here, given that the policy process is one that has remained relatively constant over time. Instead, we will describe the policy process for the express purpose of increasing understanding of those aspects of its functioning that likely are of relevance to those who want to study policymaking and broker research use. Sometimes the real-life experiences of those inside and outside the culture can be instructive.

The Policy Process Through the Eyes of State Legislators: The Routes Through Which Research Flows The insider’s view of the policy process is described here through the lens of one segment of the understudied population of policymakers—state legislators. This perspective should not be accepted as a monolithic representation of all state policymakers, or their local or national counterparts, or other policy actors in the judicial and executive branches of government. Yet it does pull back the curtain a bit on how the inhabitants of the policy culture understand the world in which they operate. Importantly, we elaborate on their perspectives later in this chapter using a more heterogeneous lens that is derived from a wide slate of policy observers, analysts, and scholars. We also briefly overview the findings of our recent studies on how state legislators describe the routes that research takes as it flows through the policy process. Somewhat surprisingly, descriptions of the policy process by lawmakers themselves often mirror the views of researchers as stated at the beginning of this chapter, even using

62

U nderstanding the P olicy P rocess

identical words in some instances. For example, a legislator told us in the interviews that one of the main reasons that the public hates policymakers is that they “look at the legislature as this black box that’s not understandable”. Another legislator explained that public understanding may be jaded because it is based on watching too many movies about corrupt politics and crooked politicians. The reality of the way that policy is made is “never as interesting or successful . . . as on your movie screen”. Moreover, contemporary understanding of the policy world also is distorted by glimpses of it that come through the typically artificial images of social media and around-the-clock media coverage (Pinker, 2018). Policymakers recognized what many researchers have failed to appreciate—the importance of learning about the policy process. Most policymakers explained that getting their legislation passed depended on a basic understanding of how the policy process works, but some policymakers became serious students of its procedures and rules. During one interview, a state legislator explained that being effective depended on knowing the rules and handed Karen the state’s rule book that he kept within easy reach on the top of his desk. Even before being elected, he was so committed to learning about the policy process that he would binge watch online recordings of legislative sessions: I wanted to know how everything worked. And so, I watched everything. I mean, my wife would wake up at 3:00, 4:00 in the morning and I’d still be watching the floor session. And she would go, “Are you nuts? Go to bed!” Another legislator, in response to Karen’s standard interview question about his area of specialization, named not a policy issue as most legislators did, but with his expertise in policy procedure. He explained that his ability to rise in the ranks of leadership depended, to a large extent, on the commitment that he had made to developing expertise on institutional procedures: No one else does this. There’s nothing sexy about that. It doesn’t produce votes. It doesn’t get publicity. But it works. And, as a result, I  .  .  . moved over to the senate  .  .  . [and] became the Assistant Minority Leader, then the Minority Leader.  .  .  . because I  was able to know more about what was going on procedurally. In general, policymakers provided a positive portrayal of the policy process and the people involved in it, as exemplified here: I am thankful we have the process we do. So, although I get so aggravated with it at times, it’s still the best process in the world. . . . Do we do some bonehead things? Yes. Have we and will we continue to pass legislation that we find out, later on, that maybe there’s some unintended consequences? Yes. But I  really have never seen a time when someone I think is doing it deliberately to really try to harm someone or harm the process. In our interviews, legislators spoke positively about the inner workings of the policy process with their observations revolving around three main themes—the people who inhabit the policy culture, the working environment in which policy decisions are made, and the rules that govern the process.

63

P roblem , P layers , and P olicy P rocess

The People Who Inhabit the Policy Culture One of the greatest things about the legislative process, according to the legislators in our study, is the people who ply the policy trade. Policymakers come with assorted life experiences and levels of expertise, all of which contribute to an understanding of the avalanche of issues that confront and confound them. Policymakers (at least most of them) also bring a commitment to solving problems for the benefit of their constituents and the people of their state (see the discussion of effective legislators in Chapter 3). As an example, one Indiana Republican, who represented a white, suburban district, was approached by a Democratic colleague seeking his support on an issue affecting his inner-city district. In Indiana, Republicans hold such a large majority in both chambers of the legislature that they can accomplish their goals without paying any attention to requests from Democratic colleagues. The response of this Republican indicated a group-oriented thinking process that moved beyond its implication for his district to what it meant for the entire state: If I can do something to help his people and it may not have any effect on mine, if I can help his people, why wouldn’t I do that . . . because they’re also citizens of Indiana. You’ve got to realize that we’ve got kind of a dual role. We represent 66,000 people [in our districts] . . . but likewise, you also represent 6.6 million people [in our state]. . . . You want to think about the people in Angola and Evansville, the far parts of the state that I don’t go to . . . or the inner-city. . . . It’s a much, much bigger picture than your district. Legislators described many instances of coming together for the common good, but also pointed out that there is some truth to the adage that “politics is theater for ugly people”. One Wisconsin legislator described it this way: We’re incredibly egotistical and selfish people. This is a group of really tough people to work with at times.  .  .  . There have been countless times that  .  .  . individual and immediate political wins have deterred critical policy items from being advanced. . . . We have this very difficult schizophrenic lifestyle of both being very much individualistic, but then also being a large legislative body and needing to be successful group thinkers. And the push and the pull between the self and the group can be incredibly difficult and often is at odds. This legislator explained that he was in a deep minority in his state, so he currently had little power to push through his agenda in the ways his Republican colleagues were currently doing. However, he explained that if he were in the majority, the temptation to use power tactics to accomplish his policy goals would be hard to resist: I say that because I live in and represent a community that’s really dying and hurting. And on their behalf, I cannot sit back and not use power to effectuate positive change in their lives. Poverty is unfair. There’s a little baby that was born into a family and they’re going to have the worst outcomes in work and lack opportunities that you and I will never dream of. . . . It’s not fair. We’re going to do something about it. In keeping with policy scholars, legislators in our study described the motivations and actions of their colleagues, but also provided a more contextualized portrayal of how 64

U nderstanding the P olicy P rocess

these individual propensities are affected by the decision-making processes of democratic lawmaking institutions (Tseng, 2012). The Working Environment in Which Policy Decisions Are Made The majority of legislators expressed appreciation that the processes of government primarily operate in the public eye. As described by one legislator: The process is open. It’s transparent. That’s the way government should be. I think it’s just a tremendous institution. I’d defend it to the umpth degree. . . . I really admire the sacrifices that people have made, so we can continue to do this. We did hear examples of times when these democratic principles were violated. Decisions were quickly made without public input, or lobbyists appeared to be pulling the puppet strings in the wee hours of the morning. Yet these examples were voiced less often and less vehemently than would be expected if one relied solely on movie narratives, press accounts, or coffee shop analyses. The Rules That Govern the Policy Process In both quantitative and qualitative assessments in our study, lawmakers strongly endorsed the rules and processes that guide democratic decision-making. One legislator likened the ways that laws are made to the “taffy pulling” established by our Founding Fathers, which he described as a “good part of the process”. One legislator explained that the policy process can remain obscure and even incomprehensible to outsiders because it represents such a stark contrast with the ways that decisions are made in other institutions: I think there is this perception that you just come up with an idea and everything else kind of takes care of itself. But that’s like saying you’re going to start a business and you have this idea for a cool mobile app. Like, who’s the developer. . . . What is the computer programming that goes into it? I  think it’s really frustrating to people at times, especially people that come from the private sector. Because I think good ideas tend to rise to the top because everybody’s going to make money. But there’s not that equivalent, so everybody has different [takes on it]. The committee has their own politics. Regions have their own politics. Caucuses have their own politics. And then the Assembly versus the Senate has its own politics. Despite the constraints of using an open process to come to agreement, this legislator concluded that paying attention to the institutional rules and caring enough to work within them actually can help you be a more well-informed policymaker and, therefore, result in better policy decisions. One Indiana Democrat characterized this back-andforth deliberation as the real “beauty of the legislative process”: At the end of the day, there have been so many eyes, and so many different positions that have looked at your language and it’s gone through so many iterations. By that time, if you are successful at getting through the entire process, 65

P roblem , P layers , and P olicy P rocess

which is committee in the Senate, committee in the House, committee in the House, committee in the Senate, conference committee . . . if you’re able to get through that whole process, then most likely, you’ve got a pretty well-vetted bill. The Routes That Research Takes as It Flows Through the Policy Process Finally, we turn to how research flows through the decision-making process as reported by policy insiders. Our study of state legislators disputes the indiscriminate allegations of many policy outsiders that research is underutilized across the board (see Bogenschneider  & Bogenschneider, 2020). Here we touch lightly on the use of research in the policy process because it has been explored more fully in other chapters of the book (see Chapters 5, 8, and 9). In a recent paper, we likened the flow of research through the policy process to the way a river flows through the landscape, winding back and forth in convoluted and circumstantial ways (Bogenschneider  & Bogenschneider, 2020). Policy issues infused with morality, ideology, or passion flow through narrow, nonnegotiable tributaries that restrict research use. However, research can navigate the policy landscape on new issues, nonemotional technical issues, or those for which the policymaker has no established position or bipartisan consensus already has been reached. Our study also revealed that any claims about research underutilization are open to criticism without considering the circuitous path of research as it winds its way through the policy process, which is seldom straight or a steady stream. Research use can be facilitated when it is introduced at the entry port of a committee where the expertise and experience on issues typically lie, and when it enters upstream where the political current is slower and more deliberative. Research use is frustrated when it enters the policy stream later in the legislative process where it is more likely to hit the rapids of unrelenting time pressures and the steep banks of entrenched policy positions. Research does not always carry the day, but there are several spaces and places where research does flow, much like a river, through the policymaking process (Bogenschneider & Bogenschneider, 2020). Those seeking to bring science to policy deliberations must remain cognizant where the most promising opportunities lie.

The Policy Process Through the Eyes of a Knowledge Broker: Welfare Reform and Science Tom observed firsthand the process whereby welfare reform legislation was enacted during his stay at the U.S. Department of Health and Human Services. On loan from the University of Wisconsin, he served as Senior Policy Analyst in the Office of the Assistant Secretary for Planning and Evaluation (ASPE). ASPE was the organizational locus around which planning for the 1996 welfare reform bill, the Personal Responsibility and Work Opportunity Reconciliation Act, took place. In many respects, this was an ideal laboratory for doing evidence-based policy development, so it is an informative case study of the promises and perils of research use. Harvard professor David Ellwood had assumed the Assistant Secretary position at ASPE, and his Harvard colleague, Mary Jo Bane, had assumed leadership of the Health and Human Services Unit with programmatic responsibility for the Aid to Families with Dependent Children (AFDC) program. Robert Reich, a Rhodes scholar and lecturer at top universities, headed the labor department. The AFDC program was an object of widespread popular revulsion and, thus, an ongoing target of reform. However, any 66

U nderstanding the P olicy P rocess

proposed changes invoke mind-numbing technical and political challenges. Proposed changes inevitably raised concerns that the most vulnerable members of society (poor children) would be further disadvantaged. The academics heading the planning effort were dedicated to the principle of bringing the best research to bear on the contentious questions at hand. This would not be politics as usual. How well did this marriage of research producers and research consumers work? For the record, the Clinton proposal was published on June 21, 1994, too late to receive serious political discussion before the fall congressional elections that turned the House of Representatives over to the Republicans who were led by Newt Gingrich. The GOP quite naturally had their own ideas about the role of government, as embodied in the Contract with America. Clinton’s plan, as originally devised, was essentially dead, but that says more about politics than either its merits or the character of the policymaking process through which it was developed. What is less appreciated is the fact that the very attempt to advance an evidence-based strategy for reforming welfare may have inadvertently contributed to the ultimate demise of Clinton’s proposal. How so, you ask? There was a mountain of research and analysis on most of the critical issues regarding the U.S. welfare system. Take the labor supply issue: Do welfare payments reduce work effort? It is a simple enough question. Yet the research offered a range of estimates depending on what methods were used, how the question was framed, which data sets were employed, how the modeling was done, and so on and so forth. It is feasible to arrive at a consensus, but the process is long and can be painful. Multiply the transaction cost (time and effort needed to arrive at a conclusion about the research) by the sheer number of issues on the table and the complexity becomes daunting. So do possible points of contention. Where the science is ambiguous, even small political differences are magnified. Complexity, however, was not the only challenge. Scores of studies had been done on specific interventions designed to reduce welfare use and increase labor market participation. Some focused on the job search process, others on enhancing human capital, still others on remediating various work impediments. Researchers naturally wanted to isolate the contribution that individual interventions might have on any observed outcomes. However, program evaluation expert Peter Rossi has often noted that the expected value of any one social experiment is zero; some work a little, some not at all, and some go in the wrong direction (Rossi, 1987). In the aggregate, none of these individual interventions changed outcomes very much. The welfare literature was less discouraging than some other social program evaluations. There was rigorous evidence that some interventions did, in fact, work according to the usual scientific standards of success. But if you took the typical (successful) evaluation back in the early 1990s, you would find something like the following in terms of net effects: examining the difference between those in the experimental and control groups, welfare use might fall a handful of percentage points, labor supply might inch up by 5 percentage points at most, and earnings might jump as much as 30 percentage points, but from a very low base, and certainly not enough to lift a struggling family out of poverty. In short, many things worked but no single approach worked very well. This was not great news for the research-driven, welfare gurus. Under political pressure to end welfare as we knew it and impose time limits on welfare receipt, they faced tough choices. The evidence suggested we could at best reduce welfare modestly. Therefore, to end welfare as the president indicated he wanted to do, we (the planners) would need lots of public service jobs, childcare, and other supports for those people who had reached their time limits. But reform was not supposed to cost money, or not much anyway. 67

P roblem , P layers , and P olicy P rocess

In fact, many anticipated budgetary savings. The available research led to very tough choices and no easy answers. So, the internal debate dragged on month after month. When Tom first arrived in Washington in late spring of 1993, the administration thought a bill would be done in mere months. By fall, the planners were no closer to a proposal. At the beginning of 1994, welfare reform was almost scuttled by the administration in favor of health-care reform. In the end, a complex bill was produced that could not match the expectations that had been climbing with each passing month. In the end, the key to successful reform was found outside of the strategies that were amenable to rigorous experimental evaluation. No single intervention nor small set of interventions were the key. Rather, real reform meant changing the very culture of welfare and the institutions through which it had been delivered. But broad cultural change was pretty much beyond what the evaluation methodologies of that era were capable of capturing. How do you randomize broad systemic change across treatment and control groups where the very intent of reform is to transform how clients and system workers conceptualize welfare in the first place? Changing broad perceptions on a communitywide basis is not the same as adding a narrow training component to an existing program. The cost, political impediments, and logistical challenges were daunting and, thus, rigorous investigation essentially eluded our best minds. Others, mostly outside Washington, were not put off by the lack of scientific evidence. The more ambitious policy entrepreneurs plunged ahead, pushing broad-based change absent evidence-based support on likely outcomes. So, what happened in the aftermath of the welfare reform law? Welfare rolls plummeted well beyond anything anticipated by the research literature, and money was thus available (for a time at least) to invest in childcare along with training and other work supports. In fact, Mary Jo Bane (2001) conceded in her presidential address to the prestigious Association for Public Policy Analysis and Management that the aftermath of the welfare law had been different than she had expected: “much larger caseload declines, significant increases in employment, fewer demonstrably adverse effects on children, and declines, though small ones, rather than increases in poverty rates for single mother families” (p. 191). Between 1994 and 2006, caseloads dropped by two-thirds—a dramatic result—and child poverty in single-parent households fell by 10  percentage points (Bane, 2009). Among female-headed families, pre-transfer poverty before any government help fell by over 20 percent in the years following reform, reflecting increases in work and earnings by these mothers (Haskins  & Sawhill, 2008). Researchers searched hard for adverse effects of welfare reform on child outcomes. Few were found and fewer were particularly robust; some positive outcomes even emerged (Chase-Lansdale et al., 2003). Bane (2009) was quick to point out that certainly the law had not “caused” these auspicious outcomes. They emanated, in part, from a good economy, an increase in Earned Income Tax Credit (EITC), and an improved child support system. Nevertheless, the law had not resulted in the devastating consequences she and others had initially predicted, in part, because society is a multidimensional phenomenon with many parts moving simultaneously. Welfare was not transformed in a vacuum. Does this little story suggest that the role, or potential role, of research is overstated or, worse, perverse? Not in the least. The real lesson is that research is not an easy panacea. Often, the findings are not easily transferable, the results may be mixed, the long-term impacts unclear, and the costs prohibitive. Most importantly, the questions researchers examined might be misplaced. The evidence available to the Clinton planners focused on modest, individual interventions when what was needed was evidence on broader transformations of the way welfare was conceptualized. 68

U nderstanding the P olicy P rocess

Ten Observations on the Policy Process: Obvious to Insiders, Surprising to Outsiders To make greater sense of our two vignettes, we now provide ten observations on the policy process that may seem obvious to insiders, but surprising to outsiders. Following the tenets of our community dissonance theory, we focus on how researchers’ understanding of the policy process may be compromised by the conflicts that arise because of incompatibilities between the cultures of the research and policy communities. We draw on our studies of research producers and research consumers (see methods in Appendix) and the observations of a broad slate of policy analysts, pundits, and scholars, particularly those of Jeffrey Henig (2008, 2017) and Ben Levin (2003, 2005). As shown in Table 4.1, these observations encompass the way the policy institution operates and how this guides the behavior of policymakers, the nature and complexity of the issues that compete for attention in the policy world, and the forces and values that drive policy decisions. Table 4.1 Ten Observations of the Policy Process: How Cultures Conflict in the Research and Policy Communities  

69

P roblem , P layers , and P olicy P rocess

Observation 1: Policymaking Is a More Rational Process Than It Appears One cultural conflict that strikes at the core of cross-cultural misunderstanding is the question of whether policymaking is a rational process or not. This question has remained elusive because it is entangled in a true paradox—two principles that are simultaneously valid, yet irreconcilable with each other (Rappaport, 1981). The first principle long embedded in Western thought is the desire to see human behavior, whether individual or collective, as goal-directed and rational. A second widely endorsed principle is a belief in democracy as an ideal system of governance for working out disagreements and conflicts of interest through compromise and consensus that does not allow exorbitant influence by any single individual or group (Albaek, 1995). When these two ideals are simultaneously upheld, the natural inclination is to overstate the premises of each, which in this case is portraying policymaking as a fully rational process or, on the other hand, as functioning with no rationality at all. If the principle of rationality is to prevail, decisions would be turned over to researchers like Ward’s scientific lawmakers or Plato’s philosopher kings. On the other hand, if the principle of democracy is to prevail, lawmaking must operate as a messy, consensus-building process in which research can be used for political purposes or to further one’s own ends (Albaek, 1995). Therein lies one of the fundamental cultural conflicts that makes the policy process seem inscrutable to researchers. Because researchers often assume that others think in the same scientific way they do (Kendall-Taylor & Levitt, 2017), they may misconstrue the role of research by looking for its influence in a systematic, step-by-step progression to a preferred policy outcome. Yet rationality in a political context must consider the role that research can play in a deliberative process where consensus must be achieved at multiple points before concrete action is taken. The real influence of research is not likely to be found in the independent power of any single individual nor any given point in the policy process. Instead, research may have considerable sway throughout the process in ways not easily transparent to outside observers such as contributing to accommodation of opposing views, coalition building, persuasion, rhetoric, and reconciliation of conflicting values (Albaek, 1995; Cairney, 2019; Prewitt et al., 2012). Like any complex process, policy decisions have multiple points of participation that make it difficult to identify who initiated them, what the drivers were, and how decisions ultimately were consummated. This difficulty in mapping out the intricate ebb and flow of policy production, however, should not be confused with irrationality. For example, in a pluralistic society, there are several competing interest groups, each with different goals, values, and preferred ends. Policy ends that are ideal for one group may be an anathema for another. This is not unexpected in a pluralistic political system where competing ends are sometimes pursued by such stakeholders as citizens, interest groups, political parties, and where differences are resolved through a back-and-forth process of negotiation and compromise. The result is not likely to be the proposal advanced by any single group, but rather a conclusion that may suit some stakeholders more than others, advantage one interest group at the expense of another, and maximize some political ends while minimizing others. So, in a roundabout manner, if all the players and points of participation are taken into account, policymaking is a rational process but not necessarily in a straightforward way that conforms to the tenets of the classic decision-making model (Nutley et al., 2007). Policymakers are interested, not only in applying research evidence to their decisions, but 70

U nderstanding the P olicy P rocess

in representing the views and values of their constituents in a political system designed to reconcile the inevitable differences through civilized debate in a public setting. Put simply, “Theirs is a political rationality, rather than a scientific rationality” (Weiss, 1978, p. 61). An Indiana Democrat gave an example of how hard it is to infer political rationality, using the example of Indiana’s Religious Freedom Restoration Act. This act, which allowed individuals and companies to use religious freedom as a defense in legal proceedings, proved highly controversial and provoked national attention: Because we would say “This is going to allow discrimination.” . . . And the other side would say, “Absolutely not, that’s not in here.” . . . To me that’s like the old movie, “The Sting”, where there’s multiple double crosses going on. Because I actually think they thought they were writing a bill that was harmless. But they wanted their right-wing supporters to think that it had real teeth and would allow them to discriminate. So, they were playing this double game where they would wink to their base and say, “Yeah, bakers, no gay cakes. . . . We’ve got you covered.” But then when the rest of the world said, “What are you doing”, they said, “Oh no, this is just a simple little thing, replicating the federal law, and there’s no impact—certainly no ability to discriminate through this bill.” And the fact that made you think they were playing that double cross is . . . they could have easily put the line in, which they eventually did after the fact, when the thing just blew up in their face. They could have easily put in there that nothing in this law shall be construed by the courts to allow discrimination. In the welfare reform vignette, the law’s passage did not meet scientific standards of rationality but still may have been rational from a political perspective. For those who designed the law and advocated for it, they would probably see political rationality in the way the process produced a law that ended up achieving at least some of its stated goals, such as reducing welfare rolls. Observation 2: Taking the Politics Out of Policymaking Is Not Possible, Nor Should It Be Another cultural disconnect that imperils researchers’ understanding of the policy process is the role of politics. Researchers hold their nose at a process that is so heavily influenced by the dictates of politics. As aptly put by Henig (2017, p. 1): Politics  .  .  . is traditionally seen as the skunk in the garden. Politics is about power and muscle and self-interest and group interest. Politics promotes policies because they suit those with advantage, and the thinking is that when politics comes into contact with research, it often ignores, coopts or corrupts the evidence that research brings to bear. Henig concludes that for those interested in brokering research for policy purposes, it will take a more sophisticated understanding of both enterprises—for research “scuffing up its heroic image as an infallible and universally applicable guide” and for politics “polishing its somewhat tarnished image” (2017, p. 4). Let’s dig into each enterprise a bit more closely. For researchers, the “search for truth is central [and] political power is merely incidental” (Smith, 1991, p. xviii). For policymakers, politics is central to negotiating decisions 71

P roblem , P layers , and P olicy P rocess

on complex issues amidst competing interests. Stated simply, researchers think about decisions in terms of “scientific control”, whereas policymakers think in terms of “democratic adaptation” (Albaek, 1995, p. 96). Researchers envision using science (a) to design policies that would achieve a democratically defined public good, (b) to oversee implementation to ensure fidelity and efficiency, and (c) to evaluate outcomes for effectiveness and continuous improvement (Henig, 2008). Policymakers, on the other hand, operate within a consensus-building process that has to accommodate the views and vantage points of a vast array of stakeholders. One legislator explained, “If everything was all black and white, they wouldn’t need us here.” This concept of consensus building may seem foreign to the inhabitants of the research culture, where technical decisions are not negotiated but are entrusted to the time-tested canons of science. A process of negotiation and consensus building, according to legislators in our study, requires a willingness to compromise (Bogenschneider, Corbett et  al., 2019). Yet, in the research culture, compromise is disdained because it is one of the least understood aspects of the political process and because it makes it seem that research is ignored, coopted, or corrupted. In the policy culture, compromise is not disdained but is explained in pragmatic terms as the way that work gets done: There’s got to be willingness to start on either end, but to gradually [progress] through a willingness to accept new information and a willingness to accept a different idea. And, of course, that word, compromise, which some people hate, but there’s just got to be a willingness to say, “Okay, I don’t agree with you on this, but I can accept that.” Legislators likened compromise to not getting “the whole loaf of bread,” but being content with getting “three quarters of a loaf”. From a policymaker standpoint, “Compromise is not hard. Not being effective in getting things done is hard. I’d rather find some compromise and get something done—move the ball down the field—versus do nothing.” In politics, the preference of legislators is to compromise with one side persuading the other and vice versa. But one legislator was frank in explaining, “If you can’t persuade them, sometimes you just have to defeat ’em . . . and that’s sort of the worst of politics.” For researchers and policymakers to take each other seriously requires a willingness to converse across these conflicting cultures (Ward, 2008). These conversations must occur with a certain degree of humility and openness to the contributions each can bring to the policy table. With increased exposure, policymakers come to appreciate the approachability of researchers and the practical value of research in their decisions (K. Bogenschneider, 2020a). With communication and contact, researchers gradually come to recognize the wisdom policymakers bring to the table such as knowing what is possible to accomplish, and knowing when to act as a trustee that makes the best decision on the available evidence and when to vote the views of constituents (Fenno, 1978; Rosenthal, 2009; Smith, 1991). Those who immerse themselves in the policy culture on an ongoing basis become less cynical about the process. Mary Fairchild worked closely with legislative institutions in several states in her job as a former senior fellow at the National Conference of State Legislatures: The legislative institution is . . . a place where we can have civilized debate, and we want to bring in different views and have those discussions and again, in some kind of civilized way, and that is really how public policy is made, by the 72

U nderstanding the P olicy P rocess

voicing of all these divergent views in the public body. . . . It’s always been quite surprising to me, frankly, that people are so cynical about the process. Without the understanding that comes from exposure, it might appear that a more efficient way to make science-based decisions might be turning policy decisions over to scientists. If truth were told, “many scientists are naïfs when it comes to policy and law” (Pinker, 2018, p. 390). Of course, the real issue here is not which priesthood should be given greater power, but an acknowledgment and appreciation that collective decisions can be made more wisely if each one contributed (Pinker, 2018). Henig (2008, p. 218) compares research and politics to oil and water and then provocatively asks, if they were “poured into the same beaker, do they necessarily separate into distinct layers? If so, is it always politics that sits on top?” The answer, according to the state legislator vignette, is that politics does not always sit on top. There are several spaces and places where research appeared to flow more freely such as nonemotional or technical issues, and those for which the policymaker had no established position or bipartisan consensus had been reached (Bogenschneider & Bogenschneider, 2020). In these instances, research does not seem to set off the “tripwire of partisan engagement” (Henig, 2008, p. 219). Observation 3: Error-Free Decisions Are Expected Despite the Pace and Pressures Another cultural axis that divides the research and policy worlds is how the rules of engagement differ in the research enterprise versus the political enterprise. In the hothouse environment of politics, decisions must be made with haste amidst a constant concern that uncertainty or contingency can be exploited. According to Henig (2008, p. 219), Speed of response is . . . of the essence. Deliberation is a potentially fatal weakness if while your side studies, the other acts. Good science depends on these very things: uncertainty and contingency and taking one’s time to draw conclusions. Making progress in research or theory is thought of as a timeless pursuit that can take years, or even a career, to achieve (Gallagher, 1990; Shonkoff, 2000). Care must be taken at each step of the scientific process—methodically testing theories, replicating experiments, and vetting results with each step subject to critical scrutiny and endless debate. Policymakers often see their world as operating at hyper-speed interspersed with periods where little appears to happen. When crunch time comes, a kaleidoscope of issues and crises can pass before their eyes on a nonstop basis, each vying for an immediate response. Levin, who has worked as a professor and a Canadian deputy minister, described the pressures on policymakers’ time as “enormous and ceaseless”: “Politicians are constantly bombarded with requests or demands to do things, stop doing things, increase funding, decrease funding, pass legislation, repeal other legislation, and so on” (Levin, 2003, p. 11). In our studies, policymakers described the pace of legislative activity in concrete terms. An Indiana legislator described it as “full throttle, pedal to the metal” and a Wisconsin legislator described it as a “marathon”, particularly during the end of session when they can deal with 150 bills a day. A senator relayed that, on one day, he testified on seven different bills in three different committees. And when he was back home in his district, there was hardly an evening that he would not need to attend five different meetings in five different locations. In our study of researchers who had effectively engaged 73

P roblem , P layers , and P olicy P rocess

policymakers, one professor reflected that he could recall working this long and hard only when he was working on his PhD and trying to get tenure. Paradoxically, the pace of policymaking can be both fast and slow. According to Levin, “Sometimes decisions were made too quickly, without adequate consideration, while at other times choices that seemed obvious were endlessly delayed” (2005, p. 6). Laws often must be made within specified legislative deadlines and session time frames, so lawmakers can find themselves forced to make decisions quickly even when data are insufficient and analyses incomplete (Chandler, 2006). One prominent example is the origins of Head Start, which was first implemented with no evidence that it would work on a national scale. Developmental psychologist Edward Zigler believed that he had won a small victory when he convinced Sargent Shriver, chief strategist for the War on Poverty, that Head Start should begin as a small pilot program. But President Lyndon Johnson, “a true Texan with Texas-size ideas”, demanded a program on a grand scale (Zigler & Styfco, 2002, p. 6). Accordingly, Head Start opened its doors to one-half million children in the summer of 1965, only a couple of months after completion of the planning. Yet other big, bold ideas may take, not a year or two, but many years to become law. The pace also can be uneven, occurring in fits and starts. Gary Sandefur, a sociologist and now Provost at Oklahoma State University, explained that policymakers may want to move quickly to draft legislation or make a public statement about something, but once this happens everything may grind to a halt: It’s like they hurry up to do something and then they may just drop it after that and move on to something else, so that can be frustrating. So, their speed tends to be [fast] early on and then . . . you don’t always know what’s going to happen after that in terms of actually developing a policy or trying to . . . draft legislation. . . . It can be a very slow process. The demands that a policymaker faces can be physically taxing as described by one Indiana legislator as he repeated the advice he had received from another member: I remember when I was campaigning and a Senator . . . made the strangest comment to me. . . . When you get elected, remember this—always take time to eat dinner. I thought, “What does that mean?” . . . I drove commercial trucks for a living, and I would take 30 minutes for lunch . . . doesn’t happen here. I’ll grab a sandwich and I’ll eat it in front of everybody at a committee meeting. You’ll eat it on the Senate floor [laughter], you eat there, you eat back here, you eat on the run. . . . I remember those words and now I understand what he was talking about. In addition to being physically taxing, these time demands can take an emotional toll, according to a professor who also had worked in the policy world: Politicians spend a lot of time listening to people, whether colleagues, opponents, constituents, or lobbyists, all of whom feel very strongly about things. They hear horror stories, sad tales, angry complaints. Sometimes they are saddened by what has happened to someone, something they cannot do anything about. Sometimes they are infuriated by someone’s stupidity. Often their feelings are stirred. Political discussion and debate, whether in the legislature or in any other setting, runs hot and gets people worked up. When you add to all that 74

U nderstanding the P olicy P rocess

an unending work schedule and the sense that there is much more to do than can possibly be done, it is not hard to understand why feelings run high. (Levin, 2005, pp. 33–34) In sum, the demands are relentless. Policymakers are forced to make important decisions under short timelines and without major errors—a next-to-impossible standard to which the public holds its elected leaders. Faced with so little down time, overly crowded schedules, and ceaseless demands for their time and attention, policymakers can quickly burn out and find they have too little energy and too few new ideas to carry out the agenda that inspired them to run for office in the first place (Levin, 2005). Observation 4: Policymaking Functions on Personal Relationships The collaborative nature of the culture in which policy decisions are made is quite different than much of the research environment. Whether documented by science or observed in practice, the human element is the primary driver of policymaking, which is embedded in a social ecology of relationships (Oliver, Innvaer et al., 2014; Tseng, 2012; Vincent, 1990). In the research culture, scientists can achieve fame by producing a good idea from investigations confined to their own lab. In fact, it may be advantageous to separate oneself from peers and stake out new ideas. The solitary genius is an iconic image in the academy. In the policy culture, coming up with a good idea is the least effective way to be effective. Without the support of interest groups and colleagues, a legislator explained that “way more infrastructure” is needed to move an idea forward, no matter how good it is (Bogenschneider, Corbett et al., 2019). In policymaking, collaborating with others is imperative (Vincent, 1990). As put by a legislator, “You, as a majority of one, you’re just not going to get much done.” A common theme in our study of state legislators is that “relationships are the biggest thing”. As explained by one legislator, being effective depended on one’s ability to build trusting relationships: I can write policy . . . but if I don’t show people back home that I know what I’m talking about or that I feel what they feel, no policy’s going to matter. Or if I don’t show legislators here that I value their input or opinion on a piece of legislation, it’s not going to matter because they’re not going to vote for it or . . . show any support for it. So, policy is the easy stuff. Given its importance, legislators described the serious commitment they make to developing collaborative relationships. In one of Karen’s interviews, a legislator pulled a cheat sheet out of his desk drawer—colored photos of the 18 freshman legislators in the last session and the 25 in the current session. During the interview, he also held up a checklist that he had developed with contact information for every member of the Assembly. He had recently introduced a bill and had diligently recorded when he made contact with every member and how each had responded. The available literature corroborates our finding that a fuller understanding of the policy process requires going beyond the substance and the politics of issues. Much of what is seminal in the policy arena is, indeed, personal (Rosenthal, 2009). By its very nature, policymaking requires that lawmakers call on colleagues to provide cues for how to vote on the hundreds of bills that come before them, to request support for the bills they introduce, and to sign on to support bills sponsored by colleagues (Bogenschneider & 75

P roblem , P layers , and P olicy P rocess

Bogenschneider, 2020; Krehbiel, 1992; Matthews  & Stimson, 1975). This legislative reciprocity is so commonplace that it is said to be almost a universal norm in lawmaking bodies (Rosenthal, 2009). The reciprocal nature of policymaking may surprise policy outsiders, but not those who have been involved in the process. Take, for instance, the observation of a community psychologist who worked as a Senate staffer on Capitol Hill: Unlike the stereotypes of insensitive, impersonal politicians who are out of touch with the people, the policymaking process actually involves a creative integration of the personal and the political and is very much infused with the human element at all levels of interaction and decision-making. (Vincent, 1990, p. 61) This integration of the personal element into the decision-making process does not mean that careful attention is not paid to the substance of policy or that legislation is not thoughtfully developed. To the contrary, legislation is negotiated as part of a network of collaborative relationships that ensure exposure to diverse considerations and to meeting a range of needs and goals (Vincent, 1990). To those who desire to engage the policy world, the need to make “human connections” rests with the scientist (Day, MacDermid Wadsworth et al., 2019; Friese & Bogenschneider, 2009; Vincent, 1990, p. 63). This important insight was first reported in the research utilization literature about 40 years ago when “people” were prioritized over “products” as the preferred vehicles for communicating research to policymakers (Haynes et al., 2011). Since that time, studies continue to show that “people are more persuasive than papers” (Haynes et  al., 2011, p.  587). Accordingly, the influence of research increases when researchers and policymakers have relationships of trust and mutual respect (Gormley, 2011). Moreover, when research is provided in forums or face-to-face consultation, this allows the researcher to participate in crafting the policy argument (Haynes et al., 2011). In policymaking, the potency of relationships was evidenced in the state legislator vignette by the group-oriented thinking of a Republican who responded to a request from a member of the minority party, even though he didn’t have to. Another state legislator provided a telling account of how a good idea is of little value unless and until it is shared by others. Observation 5: A Full-Time Opposition Guarantees Gripping Dynamics Another critical difference from the way the research culture operates is the oppositional nature of the environment in which policy decisions are made. In few institutions are people elected to their position, which of course is an indispensable component of democracies where members are elected by the people to conduct business for the people. Conducting business for the people in a lawmaking body entails collaborating with colleagues in the opposing party who will be vying for your seat in the next election. Imagine working in an environment where some of your colleagues are working hard “to make you look bad” (Levin, 2003, p. 15). A sitting state representative explained the theatrical dynamics set up by this set of conflicting demands: It’s hard to be in the building and working together, and out of the building trying to figure out how to get rid of the person that you just worked with. There are few other comparable jobs where that is the case. 76

U nderstanding the P olicy P rocess

The work environment of researchers is starkly different. Researchers can be awarded the job security of tenure and also are accorded a great deal of latitude in their work. Researchers essentially function in institutions with relatively little bureaucracy and operate much like academic entrepreneurs, who are rewarded for creative independent thinking. In contrast, each member of a policymaking body holds a seat that is a prize the opposition would like to claim in the next election. So, the opposition takes it upon itself to watch members closely and to meticulously point out any mistakes or misjudgments (Levin, 2005). Because each side believes that their point of view is surely right and that of the opposition is certainly wrong, these criticisms sometimes can be inaccurate, misleading, or downright unscrupulous. Imagine the disillusion this constant barrage of criticism creates among the public and the distrust that it engenders in the body. A state senator explained that when she is working with someone from the (loyal) opposition, she thinks about whether they are “trying to take me out”: I could be talking to a colleague who I think has my best interest at heart, but in reality they’re probably trying to find my replacement. . . . I’m not going to make you an enemy. I’m not going to make you my best friend. But we can still do good work on issues that we both care about. Over time, she has come to terms with working in an oppositional environment by developing a motto to guide her interactions: “There are no personal friends, no personal enemies, just permanent interests.” Policymakers suggested five ways that the oppositional nature of their culture guarantees gripping, sometimes dramatic, dynamics. First, policymakers do not have tenure nor any assurance they will be around when the next legislative cycle rolls around, so the system rewards them for being reactive and responding quickly to emerging issues. A  Wisconsin representative, who was elected to a two-year term, explained the time pressure that she feels: I keep telling myself that every time I get reelected, I have a one-year window. I have to work as hard as I can during that one year because it’s not a whole lot of time. I don’t know how long I’m going to be here. And I don’t want to leave this seat thinking that I didn’t do anything for Milwaukee. I want to leave this seat thinking, “Maybe I wasn’t able to accomplish everything that I wanted or as much as I wanted but I did something.” Social scientists have made similar observations. According to a professor who worked to bring data to policymakers for 20 years (Kaufman, 1993), the surest way a policymaker can jeopardize reelection is by failing to act; even a misguided response can be remedied in the next legislative session if the policymaker is returned to office. Second, the tenuous nature of their jobs also limits policymakers’ willingness to push too hard on issues that lack popular support. After all, it is public support that puts elected officials into office, and public opinion that decides which policies merit attention. Policymakers will occasionally take unpopular positions, but “staying elected depends on enough people believing that you are doing enough of the right things” (Levin, 2005, p. 18). A Democratic senator explained that if you can’t get the majority to go along with you on a bill that you are pushing, you can end up alienating people. If your colleagues don’t buy into the idea, you end up “beating your head against the wall and hurting 77

P roblem , P layers , and P olicy P rocess

yourself”. A former legislator had mentored her about the importance of backing away from provisions with too little political support: “It’s not perfect, but we gotta live to fight another day.” In districts like hers that aren’t “safe seats”, she explained that pushing too hard on a bill could hurt you in the long run: “Maybe you didn’t get the whole pie, but a piece is better than nothing at all. You can go back later and try to get more. And you still will have friends in place to help you.” Third, in the face of a full-time and attentive opposition, policymakers find themselves in the unenviable position of having to focus, not only on the long-term goals that motivated them to run for office in the first place, but also on avoiding missteps on minor issues that could wind up turning into major problems (Bogenschneider, Day, & Parrott, 2019). One misstep that the opposition gravitates toward is “perks”, especially those that are simple to track and easy for the public to understand, such as the number of gifts accepted, per diems claimed, or trips taken. Even the most egregious violations may not be large in their fiscal impact (Levin, 2005), but they could loom large in the public eye by raising questions of character and the dreaded criticism of being labelled elitist or out-of-touch with the average voter. Unfortunately, this sets up a dynamic of spending time and effort on avoiding small mistakes, which can drain energy away from the more important work of contemplating and crafting solutions to the pressing problems of the day. Fourth, because policymakers operate under the perennial cloud of reelection, legislators are forced to consider how they could explain any vote they take to their constituents and political supporters. The importance of these explanations is one of the key findings of a landmark study of members of the U.S. House of Representatives (Fenno, 1978). The rationale that policymakers use to explain their votes may be as important as the vote itself: “Members believe that it would not take too many unacceptable explanations to cost them dearly at the polls” (Fenno, 1978, p. 342). The clincher is that policymakers have to give attention to the rationale for each vote they cast, never knowing which may be the issue that comes back to bite them in their next campaign (Bogenschneider, Day, & Parrott, 2019). Fifth, as important as how policymakers explain their votes is whether their positions have been consistent over time. The public values a consistent voting record, perhaps unrealistically so. Consistency carries with it an aura of a person who is informed and resolute, whereas inconsistency can be cast by opponents as being unprepared or showing poor judgment, which are definite liabilities for those elected to make decisions (Weiss, 1989). The importance of consistent views and voting records has become even more important in the advent of social media, cable news, and hand-held phones that can record what happens almost anywhere. Even before lawmakers are sworn into office, they are encouraged to show resolve in their opinions because any evidence of inconsistency can easily be used against them in the next election. Why is oppositional politics the sine qua non of democratic decision-making? In an interview with a Democratic representative, Karen explained that she was writing a book on how to infuse research into policymaking. Without missing a beat, he quipped, “Do you think you can?” He went on to explain that we were examining the wrong question. For anyone serious about making policy change, the real power in a democracy lies with elections. “Policymakers get and keep their jobs, only if elected.” In three sentences, he put his finger on why the oppositional nature of policymaking is here to stay. Despite its oppositional nature, legislators reported acting in ways that seem counterintuitive to those looking at the policy culture from the outside. In our recent study, we heard repeatedly that, in politics, it’s important never to take things personally: 78

U nderstanding the P olicy P rocess

You can disagree and dislike it, but this is not personal. We have disagreements in caucus. We have them on the floor. We have them in committee. . . . But you know what? At the end of the day, when you rap the gavel and the session is over, we go to dinner at night, and sit and talk about my beloved New Orleans Saints or Chicago White Sox or Ole Miss Rebels. Given the unpredictability of politics and the uncertainty of elections, policymakers appear to be living in the moment. Yet legislators frequently described engaging in longterm thinking. As one example, a Wisconsin legislator explained that even though he now enjoys a leadership position in the majority party, that does not prevent him from simultaneously looking to a future scenario: I know the pendulum is here today. . . . In Wisconsin, we could have Republican control for the next 20 years. . . . But, if and when that comes back this way, how I treat people today, that’s how they’re going to treat me when that pendulum swings the other direction. Observation 6: Policymaking Is Fluid and Unpredictable Another point of friction that creates misunderstandings about the policy process is how researchers and policymakers react to events and circumstances that are fluid and unpredictable. The research culture operates under a long-standing tradition of reflexivity, whereas the policy culture rewards reactivity. Researchers are trained to be cautious, tentative, and skeptical, and they perform their work in an environment so patient and forgiving that it recognizes progress can take years to achieve. In stark contrast, policymakers are expected to respond quickly to changing circumstances, emerging crises, and unexpected events in a proactive culture that expects, even demands, progress (or at least action) within days or weeks. Policymakers work in an environment where surprise dominates. In his classic study of Congress, Fenno (1978) found that most issues were predictable, but “it was the possibility of the unexpected event that worries members” (p. 16). The policy agenda can be driven by the high probability that some low probability event will take over without warning (Dror, 1986; Levin, 2003). Two recent examples are the coronavirus pandemic that struck with little warning and the death of George Floyd that spurred nationwide protests. Both events immediately soared to the top of policymakers’ agenda, replacing other priorities, no matter their prior importance, their extent of political support, or the amount of time and resources already invested. Former Wisconsin legislator and Harvard-educated lawyer, Rebecca Young, illustrated the whopper effect that a single idiosyncratic event can have: In Milwaukee, there was something called the Intensive Sanctions Program that was set up in the 1990s to substitute for prison time. . . . People would be subject to intensive monitoring while they were out of prison and also able to get certain kinds of training or help with housing, alcohol and drug treatment, and mental health treatment—the kinds of things that would help them succeed in not committing any crimes. However, one of the persons who was on intensive sanctions did commit a murder. And that was the end of the Intensive Sanctions Program . . . even though you have the research to show that . . . [it] was quite successful in reducing recidivism compared to time in prison. 79

P roblem , P layers , and P olicy P rocess

The outsized effect of a single case like this is hard to fathom in the research culture, where inhabitants might greet it with interest but with a clear need for replication with larger sample sizes before scholars would take it seriously. At a Family Impact Seminar discussion, a neuroscientist cautioned a group of Wisconsin policymakers about putting too much stake in a single case or idiosyncratic event. Using the example of tests of a new prescription drug, he explained that if research revealed that it failed for only one person, it would be heralded as a great success. However, social programs appear to be held to a different standard, with one outlier or one deviant case able to derail an entire policy agenda. The death of a single child in the custody of the state, when it becomes a media event, can offset much of the good such programs do. The policy process responds, not only to extraordinary events, but also to recurring contingencies such as changes in administration, shifts in partisan control, the timing of the electoral cycle, the arrival or departure of individuals in key policy roles, or changes in committee functioning. A Wisconsin Democrat explained how changes in the way that legislative committees operate can open the door to the boilerplate legislation produced by national interest groups on both the left and the right: We used to have more organic public policymaking that came out of the committees. . . . It was Wisconsin-based. It was an original. . . . [Now] you get boilerplate-like legislation with all the talking points, with all the information, and with all the research. I feel like committee work has gone by the wayside. That’s where the research and the data and the slow perusing needs to happen.  .  .  . That’s where the grinding should really go on. But if we don’t have enough good people working on organic legislation, there’s more . . . junk research. The policy process also responds to novel events that the media brings to the attention of the public. Even though these novel events often are relatively minor in nature, a segment of the public becomes worried, which is followed by a cascade of additional media coverage. Soon, the issue is at the top of everyone’s mind. This “availability bias”, as it is called by cognitive psychologists, escalates this issue at the expense of others that perhaps should be a higher priority (Kahneman, 2011; Levin, 2003). For a process as fluid as policymaking, even policymakers who know it best find it difficult to explain to outsiders. They often seem to grasp for words to provide a concrete description of a fluid process with no clear transitions from one issue to the next, and one that can halt or even shift direction with little warning. According to one 25-year veteran of the legislature whose office was laden with plaques and trophies of every conceivable shape and size, policymaking cannot be captured in a flow chart because policies do not progress neatly from Point A to Point B. Even after a bill becomes law, it is hard to pinpoint just why it happened as it did. A relative newcomer to the Assembly, just completing her second term, explained the elusive nature of policymaking in quantitative terms: “1 plus 1 don’t always equal 2”; other factors that are hard to quantify, like dark money and political pressure, often come into play. A ten-year veteran of the legislative process from a well-connected political family spoke of the policy process, not as a flow chart or a mathematical equation, but more like an “art form” where every decision requires some sort of compromise. Because the process of compromise is often long and convoluted, a veteran senator explained that “change rarely happens swiftly”. Yet she immediately underscored the unpredictability of the process by explaining that the biggest accomplishment of her 25-year career

80

U nderstanding the P olicy P rocess

was a small legislative change that ironically occurred quite quickly—prohibiting 14- to 17-year-olds from signing themselves out of mental health treatment facilities. The fluidity and unpredictability of the policy process is captured in that old government saying, “That’s final . . . unless it isn’t” (Levin, 2005, p. 16). One of the “great things” about the legislature, according to a key informant, is that “you never lose forever. Even the most hotly contested issues have a way of coming back. . . . If there’s a basis for moving forward, the legislature will often find a way.” Those in the minority may like nothing better than an opportunity to raise the issue again under more favorable circumstances. One such circumstance is when the control of a lawmaking body shifts from one party to another. As explained by a Republican senator: The Democrats jammed this thing through with sex ed in schools where if you’re going to teach it, you had to teach the state’s curriculum. And as soon as we got in power, we flipped that thing—put it back to where it was. So, you want to make sure that there is bipartisan buy-in because otherwise the other side will flip it and it won’t last very long. Any legislation that involves moving from individual actions to collective decisions will be fluid and unpredictable, as illustrated by the welfare reform vignette. First, the legislation was stalled, and then it was fundamentally transformed by the Republican takeover of the U.S. House of Representatives in congressional elections. Observation 7: Policymakers Wrestle With Problems Ranging From the Wee to the Wicked A basic cultural barrier in researchers’ understanding of the policy world is a mismatch between the core technology of the two communities, which manifests in different questions of interest. The research culture produces studies to understand the way that the world works, whereas a key function of the policy community is designing, enacting, and implementing laws to make the world work better. Research questions that are driven by academic disciplines often are isolated from the real-world problems that policymakers are facing (Watts, 2017). To researchers, some of the problems policymakers face seem like alien concerns that are too inconsequential to warrant serious attention or too complex for serious scientific study by any single discipline or methodological approach. The nature of problems that policymakers address can relate to a broad swath of government functions—building roads, bridges, and dams; caring for the aged and disabled; educating its citizens; and keeping the public safe (Lindblom, 1968). With a portfolio so far-reaching, policymakers may find themselves dealing with sweeping legislation that attempts to redress the most perplexing social problems of our times. At the same time, a significant amount of time is taken up with what we have termed wee issues— commonplace, routine, administrative matters. One legislator in our study described tedious statutory details, like comma changes that are “super important”, but so boring that you easily could “start bleeding from the ears”. Often, small pieces of state laws need to be adjusted or changed in one way or the other. For these administrative issues, the information that was forthcoming was often not specific enough or sufficiently targeted to the question at hand. In the words of a Democratic caucus chair, “a lot of times the information that might come in might be too broad or too general and not fit those specific needs.”

81

P roblem , P layers , and P olicy P rocess

This caucus chair also described how he needed research to confront large issues of long-term significance that defy easy or obvious remedies, what we refer to as wicked problems (Corbett, 1991, 1993; Rittel  & Webber, 1973; Sutherland et  al., 2012). He explained the value of research when dramatic shifts in policy direction occurred: We went through the whole welfare reform issue that the governor had put before us. And there were many, many details on how you would implement sort of a dramatic shift in implementing a major social program. At times like that, it was good to have information available. When faced with wicked social problems like welfare reform that entail complexity and uncertainty, the capacity of the academy to respond has proven inadequate in many ways, three of which emerged in our study of researchers who had effectively engaged policymakers (Friese  & Bogenschneider, 2009). For example, one strategy researchers use to make wicked problems more manageable is to minimize and dissect the complexity in ways that are more compatible with the dominant methods of research and evaluation. Granted, such simplification reduces the complexity of the research process, but the wicked problems policymakers confront do not so easily yield to narrow and simplistic solutions, as explained by a former director of the Institute for Research on Poverty: In fact, there is a fundamental problem with a lot of the best research as it becomes applied to public policy. The experimental method, which is what we use to really uncover truth, works best when you can isolate a specific intervention. Then you can test whether that intervention is better than some counterfactual, whether it’s a different kind of intervention or business as usual. What we found in the welfare world . . . is that any particular intervention might work just a little bit . . . if there was any impact at all. Another inherent barrier that the academy bumps up against when addressing wicked policy problems is that they do not fit neatly into the academy’s disciplinary silos and often exceed the sensitivity of the measures and the sophistication of the methods available to social scientists (Corbett, 1991; Shonkoff, 2000). Wicked problems demand cross-disciplinary, out-of-the-box thinking that can be unfamiliar and uncomfortable for researchers trained to advance a narrow field of study one small step at a time. Instead, what is required is a type of holistic and integrative thinking that academics often do not have the time, training, or wherewithal to engage in. In the words of a member of a National Academy of Sciences panel: I think about the world that I’ve operated in, the welfare world. Academics were more comfortable when the issues were largely economic in character. That is, there were a set of principles, economic principles in terms of guarantees and marginal tax rates and labor/leisure trade-offs that lent themselves to empirical analysis and the development of a set of theoretical propositions. But now welfare is largely about changing behaviors, a much more complex endeavor. And what I found is that a lot of economists, in particular, were most willing to spend a lot of time . . . when issues were defined in ways that could be dealt with by their discipline. Now that it’s a much more complex set of propositions involving individual behavioral changes and the way institutions perform and relate to one another, and so forth, they’re not as eager to play in that. 82

U nderstanding the P olicy P rocess

Finally, a legislator pointed to a fundamental mismatch between the kinds of evidence that are valued in the two cultures. In her work as a policymaker, the kind of information that is most valuable to her is what research says about the people in her district. Even if an issue is widely regarded as a wicked social problem for the population at large, elected officials would be remiss if they failed to consider how this issue affects their own constituents. What this interchange made obvious to us is that the gold standard of rigorous research is representative samples that provide generalizable data. This is not the same as the gold standard of policy-relevant research, which is localized data on how a policymaker’s jurisdiction compares to similar jurisdictions, such as how her district compares to other districts in the state or how her state compares to other states in the nation (Kaufman, 1993; Riley, 1997). Policy problems range from the wee to the wicked. Policy issues extend from the common (e.g.,  making comma changes to correct errors in state statutes) to the complex (e.g., considering legislation to change the major social program serving the most vulnerable members of society). Observation 8: Policymaking Is an Incredibly Complex Undertaking The cultural values around the notion of complexity differ between policy and research communities. That is, complexity itself is (gasp) complex. Researchers are excited by complexity because it is associated with sophisticated thinking in a culture that is open to it and imposes no need for final closure (Henig, 2008). For policymakers, thinking too much about complexity confounds decision-making by leading to what legislators termed “analysis paralysis” in a culture in which action is demanded. Yet complexity takes an unexpected twist when it comes to how each community applies its core technology. In designing their studies, researchers tend to reduce the real-world complexities to come up with manageable questions that can be addressed with their preferred methodology. Ferguson (2015) refers to this as the “Hammer Hypothesis”. That is, if the only methodological tool you have is a hammer, every research question begins to look like a nail. In making policy decisions, policymakers cannot discount complexity (even though their preference is to do so). They must factor in all this complexity to come to a decision that will yield no major errors—a next-to-impossible standard against which the public measures its elected leaders (as discussed in Observation 3). Another observation is quite obvious to those familiar with the policy process—making laws is much more difficult than most people ever imagine. In fact, a researcher in our study who worked on a federal balanced budget bill hit upon a counterintuitive insight that policymakers face more complexity in their jobs than do researchers; researchers have the luxury of focusing on “small, very well-defined issues that they can reasonably address with the data they have available. Policymakers, by contrast, have to focus on larger questions that tend to be not nearly as well defined.” Another researcher, with a decade of experience working with lawmakers, concurred that the complexity of producing research pales in contrast to that of producing policy: Researchers are not dealing with social, complex, political systems. Politicians have to use research in a complex world in which to try and use research knowledge in order to make a difference. They have to be attuned with the political opposition. Research by comparison is so simple. We control for a lot of variables and we don’t try to change complex systems and organisms. The unintended negative side effects of legislation and policy are so huge, and the weight 83

P roblem , P layers , and P olicy P rocess

of responsibility is so great for these people. If I botch my research study, it’s going to be buried in a journal. I won’t have shaped the nature of the field with my mistake. If you’re dealing, for example, with welfare reform and family support, nobody knows yet how this will play out. You can do a lot of damage. An emeritus scholar with 30 years of experience agreed that it is not uncommon for academics to underestimate the complexity of the policy process. Members of the academy typically are unaware of all the factors that come into play in any policy deliberation. They are unfamiliar with the dimensions of decisions for which there is no available evidence, and unacquainted with the complexities of taking an idea, often untested, and making it reality. Experts, the man on the street, and everyone in between have a solution for policymakers, according to this experienced scholar, and they wonder why those dummies can’t see the obvious and just do X. But when you get in there and you try to do X, or Y, or Z, not only are you bombarded by competing visions of what the truth is, but the complexity associated with doing anything, changing bureaucratic cultures whatever it may be, is enormous and the specter of unintended consequences is always there. And so you do X, you get it instituted, you make the change, and then you find out that this generation’s solution to a problem is the next generation’s scandal. . . . Sometimes it doesn’t even take a generation. It’ll take a year or two. And people look back and say, “What idiot did that?” because we can’t always anticipate all the blowback and the unintended consequences. . . . This is not about having a vision of truth and going out and applying it. This is about slogging through thorny issues that have enormously complex trade-offs associated with them. The origins of this complexity are many, three of which will suffice for our purposes. First, policymaking by its very nature demands that policymakers be generalists, according to a researcher at a prominent think tank: “[They] have to vote on a large number of issues, so they tend to be broad, but not very deep on most issues.” Typically, policymakers in any one session are required to vote “aye” or “nay” on policies ranging from childcare to health care, from minimum wage to mining, from taxation to transportation. Ironically, not being an expert on each of these issues is one of a policymaker’s strengths: It is often the fate of experts to be unable to see a problem from any perspective but that of their own expertise. It is the special talent of politicians, on the other hand, to be able to see a matter from enough different points of view to provide them with scope for action. (Albaek, 1995, p. 94) Second, policymakers are faced with the formidable task of trying to reconcile visions of truth that are inconsistent with each other and may be inconsistent over time as well. The same constituents who demand lower taxes in one meeting may request more services in the next; later, they may even complain loudly about the consequences of the cuts for which they had previously lobbied (Levin, 2003). Consistency of positions may be a virtue among policymakers, but it is not required of their constituents. Third, the complexity of public policy defies easy and efficient solutions to perplexing problems, even when guidance from research is forthcoming. Research is much better at documenting “what is”, than in making predictions about what “ought to be” (Seeley, 84

U nderstanding the P olicy P rocess

1985) or what “might be”. Policymakers, however, are called upon to enact a response, which often is untested and may result in unintended, collateral consequences that, like the proverbial Gordian knot, could feed back upon itself. Despite the best of intentions, some policy decisions will later be found to have been mistaken and then policymaking becomes a process of remedying the errors of past decisions (Lindblom, 1968). Policymakers and researchers have responded by developing strategies for confronting and coping with complexity. One experienced legislator relayed her experience introducing bills that ended up causing problems that she had not anticipated, prompting her to rethink the processes that she employed to design legislation. To gain greater assurance that her legislation will bring about the intended change, she now brings to the table those who will be affected by the bill when it is being designed and gives them this charge: “You are the practitioner. You will have to work with the legislation. Help me design it.” I think that I am a better legislator because I do it this way. I think this is one of the most important things that legislators can do. The complexity of the policy process is illustrated by the welfare reform vignette as would-be reformers weighed options ranging from job search to enhancing human capital to remediating work impediments, all in the context of evaluation data that suggested no single approach worked very well. After a decade of working closely with policymakers, one researcher has gradually come to appreciate how complex policymaking is and has “become more sympathetic to slowness in the process and the need for pilot and demonstration projects to see if the policy works”. Observation 9: Values Should Not Be Underestimated as Policy Drivers Values are a fundamental element of the cultural clash between researchers and policymakers. Granted, everyone has a set of values and priors that they take into any debate or line of inquiry. So, the crux of the cultural clash centers on the primacy of values in decision-making—being a criterion of prime importance in the policy culture and of subprime importance in the research culture. Researchers attempt to keep their decisions as value-free as possible by relying on rigorous research methods and predetermined probability levels, whereas policymakers privilege their values and the values of their constituents in their decisions. A clash around an issue as fundamental as using values as a criterion for decisionmaking can impede communication and cooperation between the two cultures. According to a 30-year veteran of the policy process from a land-grant university: Policymakers, particularly if they’re from the political end of the policymaking process, are really driven and defined by their values, whereas researchers or academics are supposed to be relatively value-free except for the set of values that go into the scientific inquiry per se. So, I think that the fact that one tradition or culture has values of prime importance and the other one tries to at least hide their values also makes it a very difficult exchange. So, as researchers or policy analysts bring their evidence to policymakers, they’re often taken aback when that evidence is filtered through a set of very distinct . . . priors that sort of shape how the evidence is looked at and used, and sometimes even distorted. When values dominate science, those socialized in the research culture can respond with “a bit of arrogance”, according to one professor with stellar credentials in the 85

P roblem , P layers , and P olicy P rocess

world of research and 25 years of experience working with policymakers. In her work on politically charged topics like teenage pregnancy, she learned to reconcile the role values played in her work with the role they played in the work of the policymakers she hopes will use her research. Here’s how she does it: One has to accept that values are values. We all have them, but as a researcher, my values have no place in my research. But policymakers’ values count and their constituents’ values count. So, this researcher engages in the delicate balancing act of resisting the intrusion of bias into her research study, while still responding to policymakers’ values. She explained how she is intentional in presenting her research findings in ways that are respectful of diverse values and perspectives: If I went out and made a big announcement, present findings, and don’t attend to the fact that a section of the population will object based on value-driven reasons, you won’t get as far. Researchers are pretty ignorant in the political space and don’t understand where people are coming from. . . . Now I ask myself, what are the different factions? Where are they coming from? I try to get into their heads. Researchers occasionally bristle at the contention that research can be fully value-free given their discretion over the questions to ask, the methods to use, and the analysis to employ. Obviously, subjectivity can be introduced into each of these and other decision points. Yet researchers still resist reliance on values, which perhaps is best illustrated when they propose a hypothesis and then reject it based on data and analysis. This goes a long way toward saying, “Yes, I have values, but my job is to explain what science says.” Values often surface when policy goals are being established. For example, welfare reformers faced the dilemma of balancing two important, yet contradictory, value-based goals: reducing welfare dependency and reducing child poverty. For example, dependency could be eliminated by ending welfare benefits, but that might well increase the number of children living in poverty. Alternatively, child poverty could be eliminated by increasing the generosity of welfare benefits, but that comes at the risk of increasing dependency on government assistance (Corbett, 1993). Deciding which of these goals is more important often entails a judgment that rests on values, because research is seldom comprehensive and definitive enough to serve as the sole arbiter. Veteran policy educator Barry Flinchbaugh cautioned researchers about making these types of value judgments: “Just because you have studied the issue, just because you have a degree behind your name, and just because you may be the ‘expert,’ does not make your values superior to anyone else’s” (1988, p. 25). Knowledge brokers, when communicating research to policymakers, are often asked to make policy recommendations, as described by Extension Agricultural Economist Richard Barrows. He conducted policy education programs across the state of Wisconsin when the legislature was considering an innovative and ultimately successful farmland preservation program. In his sessions, he discussed who was advantaged and disadvantaged by the proposed legislation but was firm in resisting pressure from both state legislators and his supervisors to provide his own opinion. He was fully aware that the decision could not be made based on science alone: Scientific knowledge, the wisdom of the university, cannot be used to determine the “correct” policy choice for society because science cannot supply the value 86

U nderstanding the P olicy P rocess

judgment that rank the interests of one group as more important than the interests of another. (Barrows, 1994, p. 3) Policymakers, on the other hand, take the charge to represent the values and views of their constituents very seriously, despite the enormous complexities of doing so. Most policymakers represent a sizeable constituency, so the value stances of constituents often conflict with each other. It falls to policymakers to sort out and to negotiate these varying value orientations. The enormity of this task is another reason that policymaking is one of the most complex undertakings known to humankind (see discussion in Observation 8). Observation 10: Policymaking Favors the Status Quo Differences in the gravitational pull of the research and policy cultures is another fundamental difference that can muddy researchers’ understanding of the policy process. In the research culture, the gravitational pull is toward providing new and transformative ideas, whereas the strong pull in the policy culture is toward maintaining the status quo. One tenet of the research culture is the scientific worthiness of new and innovative ideas and the scientific unworthiness of replicating an existing idea or approach, which some journals even refuse to publish (Ferguson, 2015). In the academy, new knowledge builds on the past, but honor and prestige come from unearthing a new idea or upending conventional wisdom (Henig, 2008). In contrast, in the policy culture, there is a strong gravitational pull in the opposite direction toward maintaining the status quo. One of the legislators in Karen’s study put it this way: “You can change when you need a change . . . but change for change’s sake most of the time is not necessarily a good thing.” The predominant institutional pressures are for preserving existing programs and policies, many of which become entrenched in the bureaucracy amidst a throng of vested interests that will defend them to the death (Lynn, 1978). One legislator explained the unrelenting pressure that he receives from interested parties: “The status quo is good for them. They fear change. They fear disruption. They fear additional expense. And they just fear the unknown.” Several legislators spontaneously mentioned this preoccupation with the status quo. One legislator explained it this way: These programs end up piling on top of each other. Legislators have to make decisions about what programs to cut and what to expand. It is hard to say “no” to existing programs, even when there is a better alternative. Despite most social policies abiding by the gravitational pull of the status quo or incremental change (Baumgartner  & Jones, 2009), some qualitatively big jumps do occur. Milton Friedman (1982) is credited with saying that: “Only a crisis—actual or perceived—produces real change. When that crisis occurs, the actions that are taken depend on the ideas that are lying around.” As an example, David Riley cites the birth of Head Start that was precipitated, in part, by the crisis created by “Sputnik”, which underscored the need to train a competent, new generation of scientists. The Spanish Flu at the end of World War I had profound effects on medical research and the practice of medicine. The future will reveal the changes that will emerge from the pandemic known as COVID-19. The conditions required for dramatic policy shifts can be likened to those 87

P roblem , P layers , and P olicy P rocess

of a perfect storm—innovative ideas lying around that are picked up by skilled champions who shepherd them through the policy process in the wake of a crisis and public outcry for political response.

Conclusion Lawmaking bodies have been called a “policy development cauldron” (Jonas, 1999, p. 8). Cauldron seems an appropriate descriptor for an ill-defined process composed of a mix of ingredients so paradoxical that it disavows a rational scientific basis for the outcome. Policymaking can leap forward at lightning speed or saunter along at a snail’s pace. The resulting decisions can be fueled by the political calculus of a full-time opposition or inspired by the consensus policymakers are able to achieve by leveraging trusted relationships. No matter the impetus, decisions are expected to be made with haste and without error. Policy problems range from the wee to the wicked, and policy issues range from the commonplace to the complex. Decisions must weigh what facts say are real with what values say are ideal, while resisting the gravitational forces that relentlessly pull policy back to the status quo. With so many convening and contravening forces, it is no wonder that there is disdain for the policy process and disparagement of those involved in it. In large part, Levin (2005) defends government but also acknowledges that government is an imperfect institution: [Government] may fail to do what is important or right and may actually do things that are bad and dangerous. They are sometimes driven by the pursuit of wrong ends: power above purpose, the short-term over the long-term, the interests of the powerful instead of the interests of the majority. Image becomes more important than substance, and personal agendas may dominate public ones. Truth may become a casualty of political convenience. (pp. 6–7) In keeping with Levin, we see the imperfections, but we also see the potential for policymakers to come together for the common good. We argue here that the built-in tensions between the culture of the research and policy communities, which are “endemic” need not be “debilitating” (Henig, 2008, p. 221). We see the potential for research to be used in multiple decision points for a myriad of purposes throughout the process, such as accommodation to opposing views, coalition building, persuasion, and rhetoric (Albaek, 1995). We see the potential for researchers to master an understanding of the functioning, complexity, and incentives of the policy culture, not with minimal or superficial exposure to policymakers, but with one-on-one engagement with them (Chua, 2018). By no means have we done a comprehensive or exhaustive examination of the policy process here, but our hope is that it will serve to motivate the reader to become more attuned to the vagaries of that fascinating world—an endeavor that we think is important for a couple reasons. First, we agree with Levin’s assessment that without a good grasp of how government works, it will “lose its legitimacy in the eyes of citizens” (2005, p.  7). Troubling trends have emerged in recent polls that reveal only a quarter of the U.S. public has a “fair amount” or “great deal” of confidence in its elected officials (Pew Research Center, 2018). If professionals and the public better understood the inner workings of the policy process, is it possible that they would come to see it as more rational, collaborative, and committed to the public good? 88

U nderstanding the P olicy P rocess

Second, we believe it is fundamental to develop a deeper and more nuanced understanding of the lawmaking process because policymakers do not operate in isolation, but within an institutional culture that sets powerful constraints on what they can and cannot do (Weiss, 1999). We do not contend that policymaking can be reduced to a set of textbook processes that easily can be committed to memory and repeated chapter and verse. However, bringing into sharp relief the basic assumptions of the policy culture and the ways they conflict with those of the research culture can help researchers cross the cultural divide with more confidence and success. For example, the fact that policymaking is fluid means that successful knowledge brokers must be diligent in determining which issues are on or off the political agenda. Understanding the relationship-based culture of policymaking can prompt researchers to take the initiative to establish personal contacts and interact in face-to-face settings. Awareness of how values drive the policy process helps knowledge producers know how to relay the importance of a study and be more realistic about what research can contribute. Finally, though the policy process can seem arcane and obtuse from the outside, its secrets can be mastered with modest effort. Remember that the academy also looks mysterious to outsiders with its seemingly secret language and obscure patterns of communication. In the end, all communities and cultures have a common set of dimensions which, once identified and appreciated, permit us to look behind the curtain to see how things really work. Let us end this part where we began—with a quest for understanding. In the words of C. P. Snow, understanding how the policy world ticks is the first step toward making it tick better.

89

Section 2 ENVISIONING A NEW ERA OF THEORY, RESEARCH, AND EVALUATION

We initiate our search for a more complete theory of the research-policy divide by addressing four questions: First, we explore whether policymakers even want research and, if so, for which purposes (Chapter  5). Conventional typologies for categorizing research use have been useful, but lack the depth and clarity that we think is needed to move our conceptual thinking forward. Second, what is the best way to configure the communities that make up the research and policy worlds (Chapter 6)? Earlier theoretical thinking posited large community groupings that we feel obscured nuanced thinking about critical target populations. Third, how can we better understand the nature of the disconnects across the communities, so that we can do something about them (Chapter  7)? Specifically, we spend much time examining the elementary dimensions of institutional culture, along with the contributing influences of professional culture. It is along these dimensions that friction points arise that result in communication and relationship breakdowns or what we term community dissonance. Finally, we also see another role for theory beyond explaining why policymakers use research and why it is so hard for researchers and policymakers to communicate with each other. We envision a theoretical framework that informs efforts to communicate research to policymakers (Chapter 8). We propose a theory of change that we developed in our decades of work with the Family Impact Seminars that we hope can guide evaluation of the effectiveness of communication efforts and identify ways they might be improved.

5 HOW POLICYMAKERS SAY THEY USE RESEARCH A Fresh Theoretical Framework Karen Bogenschneider and Thomas J. Corbett

To reach the endpoint of evidence-based policy, this chapter  circles back to where it starts: Do policymakers really care about using research evidence to govern more effectively? Drawing on an unusual study of legislators, this chapter reveals six ways that policymakers report using research that have received little attention in prominent ­theories—providing a larger context for thinking about issues; earning the trust of colleagues; educating others; asking important questions; enhancing debate, dialogue, collaboration, and compromise; and persuading others. Building on these findings, a new theoretical framework proposes that policymakers use research for achieving policy ends, and also for achieving more immediate means to accomplish their longterm ends. What we call the beginning is often the end. To make an end is to make a beginning. The end is where we start from. T. S. Eliot (1943, p. 46) All around the world, studies have shown that research which once seemed to be relegated to the dustbin was actually influential over the long term. . . . How could it be? . . . The answer is that social scientists were looking for the wrong kinds of influence in the wrong kinds of places. Weiss (1999, p. 195) The endpoint we seek is evidence-based policymaking, but where is the starting point? Where do we begin the journey to bring research evidence to bear on policymaking? We start out in this chapter  by thinking about whether policymakers care about research evidence. Are they interested in using research to govern more effectively? In what ways is research used in the policy process? If policymakers see little value of using research in their jobs, our pursuit of evidence-based policymaking may be no more than a pipe dream. The way we think about this core issue of policymakers’ use of research may well shape how we think about everything else. This is the end where we start from. In our work with legislators and agency officials over the decades, we discovered that the receptivity of policymakers to research would surprise most scholars and shake their prevailing pessimistic presumptions about the policy community (Boehnen et al., 1997;

93

N ew E ra of T heory , R esearch , E valuation

Bogenschneider, 2014, 2020; Bogenschneider  & Bogenschneider, 2020; Bogenschneider, Day, & Parrott, 2019). Our observations are not an outlier perspective, however. They are consistent with pioneering researcher Carol Weiss (1999) who disputed that research had been “relegated to the dustbin”, and other experienced policy scholars who are reputed to have found that research can and often does contribute to policymaking (Gormley, 2011; Grisso & Steinberg, 2005; Henig, 2008). The more interesting question is how? We now look more closely at how policymakers say they use research evidence (Bogenschneider, Day, & Parrott, 2019). We draw upon a recent study of state legislators to erect a fresh theoretical framework for conceptualizing the various uses of research in the policy process.

Is Research Relevant to Policymakers? One paradox that the field of evidence-based policymaking faces is this: Why do outsiders think research evidence is irrelevant to policymaking, whereas those with more experience in the policy community often find it more influential than commonly thought (Weiss, 1999)? The persistence of this skepticism about the relevance of research is perplexing, given that science has amassed a respected record of influence in all fields, including politics (Pinker, 2018). Admittedly, in select time periods, on certain policy topics, and during moments of partisan conflict, research has receded in importance or at least retreated into the background. But does this represent reality overall and over the long haul? Or have we become overly cynical, responding more to headlines than substance? Perhaps research use is underappreciated because its use is not as straightforward as a casual observer might suspect. Perhaps its effects are pervasive, yet subtle, and thus, not “easy to see” and scarcely “visible to the naked eye” (Weiss, 1986, p. 217). As suggested by Weiss, those of us who study and practice research use may have been groping around in the dark looking in the wrong places for the wrong purposes for the wrong reasons (Weiss, 1999). To shed light on how research is used by policymakers, this chapter looks through two lenses—the lens of theory and the lens of state lawmakers. Early theorists have touched upon various meanings of research use in policymaking. The most common image, which is often the focus of the evidence-based policy agenda, is instrumental use where policymakers directly apply it to a policy or practice decision (Boaz & Nutley, 2019; Nutley et  al., 2007). We examine here whether it is possible that policymakers use research in ways other than for this straightforward and direct purpose. Does interest in research pique primarily when perplexing policy questions call for designing legislation to achieve intended outcomes? Do policymakers seek research to help form their own opinion or to justify a preexisting position? Or is research seen as a trustworthy source of ammunition to persuade others to one’s point of view? Could it be that research is a political tool for gaining leverage in the policy process when partisan interests collide and policymakers seek points of convergence through negotiation and compromise? We engage in this theoretical exercise because thinking through alternative uses of research evidence tells us a lot about how easy or difficult it might be to communicate science to policymakers and to be heard by them. Whether one is listened to depends on the utility of one’s message to the target audience. Put another way, you have input when you conform more closely to the purposes for which your audience employs science. 94

H ow P olicymakers S ay T hey U se R esearch

To get a better handle on this topic, we explore how research is used in policymaking by asking a sample of state legislators. Granted, state lawmakers are only one window into how research shapes the way that policymakers go about their business. Yet their perspective remains a vital window because empirical evidence on how legislators value research is so remarkably “thin” (Nutley et  al., 2007, p.  2). It has been quite rare in recent decades to tap the perspectives of those who make laws through interviews with lawmakers themselves (Mayhew, 2011) and particularly state legislators (Fox, 2010; Merriman, 2019). Thus, our insights and perspectives from state legislators and key informants who operate in the policy world provide rare input from a population that is underrepresented in scholarship. Granted, their insights are not likely to resolve all our theoretical and empirical quandaries, yet their perspectives will inform us regarding the directions in which we might proceed.

Theoretical Thinking About Research Utilization in Policymaking The mystique of science is its power to predict (Dunn, 1998). Theories move thinking beyond “just a description of how things are” to “an explanation of why they are the way they are” (Pinker, 2018, p. 392). Unlike the certainty found in the hard sciences, theory is more likely to fall short in the social sciences more generally (Dunn, 1998): “We are scientific by intention and effort, but not yet by achievement. We have no elegantly successful theories that predict behaviors precisely in widely different settings” (p. 45). The hard sciences predict the overall movement of gases with considerable accuracy. In the eyes of some, the social sciences too often attempt to predict the movement of individual molecules, or groups of molecules, within the gas. That is a rather daunting ambition in which expectations inevitably disappoint. That has not prevented us from developing theoretical frameworks, though theory is particularly likely to fall short in public policy (Ferguson, 2015) and even more so in research utilization. Explicit theories on the use of research are limited (Nutley et al., 2007), which is reflected in the nature of the models that we consider here. Nutley and colleagues characterize the nature of the prominent theoretical models as examining the interface between researchers and policymakers in four main categories: incremental decision-making, types of research use, two communities of research and policy, and agenda-setting theories (Nutley et al., 2007). We examine one theory in each of these four categories. Each theory has a different focus. Each captures a discrete slice of how research is used in different dimensions of the policy process such as decision-making, agenda setting, and policymaker purposes. Our intent here is not to test these theories, given that each has a specific focus, but rather to provide a conceptual guide for comparing researchers’ perceptions of policymakers’ research use that we can compare to policymakers’ perceptions. Let us briefly become acquainted with each general approach. The Politico-Administrative Decision-Making Model The politico-administrative decision-making model moves beyond classic, rational theories that have been widely refuted as too simplistic and linear. Albaek (1995) shifts away from an analysis of decision-making to an analysis of action. He theorizes research is used in policy discourse as (self) reflecting participants deliberate and debate norms and alternatives with an eye toward concrete action. In this view, the focus of debate in a 95

N ew E ra of T heory , R esearch , E valuation

Key Concepts 5.1.  Four Prominent Theoretical Models of the Researcher/Policymaker Interface 1. The Politico-Administrative Decision-Making Model (Albaek, 1995) •

The process of making decisions through debate and deliberation of norms and alternatives to come up with a practical plan of action

2. The Policy Agenda-Setting Theory or Multiple Streams Theory (Kingdon, 2011) •

The three streams of problems, policies, and politics that position issues on the decision-making agenda

3. The Conventional Typology of Research Use (Nutley et  al., 2007; Tseng, 2012; Weiss, 1999) • • • • •

Instrumental use Conceptual use Tactical use Imposed use Process use

4. Community Dissonance Theory (Bogenschneider & Corbett, 2010; Bogenschneider, Corbett et al., 2019) • • • • • •

Allocation shifting Tactics shifting Solutions shifting Salience shifting Framework shifting Awareness shifting

democracy is not on finding the scientifically best policy alternative, but rather on debating alternatives in a spirit of negotiation and compromise to come up with a practical plan to ensure that something gets done. The emphasis is on how the process shapes the product. Kingdon’s Policy Agenda-Setting Theory or Multiple Streams Theory Kingdon (2011) focuses on how policy issues rise and fall in the dynamic and chaotic policymaking process in his policy agenda-setting theory or multiple streams theory. Moving beyond previous thinking that has focused disproportionately on how policy decisions are made, he focuses on why or how an issue gains political traction (Kingdon, 2011; Nelson, 1984). Kingdon (2011) emphasizes the nonlinear nature of the policymaking process and posits that three independent “streams”—problems, policies, and politics—must come together to open a “window of opportunity” for policy change to occur. No single stream is likely to place an item on the decision-making agenda.

96

H ow P olicymakers S ay T hey U se R esearch

For example, a problem without an acceptable solution, a solution without a compelling problem, and problems or solutions that are not politically or economically feasible quickly result in the closing of a policy window. This theory conceptualizes how the setting of the policy agenda may be influenced by the role that research plays in recognizing problems, identifying available policy responses, and determining the receptivity of the political and economic climate. His thinking was a vital contribution to how research helps issues gain enough momentum to be seriously considered. The Conventional Typology of Research Use The well-known typology of purposes in using research finds its way into the core of virtually every extant theory (Nutley et al., 2007; Tseng, 2012; Weiss, 1999). There are five types of use included in the conventional typology, the first three of which were included in Karen’s study of state legislators. Instrumental Use A use is defined as instrumental when the results of research or other rigorous analyses provide input that directly shapes an identifiable policy decision, such as designing new legislation or changing existing law. This is close to what has been called the hypodermic metaphor for directly injecting research into policy (Pettigrew, 1985). There is a direct application of research collected according to prevailing canons of science and judiciously applied to the policy question at hand. Conceptual Use The slow accretion of evidence that gradually (and not always consciously) shapes ideas or enlightens understanding of problems and solutions is thought of as conceptual use (Farrell  & Coburn, 2016; Yanovitsky  & Weber, 2020b). Conceptual uses of research can contribute to “shifts in intellectual discourses, policy paradigms, and social currents of thoughts” (Nutley et  al., 2007, p.  301). This is close to Weiss’s (1987) enlightenment model or Bulmer’s (1987) “limestone” metaphor. This consciousness-raising role is widely considered the most common on-the-ground way that research is used by policymakers (Nutley et al., 2007; Weiss, 1979). Basically, the weight of evidence slowly shifts the way issues are viewed and how they are responded to. Tactical Use Sometimes termed political or symbolic use, tactical use means using research to persuade colleagues to support one’s view or to justify a decision that already has been staked out. When information conflicts with personal beliefs, policymakers resolve the cognitive dissonance by resorting to the natural human tendency to find research that confirms their beliefs rather than changing what they think (Tseng et al., 2018). Some researchers snub their noses at research being used, not before the fact to reach decisions, but after the fact to justify them. Yet others contend that whenever research is used, it plays an important symbolic role among the citizenry, who sees the purpose of politics in a democracy as coming up with good ideas to solve public problems (Shulock, 1999). A basis in scientific evidence lends gravitas to policy ideas and credibility to the decisions

97

N ew E ra of T heory , R esearch , E valuation

made by policymakers and administrators. When policy decisions can be shown to be evidence-based, voters gain confidence in the capacity of democracy to remedy problems and resolve conflicts (Dionne, 2004). Imposed Use When use is mandated by someone else, it is known as imposed use. In recent years, there has been a frenzy requiring that practitioners use only evidence-based programs. Since 2008, the federal government has invested over $6 billion in a tiered-evidence approach that allocates more funding to programs with more rigorous evaluation evidence (Dumont, 2019; Tseng, 2015). Such mandates raise questions about how the strength of program evidence is assessed and whether the mandates are scientifically substantive or political in nature. With mandated use also comes concerns about whether the same results will be achieved under different local conditions, and whether programmers learn habits of decision-making based on the quality of the science or merely acquiesce to “toeing the line” to satisfy regulatory or financing requirements (Weiss, Murphy-Graham, Petrosino, & Gandhi, 2008, p. 41). Process Use Process use involves moving research use away from one-way transmission that extends from researchers to policymakers, and toward two-way collaborations of researchers and policymakers that drive research at the outset. These research-practice partnerships, as they are known, are defined as mutual, long-term collaborations between researchers and practitioners, typically involving researchers and external agencies, or the research and program offices within a single agency. Ample anecdotal evidence suggests that these partnerships meet their goals such as improving education and human services, and enhancing child and youth outcomes, but no rigorous impact studies exist (Tseng, Easton, & Supplee, 2017). Community Dissonance Theory Our theoretical perspective, community dissonance theory, was initially introduced in 2010 (Bogenschneider & Corbett, 2010a) and revised in 2019 (Bogenschneider, Corbett et al., 2019). To improve research use in policymaking, our basic operating premise is that unless theory deconstructs the forces responsible for dividing researchers and policymakers, effective strategies for bringing them together are not likely to be forthcoming. The 2010 version of the theory posited that better understanding in the research community of the inhabitants, institutions, and cultures of the policy community can build trust and communication and, thus, increase the use of research in policymaking. By taking the perspectives of policymakers, both legislative and executive agency types that we had worked with over the decades, we proposed in 2010 six ways that research is used: 1. Allocation shifting is using research to decide how to distribute scarce resources. 2. Tactics shifting is when research is used to inform the implementation or management decisions of a given policy solution. 3. Solutions shifting is using research after policy outcomes are agreed upon to make decisions among the available policy alternatives. 4. Salience shifting means using research to put new issues on the policy agenda or to alter the queue of existing issues. 5. Framework shifting uses research to alter ways of 98

H ow P olicymakers S ay T hey U se R esearch

thinking about policy issues or choices. 6. Awareness shifting is when research brings to light a new idea or issue that contributes to the understanding of a problem, solution, tactic, or unintended consequence. Given all these different theoretical angles on the connections between researchers and policymakers, what would we find if we tapped the beliefs of a single group, such as state legislators, in much greater detail? Perhaps a more systemic exploration of the insights that reside in a specific policy population could advance our general theoretical thinking about research utilization in policymaking.

Research Contributions From the State Policymaker Perspective The notion of perspective is a seminal factor in thinking about research use. Arguably, research use may vary according to whom you ask. Instead of imposing researchers’ perspective, we turn to one specific set of policymakers, state lawmakers, to examine how they describe research use in their own language and based on their lived experience. Note that we rely here on one of our primary communities of interest—lawmakers—and one specific type of policymaker—state legislators. From the responses we secured from this population of lawmakers, we revisit the conventional theoretical frameworks used to categorize and understand research use. We overview here the findings of a semi-structured qualitative study of 225 state policymakers in Indiana and Wisconsin (see methods in the Appendix). In Round 1, we conducted qualitative, face-to-face interviews of 123 legislators (60  percent response rate), in which we asked them to nominate exemplar colleagues, including those who were exemplar research users. We report here on Round 2 interviews of 32 exemplar research users (84 percent response rate), which averaged 63 minutes in length. When the research exemplars were compared to all legislators in the study, there were no significant differences based on state, chamber, gender, party, or years of service, though the research exemplars had significantly higher levels of education, t(170) = -4.57, p = .00. Round 3 interviews included 13 key informants purposely selected to represent both parties in each state in diverse positions—former governor, lieutenant governor, and gubernatorial staffer; lobbyists; nonpartisan legislative agency analysts; and former legislators (nominated as statesmen and stateswomen by legislative leaders and nonpartisan analysts). Key informants were positioned at the highest levels of their organizations, with an average of 31 years of state policy experience. The research exemplars responded to qualitative and quantitative questions regarding their use of research and the consistency of their experience with ten contributions of research identified in Round 1. The interviews of the research exemplars and key informants were audio recorded and transcribed. Data analysis entailed line-by-line coding of each interview and an iterative process of code verification by the research team (see details in Bogenschneider, Day, & Parrott, 2019). Legislators related multiple ways that they use research, often belying the conventional wisdom that policymakers care little about rigorous evidence. The 32 exemplar research users identified 14 contributions research makes to policymaking that were mentioned 535 times, ranging from six to 102 mentions. The pattern of responses from the legislators proved both illuminating and helped shift our own thinking. As depicted in Table 5.1, the contributions of research most frequently mentioned by the legislators are discussed along with our approximation of their mentions in various theories. Some of what legislators mentioned comports with theory, but not always. 99

Politicoadministrative decision-making model Albaek, 1995

100

X X X X X X

X X X X X

Note: Based on interviews of 32 legislators nominated by their colleagues as exemplar research users (Response rate = 84 percent)

X X

X

X

X

X X

Community Policy agenda dissonance theory setting Bogenschneider & Kingdon, 2011 Corbett, 2010a

X X

X

X

Typology of research use Nutley et al., 2007 Tseng, 2012

No. of No. of Acknowledgment of contributions in four prominent theories of mentions Leg research utilization in policymaking

Research contributions frequently mentioned by legislators, sometimes predicted by theory Persuading others by developing arguments that support 102 29 X one’s view or anticipate and counter opposing views Providing a larger context for thinking about issues and an 72 29 enlightened understanding of the complexity of issues Research contributions frequently mentioned by legislators, frequently predicted by theory Designing good legislation and stopping bad legislation 61 26 X Defining problem/raising awareness of why action is needed 40 21 X Research contributions frequently mentioned by legislators, seldom or never predicted by theory Educating others 49 20 Improving the decision-making process by enhancing 41 20 X debate, dialogue, collaboration, and compromise Earning the trust of colleagues as a knowledgeable and 34 20 credible information source Asking important questions for policy or political purposes 30 13 Research contributions infrequently mentioned by legislators, frequently predicted by theory Informing one’s position on an issue 29 15 X Justifying a preexisting position on an issue 24 14 X Assessing the political and economic feasibility of policy 21 10 X decisions Changing one’s position on an issue 13 10 X Offering new and emerging ideas 13 13 X Explaining a vote on an issue 6 5 X

Research contributions mentioned by policymakers

Table 5.1 Contributions of Research to Policymaking by Number of Mentions of the Contribution by Legislators (Leg), Number of Legislators Who Mentioned the Contribution, and Acknowledgment of These Contributions in Prominent Theories of Research Utilization in Policymaking

N ew E ra of T heory , R esearch , E valuation

H ow P olicymakers S ay T hey U se R esearch

We group the responses as follows: frequently mentioned by legislators, sometimes predicted by theory; frequently mentioned by legislators, frequently predicted by theory; frequently mentioned by legislators, seldom or never predicted by theory; and infrequently mentioned by legislators but frequently predicted by theory. Within each grouping, the contributions are ordered in terms of the frequency with which they are mentioned. Contributions Frequently Mentioned by Legislators, Sometimes Predicted by Theory Persuading Others by Developing Arguments That Support One’s View or Anticipate and Counter Opposing Views This contribution of research was predicted by two theoretical frameworks and had the most mentions from legislators—102 in all. A  key informant, who was a former legislator, was explicit that research use, to him, meant constructing and formulating an argument. In a work culture in which effectiveness depends, in part, on one’s ability to persuade, a Republican legislator explained why research builds better arguments: You can make the argument, but if you don’t have the data to back you up, then nobody’s going to support you on it. And nobody’s going to trust you long-term. Kind of like a scientist, right? A scientist has a hypothesis but doesn’t have any data to support [it]. The use of research for persuasion was shaped by power dynamics in the legislature. Democrats, experiencing minority status in both states, were more likely than Republicans to report using research to counter opposing views. As explained by a seven-term Democrat: When you get into the entire General Assembly, you have people on opposite sides of the issue. Then you’re trying to convince people that your data is proving the other people wrong, so that’s a little tougher. . . . The first thing that I always try to do is if I can ever get a peer-reviewed study, I always go with that because then I can say, “Look, there are experts in this field who reviewed this and determined that the methodology worked and it actually got published in a scholarly journal”—that’s kind of like the gold standard for whether or not something is considered good research. Another use of research for persuasion was to debunk myth, which is often treated as fact in policy debate (Ferguson, 2015). A Democrat was diplomatic in referring to myth as popular wisdom: The popular wisdom is if you increase a penalty on a crime, that crime will decline, and that the higher the penalty, the more the crime will be reduced. . . . So, you start by saying, “Okay, now that might make sense to somebody because for an average person who doesn’t commit any crime, the greater the risk, the less likely we are engaged in that activity.” But . . . we’re dealing with criminals, right? Or people that are prone to crime. So, that’s when you’d say “Now, people have actually studied this” . . . and what their studies have basically showed is that . . . if you have some penalty, there is some deterrence . . . but there’s a 101

N ew E ra of T heory , R esearch , E valuation

point where any additional penalty, you don’t get any more deterrence. So, it’s really a waste of your time, energy, and money. Providing a Larger Context for Thinking About Issues and an Enlightened Understanding of the Complexity of Issues The second most frequent contribution of research mentioned by legislators was predicted in two of the four theoretical frameworks. This use, mentioned 72 times, often supplements other information that policymakers rely on. According to a Republican: [Research] helps to provide a kind of a common base of understanding, outside of whatever personal experience somebody might have. I mean, anecdotes can help illustrate a situation, but research can help connect that to a broader universal understanding of what’s going on. A Democrat illustrated the value of background research when he learned of an increase in people opting out of vaccinations as part of a “weird sort of natural movement”: I mean research is what provides meat to the bones of data. Data can tell us like . . . asking the questions, is there any other reason to explain this? . . . Is it just people opting out? . . . [Has] there has been a trend where doctors are increasingly reluctant to confront people for fear of  .  .  . making them mad? Okay, is there any intervention that we can make to increase facts and compliance without making the doctors uncomfortable? And there was. The research exemplars tended to discuss how understanding the larger context of issues can benefit policy decisions, whereas the key informants emphasized its political advantages: What I always thought was very valuable in making arguments with legislators, and winning arguments with legislators, is to help them take what appears to be just a single issue and put it into a context that at least the members of my caucus or maybe all legislators [or] people from my state, would hold. So, I don’t know much about this issue. There’s people saying pros and cons, but this guy is saying, “See this fits into all the things we were trying to do to take Wisconsin’s economy into the 21st century or whatever.” I  always found that was valuable . . . because in a way you’re helping them make the argument back home. Another way that background research on issues has political value to policymakers is that it helps them think about whether their vote could have damaging consequences in the next election. As aptly explained by a former Democratic senator: The legislature is not a perfect world. Thinking more broadly about issues could sometimes help, because a legislator often thinks in terms of if I vote for this, the narrow selfish concern is “Is this going to get me beat next time?” Legislators love to get reelected. . . . Your instinct is, if I vote for this, something that I have not spent a lot of time on . . . what’s going to come back at me from a negative standpoint that I didn’t know.

102

H ow P olicymakers S ay T hey U se R esearch

Contributions Frequently Mentioned by Legislators, Frequently Predicted by Theory Designing Good Legislation and Stopping Bad Legislation This research contribution that policymakers frequently mentioned was predicted by all four theoretical frameworks. Among the 61 mentions, policymakers described five different ways research is used to design legislation. First, research can have a direct and immediate impact on policy, though this occurs only occasionally. A Democrat described an instance when research was quickly turned into policy, and remarkably on the politically contentious issue area of the environment: Illinois’s [microbead bill] actually passed first. So, suddenly we all started copying Illinois’s language for the purpose of uniformity. . . . It almost never works out this easily, but it genuinely was about this two-hour presentation that a group of 80 legislators from across the basin got on microbeads and why we needed to phase them out of consumer products because it was having this very real adverse impact on the lakes. A second way legislators used research to design legislation was by looking to their counterparts in other states to determine what already had been tried and with what effect. A key informant described why comparisons with other states was often a frequent use of research: Research is important to find out . . . the impact. Research will give you examples of the same concept or a similar concept that has been tried someplace. What are the ramifications of that? Have they been good? Is it bad or different? How would the state of Wisconsin be different than the state of Indiana or . . . Florida coming up with that? Other policymakers described a third way that ideas from research entered into their own legislation. A Republican key informant explained how he used research to go back and “make sure that this premise that you built your entire house of cards on is actually accurate”: [Research is] a pretty important tool . . . as you think what it is we’re trying to achieve to help people. At some point, you need to start measuring those thoughts and aspirations against what is the reality of the condition today and what is really happening. Are my thoughts actually accurate or are they inaccurate? And if we’re going to solve these problems then, what changes do we need to make in order to make it be better? Do we have any data that shows that those changes make a difference? You just keep coming back to that interplay. A fourth use of research is attempting to stop bad legislation. This is a highly valued role of research, according to a key informant: “What legislators don’t want to do is pass a bill and then run back here in two weeks and repeal it because it’s all screwed up. . . . We’ve seen that happen.” A  Republican found research useful in spotting unintended consequences of decisions: “If someone else suggests that this action will create this

103

N ew E ra of T heory , R esearch , E valuation

outcome and we haven’t considered the potential downside outcomes . . . that’s what I’m using research to try and find.” Finally, a Democrat detailed how research can be used to stop bad legislation, even after it has been enacted. Research can serve to establish legislative intent in court challenges: [Research] can be a sword, but it can also be a shield. And more often than not, because we’re in the minority, we’re trying to stop things from happening. . . . So, oftentimes I’ll introduce a series of 10, 12 amendments on a bill . . . using either legal research or social science research or health research . . . to demonstrate we gave them the opportunity to fix the bill. We know with the best evidence that’s out there that this won’t work or it will have this adverse impact. “You had the opportunity to fix it and chose not to,” and that creates a record amount of legislative intent that then gives standing to perhaps challenge it. . . . [Research] appears to be a shield, but it can turn into a sword when you go into the courtroom to try to slow something down. . . . So, if a majority is just going to blatantly ignore all the evidence that is out there, that creates a different venue perhaps to challenge the validity of what you might consider bad public policy moving forward. Defining Problems and Raising Awareness of Why Action Is Needed This contribution was frequently mentioned by legislators (40 times) and predicted by all four theories. A Republican described the vital role research plays in providing verifiable data to point out a problem: “I keep coming back to data that is verifiable, and hopefully indisputable, so that ‘Okay, this is the issue. . . . This is what we want to solve.’ ” Another Republican gave an example of when research was used to raise awareness of an unintended consequence of a law currently on the books: We have certain laws in Indiana where people who have been convicted of offenses against children . . . can’t live within so many feet of a school, so many feet of a daycare, or a church.  .  .  . DePauw [University] students had done a research study where they took Marion County and they did circles—these 1,000 or 500 feet circles—and they showed how few areas in the community where people who had convictions could live. We all want to protect the children [but] these people had done their time. . . . They’ve paid their debt to society . . . but there’s this permanent ban. Then you start realizing . . . their freedom isn’t there anymore. . . . Before we put those lifetime restrictions on everybody, we need to know you can’t extend that out forever because . . . everyone has to live somewhere. A Democrat described a new issue that arose when an industrial sand mine and processing plant set up operations in her district. Anecdotal evidence emerged almost immediately after the high capacity wells began local operations: “Neighbors’ water pressure dropped dramatically, wells went dry, and water filters normally changed every 30 years had to be changed every three months. Chicken watering devices clogged with sand and chickens died. Heavy metals appeared in the drinking water.” Some research was available, such as the inability of humans to tolerate high levels of heavy metals in drinking

104

H ow P olicymakers S ay T hey U se R esearch

water, but not the research that she most needed to determine whether water quality in her district was a problem. The state’s Department of Natural Resources was not monitoring the situation, so there was little data to verify how widespread or damaging the wells were. This state senator encouraged residents to avail themselves of the county’s water testing program as a first step toward collecting data to define whether a problem existed that warranted policymakers’ attention: There’s never been this kind of an implosion of the aquifer that’s been caused by a sand mine that I  know of. It’s never happened in Wisconsin before, so how are we going to figure it out? Our purpose is to find truth with a capital T, what’s really happening. . . . not to argue one side or the other of an issue. The purpose is to understand what actually is. . . . I would capitalize what actually is, W-A-I. Contributions Frequently Mentioned by Legislators, Seldom or Never Predicted by Theory Educating Others One research contribution, mentioned 49 times, was not predicted by any of the theoretical frameworks—using research for educating others. A  Republican considered “educating” an important part of his job, which a Democrat clarified is distinct from “persuading” others to a particular point of view: It’s funny because I think it’s a lot of skills that a good professor has to have in that I think you need to be able to explain the issue in a clear and concise way, so you’ve got to make it understandable. . . . And then once you’re able to get them to understand . . . the issue, then you need to persuade them that your view on it is the correct one. . . . So, one part is like information, and the other part’s persuasion. It’s like a two-step process. A Democratic key informant clarified that the second step of persuasion may be unnecessary if you have a good relationship with someone and confidence in their ability: “If you know them well enough . . . you can give them the facts that support your point of view [and] you don’t have to talk them into it.” A Republican noted how education may play a more significant role for some colleagues than for others; he found education to be more important for legislators who view their job as being a trustee and vote based on their own judgment and understanding of issues, versus those who think of themselves as delegates who vote the views of their constituents (Rosenthal, 2009): Acting as a trustee is not without additional responsibilities, namely robust, frequent communication with your constituents to explain to them why you see something differently than they might at first blush. So, I feel like a big part of my responsibility is to take information back to them. . . . I get to spend all this time learning about issues and meeting with people who have specialized knowledge of the issue. And that puts me in a position where I can go home and say, “Look, I understand that you feel this way. And I might have even felt that way myself. But now I’ve learned some things and I’d like to share those things with

105

N ew E ra of T heory , R esearch , E valuation

you.” . . . I would say “educating” is the right word because it’s not “persuasion.” I’m not trying to change their beliefs. This education function extends beyond constituents to colleagues. A Democrat, well trained in research methods, used her newspaper columns to educate members of her caucus: I’ll say, “I put this piece out this week. . . . Here’s my source if you want to read any more about it.” I do it for a couple of reasons. I do it because I found a way to communicate something that maybe other colleagues were struggling with, or I found a little nugget of knowledge that I felt needed to be lifted up. And then I do it sometimes because I get irked at the partisan rhetoric from some of my colleagues. And I feel like I’ve come up with a better way to say what they’re saying in not such an inflammatory way. This caucus example illustrates a theme that emerged from the key informant interviews—the political overlay when research is used for the purpose of education. When legislators educate colleagues in their own caucus, a key informant noted: “It’s done, not only for internal education, but for purposes of external political reasons.” This aligns with the view of another key informant who claimed that there is a fine line between educating and persuading: “You’re always persuading, whether you think you are or not.” Even the decision to educate may be subject to political calculation: Zealous argumentation is not as useful in lots of circumstances. A softer tone is important. . . . Use of persuasion techniques essentially is a sense of trying to dominate someone. . . . [With] education, you bring them along to consent to go your way. Improving the Decision-Making Process by Enhancing Debate, Dialogue, Collaboration, and Compromise Another frequently reported contribution of research that came up 41 times was not producing a specific policy outcome, but rather improving the decision-making process itself. Only one theory predicted research could be used for improving the policy process. A member of the Republican majority explained how research is the first step toward compromise: “At first, you’ve got to agree on the context and the data set that you’re using before you can horse trade anything.” Similarly, a Democrat gave an example of how she had observed research improve policymaking by bridging ideological divides: And there was a study put out by a group who I’m usually not in agreement with—The Wisconsin Public Policy Institute—who’s a right‑wing think tank. . . . They had a paper on the investment in quality early childhood programs. And they concluded that you get the biggest bang for your buck, more so than any tax cut, in that investment. And so, I can use that sometimes to say, “Look, this is what your own people are saying about this.” A Democrat used research to reach consensus, which began when he called the university to request a fellow to work in his office one day a week. The fellow wrote a literature review on infant mortality that identified programs that were research-based 106

H ow P olicymakers S ay T hey U se R esearch

and culturally competent. The legislator proposed funding and included provisions in the statute that allowed monitoring in accord with the best practices suggested by the research evidence: We were debating that infant mortality stuff . . . and there was this initial, “No, that’s government spending. We don’t want to do that.” And there was just the basic research about . . . the infant mortality rate right now . . . that falls in between Nicaragua and East Pakistan. Like is that really where you want to be? . . . Through our own . . . failure to act here, we’re just going to continue to let children die before they reach one-year-old because we don’t want to provide adequate medical care. And there was an interesting thing that happened during the debate. . . . The women Republicans actually started saying, “Oh, hang on a second here. What are we saying we should vote down?” . . . They actually stopped the proceeding for about 10 minutes, and they had a little side huddle. It was mostly women Republicans . . . sort of shaming the men and saying . . . “We think this makes sense.” That [consensus] was sort of two things. It was the evidence of it, but then it was the morality of what happens if we don’t act. Key informants pointed out various ways research can rise above politics and create a climate that contributes to constructive dialogue. For example, one key informant explained that research can “keep the debate more civil” and take some of the edge off an argument that could become personal. Another key informant emphasized that research is “really important” in winnowing down differences of opinion: “In order to overcome all the different opinions that people have, you need to be able to either go to an outside source or construct an outside logical argument” that can move dialogue away from “philosophical or political” posturing and toward a “problem-solving exercise”. Using the example of financing education from kindergarten to Grade 12, he worked with the nonpartisan legislative service agency to develop a report that indicated how many children were enrolled in each school district in the last three years and whether attendance was growing, static, or declining. The report calculated per-child spending each year based on property tax revenue, state tuition support, English language learner funding, federal assistance, and so forth. He described the impact of using this per-child data at public forums: I mean everybody is ready to fight before I started. The first time I went to the forum and I  did that, I  mean literally, it took the air out of the whole room because it brought it back down to what they’re fighting about was pretty much covered by this fact. Earning the Trust of Colleagues as a Knowledgeable and Credible Information Source Absent from every theory was a contribution that policymakers mentioned 34 times— the role research plays in earning the trust of colleagues as a knowledgeable and credible information source. To fully grasp the significance of this contribution, it is important to understand how critical trust is in the way business gets done in the lawmaking culture. One Republican was emphatic that “You can’t get anything done without it.” Trust is 107

N ew E ra of T heory , R esearch , E valuation

critical in the policy culture, according to another Republican: “This isn’t a private business. The CEO does not say, ‘This is what we’re going to do,’ and it happens. This is management by consensus.” In policymaking, the centrality of trust extends beyond its role in reaching consensus. Reliance on trust also is a pragmatic response to the rapid-fire pace of lawmaking and the staggering volume of bills policymakers face on a sweeping array of issues (Krehbiel, 1992): We vote on about 500 bills a year around here. I’m not an expert on 500 issues, and neither is anybody else. There has to be a certain amount of trust through the committee process and through individual members that they know what they’re talking about. . . . And, so, it becomes a matter of your reputation as well. Once I develop a reputation for sweating the details on these things, my colleagues have a degree of confidence that, number one, I will tell them the truth. Sometimes the truth requires me to tell them things that are not favorable to my position. . . . Second of all, if they know that I am going to tell them the truth . . . quite often they’re willing to give me the benefit of the doubt in moving ahead. And that has been helpful to me in many, many respects. It also helps lead to bipartisan collaboration. If your colleagues . . . know that you’ll tell them the truth and that you have your facts right, it at least gives you the basis for a compromise. In the policy culture, research and relationships with colleagues can have a reciprocal influence on each other. According to a Republican, his use of research influenced his ability to build trusting relationships; in turn, these relationships also influenced the likelihood that his colleagues would trust him and pay attention to the research that he sought to advance: This is a relationship business and one of the liabilities for a guy like me who values research is that I sometimes don’t focus enough on the relationship side of being a legislator. . . . The reality is if you want other people to pay attention to your research—the research that you believe is important—then it needs to start with having a relationship where you can . . . have communication and trust. If someone won’t talk to you or won’t take you seriously, it’s difficult to present them with research. A recurring theme was the strategies legislators use to gain and maintain a reputation as a knowledgeable and credible information source. For example, earning the confidence of one’s colleagues was said to begin with trustworthy facts. A Democrat described the diligence required to leverage a mastery of trustworthy facts into a reputation as someone who can be trusted: If you become known for the kind of person who uses studies and more factbased stuff in the debate, I think people might give you a little bit more credit, but you have to be really careful that you maintain that confidence in people. So, in other words, if people kind of figure out that you’re just using bad studies to back up your point or that the data you’re presenting is just from some leftwing think tank or something, then that will undermine you. . . . If you kind of develop a reputation for somebody that’s more fact-based, and people feel like, “Okay, I may not agree with what the guy is saying but I’ve never known him to misrepresent something, or to just kind of throw garbage out there.” 108

H ow P olicymakers S ay T hey U se R esearch

A Democratic key informant echoed the thinking of many legislators about how fragile a reputation as a trustworthy information source can be, and how hard it can be to maintain: “You’ve got to be credible 100 percent of the time. . . . When trust leaves you, it gallops away, and it comes back walking very slowly.” A Republican key informant expanded on how policymakers build trust among their colleagues. For example, new legislators typically are advised to establish a niche expertise in an area important to them, an institutional practice that has been observed in other studies (Krehbiel, 1992; Matthews & Stimson, 1975): Those who take that advice to heart will almost always  .  .  . try to become a knowledgeable and credible source on a topic, and they recognize that’s a key way to build relationships and earn trust . . . I’ve just commented on how important I think relationships are. But in the end, if there’s not a substantive basis on top of that, a legislator won’t be as effective as they could be . . . you need to have both. Asking Important Questions for Policy or Political Purposes Another frequent contribution of research that did not appear in any theoretical framework was how having a good grasp of research prepares policymakers for asking important questions for policy or political purposes. On 30 occasions, policymakers mentioned using research to ask important questions for policy purposes (e.g., identifying and raising what the underlying root of the problem is) or political purposes (e.g.,  signaling incomplete or misleading information, or pointing out hidden motives or obscure agendas). A newly elected Republican quickly learned that asking questions is an important skill for being effective: “I’m still learning . . . what are the right type of questions to ask so that you get the results you are looking for, so that you know what the intent of a piece of legislation is?” Policymakers described how questions, if well-informed, can serve multiple purposes. For example, a Democrat raised questions about a bill being discussed at a hearing both to educate himself and to score political points: A witness . . . starts to advocate for or against it. The way I formulate my position and try to educate is by the questions I ask. . . . In many cases, the design is to figure out what’s going on, because it’s not always self-evident what somebody is trying to do, or you haven’t had time to read it carefully or study it. So, many of my questions are absolutely just open-ended questions. . . . But, at a certain point, my questions become very focused to show the weaknesses or the strengths. . . . I will ask questions based on the data showing how much money we’re spending on it and who’s benefitting from it, and whether we really caused any change or simply have redeployed our money in a different direction. Policymakers also used research to ask leading questions or “gotcha” questions. A Democrat described how her informed questioning exposed the unstated agenda of an agency leader: They had cited three other states as better examples. . . . I looked at the three other states they cited [and] found out they’re horrible examples. . . . So, when the Secretary came before the . . . committee, I could say, “Well, you gave these three examples as better. Why were they better?” And he could not answer the 109

N ew E ra of T heory , R esearch , E valuation

question . . . so it enabled me to kind of expose that these Secretaries actually did not know what they were talking about. The key informant interviews revealed three reasons the contribution of research to asking questions may seem obscure which explains, in part, their absence from the theoretical frameworks. First, an experienced legislative analyst explained that legislators are more likely to ask questions in private settings with two or three colleagues than in public settings. A former legislator, who rose to leadership, would ask questions in some public settings, but not in others: I would ask an important question . . . in a hearing because it’s fair to ask an important question like, “So, how many miles of state roads are there?” . . . By the time it got to the floor, personally I would never ask a question that I ought to know the answer to, because . . . that would make me appear not as competent as I want to appear. A second reason for the obscurity of questions is that they often occur behind the scenes. A Democrat described how he raises questions with a Republican colleague in personal meetings “so that I don’t embarrass him”. Similarly, a key informant said policymakers who are genuinely interested in “good legislation” will make certain to raise requisite questions with a bill’s sponsor to help ensure any unforeseen implications are considered, but deliberately does so discreetly: I’ve always appreciated and I think others appreciate to ask the question before a committee meeting or before they get on the floor so that they can be prepared. . . . I never waited until somebody got up there and then . . . asked a question that would just kill a bill. Finally, the importance of a research-focused question may be overshadowed by the immediacy of the response. If a question comes at the right time, people forget just how influential it was, given how quickly subsequent events transpired. A key informant explained how powerful research can be in propelling issues forward: You get tipping points. . . . Smoking bans went nowhere until it was established that it wasn’t just hurting the smokers, it was hurting everybody. It was secondary smoke that was the big tipping point. . . . It isn’t [that] cigarette smoke causes cancer, bang, cigarettes are gone. . . . Somebody’s got to ask the question, somebody’s got to make it into a potential tipping point. And it will always be facts. Contributions Infrequently Mentioned by Legislators, Frequently Predicted by Theory The six contributions of research at the bottom of policymakers’ list were mentioned only six to 29 times, yet were frequently acknowledged in the theoretical frameworks. That means scholars theorized about these research contributions, but they were infrequently mentioned by legislators. All four theoretical frameworks mentioned that research offered new and emerging ideas. Three frameworks mentioned contributions to informing one’s position on an issue; justifying a preexisting position on an issue; assessing the political and economic feasibility of policy decisions; 110

H ow P olicymakers S ay T hey U se R esearch

changing one’s position on an issue; and explaining a vote on an issue. Each one is considered, in turn. Informing One’s Position on an Issue Policymakers mentioned 29 times that research contributes to policymaking by informing one’s position on an issue in ways such as helping them decide how to vote to ensure they are headed in the right direction. A Republican reported relying heavily on research for forming opinions: Research has certainly impacted the way I  thought about things like road funding to school funding to where you would put increased revenue from an increase in cigarette tax. So, it just serves a variety of different things, but I think there’s probably not a single major issue that I wouldn’t say research has had a major impact on my opinion. Another Republican explained how he intentionally seeks out competing sources of information: I think you’re better served to try and find information that stretches you . . . just not reinforcing. You want to find the opposing information. . . . I’m a conservative, but I read the New York Times editorial page. I read the Washington Post editorial page. These are not people I traditionally agree with, but I think it’s important to understand. There are moments in time [when] I go, “Okay that’s interesting writing” [by] Brookings or whoever. You have to be able to say to yourself, “I need to understand the arguments of the other side.” Sometimes you go, “Hey, I understand that—that makes sense.” Justifying a Preexisting Position on an Issue Legislators mentioned 24 times using research for justifying a preexisting position on an issue. A Democrat trained in law reported why he uses research to support his priors: I’m guilty of this too. We’re guilty of acting more like lawyers. I’ve got a position. You’ve got a position. And then using research to back up a particular position instead of the other way around—letting the research inform the position. That’s not always true. It’s . . . the default . . . I hope in the future that we’ll be more scientific-minded. A Republican key informant who had experience in both states confirmed this perception: I think the research overwhelmingly says that giving parents a greater choice in where they send their children to school actually provides a benefit. There’s gold standard research that shows it, but there’s a small amount of research that raises questions about that. So, now it’s just “He said. She said.” . . . Both sides are defending a position they hold, really based upon a personal set of beliefs or a political set of conditions. And, then they look for the research to back it. Unfortunately—quite often—research is used after the fact to justify a position rather than to determine one. 111

N ew E ra of T heory , R esearch , E valuation

Assessing the Political and Economic Feasibility of Policy Decisions Policymakers mentioned 21 times using research for assessing the political and economic feasibility of policy decisions. As an example, a Republican legislator said: I’ve always advocated that we need to do more cost-benefit analysis . . . before we jump into something. It shouldn’t be just gut instinct. It ought to be more based on valid reliable research that would say, this is where it’s going to go. An example of economic feasibility was raised by a Democratic legislator when he was working on a bill to mandate HPV vaccine, which at the time was incredibly expensive: “An oncologist said to me, ‘You know if your goal is saving lives and you put the amount that you’re going to spend on the HPV vaccine into tobacco prevention, you would actually save more lives.’ ” When key informants were asked why assessing economic feasibility was not mentioned more frequently, a nonpartisan legislative analyst explained that “Economic feasibility is a bit of a tricky thing to gauge. . . . It may require some modeling and predicting that’s more or less reliable over time.” Another key informant questioned the reliability of dynamic scoring that entails predicting people’s behavior; he specifically went on to caution how meaningful economic analysis is at the state level when “what the feds do is far more influential than what [a] state . . . can do”. When asked why assessing political feasibility was not mentioned more often, a key informant who self-identified as a “junkie on polling” explained that polling can lead to good and bad answers. He prefers to use polling to sell an agenda, rather than to develop one: Too many politicians try to use it to develop their agenda. For instance . . . the polling is really high on saying we should mandate that a certain percentage of the money we give to schools ends up in the classroom. That phrase polls off the charts. So, 60 percent of the money should end up in the classroom, 70 percent of the money. Hell, you could go out and tell people 90 percent and they would be in favor of it. But, that’s not necessarily the best answer for running the schools. It just polls really, really well. So, you get politicians who are going to say, “I’m going to propose the ‘more money in the classroom’ bill.” There’s just a lot of simple solutions that poll very well that aren’t actually effective. Changing One’s Position on an Issue The use of research for changing one’s position on an issue was mentioned only 13 times by policymakers. A Democrat explained why policymakers seldom mentioned this contribution: Research can change my position, but I don’t go to the research on the assumption that I’ll change my position. So, it can inform me. But I never, I never set out to say, “Oh, I want to look into this because I want to change my view.” A Republican clarified, however, that the capacity of research to change an opinion depends on whether or not a legislator is open to views different from his own: I think you have to have an open mind to understand you’re not always right. And sometimes being able to recognize research that says, “Oh, I guess

112

H ow P olicymakers S ay T hey U se R esearch

I was wrong on that issue” and being able to accept that and not just turn your head and a blind eye. . . . You got to be able to see the other side of the equation. When a key informant was asked how often research changes a legislator’s opinion, he initially responded “not a lot”. Upon reflection, however, he offered a more nuanced and circumstantial view of how it depends on the issue: There’s hot-button issues. There’s things that are really big issues  .  .  . social issues. . . . But then there’s a million other, I’ll call them technical, issues . . . that’s really the majority of the work . . . that really no one cared much about, but were important. But [they] weren’t going to be on the news tomorrow. Now on those, all bets are off, because there’s no, quote, “Republican” or “Democratic” theory about that and there’s no big contributor to your campaign who cares about this. It’s just like you’re stripping it down to the essence of good government. It’s like, “No one really cares what we do here—let’s do the right thing”. Those are the kinds of issues . . . where research could have a lot more impact. . . . So, I think you got your big hot-button issues that are highly politicized. . . . The other 80 percent that are technical, but yet very important to the government of the state, I think you have a much more free reign . . . to sort of govern in an evidence-based way. Offering New and Emerging Ideas Policymakers mentioned 13 times that research contributes to offering new and emerging ideas. Occasionally, examples were given of research evidence that directly influenced legislation: A pediatrician said the child safety seat laws are totally out of date. They’re not consistent with the medical evidence. And so when I looked at the child safety seat laws, I realized, “Oh, gosh, I got to change these.” And we had a ton of evidence about what’s the safest for children, for infants. And so that actually spurred a public policy. In interviews with key informants, they explained that new ideas for legislation, especially more minor pieces of legislation, come most often from constituents. Research is more likely to be used for issues that are new and emerging like brain concussions and technical issues like local government tax assessments or annexation policy. Explaining a Vote on an Issue Policymakers made only six mentions of using research for explaining a vote on an issue. Both Democrats and Republicans from each state discussed using research in this way: It’s really more like a friendly . . . if somebody challenges you on why you voted for this thing. Here’s the study you can point to that shows that X, Y, and Z is happening . . . and they’re like “Oh gee, thanks for that information.” It’s not so much like “Okay, prove to me this is really right.”

113

N ew E ra of T heory , R esearch , E valuation

A Democrat used research, not only to explain why he voted for bills, but also why he did not vote for bills. He illustrated with his decision to oppose legislation on raising the speed limit: What data do you have to support that? And the bill’s author was not able to provide it at the time, so . . . my “no” vote was sort of justified to the press by saying, “This is not particularly complicated research, but it’s something that needs to be looked at and determined. Do fatalities increase when you increase the speed limit in various states? Do they go up? Do they go down? Do they stay the same? I think that’s worth knowing before taking a vote.” And in the meantime, I think the default position is . . . to vote “no” and sort of explaining my position on a particular bill with the absence of research. Reflections on Prominent Theoretical Frameworks About Research Utilization Our study of policy insiders, specifically state legislators, raised some fresh insights about strengths and shortcomings of the ways researchers have been thinking about policymakers’ use of research (see the full discussion in Bogenschneider, Day, & Parrott, 2019). As summarized earlier, we found an extensive array of research uses, probably more than most outsiders would have thought existed. Some of these uses comported with the predictions of four prominent theories of the connections between researchers and policymakers, yet others did not. The predictions of these prominent theoretical frameworks were relatively consistent with the perceptions of policymakers in terms of how research contributes to policy issue considerations—defining when action is needed, designing legislation, offering new ideas, and assessing political and economic feasibility. The frameworks also were consistent with policymakers’ perceptions of how research contributed to position taking, whether it be for forming, justifying, or changing a position, or for explaining a vote. However, these prominent frameworks did fail to include research contributions frequently expressed by policymakers in areas such as building relationships and improving the policy process itself. The predictive ability of theory can be enhanced when even our most “beloved theories” are seriously tested for falsifiability using new research evidence (Ferguson, 2015, p. 532) and, in particular, data from a broad range of policymakers. Karen’s study of state legislators raises six roles that research plays in policymaking that are cast in bit parts in the current theoretical playbook—providing a larger context for thinking about issues; earning the trust of colleagues; educating others; asking important questions; enhancing debate, dialogue, collaboration and compromise; and persuading others. As discussed next, each may prove increasingly important as we capture more data and continue to refine our theoretical thinking. First, existing frameworks did not focus on the research contribution that is most often recognized in empirical studies—providing a larger context for informing policymaker’s thinking about the complexity of issues. This enlightenment function of research is thought to be the most important influence of research on policymaking given the conceptual grounding it provides to achieving a shared understanding of what to do, how to do it, and why (Albaek, 1995; Henig, 2008; Hird, 2005; Nutley et al., 2007). In our study, the role of research in building understanding of the larger context has both policy and political benefits for policymakers. 114

H ow P olicymakers S ay T hey U se R esearch

Second, prior thought on research utilization tended to conceptualize it mostly as a knowledge-driven model that focused on the individual policymaker (Weiss, 1979), rather than the more expansive set of relationship-building contributions described by policymakers. Appreciating the relational nature of policymaking moves us away from what Mayhew (2011, p. 882) described as “the individualization” of politics and toward an understanding that research use runs on relationships—that is, research use influences relationships and relationships influence research use. A policymaker’s grasp of research earns the respect and trust of one’s colleagues that feeds into a reputation of being knowledgeable and a credible information source; in turn, the trust of one’s colleagues in a relationship-based culture parlays into confidence in the research that a policymaker seeks to advance. In the policy culture, relationships were built on substantive information—trusting relationships operate in tandem with trustworthy research and data. Third, another aspect of the collegial nature of policymaking was educating others about research on policy issues. Policymakers described how they educated the press, members of their own caucus, and constituents on the research they gathered on an issue. Using research to educate constituents was one way they connected their activities in the statehouse with their activities in their home districts (Fenno, 1978). Policymakers asserted that educating constituents was central to how they do their job but, in an elected body, it also helped policymakers keep their job (Fenno, 1978; Mayhew, 2011). Fourth, our data suggested that research prepared policymakers for raising important questions for policy or political purposes. This type of questioning could be used for policy purposes, such as informing their own position or gaining a deeper understanding of the underlying problem and viable responses. Sometimes questions were used for political purposes such as casting aspersions on the credibility of research or to expose an opponent’s obscure motives or hidden agendas. Using research for political purposes appeared to rise in importance as a negotiating tactic for those who wielded less power in the policy process, particularly members of the minority party, and those operating in more polarized environments. Fifth, prior thought gave short shrift to the policy process itself and the contributions research makes to decisions that are negotiated in a politically charged environment (Albaek, 1995). The potential of research to influence the policy process is not a new idea, having first been raised by pioneering researcher Carol Weiss (1978) over four decades ago. Yet it still has not received sufficient attention in research and theory, according to researchers (Rickinson & McKenzie, 2020) and a recent report of the National Research Council (Prewitt et  al., 2012). Typically, prior thinking has failed to acknowledge that even the most instrumental and direct uses of research occur in a political context; this disregard of the political overlay of policymaking has downplayed the potential contributions of research to civil, problem-focused discourse and negotiation strategies that can enhance debate, dialogue, collaboration, and compromise. For theories to improve their predictive power will entail recognition and reification that any contribution of research comes through “its place within politics, not its rivalry with politics” (Henig, 2008, p. 18). Finally, the most frequent contribution of research to policymaking was persuading others to support one’s view or to anticipate and counter an opposing view. Politics will “always take the form of argumentation” (Albaek, 1995, p 90). Yet research and theory devote little attention to the legitimate function that research serves for developing persuasive arguments and devising political strategies for coming to common ground (Mayhew, 2006). In a democracy, decisions are made, not by experts such as Plato’s 115

N ew E ra of T heory , R esearch , E valuation

philosopher kings, but rather by a deliberative process of discussing, negotiating, and persuading in which research can play a significant role (Albaek, 1995). Looking closely at what our legislators said about how they used research led us to think differently about the issue. Taken together, it appears that legislators characterized their use of research in two different ways. They reported using research evidence to achieve end goals such as consciousness raising, agenda setting, policy design, and resource allocation. These are the classic ends that have been the conventional way that research use has been conceptualized. However, our respondents also reported using research evidence as a means to achieve policy ends. They tended to focus on such things as building trusting relationships with their colleagues, persuading others, asking clarifying or convicting questions, enhancing policy dialogue, and so forth. These and a variety of other interim goals were more about improving how the policy process worked and less about the character of the legislation under consideration or about the zero-sum game of winning and losing. The policy process, particularly in the highly charged political world of the legislature, was too often misshapen by heated partisan and normative disputes. Often, legislators prized the role research played in shifting the dialogue away from passionate disputes and toward informed negotiation. That did not work in all policy matters, but often enough to be noticeable. For us, the study outcomes led us to think harder about the process and the means for achieving policy ends, as well as the ends themselves. It shifted our own thinking on what was critical to consider in how research is used in the policy arena.

Given Our Findings, Whither Goes Theory? Those who have thought long and hard about the utilization of research use remain skeptical about the ability of existing theories to explain research use in different contexts across policy and practice communities. For example, Nutley and colleagues acknowledge the enormous influence that the two-communities theory has had on subsequent thinking, but also raise questions about its limitations as an explanatory framework. For example, the theory minimizes the diversity among researchers and policymakers, and ignores many other key players on either side of the research/policy divide. The theory also downplays the organizational context in which these key players operate, such as the influence of politics and power in the policy world (Nutley et al., 2007, p. 111). In general, existing theories have been criticized because of their inability to predict how and which information decision-makers use (James & Jorgensen, 2009). In particular, the explanatory ability of the conventional research typology has been questioned by the National Research Council: “It is unrealistic to expect a clear and unambiguous typology for a phenomenon as complex as the use of science in policy” (Prewitt et al., 2012, p.  39). Current theorizing provides a list of factors with little attention to what their explanatory ability is, how they interact with each other, and whether they transcend specific settings. This is a complicated business. Yet this does not rule out the possibility that new research may identify factors or processes that so far have been missed (Prewitt et al., 2012). To optimize predictions about future research use, we believe theoretical antennae will need to be attuned to three key elements—identifying the specific community that is being targeted, clarifying definitions of research use, and incorporating the perspectives of policymakers themselves, the topics we turn to next.

116

H ow P olicymakers S ay T hey U se R esearch

Key Concepts 5.2.  Three Key Elements to Optimize Future Predictions of Research Use • • •

Identifying the specific community that is being targeted Clarifying definitions of research use Incorporating the perspectives of policymakers themselves

Identifying the Specific Community That Is Being Targeted In the quest to understand research use, existing theoretical frameworks often fail to appreciate the importance of identifying the target policy audience. They sometimes ignore even the simplest of user dichotomies, such as researcher producers and research consumers, each of whom has a key role to play in research utilization. As we developed our own thinking on research use, we quickly understood that those who produce studies and those who consume them are distinct communities that live on separate islands and operate within discrete cultural landscapes. Those who seek to understand the way that the world works look through different lenses than those who use research to make the world work better. That leads us to consider whether there were other salient communities of professionals bound together by common interests. We wondered whether these communities that produce different core technologies might have their own unique perspective on the value of research evidence. To optimize utilization, it is critical to deconstruct the complexities of a community’s core technology and the culture that shapes how inhabitants go about their business. One important beginning step in developing any theory of research utilization is figuring out which community to target, so key motivations and dispositions could be identified. One example of oversimplification and a failure to identify a target community is the conventional typology of research use most often used in recent years (Nutley et al., 2007; Tseng, 2012; Weiss, 1999). This typology includes five main purposes, three that obviously seem to map onto the core technology of research consumers—conceptual, instrumental, and tactical uses of research. However, two of the purposes—imposed and process—seem to better map onto the core technology of research producers. Imposed (or mandated) use of evidence-based programs is driven more by researcher producers, who typically are the ones who identify which programs pass scientific muster and warrant replication. Research consumers may be involved, to a limited extent, in some political decisions about which evidence-based programs end up on the final approved list. However, the primary role of the research consumer is enacting and implementing those programs deemed most deserving by the research producers. Imposed use is so strongly skewed to the research producer end of the equation that “those on the front lines often feel that evidence is something done ‘to’ them” (Tseng, 2015). This inexorably suggests that our theories would be more predictive if we defined more clearly which community we had uppermost in mind. Clarifying Definitions of Research Use Another complication in existing theoretical frameworks is that the uses of research are often defined in inconsistent and confusing ways. Using the conventional typology of

117

N ew E ra of T heory , R esearch , E valuation

research use (Nutley et al., 2007; Tseng, 2012; Weiss, 1999), studies frequently find that policymakers use research for conceptual and instrumental purposes (Bogenschneider, Day, & Parrott, 2019; Farrell & Coburn, 2016; Yanovitsky & Weber, 2020b). Yet, looking across studies, it is hard to decipher what these findings mean, given that researchers themselves define these purposes differently and interpret them in contradictory ways. For example, Yanovitsky and Weber (2020b) expanded on the conventional definition of conceptual use (Nutley et al., 2007; Tseng, 2012; Weiss, 1999) and proposed that it should be defined as “informing one’s own perspective and understanding of an issue as well as others’ perspective and understanding”. One of the main findings of their study of Congress is that 73 percent of conceptual uses were employing research to persuade other policymakers. Similarly, in our study, the most frequent use of research in state legislatures was for the purpose of persuasion. However, in our theoretical framework, we would define this use of research, not as conceptual, but rather as an instrumental means of using research to persuade the adoption of a particular policy position. So, persuasion was the leading use of research in these two studies of federal and state policymakers, which one defined as a conceptual use and the other defined as an instrumental use. Another example of inconsistent meanings of the purpose of research use comes from a study of education decision-makers in a major urban school district by Farrell and Coburn (2016). Expanding on Weiss’s original definition, they define conceptual use as “providing policymakers new ideas and frameworks that influenced how they individually and collectively approach their work”. Using this definition, they illustrated conceptual use with an example from a district task force charged with reorganizing the content of mathematics courses to meet Common Core State Standards. After hearing research on the impact of various strategies such as detracking (not assigning students to distinct educational paths), district leaders subsequently decided to take an honors option off the table in favor of a more heterogeneous teaching approach. In our study, we would have defined this, not as conceptual use, but as an instrumental use for the purpose of developing good policies and dropping bad ones. In the conventional research typology (Nutley et  al., 2007; Tseng, 2012; Weiss, 1999), confusion also arises in how the categories are conceptualized. Instrumental, conceptual, and tactical use are all defined as a purpose for using research. In contrast, imposed use is a method for ensuring the use of research-based programs and policies, and process use is a rationale for forming research and policy/practice partnerships. Complicating matters further, process use is often mandated by a government or funding entity, which then overlaps with the imposed category in the typology. This muddies our thinking. Incorporating the Perspectives of Policymakers Themselves Political scientists have called for on-site studies of the inhabitants of policy communities because “fresh theoretical claims might lurk out there waiting to be born” (Mayhew, 2011, p. 75). Because on-the-ground studies of policymakers, particularly state legislators, are rare (Fox, 2010; Merriman, 2019), we turned to our study of the properties and processes of legislators’ social and political worlds in an attempt to enrich our theoretical thinking. Karen’s study allowed us to leverage this creative tension between the empirical findings of research and the more abstract and universalistic propositions of theory (Bogenschneider et al., 2020). We followed the recommendation of Weisz (1978) to move on a continuum from empirical statements that are “relatively specific, concrete, and circumstantial” to propositions that are “relatively general, abstract, and universalistic. . . . 118

H ow P olicymakers S ay T hey U se R esearch

Thus, from the general we may derive the specific, from the specific the general, and so on in a continuing dialectic” (1978, p. 3). Mostly, what emerged from our study of policymakers, at the state level at least, was a deeper appreciation of the subtler aspects of how these policy types saw research. True, they did use research to help them weigh policy solutions, decide how to vote, and determine what problems should be on the policy agenda. Of equal importance, however, they saw research as assisting them in enhancing the means through which policies decisions are made and relationships are built in the collegial environment where negotiations and compromises are hammered out. According to policymakers, using research to gain and maintain a reputation as a knowledgeable and credible colleague is as popular a theme among policymakers as winning the debate on a given issue. In short, the use of research for more immediate means (persuading colleagues to one’s point of view) and for longerterm means (asking questions to clarify the underlying causes of a problem) turned out to be way more important that we initially thought. These empirical insights informed our theoretical thinking as we moved forward.

A Fresh Theoretical Landscape for Understanding the Use of Evidence Our field work taken together with other empirical findings over the last decade led us to refine our thinking about research utilization in policymaking. We argue that existing typologies, while heuristically useful, have long struck us and others as lacking predictive capacity, coherence, and comprehensiveness (Nutley et al., 2007; Prewitt et al., 2012). In 2010, we initially proposed a framework with six dimensions: allocation shifting, tactics shifting, solutions shifting, salience shifting, framework shifting, and awareness shifting (Bogenschneider & Corbett, 2010a). This framework still strikes us as conceptually cleaner than prior schemas, but we still thought it needed some reworking. Our Refined Framework of Policymakers’ Purposes for Using Research In this volume, we do not dismantle our original framework, but instead refine it given all that we have learned from research and practice. We replace salience-shifting with the similar agenda-shifting nomenclature. We continue to include tactics shifting, despite its absence in the state legislator data, because we hypothesize that it might well emerge in other policy communities such as executive agency management/support staff or front-line service providers. In addition, we added the new dimension of position shifting. Unlike most of the prominent theoretical frameworks that tend to focus on individual considerations (see Bogenschneider, Day, & Parrott, 2019; Mayhew, 2011; Weiss, 1979), we had not included the aspect of personal decision-making in our original framework. Its importance was consistently mentioned in Karen’s study, as aptly articulated by one legislator who was trained as a lawyer and specialized in criminal justice reform: Frequently I’m the only one standing in the room saying, “This is a good idea or a bad idea.” That’s a lot of pressure, but that’s exciting. If you’re wavering on your personal judgment, this is a bad job for you. You have to have a strong sense of conscience and direction. If it’s really hard and stressful for you to make a decision of yes or no, voting on 500 bills a year is . . . a bad line of work. . . . We literally do that for a living. 119

N ew E ra of T heory , R esearch , E valuation

We heard in the study that legislators use research to help them with position taking, whether it be for informing, justifying, or changing a position, or for explaining a vote. We expanded our original theoretical framework to include this position-shifting category and further developed our thinking around the key categories, now seven. Our elaborated definitions of each follow: 1. Position shifting is using research to shape one’s position, which can occur at several points along the policy process. It can be used at the front end to inform a position, such as helping policymakers decide how to vote or to help ensure them that they are doing the right thing. Or research can be used midstream to change a position on a prior stance. Research also was used at the back end to retroactively justify a position or to come up with a convincing rationale to explain a vote questioned by constituents or colleagues (Fenno, 1978). 2. Allocation shifting is using research to decide on the political and economic feasibility of distributing public resources. Allocating scarce resources wisely amidst competing demands is a key challenge in governance. Research can provide evidence of where the need is greatest or where the benefits may be most pronounced. 3. Tactics shifting is when research is used to determine the best implementation practices (tactics) for achieving a legislated end. A  given policy choice may be research-based, but it still can be a blunt instrument for achieving its goals if the implementation or management tactics are not consistent with the parameters of the research evidence or if they are not carried out in ways that achieve the legislated ends (Eggers & O’Leary, 2009; Granger, Tseng, & Wilcox, 2013; Kendall-Taylor & Levitt, 2017). 4. Solutions shifting is using research after policy goals are agreed upon to make decisions among the available policy alternatives. The focus here is on the more macrolevel choices of which is the best solution rather than decisions about which is the best tactic to implement the solution. Research can provide one set of accepted criteria for weighing solutions against each other. 5. Agenda shifting means using research to position new issues on the policy agenda or to alter the queue of existing issues in public or political discourse. Here we are talking about getting policymakers to focus on something as deserving of their attention. The positioning of an issue can shift when relevant research comes directly to the attention of policymakers or when constituents or colleagues bring it to their attention. 6. Framework shifting is using research to raise consciousness and alter ways of thinking about or understanding policy issues or choices. How policymakers look at social and economic issues goes a long way toward determining the range of possible responses that come into serious play. To use a classic example, if you used research to shift the causes of climate change from natural causes to human causes, that would radically shift the framework of the debate. 7. Awareness shifting is when research brings to light a new idea or issue previously not part of public dialogue. Also, it can encompass when a doer of public policy comes to realize that an emerging idea might contribute to understanding of a problem, a solution, a tactic, or an unintended consequence. Smoking cigarettes was known to kill many Americans, but it didn’t become a public issue until policymakers became aware that second-hand smoke was a health hazard to nonsmokers. With that awareness, a new issue was placed on the decision-making agenda.

120

H ow P olicymakers S ay T hey U se R esearch

These seven ways that research is used are positioned in a rough order, going from more discrete or distinct policy uses to those focused on larger, more abstract applications. The first use, position shifting, focuses on individual policymakers as they weigh various information to factor into forming, justifying, or changing a position, or into explaining a vote. The second use, allocation shifting, relates to making decisions about the most effective and efficient allocation of resources. With each subsequent theoretical use, the level of proposed change becomes more significant and daunting. By use #7, that of awareness shifting, we are talking about bringing a whole new issue or overall approach into the policy arena. That rarely occurs because in policymaking, long periods of stability are punctuated only occasionally with disruptive policy change (Baumgartner & Jones, 2009). A good example would be getting climate change to be taken seriously. 6. Framework shifting would be getting policymakers to appreciate the role that humans play in changing climate. 5. Agenda shifting would be research that informs policymakers of the urgency of climate change that makes it deserving of public attention and political discourse. 4. Solutions-shifting research would lead policymakers to shift among policy alternatives, such as from programs designed to encourage alternative sources of energy to imposing fines on those responsible for emitting carbon dioxide into the atmosphere. 3. Tactical research, on the other hand, would encourage a shift in program emphases or implementation strategies within a specific, agreed-upon solution. For example, if a study suggests that wind power deserves more support, tactical decisions might arise about where and how wind power can best be harvested. 2. Allocationshifting research might be directed at providing evidence as to how much we should shift public resources from other investments to wind sources of power. We preferred this taxonomy at the time since it appeared to be ordinal in character with each subsequent use appearing more consequential in some larger perspective of doing public policy, even though it is not technically an ordinal scale. The first four uses (i.e., shifting positions, allocations, tactics, solutions) are more instrumental in character, given that they look at questions that affect particular policy decisions or positions, as well as specific issues or pieces of legislation that one might find on any policy agenda. The last three (i.e., shifting agendas, frameworks, awareness) are larger, more conceptual uses. Here, research is aimed at making larger changes on the policy agenda. Our prior taxonomy had instrumental and conceptual categories but clouded the topic with other categories of a different order and failed to detect discrete differences within these two larger categories that we now think deserve attention. Initially, we thought this 2010 framework with its six dimensions was an improvement over earlier typologies, yet recent empirical evidence, particularly that from legislators, alerted us of ways that it could be improved. Karen’s extensive interviews of state legislators clued us in to a broader array of research uses than those previously predicted by conventional theories. So, we started thinking about research utilization from scratch. We were encouraged in this exploration by psychologists who have cautioned that theory can be too simplistic if human responses to stimuli are conceptualized as narrow and rigid rather than as “complex, motivation-driven, [and] agentic” (Ferguson, 2015, p. 532). Karen’s study also provided us with insights into the motivations of policymakers. Broadly speaking, policymakers have two basic motivations for thinking about research use—the final ends in terms of new societal laws, rules, and regulations, as well as the means to be used in affecting those policy ends. We realized we needed a cultural landscape that could accommodate both types of uses. In Figure 5.1, we lay out our updated thinking on research use.

121

N ew E ra of T heory , R esearch , E valuation

USING RESEARCH AS A MEANS TO AN END

USING RESEARCH TO ACHIEVE A POLICY END

INSTRUMENTAL MEANS

INSTRUMENTAL ENDS

Persuading – To convince others to support one’s view or anticipate and counter opposing views

Position S hifting – To inform one’s position, justify a a p osition, change a position, or explain a vote

Educating – To make an idea understandable to colleagues, constituents, and the press by concisely explaining it or providing an example of how to clearly communicate it

Allocation S hifting – To assess the political and economic feasibility of distributing public resources Tactics Shifting – To implement a policy in keeping with its intended goals and the parameters of the research evidence

Asking Political Questions – To develop leading questions to cast aspersions on the credibility or completeness of oppositional research, or to point out the hidden motives or obscure agendas of adversaries

Solutions Shifting – To weigh solutions among the available policy alternatives u sing scientific criteria

CONCEPTUAL MEANS Earning T rust – To gain and maintain a reputation among colleagues as a knowledgeable and credible information source Enhancing D ialogue – To improve the decision-making process by creating a climate that contributes to constructive debate and dialogue, a nd to solving problems through negotiation and compromise

CONCEPTUAL ENDS Agenda Shifting – To position new issues on the policy agenda or to alter the queue of existing issues in public or political discourse Framework S hifting – To raise consciousness and alter the ways colleagues and constituents think about policy issues and choices

Enlightening Understanding – To provide a deeper understanding of complexity regarding the nature of issues, why they are important, and how to respond

Awareness Shifting – To bring to light new and emerging ideas or issues not previously part of public dialogue that can contribute to the understanding of a policy problem, solution, tactic, or consequence

Asking Policy Questions – T o raise questions that clarify the underlying root causes of problems and the implications of various policy responses

Figure 5.1 The Revised Conceptualization of Community Dissonance Theory for Understanding Policymakers’ Use of Research Evidence

Research Use as an End in Itself We begin by discussing one logical starting point for why policymakers turn to research— the policy ends depicted in the right-hand box of Figure  5.1. Basically, what does a lawmaker hope to achieve? What kind of change does he or she want to make? As we discussed previously, there is an ordinal feel to this framework of policy ends, ranging from instrumental to conceptual ends. Instrumental ends include taking a position, allocating resources, selecting tactics within solutions, and choosing among solutions. The instrumental dimensions tap into policy questions and processes that are oriented to the end goals that have dominated prior thinking on research use. Attention tends to gravitate toward a specific bill in this term of the legislature or an issue that must be dealt with in the short term. Conceptual ends are longer term in character and deal with questions and processes that likely will extend across legislative sessions. They tap into questions of greater consequence or enduring importance such as placing issues on the front burner, getting people to think differently about a set of issues, and raising a new topic or an innovative approach. These are the big-issue items that legislators work on for years that will inform a policymaker’s longer-term success. Karen heard legislators discuss some 122

H ow P olicymakers S ay T hey U se R esearch

of these conceptual ends with great gusto as they described their most significant policy achievements. Legislators told stories of it taking 25 years to enact a smoking ban and multiple sessions to establish a children’s commission. Or think about the concept of a universal guarantee for health care. This topic was introduced into the United States at least a century ago during the Progressive Era as a few European countries began to introduce the first elements of what we now term a social safety net. It was not for decades, however, that the issue became an acceptable topic of discussion and even longer before it was taken seriously. Research Use as a Means to an End Now we move to a second logical starting point for why policymakers turn to research (if they employ it at all). When lawmakers go about their legislative tasks, one use of research evidence is as a means to help them achieve a policy end. For example, when discussing the contributions of research, they often mentioned means such as using research for persuading or educating colleagues, for earning the trust of colleagues, for enhancing policy dialogue, and so forth. These means to accomplish an end can be distinguished from the end result of their efforts. This is a natural distinction when one thinks about it, and a critical one that we initially neglected but now have built into our thinking. As displayed in Figure 5.1, policymakers use research as one of the means in the lefthand box to achieve one of the ends in the right-hand box. Again, our ordering of means is a way of distinguishing between those that are more short term, tangible, and incremental in character, as opposed to those that are more long term, abstract, and enduring in nature. An instrumental means constitutes the daily fare of persuading colleagues and political adversaries to one’s point of view, educating colleagues or constituents, or casting aspersions on the research or motives of political adversaries. They tend to be immediate uses regarding a specific issue or a particular piece of legislation. The second type of means, conceptual means, can be thought of as tools to be used in their everyday policy world to achieve longer-term and more remote ends. They are of a different order because they typically demand a slow accretion of effort over time and may require ongoing debate and consideration before gaining acceptance. For instance, earning the trust of one’s colleagues or improving the policy process does not happen overnight and takes some time to accomplish. As displayed in the left box in Figure 5.1, persuading and educating are considered instrumental means, whereas earning trust, enhancing dialogue, and enlightening understanding are considered conceptual means. Asking questions is either an instrumental or conceptual means, depending on the policymakers’ intent. We start by describing the instrumental means. Persuading Legislators reported employing research to convince others to support one’s view on a specific issue and to anticipate and counter opposing views. Research was used to help legislators to construct and formulate persuasive arguments, which is an important and unique aspect of how information is used in the policy culture. Policymakers use research, not only for developing their own positions, but also for persuading others to their point of view. Therefore, because policymakers use research in an influence-driven context, it is not surprising that one of the prominent uses is for influencing others to achieve their long-term ends. 123

N ew E ra of T heory , R esearch , E valuation

Educating Policymakers reported that an important part of their job is “educating” their colleagues, constituents, and the press. Educating, as used here, has the intent of making an idea or issue understandable by concisely explaining it or providing an example of how to clearly communicate it. In Karen’s study, policymakers differentiated “educating” colleagues from “persuading” them to adopt a specific point of view. One of the differences is that, for the purpose of persuasion, legislators relied on research that supported their positions, whereas for educating, they also used research that did not support their priors. Asking Political Questions When legislators have a grasp of research evidence, they report using this information for politically motivated purposes. Sometimes, they ask questions to which they have ready answers to bolster their own positions or that of colleagues. This is a way proponents can employ research to “make points”. They use their knowledge to develop leading questions to cast aspersions on the credibility or completeness of oppositional research or to point out the hidden motives or obscure agendas of adversaries. In policymakers’ parlance, these leading questions are known as “gotcha” questions that are intended to embarrass colleagues for political purposes. This next set of means are conceptual means. In contrast to instrumental means, conceptual means are bigger ideas or concepts that are longer term in nature. Earning Trust Legislators use research to gain and maintain a reputation as a knowledgeable and credible information source, either in general or in an area of specialization. For policymakers to build a reputation as a trusted information source, they need to be fact-driven in the way they approach their job and selective in using data respected by all sides. Becoming known as a reliable information source helps legislators earn the confidence of colleagues, which is critical to influencing policy ends in the relationship-based culture of policymaking. Enhancing Dialogue Research can improve the policy process by creating a climate that improves the general level of discourse and contributes to constructive debate and dialogue. With access to believable and credible research, policymakers can more easily engage in the negotiation and compromise that it takes if they are going to solve problems and get things done. Research evidence can be used to winnow down differences of opinion, which enhances the prospects that common ground will be found and concrete action will be taken. Consensus can be arrived at when research evidence comes from a source that is perceived as credible by political adversaries, or when research coming from different ideological perspectives reaches a similar conclusion. Enlightening Understanding Research can contribute to policy ends by providing policymakers with a deeper, more nuanced understanding of complexity regarding the nature of issues, why they are 124

H ow P olicymakers S ay T hey U se R esearch

important, and how to respond. This enlightenment function is thought to be the most important influence of research because it can provide a conceptual grounding for what to do, how to do it, and why (Albaek, 1995; Henig, 2008; Hird, 2005; Nutley et al., 2007). Asking Policy Questions When legislators are familiar with research evidence, it can help them frame questions that can clarify the underlying root causes of the problem and the implications of various policy alternatives. Policymakers told us that they try to inform themselves and their colleagues by the questions they ask in committee hearings, in meeting of their caucus, and in private discussions with their colleagues. These are the “Have you thought about this” kinds of input that transform the debate or shift it in a new direction. This is perhaps the broadest use of research and is intended to elevate the level of discussion within the political body from a debate to a dialogue. It is a marvelous thing to observe when it happens. Thinking About the Connections Between Means and Ends We outline the entire research use model in Figure 5.1 to illustrate the implicit relation between means and ends and to highlight the fact that instrumental and conceptual categories apply to both means and ends. This figure helps convey and fully recognize that policymaking is a dynamic process. For example, persuasion is a means that can be used to influence multitude ends, such as thinking about long-term awareness shifting or it might be directed toward allocation shifting of public dollars being voted on that day. The left and right boxes are independent of one another and yet intimately intertwined. Positioning them as such in the same figure  makes sense irrespective of this lack of a direct connection between a specific means and a specific end. It implies a reasonable sense of sequence that we first focus on means as an antecedent to achieving a policy end. An inherent logic is preserved. Let us illustrate our arguments with another vignette. Say we are proposing a guaranteed child support payment program where the government would establish child support orders for noncustodial parents as a percentage of their income. The government would guarantee these payments if the amount owed were not collected from the absent parent, no matter the poverty status of the recipient child and parent. Such a concept would have no precedent and would represent a radical expansion of public responsibility. Thus, it would be controversial and would likely generate skepticism and heated partisan debate. In this scenario, our policymakers are presented with something beyond a run-of-the-mill instrumental end and something better thought of as a conceptual end in the right box. In case the reader thinks the example is far-fetched, such a proposal was made in Wisconsin during the 1980s and federal approval was forthcoming to try the concept out. This proposal would require a drastic change in thinking before such a dramatic proposal would be considered for enactment on a wider basis. One means to advance this proposal would be research provided by a source who had earned the trust of colleagues with a standing as someone sensible and not a wide-eyed radical. They would trot out research, particularly credible research, to say something like “What I am supporting is not insane. I am not insane” or in the words of one of our study respondents, I am not known “to just kind of throw garbage out there”. Research on the proposal’s economic feasibility might be introduced as a means for enhancing dialogue around the issue or 125

N ew E ra of T heory , R esearch , E valuation

to open the door to compromise and consensus. Empirical data may take some of the wind out of political conflicts. It is a way of saying, “Let’s focus on what the research says rather than on our political instincts or mere partisan advantage to see if we can make any progress” or in the words of one study respondent to “move dialogue away from ‘philosophical or political’ posturing and toward a ‘problem-solving exercise’ ”. Research and science can play the role of a conversation starter and consensus builder. This revised model raises the question of whether a deeper and more nuanced understanding of research use can move us forward. Karen’s respondents suggest that prior thinking had been truncated and insufficient. We know that using research as a guide to policy decisions is a long-standing conundrum that is not easy to understand and may be difficult to resolve in the short term. Still, we hope we have moved the needle forward just a bit.

Discussion What policymakers told us about how they used research raised several questions. First, they intimated a broader array of uses than suggested by our priors. Could policy researchers be looking for research use in the wrong places? Could studies be focusing too much on the same-old, circumscribed uses of research, such as forming or justifying a position, rather than on fresh uses such as asking questions and educating others? Second, researchers may have been looking for research contributions for the wrong reasons. Has the reason for research use too often focused on its influence on individual decision-making rather than looking for whether policymakers use it to earn the trust of colleagues in a relationship-oriented, reputation-based culture? Finally, researchers may have been looking for research contributions for the wrong purposes. Have studies been slanted more to how research informs the end product—a specific policy or practice—and less on its contributions to the policy process itself? Could understanding of research use be enhanced if studies incorporated the contributions of research to civil, problem-focused dialogue and to negotiation strategies that can enhance collaboration and compromise? Pragmatically, these findings suggest that understanding research use begins with clearly defining terms, designating the policy community of interest, and engaging policymakers to identify their purpose for using research. For instance, are they interested in elevating a new and innovative issue they wish to place on the policy agenda, perhaps with less concern about its cost? Or are they seeking a rapid, rigorous, and reliable rationale for deciding whether to support an issue that is coming up for a vote? When thinking about working with policymakers, it helps to know who your audience is, what they want from you and in what form, and how they intend to use your input. Purpose matters. If a policymaker plans to use research to raise questions or to educate their colleagues, constituents, or the press, they may want information, not just from a single study, but from a review of relevant literature; this broader background allows them to offer a more enlightened understanding of the issue and to feel more comfortable responding to questions that arise. If the purpose is to earn a trusted reputation among colleagues, policymakers will want to know how rigorous and reliable the research is and how often it has been replicated. If the research will be used as a bargaining tool in political negotiations, policymakers will want to understand, not only the scientifically best policy response, but also how far a policy response can stray from the ideal that might have

126

H ow P olicymakers S ay T hey U se R esearch

been tried and evaluated in “hot house” conditions. According to a sitting state senator (Olsen, 2019): “Don’t let perfect be the enemy of good”; he went on to explain that even when research does not contribute to the best decision, it still may contribute to a good decision. All things considered, lawmakers juggle two concepts simultaneously when thinking of research use. They recognize that research is important for achieving policy ends, such as deciding what issues should be placed on the policy agenda, whether to vote for a legislative proposal, and how many resources to allocate to it. But they also use research as a means to achieve policy ends in such ways as educating members of their own caucus, asking pointed questions, or earning the trust of colleagues. Thus, lawmakers intuitively juggle ends and means when thinking about when, why, and how to use research evidence. The theoretical ideas presented earlier are provocative but have obvious limitations, four that we mention here. First, as with any typology, one inherent shortcoming is that we may be missing categories or the categories that we present may not be mutually exclusive. This can muddy understanding in unknown ways. Second, policymakers may have different motivations and use the same research finding for different purposes that fall into different categories (Nutley et al., 2007). As we point out, policymakers can use research for asking questions, some that serve a direct policy purpose of coming to a better understanding of the policy problem or viable response. However, questions can also frame research for political purposes such as embarrassing their colleagues or criticizing the credibility of the information they offer. Real life is always more protean and organic than the way it is depicted in artificial conceptual maps. Third, our typology provides only a snapshot of different purposes for which policymakers use research, rather than a process model where research that starts with one purpose ends up being used for other purposes as it winds its way through the policy process (Boaz & Nutley, 2019). For example, it seems plausible that research first provides a larger context for thinking about an issue before it shapes a policymaker’s position on an issue and eventually is used for persuading colleagues to adopt a similar point of view. Process models present intriguing possibilities but from a theoretical vantage point, it may be necessary to first identify the still shots of research use before stringing them together into a movie-length narrative of how research flows through the policy process. Finally, our theoretical thinking draws heavily on Karen’s study, which has several limitations. The shortcomings of the theories examined in the study may be an artifact of their narrow focus on different dimensions of the policy process, though each theory did address the contribution of research. Albaek (1995) specified the use of research for taking concrete action, Kingdon (2011) discussed its role for setting policy agendas, and our theory (Bogenschneider, Corbett et al., 2019) examined the purposes for which policymakers use research. The data in Karen’s study derived primarily from state policymakers who differ in unknown ways from policymakers in other jurisdictions, at other levels of governance, and in other policy communities. However, others who have reviewed the literature suggest that the forces shaping research use at the local level are similar to factors among national policymakers (Nutley et al., 2007); if so, perhaps our findings have some general applicability. Also, we do not know if legislators’ self-reports of research reflect their actual behavior. We are encouraged, however, by the triangulation of their reports in our study using different analyses, data types, methods, and sources (see Bogenschneider, Day, & Parrott, 2019).

127

N ew E ra of T heory , R esearch , E valuation

Despite all the progress that has been made in understanding research use, we have been left with “blind spots” (Tseng, 2016). Looking through the lens of policymakers themselves has revealed research contributions that previously had been invisible to the “naked eye” (Weiss, 1999, p. 216). Most of all, we see that research is used in many ways that normally have been hidden from view. This gives us hope that policymakers are a receptive audience and that researchers could arouse that receptivity if our theories could shine a brighter light on why policymakers look to research evidence. Some of the content of this article was adapted from a previous publication with permission. Copyright © 2019, American Psychological Association Bogenschneider, K., Day, E., & Parrott, E. (2019). Revisiting theory on research use: Turning to policymakers for fresh insights. American Psychologist, 74(7), 778–793. https://doi.org/10.1037/amp0000460

128

6 CONFIGURING THE RESEARCH AND POLICY COMMUNITIES Creating an Archipelago Karen Bogenschneider and Thomas J. Corbett

Prior theories have envisioned two or three big communities, often imagining them as large islands composed of rather homogeneous inhabitants. Instead, community dissonance theory envisions metaphorical archipelagoes that more accurately configure chains of related smaller islands, each bound together by common interests and distinctive cultures. These islands are inhabited by a complex array of research producers, research consumers, and intermediaries who bring a rich cultural diversity of professional preparations and institutional affiliations that shape how they think and act. To further complicate matters, professional and institutional cultures are contextually specific and interact in ways that can complement or contradict each other. We can be blind to the obvious, and we are also blind to our blindness. Daniel Kahneman (2011, p. 24) Inwardly we know that institutional values and commitments are important. But in contemporary society, it is difficult to find a vocabulary and grammar for even beginning to talk about such an old-fashioned subject. H. Heclo (2008, pp. 9–10) To refine our understanding of this chasm between research and policy, we need (no surprise) a better theory. In this chapter, we continue our theoretical journey by defining a set of distinct communities that comprise the research and policy worlds. Prior depictions typically have posed two or three big communities, often imagining them as large islands composed of rather homogenous inhabitants. This strikes us and others who have thought about it as overly simplistic (Contandriopoulos et al., 2010). Instead, we prefer to envision our world of interest as metaphorical archipelagoes, more accurately seeing our research and policy types as chains of related smaller islands in close proximity separated to different degrees by sometimes rough waters. This more complex image replaces the two or three larger land masses of researchers, policymakers, and practitioners or policy administrators that are typically found in prior theoretical depictions. Various inhabitants live on the dry land of this constellation of islands that make up our more complicated view of the research and policy worlds. Each island is distinct, sharing common interests and values that bind inhabitants together and provide a sense of identity. Thus, these are our communities. The members of each support one another 129

N ew E ra of T heory , R esearch , E valuation

within distinct cultural parameters. Of course, they try to relate to those tribes living on adjacent islands, but the waters isolating them render mutual understanding difficult at times, though not necessarily impossible. Aggregating the research and policy communities in two or three large, somewhat homogenous groups was simpler to understand and communicate to others but, in the end, tended to obscure important distinctions that we are trying to appreciate more deeply.

Culture and Community Throughout, we return to our singular obsession: Why do researchers and policymakers encounter such trouble communicating with one another? Presumably, each side has a stake in what the other does. Those doing public policy need research to ensure that their decisions work as they intend, which depends, at least in part, on an empirical foundation. At the same time, researchers can profit from knowing the kinds of management and outcome uncertainties that confound policymakers, which raises questions of empirical and theoretical consequence. In theory at least, such feedback ought to serve as critical input for refining existing research efforts as well as formulating future-oriented analytical agendas. These mutually reinforcing needs would seem to support a natural reciprocity between the major groupings in what Caplan’s (1979) termed his “two communities theory” or among those in what Shonkoff’s (2000) called a “three culture” perspective, or what Bogenschneider and colleagues initially described as a “three-communities” network (Bogenschneider, Olson, Mills, & Linney, 2006). Even though the inhabitants of each community in these models operate quasi-independently, presumably there are times when each will benefit from interactions with the other. Researchers have a product that policymakers should find useful, whereas those making public policies control the levers of power that some researchers, at least, want to influence. The way the world works, unfortunately, is a lot messier. If it were not messy, we would not be writing this book. In the real world, information can be highly subjective, data subject to diverse interpretation, and facts far more malleable than we would like. This simple insight is one reason why the hopes for a science-driven policy world that arose in the 1960s never came to fruition. What we lump together as evidence, whether called information or data or facts or science, is always filtered through real human beings. Even the most cognitively sophisticated and well-educated of us bring blind spots and ideological priors through which we organize our worlds. The reality of individual A is not necessarily the reality of person B. This is not a psychology text, so we will only briefly touch upon bias and systematic errors in thinking that affect perceptions of policymakers. For example, according to cognitive psychologists, people are typically overconfident about their understanding of those who inhabit other communities with whom they have little contact, but these intuitions tend to be “more tidy, simple, predictable, and coherent” than reality (Kahneman, 2011, p. 204). These misunderstandings mean that researchers and policymakers do not easily interact or, worse, intentionally avoid one another, primarily for reasons that are given little thought and remain largely uninvestigated. When they do engage one another, misunderstandings often strain the quality of communications to say the least. Miscommunication and mistrust occur even during interactions initiated with the best of intentions. The parties appear to march to different drummers, speak distinct languages, and see the world through unique lenses. A significant part of the miscommunication stems from oversimplifying a world about which the inhabitants of each island has scant 130

C onfiguring the R esearch and P olicy C ommunities

understanding. In regard to policymakers, this means they are all lumped together as nasty, mediocre, pork barreling lapdogs (Mayhew, 2011), which obscures many meaningful differences among them. Such tendencies are part of the human condition, but in no way suggest that such misapprehensions are immutable to change. One of the first steps toward change is that we must get a better handle of the topography of the “other”. Our biased thinking of the “other” derives from what Kahneman (2011) has popularized as fast (automatic) thinking and slow (effortful) thinking. Our fast, more intuitive thinking results in several systemic blind spots, basically biases that recur predictably in particular circumstances and are difficult to avoid without resorting to more deliberate and enhanced slow thinking. One bias that affects how the public thinks about policymakers is known as “hindsight bias” (Kahneman, 2011). Hindsight bias is the illusion that we accurately remember the past. After we adopt a new view, it is hard to recollect how little we knew before we changed our mind, sometimes called the “I-knew-it-allalong” effect (Kahneman, 2011, p. 202). This bias is particularly pernicious for policymakers because the tendency of observers is to evaluate the quality of a decision, not by the research evidence that was used or the soundness of the process, but by what we know later about whether the outcome was good or bad. This means that policymakers tend to be given too little credit for good decisions and too much blame for bad ones. When the outcomes are bad, policymakers are likely to be blamed for “not seeing the handwriting on the wall—forgetting that it was written in invisible ink that became legible only afterward” (Kahneman, 2011, p. 203).

Key Concepts 6.1.  Biases and Systematic Errors in Thinking About the Policy Community 1. People are overconfident about their understanding of those who inhabit other communities with whom they have little contact. These intuitions tend to be “more tidy, simple, predictable, and coherent” than reality (Kahneman, 2011, p. 204). For policymakers, the tendency is to lump them altogether as nasty, mediocre, pork barreling lapdogs (Mayhew, 2011), which obscures many meaningful differences among them. • “Hindsight bias” is the illusion that we accurately remember the past. For policymakers, this means the quality of their decisions are assessed, not by the research evidence that was used or the soundness of the process, but by what we know later about whether the outcome was good or bad. This means that policymakers tend to be given too little credit for good decisions and too much blame for bad ones.

The impediments of biased thinking intensify when like-minded people on the same island reinforce one another. There is a form of herd mentality operating. Thus, our worldviews are informed by the communities with which we associate, some by necessity (such as a job) and others by choice (often grounded in prior training, experience, or predilection). Each of these communities has its own norms and values and rules and languages that we refer to as culture. The importance of the concept of community 131

N ew E ra of T heory , R esearch , E valuation

is paramount since each community operates within the constraints of a culture that intimately shapes individual worldviews, often in ways that are not fully recognized or appreciated. Neither members of the research community nor the policy community has much of a chance of communicating productively with each other until they understand the community of the other. Next, we introduce these concepts of community and culture in some depth. We believe they are the foundational keys for improving the connection between research and policy, and for building evidence-based policy. We explore the professional and institutional foundations for why these two communities find it so difficult to relate to one another. We formally begin to erect a conceptual framework—our community dissonance theory—for thinking through the key dimensions in which each distinct group might view the world. Granted, any framework is simplistic, yet we hope our effort offers a starting point toward better understanding of why effective communication across the research-policy divide is so stubbornly problematic. Until we better appreciate the forces that separate and divide researchers and policymakers, we cannot effectively construct strategies for bringing them together.

The Foundations of Our Community Dissonance Theory Nathan Caplan (1979) introduced the two-communities theory, a familiar antecedent of the community dissonance theory that we develop more fully here and in Chapter 7. In our refinement of this theory, we aim to deconstruct in some detail the cultural disconnect that impedes optimal communication and trust between researchers and policymakers. Our extension of community dissonance theory is anchored in the concept of distinct communities that too often reflect incompatible or at least disconnected cultures. That is, in its simplest form, the inherent character of each community is quite different, and in ways not always apparent to the inhabitants of the group. Members of these distinct communities see things differently and, therefore, act differently. And much like a fish that does not comprehend the water in which it exists (Doherty, 1999), members operating in each world may neither appreciate their own context nor the context of others with whom they are interacting. Caplan (1979) theorizes that underutilization of social science research results from limited understanding and communication between social scientists and policymakers, each representing a distinct community and understanding of their own world. Each has different goals, interests, information preferences, communication styles, and reward systems. In a related theory, Shonkoff (2000) proposed three distinct cultures, as opposed to communities, but the meaning is essentially the same. On the research-using side, he divides people into two subgroups—those who officially make policy (typically policymakers and analysts) and practitioners (typically those who manage and deliver human services). In 2006, Bogenschneider and colleagues used group distinctions analogous to Shonkoff’s—researchers, policymakers, and policy administrators (Bogenschneider et  al., 2006). However, they did so more systematically by laying out the behavioral and attitudinal consequences of community affiliation that are relevant to knowledge ­brokering—information needs, work culture, and writing preferences. In effect, their work at that time began a process of detailing the specific domains in which cultural disconnects might be understood. The two-communities and three-cultures concepts held promise, yet they still fell short in at least three ways. First, they divide the world into a limited number of groups— research types and policy types; or researchers, policy types, and practitioners. While 132

C onfiguring the R esearch and P olicy C ommunities

global groupings keep things simple, too many obvious nuances may be obscured or overlooked (Boaz, Davies, Fraser, & Nutley, 2019a; Contandriopoulos et al., 2010). As we discuss next, the real world is far more complicated than such simplified depictions. There are important subgroups in both the world that produces research and the world that uses research. They need to be identified and elaborated. We miss much by overlooking or obscuring these subgroups and the roles they play in the doing of research and public policy. Second, community and culture were included in previous theories, but a deeper understanding of their meaning is needed because they are central premises to our main thesis. We know from earlier studies that belonging to an institutional community shapes attitudes, behaviors, perceptions, and sense of self. For example, in policy settings (Caplan et al., 1975; Mooney, 1992; Rosenthal, 2009), the underutilization of research has been attributed to institutional obstacles such as political considerations, established patterns of information gathering (Webber, 1986), and increased partisanship (Binder, 2003; Hird, 2005; Peterson, 2018). Are these organizational influences the only forces at work or do we need a broader conceptual net for understanding what is going on here? In fact, where one works or resides institutionally does not tell the whole story. The intellectual and social preparation that one absorbs through one’s educational training also exerts a profound influence. In his examination of researchers and practitioners, Small (2005) ignored institutional culture and attributed miscommunication to professional socialization that dictates what ends social scientists seek, to whom they respond, and what behaviors are considered appropriate. In effect, we must remain cognizant both of institutional culture, the omnipresent force exerted by one’s institutional setting, and professional culture, a powerful antecedent force that begins to shape personal dispositions and behaviors earlier in one’s life trajectory. Third, other theories that have recognized the existence of distinct communities tell us little about how any adverse effects associated with institutional culture or professional culture might be ameliorated. The culture in which one is immersed can be enriching, but it also contains a downside. It can narrow the way in which individuals see the world and their professional role in that world. Possibilities are missed. Miscommunication and misunderstandings can arise. And such failures go unaddressed because those involved fail to see any other way of doing business or any other lens for looking at others. This is the “we’ve always done it this way” trap. Culture can be a myopic condition with far-reaching consequences. As put by Nobel Prize winner Daniel Kahneman, “We can be blind to the obvious, and also blind to our blinders” (2011, p. 24). In this revised theory, we attempt to push thinking forward by addressing these theoretical shortcomings. Community dissonance theory attributes underutilization of research to a communication gap between the communities of researchers and policymakers who engage in different core technologies and work in disparate institutional settings. Communication breaks down because each community operates within distinct institutional and professional cultures that result in different decision-making processes, interactional preferences, favored epistemological frameworks, dominant influence loops, focal interests, and salient stakeholders. We turn next to defining  the key concepts of our theory. By community, we mean the group within which a given professional relates most closely. By culture, we mean a whole set of norms, values, reward systems, and operating protocols and premises that shape how people view their role in the world and inform how they act on a day-to-day basis. We introduce an elaborated definition of the notion of professional culture. This concept grows of out of professional socialization, education, training, learning on the 133

N ew E ra of T heory , R esearch , E valuation

job, and lived experiences. Professional cultures affect attributes such as intellectual substance along with skills and behaviors that comport with standard practices for getting their work done (Chao, O’Leary-Kelly, Wolf, Klein, & Gardner, 1994). Because professional culture functions quite differently from institutional culture, it plays a distinct formative role in the creation of one’s dominant cultural milieu. We also explore the related phenomenon of institutional culture to look for clues as to the importance of this concept in affecting how professionals perform their job and make decisions. We begin by turning to how our interest in culture first began.

Stumbling on the Importance of Culture in the Policy World At the very center of theoretical thinking is this inescapable notion of culture. Tom first became aware of the power of culture during his Peace Corps service in the 1960s (Corbett, 2020). As a recent college graduate, he was given some training in community development and agriculture, and then dumped into a rural village in India. The shock of new and radically different customs and rules of behavior was profound, something all the young volunteers felt deeply. You can be trained about what to expect, but reality imposes a much deeper understanding and appreciation of the power of culture over one’s daily life. After returning to the United States and obtaining his first advanced degree, Tom secured a position with the Wisconsin Department of Health and Social Services (Corbett, 2018a). It was at a time that previously separate program entities had been integrated into a mega or super-agency. The purpose was to improve communication and collaboration across programs that theoretically ought to be cooperating. The service systems being integrated were income-support services for low-income families and human-service efforts focused on vulnerable families. That kind of organizational marriage made perfect sense from a management point of view. In practical terms, however, the institutional marriage of these two systems did not work well at all. It turned out that mandated shotgun marriages between programs with different cultures were easier to propose in theory than to operationalize in practice. Welfare programs were staffed by personnel who focused on administrative issues and making sure that eligibility and computational errors were kept to a minimum. Human service programs focused more on client relationships and changing human behaviors. These were distinct core technologies that demanded different worldviews to be successful. Tom observed the daily conflicts that occurred between organizational inhabitants drawn from each system, which once again augmented the importance of cultural awareness. Some two decades ago, Tom’s growing appreciation for the power of culture began to crystallize. This happened when he worked with a small team of academics consulting with officials from Kenosha County, Wisconsin. This extraordinary team helped local officials conceptualize and then put together an innovative, integrated human service model (Corbett, 2018c). This innovation blended the then-cash assistance program for poor families with children (Aid to Families with Dependent Children) with various programs drawn from the workforce development systems and related services. As such, it represented one of the earliest attempts at systems integration by signaling that workforce preparation was to be a truly integral part of any welfare reform strategy. Despite the best efforts of the consulting academic team, which included wellknown poverty scholars such as Lawrence Mead and Michael Wiseman, early planning efforts to blend agencies from the workforce and welfare worlds did not go well 134

C onfiguring the R esearch and P olicy C ommunities

(Corbett, 2018a, 2018c). Communications across these two systems largely failed, misunderstandings and mutual suspicions emerged, and collaborative activities sputtered amidst accusations of incompetence, if not willful sabotage. The local welfare officials persisted, however, and eventually did assemble a one-stop agency that fully blended the welfare and workforce development functions, an innovation that gained international fame and that served as a model for many other sites in both the United States and abroad. While the project was a remarkable success, the reality of doing systems integration proved to be a lot more difficult in practice than on paper. Upon reflection, they concluded that the interinstitutional friction observed might be attributable to what they termed, at the time, conflicted institutional philosophies (Corbett, 2018a). Briefly put, the welfare and workforce systems organized the world in different ways. They responded to different constituencies and stakeholders. They organized their tasks and activities around distinct core technologies. They defined success in separate ways. In short, each system’s raison d’être, or reason for existence, made collaboration challenging. At the risk of oversimplification, the welfare system focused on low-income families and whether they had sufficient economic resources. The workforce system typically concentrated on employers and on meeting their labor force needs. The institutional antennae of these respective systems were simply not oriented on the same things. Tom and his colleague, Jennifer Noyes, later worked on human services integration around the country and in Canada (Corbett & Noyes, 2008). They came to realize that conflicts in institutional cultures across programs represented the greatest impediment to success in realizing integrated human service systems (Corbett, 2018c). The real issue was that the planners and managers proposing the match were blind to the depth, character, and intractability of cultural conflicts that threatened this prearranged programmatic marriage. Managers and workers from each system blamed their peers in the other system of duplicity, ignorance, or even worse sins. Slowly, by virtue of repeated experiences trying to foster cross-systems collaboration, we began to fully understand that even well-intentioned policy entrepreneurs always function within institutional cultures that shape how they view the world and their professional role in that world. Those cultural constraints can undermine even their best efforts and highest ambitions. In the mid-1990s, both Tom and Karen had stumbled across more examples of the power and pervasiveness of culture. We learned through trial and error that culture, much like an invasive species, informs the kinds of information each culture needs and how they prefer to receive it. Karen was riding the wave of having planned six successful Family Impact Seminars for state policymakers on topics ranging from child support to welfare reform. When state legislators identified interest in a seminar on teenage pregnancy prevention, she turned to a recent meta-analysis of teenage pregnancy prevention programs that used rigorous program evaluation standards to identify five programs that were successful in delaying sexual involvement, increasing contraceptive use, or preventing teenage pregnancy. She convinced the directors of two of these research-based programs to speak with state policymakers in the Family Impact Seminar venue. Given the stellar program and evaluation credentials of these speakers, she decided to expand the invitation list to include members of a state board that focused on the prevention of adolescent pregnancy. Many of these members were the directors and managers of programs that delivered prevention and intervention services to adolescents across the state. As was standard seminar practice, Karen coached the speakers on how to tailor their message to a lawmaking audience. She advised them to use clear and accessible language to describe their programs, weaving in an appropriate mix of program detail and 135

N ew E ra of T heory , R esearch , E valuation

policy implications. When the seminar evaluations were tallied, Karen was surprised to find lower ratings than usual. An enterprising graduate assistant reanalyzed the evaluations and noticed a bimodal pattern of results—some participants gave the seminar high marks and enthusiastic endorsements, but others expressed disappointment and dissatisfaction. The evaluations divided along cultural lines. A state legislator offered the accolades Karen had come to expect in seminar evaluations: “Your format was excellent. I have very little free time to attend briefings, but can always work around an hour and a half briefing scheduled in advance.” In stark contrast, program directors and management staff reported that the information was too superficial for their taste. They criticized the short length of the seminar and suggested that future conferences be up to two days in length. In effect, they complained that we had attempted to do something that seemed impossible—bridging two communities in the same forum. In the words of a program administrator: “Don’t invite politicians and people who deal with the real world to the same seminar; they [politicians] deal more with unreal issues and are not reality-based.” Notably, these disparate comments were from inhabitants of the policy world, but from different islands. They responded differently to the research presented, based presumably on important cultural distinctions that shaped their information preferences. The choice of words like “unreal” and “not reality-based” reveal the depth of the cultural divide. The substantive materials presented were accurate and well delivered. Inhabitants of each community, however, had been accustomed, or acculturated, to expect something quite different. From the legislative perspective, the speakers had met their expectations for a brief overview of the field that provided state-of-the art research on whether a public investment was likely to pay off. However, from the vantage point of program staff, the speakers had missed the mark by failing to specify critical program features and best practices that the programs needed to succeed—details essential for program replication and effective adaptation to local circumstances. Karen’s well-intentioned efforts to bring together two communities that shared an interest in preventing teenage pregnancy had backfired because of a misunderstanding of how the information needs of one community differed from those of another. In hindsight, if she had been more culturally aware, the Seminar planners would have parlayed these differences in culture into differences in information packaging and delivery. This humbling experience inspired Karen’s early “three communities” writing about why there may be no such thing as a competent knowledge broker, broadly defined. Effective knowledge brokering is context-specific, requiring an understanding of the culture out of which research emerges and the culture toward which it is directed. When a researcher attempts to communicate with a policymaker, it behooves him or her to know the culture of the policy community to which the target individual belongs. All policymakers are not the same. Examples abound of the disconnect between researchers and policymakers in our work at the intersection of the two cultures. Some researchers never even attempt to cross the divide. Tom remembers the time he asked a highly regarded economist on childcare design and financing to give a presentation for congressional staff. This economist admitted that before this trip, he had never set foot in the U.S. Capitol. Tom was surprised in the moment, forgetting that being a recognized expert in the academy did not equate to any prior contact with the policy world. Karen recalls the Harvard-educated educational policy professor who was asked to speak at a Family Impact Seminar. He asked Karen for advice on his presentation, confiding that over the last 20 years he had frequently

136

C onfiguring the R esearch and P olicy C ommunities

lectured to students and colleagues on education policy, but had never addressed a group of policymakers. Quite remarkably, this policy expert had never given a presentation to those whose decisions he studied—policymakers. Such vignettes exemplify what we learned in painful ways throughout our careers. As noted, Tom stumbled on the concept of institutional philosophy, as he called it, in the 1980s. Later, he often referred to it as institutional culture. Jodi Sandfort (2004) used the term the “deep structure” of an organization to denote a similar phenomenon. And by 2005, in a coauthored paper by Tom and his colleagues, the term institutional milieu briefly surfaced when the authors could not agree on any other label (Corbett, Dimas, Fong, & Noyes, 2005). By 2005, Tom and his colleagues had become convinced that this concept of philosophy, culture, or milieu—whatever it was to be called—was indeed a critical factor in doing service integration (delivering social services to families in a holistic, coordinated manner). By 2006, the thinking of Karen and her colleagues had evolved to the point that they began writing about how understanding the culture in which policymakers operate was a foundational element of effectively communicating research to policymakers (Bogenschneider et al., 2006). At the same time, this important notion of institutional culture, as we called it at the time, remained poorly understood either by those involved in cross-systems integration or by those engaged in knowledge brokering. Still, it was clear that we had tapped into something fundamental. This became increasingly evident as we interacted with agency officials and knowledge brokers in a variety of venues. Time and again, as we introduced the topic of institutional culture, they would affirm (typically with gusto) that this concept appeared to touch on the most problematic barrier to their progress. Exactly what does this concept of culture have to do with how we bring research to policymaking? In truth, quite a bit. Based on our experience, it fundamentally shapes the way public (and private) officials think and act. It shapes their worldview, their perception of self, their relationships with peers, and their interactions with customers and stakeholders. At the same time, we became increasingly sensitive to another seminal cultural influence over how individuals view themselves and others in the work world. We call this professional culture. For some, this concept of professional culture usually precedes the influence of institutional culture and can magnify or confound its effects. We now proceed to flesh out both concepts, starting with the professional variant.

Introducing the Concept of Professional Culture The concept of professional culture grows out of the personal background, education, training, and actual experiences that a person receives in preparation for a vocation or career. Academic research tends to be discipline-oriented, so it embodies numerous inherent signals and rewards that are associated with a certain regimen undertaken to prepare future members of the academy for scholarly career trajectories. Think back, for a moment, to the Middle Ages when guilds were prominent. Youth apprentices in a guild would learn the specific technical skills necessary to master a certain trade. But they would also incorporate a form of socialization that developed the values, attitudes, social skills, networks, and etiquette appropriate for a member of that specific society. Apprentices were trained by a master in the technical skills of a trade and mentored into a whole new way of life. You became a member of a community, not just a purveyor of a specific skill set.

137

N ew E ra of T heory , R esearch , E valuation

The training of professionals into a way of life continues to this day. In his book on institutions, Heclo (2008) argues that to enter a professional mindset is to embrace a “calling”: In this view the professional embodies a deeply held sense of responding to a call of worthwhile purposes beyond oneself. That calling and their response are identity forming. Being a member of a profession means more than acquiring a body of theory and set of techniques, though it does mean that. It also means a principled way of being in the world. To invoke claims of professionalism is to appeal to standards that go beyond technical knowledge or personal career success. One enters a professional community whose common technical skills are understood in light of ethical standards and fiduciary responsibilities that are endorsed, monitored, and enforced by one’s peers. (p. 134) Contemporary examples abound of professional preparation into a way of thinking and being. We are all familiar with the training regimens for military personnel. There is a boot camp experience or the plebe year for the service academies. The operating concept is to break down the way a candidate thinks about the world when they enter military service to build up in them a new way of thinking about things. Some academic professions simulate, at least in part, the military experience. Law school devotes the first year of study to getting future members of the bar to think like lawyers. This is, in part, a pedagogical process that goes well beyond the mere acquisition of substantive knowledge about the law or how to function in various legal situations. The first year is dedicated, in part, to framing how prospective lawyers organize facts, think through questions of law, and express themselves in the unique vocabulary of attorneys. Lawyers, who frequently populate lawmaking bodies, learned that “precedent and the language and intent of statutes are what count” (Weiss, 1999, p.  223). That first year of law school is as much about how to think as it is about what to know. Likewise, in training to become a social worker, you first embrace the culture and then move on to acquiring the requisite skills. In many programs, the first year of graduate work is oriented toward grounding students in a basic understanding of what it means to be a social worker; the second year permits students to explore their individual career path, as well as absorb a more advanced theoretical foundation for pursuing their profession. Similarly, the curricula of public policy schools are often dichotomized into technical and professional foci. The technical preparation concentrates on things like microeconomics, planning models, program evaluation methods, governmental structures and processes, and policy analysis. At the same time, these schools often employ lecturers with real-life experience in legislatures and bureaucracies. Some even require students to gain applied experiences through internships or class assignments situated in actual policy settings. These schools recognize that technical virtuosity is only part of the skill set needed to perform in the policy world. Doing policy is part craft, but it is also part art. Knowing the culture of the policy world may be as important as knowing the substance of the issues or the latest analytic techniques. You can be smart as a whip, yet fail if you do not know the “hidden” rules of the game. The point of these examples is that the concept of professional culture is powerful indeed. Preparing someone to function in an academic discipline is not just about intellectual development and the acquisition of disciplinary substance; it is also about socialization into a whole set of cultural norms and mores that shape attitudes about what is 138

C onfiguring the R esearch and P olicy C ommunities

valued and what behaviors are considered appropriate. It is about setting a relational context that governs interactions between members of a kind of guild (the academic world) and the real world outside the ivory tower. Recruits learn a language associated with the discipline and a set of signals about how to govern oneself in ways that comport with professional standards. The real emphasis of most doctoral study in the social sciences, however, is on becoming a researcher, who learns the canons of science as well as the norms and values of the academy. This is more than just learning methods and statistics, though that is critical. It is about thinking like a researcher. This is very close to thinking like a lawyer or a social worker in that the socialization process shapes one’s approach to the world tapping into views such as how we know truth, how we assess competing claims of what to believe, and how we distinguish good input from bad. Consider, for a moment, the whole notion of a null hypothesis. A properly socialized researcher does not accept plausible propositions at face value. He or she is trained to hypothesize that any proposed relationship in the real world is false, until this so-called null hypothesis is rejected by rigorous analysis using conventional standards of statistical significance. Real-World Examples of Professional Culture For those grounded in the culture of science, there is a rather clear demarcation between thinking like a civilian and thinking like a scientist (or an academic). A so-called civilian might well give play to the values and passions that policy issues bring into focus, whereas an academic researcher would (or at least should) give greater weight to scientific evidence to support conclusions. When scientific thinking trespasses into civilian territory, the clash of the cultures can be amusing. Take for example, a wine-tasting party that Karen organized for friends. Most of the partygoers enjoyed tasting the wines, rating each one, and quickly zeroing in on their personal favorites. However, the quantitative scientist among them took the scoring sheets, criticized the rating scale, and then wondered aloud about the standard deviations for each wine and whether the preferences would vary by age and gender. A scientist can entertain any number of hypotheses about what causes a certain outcome of interest. When arriving at a conclusion, however, he or she should be guided by the research evidence. There is a wonderful example from the scholarship of Sara McLanahan, a professor of sociology and public affairs at Princeton University. Early in her career, she embarked on a course of research into what accounted for differences in children’s well-being and their later success in life. Her hypothesis was that, given proper income and other supports, single mothers could raise children as well as mothers from two-parent households. She herself had been a single mom for several years. No matter how she ran the numbers, however, the data did not support her priors. Being a good scientist, she accepted in the end what the research evidence told her. Her intellectual journey was eventually captured in the popular press in an article titled “The Education of Sara McLanahan” (Whitehead, 1993). As these examples illustrate, researchers as a group tend to be cautious. They look to build carefully and systematically on existing theory. They entertain new interpretations only when the preponderance of research evidence brings conventional wisdom into doubt. Great pains are taken to identify and reject spurious relationships. Those grounded in the culture of research are steeped in the mindset that the world out there, beyond the borders of the academy, is replete with alleged examples of causal relationships that are often erroneous. To researchers, people easily look at temporal associations 139

N ew E ra of T heory , R esearch , E valuation

and assume causality. Those immersed in the culture of science know full well that correlation is not causation, and demand empirical support derived from rigorous experiments and analyses before drawing causal conclusions about how the world operates. It is an ingrained attribute that a scholar views the world with suspicion. When an inbred skepticism dominates one’s worldview, what appears true on the surface cannot be accepted as fact. In his undergraduate training at Clark University, Tom recalls sitting with two professors who were discussing a series of experiments involving how different reinforcement schedules affected the learning curve of rodents negotiating complex mazes with variable incentive structures. The research was designed to determine which reinforcement schedules were more efficacious. As Tom observed their dialogue, he was taken by the fact that these scholars stubbornly refused to accept their initial findings at face value. They continued to search for spurious reasons for observed results unconnected to their early results and perhaps even their theoretical understanding of the world. That is, were there any coincidental factors that might explain away what they had observed to date? Perhaps there were visual cues the rats were using that could be controlled in future experiments by randomly rotating the position of the maze. Or perhaps the maze should be cleansed after each trial, so that olfactory cues were not present. Their list of potential confounding factors went on. The lesson Tom absorbed that day was that a “real” scientist never relaxed in the pursuit of truth. One could never be satisfied. A good scientist kept looking for reasons to discount even positive empirical results that supported their prior predictions. Unlike policymakers who sought evidence to support their priors, scholars searched for reasons to discard their priors. Success was always tenuous and uncertain. By virtue of their training, scientists fully embraced the canons of their discipline and the culture associated with being a researcher. Methods may vary, but the procedures of scientific inquiry are well understood. When speaking as a researcher, opinions become more guarded, and conclusions drawn less quickly. There is a sense of responsibility that we bring to our public voices, or at least there should be. By the time the newly minted doctorate enters the work world, they bring with them a whole array of internalized norms and externally accepted values. These should, if they are to remain faithful to their vocation, shape the way they view the world, albeit with some adaptation depending on where they happen to be employed.

Introducing the Concept of Institutional Culture We view the concept of institutional culture as located within a specific agency, program, or organization. Exactly what is institutional culture? Basically, the institutional culture of an organization is a fancy term for the underlying values that inform the way it does business and gets work done. It also embraces the rules, explicit and implicit, that govern relationships among the inhabitants of the institution or program. These values, in turn, shape the way an institution functions and makes decisions. What shapes the so-called culture of an institution? One determinant is “core technology”, the central functions performed by an agency or program. This notion, first introduced by James Thompson (1967) over five decades ago, is still useful today. For policymakers, their core technology is to make laws, often using a sifting and winnowing process that tries to make sense of the multitude of ideas that are floated into the policy process. Policymakers seldom sit down and think up ideas themselves; instead, they find themselves in the position of evaluating the ideas that come to them from 140

C onfiguring the R esearch and P olicy C ommunities

sources such as advocates, constituents, lobbyists, policy analysts, and special interests, all willing to tell them what needs to be done and how it could best be accomplished. Policymakers sort through these ideas, screen them for consistency with their constituents’ views, partisan pressures, and their own ideology and values. Then they decide which of them are meritorious. Decisions on the merit of an idea are seldom static, however. Ideas are tugged at in the policy process with policymakers forced to continually reassess when to compromise their initial position and when to hold firm. It should surprise no one that much of policymaking is an organic process with many actors and factors playing a role. Let us concretize the discussion of core technology with some examples. In the world of publicly provided human services, each program has a distinctive core technology. It can be thought of as the essence of what they do. Some programs essentially issue benefits, such as food stamps or childcare subsidies, which are highly routinized functions associated with formal bureaucracies. Other programs deliver a comparatively routinized service (job club or job search assistance), but also contain service dimensions that require professional judgment and front-line discretion. We think of these as semi-routinized and their institutional environments as being less formal. Finally, we have programs that, for example, intervene in the lives of dysfunctional or challenged families to remedy problems or transform behaviors (e.g., family violence, child neglect). Such programs are fully nonroutinized. Here, typical institutional environments are open and characterized by a strong sense of professionalism, considerable discretion, and a bottoms-up management style. These style differences tell us a lot about what knowledge is sought, how aggressive the seeking is, and who does the searching. In effect, core technology helps shape how the institution looks and the way it functions. Powerful stuff. Academic researchers, by definition, locate themselves in universities. These are highly decentralized institutions with great latitude afforded the tenured faculty. As institutions, universities are slow to change. Edward Zigler (1998) notes the response of an acting president of Yale University, when asked how Yale would fare during a period of rapid change in leadership. The president replied that “Yale had remained great for nearly 300 years not because it was resistant to change, but because it changed only one brick at a time” (p. 532). Ironically, universities are filled with structural and cultural rigidities, yet the day-to-day working environment is one of flexibility and creativity. Researchers essentially operate as academic entrepreneurs in environments of relatively little bureaucracy. They are rewarded for creative, independent thinking, and come to expect faculty input into decisions, whether large or small. Like academics, many of those administering public policy at high levels of executive agencies have doctoral-level training and have developed an area of specialization. These administrators, charged with turning policy concepts into actual programs, operate in bureaucracies structured around rules and regulations. Public agencies tend to be hierarchical and rule-driven, conforming to the Weberian vision (Weber, 1905/2002) of a top-down bureaucracy. The first inclination of bureaucratic officials is not a creative or independent response, as with academics, but rather a response that would be consistent with and acceptable to their superiors (Holzer, 2009). They are more attentive to treating clients uniformly and fairly than in being creative. As we suggest elsewhere, there are exceptions to any rule and one can point out examples where the dominant institutional form does not prevail. Still, the general rule does prevail; those administering policy enjoy less flexibility in the exercise of their professional roles. Karen has found that any report that a state bureaucrat wrote for policymakers had to be approved by those higher on the administrative food chain. Scholars almost never need to submit their academic 141

N ew E ra of T heory , R esearch , E valuation

work to university administrators. One exception is mandatory methodological reviews when human subjects are involved in research studies. Policymakers fall somewhere between the relaxed institutional culture of a university and the rigid form of a bureaucracy. Over the last 100 years, Congress has evolved from more centralization to less (Bimber, 1996). Prior to 1910, power was consolidated in the speaker. However, after the “Revolution of 1910”, power flowed downward to committees and in the 1970s devolved further to subcommittees. However, the power hierarchy in legislative bodies is not static, as exemplified by the case of Congress in 1995 when power reverted to the speaker, Newt Gingrich. The bottom line is that the influence of institutional culture can be pervasive and encompassing. Organizations embody and express values, some explicit and some not. These values are reflected in the rewards that are allocated for appropriate behavior, the leadership style that develops, and the nuanced signals transmitted throughout the organization in so many ways. When organizational culture changes or clashes, incumbents either respond or exit. This happened in many instances as agencies underwent dramatic transformation as part of a welfare reform dynamic in the 1990s. Some saw wholesale staff turnover as programs migrated from a benefits-issuing function to a behavior-changing focus. Institutional culture is a powerful and undeniable force. Key to understanding the force of institutional culture is the awareness that formal rules and observable power relations represent only part of what governs internal behaviors. Inhabitants of institutions embrace norms, acceptable language, and permitted behaviors, sometimes by being rewarded for doing things in the right way but mostly in unconscious ways. A newly minted PhD might look forward to teaching the next generation all that he knows. However, it will soon be clear to him, if he finds himself in a top research university, that teaching is not valued as highly as publishing in peer-reviewed journals. The cues he or she is exposed to in this new culture are unambiguous—research is everything.

The Intersection of Professional and Institutional Cultures All of us, then, bring two sets of cultural influences to the table. One set is framed by our professional or academic training, and the other by the institutional setting in which we work. These influences constitute the contextual framework through which we function. They provide us with signals about the most critical dimensions of our lives as professionals. What salient topics will attract our attention? To whom should we pay attention? How shall we interact in given situations? What rewards will mean the most to us? The list goes on. We argue that each member of the research-producing and consuming communities, as well as the subgroups within these larger communities, brings a distinct set of cultural baggage with them to their professional roles. As we have argued, the influences are drawn from some combination of disciplinary preparation and institutional affiliation. That is, they are drawn from acculturation into a specific institution, as well as from the more permanent norms and values inculcated during the disciplinary socialization process associated with one’s academic preparation and lived experience. At the end of this process in which individuals are prepared to carry out their roles and responsibilities, they become a member of a group of similar individuals or what we call a community. A community is a group of like-minded persons bound together by common perspectives, goals, and institutional constraints. For researchers, some will become a member of the community of the academy in the form of a professorship in a 142

C onfiguring the R esearch and P olicy C ommunities

top research university. For policymakers, some will become members of Congress, some governors, and others political appointees to head executive agencies. Granted, there are short cuts to full membership in a community, but most arrive after much preparation and relevant experience. Individuals rise through academic ranks, through bureaucratic layers, and into political positions of leadership (head of an important committee) by virtue of playing the game successfully over time. They become accepted members of their community. Deviation from accepted norms and behaviors may well lead to one being expelled from a community or ignored into irrelevance. Denial of tenure is a form of expulsion, but there are many ways in which displeasure can be signaled after the granting of tenure. How does all this work in reality? How do these sources of influence interact to make one an accepted member? Take the example of Professor X, who is a tenured professor at a top research university. He or she would bring the academic norms absorbed as part of his or her academic training, shaded somewhat by one’s specific disciplinary preparation. For example, in making causal inferences, most economists feel comfortable using econometric modeling that accounts for intergroup heterogeneity statistically, whereas other social scientists feel more comfortable with experimental designs that employ random assignment techniques or some form of group comparisons where observable or suspected heterogeneity is statistically corrected. In addition to the cultural influences drawn from their academic training, university professors would also respond to the culture of the specific academic institution in which they were located. Being in a top research university would only amplify the signals and reward structures associated with being an academic researcher. Their professional antennae would be sensitively attuned to their research peers—behaviors and investments that resulted in peer-reviewed journal publications would trump activities directed at teaching or public service. Some of this would simply be “in the atmosphere” of a research university. But in other ways, Professor X would know full well that the prospects for advancement, honors, and the coveted corner office would rely heavily on publications, no matter what the institution might say about the relative importance of teaching and public service. For Professor X, professional training and institutional affiliation complement one another. But what happens when that interaction is less complementary? Professor Y may also have a doctorate in the same discipline as Professor X. He or she, however, does research in a university that highly values teaching and public service. Surely, they do not jettison the norms and values that were ingrained during the socialization process while in graduate school. However, the institutional signals and rewards arguably are less complementary and, we would hypothesize, less powerful. Not surprisingly, colleges and universities espousing a teaching mission value teaching. They recruit, retain, and promote based upon excellence in teaching. They highly value student evaluations of faculty performance online or in the classroom. Now, take a third researcher who has similar academic credentials. Professor Z is on a different career trajectory. He or she is professionally situated in a policy organization. The institutional culture here is quite distinct from what one would find in a university setting. Here, the core technology of the organization is oriented toward the policy community and not toward fellow academics. The product is not conducting research per se, but using scientific methods to evaluate public policies and programs, and perhaps advocate for a specific proposal informed by a partisan or ideological perspective. Clearly, the stakeholders differ and accordingly, the criteria for determining both professional and organizational success will vary. 143

N ew E ra of T heory , R esearch , E valuation

We are trying to make a simple point. Some professional trajectories are mutually supportive in a cultural sense. That is, they tend to lead one toward a homogeneous view of the world where few countervailing inputs interfere with the development of a coherent cultural perspective and a fully internalized set of norms and expectations. Some trajectories may be less integrated in the sense that other, perhaps competing, values or ways of looking at things may intrude from time to time. There are ways of accommodating such inconvenient inputs without upsetting the dominant culture, but dissonant cultural signals can influence things at the margins. In Figure 6.1, we lay out a stylized depiction of the intersection between professional and institutional cultures on a more macro level. In this simplified 2 X 2 figure, we lay out the extremes. Along the horizontal axis, we have two professional cultures. One requires minimal preparation as a researcher, perhaps a couple of research courses on the way to a professional degree. The other requires rigorous preparation in research methods and statistical analysis. Along the vertical axis, we posit two institutional cultures. The first is policy-oriented organizations or practitioners that design, implement, and manage programs. The other is top research-oriented organizations that employ rigorous research and evaluation methods. We argue that cells A and D involve the least potential conflict between professional and institutional norms. That is, minimal research training would meld more seamlessly with the institutional culture in an applied policy setting (cell A), just as rigorous research training would mesh more systematically with the institutional norms in a top research setting (cell D). Similarly, we argue that cells B and C offer possibilities for conflict where the extent of research training may clash with the dominant institutional culture. Whether or not such conflict arises depends on the circumstances, but the potential for strain on communication and collaboration is very real. Conflicts may occur when a researcher with rigorous research training works in an applied policy organization, particularly one that has an advocacy agenda (cell B). Similarly, conflicts may occur when a researcher with minimal research training works in a top research setting, such as an Professional culture

Policy-oriented

Institutional culture

Research-oriented

Minimal research preparation

Rigorous research preparation

A

B

Minimal research training

Rigorous research training

Applied policy setting or practitioner

Applied policy setting or practitioner

C

D

Minimal research training

Rigorous research training

Top research setting

Top research setting

Figure 6.1 The Intersection of Institutional and Professional Cultures: A Macro Look

144

C onfiguring the R esearch and P olicy C ommunities

evaluation firm that uses rigorous scientific methods (cell C). The conflict may be greatest when the cultural differences are most extreme, such as a researcher with minimal research training working in a top research university. So far, we have been simplistic in our exploration of the general effect of culture on the use of research in policymaking and management. Let us next turn our attention to the reality of the institutional contexts in which research is both done and used. The world is much more complex than it has been treated in discussions of the research-policy divide to date. The closer we look, the more obvious it is that there exists much more heterogeneity on both sides of the traditional equation. Simply put, there are multiple communities among researchers and policymakers.

Toward a Multi-Community Understanding of Culture There is an old saying that where you sit determines where you stand. Organizational position influences preferences and behaviors. Next we extend our understanding of the complex array of institutional settings in which research production and use take place. As a point of departure, we attempt to create a much broader understanding of the institutional settings in which knowledge is generated and used. Prior conceptualizations were simple. For the sake of clarity, a binary world was proposed where new studies emanate only from our research universities and public policies are determined solely in our legislative bodies. That convenient view of the world made it easier to develop theoretical views, but also served to obscure the complexity of the real world. The real world is much more complicated than that (Boaz  & Nutley, 2019). In Table  6.1, we lay out a more nuanced depiction of institutional settings in which research is created,

Table 6.1 An Illustrative Overview of the Archipelago: A Micro Look Professional Culture Research producers

Research intermediaries

Research consumers

X X X

X X X X X

X X

X

X

Institutional Context University-based settings Tenure track/traditional disciplines Policy/Professional school/Nonfaculty researchers Teaching-focused colleges Intermediary organizations Think tanks Evaluation firms Policy organizations Philanthropic organizations Government trade organizations Public policy settings Government-based research/analysis Front-line service provision Executive agency management/support Executive agency/cabinet leadership Legislative/judicial systems

145

X X X

X X X X X

N ew E ra of T heory , R esearch , E valuation

transmitted, vetted, and used. In effect, this is a look at the intersection of institutional and professional cultures on a more micro, or detailed, level. In a crude and abbreviated way, it is an overview of our archipelago. Along the horizontal axis, we keep our framework relatively simple. We suggest three distinctions in our view of community culture. First, we have research producers, the people who do research, evaluate programs and policies, and plumb secondary data sets to glean new ideas or confirm hypotheses about what is happening in the world. But this world of research producers is far more complex than most have treated it. We are talking about a variety of individuals working in a range of institutional contexts from top research universities to government agencies. The point we must never forget is that the production of research is not the exclusive province of the academy. Research production exists within government as well, including the very legislative bodies we see as primary research users. The second category we term as research intermediaries. These folks also produce research, in a sense, but by identifying it, organizing it, and translating it for the end user. Often, research intermediaries pull together information developed elsewhere, and integrate and interpret it in new ways geared toward the information needs of policymakers and practitioners. Intermediaries may also intuit emerging research questions and initiate analyses to address those issues, but the actual work may be outsourced to others. Intermediaries routinely scan for new studies and perform a range of useful translational functions or tasks that make research evidence accessible to key target audiences. In this sense, government-based researchers, evaluators, and analysts typically serve as intermediaries. They respond to specific questions from those in the policy realm, thereby serving as research emissaries to the policy community, particularly for those policymakers who find it difficult to cross the cultural divide. In general, intermediaries do just that—work across the cultural divide. This role is increasingly irreplaceable as the sheer magnitude and complexity of analytical work become overwhelming. Finally, the third category in our cultural typology is research consumers. Traditionally, some have thought of research consumers as those who create laws—legislators or top appointed political officers. But that view would be far too narrow. It is all too easy to focus on the elected officials who make a new rule or law, forgetting that so many essential functions exist between the promulgation of research and its impact on society. Among the many functions included here are those who implement policies, develop programs, manage agencies, evaluate outcomes, monitor performance, determine legal questions, and interact with customers where ad hoc policy decisions might be made. A law or rule may guide what is done, but much can be left to the discretion and professionalism of so many others. All these implementers need research, though the types of research they need and the ways they use that research may differ dramatically. This chart is a crude and shorthand way of capturing professional preparation (along the horizontal access) and how it interacts with institutional context (along the vertical axis). Though the correlation is far from unified, we believe there is a correspondence between the kind of academic preparation received and the professional orientation assumed throughout one’s career. That is, the academic preparation of research producers grounds them in both the culture and methods of science. They tend to accord greater weight to hard evidence collected through rigorous analysis than through direct experience, isolated vignettes, and subjective interpretation. At the other extreme, research consumers often were exposed to academic regimens that were unrelated to many of the substantive and technical issues that they most often deal with in their future professional lives. Policy positions, for the most part, 146

C onfiguring the R esearch and P olicy C ommunities

demand a more general set of competencies than that demanded of a research scientist. Namely, they must develop a command over the core technologies, politics, target populations, and trade-offs inherent in the content area they will manage or develop rules about. A  variety of professional schools prepare individuals for careers in the human services arena. Most require that their students be exposed to research methods and learn how to use research for policy and practice. Still, these schools do not thoroughly ground their students in what the culture of science is, nor how to think like a scientist, and certainly not how to conduct cutting-edge research. They prepare students in a broader array of essential skills at a greater breadth, but a shallower level of understanding. Of course, this is a gross generalization. Many who wind up in the policy world have deep training in a specific area, such as medicine or science, whereas others have a general liberal arts training or academic preparation in a specific aspect of the policy process such as law, accounting, political science, and so forth. Many legislators feel advantaged by the fact that they have no grounding in any specific substantive discipline. This lack of specialization frees them to listen to constituents and to move easily across topical areas with ease and fluidity. Being a generalist makes it easier for some to see the bigger picture and make important pragmatic connections (Albaek, 1995). It frees them from simply seeing all issues through, for example, a narrow economic viewpoint or an equally rigid legalistic perspective. Our columns, then, provide us with an easy, though admittedly crude, first cut at determining the orientation a person brings to the table. Next, we turn to the specific institutional context in which individuals are positioned. Along the vertical axis, we lay out a range of institutional positions arrayed along a conceptual continuum loosely corresponding to a theoretical research production-consumption scale. Through this depiction, we attempt to provide greater clarity on the variety of institutional settings in which those involved in shaping and doing policy find themselves (see Goodvin & Lee, 2017; Mayhew, 2011). Let us start at the top and move to the bottom. University-Based Settings At the top of the scale, we enumerate the usual actors associated with what we have termed research producers. These research producers are typically situated in university settings. Yet the diverse nature of these institutional settings brings a different institutional spin to what research activities are expected and rewarded. We describe selected examples here. Tenure Track/Traditional Disciplines Many research producers involved in the policy process often, though surely not always, are situated in research universities or other institutions where cutting-edge work is produced. Typically, we tend to think of these top researchers as being embedded in tenuretrack and discipline-focused academic departments such as economics, family studies, political science, and sociology; however, many have appointments in other locations inside or sometimes outside the university that are more policy focused. Notably, being professionally situated in a university known for producing research does not preclude one from being a research consumer in our wider sense of the term. Obviously, the very process of doing good science demands that you regularly consume the output of your peers. All top scholars review the work of their peers, both to learn 147

N ew E ra of T heory , R esearch , E valuation

and to look for new research opportunities. Some scan the real world, or should scan the real world, for ideas about what to look at for formulating research questions. Policy/Professional Schools and Nonfaculty Researchers Not all policy-oriented researchers are in traditional academic settings or in the traditional disciplinary departments. In fact, that has become less likely over time, as more and more are positioned in policy or professional schools. Others may be professionally situated in the academy but not hold tenure-track faculty appointments, serving more exclusively in academic staff positions such as research scientists. This is particularly true of your larger schools where research activities are the dominant expectation. Though Tom taught policy courses for many years at the University of Wisconsin-Madison and helped run the Institute for Research on Poverty for over a decade, his role was that of a senior scientist, not a faculty member. While he did much research, his dominant focus clearly was public policy. This greatly helped him to mediate between these two worlds. Academics also may find themselves doing policy or engaging policymakers in one way or another. They may consult with government, serve on commissions, and accept public positions (at least temporarily and without forfeiting their academic ties). For example, Tom served as a senior policy adviser in the U.S. Department of Health and Human Services working on welfare reform during the early Clinton administration. In effect, he was outsourced to the federal government for a prescribed period of time. Teaching-Focused Colleges Not all policy-oriented academics are conducting research. In Boyer’s now-classic conception of what it means to be an engaged university in contemporary society, the most groundbreaking element of his model was teaching (Boyer, 1990, 2016). Over time, Boyer’s scholarship of teaching has come to mean that research becomes relevant as it is communicated to and understood by others. In Boyer’s words, “Without the teaching function, the continuity of knowledge will be broken and the store of human knowledge dangerously diminished” (2016, p.  69). Faculty in teaching-oriented universities routinely scan the literature to identify classic and cutting-edge studies that will prepare the next generation of professionals to be competent producers and consumers of research. Karen taught college courses on family policy for many years at the University of Wisconsin. Whether teaching at the undergraduate or graduate level, she began the first day of each semester with some trepidation about how the course would go, but her experience on the last day of class was always the same. She never failed to find herself inspired by these eager and enthusiastic learners whose questions sharpened her thinking, whose insights enriched her understanding, and whose passion never failed to give her hope that the future is in good hands. However, the importance of teaching students about policy was never more evident than when Karen was writing the third edition of her family policy textbook (Bogenschneider, 2014). The publisher had sought reviews from over a dozen instructors across the country. They all shared a similar concern: What could they do to motivate and inspire the reluctant students of public policy that populated their classrooms? Students came to class with preconceptions that politics was dirtied and with serious misgivings about getting involved in such a corrupt, influencepeddling, power-driven process. Helping the next generation overcome these misgivings and comprehend the power of politics when people work together to shape a better 148

C onfiguring the R esearch and P olicy C ommunities

society is important scholarship. Teaching is as important a policy function as other pursuits (Bogenschneider, 2014). Intermediary Organizations As mentioned earlier, policy-relevant research is no longer confined to universities. By the end of the last century, the research landscape was much different. When the Clinton team was putting together its welfare reform proposal in 1993–94, where did it most often turn for research, analysis, and evidence to help with its planning? Academics types were brought in as consultants, but the research and analytical competition had grown in number, sophistication, and availability (Smith, 1991). Simply put, policymakers had more options. Tom recalls watching Harvard academic David Ellwood who was tapped to head the planning for the welfare reform initiative in the U.S. Department of Health and Human Services. When an analytical issue needed to be addressed, seldom did Ellwood or his advisors suggest calling an academic source, though it did happen on occasion. The perception was that academics were too slow in responding and that their first allegiance was to their academic peers and whatever current research study they were doing. Moreover, their understanding of the policy issues was likely to be dated and their appreciation of critical political realities misinformed or outright naïve. No, the planners turned to places such as the Brookings Institution, the Center for Law and Social Policy, Center for Budget and Policy Priorities, Manpower Development and Research Corporation, the Urban Institute, and similar organizations. Academic institutions were not viewed as the place to go for immediate answers to prevailing questions. We think of these go-to institutions primarily as intermediary organizations. They do research and evaluation, but they also keep one foot firmly planted in the policy world. They see policymakers as their primary clients. In addition to producing research, they also see their role as translating extant research and analysis in ways that enhance its utility for those doing public policy. Still, some of these organizations, or the incumbents populating these firms, do high-quality research and publish in respectable academic outlets. To greater and lesser degrees, these firms bridge the research-producing and research-consuming worlds. Of course, these intermediary organizations vary dramatically along a number of dimensions including substantive interests, ideological affiliation, technical sophistication, and objectivity. That is, not all are alike or fungible. Some orient themselves more around ideas than policy, and some are driven more by ideology than data. Some do original research and analysis, and others work closely with government officials. Nonetheless, each contains a common element—serving as a bridge between rigorous research and those seeking guidance for policy purposes. Because they play such an important role in bridging producers and consumers, we look closely at these intermediary organizations. Admittedly, our categorization is subjective and open to debate. Moreover, our examples are players at the federal level even though comparable examples do exist at the state level, though to a lesser extent. Furthermore, our illustrative players are limited to those that focus largely on social policy issues. Our purpose here is not to cover the waterfront, but rather to capture seminal insights about the nature and role of these intermediaries. Despite these caveats, the exercise is important because not all intermediaries are equal, and some of the differences can be critical to facilitating communication across the research production— consumption divide. 149

N ew E ra of T heory , R esearch , E valuation

Think Tanks Examples of these entities include the Brookings Institution, the RAND Corporation, the Urban Institute, and so forth. We define them as institutions dedicated to the creation of new ideas, insights, and innovative analyses using the highest research standards and sophisticated methods. Moreover, they tend toward agenda-free work, at least relatively speaking, with fewer political litmus tests for the selection of staff, topics, and internal review processes. In many cases, senior staff are competitive with faculty at top research universities and do work on par with these institutions. At the same time, their target audience is not primarily the community of scholars. Their raison d’être is to shape the larger set of ideas that inform public discourse. Evaluation Firms Examples of these organizations include Abt Associates, Manpower Demonstration Research Corporation (MDRC), Mathematica, and a few other respected firms focusing on social policy evaluation and analysis. These kinds of firms focus on evaluations of existing and proposed policies and programs and, in some instances, conduct higherlevel analyses of secondary data to inform public policy. They compile existing research and program evaluations for policymakers and do briefings at the state and national levels for key policy types. They operate comfortably in the academic world, but they clearly see their target population as the policy world. Some are serious about applying the most rigorous scientific methods possible. MDRC was created in the 1970s with help from the Ford Foundation to facilitate the conduct of large federally supported evaluations regarding the controversial welfare reform strategies of that era. Over time, MDRC both refined and became a proselytizer for rigorous experimental evaluation methods as the gold standard for determining what works in the social policy arena. Policy Organizations Examples of such organizations include the American Enterprise Institute, CATO Institute, Center on Budget and Policy Priorities, Center for Law and Social Policy, Children’s Defense Fund, and Child Trends. These are institutions that, in general, are organized around the policy process and are driven by a desire to influence policy. They prize access to policymakers. They select both topics and personnel based on their ability to work in the real world. They spend a lot of time thinking about how to communicate with policy types. The mission and philosophies of these policy organizations vary with some organized around specific target populations, others around topical or substantive areas such as poverty or children’s issues, and others around driving political concepts such as free markets or libertarian principles. Organizations such as the Center for Law and Social Policy and the Center for Budget and Policy Priorities have reputations for doing sound work. But they organize their work around specific motivating agendas. Each sees a strong public role in assisting vulnerable populations. Still others take a conservative view of the world. The American Enterprise Institute generally does research that supports a response to public issues that is based on free-market principles. The CATO Institute analyzes social problems from a libertarian perspective. We could go on. The work done in these policy-oriented organizations may be sound, but the underlying differences in perspectives or orientations cannot be discounted. Therefore, those 150

C onfiguring the R esearch and P olicy C ommunities

settled in agenda-focused organizations will drift in the direction of ideological advocacy rather than methodological rigor and dispassionate interpretation. As such, science may not be the governing value, and truth may be balanced against other competing purposes. The institutional environments may trump academic norms and scientific objectivity. It is not always that science is jettisoned outright in the pursuit of political truth, though that can and does happen. Rather, research tends to be selected, interpreted, and organized in ways shaped by ideological or partisan priors. Sometimes the research of organizations with a politically motivated agenda can raise red flags among the very people they want to influence. Philanthropic Organizations In addition to the well-known examples such as the Ford and Rockefeller Foundations, good examples here include the Annie E. Casey Foundation, the Charles Stewart Mott Foundation, the David and Lucile Packard Foundation, the Foundation for Child Development, the Joyce Foundation, the W. K. Kellogg Foundation, The Lynde and Harry Bradley Foundation, The Pew Charitable Trusts, the Robert Wood Johnson Foundation, and the William T. Grant Foundation. Some foundations explicitly work in the policy arena. Why do we include them since they mostly fund programs and policy development where the work is done by others? In our experience, foundations spend a lot of time and energy tracking policy research, thinking about what policy movers and shakers need to know, and exploring better ways to conduct policy-relevant research and convey that knowledge in effective ways. Thus, they serve as conduits between the production and consumption communities. Think of the Annie E. Casey Foundation Kids Count initiative, a project that assembles social indicators on children in every state and nationally, to keep child well-being on the public agenda. The William T. Grant Foundation reignited interest in the study of research utilization by funding a new generation of studies on improving the use of research evidence in policy and practice decisions that affect young people (Tseng, 2012). Government Trade Organizations Here we might include the American Public Human Services Association, the Council of Chief State School Officers, the National Association of Counties, the National Association for Welfare Research and Statistics, the National Conference of State Legislatures, and the National Governors Association. This set of institutions, located close to the public sector, represents either levels of government (e.g., counties or states), specific policymakers (e.g., governors or legislators), or selected governmental functions (e.g., public education or welfare). They are separate from government, but often exist through public fees and dues. They do play a role in bridging the production-consumption gap. Their websites and conferences expose members to the latest research and analyses. They seek out emerging questions and scan the landscape for relevant work and, just sometimes, help shape emerging research agendas. Public Policy Settings We now turn to those communities that we have referred to as research consumers— policymakers of various stripes. For our purposes, we constrain this grouping to those located in government itself though policies are made throughout society. Even though 151

N ew E ra of T heory , R esearch , E valuation

we are limiting our interest to those that make public or governmental policies, this is not a homogenous group. Most observers clearly see the legislative branch as the makers of public policy. Researchers interested in shaping policy would love access to this group. Yet lawmakers are merely one aspect of the policymaking world. Next, we take a broader look at several diverse and distinctive populations or communities that make up the world of public policy. Government-Based Research/Analysis Within this community, we might list in-house organizations such as the Congressional Research Service and the U.S. General Accountability Office. Also part of this group would be the Bureau of Labor Statistics, the Census Bureau in the Department of Commerce, the National Institutes of Health, the Office of the Assistant Secretary for Planning and Evaluation in the U.S. Department of Health and Human Services, and so many more. Most state legislatures too have (typically nonpartisan) in-house staff dedicated to collecting background information and conducting analysis on topics before them. Executive agencies have research, evaluation, and/or statistical units to help guide policy and assess results. There are a host of government or quasi-public officials whose very job is to generate and utilize policy research and analysis. Their functions vary. In some instances, they do original research and, in many cases, they identify and fund basic research irrespective of its immediate utility. However, because a great deal of their work is focused on answering questions of interest to their larger organizations, they often outsource the work to outside researchers or conduct environmental scans to assemble and organize the work of others. In most cases, particularly at the state level, they analyze data on the operation and management of government policies and programs. By virtue of their positions, they also spend a lot of time thinking through the institutional and political contexts that might mediate what the research means. They often are very attuned to the real-world caveats that might limit the application of pure science or rigorous analysis to a societal challenge. At the federal level, research capacity has waxed and waned but at times has been very sophisticated. In the social policy arena, Vee Burke exercised sophisticated scholarship in her long career at the Congressional Research Service. Legislators on both sides of the aisle sought out her work. Howard Rolston headed the research shop in the Administration for Children and Families in the Department of Health and Human Services for many years. He was a champion of doing rigorous experiments to enhance our knowledge base in social policymaking. At the state level, Karen heard frequently in her study about Robert Lang, the 43-year director of the Wisconsin Legislative Fiscal Bureau. The bureau’s reports and publications were consistently referred to as the “gold standard” of nonpartisan and nonpolitical research and analysis. Even as Wisconsin has become increasingly polarized, Lang was commended for managing to keep the bipartisan credibility of his institution and himself. One former legislator characterized him as “the epitome of a public servant [whose] picture should be next to public servant in the dictionary”. Front-Line Service Provision This group comprises those who interact with clients and deliver (or manage those who deliver) public goods and services. Normally, we do not think of these as policymakers, but any honest appraisal of how the real-world works disabuses us of this simplistic 152

C onfiguring the R esearch and P olicy C ommunities

view. Even in the most routinized benefits-issuing system (where computers determine eligibility and benefits), there exists a lot of front-line flexibility in the way people are treated and information is collected. In more complex systems, discretion and professional judgment are unavoidable (Eggers & O’Leary, 2009). Lipsky (1980) has written about street-level bureaucrats, where policy is made at the place where customers and systems interact. These practitioners should be a huge audience for research and analysis given their latitude in making sanctioned and unsanctioned decisions that affect how policies are carried out. They become de facto policymakers on a microscale. Executive Agency Management/Support This group of government users encompasses a broad set of public officials who constitute the core civil service at the management, service delivery, and support levels. They are often thought of as implementers or managers, but not makers of policy. However, this view is naïve. The core civil service types are the repositories of institutional knowledge and typically know a great amount about policy and program content. Given these comparative advantages, they can do much to steer policy and programs in directions of their own choosing. Besides, they know that they will be around for a long time and political appointees will not, especially in the era of term limits. Therefore, they have a vested interest in keeping themselves informed. Executive Agency/Cabinet Leadership These are the agency heads and their closest deputies and advisors, all typically political appointees. They have close access to governors or the White House. They interact with legislative bodies. They often have the trust that comes from shared values and common political allegiances. They are well positioned to do boundary spanning—looking outside their own institutional domains for new ideas and directions. They are a prime target for those seeking to influence policy. One prime example is Donna Shalala, who left her position as chancellor of the University of Wisconsin-Madison to become U.S. Secretary of Health and Human Services in the Clinton administration. Before leaving, she met with researchers at the Institute for Research on Poverty for a briefing on poverty and welfare issues. She remarked during that session that one of her goals as secretary would be to strengthen the agency’s analytic capacity, because she could not imagine running an operation of that size absent a robust access to research. Even as she was leaving the academy, the value of research remained deeply engrained. We suspect this trust of research remained when she became the congresswoman from Florida’s 27th district. Legislative/Judicial Systems There is little doubt or dispute about the role of the legislative branch in the doing of public policy. There is less agreement on who the target population really is. Is it the legislators themselves, their staffs, or the interest groups and lobbyists that have real and substantive access to legislators? The legislative branch has grown into a complex multi-headed phenomenon as witnessed by the growth of K Street in Washington, DC, where lobbyists and lawyers tend to congregate (Smith, 1991). We also include the judicial branch here since they interpret the constitutional legality of policies and programs, often playing a substantial role in overseeing controversial policies or intruding into the 153

N ew E ra of T heory , R esearch , E valuation

management of programs where legal errors occur or are suspected. The judiciaries’ importance to policymaking is highlighted by the vast sums now spent on state-level races and the partisan fights that take place at the federal level. In addition, lawyers typically draw on research to prepare briefs and legal arguments on constitutional issues. Research evidence was a draw for the former Chief Justice of the Wisconsin Supreme Court, who frequently attended the Family Impact Seminars and participated in discussion sessions.

Conclusion Those discussing underutilization of research in policymaking started out with a simple dichotomy. There were those who did research and those who used research, basically those who pondered how the world worked, and those who sought to make the world work better. This binary framework—research producers and consumers—was founded on a theoretical distinction that easily conformed to the straightforward way that the earlier two-communities theory conceptualized the world. But that bifurcated view of the world, while useful in those early years and generative in later years, has always been much too simplistic, as recent theorists have recognized and as we explicate throughout this book (Contandriopoulos et al., 2010; Nutley et al., 2007). We pose different kinds of research producers and research users, and a whole industry of individuals and institutions devoted to bridging the patterns and content of communications across these two universes. In many ways, Tom and Karen performed “bridging” roles even though they were located within research universities. Further, we see information flows going in multiple directions (Bednarek et al., 2018), although the flow of rigorous research and sophisticated analytical work is generally from the research world to the public decisionmaking arenas. Understanding these complexities is critical to advancing any agenda for understanding why research may be underutilized and for optimizing the use of research in the policy world. We also started out with a simple notion of culture. Researchers and policymakers thought about things differently and those differences confounded communications and cooperation. Fair enough, but we are now suggesting that culture is a more complex, contextually specific phenomenon than we realized initially. Simply put, we started with a world composed of two distinct islands, in which the inhabitants of each had trouble communicating across the waters that separated them. That evolved into three islands with suggestions of more. Eventually, we realized that an archipelago served as a more appropriate visual metaphor with sets of distinct islands cluttering our increasingly complex map. The way we organize our professional worlds emerges from the institutions in which we are located, from the way we were professionally trained and socialized, and from what is learned from life experiences along the way. At the end of the day, both researchers and policymakers embrace a dominant community with which they identify. To communicate with members of each community, one must understand the norms and expected behaviors that govern life in that community. That is, we must understand its culture. Do we have these worlds laid out correctly? We do not know for sure. Other conceptual arrangements are both possible and plausible. But we think our framework advances and illuminates the theoretical discussion. The real question is how our more complex model plays out in the ways that real people think and act and communicate. We segue into that in the next chapter that presents our community dissonance theory where we pull our thinking together. 154

7 WHY RESEARCH IS UNDERUTILIZED IN POLICYMAKING Community Dissonance Theory Karen Bogenschneider and Thomas J. Corbett

Community dissonance theory has the potential to restore political confidence, given its premise that researchers and policymakers are not destined to remain isolated on their own island. If researchers had a better understanding of the inhabitants, institutions, and cultures of the policy community, this could improve communication with policymakers and increase the use of research in policymaking. This chapter  elaborates on the professional culture of policymaking and six domains of institutional culture, including preferred decision-making processes, interactional preferences, favored epistemological frameworks, dominant influence loops, focal interests, and salient stakeholders. Without awareness of these cultural differences, friction points are bound to occur across and within communities that can imperil communication and collaboration. Enabling cultural change may be as important a focus as enabling change in practice. Isabel Walter, Sandra Nutley, and Huw Davies (2005, p. 348) Exactly what constitutes this so-called expression of community will vary across analysts, of course. . . . In public policy making, many suppliers and users of social science research are dissatisfied, the former because they are not listened to, the latter because they do not hear much they want to listen to. Charles Lindblom and David Cohen (1979, p. 1) For many years, almost like a rite of spring, the Wisconsin Legislature squared off with university leadership over the amount of time faculty spent in the classroom. This represented an inevitable flash point between politicians grudgingly allocating scarce tax dollars to educate the state’s youth versus university leaders desperately seeking to maintain the school’s status as a world-class research institution. The basic disconnect between these two institutions—the university and the legislature— was easy to infer. The former saw the university’s primary purpose as creating new knowledge, and the latter saw it as a vehicle for transmitting vocational skills to the next generation’s workforce. When two communities disagree so fundamentally on an institution’s core technology, clashes are not surprising though they can prove amusing on occasion.

155

N ew E ra of T heory , R esearch , E valuation

One apocryphal story deserves mention. It concerns a legislative hearing on university teaching loads at the University of Wisconsin-Madison, the system’s flagship campus. During this hearing, a conservative legislator from rural Wisconsin suspiciously asked a professor appearing before the committee just how many hours he spent in the classroom. “Six” was the response, given that two three-credit courses are considered the typical full-time teaching load at a research institution of higher learning like Wisconsin. Satisfied, though still suspicious, the legislator eased back in his chair. Suddenly struck by a thought, he straightened up again and fixed the professor with a disbelieving stare: “You DO mean six hours a day, don’t you?” Slack-jawed, the professor could only return his interrogator’s hostile gaze with stunned silence. Sometimes, professional worlds are so different that communication borders not only on the improbable, but also on the totally hysterical. In Chapter 6, we developed an organizational landscape for our quest to explain why research is underutilized in policymaking. We proposed a conceptual archipelago—a chain of islands in close proximity with some clustered around the core technology of research production and others around research consumption. In this chapter, we lay out a theoretical framework for addressing two key questions: How should we think about the cultural influences associated with community identity and how do these influences potentially impede cross-cultural communication? Laying out a theoretical framework for describing the world is a stimulating intellectual exercise. But any such theory is consequential only to the extent that it informs the way we see the world and, as such, leads to a more accurate appreciation of why the world operates as it does. Strong theories are consequential because they furnish explanatory frameworks that increase the odds that “promising ideas will turn into practical results” (Gamoran, 2018, p. 180; Tseng, 2012, 2019b). Given our long-standing interest in research utilization, our initial thinking about the disconnect between the research and policy communities appeared in the first edition of this book. We built on Nathan Caplan’s (1979) two-communities theory, a forerunner of our own conceptual framework, that we termed community dissonance theory. We recently published our revised thinking in an article in a theory journal that we expand upon in even more detail here (Bogenschneider, Corbett et al., 2019). Starting with the two-communities concept seemed reasonable given that many studies of the researchpolicy divide, and most studies in the field of education, have been conceptualized using two-communities theory (Ness, 2010). Also, the theory has been used as a conceptual frame in publications of policy insiders (Vincent, 1990) as well as studies in the fields of health policy (Orton, Lloyd-Williams, Taylor-Robinson, O’Flaherty, & Capewell, 2011), knowledge-brokering (Cooper, 2014), legal policy (Steinberg, Cauffman, Woolard, Graham, & Banich, 2009), and psychology (Maton, 2017). Yet theorists have seldom refined the theory or its predictive ability beyond its conceptual appeal. It does a credible job of suggesting why research is underutilized by inferring that researchers and policymakers come from two worlds that differ in some amorphous ways. Yet more incisive thought is needed to transform this abstract idea into concrete concepts and key elements that could better guide research and practice. In this chapter, we attempt such a refinement by responding to earlier criticisms of the two communities theory that questioned whether research producers and research consumers ought to be conceived as homogenous groups to the exclusion of other key players (Contandriopoulos et al., 2010; Nutley et al., 2007). The theory also has been criticized as being too simplistic in its focus on the individual policymaker and the individual researcher, which downplays the influence of the political, social, and organizational contexts in which they operate (Nutley et al., 2007). In response to these 156

W hy R esearch I s U nderutilized in P olicymaking

criticisms, we have substantially extended our earlier thinking (Bogenschneider & Corbett, 2010a; Corbett & Noyes, 2008) and incorporated additional research over the last decade, including our own study of 225 state policymakers (see details in the Appendix). This empirical evidence, taken together with our experience working at close quarters with policymakers, has deepened our understanding of policymakers (Chapter  3), the policy process (Chapter 4), and the distinctive cultures that give inhabitants of the policymaking world meaning and identity (Chapter 6). In our revision, we carefully erect each building block of our community dissonance theory beginning with a vignette that illustrates how the character of community dissonance plays out in the policy world. We define what is meant by the key concepts of community, culture, and core technology. We elaborate on the key elements of understanding community—its institutional setting, inhabitants, and professional and institutional cultures. We examine several classic friction points where cultures collide that might explain the underutilization of research in policymaking. We conclude by discussing the value of a theoretical framework that provides touchpoints for identifying cultural barriers to communication along with actionable strategies that can inspire scholarly confidence and offer a message of hope. Community Dissonance in the Real World Tom recalls a time when he (along with his colleague Irv Garfinkel) were asked by the majority leader in the Wisconsin State Assembly to participate as a nonvoting member on a special legislative committee on welfare reform. Such an invitation to participate on an official legislative committee was rare. Over the course of the committee’s work, Tom developed a comfortable working relationship with the chair of the committee, John Antaramian, a Democrat representing a blue-collar, working-class district in the southeastern part of the state. Even after the committee finalized its work, John would call on Tom for advice on welfare issues. It was clear, however, that such a trusting relationship between a member of the academy and a member of the legislature was an outlier case. Whenever the conversation within this informal working group drifted to the university, deeply held prejudices emerged on the part of lawmakers. The dominant perception among state legislators was that academics were arrogant, distant, elitist, greedy, naïve, self-absorbed, and totally clueless about real life. More than once John claimed to really like Tom but had less use for the rest of those eggheads on campus. Later, when John was leaving the legislature, a goodbye party was held in the state capitol for him and others who were then leaving. The festivities were attended by the usual set of colleagues, staff, lobbyists, and reporters. Ironically, on that very morning, yet another newspaper article “exposed” how little time the typical university professor at the University of Wisconsin-Madison campus spent in the classroom. As Tom was exchanging warm wishes and reminiscences of the welfare wars they had been through together, he reminded John of the morning’s headlines. Suddenly, the politician became animated, pointed his finger at Tom, and yelling in a loud voice so that all might hear, “university professor, university professor”. Apparently, he thought these words would prompt all the politicians in the room to descend upon the hapless academic and pummel this poor sinner to a premature demise. It was all an affectionate jest, of course, but the underlying sentiment was clear. From a legislative perspective, the academy looks like a remote land populated by a strange and incomprehensible tribe. The cultural divide not only impedes communication but fosters negative stereotypes. Misunderstandings quickly can escalate to outright hostility. Initial negative feelings 157

N ew E ra of T heory , R esearch , E valuation

soon solidify into intractable prejudices. They can be ameliorated, as the working relationship with Tom and John suggest, but not without effort. In this instance, a research producer and a research consumer were thrown together in a way in which prior feelings and stereotypes could be addressed and mitigated. The story hints at a powerful lesson— one-on-one contact can lead to personal relationships that can be a powerful antidote to established prejudices (Chua, 2018). The tale may also suggest, however, that negative priors are not easily dispelled.

Erecting the Building Blocks of a Theory of Community Dissonance In our earlier thinking, we used the term community confusion to label this amorphous phenomenon that impeded successful cross-group exchanges. We soon realized, however, that we were talking about something more meaningful and intractable than a temporary disorientation. Confusion implied a lack of clarity or direction. Our experience suggested that the gap between those who do science and those who make policy involved something more profound. It was more than a little misunderstanding. At heart, we were talking about different ways of seeing the world in some larger and more fundamental sense. We were talking about private worlds that have been embraced with such breadth and depth that it becomes difficult to understand or appreciate those residing outside our own cultural bubbles, often without being able to discern why. Our first introduction of community dissonance theory in 2010 provided glimmers of the communication breakdown between communities. Now we see it in a more sophisticated manner, as much more than a dichotomous division, but rather as an array of semi-isolated communities generally situated on either side of the research-policy divide. There are numerous opportunities for miscommunication, and not just between researchers and policymakers. Communication and understanding break down within each community—whether on the science side or the policy side of our conceptual view of the public arena. As we introduced in Chapter 6, there exist many variations on each side of the main barrier as naturally occurring subcommunities can be discerned within the research and policy worlds. For example, there are important points of institutional dissonance between those inhabiting academic positions at a research university and those located in a policy evaluation firm such as MDRC. Likewise, there are points of dissonance between those serving in legislative institutions where laws are passed and those operating out of executive agencies where many of those laws are implemented and managed. Yet the biggest potential disconnects, where understanding and communication most often fail, remain between those who see themselves as investigators of the mysteries of life and those who define themselves as inhabitants of the policy world where things of consequence get done. That remains our biggest chasm that begs for deeper understanding. We have long argued that each community can be thought of as an island in some important respects, having attributes and patterns that make them relatively distinct and, thus, vulnerable to becoming isolated. When viewed from afar, this chain of islands makes up a complex archipelago composed of two aggregations of related islands. Within each aggregation, there are degrees of separation among the constituent islands that make communication difficult. These members of distinct communities approach one another warily on occasion, sometimes with distrust or hostility. We think the word dissonance conveys a more nuanced sense of what we intuitively appreciate, a disconnect not fully understood that renders good communication a chancy thing at best for reasons difficult to isolate and, therefore, understand. 158

W hy R esearch I s U nderutilized in P olicymaking

We continue our theory-building exercise by examining the somewhat ambiguous terms—community, culture, and core technology. Then we turn to three foundational building blocks of our theoretical architecture—the inhabitants of a community, the institutional settings in which they function, and how inhabitants are shaped by their professional and institutional cultures.

Revisiting the Concepts of Community, Culture, and Core Technology As with many theoretical understandings, specifics evolve over time with various theorists highlighting different communities and suggesting nuances that they believe most important. The building blocks for a comprehensive theory of community dissonance were there. For example, Small (2005) focused primarily on incongruities in epistemological differences across the realms of research and policy, methodological and statistical limitations of science, and incongruities in cultural practices and situational demands. Myers-Walls (2000) framed the communities of evaluators and program staff as being an “odd couple” that differed along six domains: temporal, cognitive, definition of excellence, patterns of communication, daily lifestyle, and how each used the tools of their trade. Shonkoff (2000) took a slightly more nuanced version of research users, distinguishing between those who make policy and those who carry out policies that he labelled science, policy, and practice. Bogenschneider and her colleagues (Bogenschneider, 2014; Bogenschneider et al., 2006) laid out more systematically three communities, three broad cultural domains, and three behavioral elements within each domain. Other theorists might be cited, but their offerings would not differ substantially from these. One glaring deficiency with existing thought during the previous era is that the integration of such concepts as community and culture is broad, overly general, and might be construed as too superficial to be of great use. The terms community and culture are related to each other and are seen by some as fungible, yet an important distinction can be drawn. Defining the Concept of Community A community is a grouping of like-minded people organized around purpose, allegiance, affinity (personal bonding), or necessity that one interacts with in some ongoing fashion and forms either an emotional or utilitarian bond. A community says to each inhabitant, “Here is where I belong. This is what I am and these are my people. Now I know how to act, how to see the world, and even what to believe.” Maintaining that connection results in some level of acceptance of prevailing language, cultural norms, and behaviors. Most important of all, community provides individuals with a sense of belonging. Defining the Concept of Culture The notion of community is the setting, or environment, in which culture operates. Culture, in effect, is the expression of community, the glue that holds inhabitants together, provides meaning and identity, and even helps separate them from outsiders. A culture, as we also introduced earlier, captures a whole set of norms, rewards, and operating protocols and premises that shape how people view their role in the world and act on a dayto-day basis. That is, the concept of culture is the milieu that surrounds us and influences our behavioral patterns in ways that may escape conscious awareness (Corbett, 2018b). 159

N ew E ra of T heory , R esearch , E valuation

Key Concepts 7.1.  Definitions of the Key Elements of Community Dissonance Theory Community—a grouping of like-minded people organized around purpose, allegiance, affinity (personal bonding), or necessity that one interacts with in some ongoing fashion and forms either an emotional or utilitarian bond. Most important, a community provides individuals with a sense of belonging. Culture—the expression of community, the glue that holds inhabitants together, provides meaning and identity, and even helps separate them from outsiders. A culture captures a whole set of norms, rewards, and operating protocols and premises that shape how people view their role in the world and act on a day-to-day basis. Professional Culture—a set of dispositions and tendencies derived from personal background, education, training, learning on the job, and lived experience. Institutional Culture—how an institution functions and the way work gets done, which is shaped by history, traditions, norms, and rules. Core Technology—what an organization produces and for whom, which may be ideas, advice, resources, or human services of some sort from crisis intervention to human capital development. The critical notion is that a program’s product shapes purpose which, in turn, shapes essential tasks that influence the operating culture and structural architecture of an organization.

Remarkably, while culture often is expressed in formal rules or even manuals expressing appropriate behavior, much of the force behind this concept is unstated and even preconscious. We just know! We also argued that there are two sources of culture. First, professional culture is a set of dispositions and tendencies derived from personal background, education, training, learning on the job, and lived experience. These are behaviors, rules, and values repeatedly drummed into inhabitants by virtue of sitting in classes, participating in apprenticeships, working with mentors, associating with colleagues, and observing those inside and outside the community. Second, there is an institutional source of culture one is exposed to by virtue of one’s job and work location. Institutional culture is how an institution functions and gets work done, which is shaped by history, traditions, norms, and rules. A few observers have commented on how the ways that the research and policy worlds function affects the use of research evidence (Bogenschneider, 2014; Bulmer, 1987; Maynard, 2006; Myers-Walls, 2000; Shonkoff, 2000; Small, 2005). All of these commentators start from a single perspective: “Research use does not take place in a vacuum. It is always embedded in specific institutional contexts whose idiosyncratic cultures, structures, and politics temper the ways in which research gets taken up by policymakers and practitioners” (Nutley et al., 2007, p. 303). These institutional forces grow out of the core technology inherent in the professional niche that they occupy. Defining the Concept of Core Technology Let us examine the construct of core technology more closely (Thompson, 1967). What does an organization produce and for whom? What are the essential tasks required to 160

W hy R esearch I s U nderutilized in P olicymaking

generate those products? Products, in this sense, do not necessarily mean physical outputs. They might just as well be ideas, advice, resources, or human services of some sort from crisis intervention to human capital development. The critical notion is that a program’s product shapes purpose which, in turn, shapes essential tasks that influence the operating culture and structural architecture of an organization. What an institution feels like and how it really functions, as opposed to what appears on the formal organizational chart, emanates from this core technology at the center of its operations.

Toward the Concept of Community Dissonance Our work has moved us beyond the well-known two-communities theory (Caplan, 1979) toward the development of this new conceptual framework that we have been calling community dissonance theory. This refinement expands our thinking beyond the notion of two or three communities. We envision numerous community groupings where each is formed around a core technology that defines the tasks professionals perform as well as the organization’s purpose, structure, and operational premises. Each of these communities, whether they be basic researchers, applied researchers, intermediaries, policy doers, or policymakers, is shaped by a distinct culture in ways that differentially affect how their inhabitants think, act, and behave. We all live in some type of cultural stew—a professional and institutional mix of beliefs, values, dispositions, and perceptions. Most enjoy more than one cultural tradition stemming from family, religious, or ethnic background, though we focus mostly on the culture associated with one’s work life. Cultural environments are the waters in which we swim daily. The allegorical waters about us are so encompassing that we can barely appreciate the real nature of that supportive fluid and what it does to shape our world. Even though culture is not something we think of explicitly most of the time, we believe that we can understand its main mechanisms, at least in our work worlds. To appreciate the research-policy divide, we focus on a framework grounded in the professional and institutional forces that shape the reality of our working roles. At the same time, these cultural realities result in points of possible friction or incompatibility that exacerbate misunderstandings among inhabitants of each community when they intersect outside their own bubbles. Community dissonance theory attributes the underutilization of research in policymaking to a lack of communication between inhabitants of the various islands that comprise our archipelago. The inhabitants of each island share more with one another than with inhabitants of other islands. The point is, though, these points of difference, or potential friction, can be understood as the key to changing the situation. The tragedy is that they often go unrecognized amidst confusing signals and misunderstandings. This communication failure and its underlying friction points are rooted in several key dimensions, associated with all communities, that emerge from different core technologies (i.e., essential institutional purposes) and different professional and institutional cultures. This preparation starts with training in a specific career trajectory, whether business, government, politics, science, and so forth. This professional orientation is shaped by the dominant values and ways of thinking found in all institutional settings such as decision-making processes, interactional preferences, favored epistemological frameworks, dominant influence loops, focal interests, and salient stakeholders. Inhabitants of each community and culture cannot talk to one another effectively because they are not fully aware of the barriers that separate them (see Chapter 4 on the policy process). Many of these barriers are unseen and unappreciated. In the end, our natural 161

N ew E ra of T heory , R esearch , E valuation

human tendency is to make it all fit together. Too often, inexperience with the other world leads the individual to presume, albeit falsely, that inhabitants of the other island see the world as they do (Kendall-Taylor & Levitt, 2017). We realize that we have simplified a complex topic. Without question, we have obscured the rich, within-group variation that we know exists in the real world. Not all research producers, even those focusing on basic research, are alike. Similarly, some lawmakers in the same settings could care less about research and others aggressively seek it out to help them make the best decisions possible (see Bogenschneider et al., 2013 and Chapters 3 and 5). Just as clearly, we have only touched upon the attributes and characteristics that would be encompassed by a more comprehensive treatment of the topic. These shortcomings do not detract from our central message, however. Culture shapes how we think, act, and perceive our world in ways that we never fully appreciate. The perspectives embraced within cultures facilitate or impede communication across communities and worlds with alternate cultures. Unless we have some sense of the institutional and professional cultures in which we operate, we may never really grasp why we struggle with those inhabiting other islands. Languages collide, nonverbal gestures fail to communicate, expectations go unfulfilled, and the scripts we rely upon daily no longer work quite as well. As we worked to refine these concepts, we struggled with the words that best captured what we were talking about. In the end, we concluded that dissonance best taps a notion of friction between worlds we are struggling to identify. Words like confusion and conflict suggest a level of awareness we don’t believe always exists. The term dissonance, on the other hand, implies a level of disconnect that is not always conscious or intended. No one party is at fault. It is more of a lack of alignment where the diverse communities do not mesh well despite mutual needs and interests, and for reasons not always appreciated. Inhabitants of the separate islands merely pass by one another as they operate from their individual understandings of the world. Three key elements of our theory that contribute to a more complete understanding of community—its institutions, its inhabitants, and its professional and institutional cultures—can lead to actionable strategies to improve the situation.

Community Institutions: Their Meanings and Traditions One foundational element of a community is an understanding of the meaning and traditions of the institution. We start with a deeper discussion of what we mean by an institution. In his seminal work, Heclo noted that the nature of an institution is a protean concept. He laid out some 21 different definitions for how an institution might be defined. He then goes on to note: We can find the meaning of institutions being conceptualized by genealogy (where they come from), or function (what they do), or purpose (what they are meant for), and/or essence (what they are). Some of the definitions we read are explanatory (seeking to clarify important aspects of a concept). Others are analytical (specifying the type of thing the term applies to and the differences from other things of the same type). Or we might resort to metaphorical examples of the term, as in saying that institutions are “history encoded into rules”, or “frozen decisions,” or “congealed tastes”. (Heclo, 2008, p. 51) 162

W hy R esearch I s U nderutilized in P olicymaking

Whatever definition is used, institutions typically are viewed from the perspective of an outside observer looking in. Heclo (2008), however, conceptualized what institutions mean to those on the inside looking out. Basically, thinking about institutions is not the same as institutional thinking—inhabitants who carry around in their head the relevant norms, practices, and purposes that make the institution an important part of who they are (even though every inhabitant may not conform to these expectations). For some inhabitants, an institution is no more than a place to demonstrate individual competencies and record personal achievements. For others, there is a deeper respect for the whole, inherited tradition handed down over the generations that extends well beyond rewards for any individual member to collective service for a larger purpose. Most Marines identify with that institution for the remainder of their lives. Semper Fi! Let us make this abstract discussion a bit more concrete with two examples of institutional settings on the policymaking side of things that we know well—state legislatures and human service executive agencies. We rely heavily upon the words of legislators in our recent study and our experiences in the policy arena to flesh out this abstract concept. State Legislatures In the existing literature on state legislatures, the lawmaking institution means more to its inhabitants than simply a place to pass laws (Rosenthal, 2009). In our recent study of state legislators, they confirmed the special meaning the legislature holds for them and what a privilege it is to serve. They described an inherited tradition that leads them to think about honoring or dishonoring the institution itself. Legislators spontaneously noted their respect for the institution, portraying it as something greater than themselves. A Republican explained: “The title belongs to the district. I don’t own the title. I just happen to be a body here that’s using it for the moment.” There is a sense of continuity and history about some public institutions that is not found in the private sector. This institutional respect, grounded fundamentally in the culture of public service, extended to how legislators think about their colleagues and act toward them. For one legislator, the institution evoked a “sense of diplomacy” so strong that she routinely addressed people by their title out of respect for the fact that “we are privileged to be in this building and call it our workplace”. Another legislator took her children to a bill signing, despite their complaints that they did not like the governor. She told them, “We might not like him, but we have respect for the office.” One legislator, on the “downhill” side of the time he would be serving in the legislature, made a personal commitment to setting an example for newcomers to help ensure that what is good about the institution endures for his kids and grandkids: We [have a] responsibility to leave this place in terms of being able to work across the line and so forth to younger members. You do it by example.  .  .  . Because if we don’t, I don’t know what could happen down the road. . . . That’s one of those things where probably the public at large doesn’t understand. Key informants in our study, many who had decades of experience in state policy, corroborated legislators’ respect for the institution, but also described how legislators are sometimes boxed in by institutional norms. Legislators subscribed to the natural human tendency that surfaces in any institution to “go along to get along”. This tendency affects not only how they do their job but also whether they keep their job: “There’s a lot of camaraderie. ‘You scratch my back, I’ll scratch yours,’ so you are not a ‘one-horse pony.’ 163

N ew E ra of T heory , R esearch , E valuation

You’ve got to . . . help those you like.” One noted that when your career is at stake, “Partisan discipline and loyalty trump a lot. . . . In the caucus [when] they say, this is a party thing, and you start arguing against it, and you’re the lone ranger, it’s much more difficult.” In this respect, institutional norms strongly enforce behavior. Executive Agencies Another key island among the archipelago that constitutes the policy world is executive agencies (Corbett, 2018a). For illustrative purposes, we look to an era when the influence of institutional culture was stark. In a relatively short time, largely during the 1990s, welfare in some U.S. jurisdictions evolved from a rigid, routinized program to a relatively nonroutinized system. Welfare for several decades had been a benefits-oriented system. The underlying worldview was uncomplicated—the compelling misfortune of disadvantaged families was a shortfall of resources and the public obligation was to make good that deficit. Program purposes were limited—get the checks out the door to eligible families accurately and efficiently. The time horizon was short. Each month was a new accounting period where assets and income were, in theory at least, recalculated to determine if a shortage of resources existed and the magnitude of that shortfall. In such a system, horizontal equity was paramount; each family was to be treated in the same way. The worker-client interaction tended to be uniform, episodic, and regularized. The gatekeeping function was to separate the eligible from the noneligible; the ongoing case management function was to calibrate monthly income need (according to complex rules) for eligible families. This way of doing business encouraged vertical, or top-down, command-and-control authority patterns, which in Weberian bureaucratic norms were an institutional virtue (Weber, 1905/2002). Casework discretion was further constrained by the introduction of computerized decision-making systems. Existing manuals translated all discretion into clear-cut, binary choices: if the situation is A, you must do X, and if the situation is B, then you must do Y. Front-line workers carried out a narrow range of responsibilities for their own caseloads. In the 1990s, federal reforms of the cash welfare system ushered in new goals and new ways of doing things that focused much more on behavior change in clients rather than their immediate income shortfalls (Corbett, 2018a). The new program purpose was to encourage positive behaviors and discourage those perceived as being counterproductive. Such behavior-modifying programs tend to be dynamic and based on change over time, not static systems in which each month is an independent accounting period. In fact, they were so multidimensional and individualized that, by necessity, much more discretion was allocated to front-line personnel. Not surprisingly, such a change in purpose transformed the institutional architecture of both the system and the culture that supported it. When you moved from a formal, benefit-issuing system to one that focused on changing behaviors, all kinds of consequences became apparent. To start with, agency boundaries became more porous as service integration and one-stop program models emerged. Entrepreneurship and outcome-oriented institutional philosophies began to supplant risk-averse thinking. Operational discretion replaced traditional command-and-control management strategies. This new culture demanded a different kind of worker. In the past, welfare managers wanted “left-brain dominant” workers who excelled at routinized detail and worked well at linear, unvarying tasks. Suddenly, the need was for “right-brain” workers, who could see the big picture and were creative problem solvers. In just a few years, everything had changed. 164

W hy R esearch I s U nderutilized in P olicymaking

This example, drawn from welfare reform, brings home the crucial lesson of core technology: our professional lives are shaped by what we are expected to do and by what our institutions are designed to produce. In short, purpose leads to a search for compatible organizational forms which, in turn, leads to appropriate behaviors consistent with that form. Form follows function; culture follows form. New behavioral demands often require new people who think about the world in new ways. We have labored over this vignette since it is critical to appreciating the relation between core technology and institutional culture in shaping the expectations and behaviors of inhabitants.

Community Inhabitants: Their Character and Commitment Another foundational element of community dissonance theory is the inhabitants of our allegorical research and policy islands or, more accurately, archipelago. Through better understanding of one another, misperceptions and unwarranted suspicions can be allayed and even eliminated. Researchers generally appear conflicted about what to think about policymakers, an ambivalence reflected in the academic literature. For instance, Rosenthal (2009) portrayed state legislators as “pretty decent people” who give the public “more than its money’s worth” (p. 67). In contrast, Mayhew (2011) called members of Congress “mediocre slackers given to nastiness, pork-barreling, corruption, extremism, broken processes, lapdog behavior toward presidents of their own party, and other behaviors that vitiate policymaking and leave the public cold” (p. 890). How can it be that two researchers, both who study policymakers, can reach some dramatically different conclusions? These conflicting views are illustrative of Rosenthal’s (2009) warning that it is easy to second-guess the performance of policymakers, which underscores the critical importance of hearing their voice. The predominant view among the legislators we interviewed was that the public “wrongly” has less trust in them than they should. An occasional legislator disparaged a colleague as lazy or for using his or her current position as a stepping-stone, but these characterizations were outliers. A Republican with a background in human resources said: People aren’t doing bad things here.  .  .  . In the private sector [there’s more things] we would end up having to fire people [over] than I’ve seen being done here—90 percent of the legislators here on both sides of the aisle—they’re trying to do the right thing. They’re trying to do what they believe is in the best interest for the people of Wisconsin. Democrats agreed, despite being a minority voice in both states on many of the decisions being made. One Democrat described it as “freakonomics”, the unseen reality of being a legislator: Everybody thinks that legislators are liars and cheats, but the fact of the matter is, regardless of what party it is, 90-plus percent of the people here are some of the nicest, finest people you could ever meet. They all have similar-type goals of trying to improve the lives of ordinary people and they’re very honest and trustworthy. Legislators were consistent in reporting that their colleagues were well-meaning people who ran for office for the “right reasons”. Conceivably, these self-reports could 165

N ew E ra of T heory , R esearch , E valuation

suffer from a form of social desirability bias—the tendency during interviews to present oneself and one’s profession in a favorable light. So, we checked with our key informants, none of whom was holding elected office at the time they were interviewed, and all who were highly experienced and serving in the top echelon of their organizations. All 13 informants, who averaged 31 years of state policy experience, corroborated the positive reports of legislators, although one qualified that we might be at the end of an era. One key informant said anyone with “the guts to run is admirable”. Policymakers also were characterized by key informants as “honorable”, “amazing”, “public interested”, and “dedicated good people who are doing the best they can”. One key informant said legislators’ “first and foremost consideration” was whether “it will be good for their constituents”. Another described the commitment it takes to run: One thing, you’re getting $50,000. . . . And even if you get some per diem and stuff, you’re not getting rich. . . . Boy, you’re campaigning all the time, knocking on doors. . . . And the other thing, with all the social media and stuff, everybody checks everything in your background. I mean any little thing that you did when you were in high school, “Boom, it just blows up!” Who wants to put their families and themselves through that? Despite all it takes to run for elected office, this key informant continued to assert that all members are proud to be part of the legislature: “I’ve witnessed hundreds and hundreds of legislators and I have never had a legislator say to me, ‘I wish I would have never run for this job. I have never, ever heard it.’ ”

Professional Culture: Personal Background and Preparation Many factors go into the formation and maintenance of a culture. We focus here on the factors that stem from one’s prior training and preparation. We limit our conceptual discussion about the dimension of professional culture to policymakers, primarily those who inhabit the community where laws are made. The people who inhabit the lawmaking community, and thus embrace its culture, are different in one important aspect. Communities that create new laws and rules do not look to recruit candidates with common backgrounds. Rather, they come from many different disciplinary and professional traditions. This broad recruitment of inhabitants from different backgrounds and, thus, distinct professional cultures, onto the lawmaking island is rather unique. Law firms, for example, look for those with legal training, medicine selects those with medical training, most counselors have preparation in therapy or social work traditions, and researchers are all grilled in the methods of science even if their substantive or disciplinary interests vary. Because lawmakers are more heterogeneous than our other communities, this suggests professional culture is a bit less germane to our overall discussion, though not entirely irrelevant. We assume there is some self-selection into such a career path that favors individuals with pre-selected proclivities. The notion of professional culture remains a key element of community dissonance theory. Arguably, no one enters their work life as a blank slate. Each person brings important priors to the job that they absorb as part of growing up or their formal education and training. Learning about the professional backgrounds of those inside an institution can help those outside the institution understand the way inhabitants think, behave, and solve problems. Professional culture—shaped by personal background, education, 166

W hy R esearch I s U nderutilized in P olicymaking

training, learning on the job, and lived experience—affects attributes such as intellectual substance along with skills and behaviors that comport with standard practices for getting work done (Chao et al., 1994). Studies of professional culture in lawmaking bodies have waned (Mayhew, 2006) and have failed to produce conclusive evidence on how it frustrates or facilitates research use. Exceptions include studies that find less research use the longer a policymaker holds office (Weiss, 1999), and more research use among policymakers with higher levels of education, especially when they have had prior engagement with research (Nutley et al., 2007). Also, an institutional body is more likely to use research evidence when it has a higher representation of social scientists (Weiss, 1999). As expected, given the diverse backgrounds and professional preparation of lawmakers, our study of 212 state legislators revealed that they brought a diverse set of cultural antecedents to their institutional roles: “We’re moms. We’re dads. We’re grandparents. We’re schoolteachers. We’re business owners. We’re doctors. We’re lawyers.” When we counted the professional backgrounds that legislators in our study mentioned (without prompting), there were more than 35. Legislators, Republicans and Democrats alike, were farmers, journalists, law enforcement officers, lawyers, local government officials, medical professionals, military personnel, pastors, professors, real estate agents, smallbusiness owners, and much more. Legislators relayed how these diverse professional skill sets were useful in policymaking. For example, business owners reported bringing skills that were valuable on the job—problem solving, budgeting, and seeing constituents as customers no matter their political affiliation. Law enforcement officials reported the utility of their professional skill set of valuing honesty, the ability to give others bad news, and not taking things personally. Legislators also discussed the import of their “lived experience”, including their religious, family, and childhood backgrounds. Legislators reported being influenced by their parents, by parenting their own children, and by early life experiences (e.g., growing up in poverty, on farms, in various religious traditions, in a big family, as a middle child, with divorced parents). These experiences led to personal qualities that contributed to their effectiveness. Legislators described themselves as “a nerd”, an “avid reader”, a “fixer”, or they said, “That’s the athlete in me”, “I go into teacher mode”, or “I have a love of people”. Most legislators thought populating an institution with such an “amazing mix” of inhabitants was a strength. Yet they also discussed how such vast diversity in attributes such as cognitive intelligence, emotional and social intelligence, experience, personality, research background, and worldviews brought inherent challenges in getting to know their colleagues and in working with them: There are some people that their magic power is throwing the javelin. And there’s other people that are better at being the peacemakers or the bridge builders. . . . [Y]ou have to figure out what role are people going to play—sniff each other out. In short, to be part of a professional tradition means that you will arrive in your adult role with an ingrained set of dispositions, attitudes, and behavioral tendencies. In a sense, you come prepackaged. While many professional settings may prefer greater homogeneity of backgrounds, diversity is seen as a strength among lawmakers. The matters under consideration span a wide range of possibilities, and outcomes are part of a negotiated process, not the work of a single individual. 167

N ew E ra of T heory , R esearch , E valuation

Institutional Culture: Operational Practices and Procedures Beyond the cultural dispositions of inhabitants, all institutions possess a culture unique to themselves in some fashion, which Heclo refers to as an “institutional soul” (2008, p. 9). Why is this important? For the same reason that professional culture is an important concept in easing an individual into their adult roles, each institutional culture does much to determine the ease of communication and understanding within an organization. Institutional culture refers to how an institution functions and the way work gets done within it, which is shaped by history, traditions, norms, and rules (Weiss, 1999). These embedded cultural factors move beyond superficial knowledge to a deep and discerning understanding of how the decision-making process works, what and who captures inhabitant’s attention, how information is screened and communicated, and which factors influence decisions. Sandfort (2004) called it an organization’s “deep structure”. Knowing about the dominant culture of an institution and the ways it produces its core technology bring into sharp relief underlying assumptions that are essential to effectively communicating across the cultural divide. To engage policymakers with more confidence and success takes a basic understanding of the institutional culture of policymaking. From the many possible vantage points through which we might understand institutional culture more fully, which merit inclusion in our theory? Our search revolved around identifying those dimensions of a community’s institutional culture that are universal and significant. That is, they are likely to be important in defining culture in a variety of institutions and, important in identifying the behaviors and belief sets along which significant differences may arise across communities. To identify these domains, we began by examining those included in our 2010 theory. We took seriously Popper’s (1959) assertion that one hallmark of a scientific theory is the possibility of falsification. Failing to test a theory is to abandon science itself and the critical scrutiny that its methods bring for verifying the validity of its predictions (Henig, 2008). To fully test this theory, however, we would need to construct an empirical study where theorydriven hypotheses or specific interventions are rigorously evaluated by some measures of research receptivity and use. That day is still before us. But we can assess our theory by how well it comports with empirical studies and the views of policymakers regarding how their world works. This is a less exact test, but it is a place to start. We found that some of the domains of our theory, as specified in the first edition, were incomplete or insufficient, which resulted in adding some new domains, dropping some original ones, and combining others in ways more reflective of current research and practice. We identified six domains that we believe cut across diverse institutional settings: work context (the time frames and decision-making processes that exist); interactional preferences (the interpersonal qualities that contribute to productive interactions and the preferred styles of communicating); favored epistemological frameworks (how inhabitants know what to believe and what criteria are used to screen and sort out conflicting information); dominant influence loops (which signals and rewards define effectiveness and how to be effective in shaping the attitudes and beliefs of others); focal interests (salient topics and goals); and salient stakeholders (what audiences are of prime interest and to which voices do inhabitants pay attention). The important point is that differences in how the inhabitants of each community or island fall across these domains is the key to understanding cultural variation. In Table 7.1, we lay out these foundational domains of institutional culture. 168

W hy R esearch I s U nderutilized in P olicymaking

Table 7.1 Domains and Themes of the Institutional Culture of Policymaking Domains I. Work context

II. Interactional preferences

Themes (A) Time frames—Which kinds of pressures do inhabitants face in their world? What is the dominant pace of activity and to which kinds of time pressure are they exposed? (B) Decision-making processes—How does the decision-making process work? Which factors and processes influence how decisions are made and the ways the work product gets done? (A) Interpersonal qualities—Which qualities or style contribute to productive interactions? Which qualities or style can interfere with effective communication? (B) Communication style—Which communication channels are preferred? How important are interpersonal relationships, and how do relationships compare to other influences on getting the job done?

III. Epistemological frameworks

(A) Credibility of knowledge—Which processes and methods do individuals use to know what to believe and trust? (B) Decision-making criteria—How does this individual evaluate which types of evidence to acquire to factor into decisions? How do inhabitants screen and sort out conflicting information?

IV. Influence loops

(A) Organizational signals and rewards—What defines effectiveness in their world? How does an individual define success? Which institutional cues do they respond to and which indicators signify success? (B) Effectiveness strategies—How do inhabitants shape the attitudes and beliefs of others, and how are their attitudes and beliefs shaped by others? Which ways of presenting information are seen as being most effective?

V. Focal interests

VI. Salient stakeholders

(A) Salient topics—Which substantive topics or challenging problems attract interest and attention? (B) Salient goals—What is the nature and purpose of policy goals? From whom do ideas come? (A) Salient audiences—Which constituencies are of prime interest? To whom do they direct work products? (B) Salient voices—To whom do inhabitants pay attention and from whom do they seek input?

We propose these six domains of institutional culture are seminal to identifying where dissonance arises between the communities of researchers and policymakers. For each domain, we insert two themes that give a sense of which aspects of the work world it encompasses. The themes are meant to be merely illustrative, not exhaustive. Guided by the table, we describe each domain, and then review empirical evidence on how it plays out in the research, policy, and intermediary communities. We also provide a real-life example of how policymakers perceive each domain. Because on-the-ground studies of lawmakers are rare (Fox, 2010; Mayhew, 2011; Merriman, 2019), we turn to our study of state legislators. We let their insights inform us on how these institutional factors shape their views and behaviors, drawing from one of our recent studies (Bogenschneider, Corbett et al., 2019). 169

N ew E ra of T heory , R esearch , E valuation

Domain 1—Work Context Work context taps aspects of the institutional culture that govern what the world of its inhabitants looks and feels like. This domain captures the circumstances under which inhabitants are likely to feel comfortable or essentially in their element in the workplace and the circumstances in which they are they likely to feel out of their element. Of many institutional factors, we focus here on the time frames of institutional life and the ways that work decisions get made. Time Frames Time frames vary across communities (Bogenschneider et al., 2006; Myers-Wall, 2000; Small, 2005). Researchers sometimes think short term, given their focus on academic semesters and grants funded for a year or two (Myers-Wall, 2000), but they also think long term on issues such as their research agenda. Making progress in research or theory is thought of as a timeless pursuit that can take years, or even a career, to achieve (Gallagher, 1990; Shonkoff, 2000). At each stage, care must be taken—­theories methodically tested, experiments replicated, results vetted, and all steps subjected to endless scrutiny and debate. In academia, there are payoffs “for following a path that will prove you to be right 10  years from now, not necessarily tomorrow” (Henig, 2009, p. 148). Policymakers, on the other hand, typically are called upon to respond quickly in a fast-paced, fluid environment where progress can occur within weeks. They must shift attention from topic to topic as a kaleidoscope of issues and crises can pass before their eyes on a continuous basis. And they hope to have something to show constituents and campaign donors in real time. Intermediaries situated at the nexus between research and policy put a high premium on timeliness. They are intentional in following the lead of the policy community to determine what issues they are thinking about, not what the research community wishes they were thinking about (K. Bogenschneider, 2020a; Shonkoff, 2000). Yet they also remain realistic in the realization that attention to timing is absolutely necessary, but seldom sufficient: “Even when scientific information is tailor-made, properly packaged, and communicated in ‘Dick and Jane’ language”, there is no guarantee that it will be used (Caplan, 1983, p. 257). Nathan Caplan relayed his experience providing a rigorous study on a pressing issue of the day—improving the educational performance of low-income refugee children living in U.S. cities. It turned out that refugee children from Southeast Asia, who knew little or no English upon arrival, experienced remarkable school success three years later with 27 percent achieving an A average and 52 percent a B average. In follow-up studies, Caplan was able to identify the pathways through which this school success was realized. Yet even though this research on a timely topic came to the attention of educational policy leaders in influential government positions, it was largely ignored (Caplan, 2014). Decision-Making Processes Across communities, great variation also exists in how decisions are made. Some work environments rely on a highly formulaic processes, whereas others incorporate consensus-type means for arriving at decisions (Corbett, 2018a). Despite these differences, inhabitants of one community are prone to projecting the processes they are familiar 170

W hy R esearch I s U nderutilized in P olicymaking

with onto another community (Kendall-Taylor & Levitt, 2017). For instance, researchers, who are familiar with the scientific method, may misconstrue the decision-making process in a policy community by looking for a systematic, step-by-step progression to a preferred policy outcome. However, in the policy world, decisions are made through a consensus-building process of negotiation and compromise among competing interests (Albaek, 1995; Corbett, 2018c; Shonkoff, 2000). This deliberative process of policymaking is not transparent nor linear because consensus may be achieved at multiple points along the way before concrete action is taken. The vagaries of the policy process confound the efforts of research and intermediary communities in tracking the contributions that research evidence makes to policymaking. The influence of research is not likely to be found in the strength of any particular study, in the independent power of any single individual, nor at any given point in the policy process. Instead, research may have considerable influence throughout the process in ways not readily transparent to outside observers such as contributing to the accommodation of opposing views, coalition building, persuasion, rhetoric, and reconciliation of conflicting values (Albaek, 1995; Prewitt et al., 2012; Shonkoff, 2000). Researchers who have come to understand how research flows through the policy process have suggested a new generation of scientific study. The National Research Council (Prewitt et al., 2012) recommended that science should move beyond examining how research is used to achieve specific policy outcomes and toward how research is used to improve the decision-making process itself. This is not a new idea, having been raised by a pioneering researcher Carol Weiss (1978) over 40 years ago, and supported in several recent studies in which policymakers expounded on the role of research in argumentation and compromise (Bogenschneider, Corbett et  al., 2019; Bogenschneider, Day,  & Parrott, 2019; Yanovitsky & Weber, 2020a). Understanding how research can contribute not only to the products of policymaking, but also to the policy process is central to advancing the study and evaluation of research use (see Chapter 8). Decision-Making Processes in the Policy Community of State Legislators To outsiders, this nonlinear and even chaotic policy process sounds unpredictable and downright inefficient. But our state legislators interpreted this ebb and flow as an integral part of a rule-governed process for fine-tuning ideas and making better laws. The necessity of being attuned to political sensitivities from so many sources shaped how policymakers defined their job. One legislator saw an integral part of his job as building consensus among opposing views: One day I can have all the ag partners around the table. The next day I have all the environmental groups around the table and tell them exactly where the ag guys are. The next week, the ag guys come back. I tell them exactly where the environmental folks are. So, you’re here. . . . If you guys want something done that benefits you, you’ve got to come here. . . . Otherwise, nobody’s going to get anything. . . . We had a hearing on a bill in the Senate—one of the first times in my recollection that every major group registered undecided because we finally brought them all to the point where they’re going to listen to testimony, instead of being against it. . . . That’s the biggest goal. . . . Nobody’s ever going to win everything. No bill is perfect. Everybody has to understand you can get as much as you can, but you’re never going to get everything. 171

N ew E ra of T heory , R esearch , E valuation

Legislators who are most effective in consensus-building possess the personal qualities of curiosity, the ability to listen to the ideas of others, and openness to compromise. Policymakers were consistent in describing the benefits of compromise for politics, but also for policy. To cite one example, a Republican committee chair from a state where his party held a quorum-proof majority still met regularly with the ranking Democrat: [I] go through their ideas about how they think everything ought to be handled. And I  evaluate how much of what they’re saying is worthy enough so that I should make some concessions, even though I don’t have to. . . . I don’t think the whole population of Indiana . . . think all like Republicans or all like Democrats. So, if I don’t moderate the strength of our position on things where it looks like there’s a justifiable reason to moderate it, then I think we will lose control because we are showing them we cannot handle being the majority and doing . . . the right thing for most of the people. Legislators thought about compromise in pragmatic terms. They explained that “those who come to the [legislature] demanding all or nothing almost always get nothing” because the reality is that there has to be compromise “to get anything done here”: At the end of the day, that half glass is better than the empty glass. . . . You never want to stand in the way of having something passed. . . . Be satisfied with what gets passed, even if it’s not everything you wanted and then, next session, move on. Yet these benefits of compromise do not come without costs. Policymakers acknowledged that not everyone would agree with the compromises that had to be made to move the issue forward. One legislator came to understand the nature of these disagreements based on his experience as a parent: I have six kids at home. . . . None of them ever get everything they want, so why should anybody here. It’s easy. The politics at home is the same as the politics here. Somebody’s always upset. . . . Nothing’s ever fair. You just have to be the arbitrator of that and just put an end point to it and say, “Okay, this is the best I’m going to get for everybody.” Otherwise it would just endlessly go on. Legislators did complain that the checks and balances of policymaking make the whole process appear inefficient and arcane to those looking in from the outside. When compared to running a business, the legislative process is “just long. It’s like turning around a barge versus turning around a speedboat.” Yet there are built-in institutional constraints that force policymakers to make decisions (e.g., election cycles, session deadlines, and pending votes), often with incomplete or inconclusive information: Avoiding analysis paralysis is part of the challenge in this business as well. . . . You could research every issue to the end and at some point, you need to make a decision. . . . You have to be willing to work with what you got and be willing to move forward. Domain II—Interactional Preferences Interactional preferences tap into the patterns of communication and exchange that are preferred in an institutional setting. These are dictated, in part, by the personalities and 172

W hy R esearch I s U nderutilized in P olicymaking

personal attributes that policymakers bring to the policy institution, which are reinforced by their previous education and training. Preferences also are dictated by the types of institutional interactions that are most effective in producing the core technology. Interpersonal Qualities A critical question is what qualities or styles are effective in the workplace and contribute to productive interactions and communication. For researchers, the human element, while not irrelevant, is viewed with suspicion. Truth is located in the data that must be employed carefully to avoid human manipulation in accord with prevailing value premises or subjective prejudices. This is the reason for statistical tests, replication of results, and continuous vetting of findings by one’s peers (Shonkoff, 2000). In contrast, those in the policy community typically function within a web of interpersonal relationships. Because no single individual has enough power or authority to implement a decision on their own, policymakers pay as much attention to the preferences of their colleagues, the values of their constituents, and pressure from their party leaders as they do to empirical evidence. Researchers are likely to share information with others in writing, whereas policymakers and practitioners are likely to communicate with others in person or by phone so that they can build a relationship (Myers-Wall, 2000). Policymakers referred to themselves as “people-people” (Bogenschneider, Corbett et al., 2019), who operate in interdependent work environments, which elevates relationships to paramount importance (Caplan, 1979; Contandriopoulos et al., 2010). In a culture that relies heavily on relationships, trust is “the glue that holds relationships together” (Tseng et al., 2018, p. 8). It is trust that facilitates communication and communication that, in turn, facilitates trust (Caplan et al., 1975; Shonkoff, 2000; Contandriopoulos et  al., 2010). Interpersonal trust is so important that trust in research evidence often depends as much on the credibility of the source as on the rigor of the methods (Bogenschneider & Bogenschneider, 2020). Those operating in the nebulous world of intermediary organizations balance multiple factors, depending on where they are institutionally located. Agenda-driven intermediary organizations, even if they pretend to be nonpartisan and research-driven, often act more like policymakers. Perceptions and positions may be shaped by empirical evidence, but also are driven by factors such as values and political preferences. Intermediaries who see themselves as honest knowledge brokers pay more attention to methods and the canons of science in pursuing truth. Yet they still expend a great deal of energy in cultivating relationships with policymakers. They realize that the more one gets to know policymakers as people, the more likely it is that they will trust you and the easier it becomes to target their interests and purposes for using research (K. Bogenschneider, 2020a). Communication Style Economic theories have dominated our understanding of the way that interpersonal interactions work in policymaking, yet the marketplace does not always map neatly onto the policy world in meaningful ways. In the market model, people independently define their interests and know specifically what they are, whereas in the policy world the formation of interests is much more dynamic and shaped by the process itself (Stone, 2012). In the private sector, market entrepreneurs have more sway over the outcome of an innovation than policy entrepreneurs do in the public arena; unlike market decisions where sellers design products and unilaterally pitch them to buyers, policy decisions are 173

N ew E ra of T heory , R esearch , E valuation

arrived at through deliberation and negotiation among diverse interests with divergent views. The “two-person models so prominent in economics and game theory are politically empty” (Stone, 2012, p. 27). In economics, interactions occur between buyers and sellers, whereas in policymaking, interactions occur between friends and enemies within a shifting array of relationships. In contrast to market-based relationships, the norms in policymaking are that people cooperate with friends, forgive enemies, and, once alliances are formed, stick by them (Stone, 2012). Communication in lawmaking bodies transpires in an oral tradition that is long and strong (Weiss, 1987). Ideas are collected and communicated by listening to testimonies, meeting with constituents and stakeholders, debating with colleagues, and so forth (Fenno, 1978). In lawmaking, a market-based model of members trading prefabricated ideas with each other for personal gain is less useful than a community-based model of members working out ideas in a social ecology of relationships (Tseng, 2012). A political scientist described the nature of communication in the relational policy culture: [Policymakers] live in a dense web of relationships, dependencies, and loyalties; where they care deeply about at least some other people besides themselves; where they influence each other’s desires and goals; and where they envision and fight for a public interest as well as their individual interests. (Stone, 2012, pp. 10–11) Interactional Preferences in the Policy Community of State Legislators The legislators in our study rated relationships as more important than other influences on decision-making such as policy expertise, research evidence, and good ideas (Bogenschneider, Corbett et al., 2019). Coming up with a good idea, according to a Democrat, is really the least effective way to be a legislator. She explained that without the support of interest groups, a legislator needs to erect “way more infrastructure” to move an idea forward, no matter how good it is. A Republican also highlighted the role of relationships in turning good ideas into law: “If you don’t build those relationships, at the end of session, when your bill doesn’t get anywhere, all you have left is a good idea.” When first elected, one legislator had assumed that an important ingredient of success was being knowledgeable about policy, but he quickly learned that “relationships are the biggest thing”. In my first session here, I  had thought that the most effective use of my time would be not be here at all—to be at home all the time just being among the people and coming here to vote every so often. I didn’t get anything done because I didn’t build any relationships here. I didn’t have anybody to work with that idea. I tried to get a bill out into the Senate. It was a wasteland because I never talked to a senator about anything. Of the bills that are introduced each session, only a fraction ever become law. One legislator explained why the institution is littered with so many unrealized dreams: I think a lot of us like to pretend that politics is all about issues. It’s all about black and white and what the numbers look like and what the data tell us. But it’s a really personal business, both in this building and when you’re trying to build support in your community, so that they send you back here. It’s not all about the facts and figures. 174

W hy R esearch I s U nderutilized in P olicymaking

Policymakers described the nature of relationships as being much like those with friends, where you forgive shortcomings. According to a Republican: “With your colleagues, you’re going to need their support and they’re going to need yours. . . . You better have a short memory.” A Democrat, who had strong relationships within his own party and across the aisle said: “What is shared, I think, in all of these is the ability to forget yesterday and focus on the task at hand today or tomorrow.” Domain III—Epistemological Frameworks Epistemological frameworks tap ways in which members of distinct communities think about and acquire knowledge. How do we know what we know? What information is accepted as valid, usable, and worthy of being included in one’s work? What calculus does one goes through in arriving at what is truth? What is the basis for deciding how one should act based upon knowledge? Credibility of Knowledge The academy and other research producers have strict methodological canons to be followed before purported evidence is accepted as truth (Shonkoff, 2000). Anonymous peer-reviewed assessments are done, studies are replicated, and theoretical positions slowly yield to new interpretations as knowledge gradually accumulates (Henig, 2009). Basic researchers approach knowledge building from a skeptical perspective. The starting point is the null hypothesis—the premise that a hypothesized truth is “not” true until alternative explanations for the observed reality have been ruled out as false. “Real” researchers always look for reasons to discount purported findings. Skepticism is the dominant posture in the academy. David Olds and his colleagues have modeled this patient pursuit of the truth in their development and evaluation of the Nurse-Family Partnership program. Over the last 30 years, this program was designed based on scientific evidence and evaluated in multiple locations using the gold standard of scientific methods, the randomized controlled trial. The evaluation results have been published in the top medical journals with impressive results. When contrasted with comparison groups, nurse home visiting has been shown to prevent child abuse and neglect among high-risk, low-income families who are visited prenatally and during the first two years of a child’s life (Eckenrode et al., 2010). In 2009, the nonpartisan Coalition for Evidence-Based Policy issued a statement supporting the effectiveness of Olds’s program and suggesting that six other home visiting models had fewer validated effects (Haskins, Paxson,  & Brooks-Gunn, 2009). Despite these authoritative endorsements, Olds (2009) remained cautious in his assessment of the program’s effectiveness: There is a web-based information system in which family nurses—program sites—enter data on every visit that allows us to monitor the ongoing performance of the program, to use that information for continuous quality improvement, and for research on improving the basic model because the basic model is not good enough. We are pleased with it, but . . . it will always be a work in progress. . . . There are a number of things that we think that we can do better as we roll this program out. Statistical significance is of utmost importance to researchers who use data to produce knowledge; policymakers, on the other hand, may use data to support an agenda 175

N ew E ra of T heory , R esearch , E valuation

(Shonkoff, 2000), to avoid embarrassing mistakes, or to circumvent unintended consequences, (Bogenschneider, Corbett et al., 2019). For policymakers, research is not the only or even the preferred source of guidance, but competes with a multitude of other inputs such as common sense, intuition, lived experience, the positions of stakeholders, and one’s personal values (Myers-Wall, 2000; Shonkoff, 2000). What’s more, policymakers are always balancing other factors like cost, economic feasibility, and political acceptability (Bogenschneider et al., 2020; Caplan et al., 1975). For policymakers, there is a huge difference between what is statistically significant and what is substantively important in making a policy decision (Goodvin & Lee, 2017; Small, 2005). Intermediaries, particularly those of the nonpartisan variant, recognize the importance of statistical significance, so they screen for studies of the highest quality (K. Bogenschneider, 2020a). Yet they also understand the importance of substantive significance. For example, if a Family Impact Seminar has a large effect on only one legislator in a statehouse, that level of research use would certainly not approach accepted levels of statistical significance. However, if that single legislator used Seminar research to design and pass a research-based policy that benefited thousands of families in the state, that surely would be considered substantively significant (see Chapter 8). Decision-Making Criteria Ideas are a “key form of power in policymaking”, according to award-winning political scientist Deborah Stone (2012, p. 36). Political observers argue that the “quality of our political ideas may be every bit as important as the quality of our political leaders” (Eggers  & O’Leary, 2009, p.  27). Decisions about how to acquire ideas and sort out their quality are shaped primarily by factors identified in the psychology literature— the context in which decisions are made, the source of the message, and the medium through which it is delivered (Stone, 2012). Starting with context, the heavy workload of lawmaking bodies shapes decision-making criteria. For legislators to stay on top of all the issues they face is virtually impossible without relying on sources they trust—the advice of colleagues, constituents, lobbyists, and friends (Rosenthal, 2009). Lawmakers’ positions and decisions are shaped by many inputs ranging from science to stories, from politics to principles, from intuition to special interests, often depending on which is most politically persuasive (Prewitt et al., 2012). The medium preferred by policymakers is the spoken word; the written word, no matter how slickly reported, usually winds up “gathering dust” (Weiss, 1999, p. 201). Weiss famously said that policymakers “take pride in their ability to ‘read people’ rather than ‘read reports’ ” (Weiss, 1989, p. 414). Lawmakers see their task as listening to multiple sources of information and figuring out the truth by sifting through the strengths and weaknesses of competing arguments (Weiss, 1989). In sorting out the truth, many policymakers often rely on intuition, which can fail spectacularly (Watts, 2011). One of the most robust findings in the history of psychology is the superiority of statistical to intuitive judgments (Pinker, 2018). Science can bring to policy decisions an aura of authority because its norms and procedures set it apart from other information sources (Albaek, 1995; Henig, 2017). As aptly put by the National Research Council: What science has to say about policy choices results from investigations governed by systematic and rule-governed efforts that guard against . . . self-deception . . . believing something is true because one wants it to be. . . . Science is generally 176

W hy R esearch I s U nderutilized in P olicymaking

a more dependable and defensible guide than informed hunches, analogies, or personal experience. (Prewitt et al., 2012, p. 3) Favored Epistemological Frameworks in the Community of State Legislators The epistemological frameworks legislators use to sort out what to believe is shaped by the contours of the policy context. In our study, the sheer workload was one factor. Legislators explained that it is literally impossible to read and research every bill, so they turned to the readily available sources they knew and trusted—their colleagues. Their colleagues possessed a plethora of professional training and real-life experience that (for some legislators) were the hallmarks of credibility. You’re going to trust your colleagues. You’re going to listen to their view on it . . . because there’s too many—the volume is too great. You’re pulled in too many directions. . . . So that’s why those relationships with your colleagues are most important. Sorting out which colleagues you can trust is a big challenge, according to one legislator: At the end of the day, people have to . . . feel that they can trust you. And that trust has to come from building some kind of a relationship. It can’t be, I don’t know this guy—never talked to him. They’ve got to have some sort of connection that gives them the confidence to say, “I’m going to go out and, even though I’m not an expert on this issue, I’m going to vote with this guy because I know him, and I trust him. And I think he knows this issue.” In a system that operates on relationships, legislators do not lightly bestow trust in their colleagues. Using information that proves to be wrong can harm their reputation, often irreparably, with colleagues and constituents: Somebody being dishonest and giving you the wrong information . . . and knowing that you might rely on that, and then personally be made to look like an idiot when somebody corrects you. That’s not cool at all. . . . People just won’t deal with you. Given the barrage of people that deluge lawmakers with information, legislators shared several ways of determining who can be trusted, with one being the “BS Detector”: “I’m not the smartest guy here. I’m not the most politically savvy, but I do think I have a pretty good sense of people.” This legislator went on to explain when he was most apt to rely on this internal gut check. “I can normally tell when things just don’t smell right or when it seems like somebody’s got an ulterior motive—that seems to be more where I [rely] on my feel about whether I trust them.” Another legislator explained that his BS detector works best when applied face-to-face: One of the things I take a lot of credit in is meeting with individuals who provide that information, so I can see who they are and kind of gauge what kind of person or what kind of [information]. . . . If somebody comes in . . . and say[s], 177

N ew E ra of T heory , R esearch , E valuation

“Hey, can I have like 10 seconds?” . . . It makes a lot more sense to me than a form letter of research that I get from other interest groups. Intuition and gut instinct play a big role in what policymakers believe (Watts, 2011). And intuition is informed by how the policymaker responds to those delivering the information. One legislator combined this ability to read people with personal agendas into a barometer that he calls the “trust meter”. In relationships with friends and loved ones, his trust meter is 100 percent. It is not that high with a lobbyist who clearly has a job to do and an agenda to pursue: When you’re dealing with someone who has an agenda . . . it’s not that I’m not believing them, I do. . . . It’s not that I don’t trust them . . . or I think that they’re dishonest. . . . [But] their agenda is here and I’m giving them credit for 50 percent . . . or 75 percent or 25 percent or 10 percent. Not all legislators were confident in their ability to “read people”. Some registered doubts about intuition: “Good policy isn’t always intuitive and it’s often not good politics. . . . That’s I think one of the biggest challenges. Good policy is often counterintuitive”. For example, a former governor did not rely on his “gut check” alone, but also factored in stories, education, and research. In our study, legislators frequently spoke about how they use research to ferret out truth: There’s a huge misconception that the public has that we just base our decisions based on what we’re told to do, or a quick perusal of the information. And I really take issue with that. I have found my colleagues, myself—I mean, are we perfect? No. But are we diligent and do we try to understand the issues? Absolutely! And there is good nonpartisan research. . . . I was impressed as a newcomer. I had no idea that there would be this much non-biased information provided to me. One key informant with over four decades of policy experience explained that the legislator’s use of research runs the gamut. On one extreme is the policymaker “who is just interested in making sure that constituents are happy, probably doesn’t use any research, doesn’t care at all, and typically will go along with the leaders on votes, or rely on somebody else to tell them how to vote”. On the other extreme are policymakers who are committed to making decisions that research suggests will work and who are willing to “run the risk of alienating their constituents from time to time. . . . They just say, ‘Hey, I’m here for six years, I’m here for eight years, that’s it! . . . I want to make a difference.’ ” Lawmakers also fell between these two extremes, according to this key informant, who clarified that the ones with the most influence “have knowledge, and knowledge undoubtedly comes from getting information, whether it’s research or whatever else you want to call it. And they will be very careful not to just jump into something without having some knowledge of it.” When specifically asked about legislator’s use of “university” research, nine of 13 key informants said its importance had declined in the last couple decades because of unavailability, perceived bias, and “distance from the customer”. The source that legislators overwhelmingly cite as most trustworthy is in-house—the legislative service agencies responsible for bill drafting, budget and fiscal analysis, committee support, issue research, etc. The epistemological factor that fosters this trust is nonpartisanship: 178

W hy R esearch I s U nderutilized in P olicymaking

The reality is there are very, very few completely nonpartisan, nonpolitical data sources.  .  .  . Anyone who’s doing the research has got a stake in the game. Otherwise, why are they doing it?  .  .  . I  haven’t found a lot of [nonpartisan sources] . . . unless it’s one that is already been funded by the state . . . the Fiscal Bureau [a Wisconsin nonpartisan legislative service agency].  .  .  . We consider that to be the gold standard for data. Domain IV—Influence Loops Influence loops tap into the way inhabitants of a community effectively shape each other’s attitudes and beliefs. Institutional environments provide a continuous stream of feedback designed to control and shape behavior in acceptable ways. Which ways of communicating information, whether conscious strategies or inbred patterns, are most effective? Information is not merely data or facts, but substance that gets filtered through the norms and expectations of one’s cultural envelope. That is, what kinds of things do they look for in their professional and institutional cultures to determine if they are playing by the formal or informal rules? How are inhabitants rewarded and punished for their activities, and to which cues must they pay attention? Organizational Signals and Rewards For academics to gain influence in their culture, publications are what counts with soleauthored works in high-impact, peer-reviewed journals being the best way to obtain professional glory. Anonymous reviewers of journal articles are life and death to them. Academics and top researchers operate in a highly competitive environment. People get to the top of this world after years of competing with others to graduate to the next academic level, to obtain advanced degrees, to land jobs in top universities, to get tenure, to secure research support, to publish in the best journals, to win prestigious named positions, and so forth. In such a world, scientists more highly prize the statistical significance of their findings than what specifically is found (Shonkoff, 2000). Policymakers also operate in highly competitive environments. For policymakers to gain influence, they must compete in and win elections (Mayhew, 2011). In Karen’s study, key informants questioned why anyone would want to run the election gauntlet knowing full well the embarrassment that any past indiscretions could bring to themselves and their families. One state legislator deplored campaigning, which she explained was the worst part of the job: Every politician in America has to do it, so, I can’t whine. But I hate it. Would you like to pick up the phone and ask people for money to help you retain your job every two years for the kind of money we make and the hours we put in? You’ve got to be a little insane. We’re deeply committed. In policy settings, the signals and rewards may not be uniform, but may be shaped by levels of partisan polarization (Binder, 2003; Haidt & Hetherington, 2012; Peterson, 2018). Its influence varies across institutions and issues, and depends on local circumstances. Effectiveness Strategies Researchers are most effective in shaping the attitudes and beliefs of their peers by publishing rigorous research in high-impact journals read by colleagues in their area of 179

N ew E ra of T heory , R esearch , E valuation

specialization. The most effective way to impress one’s colleagues is by fastidiously following the formulaic structure of a journal article. These articles start with extensive literature reviews and a rationale for why this work will advance our theoretical and empirical understanding. Great attention is paid to describing the data used, the soundness of one’s methods, and the rigor of one’s analysis. Because the most impactful work suggests causality, one must carefully establish the credibility of such claims. Researchers feed on sophisticated analysis, complexity, nuance, and intellectual challenge (Shonkoff, 2000). Policymakers shape the attitudes and beliefs of their colleagues through two dominant influence loops characterized as a policy paradox. In a democracy, influence occurs when policy decisions are formulated through channels of “debate, argumentation, and the play of opinions”, or, alternatively, through channels of “power” and “horse-trading” (Albaek, 1995, p. 90). Much perspective is lost if policymaking is considered through only one of these channels—either argumentation or power—without attending to the fuzzy line between the two; in policymaking, influence often “spills over into coercion” (Stone, 2012, p. 27). In a democracy, “politics will always take the form of argumentation” (Albaek, 1995, p. 91) and deliberation (Benhabib, 1996). Policymakers argue and deliberate to inform and critically reflect on the merits of their own positions, but also to rehearse and refine good reasons for their positions that can arouse their audience and persuade them to their point of view (Albaek, 1995; Benhabib, 1996). Because legislators compete in the political marketplace of ideas, communicating well is a valued commodity (Friese  & Bogenschneider, 2009). Great value is placed on simplifying complex matters so they can be communicated effectively (Weiss, 1989). Cut to the chase quickly. Start with the emotional impact and bring it home with data (Bogenschneider et al., 2020). Arguably, some of the smartest national politicians never connected with the public because they lacked that ability to keep it simple, and some of the best communicators never connected because their simplistic messages lacked substance (Heath & Heath, 2007). In this domain, intermediaries, as in so many other areas, are torn. On the one hand, they are drawn to sophisticated methods. After all, many were professionally socialized in ways similar to researchers in top universities or evaluation firms. They want any original work they do to stand up to serious scrutiny. At the same time, they see policymakers as a primary target audience. Therefore, they want to reduce the complexity of research and assert findings with a boldness not found in the genteel scholarly world. This is quite a balancing act—to be correct, comprehensive, concise, and confident, all at the same time (K. Bogenschneider, 2020a; Heath & Heath, 2007; Maynard, 2006). Influence Loops in the Policy Community of State Legislators In our study and others, the primary way policymakers used research was for persuasion (Bogenschneider, Day, & Parrott, 2019; Yanovitsky & Weber, 2020b). Sometimes this use is post-decisional. A preferred path is selected, based on constituent interest or partisan preference, and then research support is sought to argue for that direction. However, a former legislator reported that he used research upfront to construct and formulate an argument. In a work setting in which effectiveness depends, in part, on one’s ability to persuade, a Republican described how powerful research can be in building an argument: “I think the intersection of policy and politics is that you don’t get your policy enacted unless you can convince people—and research helps convince them.” Legislators also gave examples of when “sound, thorough research” ended up taking “politics out of the discussion”. A  Democrat described how he and a Republican 180

W hy R esearch I s U nderutilized in P olicymaking

colleague used research that had a direct effect on the passage of a law to allow family and medical leave for organ donation: That was very much driven with the help of the hospitals and the medical schools about where we rank compared to other states.  .  .  . I  mean there were some Republicans who had said “That’s a government mandate. I’m opposed to all government mandates”. . . . But [having] doctors and medical researchers standing around both of us to say, “This will save lives.” And all the evidence is that when you go to these policies, the more people can be organ donors. . . . So, it passed, and Governor Walker signed the law. . . . I could write a couple volumes of how much I disagree with Governor Walker on both his policy and his politics. But on that day, he was signing a bill into law that I worked on a bipartisan basis and was able to use good research and good evidence-based publications out there about why this works and why it will actually be a solution. Legislators also explained how power politics—partisan agendas and campaign strategies—can trump efforts to use research to persuade colleagues (Bogenschneider, Corbett et al., 2019). When persuasion no longer works, policymakers turn to coercion: In that situation, no amount of research, no amount of facts will change their mind. Because they were not to be persuaded. Ultimately in that kind of situation, all you can do in politics, if you can’t persuade them, sometimes you just have to defeat them. When research bumped up against the sacred—the brick walls of ideology, fixed positions, and partisan agendas—the integrity of the research and researcher was attacked: If it fits our strategy, our outcomes, it’s good. If it’s not, then we say, “No, this doesn’t fit where I wanted this train to go.” Then you start questioning was it really good research? . . . Were there things they didn’t measure that they should have? Research also was used selectively as a weapon in a larger political game. Legislators strategically decided whether research should be “thrown out first” or the “last card” played. One explained, “We play games with research. We play games with facts— it’s just what politics is. [Interviewer: “How do you work around the games?”] You don’t. . . . You play them.” Domain V—Focal Interests Focal interests give us some understanding of what substantive issues or more theoretical questions dominate institutional life. What topics or salient issues are on an institution’s front burner? How do they get there? How much discretion or freedom does an inhabitant of an institutional island have in selecting salient goals to guide what they or their organizations will do? These are critical differences that need to be appreciated. Salient Topics We know, for example, that topics are salient for different reasons in the research and policy communities. Researchers tend to gravitate toward more theoretical and abstract 181

N ew E ra of T heory , R esearch , E valuation

topics, whereas policymakers are moved by everyday problems that need concrete solutions (Caplan, 1979). Researchers are attracted to topics that cut across time and space knowing full well that their quest for knowledge never seems complete (Shonkoff, 2000). The interest of policymakers, however, can run hot and cold. A salient topic suddenly loses political momentum and falls off the policy agenda, which can happen in an instant (Bogenschneider, 2014). Researchers search for ultimate answers that are irrefutable, whereas policymakers need fast actions that can be fixed later if they fall short (Shonkoff, 2000). Yet not all policy questions function on a real-time basis. Policymakers may have longer time horizons for more complex questions. Take, for instance, when blue ribbon commissions or study committees are established that convene experts and citizens to examine challenging problems facing a state or the nation. These more in-depth investigations develop policy options or specific pieces of legislation that often are enacted in future sessions (Rosenthal, 2009). With more time for investigation, research evidence is more likely to be used (Bogenschneider & Bogenschneider, 2020). Also, research use is more likely on topics of low or moderate salience (Gormley, 2011), and less likely on highprofile issues driven primarily by passion (Bogenschneider & Bogenschneider, 2020). The communities also differ in the ways that salient topics come to their attention. Research topics often arise from the literature, from questions being debated in top journals, or from issues for which funding is available. To select the topic, the research community follows the linear models of rational decision-making by precisely identifying what the problem is, whether it is defined correctly, and if it is a problem that could or should be addressed with the tools of science. For policymakers, identifying salient issues is much more fluid and erratic. In the fastpaced world in which policymakers operate, they always need to be ready for any emerging issue that the public may see as a priority that deserves government action (Henig, 2008; Stone, 2012). Psychologists have identified a cognitive bias in the way issues come to policymakers’ attention that sometimes has resulted in erratic and misplaced policy priorities. The availability bias, as it is known, errs by judging the importance of a risk, not by how often a situation occurs, but by how easily (and emotionally) it can be brought to the conscious mind (Kahneman, 2011). With public policy issues, an availability cascade can occur when the media brings a relatively minor event to the attention of the public. A  segment of the public becomes worried, which prompts additional media coverage and a flurry of attention-grabbing headlines. Scientists and others who try to dampen the fear are ignored or, worse yet, accused of being involved in a cover-up. This cascading of events escalates the political importance of an issue because it is on everyone’s mind. This availability bias can reset priorities as other more important uses for public resources fade into the background. Herein lies one of the main problems of government, which academics refer to as Type III errors—successfully solving the wrong problem (Dunn, 1998). For intermediaries, the salience of topics is a prime concern (K. Bogenschneider, 2020a; Contandriopoulos et al., 2010; Maynard, 2006). At the same time, they recognize how extraordinarily difficult it is to identify the most salient policy topics because they are always changing, which also makes it difficult to identify the most relevant research (Maynard, 2006). Intermediaries are perhaps more conscious than most regarding how the prevailing political winds can shift direction in each election, how the players change as policymakers are sworn in and voted out, and how policy issues quickly can rise and fall on the political agenda (Bogenschneider & Corbett, 2010a). For example, in 2006, Karen spent six months planning a Family Impact Seminar on long-term care. A  couple of weeks before the scheduled event, Wisconsin’s governor scaled back the reach of his long-term care initiative from a statewide effort to a pilot 182

W hy R esearch I s U nderutilized in P olicymaking

project in a couple of counties. In 2010, two weeks before a recipient of the Nobel Peace Prize spoke at a Seminar about jobs for a clean energy economy, green jobs became so highly politicized that all parties agreed no legislation would move forward that year (K. Bogenschneider, 2020b). Partisan polarization can quickly push relevant issues into irrelevance (Contandriopoulos et al., 2010). Similarly, Tom and his colleague Jennifer Noyes had been invited to work with Indiana state welfare and workforce development officials to integrate their agencies and programs. Things were going very well until the governor of the state suddenly decided to privatize the welfare administration function. Program integration completely fell off the radar screen. Focal interests are no different than other domains—cultural confusion is inevitable. Put simply, researchers prefer to focus on questions—what we do not know. Policymakers prefer to focus on answers—what we do know. Intermediaries prefer to focus on figuring out what policymakers need to know (Bogenschneider, 2014; Shonkoff, 2000). The list of potential conflict points is lengthy. To stay in touch with the topics salient to another community, researchers may need policy updates, just as policymakers need research updates (Doherty, 2000). Salient Goals To further complicate matters, many topics that become salient to policymakers have multiple causal models that spawn a mix of alternative policy goals. In the rational decision-making models of the research community, goals are causal statements of what is needed to address a specific problem. In the policy community, however, goals are actionable statements about who would benefit. To help pinpoint political motivations, the first question that policymakers face is not what the policy goal should be, but who benefits from it and where did it come from (Stone, 2012). In contrast to the specificity of goals in the research community, policy goals are often written ambiguously, so the language can be adapted as needed to muster political support from those who might disagree on specifics (Stone, 2012). Vague goals can also allow a policymaker to vote for a bill and then punt its conflictual aspects over to executive agencies for interpretation and implementation (Stone, 2012). In the end, the specific language of a policy goal is less important than the intent—whether the goal improves people’s happiness, which Rosenthal (2009) maintains is the ultimate satisfaction for most state legislators. Of course, when legislators think of keeping people happy, they think first of the constituents in their district, who elected them to office and who will decide whether to reelect them to another term (Fenno, 1978; Mayhew, 1974). In a political context, power tactics are ever-present, none more important than controlling the number and kinds of policy goals and alternatives (Stone, 2012). Here, the language and listing of choices may be critical. One power tactic is to place the preferred policy goal in the middle, which is the most favorable position. Another power tactic is a carefully constructed Hobson’s choice where the desired option is presented positively amidst a list of other alternatives, all portrayed in a negative light. Even better is keeping alternatives off the list altogether: To keep it off is effectively to defeat it. In fact, keeping an alternative from explicit consideration is even better than defeating it, because an alternative that remains unarticulated, unnamed, and unexamined doesn’t lurk around as the focus of discontent and hope. (Stone, 2012, p. 253) 183

N ew E ra of T heory , R esearch , E valuation

Salient Goals in the Policy Community of State Legislators In our study, the optimal policy goal was not always obvious. One legislator raised a textbook example of the complexity of goal setting on an issue she faced with multiple causes and no widely accepted solution. Sexual violence against children came to the attention of this Indiana legislator based on data from the Centers for Disease Control and Prevention: Indiana had the second highest rate in the nation when it comes to rape and sexual assault against girls. So, by the time they’re in high school, one in six girls in Indiana have been raped or sexually assaulted. . . . This was just one of those issues that made me want to set my hair on fire. I thought if one in six girls in Indiana disappeared or had a mystery disease, we’d do something about it. . . . When you say, “Okay, rape or sexual assault, well, that can mean a number of things. . . . How do you combat that? Because, is it random acts of violence? Is it date rape? Is it child seduction?” You know, “I’m not a victim. I’m in love with my volleyball coach.” Or is it incest, et cetera? Well, a lot of different causal relationships there. A lot of different sorts of solution-based legislation. . . . [So] I pushed to get a further study conducted so that Indiana could be using Indianaspecific data. In interviews, legislators elaborated on the varied nature of goals in a political world, describing them as action-oriented, ambiguous, reactive, and long term. For setting action-oriented goals, one legislator found research most useful when it addressed “malleable” factors “that are actionable”: “We must be able to control it in some way. Otherwise, it’s just interesting . . . but it’s not useful to me in my job.” Action orientation, by its very nature, requires compromise, which is best served by ambiguous goals that always seem to be in flux: I don’t deal with black and white very often. The issue I’m dealing with at home right now is going to be 100 percent issue or 0 percent issue for whichever side wins or loses. I keep telling both sides, you’d rather be in that 50 percent area than the zero or the 100 percent. You don’t want to risk losing everything or gaining everything. You got to work together, and each be a little unhappy. . . . Yeah, there’s no absolute outside this building. So, why people think there’s an absolute inside the building just escapes me. Policymakers are often called upon to react and respond to the emergency of the day. “We are very reactionary. . . . If there are fires, we’re going to put water on the one that’s going to burn us first.” For instance, when Indiana hosted the Super Bowl, a law quickly passed to prevent the human trafficking of minors that often comes along with large sports events. Beyond responding to these emergency fixes, legislators also set goals that are longer term in nature. One Republican prided himself on foresighted bills: “I’m looking ahead and I’m being proactive rather than reactive. I’m not waiting for the problem to happen. I’m trying to nip it in the bud.” Legislators told stories of it taking 25 years to enact a smoking ban and multiple sessions to establish a children’s commission. One legislator explained that she strategically engaged in long-term planning because of the incremental nature of policymaking. Another legislator was strategic in seeking partners who shared her long-term commitment: 184

W hy R esearch I s U nderutilized in P olicymaking

It can take years to get something done and I always think long term. So, finding the right partner on an issue knowing that maybe five years from now your idea will become law, you have to have someone who’s just as diligent as you. . . . So, you don’t want to just throw spaghetti at the wall and see what sticks. . . . So, moving the ball forward and finding someone who will carry the ball with you and pass it back and forth is really important. Finally, a Republican who came from a corporate background wanted the legislature to spend more time looking to the future to understand how trends are changing, so those changes could be incorporated into decision-making: Social issues are changing. And as much as people, particularly on the conservative side, don’t like same-sex marriage or transgender issues, these are issues that are coming up. And the younger people that are conservatives really don’t care. . . . They don’t see it as an issue. . . . So, that’s one of the things that I see that we have to . . . not waste our time legislating. Domain VI—Salient Stakeholders Salient stakeholders tap the people and groups that dominate the attention and energies of the inhabitants of any community. Whom do they think about as they do their work? Which voices have the relatively stronger claims on their attention as they scan their respective environments? Salient Audiences For scholars, the most important audience is their disciplinary peers both within and outside their university. Researchers organize their work and expend their energies with an eye on other researchers. (Would we expect anything else?) They look to their peers. The more ensconced a researcher is in the academy, the more likely he or she will focus on a small set of peers with a narrow sphere of expertise, who typically are dispersed across the country and the world. Academic careers are sustained and advanced by one’s reputation among these peers. For intermediaries at evaluation firms and think tanks, they pay most attention to those whom they want to influence (mostly policymakers of various stripes) and those who pay the bills. For state legislators, their most important stakeholders are the constituents they represent and the citizens of the state (Mayhew, 1974; Rosenthal, 2009). When policymakers think of constituents, they think of the people in their district who elected them to office and will decide whether to reelect them. Voters in their district are the most common source of policy goals and often a primary influence on policy priorities. Granted, constituents do vary substantially depending on the heterogeneity of the district, but nevertheless they still are a salient audience with a substantial influence on a lawmaker’s style of legislating and legislative behavior (Fenno, 1978). Salient Voices In a representative democracy, lawmakers are expected to make themselves accessible to pressures and demands from a variety of stakeholder voices (Rosenthal, 2009). By the very nature of their position, lawmakers face the challenge of listening to these demands 185

N ew E ra of T heory , R esearch , E valuation

and then deciding whether to respond to requests that government do something and, if they don’t, wondering whether they will get voted out of office. Depending on their specific organizational location, lawmakers continuously scan their environment to keep track of several sources of potential influence—constituents, interest groups, partisan players, political friends and enemies, the press, and the public (Stone, 2012). Yet these sources of influence are not all equally salient. The Black Lives Matter movement has been the most recent manifestation of disparities across interest groups, with some being stronger, wealthier, and more well established than others (Stone, 2012). Not surprisingly, policymakers’ responsiveness to stakeholders also affects their preferences for research. Research needs flow out of different cultures and expectations. Researchers strive for representative (preferably random) samples, whereas policymakers prefer data on how their constituency stacks up to a similar city, county, state, or region (Kaufman, 1993). Researchers seek population estimates, whereas policymakers seek district estimates to assess its meaning to the constituents they represent. One of the most common ways that state legislators use research for designing legislation is looking to their counterparts in other states to determine what already has been tried and with what effect (Bogenschneider, Day, & Parrott, 2019). Knowing the layout of stakeholders that matter to a community’s inhabitants and have a significant influence on their decisions can be critical to effectively communicating with them. Salient Stakeholders in the Policy Community of State Legislators In our interviews, legislators mentioned a breathtaking number of salient stakeholders, almost equally divided between those inside and outside the legislature. The inside stakeholders include colleagues of both parties and both houses, committee chairs, leadership, legislative service agency analysts, office staffers, and so forth. Outside stakeholders included the two that are mentioned most frequently—constituents and citizens of the state—along with legislators in other states, lobbyists, media, national political organizations, nonprofit advocacy groups, political donors, primary voters, special interests, and so forth. Legislators in the study reported responding differently to issues depending on the stakeholder. In general, legislators defer to constituents in their district in identifying policy goals: “The legislators first will listen to their constituents. . . . If you’re not hearing it’s a problem in the district or it’s something that is not in your principles, then you’re probably going to disregard it.” The most partisan issues where the “real dialogue breaks down as far as the battle on the floor” were the ones “brought in by outside associations or outside resources that have a stake in the game [or] have a dollar in the game”. Legislators also revealed some of the complexities they face in making decisions about how to best represent their constituents. You’ve got to support most of your constituents most of the time or you’re not going to be around. . . . But you don’t have to support ’em all the time . . . You don’t want to be a drone. . . . You don’t want to be a yes person. You have your own views. Another related decision that legislators frequently face is whether to approach their job as a delegate who votes the views of their constituents or as a trustee who votes their own judgment:

186

W hy R esearch I s U nderutilized in P olicymaking

I don’t buy the argument that you can simply be a delegate. Because if you believe that your role is simply that of delegate, then what you’re saying is that all of your constituents know just as much about all of the issues as you do. And that means that either you have some very informed and sophisticated constituents, or you’re doing a very poor job keeping up with the issues yourself. This legislator went on to explain that he votes as a trustee on those issues he deems most important. Even though he realizes that some of his constituents will struggle with his vote, he has found they will accept his decision if he has good reasons, and if he can back up his reasoning with research evidence. One characteristic of lawmaking bodies that differentiates them from other institutions is the extent of influence exerted by outside stakeholders. A gubernatorial key informant described her experience in recruiting top business talent to run state agencies. She observed what a big cultural shift it was for these top executives as they transitioned from running a business to heading an executive agency: “They just couldn’t get accustomed to six million people caring about what they were doing and thinking they had a right to say [so].” In sum, if researchers commit to understanding an institution’s soul, it can do much to ease communication and build trust across cultures and, in so doing, increase the utilization of research in policymaking. Yet understanding culture is a difficult undertaking given its complexity and all the potential conflicts that can arise both across and within communities. We discuss in the next section a framework for advancing our thinking about the complexities of community culture.

Exploring the Complexities of Culture to Improve Research and Practice In our hypothetical policy world, we extended previous theoretical thinking beyond the original two islands (loosely defined as research producers and consumers) into an intricate archipelago of multiple island communities. Our theory proposes more complex thinking about the number and diversity of communities, but our empirical understanding lags far behind. For those serious about improving research and practice, we discuss here what it would take to develop a more sophisticated understanding of the influence of professional and institutional cultures both across and within communities. Cultural Conflicts Across Communities In our archipelago, we envision groups of inhabitants organized into distinct island communities. We have scientists and evaluators, and a host of intermediary institutions along with agency managers and heads, and policymakers and their staffs. Within these major aggregate communities, we can see smaller related communities, such as applied scientists, policy teachers, philanthropists, government analysts, front-line service providers, state legislators, and so forth. One premise of community dissonance theory is that each of these islands is defined by a unique culture that stems from its inhabitants’ professional preparation and its institution’s norms and practices. Slowly, we have created the architecture for understanding the pervasive and nuanced influences that professional and institutional culture play in shaping the disconnect among the various communities that sustain the gap between the production and use of scientific research. We argue that closing this gap will depend on understanding the dispositions and motivations of

187

N ew E ra of T heory , R esearch , E valuation

the inhabitants on each of these islands. Yet to date there is an embarrassing paucity of research on what these inhabitants are like and how their world operates. How can we get a handle on this dizzying assortment of communities and cultures? We begin by collapsing our archipelago into the following five distinct categories of aggregate target communities: basic researchers, applied researchers, intermediaries, policy doers, and policymakers. In Table 7.2, we define a select set of community inhabitants that range from those who build intellectual theory through those who bridge the research producer and user worlds to those who shape the overall course of public policy. These categories are not always as discrete in the real world as depicted here, but the distinctions are a shorthand way of capturing professional culture. As discussed in Chapter 6, we expect there is a correspondence between the professional orientation that emerges from one’s academic preparation and one’s later choice of a career path. Each aggregate target community has a similar core technology that shapes purpose, essential tasks, and primary audiences. We capture six domains of institutional culture that can be applied universally across communities. The domains encompass several beliefs and behaviors that tend to separate communities—work context, interactional preferences, epistemological frameworks, influence loops, focal interests, and salient stakeholders. In Figure 7.1, we attempt to provide a crude conceptual matrix of how a community culture is likely to influence how inhabitants think, act, and perceive the world. Our Table 7.2 The Professional Orientation of Aggregate Target Communities Aggregate Target Communities

Professional Orientation

Basic researchers

Focused on conducting research irrespective of whether or how that information is applied in the real world

Applied researchers

Focused on issues and questions with real-world applications

Intermediaries

Oriented toward trying to bring research and analysis to policymakers and/or those who try to stimulate appropriate research and analysis on current management or policy questions

Policy doers

Responsible for designing, managing, or applying the intent of policies

Policymakers

Responsible for making basic decisions about the direction of policies

Institutional Culture Domains

Professional Orientation Research Orientation Policy Orientation A B C D E Basic Applied Intermediaries Policy Doers Policymakers

I—Work context I-A II—Interactional preferences II-A III—Epistemological frameworks III-A IV—Influence loops IV-A V—Focal interests V-A VI—Salient stakeholders VI-A

I-B II-B III-B IV-B V-B VI-B

I-C II-C III-C IV-C V-C VI-C

I-D II-D III-D IV-D V-D VI-D

Figure 7.1 The Intersection of Institutional Culture and Professional Orientation

188

I-E II-E III-E IV-E V-E VI-E

W hy R esearch I s U nderutilized in P olicymaking

proposed matrix intersects the five expressions of professional culture in the aggregate target communities, and our six domains of institutional culture. Our intent is to achieve a delicate balance between acknowledging the complexity of the real world, and describing it in ways that are comprehensible. In that spirit, Figure 7.1 identifies 30 cells that represent interactions between discrete domains of institutional culture and distinguishable dimensions of professional culture. We do not attempt to describe each cell in detail. Such minutiae would surely constitute a narcotic that would numb the most studious of readers. Rather, our intent is to provide an exploratory cultural landscape that might serve as a guide for more systematic research and effective practice. We are proposing that researchers consider embarking on studies to gain an in-depth understanding of the social and political world of policy inhabitants in these aggregate target communities and other related communities as well. As an example, Karen studied the cells in Column E in the aggregate target community of policymakers, specifically focusing on the related community of state legislators. We also advise that researchers and intermediaries begin any effort to connect with a foreign community by conducting an environmental scan of what the inhabitants of each community are like across each of these domains by locating whatever literature about them is available. Better yet we propose travelling across the deep waters to foreign islands to personally acquaint oneself with its inhabitants and fully immerse oneself in its culture. We have erected our building blocks in the hope that it would contribute in a more concrete way to understanding why various communities in our archipelago have so much trouble getting along. With our theoretical building blocks in place, perhaps the nature of the amorphous cultural divide is a bit less puzzling. By considering the factors identified by community dissonance theory, we anticipate the cultural distinctions in beliefs and behaviors across our islands will become more accessible and understandable. Perhaps a couple examples can illustrate how the dimensions underpinning our understanding of culture play out in the real world. First, it is important to remember that ease of communication or collaboration across the islands in our archipelago depends on the distance, culturally speaking. The basic researchers, depicted in the cells in Column A, are most distant from the policymakers, depicted in the cells in Column E, and can expect the most cultural dissonance. Those attempting to connect across closer cells, such as in Column A and B, can expect less culture shock. As another example, let’s use the domain of epistemological frameworks to explore differences in how communities acquire knowledge and decide what information is valued. Those of us who have observed epistemological conflicts between basic researchers (cell IIIA) and policymakers (cell IIIE) have witnessed variants of this sad drama. The researcher cannot understand the lack of interest in new empirical findings that have stood up to rigorous scrutiny by peers. The researcher has achieved a breakthrough in identifying a program that increases effectiveness by ten percentage points over any other current program. The policymaker is likely to ask how much it will cost. The policymaker is astounded at the naiveté of the researcher, who is unable to answer this basic question. Why can’t this egghead see what is so obvious—a program that is too costly could incur untenable political costs if it forces budget cuts in other popular programs? Perhaps apriori understanding of epistemological differences could help avoid catastrophic misunderstandings and build better cross-community connections. Without this knowledge of how communities differ on epistemological frameworks and other dimensions of institutional culture, inhabitants of one community are not always sure how to effectively relate to or connect with inhabitants of another. There is always some element of uncertainty. Venturing into another community is a risky 189

N ew E ra of T heory , R esearch , E valuation

undertaking. It is easier to stay put and to fight the inevitable professional struggles that arise in a playing field that is known. Yet cultural conflicts can arise even when inhabitants play on familiar ground. Cultural Conflicts Within Communities Cultural conflict and misunderstanding can and do occur within communities, not just across them. What happens when inhabitants of a policy community confront misunderstandings, even conflict, with respect to their own institutional culture? Studies have been conducted regarding conflict across communities (Myers-Wall, 2000), but the only study to address within-community conflict, to our knowledge, is our study of state legislators (Bogenschneider, Corbett et al., 2019). We draw on that study here. The cultural conflicts within communities are legion, but one such conflict that can occur is over loyalty to one’s institution. Loyalty to the higher purposes of an institution can be admirable, yet institutional loyalty is “not a universal moral obligation to act justly” (Ward, 2008, p. 299) and can go awry (Heclo, 2008): However good the institution and its organizational manifesto might be, there is no morally simple way of deciding the limits of loyalty. Group loyalty can too easily become the highest form of morality. Leaders can have a knack for exploiting the feeling that everyone ought to be a “team player”. (p. 91) Loyalty holds different meanings in private and public institutions, such as government bodies, which are among the oldest institutions in the United States, notable for the duration of their basic governing processes (Mayhew, 2011). In the public sector, loyalty is said to be more important than in the private sector, because disloyalty in a political context can end up ruining a career (Cullen, 2015). Perceptions of the legislature differ dramatically depending on whether one is inside the institution looking out or on the outside looking in. Those on the inside have confidence that their colleagues are “honest” and “trustworthy” public servants who are “trying to improve the lives of ordinary people”. Our experienced key informants corroborated the views of sitting legislators that policymakers are well-meaning people committed to doing the right thing. Then why does only a quarter of the public have a “fair amount” or “great deal” of confidence in elected officials (Pew Research Center, 2018)? We asked our experienced key informants how they reconcile these negative public perceptions with the positive views held by those who know policymakers best. Responses included misleading media portrayals, public misperceptions, individual shortcomings, and institutional constraints. We focus here on institutional constraints. A Republican political operative with more than four decades of experience claimed that “the system is broken more than the people are”. Another Republican explained why this is so: By and large, [legislators] came for the right reason. They do realize once they get here that the right reason isn’t always the reason things happen. . . . It was for me, disillusioning, a little bit, to know that a lot of times the thing that happens isn’t probably the right thing, but it happened because it was doable or because some powerful interest wanted it to happen or didn’t want it to happen. . . . That’s why they call it sausage making, right? 190

W hy R esearch I s U nderutilized in P olicymaking

Key informants also explained that legislators, once elected, find themselves in a “polarized machine” controlled by leaders who demand loyalty and do not “give them any freedom of movement on anything important”. A key informant contended this consolidation of power in the leadership began in the 1970s as an unintended consequence of the post-Watergate reforms that took authority away from political parties at a time when money emerged as a dominant political force. The power shifted to political action committees and the leaders of legislatures, who inserted themselves in the money distribution process by recruiting candidates and directing cash to campaigns: The leaders are more involved in individual legislative races and, therefore, develop some loyalty, in particular, with . . . first termers who are out campaigning. And when you turn around, and there’s the majority leader, or the minority leader, or the speaker with you, it develops a bond of loyalty. After getting elected, legislators have a choice to make. As a former Democratic leader relayed: Some people go there with good intentions, and then they have to make the deal. “Am I going to stick out there and oppose my leadership . . . or am I going to get on the team?” Most of them . . . figure out they want to have influence and they got to get on the team. Beyond a desire for influence, there are numerous ways that institutional rewards can be used to enforce compliance to prevailing norms. The leadership has control over what happens within the building, such as the scheduling of bills for a vote, and also what happens outside the building, such as providing resources for the next campaign. Going it alone is a precarious route to take. One informant detailed the various forms of power that leaders possess in their political arsenal: If you’re not on the team . . . forget your committee assignments, forget your office space, where your office is  .  .  . how much staff you get.  .  .  . And who knows? . . . There’s always that fear of hey, you get a primary next time around. “We got a guy over here, he’s not part of the team, not helping us.” Boy, you get that message out, it makes it tough. Even policymakers who want to do the right thing find themselves boxed in by the constant need to raise money and by political pressure to be loyal to leadership. They described their frustration when leaders made them an instrument of the party by demanding partisan loyalty. Conflicts can arise within cultures as well as between them. These examples of cultural conflict suggest a caution for those who study and do public policy. Before engaging inhabitants of another community, it is important to conduct research and environmental scans to become familiar with the dominant professional and institutional cultures. Equally important is keeping in mind that communities are not clad in one cultural cloth. What is generally true across the culture of a community may not be true for every inhabitant.

Conclusion Ultimately, community dissonance theory inspires some political confidence and a message of hope. As politics and policymaking have become more cynical and partisan 191

N ew E ra of T heory , R esearch , E valuation

(Binder, 2003; Haidt & Hetherington, 2012; Peterson, 2018), scholars have despaired about whether engaging policymakers is worth their time or effort. One of the biggest challenges of our times is whether scholars can reverse their political pessimism about the prospects that science can inform public policy (Ward, 2008). This malaise of confidence in science exists, despite its historical record of accomplishment in all fields including evidence-based policymaking (Pinker, 2018). Such feelings of pessimism and despair can easily morph into suspicion and hostility. Political pessimism, according to Rorty (1999), is a predictable byproduct of the lack of communication between the research and policy communities. Community dissonance theory has the potential to restore political confidence, given its premise that inhabitants of each community are not destined to remain isolated on their own islands. Researchers can communicate across the deep waters to bring science to bear on policy decisions (Ward, 2008). The first step in opening dialogue and improving communication is for researchers to come to better understand the way of life on an array of policy islands. This is hard for outsiders to accomplish without a theoretical framework (complete with charts and matrices) that orients them to the pivotal touch points. Community dissonance theory enriches its theoretical predecessors by designating three key elements to understanding any policy community: the topography of the policymaking institution, the character of its inhabitants, and the dominant cultures that shape its functioning. First, the institutions of policymaking matter more than we ordinarily credit. Legislators, regardless of state or party, spoke of the policymaking institution much like they spoke of family, as a deeply valued tradition that they inherited from their predecessors and that they are committed to defending and maintaining for future generations. This respect for the institution inspires confidence in our elected leaders because it means that self-serving inclinations are outweighed by the honor of being part of a long-lasting historical and political tradition larger than themselves (Heclo, 2008). Yet policymakers also were explicit about how unwavering loyalty to the institution can sometimes make them an instrument for advancing partisan and ideological agendas (Rosenthal, 2009). Second, it is one thing to understand the deliberative and relationship-based environment in which policymakers operate and quite another to know with what character and motivations inhabitants of the policy community maneuver the policy process. Without personal experience working with policymakers or the benefit of the knowledge of others who have, one can easily be seduced into maligning all policymakers as a homogenous, faceless population. We in the academy tend to paint all policymakers with a broad brush even though, if the tables were turned, we would bristle at all researchers being characterized as eggheads, nerds, and ivory tower leftists. Representations of policymakers in our study report quite a positive story. With occasional exceptions, those who work closely with policymakers come to respect them as decent people of action (Albaek, 1995). Granted, policymakers have human frailties and do disappoint, but they often try to do the right thing under enormous pressures amidst competing agendas. Third, familiarity with the policy culture exposes impediments to cross-cultural communication, many that are behavioral and, as such, amenable to change. Many policymakers in our study were open to change. Many implicitly understood the structural impediments within their institutional setting that undermined good policymaking, such as political constraints and agenda-driven information. They appreciated that rigorous research could help them circumvent these impediments. For researchers who desire to communicate with policymakers, community dissonance theory suggests several cultural barriers that need to be better understood. The theory elaborates on professional culture 192

W hy R esearch I s U nderutilized in P olicymaking

and six domains of institutional culture—preferred decision-making processes, interactional preferences, favored epistemological frameworks, dominant influence loops, focal interests, and salient stakeholders. When the cultures of researchers and policymakers clash, friction is bound to occur. If these friction points remain obscure, mistrust and miscommunication are exaggerated. We review here several of these friction points because understanding them is a key to change. One friction point with pragmatic implications is different understandings across the two communities regarding how the decision-making process should work (Domain 1). The peril is that researchers are accustomed to looking for a straight-line influence of research in advancing the “scientifically” best policy outcome. In policy settings, research is used in a deliberative process that encompasses multiple decisions points at which negotiation occurs for the purpose of taking concrete action (Albaek, 1995). Could it be that those interested in the influence of research in the policymaking process have been asking the wrong question? Perhaps the question needs to be reframed from how to increase the use of research in policy decisions to how research can improve the policy process. Could it be that the use of research might be less for “better policy” and more as a “better guide” to the process itself (Prewitt et al., 2012, p. 19)? A second friction point is differences in how each community prefers to interact (Domain II). Policymakers’ interactions are often conceptualized as a market-based model of two people trading ideas, but what is surprising in our study and others is how deeply embedded political decision-making is in a social ecology of relationships (Tseng, 2012). To know how to enter this social ecology, science offers a guide. Studies show connections will come, not from minimal or superficial exposure to policymakers, but rather from one-on-one engagement with them (Chua, 2018). Scholars have long called for research dissemination that involves face-to-face presentation of research in forums like seminars that create a safe space for discussions among researchers and policymakers (Nutley et  al., 2007; Sabatier  & Jenkins-Smith, 1993; Smith, 1991; Weiss, 1999). Regrettably, few such policy forums have been created for communicating research evidence. Even fewer have created neutral, nonpartisan opportunities for policymakers to engage in open dialogue with each other that allow fostering relationships and finding common ground (for an exception, see K. Bogenschneider, 2020a). Now more than ever, the drift toward hyperpartisan policymaking cries out for such civilized conversation. The peril that researchers face is their widespread cynicism of the policy process and whether they could enter into dialogue with policymakers without disdain and suspicion. A third friction point is epistemological differences in how inhabitants assess the credibility of evidence (Domain III). For the prototypical researcher, the best route to truth is adherence to the canons of science. However, legislators live and operate in a radically different world. Truth is not determined by the technical judgments of anonymous peers, but by intuition and negotiation in a world dominated by human connections and trust. To assess the trustworthiness of people, including researchers, legislators prefer face-toface interactions. Research use may depend on ratcheting up such interpersonal connections. Open dialogue holds promise for exposing policymakers to the statistical judgment of science that can help guard against the self-deception of intuition. The peril is staying true to the science to avoid feeding into perceptions of a liberal bias in the academy. A fourth friction point is how policymakers influence the attitudes and beliefs of others (Domain IV). The most frequent way policymakers in our study used research is for argumentation—to persuade others to support one’s view or to anticipate and counter an opposing view (Bogenschneider, Day et al., 2019). Academics are quick to snub research being used for argumentation, failing to fully appreciate the extraordinary importance 193

N ew E ra of T heory , R esearch , E valuation

of persuasion in a governing process that privileges deliberation and consensus building over brute force (Albaek, 1995). Our understanding of research utilization and the effectiveness of research dissemination will remain incomplete without considering how research is used in argumentation and with what effect. The peril is holding modest expectations in those circumstances where persuasion is trumped by power politics. A fifth friction point arises with the topics that strike the members of each community as salient or worthy of their attention (Domain V). Researchers love brighter questions that might result in new breakthroughs that cut across time and space, whereas policymakers are moved by practical problems that need concrete solutions. The peril is that research will focus on questions that are cutting-edge in the research community, but isolated from the pressing problems confronting policymakers. A final friction point is recognition of a community’s salient stakeholders (Domain VI). In the academic community, researchers are most concerned with their academic peers (Ward, 2008). In the policy community, inhabitants respond to a mind-boggling array of stakeholders inside and outside the lawmaking body, some who provide bill ideas, some who provide political strategies, some who provide research and analysis, etc. Henig (2009) suggested scanning these salient stakeholders to strategically identify who has the ear of policymakers and might be tenable partners. This sage advice avoids the peril of trying to go it alone in charting new ground in foreign terrain. Research studies converge with what we heard from legislators that one such ally might be nonpartisan legislative service agencies (Hird, 2005; Peterson, 2018). Researchers could leverage these agencies’ relationships with policymakers, and they could capitalize on the academy’s access to rigorous research in pursuit of our mutual purpose—helping policymakers reach their long-term goal of benefiting the families and citizens they serve. Our theory has several limitations, one being that it tends to classify inhabitants as being fully ensconced in one community or the other. The real world, however, is not this pure. According to Henig, “Plenty of political actors appreciate science and wrestle with complexity. Plenty of researchers are eager to change the world and grow impatient with fellow researchers who split hairs and hedge bets” (2008, p. 222). The study of state legislators that informs our theory also has several limitations including the potential for self-report bias, though the results of the triangulation of data with experienced key informants is encouraging (Bogenschneider, Day et al., 2019). Also, the analysis was based on policymakers in two Midwestern states that varied widely on a number of relevant domains, yet the results still must be cautiously applied to other local or national policymakers and other policy actors such as agency or judicial officials. Community dissonance theory has pragmatic implications for academics working as idea traders (who design research studies that produce findings that are relevant on the island of policy), linguists (who translate research findings for policy consumption), or diplomats (who voyage across the deep waters to build relationships with policymakers on distant shores; Locock  & Boaz, 2004). At the end of the day, friction points are not cast in stone; they are not certainties. Better communication across our islands can resolve many issues, especially when trusting relationships are formed. With some effort, we can envision a day of better communication between members of the research and policy communities and more frequent use of research in policymaking. “There are worse things to hope for and worse ways of hoping” (Ward, 2008 p. 305). Some of the content of this article was adapted from a previous publication with permission. © 2019, National Council on Family Relations Bogenschneider, K., Corbett, T. J., & Parrott, E. (2019). Realizing the promise of research in policymaking: Theoretical guidance grounded in policymaker perspectives. Journal of Family Theory and Review, 11(1), 127–147. https://doi.org/10.1111/jftr.12310 194

8 EVALUATING EFFORTS TO COMMUNICATE RESEARCH TO POLICYMAKERS A Theory of Change in Action Karen Bogenschneider, Heidi Normandin, Esther Onaga, Sally Bowman, Shelley MacDermid Wadsworth, and Richard A. Settersten, Jr.

A systematic review of knowledge-brokering interventions found no empirical evidence that efforts to communicate research to policymakers have been effective in bringing about nonpolitical use of research. Still, experienced knowledge brokers remain enthusiastic about their effectiveness in bringing research to the policy world. Evaluating the effectiveness of research communication efforts faces conceptual complexities around determining the appropriate stakeholders, outcomes, methods, and time frames. In response, a theory of change is proposed that identifies seven key elements that are fundamental to increasing policymaker’s use of research. The chapter  presents protocols developed by the Family Impact Seminars to evaluate these seven key elements and provides preliminary evidence on each one. It is essential that the dissemination of research be viewed as a science unto itself. Jack Shonkoff (2004, p. 4) We should be on our guard not to overestimate science and scientific methods when it is a question of human problems. Albert Einstein (1949) We write this chapter because we have encountered a conundrum that may well comprise one of the most important intellectual challenges of our times: What will it take for research to influence policy and practice in ways that improve social outcomes (Tseng, 2019b)? This conundrum can be thought of as a three-legged stool. The first leg is policymakers’ growing interest in research evidence. The reason for this rise in receptivity was reported recently in a special issue on evidence-based social policy edited by Ron Haskins. Haskins was the staff director of the House Ways and Mean Human Resources Subcommittee for 14 years and was named by the National Journal one of the 100 most influential people in the federal government. In his words: More and more policymakers are convinced that they can use evidence to develop and provide political support for policies that work or are promising. . . . Few 195

N ew E ra of T heory , R esearch , E valuation

things are more important to policymakers, especially those in the majority party who initiate and are responsible for passing legislation, than citing reasons why their legislative actions are good for the country. In recent years, claiming that a policy is evidence-based fits this requirement. (Haskins, 2018, p. 10) A state legislator agreed with Haskin’s assessment. Former Indiana Senator Luke Kenley wrote recently that he thought researchers may not be familiar with how important research is to legislators and the ways it is used to guide the policy process: Research is a pretty important tool when thinking about what we’re trying to achieve to help people. . . . I find myself asking questions like the following: Are my thoughts actually accurate, or are they inaccurate? If we’re going to solve these problems, which changes do we need to make for things to be better? Do we have any data that show that these changes make a difference? I keep coming back to the interplay between what I want to achieve and whether the change I am promoting will work. So, that’s where the idea of research comes in. It is going back and making sure that this premise that you built your entire house of cards on is actually accurate. (2019, p. 154) So, the first leg of the conundrum is that a hunger for research exists, granted not for every policymaker, but for the most effective among them (Bogenschneider & Bogenschneider, 2020; Bogenschneider, Corbett et al., 2019; Bogenschneider, Day, & Bogenschneider, 2020). The second leg of the stool is creating the conditions to facilitate more widespread use of research by policymakers (Orr, 2018; Tseng, 2019a). Meeting this challenge, according to Adam Gamoran, the president of the William T. Grant Foundation, will involve strengthening the infrastructure to produce and use research at the federal, state, and local levels. Without an infrastructure to support relationships between researchers and policymakers, the chances of improving the use of even the most rigorous and relevant research evidence will be slim (Gamoran, 2018). Echoing this assessment, Wisconsin Senator Mark Miller called for the creation of “institutional structures” in a recent special issue on family policy: “Creating trust between legislators and researchers is an essential component for legislators to welcome scientific research into the policymaking process” (2019, p. 148). So, the first two legs of our metaphorical stool are that policymakers want research evidence, but there is little infrastructure for them to engage with researchers around it. The third leg in our conundrum is that no evidence exists that infrastructures for communicating research to policymakers are effective. This was the conclusion of a large-scale systematic review of knowledge-brokering efforts: “Our review failed to find any knowledge transfer techniques likely to bring about significant nonpolitical use [of research]” (Contandriopoulos et al., 2010, p. 467). This flies in the face of what we have experienced in the knowledge-brokering effort that we have been involved in for over 25  years—the state Family Impact Seminars (see Chapter  8). The response of legislators to the research that we have communicated and the relationships we have built are encouraging. For example, in 2019, an Oregon Family Impact Seminar on housing was likely one influence on the passage of the first statewide rent control law. Confidence in the research provided by the Wisconsin Family Impact Seminars was expressed by one legislative aide in a follow-up evaluation: “I always have confidence in the Family Impact 196

E valuating E fforts to C ommunicate R esearch

Seminar research. There is a high level of trust.” In the same evaluation, a legislator endorsed the Seminars as a valuable service to the legislature: “Really thrilled that this already exists. If it didn’t, I’d make it an initiative on my own” (N = 35 legislators and aides, response rate = 76 percent). How can it be that knowledge-brokering efforts can produce such an enthusiastic response from legislators, yet so few investigations and so little scientific evidence on their effectiveness (Boaz  & Nutley, 2019)? Three possible explanations are raised by policy scholars. First, could it be as Rossi (1987) suggests that evaluations of knowledgebrokering efforts fall under the “zinc law of evaluation”—effective communication efforts are out there, but they have not been evaluated? Or, alternatively, could it be as Tseng (2019a) posits that evaluators have pursued research utilization with a narrow technocratic focus and misunderstood the many ways it is used as part of a social and political process? Or, finally, could it be, as Guba (1981) contends, that evaluation methods have become a kind of orthodoxy that dictates “no longer a way but the way” (Guba, 1981, p. 90)? Is the evaluation enterprise open to circumstances when the context may need to drive the methods? Using our metaphor of the three-legged stool, this conundrum has three components: First, policymakers want research. Second, the infrastructure for connecting researchers and policymakers is limited. Third, evaluation methods for examining the effectiveness of research communication efforts are woefully inadequate. We will not be able to resolve this conundrum in a sole chapter of a single book. Yet by devoting attention to it, we hope to underscore that we think generating data on the communication of research evidence to policymakers is a legitimate and worthy pursuit. We believe, like Shonkoff, that the communication of research should be thought of “as a science unto itself” (Shonkoff, 2004, p. 4). This belief has compelled us to evaluate the efforts of the Family Impact Seminars to move research evidence from the hallowed walls of the academy to the harried halls of the statehouse. As Seminar directors, we approached this challenge by first taking stock of the complexities and challenges of conducting evaluations of research use in a policy context. For a decade, we engaged in many stimulating conversations about how to overcome some of these challenges, not perfectly, but thoughtfully. We present here the outcome of those conversations and our endless experimenting with novel methods for evaluating research use, broadly defined. The theory of change that we came up with proposes seven key elements regarding how research might be used by policymakers. We present each element of the theory by drawing upon the scholarly literature and by considering how each might be influenced by a knowledge-brokering effort, such as the one that we have been involved in. To transform our theory into action, we provide evaluation results on each element from our experience convening Family Impact Seminars in our respective states and evaluating when they were effective and when they were not. In our view, evaluating socially significant work like communicating research to policymakers should be a high priority and not “handcuffed” because it lacks perfect approaches, measures, and methods (Harvard Family Research Project, 2007, p.  14). This chapter is a first step to advance that vision. Of course, these ideas are based primarily on one knowledge-brokering effort, so the lessons we have learned are offered with humility. Our hope is to inspire ongoing conversations and to generate hypotheses that can be tested going forward. We do not have definitive answers, but our experience has acquainted us with the key questions that other evaluators might need to resolve. We begin by illustrating the complexity of this challenge with some real-life stories of how knowledge brokers have communicated research to policymakers. 197

N ew E ra of T heory , R esearch , E valuation

How Research Is Used by Policymakers: Three Vignettes We present three vignettes: the first based on the experience of a nonpartisan organization providing technical assistance to members of the U.S. Congress, a second based on Wisconsin’s 18th Family Impact Seminar for communicating research to state policymakers, and a third based on Oregon’s 1st Family Impact Seminar. These examples bring to light many of the dizzying challenges that confront and confound evaluators. Even when policymakers appear to use research, the process remains largely a mystery. The U.S. Congress Office of Technology Assessment The rise and fall of the Office of Technology Assessment (OTA), described in a 1996 book by Bruce Bimber, is particularly relevant for two reasons. First, 75 percent of its staff were researchers. Second, it is one of the only nonpartisan legislative service agencies that Congress voted to enact and then, 23 years later, decided to eliminate. OTA’s mission was to provide information and expert analysis to Congress on technology policy, but also on such issues as defense, energy, the environment, health care, and land use. OTA produced about 20 to 30 reports annually, with each study taking about two years to complete. Its studies were designed, conducted, and reported in order to be relevant to and read by busy policymakers. Albert Gore Jr., jokingly nicknamed “Senator Science”, is alleged to be the only known member to have been spotted carrying a fulllength OTA report on the Senate floor. Most policymakers became familiar with OTA studies by reading the one-page summaries or by being briefed by OTA staff. Threats to the agency’s existence in 1975 prompted efforts to trace the use of the agency’s research and analysis in congressional decisions. The most direct and obvious ways to document the impact of the agency—linking an OTA study to legislation or laws—came up empty-handed. In interviews with policymakers, congressional staff, and agency personnel, not one vote cast by a member of Congress could be traced to an OTA study. In a content analysis of the congressional record (which includes oral statements, revisions to these statements, and written insertions), the influence of the agency is nearly absent. During the 1980s and 1990s, references to OTA studies occurred less than once a month among the 535 members of Congress. After striking out on these three visible and verifiable fronts—legislation passed, votes cast, and comments amassed—it would seem safe to conclude that OTA had little, if any, influence on policy in its 23-year history. Reviewing this questionable track record, a San Diego newspaper published an article on OTA titled “The Office of WHAT?” (Bimber, 1996, p. 38). Upon further examination, however, Bimber (1996) concluded that it was misleading if not mistaken to dismiss OTA’s influence. In a broad analysis of anecdotal evidence on several bills, Bimber (1996) concluded that OTA’s influence on the policy process was both rhetorical and analytical. On several occasions, members of Congress such as Rep. Pete Stark and Sen. John Tower used OTA studies to reinforce their rhetoric on issues ranging from pharmaceutical prices to energy policy. Observers do not believe that these elected officials came to their positions as a result of OTA studies, but that they had instead selectively mined these studies for the purpose of persuading others to their point of view. In other instances, OTA studies performed more of an analytic function. In 1974, James Schlesinger, the Secretary of Defense, requested funding for Counterforce, the nuclear weapons targeting strategy. Central to this decision were estimates of the casualties that 198

E valuating E fforts to C ommunicate R esearch

might result from a nuclear exchange between strategic sites rather than population centers. Several members of Congress questioned the estimates provided by the Department of Defense and asked OTA for estimates. The OTA study criticized the department’s numbers, which provided a rationale for the chair of the Senate Foreign Affairs Committee to request revised estimates. So, in this instance, OTA helped Congress better grasp whether a problem existed and what the policy response might be. Many major bills (e.g., the Brady Bill and the appropriation for the Strategic Defense Initiative) and dozens of minor bills actually bear OTA’s impact, even though the influence of OTA was not immediately apparent. These examples are instructive for four reasons related to when, where, what, and how evaluators should look for impact. First, these examples illustrate that evaluators should think about when to look for the influence of research on policy decisions. For a nonpartisan purveyor of information like OTA, its influence seemed to occur earlier in the policy process when issues were being framed and the policy agenda was being set. For example, the influence of OTA on a bill that required chemical markings in the marketing of commercial explosives appeared in the framing of policy alternatives as “OTA Option 1”, “OTA Option 2”, and so forth. Second, the OTA examples indicate the importance of considering where to look. OTA staff did not always communicate directly with elected officials, but often provided information indirectly to aides. These indirect lines of communication would not be reflected in voting records, bill sponsorship, or the Congressional Record. Third, the examples also illustrate what to look for. Votes cast by policymakers may not be the best indicator of the effectiveness of this agency, which was not charged with providing voting recommendations, but rather with producing a menu of options for policymakers to consider. Finally, the examples illustrate how fine-tuned indicators need to be if they are to detect influences that may be indirect or modest in impact rather than monumental. No reasonable person expects policy decisions to be based solely on research evidence or expert recommendations. Rather, research is one of many inputs in a multiply determined process in which no single factor prevails over all the rest. Further complicating matters, once scientific evidence is incorporated into a policymaker’s knowledge base, it becomes difficult to remember its origins (Tseng, 2008). Wisconsin’s 18th Family Impact Seminar Wisconsin’s 18th Seminar was planned at a time when legislators were hearing a groundswell of concern from families about how rising health-care costs were affecting access to and affordability of health care. In particular, small employers were facing sticker shock at the cost of providing health care for their employees. One policy option that was under consideration, employer purchasing pools, was identified by Family Impact Seminar advisors (i.e., ten legislators and one Governor’s Office staffer) as the issue most likely to be debated and acted on in that session. Wisconsin already had legislation on the books, although it had not been implemented, to establish employer purchasing pools for small businesses (i.e., grouping small employers into a larger group to spread risk and offer a larger choice of benefits with more stable rates). At a Seminar planning meeting, which included several experts on the issue, one state agency staffer revealed that they had commissioned a study on the feasibility of setting up employer purchasing pools in the state. They agreed to release their findings at the Seminar and in the accompanying briefing report. The Seminar attracted 110 participants, including 28 legislators and 35 legislative aides. An additional five legislators attended a luncheon discussion with the speakers, 199

N ew E ra of T heory , R esearch , E valuation

bringing the total to 33. During the day’s activities, 59 legislative offices were represented. Another 15 ordered copies of the briefing report or an audiotape of the Seminar, which meant this Seminar reached 74 legislators and their staff—over half of the offices in the Wisconsin legislature. Participants gave high marks to the quality of the research in the end-of-session evaluation. On a scale of 1 (poor) to 5 (excellent), participants rated the objectivity of the research as 4.4, its usefulness to their current role as 4.4, and its relevance to their needs as 4.3 (N = 88, response rate = 80 percent). In paired sampled t-tests, participants also reported knowing significantly more after the Seminar than before about the factors responsible for rising health-care costs and the features of successful small-employer purchasing pools (both significant at p < .001). As part of the day’s activities, the governor’s top health-care advisor requested a special one-hour meeting with two Seminar speakers and two high-ranking state agency officials. In this meeting, he relayed that he needed four pieces of information in five days to put before the governor for possible inclusion in his state budget proposal. After an hour, he asked that the meeting be extended, but 15 legislators were waiting to spend 1¼ hours interacting with the speakers. After extending the discussion with legislators to 1¾ hours, the chair of the Assembly Health Committee asked if it could continue even longer, because they so seldom have the opportunity to talk with experts, especially those from out-of-state without a vested interest in what happens in Wisconsin. However, 20 high-ranking state agency officials were waiting to meet with the speakers. The state agency discussion session was organized around the four questions that the governor’s advisor had raised earlier. After the scheduled 90 minutes, the head of the state health agency asked what time the speakers were flying out and whether the meeting could be extended another 30 minutes. At the Seminar, preeminent researchers had reviewed the available evidence that voluntary, unsubsidized insurance pools have little or no effect on health insurance costs or coverage rates. Consistent with this research, no further legislation was passed and the legislation on the books was not implemented. This vignette brought home to us the issues of the complexity of identifying outcomes for evaluating efforts to communicate research to policymakers. Can an outcome of deciding not to take legislative action be counted as a success? This vignette revealed other outcomes that could be indicators of effectiveness, such as policymakers’ receptivity to research evidence and to opportunities to discuss it with key policy actors. Oregon’s First Family Impact Seminar Oregon’s first Family Impact Seminar on “Helping Families Help Themselves” was strategically conducted early in the 2001 legislative session. One featured speaker was John Karl Scholz of the University of Wisconsin-Madison, who provided the latest research on the consequences of several policy options including providing tax credits to the working poor. Scholz has published extensively on Earned Income Tax Credits (EITCs), directed the University of Wisconsin-Madison Institute for Research on Poverty, and served as Assistant Deputy Secretary for Tax Analysis at the U.S. Department of Treasury. The Seminar, held in conjunction with an Oregon House Revenue Committee, was attended by six legislators (in a body of 90 members) and 64 members of the public. Participants heard a 60-minute presentation and received a briefing report with the latest research and relevant policy implications. The session was broadcast live over the Internet and on closed circuit television throughout the capitol building. Also, in a follow-up discussion session, Scholz met with top personnel from the Legislative Revenue Office, 200

E valuating E fforts to C ommunicate R esearch

legislative staff, and representatives of key advocacy groups to examine the effectiveness of tax credits and other policy options for the working poor. In end-of-session evaluations, participants’ knowledge of the EITC had significantly increased. The Republican chair of the House Revenue Committee and a Democratic representative ended up introducing HB 2716, a bill to establish refundable childcare tax credits for low-income working families. Before the session adjourned, the bill was passed by a Republican-controlled legislature and signed into law by a Democratic governor. Under the law, families who paid for childcare received 40 percent of their costs as a refundable childcare credit. Previously, the credit could only be taken as a deduction on a family’s state income taxes, which limited its reach to only those earning enough income to pay taxes. The chair of the House Revenue Committee credited the Family Impact Seminars for making all the difference in crafting a bill that could be passed. In his words, the Seminar “raised the level of the conversation and helped us keep tax credits in the forefront”. This Seminar raised for us the methodological challenges of conducting evaluations of efforts to communicate research in policy contexts. With so few legislators attending, analysis of the effects of the Seminar across the entire legislature is unlikely to be statistically significant. Are there alternative methods that could tap into the substantive significance of the Seminar? Is a randomized controlled trial feasible or are there other methodologies that could produce outcomes convincing to researchers, intermediaries, policymakers, or other stakeholders?

The Complexities of Evaluating Efforts to Communicate Research to Policymakers The most important program evaluation questions are often the most difficult to answer (Shonkoff, 2004; Scott, 2020). At the top of that list is the difficulty of evaluating efforts to build evidence-based public policy. According to veteran evaluator Larry Orr (2018), one of the major barriers of applying program evaluation to public policy is persuading policymakers to act on evaluation results. Scientists have long had mixed responses to the feasibility of evaluating the communication of research to policymakers for their use. Some react with pessimism, contending that it is futile to even think about causality because when research is invoked to justify policy decisions, it is tainted by politics in inscrutable ways that defy rational explanation (Brodkyn, cited in Weiss, 1995). Others are similarly cynical, but argue the impediments are mainly methodological (Segal, 1983). For example, some reject the study of questions that do not readily conform to randomized experiments, considering them unevaluable and unworthy of the evaluator’s time (Brown, 1995). Others criticize the overvaluing of randomized experiments, which they believe undermines the value of alternative methodologies that may be more appropriate for certain research questions in certain contexts (Guba, 1981; McCall & Green, 2004; Shonkoff, 2004). We borrow from common principles, models, and methods of evaluation (Coffman, 2007; Hawkins, Clyde, Doty, & Avellar, 2020; Rossi, 1987). However, we deliberately resist dominant paradigms that evaluation research methods are static and relatively stable across contexts (Collins & Overton, 2006; Rossi, 1987; Scott, 2020). In a rigorous review of knowledge-brokering efforts, the main conclusion was that it is context that actually “dictates the realm of the possible” for efforts to communicate research to policymakers (Contandriopoulos et al., 2010). Yet evaluators tend to disregard the ways that research is used in the policy context, which differs from other settings in several important ways (Contandriopoulos et al., 2010). We discuss three of these differences here. 201

N ew E ra of T heory , R esearch , E valuation

First, in evidence-based medicine, effectiveness depends on the actions taken by autonomous individuals, whereas in evidence-based policy no single individual on his or her own has enough autonomy to produce a collective decision in a system characterized by high levels of interdependence (Contandriopoulos et  al., 2010). Second, unique to policy settings is that individuals use information, not only for their personal benefit, but also for shaping their rhetoric and informing their arguments to persuade colleagues to their point of view. Third, for evaluating interventions that focus on individuals, robust quantitative methods can be used to detect effectiveness. However, in a policy setting, when an intervention depends on collective knowledge, system change cannot be so easily specified. As the vignettes demonstrate, evaluating efforts to communicate research to policymakers brings contextual complexities regarding stakeholders, outcomes, methods, and time frames (Corbett, Danziger, & Werner, 2005). We review here empirical and experiential evidence regarding each of these complexities.

Key Concepts 8.1.  The Complexities of Evaluating Efforts to Communicate Research to Policymakers • • • •

Stakeholder complexity Outcome complexity Methodological complexity Time complexity

Stakeholder Complexity At the heart of stakeholder complexity are differences regarding what constitutes evidence. How evidence is used by policymakers was aptly reflected by Tommy Thompson, former Wisconsin governor and former U.S. secretary of Health and Human Services. In a talk at the Heritage Foundation, he described Wisconsin’s groundbreaking experience transforming welfare from an income support program to a system based on work and selfsufficiency with accompanying reforms in childcare, child support, and work preparation. As evidence of the success of his initial reforms, Thompson explained that caseloads dropped by 60 percent statewide and 32 percent in the largest urban center, Milwaukee. According to his estimates, families were earning $5,000 more per year through work than they had previously received on welfare; moreover, the state’s payments to welfare recipients dropped from $46 million per month in 1987 to $21 million per month in 1997. Drawing upon Kids Count data from 1986 to 1993, the time period when the welfare reforms were launched, Thompson claimed that child poverty in Wisconsin dropped 13  percent and child abuse declined by 15 percent. He also read a letter from a welfare recipient who had kept in touch with him after he helped her land a job several years earlier: Governor Thompson, I want you to know where I’m at. Me and my daughter have left our beloved Wisconsin. We are now living in Phoenix, Arizona. I just got promoted and I had to leave Wisconsin for this promotion. Now, I’m running the office in Phoenix, Arizona. And I want you to know that the next time you come to Phoenix, I want to take you out for lunch, and I can afford to pay for it. (Thompson & Bennett, 1997, p. 9) 202

E valuating E fforts to C ommunicate R esearch

According to Thompson, this letter brought tears to his eyes. For him, it was a convincing form of evidence: A letter like this made everything that I had been doing to reform welfare realistic. It put a face on welfare reform. And I knew from this letter and from her example that we were indeed going in the right direction. (Thompson & Bennett, 1997, p. 9) This example shows how stakeholders look to different standards of evidence. Accordingly, evaluations may need to be designed to collect different kinds of evidence to suit different stakeholders (Patton, 2001; Weiss, 1972). Policymakers like Governor Thompson want to know if a program works and whether its benefits are worth taxpayer costs. As our studies and others have shown, policymakers find stories, like the vignettes of the Oregon and Wisconsin Seminars, to be convincing evidence for their colleagues and constituents (Bogenschneider et al., 2020; Mayne et al., 2018; Nutley et al., 2007; Oliver & Cairney, 2019). Researchers look to different kinds of evidence. They are interested in using statistically based methods to indicate the effectiveness of a program in achieving its intended outcomes, and to determine if the program itself was the cause of improvements and not some other factor. Because researchers are interested in rigor, they prize the “gold standard” of evaluation—experimental methods that allow them to make causal claims and rule out alternative explanations (Hawkins et al., 2020; Orr, 2018; Rossi, 1987; Rossi, Lipsey, & Henry, 2019). In contrast, program administrators are looking for better ways to serve their clients, so they want to know if, how, and under what conditions the program is working and whether it could be replicated in their setting. Thus, program administrators are less impressed with experimental studies because it is harder to determine what elements of the program worked and how each contributed to the program’s success (Deaton & Cartwright, 2018; Granger et al., 2013; Knox, Hill, & Berlin, 2018; Weiss, 1972, 1997). To cut through this complexity, evaluators first need to get a firm fix on the purpose of the evaluation. Who are the stakeholders, what questions do they want answered, and which types of evidence do they value? Outcome Complexity The critical outcome for measuring the impact of research communication efforts seems obvious: whether a policy is passed. But this may not always be the most appropriate indicator. As illustrated by the Wisconsin vignette, available research evidence on employer purchasing pools for small businesses supported defeating, not passing, the policy (Coffman, 2007). Indeed, marking the success of a research communication effort only by the passage of a policy is based on several faulty assumptions regarding policymakers, policy responses, communication approaches, and inputs into policy decisions. We comment on each in turn. First, even if research is effectively communicated, not all policymakers value research, let alone agree that policy decisions should be research-based. The incorporation of research into a bill is not a reasonable outcome when some policymakers who will vote on it do not value research and are prone to discount it, no matter its rigor or relevance (see Chapter 3; Bogenschneider et al., 2020). 203

N ew E ra of T heory , R esearch , E valuation

Second, even if policymakers value research evidence, they can have legitimate disagreements about the optimal policy response, which often revolves around whether government is perceived as causing problems or resolving them (Bogenschneider et al., 2020). Karen worked with a Governor’s Commission on Children and Families, composed of a broad mix of policymakers and the public from different walks of life, many appointed by a conservative governor. The members of the commission agreed that an important policy direction for the state would be to support children in single-parent families. Karen reviewed with the commission 29 research-based policies described in a Family Impact Seminar briefing report on the topic (Bogenschneider, Kaplan, & Morgan, 1993). How many of these options did the commission decide to endorse? To Karen’s great surprise, not one. The research-based alternatives were largely governmental solutions to a problem that commission members believed was primarily private in scope and cultural in nature. Third, policy enactment is also a problematic measure of success when knowledge brokers use an education approach that communicates the consequences of alternative policy options, including the status quo (see Chapter  11). In keeping with this approach, not passing an option with little research support would be a researchinformed outcome. Finally, the enactment of a policy may be an unrealistic outcome when one considers the multitude of inputs into a policy decision. As illustrated in the vignette of the Office of Technology Assessment (Bimber, 1996), research is only one of many policy levers that include ideology, values, political interests, and other influences beyond the reach of the knowledge broker (Weiss, 1999). Thus, if a policy fails to pass, is it fair to conclude that the communication of research was ineffective? Conversely, if a researchbased policy is enacted, is it reasonable for the knowledge broker to take all the credit, some credit, or any credit at all? System-wide impacts, such as the enactment of laws by an entire legislative body, are a grandiose outcome that is long-term in nature, difficult to measure, and hard to interpret even when it does occur (Contandriopoulos et al., 2010; Weiss, 1972). It seems prudent for knowledge brokers to also consider more immediate outcomes at the individual level that fall short of ultimate legislative approval, but might occur en route to a policy decision. One set of immediate outcomes that likely would precede other outcomes deals with the research itself, such as whether policymakers have an opportunity to access it, whether they understand it, and whether it is perceived as being objective and relevant to the issue at hand. If it passes this first set of screens, another set of intermediate outcomes that might precede passage of a law is whether policymakers use the research to inform their position, draft new bills, or evaluate existing policy proposals. In the policy context where research use is a collective process, another obvious set of outcomes would be whether the individual policymaker communicates research to others by incorporating it into speeches or presentations, sharing it with colleagues, or responding to constituent requests. When information that is objective and relevant is shared with others, is it possible that this raises their stock among colleagues and affects their relationships with them? Also, one wonders whether the use of research in these ways affects their attitudes about research and researchers. The basic point is that knowledge brokers can measure how the research is received by an individual policymaker, how that individual communicates it into the collective decision-making process, and what effect its communication has on individual policymakers and their colleagues. 204

E valuating E fforts to C ommunicate R esearch

Methodological Complexity In the research and evaluation communities, the basic methodological ground rules are clear. In some education and policy circles, “the term ‘evidence’ has become synonymous” with randomized controlled trials (Tseng & Coburn, 2019, p. 353). In the policy community, however, these highly regarded methodologies do not always map neatly onto the question at hand. Yet complaints about the shortcomings of dominant evaluation methods have been roundly rejected by mainstream evaluators. For example, Rossi (1987) claims they deserve very serious consideration, but finds the revisionist arguments unconvincing. The Family Impact Seminar vignettes raised the shortcomings of three common methodological practices when they are applied to evaluations of efforts to communicate research to policymakers. One common evaluation practice is calculating the average effects of a program across the entire population. At the Oregon Family Impact Seminar, only six legislators out of the 90 members of the legislature heard the research evidence that tax credits are effective because of their work-incentive features, low administrative costs, and targeting to the bottom quartile of wage earners with children (HalpernMeekin, Edin, Tach, & Sykes, 2015; Hotz & Scholz, 2000). Even if the Seminar had a large effect on these six legislators, the average effect across the legislature was likely be nonsignificant. However, the vignette reveals that the program had a “whopper” effect on one state legislator, the chair of the House Revenue Committee. He introduced legislation on a childcare tax credit that was eventually passed by the legislature and signed into law by the governor. This shortcoming of dominant evaluation paradigms to detect whopper effects has been pointed out by evaluators. Randomized trials are better able to provide unbiased estimates when the intervention produces homogeneous effects in the target population; however, when the effects are more heterogeneous, as they are in this vignette, the evaluation may obscure more than it reveals (Fraser & Davies, 2019). A second shortcoming is that dominant evaluation paradigms prize statistical significance, so a communication effort that failed to produce a significant effect on policy across the entire sample would be considered a dismal failure. From a policy paradigm that prizes substantive significance, however, passing a research-informed law would be considered a resounding success. Success would be measured, not by probability levels alone, but by its impact on people. In 2012, Oregon’s childcare tax credit returned $22  million to over 25,000 taxpayers (State of Oregon Legislative Revenue Office, 2015). What knowledge broker would not consider it a success if he or she could document playing some part in a research-based policy that benefits thousands of families in a state? The Family Impact Seminars also revealed a third methodological shortcoming of common evaluation practices. In an ideal world, evaluators use methods that allow making causal claims (Rossi et al., 2019), but the feasibility of using randomized controlled trials may depend on the context. To our knowledge, only two randomized controlled trials have been used in legislative settings, one in Congress and one in a state legislature. One study examined the effectiveness of the Research Policy Consortium, a nonpartisan knowledge-brokering effort that connected researchers and Congressional offices around child and family policies. Greater use of research evidence was found in the offices randomly assigned to the intervention group than the control group (Crowley et al., 2021). Another study randomized pending veteran’s bills, so that state legislators received a face-to-face research briefing from a trusted legislative analyst on the “intervention” bills, but not the “control” bills. The evaluation then was able to demonstrate 205

N ew E ra of T heory , R esearch , E valuation

that legislator’s cosponsorship and votes on these bills were “caused” by the intervention (Zelizer, 2018). Yet it is hard to imagine how a randomized trial could be used to evaluate effectiveness in other policy education contexts. In the Seminars, university researchers are communicating evidence on a fluid and evolving issue to an entire body of legislators that includes a number of key policy actors. The Wisconsin vignette illustrates the absurdity of randomly assigning these key actors to a “seminar” or “no seminar” condition. The obvious challenge is that there is only one head of the Department of Health and Human Services, one top health advisor to the governor, and one chair of the Assembly Health Committee. An additional complication is that there are only a couple “go-to” or “cue-giving” legislators on health care in each legislature. These “go-to” legislators who develop specialized expertise on an issue provide advice to their colleagues on what position to take or how to vote (Bogenschneider  & Bogenschneider, 2020; Krehbiel, 1992; Matthews & Stimson, 1975). Conceivably, a “go-to” legislator who was assigned to the treatment condition could easily contaminate the study design by giving advice to colleagues assigned to the “no treatment” condition. In the knowledge-broker world, the ethical constraints of random assignment are also apparent. For example, what are the political ramifications if a university or other public knowledge broker invites only some policymakers to a forum featuring the latest research from top scientists on a high-profile issue? Instead of random assignment of individuals, another well-accepted methodology for determining causality would be randomly assigning states to a “treatment” or “no treatment” condition (Fraser & Davies, 2019). The upside of having such a counterfactual, when feasible, is the many benefits that such a comparison brings to the rigor of an evaluation and the confidence in any claims about impact. The downside is that each state is an N of 1, which yields a maximum population of 49 if every single state could be considered a legitimate comparison. They obviously cannot. For example, in the Wisconsin vignette, it strains credulity to imagine another state demographically and politically similar to Wisconsin that is looking at employer purchasing pool for small businesses and that is considering implementing a law already on the books. Finding a legitimate comparison state is challenging, when they vary in so many obvious and not-so-obvious ways that may affect policymakers’ use of research. Some potential differences include: • • • • • •

State characteristics (e.g., size, region, population demographics) The lawmaking body (e.g., number of members, part-time or full-time status, limited or unlimited terms of office, length of legislative sessions that vary from one month every two years to several months annually) Political history and configuration (e.g., Blue State or Red State or Purple State, partisan control of the executive and legislative branches of government) Availability of and access to information inside and outside the legislature (e.g., level of support from nonpartisan service agencies, the extent of knowledge-brokering efforts in the state, the saturation of advocacy groups) The quality of the communication effort (e.g., legislative involvement in design and execution, marketing and timing, relevance of the research, skill of speakers) The salience of an issue (e.g., a crisis or serendipitous event that creates interest in an issue, availability of funding to seed a response, local activism behind the issue)

Given the many constraints of conducting evaluations in a political context, is a randomized experiment truly the “gold standard” or is this seemingly simple truth no more 206

E valuating E fforts to C ommunicate R esearch

than “fool’s gold”, as some have suggested (Gueron, 2007)? Is it reasonable to search for causal impact when the Seminars themselves make no claims of causality? Even if research of the highest quality is communicated to the right policymakers at the right time, knowledge brokers would not claim that a single input into a multiply influenced policy decision would “cause” a given political outcome. Are there other methods that might be more appropriate such as time series analysis or regression discontinuity designs that would allow evaluators to compare the program against itself over time (Corbett, Dimas et al., 2005)? Time Complexity Timing is a “fickle” factor in the policy culture. Policy issues rise and fall on the political agenda. Policymakers are sworn in, and they leave office when their terms end or they are voted out. Evaluation is challenged by the slow way that research is thought to seep into policymakers’ frames of reference, conventionally referred to as enlightenment (Weiss, 1999, p. 195). [Research] brings new information and new perspectives into the policy system. It often shows that old shibboleths are misguided and it punctures old myths. Over time, the ideas from research seep into people’s consciousness and alter the way that issues are framed and alternatives designed. . . . The slow trickle of enlightenment is hard to see and harder still to identify as the product of social science. But the cumulative effect may be a major recasting of the policy agenda and the embrace of a very different order of policy. Examples abound of just how long it can take for this redefinition, recasting, or reframing to occur. Based on 22  years as a nonpartisan fiscal analyst, Jere Bauer Jr. explained that “certainly over time, a low dose, low dose, low dose” of good information eventually has an impact. One researcher explained that it took 15 years for affordability in drug therapy to become part of the policy agenda. Kristin Anderson Moore of Child Trends described the gradual recognition in research of the importance of fathers in children’s lives, and how this slowly became part of policy debate. In her words, it does happen, but it is rare to see something change the world directly and immediately: I think the thing that was really a surprise to me earlier in my career was presenting testimony on teen childbearing . . . and actually getting into the newspaper and having that affect policy and then realizing, “Oh, that doesn’t happen every year.” Many years go by before that happens again. . . . You have to just accept the fact that those kinds of mountain-top moments where you feel like you’ve really gotten something across clearly, and they heard it, and it had an impact— those moments are pretty rare for any of us. Some issues lend themselves to quicker action than others, but with controversial issues one could spend a lifetime and not see the desired changes. An academic with 30 years of experience working with policymakers drew on the example of U.S. health care to illustrate the excruciatingly slow pace of policy change: If you were someone who really thought that this country ought to have publicly supported health care for everyone, you probably would have gotten excited 207

N ew E ra of T heory , R esearch , E valuation

when Franklin Delano Roosevelt included that in his charge to the committee that developed the Social Security Act and thought, “Ah, we’re going to have that now.” And you would have been excited when Harry Truman raised the idea again, and when Lyndon Johnson eventually pushed through Medicaid and Medicare, and when Bill Clinton revisited the topic during his presidency. . . . And, of course, your career would have long been over and you would have now been dead and not seen that come to pass. And yet, slowly we have seen an increase in public coverage of children and families and other individuals, piece by piece by piece. So, a lot of it depends on how you define what the issue is and how you define progress. If your solution was a single-payer system, publicly funded and administered, you would feel probably betrayed and very discouraged having spent the last 65 years working on this. Yet if you could be satisfied with inexorable progress, a little bit here and there, sometimes a step back, sometimes two steps forward, you might look back on your life and say “Hey, we left it somewhat better off than where we started . . . there are seldom opportunities or moments when you get very clear transitions or changes in course that say, “Ah, this is it! We’ve reached the Promised Land!” Timing raises two primary questions for evaluators. First, what criteria can evaluators use to determine the best time for assessing the influence of research on policymakers and policymaking (Hawkins et al., 2020)? Basically, when should one look for whether research has arrived in the “Promised Land”? In the OTA example, the influence of research seemed to occur earlier in the process when issues were being framed and the policy agenda was being set. Tracing a process that can be this indirect and diffuse takes patience (Farrell & Coburn, 2016). For example, the Early Head Start program for children, aged 1 to 3 was launched, in part, by the accumulation of decades of research on the importance of the early years to human development (Zigler, 1998). If the influence of these studies had been evaluated too soon, the impact would have been said to be minimal. Second, how can the evaluator detect small effects that accumulate over time (Hawkins et  al., 2020)? When looking for blockbuster effects, even imperfect measures can detect them. However, fine-tuned indicators like those eventually used to evaluate the OTA are needed to reveal the more subtle enlightenment functions of research such as projecting ideas into political discourse and framing the terms of policy debate (Weiss, 1999). Tracking modest effects that add up to something over time can be extraordinarily difficult, particularly in a field where standard measures with acceptable psychometric properties are not widely available (Hawkins et al., 2020).

Baby Steps for Evaluating Research Communication Efforts The Family Impact Seminars has a long history in communicating high-quality, nonpartisan research to policymakers first in Congress and now in state legislatures. Since the state-focused program started in 1993, the Family Impact Seminars have wrestled with the challenges of program evaluation, in general, and the complexities of stakeholders, outcomes, methodology, and timing, in particular. Over time, we developed and began testing a theory of change to guide our evaluation efforts. To contextualize the theoretical framework that follows, we expand on the methods and purposes of the Seminars described in Chapter  9. We also detail the preliminary protocols and procedures for evaluating each element of our theory of change. 208

E valuating E fforts to C ommunicate R esearch

The Family Impact Seminar Goals and Execution Family Impact Seminars are an ongoing series of presentations, discussion sessions, and briefing reports that communicate rigorous and relevant research to state policymakers on timely topics they identify. The Seminars are attended by state legislators, legislative aides, governor’s office staff, nonpartisan legislative service analysts, state agency representatives, justices of the Wisconsin Supreme Court, and university/extension faculty. To foster consensus building, lobbyists and the media typically are not invited. The Seminars are designed (a) to build greater respect for and use of research in state policy decisions, (b) to provide neutral, nonpartisan opportunities for legislators to engage in open dialogue for fostering relationships and finding common ground, and (c) to encourage policymakers to examine policies and programs through the lens of family impact.

Key Concepts 8.2.  Goals of the Family Impact Seminars 1. To build greater respect for and use of research in state policy decisions 2. To provide neutral, nonpartisan opportunities for legislators to engage in open dialogue for fostering relationships and finding common ground 3. To encourage policymakers to examine policies and programs through the lens of family impact

The Seminars are currently being conducted in 15 states. Since 1993, a couple dozen states and the District of Columbia have conducted over 234 Seminars on a wide range of topics identified by state policymakers, including controversial issues (e.g., gambling, Medicaid, tax policy) and issues affecting vulnerable children, youth, and families that are often neglected (children of incarcerated parents, homelessness, immigrant families, racial and ethnic infant mortality). To date, 51 Seminars have been conducted on parenting and youth issues; 50 on education and early childhood education; 43 on health care; 41 on family poverty and welfare reform; 25 on jobs, the economy, and workforce development; and 24 on corrections and juvenile crime. The Family Impact Seminars are a manualized process that includes face-to-face training, technical assistance, and implementation manuals to help ensure the program is implemented with fidelity to the theories that guide its design and the core principles that govern its operation. Seminar organizers have written about the macro mindset (the big picture thinking) and the micro mindset (operational practices on which successful execution depends; Bogenschneider, 2014, 2020a; Bogenschneider, Shager, Little, & Eddy, 2017; California Research Bureau, n.d.; Mayer & Hutchins, 1998; McClintock, 1999; Melton, 1995; Tomayko et al., 2019; Tran, Hines, & Bell, 2019; Wilcox, Weisz, & Miller, 2005; Wise & Brandon, 2014). Typically, the Seminars are a low-budget operation with the leadership consisting of minimal staff. In some states, the Seminars are led by a faculty team (whose time is an inkind contribution) and graduate student assistants (Tomayko et al., 2019). Other states have a faculty director, who makes an inkind contribution of about 25 percent time and works with a grant-funded coordinator, graduate student, or consultant, each of whom devotes part-time effort over a couple months (Bogenschneider et al., 2017). 209

N ew E ra of T heory , R esearch , E valuation

The traditional format of the two-hour Seminar consists of two or three 20- to 30-minute presentations given by a panel of premier researchers, program directors, and/or policy analysts. Typically, the presentations are followed by discussion sessions led by a Seminar facilitator in a neutral, off-the-record setting that allows policymakers to discuss issues and seek common ground where it exists. For each Seminar, a background briefing report summarizes policy-relevant research along with family and policy implications in a succinct, accessible format. Some states are intentional in offering Seminars in advance of legislative sessions and arranging for speakers to give testimony to committees that have authority over the topic, which allows legislators to engage with researchers as bills are being developed and debated (Tomayko et al., 2019). In some states, the first Seminar is followed by a second “by-invitation” session that engages state agency leaders, research and social service organization directors, and program officers of philanthropic foundations (Tomayko et al., 2019). Some states sponsor a special discussion session for agency officials the day of the seminar and other states sponsor a follow-up discussion six to eight weeks after the Seminar for legislators to continue the conversation with local researchers.

Key Concepts 8.3. Core Principles of the Family Impact Seminars • Easy access to research on timely topics identified by legislators • Varied delivery formats, including presentations, discussion sessions, briefing reports, and web posting of Seminar presentations and reports • Use of the family impact lens • Objective, nonpartisan approach • Opportunities for discussion in a neutral, nonpartisan, off-the-record setting • Partnership with high-ranking state agency officials, nonpartisan legislative service agency analysts, university/extension faculty, and others with expertise on the Seminar topic

Family Impact Seminar Evaluation Protocols and Procedures Common evaluation protocols and procedures have been developed and used across sites. First, we decided that any summative evaluation method, such as randomized experiments, was not feasible. Our preferred methods are based on five evaluation procedures: careful record keeping, end-of-session evaluations, follow-up phone evaluations, testimonials, and tracking of Seminar issues as they progress through the policy process. We target many stakeholders for the evaluation data, including the policymakers we try to attract, funders we strive to convince to underwrite the Seminars (Hawkins et al., 2020), university faculty and legislative partners we seek as collaborators (K. Bogenschneider, 2020a), and university administrators to whom we are accountable (K. Bogenschneider, 2020b). For policymakers, stories about how research is used in policymaking have proven to be believable, trusted, and influential evidence (Bogenschneider et  al., 2020; Mayne et al., 2018; Nutley et al., 2007; Oliver & Cairney, 2019). For university faculty and administrators, objective and statistically based evaluation methods are most convincing (Patton, 2001; Weiss, 1997). Because the Seminars have multiple purposes, we collect several kinds of evaluation data. 210

E valuating E fforts to C ommunicate R esearch

The Rationale for a Theory of Change to Guide Evaluation of a Policy Education Effort When we first began, our goal was to evaluate the Family Impact Seminars in a way that moved beyond simply bean counting—the number of presentations planned, discussions held, and publications produced. Among the key reasons why assessing program effectiveness stumps evaluators is that the assumptions regarding the nature of the problem and the programs are poorly articulated (Hawkins et  al., 2020; Rossi, 1987; Weiss, 1999). In addition, David Olds (2009), developer of the evidence-based Nurse-Family Partnership program, warned about the inherent tension that evaluators must resolve between the “problem” and the “program”: Are we here to promote a particular program or are we here to solve a problem? It’s a fundamental question; in fact, it’s an issue that I had to struggle with when I first began this work. And I eventually came to realize that I was not here to promote a program; I was here to solve a problem. . . . It’s also important for us to realize that . . . you have theories, you have ideas of how this is going to work. Consistent with this line of thinking, we developed a theory of change to guide the evaluation of our response to a complex societal problem—the failure of research to fully realize its promise in contributing to public policymaking (see Anderson, n.d., 2004; Weiss, 1995). Our theory of change articulates how research evidence flows through the policy process and the ways a policy education effort like the Seminars might influence this process. Specifically, we hypothesize seven elements that occur when policymakers use research and we detail which indicators need to be assessed to systematically document how a knowledge-brokering effort might affect each of these elements. We doubt that we have identified every relevant element and we realize that our measurement is preliminary. Nonetheless, we found it informative to take a mysterious process like research utilization and attempt to break it down into several discrete early and intermediate outcomes that must occur if the long-term outcome is to be realized—policymakers using research and a family perspective in performing their jobs (Anderson, 2005). As a point of clarification, a theory of change differs from a logic model, which is a more tactical, program-level explanation of program inputs, outputs, and activities for producing outcomes. In contrast, a theory of change is a more “strategic picture of the multiple interventions required to produce the early and intermediate outcomes that are the preconditions of reaching an ultimate goal” (Anderson, 2005, pp. 12, 19). It is a macro perspective or holistic vision of how and why big things happen. A theory of change could encompass several logic models that explain on a more micro programmatic level how to produce each outcome.

The Family Policy Education Theory of Change The family policy education theory of change, as we have dubbed it, has seven key elements that can help knowledge intermediaries assess whether the time and resources they expend on communicating research to policymakers will achieve its anticipated early, intermediate, and long-term outcomes. This theory of change draws upon several theoretical traditions: constructive theories of learning, Kingdon’s open policy windows, social learning, and community dissonance theories (e.g.,  see Bogenschneider, Corbett et al., 2019; Kingdon, 2011; Walter et al., 2005). 211

N ew E ra of T heory , R esearch , E valuation

Engage Policymakers & Experts to Identify Timely Family Issues

Provide Research

Early Outcomes

Attract the Policy Community to Dialogue in a Nonpartisan Educational Setting

Gains Knowledge

Builds Relationships Intermediate Outcomes

Changes Attitudes

Policymakers Use Research and the Family Perspective in Their Jobs

Long-Term Outcome

Figure 8.1 Family Policy Education Theory of Change: How Policymakers Use Research and the Family Perspective in their Jobs

Our theory of change is depicted in Figure 8.1. The early outcomes entail engaging policymakers and experts to identify timely family issues and then providing rigorous, relevant research on these issues in ways that attract the policy community to dialogue in a nonpartisan, educational setting. Intermediate outcomes include knowledge gained, attitudes changed, and relationships built. That is, providing research on timely issues in a nonpartisan setting starts a process whereby policymakers gain knowledge on both research and the family impact of policies that, in turn, changes their attitudes about the value of research and a family perspective in policymaking. These attitude changes subsequently increase their propensity to build relationships with researchers and their colleagues in multiple ways that are useful in their jobs. This process is surely multidirectional with several possible causal or even circular paths. Conceivably, this process may begin with policymakers building relationships with researchers; in the context of these trusted relationships, policymakers may seek out research-based information and, in so doing, change their attitudes about the value of research and a family perspective in policymaking. It also could be that the process begins when policymakers gain knowledge of research and its family implications from Seminar activities, which prompts them to build relationships with researchers and colleagues and, in turn, contributes to changes in attitudes about the value of research and 212

E valuating E fforts to C ommunicate R esearch

a family perspective in policymaking. The long-term outcome is that policymakers use research in their job in several ways, such as developing policy positions, sharing it with their colleagues and constituents, incorporating it into their speeches, or using it to draft legislation and enact laws to support families. The family policy education theory of change has the potential to inform other policy education efforts in multiple ways, three of which are described here. First, the theory introduces some early and intermediary outcomes in the research utilization process that are sometimes overlooked. Because it can take years for research to seep into policymakers’ consciousness or for a policy to be enacted into law (Weiss, 1999), policy educators face the challenge of maintaining support from funders and other stakeholders with little or no interim evidence that their efforts are having an impact. Second, we believe this theory of change may have applicability beyond the Seminars because it is not framed from the perspective of “program testing” but rather from that of “variable testing” (Hawkins et al., 2020; Weiss, 1972). The variables we test and the early, intermediate, and long-term outcomes we identify could apply to any reasonably similar effort to communicate research to policymakers. Third, this theory can help policy educators identify which theoretical elements are being measured and which are not. For example, we constructed a grid that included, on one side, the elements of our theory of change and, on the other side, the evaluation protocols we were using. The gaps between the two sides of this grid revealed which elements were not being evaluated, prompting us to develop new questions and procedures to ensure testing of every theoretical link. In this section, we present the underlying rationale and the empirical research for the seven elements of the family policy education theory of change. For each element, we explain the indicators, how they are operationalized, what evaluation protocols are in place (see Table 8.1), and the preliminary evidence that exists. Figure 8.2 is an expanded schematic of the theory of change, which includes each indicator that we assess in the evaluation. Engages Policymakers and Experts to Identify Timely Family Issues The processes of identifying timely Seminar topics vary slightly across states, but are always a high priority. For the organizers of the Oregon Family Impact Seminars, selecting and refining the topic is the most time-consuming aspect of the entire planning process (Tomayko et al., 2019). Why is timing so important? The reality of policymakers’ lives is that they are faced with literally hundreds of issues that they must consider, assess, and vote on each session (Bogenschneider, Corbett et al., 2019; Bogenschneider, Day, & Parrott, 2019). Not all bills that are introduced have momentum behind them, nor will there be enough time to permit each to be examined with equal diligence. To address this political reality, we draw on Kingdon’s (2011) theory of open policy windows to identify the issues for which research may have the best chance of influencing a policy outcome. When policy windows are open, the conditions are right for social change on an issue and policymakers are willing to invest their time, energy, and political capital because their efforts may pay off. The opening of a policy window usually occurs with the convergence of three separate streams: problems are recognized, policy solutions are available, and the political climate supports the change. No single stream, on its own, is likely to position an item on the decision-making agenda. For example, a problem without a solution, a solution without a compelling problem, and problems or solutions that are politically unacceptable 213

N ew E ra of T heory , R esearch , E valuation

Table 8.1 Evaluation Methods for Assessing Each Element of the Family Policy Education Theory of Change Elements of the Theory of Change

Program evaluation method

Engage policymakers and experts to identify timely family issues

Examination of whether program organizers take the following steps to identify timely issues for policy education: 1. contacting policymakers to identify the problem, 2. consulting with experts to generate researchbased policy solutions, and 3. recontacting policymakers to assess the political and economic viability of the policy solutions. Questions in the end-of-session evaluations assess whether the research presented at the program is relevant, useful, objective, practical, innovative, economically feasible, and politically feasible. Careful record keeping of attendance at the program and discussion sessions tracks how many policymakers participate and whether the information reaches influential lawmakers such as committee chairs, bill sponsors, legislative opinion leaders on the issue, etc. End-ofsession evaluations assess the usefulness of the discussion sessions. Retrospective pre- and post-test questions in the end-of-session evaluation assess participant perceptions of their understanding of the research presented at the program and how it affects families. Attribution questions in phone interviews six to eight weeks after the program examine changes in attitudes about the value of research and the importance of a family perspective in policymaking. Phone interviews six to eight weeks after the program examine changes in attitudes about how approachable researchers are, their ability to identify others interested in the issue, and the contacts made with researchers and colleagues on the program topic. Testimonials indicate the value of the dialogue created with researchers and among policymakers of opposing parties. Follow-up phone interviews six to eight weeks after the program assess research utilization and participant ratings of the usefulness of several information sources, including one’s own program. Tracking policy issues assesses if and how the information was used. Available research studies are used to indicate whether laws that are enacted are effective, efficient, and supportive of families.

Provide research Attract policymakers to dialogue in a nonpartisan setting Gains knowledge Changes attitudes Builds relationships

Policymakers use research in their jobs

quickly result in the closing of a policy window. Typically, more visible policy participants like the governor or legislators set the legislative agenda, whereas hidden participants like academics, legislative service agency personnel, and state agency staffers identify policy alternatives. Political feasibility is determined, to a large extent, by policymakers (Kingdon, 2011). The organizers of the Wisconsin Family Impact Seminars use a three-step planning process that closely follows Kingdon’s framework for linking problems with policies and politics (see Figure 8.3). First, we use topics from seminar evaluations and an environmental scan to interview a bipartisan group of legislative and gubernatorial advisors who identify the single most compelling problem with bipartisan and bicameral interest. Typically, this committee includes an equal number of Republicans and Democrats representing a range of ideological perspectives, along with a representative of the governor’s office. Second, to identify research-based policy options, we form a planning committee composed of those with relevant topical expertise, including university faculty, agency officials, legislative service agency analysts, and other experts relevant to the topic. For example, the expert planning committee for a jobs Seminar included the secretary and 214

E valuating E fforts to C ommunicate R esearch

Engage Policymakers and Experts to Identify Family Issues

Relevant Useful Objective Practical Innovative Economically feasible Politically feasible

Early Outcomes

Provide Research on Family Issues

Attracts the Policy Community to Dialogue in a Nonpartisan Educational Setting

On research On family impact of policies and programs

Problems recognized Policy solutions are research-based Politics favorable

Gains Knowledge

Policymakers Policy administrators Legislative aides Legislative service agency analysts Governor's staff

Builds Relationships

With researchers With colleagues

Intermediate Outcomes On the value of research On a family perspective in policymaking

Changes Attitudes

Policymakers Use Research and the Family Perspective in Their Jobs

Evaluate legislation

Incorporate into speeches

Share with colleagues

Draft legislation

Long-Term Outcome

Develop policy positions

Enact laws that support families

Figure 8.2 Detailed Map of Family Policy Education Theory of Change: Measuring Policymakers’ Use of Research and the Family Perspective in their Jobs

chief economist of the Wisconsin Department of Revenue, the president of the Wisconsin Technical College System, the chief economist and an executive leader of the Wisconsin Department of Workforce Development, a division administrator of Family Economic Security in the Wisconsin Department of Children and Families, the president of the Wisconsin Technology Council, the director of the Wisconsin Legislative Fiscal Bureau, and several heads of university institutes (Bogenschneider et al., 2017). The third step is taking the top policy options identified by the expert planning committee and running them by the Seminar’s legislative and gubernatorial advisors. These advisors rank the political and economic feasibility of each option to determine those currently most relevant in the state that will be featured at the Seminar. One way to evaluate whether a research communication effort is implemented with integrity is conducting a simple check to see if each of the three planning steps is taken. This ensures that a recognized policy problem is addressed with research-based responses consistent with the prevailing political and economic climate in the state. The steps are detailed in Figure 8.3. Legislators commend the Seminars for this method of identifying topics for which research might be useful to them. In an open-ended interview, a Wisconsin Democrat 215

N ew E ra of T heory , R esearch , E valuation

Seminar participants identify problems, which are ranked by state legislators and gubernatorial staff.

Problems 1

3

Politics

State legislators and gubernatorial staff rank politically viable policy alternatives.

2

Policies Academics, agency officials, and policy analysts identify policy alternatives.

Figure 8.3 Planning Timely Family Impact Seminar Topics: Identifying Open Policy Windows Source: Adapted from Bogenschneider, K. (2014). Family Policy Matters: How Policymaking Affects Families and What Professionals Can Do (3nd ed., p. 267). Mahwah, NJ: Erlbaum.

spontaneously relayed that she “really respects” the planning process, a view echoed by a Wisconsin Republican: As I look at how we’ve worked with the Family Impact Seminars, it’s been . . . bipartisan [with] both parties participating and identifying what the issues are we see coming up this session. If you were just to think about it from the outside, I don’t think that you would have the inside feel for what we would identify as areas that we need information on now, because we’ll be making decisions on that particular issue in the course of this legislative session. In Massachusetts, a former Senate president appreciated the timeliness of the Seminar topics: The topics covered at the seminars have always been very timely and have provided my colleagues and me with high-quality information in order to make solid decisions about issues and bills we are debating. On several occasions, the seminars were ahead of the curve and sparked new ideas. A Wisconsin Republican legislator described how central timing is to legislators’ use of research in their decisions: Prescription drugs was an excellent . . . seminar that we had because we dealt with it shortly afterward and we heard from folks saying, “This is what you should do and this is what you should not do”, and we tailored some of our legislation in this area. Provides Research In the highly politicized world of policymaking, studies show that research evidence can and often does contribute to policymaking (Gormley, 2011; Grisso & Steinberg, 2005; 216

E valuating E fforts to C ommunicate R esearch

Henig, 2008; Hird, 2005; Huston, 2008). At the Seminars, we work hard to identify high-quality and nonpartisan research that is relevant to the Seminar topic and that has implications for policy responses that are politically and economically feasible. During the planning, we also look at the topic through the family impact lens because families appear to be a shared value that has the potential to rise above politics and build common ground (Bogenschneider et al., 2020). Guided by the family impact lens, we consider how families may contribute to the Seminar topic, how they are affected by it, and whether a family approach might result in more effective and efficient policies (Bogenschneider et al., 2012). Making research available responds to a frequent complaint from legislators that nonpartisan research is hard to find (Bogenschneider & Bogenschneider, 2020). As voiced by a Wisconsin Republican: The reason that I go to the Family Impact Seminars is they bring in speakers that we normally never get to hear from in the State Capitol. The State Capitol is loaded with politics. It is not loaded with scientific research or people who have studied these issues from across the country. So, what happens is we hear from a number of well-respected folks who know the issue inside and out, and we get information that we never ever get an opportunity to hear otherwise. However, we have learned that not just any research will do, so we turned to the empirical literature on the characteristics of research used by policymakers. In Table 10.1 (see Chapter  10), we summarize evidence on the variables associated with policymakers’ use of research. Of the 14 variables that emerged from previous studies, we selected three that appeared especially relevant to the Seminars. First, science corroborated what policymakers told us about the importance of nonpartisan information. In addition, the empirical evidence suggested the importance of research that is applicable to questions of interest to policymakers, and research that reflects knowledge of the policy process. The value of nonpartisan information emerged from several empirical studies (Bogenschneider & Bogenschneider, 2020; Bogenschneider et al., 2020; Goodvin & Lee, 2017; Mayne et al., 2018; Peterson, 2018), which legislators typically describe in their Seminar evaluations as being “objective”. The applicability to questions of interest to policymakers also emerged from empirical studies (Goodvin  & Lee, 2017; Mayne et  al., 2018; Tseng et  al., 2018; Zigler, 1998), which legislators typically describe in their Seminar evaluations as “relevance”. Being knowledgeable about the policy process was also frequently mentioned in empirical studies (Bogenschneider  & Bogenschneider, 2020; Mayne et al., 2018; Orton et al., 2011), which in the context of the Seminars signals the “usefulness” of the information to policymakers in their jobs. A common end-of-session evaluation protocol used across sites assesses a Seminar’s overall rating along with ratings of its objectivity, relevance, and usefulness. We calculated the mean participant ratings along these dimensions for 43 Family Impact Seminars held in Indiana, Michigan, Oregon, and Wisconsin (88 percent response rate). On a scale of 1 (poor) to 5 (excellent), the mean objectivity rating across 43 Seminars in these four states was 4.35, which is a surprising level of consensus given that the Seminars are attended by some of a state’s most liberal and conservative policymakers. The value policymakers place on objectivity is expressed by a veteran Republican legislator, Sen. Luther Olsen: That’s the best part of the Seminars, they’re nonpartisan, they are research based. . . . It really makes no difference if you’re Republican or Democrat. 217

N ew E ra of T heory , R esearch , E valuation

The information is good, and it is very, very well balanced. That’s very important in being a credible source. You have to, in my estimation, [give us] just the facts ma’am, just the facts, as they used to say in Dragnet. The valuing of objectivity transcends party lines. The late Democratic Representative Tamara Grigsby described the uniqueness of Seminar research in the policy culture: It’s objective. It’s neutral. It’s not an advocacy approach. It’s “Here’s the information. Here’s the data. Here are some things that have worked some places. Here are some things that haven’t. You make the decision.” That really leaves people feeling very comfortable that they’re not in a partisan environment. And that no one’s lobbying them, which is naturally what everyone . . . expects everyone is trying to do in this building. These high mean ratings of objectivity did not come at the expense of participant ratings of relevance (M = 4.33) or usefulness (M = 4.12) across 43 Seminars held in four states. The relevance of a Wisconsin Seminar on early brain development was obvious to a legislator in an anonymous follow-up evaluation: Poverty, education, and health issues are all linked through early brain development. Poverty affects brain development which, in turn, affects health which, in turn, affects school which, in turn, affects poverty. I want to work toward legislation that connects those dots. The usefulness of the research presented at a Seminar reflects how effective the planning process has been in tracking the policy process to determine timely questions that policymakers will be debating and deciding. The usefulness of a Seminar on jobs was expressed by a legislative aide in a follow-up evaluation: The really useful discussion at this last seminar was the mismatch between the skills that are available in the workforce and the skills that are needed in advanced manufacturing and other technical areas, and how to address that. That has turned out to be one of the major economic development themes of this year. We used the information from this seminar to support a bill that we introduced on advanced manufacturing. Providing research that is relevant and at the right level of detail is a challenge for Seminar organizers. Sometimes legislators find Seminar information is too technical (e.g.,  “this was over my head”, “too much started from an assumed knowledge base that most don’t have”), and other times not technical enough (e.g.,  “not very much meat in today’s seminar; just an overview, not very deep”). Other legislators report that the Seminar information was not tailored enough to the state and did not offer enough policy responses and new ideas. In response to how the Seminars may have missed the mark, we also developed evaluation questions that assessed whether policy options were discussed at the Seminar, how practical and innovative they were, and to what extent they were politically and economically feasible. For example, a follow-up evaluation of a Wisconsin Seminar on early brain science was completed by 18 legislators (69  percent response rate). Almost all legislators (94 percent) said a policy option (defined as a program, law, bill, or legislative 218

E valuating E fforts to C ommunicate R esearch

action) was presented at the Seminar. On a scale of 1 (poor) to 5 (excellent), legislators rated the policy options as practical (4.3), innovative (4.2), economically feasible (3.9), and politically feasible (3.2). Attracts the Policy Community to Dialogue in a Nonpartisan Educational Setting The consensus of scholars is that the most effective method of disseminating research to policymakers is face-to-face forums that create a safe space for discussion between researchers and policymakers, which allows for the discussion of next steps with each other (Nutley et al., 2007; Sabatier & Jenkins-Smith, 1993; Smith, 1991; Walter et al., 2005; Weiss, 1999). Consistent with constructivist theories of learning, meaning is created through interaction, especially in the relationship-based culture of policymaking (Bogenschneider et al., 2019). In the words of a former Wisconsin Democratic caucus chair: Physical presentations by other human beings are what grab you. Otherwise, there’s the strong likelihood that one of the legislators will write “File” on the report that just came across their desk, as . . . I did and we all do. And it’ll go in a file somewhere. But when there’s a presentation, and . . . if it’s a good speaker— entertaining, funny, or challenging . . . that’s what gets us hooked. We’re human beings. We’re “people-people.” We like to interact with folks. That’s one of the reasons we’re in politics. So, you have that dynamic of physically hearing and seeing someone, which is very important, and being able to listen to what they’re saying. You also have the report in front of you and a lot of times they can explain the report that we couldn’t dig through otherwise. The opportunity to discuss research evidence is what makes the Seminars effective and engaging, according to Massachusetts Republican Joseph McKenna: If I get a policy brief delivered to my desk with no discussion attached, I may just put it aside and not get to it. But, if it’s delivered with the discussion from the professor or researcher as to the context, you . . . are able to go more in depth and it just helps the process. (Tran et al., 2019, p. 14) A fundamental first step in evaluating knowledge-brokering efforts is careful record keeping—tracking requests for reports and information, website usage, and attendance for each Seminar and across all Seminars. To illustrate, we examine a specific seminar, a 2014 Wisconsin Seminar on early brain science. It attracted 98 participants, including 26 legislators (about half Republican and half Democratic), 20 legislative aides, Wisconsin’s First Lady, ten staff of the nonpartisan legislative service agencies, 22 executive agency staff, six members of the court system, and five university or extension faculty. Looking across the first 34 Seminars in Wisconsin, 80 of the 132 legislators serving in the 2015– 16 biennium had participated in Seminar activities, and 13 of those not participating had sent an aide. In other words, the Seminars engaged 70 percent of legislative offices in the state. A similar level of policymaker penetration occurred in a less experienced Seminar site in an eastern state. After a decade of operation, the Massachusetts Family Impact Seminar had reached 55 percent of the members of the Senate and the House of Representatives (Tran et al., 2019). 219

N ew E ra of T heory , R esearch , E valuation

Other evaluation strategies include tracking whether the Seminar information reaches the right people in the right places and in the right way. First, getting information into the hands of the right people, begins with understanding one of the biggest mistakes people outside of the legislature make. Mary Fairchild, senior fellow at the National Conference of State Legislatures, explains that oftentimes knowledge brokers gravitate toward developing “a relationship with a friendly legislator, somebody who is their friend. . . . And that person may not have an ounce of influence.” So, we track not only how many policymakers come to the Seminar but also who comes, such as committee chairs, bill sponsors, and legislative opinion leaders on the issue. At a Wisconsin Seminar on early childhood education and care, a luncheon discussion included many of the key players on the issue: the Assembly Democratic caucus chair, the chairs of the Education Committee in both the Senate and the Assembly, and the chair and ranking minority member of the Assembly Children and Families Committee. Second, based on our research and experience, getting research into the hands of policymakers depends on knowing the place in the legislative process where the experience and expertise on issues lie—the committee. Virtually every bill goes through a committee and it is at the committee level where bills get the most scrutiny and research is most likely to be used (Bogenschneider & Bogenschneider, 2020.) The organizers of the Oregon Family Impact Seminars attribute their remarkable legislative success to the presentations given by their speakers before relevant committees. Seminar staff consult with the speakers to nimbly adapt and compress their presentations, which are strategically scheduled during the legislative days prior to session when lawmakers are planning their scope of work (personal communication, R. A. Settersten Jr., July 17, 2020). Consequently, another metric in Seminar evaluations is tracking whether the research is presented at the relevant committee or whether it gets into the hands of relevant committee chairs, ranking minority members, and committee members. Third, another way to evaluate a policy effort is its ability to provide information in ways that add value in a polarized political environment. According to Smith (1991), the most valuable service a professional could provide policymakers may be providing a neutral space for dialogue outside the contentious, interest-group-dominated environment in which policymaking typically takes place. Legislators have few opportunities for discussion on a bipartisan basis, so providing an off-the record space for dialogue and negotiation can be conducive to seeking common ground and reaching compromise where it exists. Accordingly, we use quantitative and qualitative measures to track the usefulness of opportunities to discuss research and next steps with researchers and colleagues. The value of these Seminar discussions was described by Sen. Mark Miller, longtime co-chair of Wisconsin’s powerful budget committee: The other thing that I really, really like is the fact that legislators from different political persuasions who are sincerely interested in dealing with that particular policy issue get a chance to understand what the concerns of the other members are. We oftentimes don’t understand people on the other side of the aisle because we don’t get a chance to talk to them just on policy. That’s one of the real benefits of the Family Impact Seminars. Gains Knowledge Our theory of change proposes that policymakers must understand the research that is presented if they are going to act on it. To assess policymakers’ understanding of Seminar 220

E valuating E fforts to C ommunicate R esearch

research on the issue and how it affects families, we use the retrospective pre- and posttest methodology first proposed by Campbell and Stanley (Lamb, 2005). At the end of the session, participants report their understanding of the same items before and after the Seminar. The retrospective pre- and post-test methodology is not considered a perfect evaluation tool because of a potential response bias when participants want to appear like a “good subject”. However, in previous studies, little difference has been reported between the results of evaluations using retrospective pre- and post-tests and traditional pre- and post-tests (Lamb, 2005; Pratt, McGuigan, & Katzev, 2000). Retrospective pre- and posttests can be useful in situations in which it is difficult to use more traditional methods, four of which seem relevant to the policy context. First, this method is appropriate when a less complex and time-consuming evaluation is desirable. Busy policymakers, who are hesitant to participate in studies, might not be willing to commit the time to complete the two evaluation protocols required in a traditional pre- and post-test. Second, this retrospective method is recommended when you are asking people to rate their knowledge of items that they do not grasp but think they do (Laura Hill, personal communication, June 3, 2009). Thus, pre-test results can be inaccurate when participants initially think they know more than they do, an overconfidence that has been shown to contribute to cognitive bias (Kahneman, 2011). Third, one goal of the evaluation of policy education is exactly what the retrospective pre- and post-test evaluation does—to explain whether participants feel they have gained knowledge (Hill & Betz, 2005; Pratt et al., 2000). For some stakeholders, such as funders and legislators who advise the Seminars, knowledge change stated by legislators may be one of the most persuasive forms of evidence. Finally, a retrospective pre- and posttest can help avoid “response-shift” bias by clarifying misconceptions in understanding before participants complete evaluation protocols. After the concepts are presented at the Seminar, participants assess their new level of understanding and then, with the response set in mind, reflectively assess their understanding before the event (Lamb, 2005). Figure  8.4 displays the results for six items assessing perceived knowledge change from a Michigan Family Impact Seminar on Medicaid. Paired sample t-tests examined whether the pre- and post-test means differed significantly. Overall, 61 participants (57 percent response rate) reported significant (p < .01) increases in knowledge on every item, including knowledge of both (a) the research presented at the Seminar and (b) the family implications of the issue. We have some circumstantial evidence of the validity of this retrospective methodology. The results are not always significant, and they are sometimes significant for some participants (e.g., legislators) but not others (e.g., legislative staff). Also consistent with our expectations, legislators typically report lower pre-test knowledge on the Seminar topic than do more specialized state agency officials. Legislators have explained what these knowledge gains mean to them. For example, one Republican legislator reported that Seminar information “raises the level of understanding across different issues in the Capitol. So I use it to question what we are doing, to say ‘You know this is a good thing.’ ” Evidence also suggests that research plays an important role in destroying myth (Bogenschneider, Day, & Parrott, 2019; Flinchbaugh, 1988; Heckman, 1990). For example, a Republican legislator commented after a Seminar on parenting: “I wasn’t sure that government had a role in parenting, but now I am sure we can no longer stick our head in the sand.” Similarly, a legislative aide wrote on an end-of-session evaluation that “[the seminar] put many of our presumptions in a new light”. 221

N ew E ra of T heory , R esearch , E valuation

The key factors driving the increase in state Medicaid expenditures (t = 5.164)***

3.37 3.88

What states can do to control Medicaid costs (t = 2.793)**

3.66 4.20

The role that family caregivers play in providing care to frail elderly and persons with disabilities (t = 4.676)***

2.59 3.75

What states can do to support family caregivers (t = 6.416)***

3.34 4.28

Current picture of the impact of obesity (t = 3.565)***

3.02 3.42

Community-level interventions to prevent obesity (t = 3.849)***

3.51 4.08 0

Low

0.5

1

1.5

2

Pre-Test Before the Seminar

2.5

3

3.5

4

4.5

High

Post-Test After the Seminar

Figure 8.4 How Participants Rated Their Understanding of Research and the Family Perspective **p < .01 ***p < .001 Note: Based on responses from 35 participants (57 percent response rate) attending a 2005 Michigan Family Impact Seminar, “Supporting Children and Families While Controlling Medicaid Costs”

Changes Attitudes The utilization of research may depend upon the policymaker’s attitude toward research and, for our purposes, toward families. Empirical evidence, albeit limited, finds that policymakers who most frequently use social science research have the most positive attitudes about its value to policymakers and to the policy process (Bogenschneider et al., 2013; Caplan, 1979; Webber, 1986). In recent studies of state legislators’ attitudes toward youth and families, Republicans and Democrats alike widely endorsed them as a population deserving of support. Legislators also reported that research can sometimes play a more important role for youth and family policy than for other issues (Bogenschneider et al., 2020). Given the importance of attitudes to policymakers’ research use, we track whether the Seminars change attitudes toward research or families. To assess attitudes about the value of research, participants are asked to attribute changes in attitudes to the Family Impact Seminars (see this approach in Riley, Meinhardt, Nelson, Salisbury, & Winnett, 1991). These items are written in a way that asks respondents to attribute causality (e.g., “Because of the Family Impact Seminars, I am more likely to consider how pending legislation might affect families”). Because policymakers are called by staff associated with the seminars, the results could contain biases related to social desirability—the tendency of respondents to paint themselves in a favorable light or to respond in ways that they think will please the interviewer. We address these concerns, in part, by using a three-point response scale of “not at all”, “a little”, or “quite a bit”. The response “a little” gives participants a socially desirable way to indicate that the Seminars have benefited them, but not very much. Thus, we 222

E valuating E fforts to C ommunicate R esearch

consider “a little” a negative response; only respondents who report “quite a bit” are counted as affirmative. In 2014, we conducted phone interviews of 18 state legislators (69 percent response rate) who attended a Seminar on early brain science. Because of the seminars, 94 percent of legislators indicated they were “quite a bit more likely” to consider how pending legislation might affect families and 94 percent said they were “quite a bit” more likely to consider how new legislation that I am drafting might affect families. Also, because of the Seminars, 83 percent of legislators reported being “quite a bit” more likely to see the practical value of research. The results are displayed in Figure 8.5. Potential reasons for these attitude changes are suggested in the feedback from Seminar participants. For example, one state agency section chief wrote: The Family Impact Seminars are something we make time for. Many people from around the department, including division administrators, attend regularly. Most important, this is the only time such a broad group gets together to really discuss family-related issues in an atmosphere that encourages good public policymaking over politics. A Republican legislator expressed the value to him of receiving research on a representative sample of families in a culture where anecdotes are commonplace: In the Capitol, we get removed three or four steps from the folks who actually have to deal with a lot of the laws that we pass. We don’t have to meet them at the counter when they go to the family services department at the county or the city or the school. . . . [The Seminars] gives us an opportunity to see through studying how this affects large groups of people. If we do get involved, it is usually anecdotal information and so we deal with one or two people that talk to us about an issue. But we don’t have the numbers and research to say how does it affect folks in a large scale and that is very, very important. “Because of the Seminars”, Percent of Legislators Who Were “Quite a Bit” More Likely to . . .

Consider how new legislation that I am developing might affect families

94%

Consider how pending legislation might affect families

94%

83%

See the practical value of research

View researchers as approachable

72%

Figure 8.5 Attitude Changes Attributed to the Family Impact Seminars Note: Based on phone interviews of 18 Wisconsin legislators (69 percent response rate) following the 2014 early brain development Family Impact Seminar

223

N ew E ra of T heory , R esearch , E valuation

Builds Relationships Any effort to engage policymakers entails entering a culture in which relationships are essential for getting anything done (Bimber, 1996; Bogenschneider, Corbett et al., 2019; Hird, 2005; Oliver, Innvaer et al., 2014; Nutley et al., 2007; Mayne et al., 2018; Oliver  & Cairney, 2019; Tseng et  al., 2018; Weiss et  al., 2008). Thus, it seems safe to assume that efforts to connect research and policy should be developed with an eye toward tracking and assessing impacts on relationship building. The tracking begins with monitoring efforts of Seminar organizers to build relationships with policymakers. The Oregon Family Impact Seminars has found that building relationships takes a substantial amount of time and effort, but is an investment that is vital to their success. Based on their experience, email communication was less effective for initial contacts than in-person meetings and personal connections (Tomayko et al., 2019). We also have some preliminary indications that the Seminars may help build relationships by dispelling stereotypes that both policymakers and researchers hold of each other. For instance, researchers report being impressed by how informed and perceptive policymakers are, and policymakers report being surprised at how practical and downto-earth researchers are. These perceptual shifts appear to translate into more frequent contact. For example, based on interviews following the Wisconsin Seminar on early brain science, 72 percent of policymakers reported that because of the Seminars, they were “quite a bit” more likely to view researchers as approachable (see Figure 8.5), with 17  percent of policymakers reporting that they had contacted researchers for further information (N  =  18, 69  percent response rate). The ability of the Seminars to dispel stereotypes and usher in mutual respect was aptly stated in an unsolicited blog by a longtime political operative, Bill Kraus (2013): One of the most noticeable and welcome side effects of these sessions is that they dispel the notions that anti-intellectualism is rampant on one end of State Street [the legislature] and unintelligent intellectualism dominates the other end [the university]. Because the Seminars are only about solutions, they somehow engender civility and mutual respect as well. Consistent with social learning theory, empirical studies reveal that the use of research also depends on policymakers’ relationships with each other (Bogenschneider & Bogenschneider, 2020; Bogenschneider, Corbett et  al., 2019; Krehbiel, 1992; Matthews  & Stimson, 1975; Walter et al., 2005). One side benefit of the Seminars that we had not initially anticipated is the unique ability of the Seminars to provide an opportunity for policymakers to talk to each other. The role the Seminars play in building collegial relationships has emerged from both quantitative and qualitative data. For example, in phone interviews following a Wisconsin Seminar on early brain science, most legislators (89 percent) reported that the Seminar helped them identify others interested in the issue and about three-fourths (72 percent) reported sharing the information with colleagues (N = 18, 69 percent response rate). One legislator specifically commented on the political value of a nonpartisan organization that can pull such bipartisan participation: The Family Impact Seminars provide a catalyst for problem solving across partisan sides. It helps legislators identify others from across the aisle. Oftentimes, I  find I  am surprised by who attends. I  wouldn’t have otherwise anticipated 224

E valuating E fforts to C ommunicate R esearch

that a legislator from across the aisle might have a response similar to mine. In general, bipartisan support is necessary [for policies] to stay intact and not have them repealed with leadership changes. A Democrat explained that her colleagues were willing to listen to Seminar research because it did not carry “partisan” baggage. This credibility across party lines is key in the policy culture where research has to be believed, not only by your allies, but also by your political adversaries (Bogenschneider & Bogenschneider, 2020). A former gubernatorial staffer with a long reputation for working with members of both parties (Kraus, 2013) offered one reason that legislators believe research from the Seminars: [Karen] asks the legislators what subjects the Seminars should be about instead of vice versa, instead of telling them what she or others think is important to them. Once Seminar topics are selected, Karen finds appropriate experts to participate in a seminarial discussion with the legislators and staffers. This is a discussion not a lecture. That too they find endearing. What the experts bring to the discussion are facts. What the legislators and staffers do is turn the facts and research into legislative policy. The magic is that people who disagree philosophically and politically see that collaboration is not only possible, but desirable when the focus is on the facts. The Seminar discussions are beneficial, according to Wisconsin Senator Mark Miller, because they allow for a broad-ranging discussion of research without having to defend or attack a particular piece of legislation (Miller, 2019). He also emphasized to a group of Seminar organizers what a valuable service it is to provide an institutional structure for discussing issues across party lines: Now interestingly enough, in the 10 years that I’ve been in public office, I’ve also come to realize that we tend not to make decisions either as voters or as legislators based on facts. We tend to make them based on emotion. Emotion tends to override reason. So, we need to find ways of delivering facts, information, and research in a way that breaks down those barriers, that opens our minds to what the research tells us. And that is the challenge I think that you’re all going to face as you try to implement these kinds of programs in your communities. How do you break down the ideological and the partisan barriers that prevent us from communicating effectively? I think that institutional frameworks, such as the Family Impact Seminars, are extraordinarily beneficial in making that happen  .  .  . in creating that institutional framework by which we can have a discussion across the partisan and ideological divides that often separate us, unnecessarily. Because what I find . . . is that we have more commonality than differences. But our differences are what keep us from coming to effective solutions. And you can help them get that together. One Republican legislator explained how involvement in the Seminars helped him develop a reputation as a “problem solver” with members of the minority party: I am one of the leaders in the Republican side of the aisle. However, I  think that I’m viewed and well respected by members on the other side of the aisle. I really think one of the reasons why that is the case is because of things such as 225

N ew E ra of T heory , R esearch , E valuation

the Family Impact Seminar where you’re really trying to reach across the aisle, reach across the great divide to people and other legislators that may have a completely different worldview of how you solve these difficult problems. So, it has helped me tremendously in terms of building my credibility and integrity with members in the other party. Policymakers Use Research in Their Jobs Research is sometimes thought to be ignored by policymakers when, in fact, it may have indirect effects or may have slowly seeped into the policymaking process (Weiss, 1999). In response, the end-of-session questionnaire asked policymakers to provide retrospective reports of research use from the five or six most recent Seminars. As an example, one legislator reported how he used information from a previous Seminar on economic development (Parrott & Taub, 2014): I carry around the report on the economic development programs that states can do. It has useful information on the most effective tools. Tax incentives are not effective. When I need research to back up a point, I rely on the Family Impact Seminars. We also let time lapse before we conduct follow-up evaluations. A couple months after a Seminar, we (sometimes) conduct 10- to 30-minute phone interviews to find out how useful the information has been after policymakers have had time to reflect on it and the issue has advanced along the legislative process. In Wisconsin, three to five months after an early brain science Seminar, almost 8 in 10 legislators (79 percent) reported incorporating the research into speeches and presentations, over half (56 percent) reported using Seminar information to find references for further information, almost 4 in 10 (39 percent) to evaluate pending legislation, one-third (33 percent) to respond to constituents’ questions, and almost 3 in 10 (28 percent) to draft legislation (N=18, 69 percent response rate). The multiple ways that policymakers factor seminar research into their jobs signal its usefulness and policy relevance. We also track how Family Impact Seminar information is taken up in the legislative process. Policymakers often report in written questionnaires, interviews, and testimonials that Seminar information influenced a policy decision, sometimes to enact laws and sometimes to defeat legislation. Even though we make no claims that the Seminars, in and of themselves, cause a policy change, several examples exist where policy decisions may have been influenced by Seminar research: • Following a Seminar in Nebraska on ways to provide access to health care among low-income families, the legislature passed a State Children’s Health Insurance Program discussed by a Seminar speaker; in 2018, over 59,000 children were served by the program (National Academy for State Health Policy, 2019); • In Oregon, following a 2015 Seminar on two-generation approaches to poverty, the state became the first in the nation to allocate a greater percentage of the Earned Income Tax Credit to families with children under 3 years of age; in 2017, the tax credit was claimed by 265,000 low-income parents with an average refund of $2,130 (State of Oregon News Room, 2018); • In Indiana, after a middle school violence Seminar, the legislature passed a law requiring a safety specialist in the 25 Indiana public schools that serve over 1 million students; 226

E valuating E fforts to C ommunicate R esearch

• In Michigan, a juvenile justice Seminar highlighted judicious use of assessment and the implementation of evidence-based programs in local communities; in a followup summit, the Department of Human Services, the courts, and local offices reported on the positive results of implementing some of these changes; and • In Wisconsin, four features of other states’ prescription drug programs that were presented at two Seminars were incorporated into a prescription drug law enacted shortly thereafter; in 2018, enrollment was over 107,000 seniors (University of Wisconsin Population Health Institute, 2019). The value of the information provided by the Seminars is illustrated here in reference to two Wisconsin Seminars on prescription drugs. As explained by a Wisconsin senator: We didn’t have a prescription drug plan in the state, so this was a timely seminar because we were in the process of making decisions on what it should be. And I served in the Assembly on the Health Committee and we were looking at one that was going to cost a million bucks. Well, I think it was like 20 bucks a person, that’s nothing. . . . [The Seminars] had experts come in and talk to us and tell us what other states were doing, what it cost . . . so, that helped me as a legislator sort of working on the fringes of that thing saying, “Well, if you’re going to do it, you have to spend enough money to make it worthwhile. Otherwise, don’t do it.” And realizing if you start it . . . it’s going to cost a heck of a lot more in a couple years than you ever dreamed. So don’t go with the illusion that you spend $14 million today and in 3 years, it’ll be $14 million. It will be a lot more. And guess what? They were right. Nobody was shocked. Well, some people were shocked, but not anybody who studied it. Comments like this and much of our evaluation data derive from self-reports, which raises concerns about social desirability bias. To partially address this concern, we created contrast groups against which respondents could compare the Seminars (Riley, 2008). In follow-up interviews, we asked legislators and aides to rate several sources of information on family and social issues (see methods in Riley et al., 1991). We intentionally placed the Family Impact Seminars last on the list so the respondents have a frame of reference in mind before they complete the rating. As shown in Figure  8.6, legislators reported the Family Impact Seminars were the most useful source of information related to families and social issues; websites and media reports were the lowest rated information source. The difference in means between the Family Impact Seminar and the two lowest ­categories—media and websites—was statistically significant, F(8, 9) = 8.35, p < .003. We also receive unsolicited testimonials from key informants, both inside and outside the university, that compare the Seminars to other policy efforts. For example, Dave Riley, Associate Dean of Outreach, in the University of Wisconsin-Madison (UW) School of Human Ecology, the academic home of the Seminars, described the value of the Seminars to state government: In Wisconsin history, this is the only successful, ongoing, educational seminar series provided by the state university for state legislators and agency heads. Of the 10,000+ faculty in the UW system, it would be hard to point to more than one or two who could claim as great an impact on Wisconsin state policies . . . to show state government leaders the value and utility of their state University and of research-based knowledge generally. 227

N ew E ra of T heory , R esearch , E valuation

Family Impact Seminars

4.6

Family Impact Seminar briefing reports

4.4

State agency staff working on family and social issues

4.1

Constituents with concerns about family and social issues

4.0

Committee hearings related to family and social issues

3.8

Lobbyists who work on family and social issues

3.5

National legislative organizations

3.4

Websites related to family and social issues

3.1

Media reports on family and social issues

2.8 1

Low

2

3

Usefulness

4

5

High

Figure 8.6 Average Legislator Usefulness Ratings of Information Sources About Family and Social Issues Note: Based on evaluations completed by 18 legislators (69 percent response rate) attending a 2014 Wisconsin Family Impact Seminar on early brain development. Family Impact Seminars were rated significantly higher than websites and media reports, F(8, 9) = 8.35, p