Secondary Data in Mixed Methods Research 1506389570, 9781506389578

Citation preview

Secondary Data in Mixed Methods Research



Mixed Methods Research Series If you enjoyed this book, you may also enjoy these titles

Mieke Heyvaert, Karin Hannes, Patrick Onghena

Jessica T. DeCuir-Gunby, Paul A. Schutz

ISBN: 9781483358291

ISBN: 9781483365787

©2017 / 344 pgs.

©2017 / 288 pgs.

Leila C. Kahwati, Heather L. Kane

Michael D. Fetters ISBN: 9781506393599

ISBN: 9781506390215

©2020 / 312 pgs.

©2020 / 312 pgs.

Visit www.sagepub.com/mmrs to view all the books in this series.

Secondary Data in Mixed Methods Research

Daphne C. Watkins University of Michigan

FOR INFORMATION:

Copyright © 2023 by SAGE Publications, Inc.

SAGE Publications, Inc. 2455 Teller Road Thousand Oaks, California 91320 E-mail: [email protected]

All rights reserved. Except as permitted by U.S. copyright law, no part of this work may be reproduced or distributed in any form or by any means, or stored in a database or retrieval system, without permission in writing from the publisher.

SAGE Publications Ltd. 1 Oliver’s Yard 55 City Road London, EC1Y 1SP United Kingdom SAGE Publications India Pvt. Ltd. B 1/I 1 Mohan Cooperative Industrial Area Mathura Road, New Delhi 110 044 India

All third-party trademarks referenced or depicted herein are included solely for the purpose of illustration and are the property of their respective owners. Reference to these trademarks in no way indicates any relationship with, or endorsement by, the trademark owner. Printed in the United States of America ISBN: 978-1-5063-8957-8

SAGE Publications Asia-Pacific Pte. Ltd. 18 Cross Street #10-10/11/12 China Square Central Singapore 048423

This book is printed on acid-free paper.

Acquisitions Editor: Leah Fargotstein Product Associate: Ivey Mellem Production Editor: Prachi Arora Copy Editor: Integra Typesetter: Hurix Digital Cover Designer: Victoria Velasquez Marketing Manager: Candice Harman

22 23 24 25 26 10 9 8 7 6 5 4 3 2 1

• Brief Contents • Series Editors’ Introduction

xiii

Preface xv Acknowledgments xxi List of Figures, Tables, and Case Studies About the Author

PART I

• AN INTRODUCTION TO SECONDARY DATA IN MIXED METHODS RESEARCH

Chapter 1 • The Science of Secondary Data

xxv xxix

1 3

Chapter 2

• Using Secondary Data in Mixed Methods 21

Chapter 3

• Evaluating Secondary Data for Mixed Methods 37

PART II

• DESIGNING AND CONDUCTING MIXED METHODS WITH SECONDARY DATA

Chapter 4

• Convergent Design With Secondary Data 59

Chapter 5

• Exploratory Sequential Design With Secondary Data 89

Chapter 6

• Explanatory Sequential Design With Secondary Data 115

Chapter 7

• Complex Applications of the Core Designs With Secondary Data 143

PART III

• WRITING MIXED METHODS WITH SECONDARY DATA: BEFORE, DURING, AND AFTER

Chapter 8

57

161

• Early-Stage and Active Project Writing for Mixed Methods With Secondary Data 163

Chapter 9

• Reporting Mixed Methods With Secondary Data 185

Glossary 201 References 207 Index 219

• Detailed Contents • Series Editors’ Introduction

xiii

Prefacexv How This Book Came to Be

xv

Necessity Is the Mother of Invention

xvi

What This Book Covers

xvii

How Readers Can Use This Book

xviii

Audiences for This Book

xix

Pedagogical Features

xix

Structure of the Book

xx

Part I: An Introduction to Secondary Data in Mixed Methods Part II: Designing and Conducting Mixed Methods With Secondary Data Part III: Writing Mixed Methods With Secondary Data: Before, During, and After

xx xx xx

Acknowledgmentsxxi List of Figures, Tables, and Case Studies About the Author

xxv xxix

PART I  •  AN INTRODUCTION TO SECONDARY DATA IN MIXED METHODS RESEARCH

1

Chapter 1 • The Science of Secondary Data

3

Imagine3 The Science of Secondary Data

4

The Epistemology of Secondary Data

8

Types of Secondary Data Secondary Quantitative Data Secondary Qualitative Data

Scientific Developments With Secondary Data Knowledge Pursuit as a Scientific Development of Secondary Data Technology as a Scientific Development of Secondary Data Big Data as a Scientific Development of Secondary Data

10 10 12

14 16 16 17

Chapter 2 • Using Secondary Data in Mixed Methods

21

Imagine21 The Knowledge-Level Continuum as a Roadmap for Mixed Methods

22

What Is Required to Do Mixed Methods With Secondary Data?

26

Secondary Data in Mixed Methods: Three Reasons Why

28

Examine the Potentially Untapped Possibilities to Expand Knowledge

28

Gauge the Depth of a Topic, So You Know How to Proceed Quantitatively

29

Gauge the Breadth of a Topic, So You Know How to Proceed Qualitatively

30

Examples for How Mixed Methods Could Benefit From Secondary Data 31 Mixed Methods Solo, Semi-Solo, and With a Team

Chapter 3 • Evaluating Secondary Data for Mixed Methods

33

37

Imagine37 The Life Cycle of Research Data and a Place for Secondary Data

38

Three Considerations When Selecting Secondary Qualitative and Quantitative Data

40

First Consideration: Operationalizing the Terms in Your Research Question

40

Second Consideration: Locating Secondary Data

42

Third Consideration: Evaluating Secondary Data

43

Ten Questions When Evaluating a Secondary Qualitative or Quantitative Data Source

44

Question 1: What Was the Purpose of the Original Research?

45

Question 2: Can You Retrieve the Data?

45

Question 3: How Timely Are the Data?

46

Question 4: Who Collected the Data? How Were They Collected?

49

Question 5: What Types of Questions Were Asked?

49

Question 6: How Relevant Are the Data to Your Research Questions?

50

Question 7: Do The Planned Variables (or Concepts) Match Those of the Original Study?

51

Question 8: What Are the Sampling Strategies? Response Rates? Missing Data?

51

Question 9: Are You Equipped to Handle the Analysis?

52

Question 10: Which Component of Your Mixed Methods Project Will the Secondary Data Help Complete?

53

Applying the Ten Questions to Secondary Data

53

PART II  •  DESIGNING AND CONDUCTING MIXED METHODS WITH SECONDARY DATA

57

Chapter 4 • Convergent Design With Secondary Data

59

Imagine59 Features of the Convergent Design

60

Bringing Secondary Data Into the Convergent Design Preparing Secondary Data for a Convergent Design Data Analysis and Integration for a Convergent Design With Secondary Data Analyzing Data and Integrating the Results Interpreting the Results

Planning and Implementing a Convergent Design With Secondary Data Convergent Designs With Secondary Quantitative Data Sampling for a convergent design with secondary quantitative data Qualitative data collection instruments when the quantitative data exist Example of a convergent design with secondary quantitative data Convergent Designs With Secondary Qualitative Data Sampling for a convergent design with secondary qualitative data Quantitative data collection instruments when the qualitative data exist Example of a convergent design with secondary qualitative data Convergent Designs With Secondary Quantitative and Qualitative Data Sampling for a convergent design with secondary quantitative and qualitative data Example of a convergent design with secondary quantitative and qualitative data

Challenges (and Solutions) When Using Secondary Data in the Convergent Design

Chapter 5 • Exploratory Sequential Design With Secondary Data

61 63 65 66 68

72 72 72 73 74 77 77 78 79 82 82 82

85

89

Imagine89 Features of the Exploratory Sequential Design

90

Bringing Secondary Data Into the Exploratory Sequential Design

92

Preparing Secondary Data for an Exploratory Sequential Design Data Analysis and Integration for an Exploratory Sequential Design With Secondary Data Analyzing Data and Integrating the Results Interpreting the Results

Planning and Implementing an Exploratory Sequential Design With Secondary Data Exploratory Sequential Designs With Secondary Qualitative Data Sampling for an exploratory sequential design with secondary qualitative data Quantitative data collection instruments when the qualitative data exist

93 96 96 98

99 99 99 100

Example of an exploratory sequential design with secondary qualitative data Exploratory Sequential Designs With Secondary Quantitative Data Sampling for an exploratory sequential design with secondary quantitative data Qualitative data collection tools when the quantitative data exist Example of an exploratory sequential design with secondary quantitative data Exploratory Sequential Designs With Secondary Qualitative and Quantitative Data Sampling for an exploratory sequential design when both qualitative and quantitative data exist Example of an exploratory sequential design with secondary quantitative and qualitative data

Challenges (and Solutions) When Using Secondary Data in the Exploratory Sequential Design

Chapter 6 • Explanatory Sequential Design With Secondary Data

100 103 103 104 104 107 107 108

110

115

Imagine115 Features of an Explanatory Sequential Design

116

Bringing Secondary Data Into the Explanatory Sequential Design

117

Preparing Secondary Data for an Explanatory Sequential Design Data Analysis and Integration for an Explanatory Sequential Design With Secondary Data Analyzing Data and Integrating the Results Interpreting the Results

121 122 122 126

Planning and Implementing an Explanatory Sequential Design With Secondary Data 127 Explanatory Sequential Design With Secondary Quantitative Data Sampling for an explanatory sequential design with secondary quantitative data Qualitative data collection instruments when the quantitative data exist Example of an explanatory sequential design with secondary quantitative data Explanatory Sequential Design With Secondary Qualitative Data Sampling for an explanatory sequential with secondary qualitative data Quantitative data collection instruments when the qualitative data exist Example of an explanatory sequential design with secondary qualitative data

127 127 128 131 131 131 132 133

Explanatory Sequential Designs With Secondary Quantitative and Qualitative Data Sampling for an explanatory sequential design with secondary quantitative and qualitative data Example of an explanatory sequential design with secondary quantitative and qualitative data

135 136 136

Challenges (and Solutions) When Using Secondary Data in the Explanatory Sequential Design

139

Chapter 7 • Complex Applications of the Core Designs With Secondary Data

143

Imagine143 Features of a Complex Design

144

Bringing Secondary Data Into Complex Designs

145

Preparing Secondary Data for Complex Designs

146

Case Studies of Complex Mixed Methods Designs With Secondary Data 149 Case Study 1: Complex Evaluation Design With Secondary Data Case Study 2: Complex Embedded Design With Secondary Data Case Study 3: Complex Longitudinal Design With Secondary Data

149 152 155

Challenge (and Solution) When Using Secondary Data in Complex Designs158

PART III  •  WRITING MIXED METHODS WITH SECONDARY DATA: BEFORE, DURING, AND AFTER

161

Chapter 8 • Early-Stage and Active Project Writing for Mixed Methods With Secondary Data

163

Imagine163 Early-Stage and Active Project Writing: Defined

164

Examples of Early-Stage Writing

165

Early-Stage Writing Example 1: Journaling to Plan 165 Early-Stage Writing Example 2: Developing Research Proposals 167 Early-Stage Writing Example 3: Planning With Diagrams, Tables, and Figures168 Early-Stage Writing Example 4: Drafting Study Protocols 169 Early-Stage Writing Example 5: Developing Training Materials 170

Examples of Active Project Writing Active Project Writing Example 1: Journaling to Track Active Project Writing Example 2: Keeping Field Notes Active Project Writing Example 3: Tracking With Diagrams, Tables, and Figures Active Project Writing Example 4: Developing Codebooks

171 172 173 174 175

Tips to Maximize Early-Stage and Active Project Writing Tip #1: Start Early Tip #2: Review Previous Literature Tip #3: Use Templates for Inspiration Tip #4: Use Technology Tip #5: Collaborate

Chapter 9 • Reporting Mixed Methods With Secondary Data

177 177 178 179 180 181

185

Imagine185 Reporting Mixed Methods With Secondary Data

186

Project Deliverables for Mixed Methods With Secondary Data

187

Challenges When Reporting Mixed Methods With Secondary Data

189

Tips for Reporting Mixed Methods With Secondary Data

191

Tip #1: Introduce the Value-Added by Using Secondary Data to Achieve the Study Purpose and Answer the Research Questions 192 Tip #2: Outline Advantages of Using Secondary Data in the Study Design 193 Tip #3: Discuss Alignment Between the Secondary Data and the Research Question 193 Tip #4: Describe the Appropriateness of Secondary Data for Successful Analysis and Integration 194 Tip #5: Align the Secondary Results With Study Findings and Inferences195 Tip #6: Present the Secondary Results So They Clearly Illustrate Their Contributions to the Study 195 Tip #7: Report How Using Secondary Data Contributed to Theory 196 Tip #8: Share Limitations When Using Secondary Data 196 Tip #9: Discuss How Using Secondary Data Contributed to Furthering Knowledge on the Topic 197 Tip #10: Discuss How Using Secondary Data Contributed to Furthering Mixed Methods 198

Glossary201 References207 Index219

• Series Editors’ Introduction •

R

esearchers today have an amazing array of possibilities for the data that they use in their research studies. Traditional sources such as surveys, interviews, and documents have expanded into a wide range of potential data forms including digital records, social media posts, and various multimedia forms, to name a few. Furthermore, researchers have also enhanced their interest in creating data repositories and undertaking secondary analyses of existing databases including “big data” applications. These developments have expanded the kind of questions that researchers have been able to examine and provided opportunities for additional insights from data that have already been gathered. The opportunities to reuse existing data have also added new complexities to the methodological considerations and contextual influences that are involved in the conduct of research. The expanded availability of existing data for secondary analyses is of high interest to researchers using mixed methods approaches. More and more researchers are turning to existing databases as a source of salient information to be included within mixed methods studies. These databases have unique potential for addressing complex questions with mixed methods approaches, such as by combining trends within national data sets with in-depth case descriptions or analyzing existing interview or focus group transcripts to identify variables from existing data sets for statistical testing. Although mixed methods research has always supported the use of diverse types of quantitative and qualitative data, for too long discussions in the field have given limited attention to the use of existing data. This lack of attention has left researchers with little guidance about the methodological considerations and logistical challenges of integrating existing data and secondary analyses within mixed methods designs. We are therefore pleased to welcome the addition of Secondary Data in Mixed Methods Research by Daphne C. Watkins to the Mixed Methods Research Series. This book is the first to offer practical methodological guidance specifically targeted to the use of existing data within mixed methods research studies. This innovative and timely book provides an essential bridge between established discussions of mixed methods designs and the unique considerations and practical realities of researching with existing data. In crafting this book, Dr. Watkins draws extensively on her experience using existing data within mixed methods studies as well as her passion for the unique value and benefits of this approach. She creatively uses her expertise to provide practical guidance for scholars who plan to incorporate secondary data within their mixed

xiii

xiv   Secondary Data in Mixed Methods Research methods studies. Her accessible writing style, powerful graphics, and practical advice make the book feel like a personal consultation with the author as if readers were discussing the approach while sitting in her office. Furthermore, the book advocates for all scholars to consider the possibilities for mixing methods with secondary data to generate opportunities for their own research agenda and topics of interest. Without question, increased use of existing data within the big data movement is a future trend for mixed methods research, and this book provides valuable guidance for researchers to get started and complete mixed methods studies with secondary data. Vicki L. Plano Clark and Nataliya V. Ivankova Editors, Mixed Methods Research Series

• Preface •

W

riters have the difficult task of trying to anticipate why someone would decide to read their book. This can make the act of writing stressful, as good writers would hope to elicit a healthy combination of excitement about the topic of interest and insight into why the reader should be engaged in the topic. I can only assume that if you are reading this book, it is because you want to answer an essential question in your research program: “Should I use secondary data in my mixed methods research?” Or perhaps you are reading this book because you are already convinced that mixed methods are for you and, more importantly, that you want to incorporate some secondary data sources into your mixed methods design. Or maybe someone has suggested that you consider using secondary data in your mixed methods study, and you politely declined their offer (why do they want you to make your research project more complicated than it already is?). But, after giving it some thought, you decided that you cannot completely rule out the idea of using secondary data in your mixed methods study without looking deeper into what such an effort would require. Thus, you are coming to this book with a different question. You want to know, “How can I use secondary data in my mixed methods research?” Guiding procedures used for secondary data studies can serve as benchmarks for the conduct of all secondary data studies that follow. Why? This is because your ability to compare these procedures across different studies can help standardize the steps for working with secondary data and help deepen your understanding of the replicability (for quantitative studies) and transferability (for qualitative studies) of these procedures within and across research communities. Researchers who work primarily with secondary quantitative data are well-equipped to do so because most resources for conducting secondary data analyses are geared toward researchers who work with quantitative data. Researchers interested in working with secondary qualitative data have fewer resources, and until now, no resources existed for how to work with secondary qualitative and quantitative data in mixed methods research.

How This Book Came to Be A few years ago, I searched high and low for a book that would teach me how to address the unique challenges of using secondary data in mixed methods research. But, unfortunately, I never found such a book. So now I am writing the book that I had hoped to read. But, if you are new to mixed methods or new to using secondary data in your research projects, you may be asking xv

xvi   Secondary Data in Mixed Methods Research yourself, “Why does the world need a book that teaches readers how to use secondary data in mixed methods research?” I believe there are several reasons for this. First, data collection, data management, and data analysis procedures have improved with the expansion of technology. More so now than ever before, research teams are beginning to explore more efficient ways to collect, manage, and analyze their quantitative and qualitative data to maximize its use and dissemination for research purposes. These data sets can be large, publicly available data sources or smaller, privately owned ones. Second, more people are storing and managing their data sources on the Internet, but they are also engaged in data-sharing practices so that more than just their teams have access to their data files. Over the past few years, sophisticated data-sharing plans have become a requirement for federally funded research projects, and thus, the expectation to provide access to one’s data has heightened. Similarly, more complex research topics in the health and social sciences are being explored. The methodological details surrounding these explorations are shared across various technology, cloud, and social media-based platforms. An example is how online scientific journals now allow authors to publish resource materials to supplement their accepted publications. This type of information sharing has been expanded with the increase of Internet-based scientific journals. Finally, when one realizes they have access to secondary data, it is only natural to seek resources that explain how to incorporate secondary data into their single-method and mixed methods projects. Unfortunately, most books cover secondary data for single-method studies and are geared more toward quantitatively oriented audiences. The good news is that books on secondary qualitative data analysis are increasing. While working on this book, three new books on secondary analysis of qualitative data were published (Beck, 2019; Hughes & Tarrant, 2020; Largan & Morris, 2019). My book seeks to fill this gap in knowledge by providing readers with comprehensive and practical guidelines for using secondary data in their mixed methods projects.

Necessity Is the Mother of Invention Despite popular belief, mixed methods research is not a new phenomenon. Those of us who have been in the business of combining qualitative and quantitative data to address our research questions for many years are usually taken aback when someone refers to mixed methods research as a “new“ method (or methodological) technique. Other books chronicle the history of mixed methods as a study design (Creswell, 2009; Creswell, Goodchild, & Turner, 1996; Maxwell, 2016), as well as the differences and commonalities across disciplines that use mixed methods (Curry & Nunez-Smith, 2015; Haight & Bidwell, 2016). So I will not cover these topics in this book. Instead, I will use the pages of this book to delve into using secondary data in mixed methods, which is a topic rarely covered in mixed methods texts and is an unwritten expectation for mixed methods researchers who want to streamline their data, resources, time, and money.

Preface  xvii

I remember writing my first proposal for a mixed methods study that used secondary data. About three years after I defended my mixed methods dissertation using new qualitative and quantitative data, I found myself at a large research institution with many secondary data sources that postdoctoral fellows and faculty were encouraged to use. I had also recently met new colleagues from other departments excited to work with me, an early career scholar who had just as much qualitative methods experience as I had quantitative methods experience. So it was only natural for them to offer their qualitative and quantitative data sets to help me answer some of my lingering research questions. During that time in my career, I was familiar with collecting my own qualitative and quantitative data for my mixed methods studies. Similarly, I knew how to use a secondary data set as the first phase of a mixed methods study. But developing a new mixed methods proposal that included two secondary data sources intrigued me, so I did what any researcher in my position would do: I wrote a grant proposal for a project idea using a secondary qualitative data source and a secondary quantitative data source. After sketching my preliminary grant ideas on a legal pad, I was excited to immerse myself in the literature on using secondary/existing data sources. So I searched library databases for peer-reviewed articles, books, dissertations, reference manuals, and research textbooks. And guess what I found? Nothing. Well, maybe that is not entirely true. I did uncover a few articles that alluded to secondary data for one phase of a mixed methods study. I also located multiple resources (books primarily) that discussed using existing (or secondary) data sources to do social science, health science, and education research. But unfortunately, few resources were suitable for helping me combine two preexisting data sources to address a research question. So, rather than giving up, I pooled resources from my doctoral program and postdoctoral training to create a mixed methods study design that used a secondary qualitative data source to construct a conceptual framework that I would test with a secondary quantitative data set. To my surprise, the grant was funded. And that is when the hard work began. So, if necessity is the mother of invention, this book on using secondary data in mixed methods research is long overdue.

What This Book Covers This book, Secondary Data in Mixed Methods Research, emphasizes how using existing, or secondary (qualitative and quantitative) data sets in mixed methods research can help answer new and ongoing research questions. Readers should note my use of “existing data” and “secondary data” throughout the book. Though the book is titled Secondary Data in Mixed Methods Research, while writing, I grappled with whether I should use “secondary data” or “existing data” to describe the types of data one might incorporate into a mixed methods project. I quickly learned that while researchers tend to refer to existing data as “secondary data,” discussing secondary data alongside other secondary components of mixed methods research such as “second priority,” “secondary study

xviii   Secondary Data in Mixed Methods Research phases,” and “secondary analysis” could be challenging for novice researchers to follow. To limit confusion, I decided to use “secondary data” and “existing data for secondary purposes” interchangeably throughout this book. Using existing qualitative and quantitative data for secondary purposes is still evolving and not without its limitations; the subsequent chapters cover the strengths and weaknesses of using existing data in mixed methods research. For instance, researchers and analysts interested in using secondary data may find that the methodological decisions for collecting the original study data are limited or inaccessible. This book provides easy-to-follow instructions for working with data under such conditions and navigating between and across unfamiliar data repositories. This book also describes strategies for sifting through data records and making sense of the available information associated with secondary data. Increasing our use of secondary data sources adds to the unique possibilities for further defining and operationalizing mixed methods research. Previous studies suggest secondary data are usually used initially in (or for the first single-method phase of) a mixed methods study (Gray & Geraghty, 2020; Smith, 2008). However, as qualitative and quantitative data sources become more robust and sophisticated, the potential for implementing entire mixed methods studies using previously collected data for purposes aligned (and not aligned) with the original research is promising. Integrating secondary qualitative and quantitative data can make connecting two different data sources with different purposes a reality.

How Readers Can Use This Book This book teaches readers how to (1) identify key characteristics of secondary data that can be used to accomplish the goals of a mixed methods project, (2) follow a step-by-step procedure for incorporating secondary data into various mixed methods research designs, and (3) expand their use of secondary data to address more social sciences and health sciences research questions. The emphasis of this book will be on practical guidelines that help readers conceptualize, design, and analyze mixed methods studies that include secondary data. I suspect readers could also use this book to propose mixed methods theses and dissertations where one or both study components contain secondary data. I anticipate this book being used alongside other mixed methods books. For example, this book could serve as supplemental reading to books on designing and conducting mixed methods research. This book could be paired with other books in this or other mixed methods series to teach readers the advantages of using secondary data to obtain baseline information about a community of interest. This information could then be used to design, implement, and evaluate (using mixed methods) a culturally sensitive intervention to improve community members’ health. This book could also be paired with theses and dissertation proposal handbooks to help students expound on using secondary data in mixed methods proposals and help them think through whether

Preface  xix

additional time and money associated with more primary data collection are needed for their theses and dissertations. Finally, this book could accompany grant writing books and help researchers think through their various research design options to achieve their research goals. Though many mixed methods topics will be covered in this book, it is a short volume that will include references to more comprehensive texts.

Audiences for This Book This book, Secondary Data in Mixed Methods Research, provides a springboard for more innovative and cutting-edge discussions around mixed methods research in classrooms and on research teams and encourages researchers to think more broadly about their extensive use of secondary data in their mixed methods research. A primary audience for this book includes (1) doctoral students from anthropology, education, psychology, political science, nursing, sociology, public health, social work, psychiatry, health care management, and health services interested in using secondary data for their mixed methods dissertations; (2) faculty members looking to add readings to their qualitative, quantitative, and mixed methods courses so that students can gain an appreciation for integrating different data sources to address their research questions; and (3) researchers, analysts, and teams who need a basic understanding of mixed methods in their research design decision-making and grant writing. Examples of courses for which this book could be used as a supplemental book are qualitative research methods, quantitative research methods, introduction mixed methods, and advanced research methods. Other audiences may include researchers and analysts who oversee big data and work with publicly available, government-supported databases. This book could also be used during a mixed methods workshop, seminar, or class administered outside academia.

Pedagogical Features This book has important features you will find helpful as you navigate through the chapters. For example, all the chapters follow a similar structure and include seven features: an imagine section, learning objectives, key terms and definitions, callout boxes, tables and figures, summary, and chapter application questions. As the reader, you will find that the consistent chapter structures make it easy to focus on each chapter’s learning objectives and underscore the key terms and their definitions. The callout boxes are nice in that they help guide your eye to important statements made about concepts throughout the book; they can also help provide context for some of the tables and figures. Ending each chapter with a summary will ensure you take away the most important concepts for each chapter before moving to the next. Finally, the chapter application questions are a great way to check your competency and grasp of the material covered in each chapter.

xx   Secondary Data in Mixed Methods Research

Structure of the Book Part I: An Introduction to Secondary Data in Mixed Methods Part I of the book begins by addressing two important questions: “What are secondary data?” and “why should you consider using secondary data in your mixed methods research?” I then provide a basic introduction to secondary data and how to use secondary data to achieve one’s mixed methods research goals. This part of the book also outlines some advantages and disadvantages of using secondary data in mixed methods research, introduces big data (Basken, 2014; Mayer-Schonberger & Cukier, 2013), and provides evaluative questions for reviewing secondary data for potential inclusion in your mixed methods research.

Part II: Designing and Conducting Mixed Methods With Secondary Data In part II of the book, my goal is to answer the question: “How can secondary data be incorporated into mixed methods designs?” These chapters focus on describing the core mixed methods designs described by Creswell and others (Creswell, 2015; Creswell & Plano Clark, 2018; Plano Clark & Ivankova, 2016) and the ways secondary data can be incorporated into one or both components for convergent and sequential designs. Practical examples from the literature illustrate the strategies involved in each mixed methods research design and how to successfully execute the mixed methods designs using secondary quantitative and qualitative data for one or both study components. Each chapter in part two is written to fit the utility and function of incorporating secondary data into core and complex mixed methods designs.

Part III: Writing Mixed Methods With Secondary Data: Before, During, and After Part III of the book will address the important question: “How should I write when I have a mixed methods project that uses secondary data?” This section will discuss the writing of mixed methods with secondary data, before the project begins, during an active project, and after the project has ended. For example, during the early and active writing stages for your mixed methods with secondary data, I tap into the need to: start early, review previous literature, use templates for inspiration, use technology, and collaborate. This advice is also helpful when writing reports for mixed methods with secondary data.

• Acknowledgments •

W

hen I think about all the people who supported me, gave me an encouraging word, or simply cheered from afar, the list is extensive. So I will do my best to thank as many people as I can within the confines of these pages.  First, I want to thank the series coeditors, Vicki Plano Clark and Nataliya Ivankova, two of the most patient people I know. This book is probably the most challenging project I have completed to date, and most people do not realize the amount of writing and rewriting that goes into an advanced research methods book. So I am thankful for Vicki and Nataliya’s patience, encouragement, clarity, and support throughout this process. Next, I want to thank my colleagues at the University of Michigan School of Social Work (UM-SSW), V ­ ivian A. and James L. Curtis Center for Health Equity Research and Training, for their encouragement and support. In particular, people like Jamie Mitchell, Jaclynn Hawkins, Kirstn Tatar, Jamie Abelson, Sharon NorrisShelton, Kate Kloss, and Keith Miller cheered me on as I pushed this book over the finish line. They also encouraged me to celebrate every milestone. I want to thank the UM-SSW Continuing Education Office staff, Jaclyn Ruffolo and Alia Wesala. They are the super(s)heros who support my mixed methods certificate program, and the program would not be a success without them. I also want to thank my mixed methods certificate program teaching assistants/ associates, Janelle Goodwill and Natasha Johnson. Their support of our over 350 participants over the past eight years has been invaluable, and their dedication to advanced research methods for communities of color is unparalleled. Big thanks to Natasha, who also read early drafts of my chapters; I appreciate her insight and curiosity on this mixed methods journey.   I want to thank my colleagues at the Michigan Mixed Methods Program, John Creswell, Tim Guetterman, and Mike Fetters, for their support and leadership in the field. I want to thank the UM-SSW joint Ph.D. program director, Willie Elliott, for believing in my doctoral-level mixed methods course. But I must also thank the students who enrolled in the course. These students challenged me in unimaginable ways and consistently held the needs of communities of color at the forefront of every methods jewel I taught them. The future of mixed methods is bright.  I want to thank some fantastic colleagues and friends whose unwavering support has no doubt helped me be a better researcher: Kendra Hearn, Tanya Sharpe, Jodi Jacobson Frey, Briana Mezuk, Derek Griffith, Desmond Patton, Stephanie Rowley, Nkemka Anyiwo, Kara Zivin, Matthew J. Smith, Lynn Videka, Trina Shanks, Barb Hiltz, Katie Doyle, and Elizabeth Koschmann.

xxi

xxii   Secondary Data in Mixed Methods Research Similarly, I want to thank my dear colleagues, Jacquie Mattis, Robert Joseph Taylor, Linda Chatters, Cleopatra Caldwell, Mary Ruffolo, Earl Lewis, and B. Lee Green, for always being there when I need them. This means the world to me, and I cherish your candor. And speaking of honesty, I must also thank my parents, Donald and Lucy (Butts) Watkins, my sister-in-law, Shirl, and my brother Chris. Every year when I said, “I’m still working on this book,” you offered an encouraging word (or, in Chris’s case, a joke) that was always right on time. Thank you for your encouragement and for helping me “check out” of the work whenever I needed to replenish.   In his book On Writing, Stephen King writes that authors who thank their spouses and partners actually get it. This is because spouses and partners are there to witness every moment of the process of writing: the good, the bad, the frustrations, and the triumphs. So I have to take a moment to thank my husband, Paul Jacobs, for everything. Thank you for waking up early with the kids so I could write, thank you for “carrying the load” so I could steal away for a weekend writing retreat here and there, thank you for letting me run my ideas by you over breakfast, and thank you for just being you. I birthed a whole human while also birthing this book, and Paul supported me at every turn. I needed every nod of affirmation, every joke, every compliment, every snarky remark, and every cup of coffee you brought me while I was typing to my heart’s content. I could NOT have done this without you, and frankly, I would not want to. To my heartbeats, Quinn and Clay, your welcomed distractions were everything I needed to push through the final stages of writing this book. Seriously, I mean that.  Finally, I want to thank senior acquisition editor Leah Fargotstein, the incredible team at SAGE, and the reviewers whose feedback both challenged me and inspired me to write the book I wanted to see when I started using secondary data in mixed methods a decade ago: Steve Bilham, University of Bedfordshire Scott Bucker, University of Worcester Rhonda Erica Celey, Morgan State University Niyazi Ekici, Western Illinois University Jennifer Esposito, Georgia State University Sarah Ferguson, Rowan University Tamar Ginossar, University of New Mexico Susan Gunby, Mercer University Kelly S. Hall, Texas A & M University—Kingsville Larry Hearld, University of Alabama Su-I Hou, University of Florida

Acknowledgments  xxiii

Jill M. Humphries, University of Toledo Amy Roth McDuffie, Washington State University Courtney McKim, University of Wyoming Heidi A. Mennenga, South Dakota State University Phil Murphy, Middlebury Institute of International Studies Leah C. Neubauer, Northwestern University Antigoni Papadimitriou, Johns Hopkins University Beth Winfrey Shindel, Saint Louis University Minerva D. Tuliao, Texas Tech University Liyun Wu, Norfolk State University Julie Zadinsky, August University

• List of Figures, Tables, and Case Studies • Chapter Item

Title

CHAPTER 1 Table 1.1

The Evolving Definition of Secondary Data

Table 1.2

Examples of Secondary Quantitative Data Sources

Table 1.3

Examples of Secondary Qualitative Data Sources

Table 1.4

Real-World Examples for When to Use New Data Versus Secondary Data

CHAPTER 2 Figure 2.1

The Knowledge-Level Continuum

Table 2.1

Core Mixed Methods Designs and Definitions

Table 2.2

Sample Mixed Methods Projects That Could Benefit From Secondary Data

Table 2.3

Team Options for Doing Mixed Methods with Secondary Data

CHAPTER 3 Figure 3.1

The Life Cycle of Research Data

Figure 3.2

Decision Tree for Selecting Secondary Data Sources

Table 3.1

Ten Questions When Evaluating Secondary Data for Your Mixed Methods Research

Table 3.2

Sample Data Repositories in the Health and Social Sciences

Table 3.3

Applying the Evaluation Questions to a Sample Secondary Data Source

xxv

xxvi   Secondary Data in Mixed Methods Research CHAPTER 4 Figure 4.1

Convergent Design With New or Secondary Data

Big Data Break 4.1

Metadata in Convergent Designs with Secondary Data

Big Data Break 4.2

Analytics for Secondary Big Data in Convergent Designs

Table 4.1

Preparing Secondary Quantitative and Qualitative Data for a Convergent Design

Table 4.2

Sample Joint Data Display Combining Qualitative and Quantitative Findings

Case Study 4.1

Convergent Design With Secondary Quantitative Data

Case Study 4.2

Convergent Design With Secondary Qualitative Data

Case Study 4.3

Convergent Design With Secondary Quantitative and Qualitative Data

Figure for Case Study 4.1

N/A

Figure for Case Study 4.2

N/A

Figure for Case Study 4.3

N/A

CHAPTER 5 Figure 5.1

Exploratory Sequential Design With New and Secondary Data

Big Data Break 5.1

Types of Big Data With Sequential Designs

Big Data Break 5.2

Integrating Secondary Big Data Into Sequential Designs

Table 5.1

Preparing Secondary Qualitative and Quantitative Data for an Exploratory Sequential Design

Case Study 5.1

Exploratory Sequential Design With Secondary Qualitative Data

Case Study 5.2

Exploratory Sequential Design With Secondary Quantitative Data

Case Study 5.3

Exploratory Sequential Design With Secondary Qualitative and Quantitative Data

Figure for Case Study 5.1

N/A

Figure for Case Study 5.2

N/A

List of Figures, Tables, and Case Studies  xxvii

Figure for Case Study 5.3

N/A

CHAPTER 6 Figure 6.1

Explanatory Sequential Design With New and Secondary Data

Big Data Break 6.1

What Are Basic Big Data Analytics?

Big Data Break 6.2

What Are Advanced Big Data Analytics?

Table 6.1

Preparing Secondary Quantitative and Qualitative Data for an Explanatory Sequential Design

Case Study 6.1

Explanatory Sequential Design With Secondary Quantitative Data

Case Study 6.2

Explanatory Sequential Design With Secondary Qualitative Data

Case Study 6.3

Explanatory Sequential Design With Secondary Quantitative and Qualitative Data

Figure for Case Study 6.1

N/A

Figure for Case Study 6.2

N/A

Figure for Case Study 6.3

N/A

CHAPTER 7 Figure 7.1

Complex Evaluation Design With Secondary Data

Figure 7.2

The Context for Preparing Data for a Complex Design

Case Study 7.1

Complex Evaluation Design With Secondary Data

Case Study 7.2

Complex Embedded Design With Secondary Data

Case Study 7.3

Complex Longitudinal Design With Secondary Data

Figure for Case Study 7.1

N/A

Figure for Case Study 7.2

N/A

Figure for Case Study 7.3

N/A

xxviii   Secondary Data in Mixed Methods Research CHAPTER 8 Table 8.1

Examples of Early-Stage Writing for Mixed Methods With Secondary Data

Table 8.2

Examples of Active Project Writing for Mixed Methods With Secondary Data

Table 8.3

Considerations When Working With a Secondary Quantitative and Qualitative Data Codebook

Table 8.4

Tips for Maximizing Early-Stage and Active Project Writing

Figure 8.1

Time Response Continuum

CHAPTER 9 Table 9.1

Potential Deliverables When Reporting Mixed Methods with Secondary Data

Table 9.2

Ten Tips for Writing Mixed Methods Reports With Secondary Data

• About the Author • Daphne C. Watkins is a professor of social work and a University Diversity and Social Transformation Professor at the University of Michigan. She became interested in mixed methods research during her quest to uncover the “voices behind the numbers” as a doctoral student. Professor Watkins has taught research methods to health and human service professionals for more than two decades. After noticing the lack of postgraduate research training, Professor Watkins developed the first certificate program in mixed methods research at the University of Michigan School of Social Work. The certificate program— designed for researchers and practitioners interested in doing mixed methods research in practice settings—was the motivation behind her first book, Mixed Methods Research (2015, Oxford University Press) for the Pocket Guides to Social Work Research Methods Series. In addition to developing culturally appropriate strategies for conducting mixed methods, Professor Watkins developed the rigorous and accelerated data reduction (RADaR) technique, an individual and team-based approach to organizing, coding, and analyzing qualitative data. Professor Watkins is particularly interested in using secondary data in mixed methods research to address health disparities and achieve health equity. She is the founding director of the Gender and Health Research (GendHR) Lab and the award-winning Young Black Men, Masculinities, and Mental Health (YBMen) Project, which leverages technology to provide mental health education and social support for young Black men. Professor Watkins currently directs the Vivian A. and James L. Curtis Center for Health Equity Research and Training at the University of Michigan.

xxix

PART I

An Introduction to Secondary Data in Mixed Methods Research

1

1 The Science of Secondary Data Imagine Your friend, Paul, has been racking his brain all month. He is a graduate student and wants to complete his master’s thesis exploring the effects of racial injustice on the likelihood that people of color will seek mental health treatment. Though his original plan was to collect his own data, his professor gave him access to a recent data set collected by the university counseling center on mental health diagnosis, assessment, and treatment. This data source includes 3,000 students, and 1,000 of them are students of color. “But wait,” Paul says to you one Wednesday afternoon while the two of you are writing at Starbucks, “I didn’t think I could use someone else’s data for my master’s thesis? Is that even allowed?” You smile as you sip your caramel macchiato. “Sure, it’s allowed. But have you ever used someone else’s data before?” Paul slowly shakes his head and looks down at his laptop. “Well, then…. Are you done with classes for the day?” You ask as you lean back in your chair. Paul gives you a disturbing look as he slowly responds, “Yes… why?” “Because,” you smile, “I’m about to tell you everything you need to know about using someone else’s data.” This chapter will introduce you to secondary quantitative and qualitative data and give you valuable examples to share with Paul.

3

4  Part I  •  An Introduction to Secondary Data in Mixed Methods Research

Learning Objectives This chapter provides a brief overview of secondary data, describes three scientific advances in using secondary data, and discusses the use of big data for knowledge acquisition. By the end of this chapter, you will be able to: 1. Discuss the science of secondary data, 2. Describe ways to be epistemologically sound in your use of secondary data, 3. Describe types of secondary data and relevant examples of secondary data, and 4. Name three scientific developments in secondary data and how they have shaped knowledge acquisition.

The Science of Secondary Data What exactly are secondary data? What role do they play in the advancement of science? Understanding the science of secondary data begins with understanding the distinct differences between primary data and secondary data (Corti, 2012; Johnston, 2014; Kitchin, 2014). Primary data are new qualitative or quantitative data collected to address a fundamental research question. Primary data are also called “new” or “original” data. On the other hand, ­existing data, or secondary data as it is often called, are not new and undergo secondary analysis to address a research question. Secondary data are usually related to the original, primary study goals and research questions. However, they are existing data used for secondary purposes (Corti & Backhouse, 2005; Corti, Van den Eynden, Bishop, & Woollard, 2014; Trinh, 2018). Using data collected by someone else to answer your research questions is a common and accepted practice in research and academic settings. ­Secondary data can range from large, publicly available data sources (e.g., U.S. Census data) to smaller, privately-owned data sources (e.g., local social service agency data for the client population). Whether they take the form of publicly available data points, papers, artifacts, or electronic documents, secondary data are valuable in health and social science research. They can provide answers to your new and preexisting research questions (Corti, 2014; Johnston, 2014). All secondary data were primary data at some point. Adding a temporal element to the definition of primary and secondary data is critical when distinguishing between the two because you can collect primary data now that you use for secondary purposes later. There is no hard and fast rule to the length of

Chapter 1  •  The Science of Secondary Data  

5

time that must pass before primary data are considered “secondary.” The only real distinction involves whether someone has a fundamental research question they want to answer and whether the data to address that question exists or not. For example, if you collected data two years ago and those data can help you answer a new research question today, you can use the data to answer the new research question. Perhaps the second research question might be unrelated to the original study and its intent. In this case, you would be using the existing data for a secondary purpose. The definition of secondary data has evolved. Table 1.1 chronicles the evolution of the meaning of secondary data beginning in 1963 and ending in 2020. Under some circumstances, using secondary data to address your research questions may save you time, money, and the resources required to initiate a new primary data collection project. Though secondary qualitative and quantitative data are collected for a different purpose, certain features may help answer your research questions, especially your mixed methods questions. To lay the foundation for the remainder of the book, here I define mixed methods as the rigorous and epistemological application and integration of qualitative and quantitative research approaches to draw interpretations based on the combined strengths of both approaches to influence research, practice, and policy (Plano Clark & Ivankova, 2016; Creswell, 2015; Watkins, 2017a; Watkins & Gioia, 2015). I discuss mixed methods more in Chapter 2 of this book. So refer to this definition from time to time, as your understanding of how to incorporate secondary data into your mixed methods crystallizes, and you make stronger links between these key terms and their definitions. Consider this scenario: You are a first-semester master’s student in public health, with hopes of doing a research project on statewide differences in alcohol consumption with your major professor. Unfortunately, your professor is swamped with a grant proposal for the next three weeks, so she has asked you to choose a research topic and locate some data to analyze. You recently overheard a classmate talking about some free and publicly available data on the Substance Abuse and Mental Health Services Administration1 website, so you decided to check it out. Given your interests in alcohol consumption, you choose to review the Substance Abuse and Mental Health Data Archive,2 where you find some online data analysis tools. After clicking “Analyze Data Online” on the left side of the screen, you are taken to the Public-use Data Analysis ­System (P-DAS), where you can explore basic descriptive statistics for alcohol consumption across each state in the country. You do not have much experience with statistics, so this option suits your needs and your professor’s needs. All in all, this feels like a great start to your research project on statewide ­alcohol consumption.

1

https://www.samhsa.gov/data/node/20

2

https://datafiles.samhsa.gov/

Definition

“The study of specific problems through analysis of existing data which were originally collected for another purpose.”

“Secondary data is the extraction of knowledge on topics other than those which were the focus of the original study.”

“Secondary analysis is the re-analysis of data to answer the original research questions with better statistical techniques or answer new research questions with old data.”

“A collection of data obtained by another researcher which is available for re-analysis.”

“Secondary data analysis is any further analysis of an existing dataset which presents interpretations, conclusions, or knowledge addition to, or different from, those produced in the first report on the inquiry as a whole and its main results.”

“Neither a specific regime of analytic procedures nor a statistical technique, [but] … a set of research endeavors that use existing materials.”

“Should be an empirical exercise carried out on data that has already been gathered or compiled in some way.”

“The further analysis of an existing dataset with the aim of addressing a research question distinct from that for which the dataset was originally collected and generating novel interpretations and conclusions.”

“Even re-analysis of one’s own data is secondary data analysis if it has a new purpose or is in response to a methodological critique.”

Year

1963

1972

1976

1981

1982

1985

1988

2006

2007

TABLE 1.1  ● The Evolving Definition of Secondary Data

Schutt, p. 4127

Hewson, p. 274

Dale et al., p. 3

Kiecolt & Nathan, p. 10

Hakim, p. 1

Sobal, p. 149

Glass, p. 3

Hyman, p. 1.

Glaser, p. 11

Author(s)

6  Part I  •  An Introduction to Secondary Data in Mixed Methods Research

“Secondary data are data made available to others to reuse and analyze that are generated by someone else.”

“…using data from a previous or ongoing study to test new hypotheses or answer questions not initially envisioned.”

“Secondary data” are data that were formerly collected for other purposes than that of the study at hand.”

“Secondary data analysis is commonly defined as the use of datasets, which were not collected for the purpose of the scientific hypothesis being tested.”

“Secondary data is collected by someone other than the researcher and with another purpose.”

2014

2016

2017

2018

2020

Panchenko & Samovilova, p. 1

Trinh, p. 163

Prada-Ramallal, Takkouche, & Figueiras, p. 352

Polit & Beck, p. 244

Kitchin, p. 7

Pienta, O’Rourke, & Franks, p. 13

Note: Authors often write about secondary data and secondary data analysis together (and interchangeably), noting that “secondary data” are the existing data on cases themselves. In contrast, “secondary data analyses” are the decisions and procedures associated with analyzing existing data for a secondary data.

“Secondary data are those data that have been made available for use by people other than the original investigators.”

2011

Chapter 1  •  The Science of Secondary Data  

7

8  Part I  •  An Introduction to Secondary Data in Mixed Methods Research In the previous scenario, you can access secondary data … if certain features of to initiate a research project the secondary data source about differences in alcohol consumption by state. could be used to advance your Though you had to search for understanding of a research this information on a publicly topic, why would you not available website after you consider using the secondary overheard a classmate talking about it, you feel lucky source? that things worked in your favor. You have the data you wanted, and you did not need to assemble a research team or travel state to state to collect this information. Why? This is because someone did this for you. Now ask yourself: What if you could access a qualitative or quantitative data source for a topic in your area of interest? What would you do? Would you forget about it and collect new data? Perhaps. But if certain features of the secondary data source could be used to advance your understanding of your research topic, why would you not consider using this secondary source? You can answer these questions by understanding the epistemology of secondary data.

The Epistemology of Secondary Data Understanding the benefits of using secondary data is key to advancing your research trajectory and productivity. Knowing how to use secondary data for a single-method study is a first step toward integrating two secondary singlemethod qualitative and quantitative data sources for mixed methods purposes. In alignment with my introduction to the science of secondary data, I would also like to discuss the epistemology of secondary data. Epistemology is the study of the distinction between acceptable belief and opinion (Cameron, 2011; Johnson & Onwuegbuzie, 2004). It unpacks belief, truth, and justification for each. Let’s consider an example of epistemology using belief. Let’s say you believe that 2  +  2  =  4, and your belief is based on truth supported by facts. If your beliefs were based on reliable information (e.g., asking someone to count the number of writing utensils on your desk and they count two pencils and two pens), it would qualify as knowledge. However, if your belief in this mathematical equation were based on unreliable opinions (e.g., asking someone to guess the number of writing utensils on your desk with their eyes closed), it would not qualify as knowledge. Data alone is not knowledge; the interpretation of data is knowledge. But how does this connect to our interest in secondary data? How can you be epistemologically sound in your research with secondary data? How can you be epistemologically sound with secondary data in your mixed methods? First, you must be explicit about your engagement with the

Chapter 1  •  The Science of Secondary Data  

9

secondary data (or lack thereof). If you did not help develop the original study questions or choose the eligibility criteria for the participants, then be straightforward about this and your desire to answer your research questions using data you did not help collect. In other words, be realistic about your limited knowledge of the original study development, protocol, and lack of interactions with the study participants, acknowledge that your analysis of their information is really “secondhand.” Next, using secondary data in an epistemologically sound way means remembering that knowledge is socially and historically located within a complex cultural context. Therefore, determining the degree to which the secondary data you plan to use aligns with your anticipated research question is more of an art than a science. The focus of your alignment should be less about perfecting the fit and more about what the secondary data source adds to the larger body of work on your topic. If you cannot perfectly fit your research question onto a secondary data source, what can you discern from the secondary data source that will answer a related research question? Or what can you determine that contributes to your overall research program’s short- or longterm goals? Finally, using secondary data in an epistemologically sound way means respecting the culture of the primary research (e.g., the culture of the research team and the culture of the research participants) and acknowledging the power dynamics between the researchers and the people being researched. The origin of the secondary data does not disappear because you are using the data for a different purpose than that of the original study team. One example is research conducted with incarcerated populations. It makes sense to acknowledge institutionalized populations when the data are recent, but sometimes, we have access to data from people in prison that are a few years old. Just because these secondary data are a few years old does not mean that the power dynamics between incarcerated individuals and the people who oversee them Be mindful of the culture have changed. Be mindful of the culture and power and power dynamics of the dynamics of the researchers researchers and those being and those being researched, researched, regardless of when regardless of when the secthe data were collected. ondary data were collected. Doing so means you can analyze and interpret the secondary data in their social, political, and economic contexts, considering the similarities and differences between when the data were collected and the present day. Your use of secondary qualitative and quantitative data in an epistemologically sound way can strengthen your inquiry into a topic of interest and increase the utility of secondary data for single-method or mixed methods purposes.

10  Part I  •  An Introduction to Secondary Data in Mixed Methods Research

Types of Secondary Data Secondary data scholars have promoted its utility and value for expanding knowledge for many years. Five decades ago, Herbert Hyman (1972) affirmed that “… existing data is the extraction of knowledge on topics other than those which were the focus of the original study” (p. 1). Extending this definition beyond that of secondary data in single-methods studies, one could argue that using secondary data in mixed methods is also a worthwhile strategy for extracting knowledge. As we begin our journey toward obtaining knowledge, let us review two types of secondary data: secondary quantitative data and secondary qualitative data.

Secondary Quantitative Data When I hear researchers discussing the possibilities of their secondary data, they usually refer to their secondary quantitative (e.g., numeric) data. Secondary quantitative data come in many forms, ranging from thousands of individual cases across a hemisphere to a dozen or so individual cases right there in your office building. At present, many groups collaborate to collect and archive massive amounts of data, so one may argue that it is reasonable to use those data sources to answer some of our research questions. Moreover, some scholars say that the availability of secondary data is so robust that its use is becoming more customary across various research settings (Andrews, Higgins, Andrews, & Lalor, 2012; Smith, 2008, 2011). When the layperson hears the term secondary data, they may think about government data sources, such as the United States Census Bureau or other data collected to track population characteristics. While writing this chapter, I searched Google for “Census Data.” I found it easy to access several demographic descriptors of people in the United States using the American ­FactFinder, Quick Facts, Data Tools, and Data Visualization. Other data sources are moving in this direction, as public and private agencies are now more transparent about their data collection, analysis, dissemination, and sharing processes. You may be interested in locating secondary quantitative data that are thorough and provide information about the primary study. Secondary data include, but are certainly not limited to, population-based surveys, cohort and longitudinal surveys, administrative records, and medical records. Table 1.2 provides examples of secondary quantitative data sources you should consider in your research. You will notice there are at least four different types of secondary quantitative data sources in the table, complete with definitions and examples for each. Other secondary data sources could vary, given diverse career and occupational settings. For example, let’s say you worked at a community center for seniors, and you have decided to study the types of people who use the center. The community center’s multiple-choice membership surveys collected when a senior member first arrives might be an appropriate place to start. These are primary data used for community center purposes. But, for your purposes, it is secondary quantitative data. If you want to use

Chapter 1  •  The Science of Secondary Data  

11

TABLE 1.2  ●  Examples of Secondary Quantitative Data Sources Type

Definition

Example

Populationbased

Population-based studies are epidemiology studies in which a defined population is followed up and observed longitudinally to assess exposure and outcome relationships.

United States Census Bureau

Cohort and other longitudinal surveys

A cohort study is a particular form of a longitudinal study (panel study) that samples a cohort (a group of people who share a defining characteristic, typically who experienced a common event in a selected period, such as birth or graduation), performing a cross-section at intervals through time. A cohort study is a panel study, but a panel study is not always a cohort study as individuals in a panel study do not always share a common characteristic.

Framingham Heart Study1

Administrative records

Documents related to organization functions (such as managing the facilities, finances, and personnel) and agreements, contracts, meetings, legal actions, and so on.

Staff and personnel files

Medical records

The terms medical record, health record, and medical chart are used interchangeably to describe the systematic documentation of a single patient’s medical history and care across time within one healthcare provider’s jurisdiction.

Patient medical records

You can access a complete description and details for the Framingham Heart Study at https://www.framinghamheartstudy.org/ 1

these secondary data for a single-method or mixed methods project, you first need to obtain permission to access the data and use it for research purposes. Then you would need to remove the identifying information from the data (if this were not already done before you accessed the data), carefully review the demographic characteristics of the senior members of the community c­ enter, and perform a secondary analysis with those data. Naturally, depending on the size of the sample and the dependent variables you hope to examine, fascinating insight could be acquired from analyzing the surveys from the senior members of the community center. Let’s assume you were recently hired as the social scientist for a community college in your hometown. Your department chair has issued your first assignment: to analyze student demographics across the science, technology,

12  Part I  •  An Introduction to Secondary Data in Mixed Methods Research engineering, and math (STEM) units on campus, including reviewing students’ current academic records, transcripts from their previous schools, and test scores to determine their performance levels. This is primary quantitative data for the school, but it is existing quantitative data for your secondary purposes. The first step is to obtain permission to access the data and use it for research purposes. Then you want to make sure all identifying information has been removed before you begin your analysis. Suppose your goal is to impress the department chair and justify the need for more resources for the school. In that case, you might consider analyzing the secondary quantitative data to demonstrate that math scores for STEM students at the community college have gradually increased over the past two or three years. If you were interested in justifying the need for more funding, for example, providing statistics that frame the status of the STEM students enrolled at the community college and why the funding might benefit them would be valuable. Possibilities for application are promising when you can access comprehensive secondary quantitative data. The same is true about secondary qualitative data.

Secondary Qualitative Data Secondary qualitative (e.g., text, image, etc.) data can provide insight and help you address some of your unanswered qualitative research questions. Historically, secondary qualitative data have not been as popular as their quantitative counterpart. However, between the time I started writing this book and the time I finished it, three books were published on secondary qualitative analysis (Beck, 2019; Hughes & Tarrant, 2020; Largan & Morris, 2019). Some researchers have also tried to generate traction by exploring ways to maximize secondary qualitative analysis across various studies (Bishop 2007, 2009; Fielding & ­Fielding, 2000; Hammersley, 2009; Hinds, Vogel, & Clarke-Steffen, 1997). Given the increased respect, rigor, and credibility of qualitative data in recent years, more scholars are interested in doing secondary analysis of existing data. A deeper appreciation Unfortunately, a deeper apprefor secondary qualitative ciation for secondary qualidata lags a few years behind tative data lags a few years secondary quantitative data. behind secondary quantitative data. This is despite the growing number of resources and guidelines for using secondary qualitative data in research studies (see Beck, 2019; Hughes & Tarrant, 2020; Largan & Morris, 2019). Many possibilities materialize with secondary qualitative data. Table 1.3 illustrates some examples of secondary qualitative data sources you may find helpful. Smith’s (2008) book Using Existing Data in Education and Social Research provides a concise account of the use of secondary qualitative data in social science research. The author states secondary data “…can be numeric or non-numeric. Non-numeric, or qualitative secondary data can include data

Chapter 1  •  The Science of Secondary Data  

13

TABLE 1.3  ●  Examples of Secondary Qualitative Data Sources Type

Definition

Example

Individual interviews

One-on-one interviews between two people, one being the interviewer and the other being the interviewee

A doctor’s initial appointment with a new patient

Focus groups/ group interviews

A discussion by three or more people led by a group moderator, or facilitator, on a specific topic

A group of six new mothers asked to describe their experiences giving birth at a regional hospital

Documents/ text

A written record of information

A student’s school record

Audio recordings

An electronic sound file used to capture activity or human experience

A recorded telephone call made by a telemarketer to a homeowner to solicit interest in switching Internet providers

Video recordings

An electronic video file that is used to capture human experience

A YouTube video of how passers-by respond to a child in distress at the zoo

Photos and images

Pictures used to capture a concept, idea, experience, or interpretation of a human experience

Historical photographs of a town’s first erected buildings, architecture, and topography plans and layout

retrieved secondhand from interviews, ethnographic accounts, documents, photographs, or conversations” (pp. 4–5). Though most secondary data tend to be quantitatively oriented, Smith reminds us of the possibilities of secondary qualitative data in our education and social research endeavors. Let’s say you are a tenure-track assistant professor conducting qualitative interviews to learn more about the experiences of parents whose children are in the foster care system. You hope to understand the biological parents’ knowledge, attitudes, and beliefs about foster care. You begin by working with your ethics board to ensure that interactions with the study participants are handled appropriately, and then you collect the data. You also de-identify the data and make the appropriate ethical considerations for data-sharing, as instructed by your ethics review board. Once the answers to your questions are finalized (and if you were willing), you might consider sharing the de-identified qualitative data with colleagues and students to help answer their related research questions. For example, let’s assume I am a third-year Ph.D. student in your department. I am interested in using your data to learn about the biological parents’ demographics and their beliefs about the foster care system. You might consider sharing your

14  Part I  •  An Introduction to Secondary Data in Mixed Methods Research primary data for my secondary purposes. As a first step toward addressing my research question, I might schedule a meeting to discuss the participants’ demographic information you collected. Though this was not a primary focus of your research, participants might have provided this information in a short survey or during the qualitative interviews, so it would be great to maximize your secondary data to advance my research. Studies that use secondary data are not excluded from an ethics review, so be sure to check with the ethics review committee to learn about the process for undergoing an ethics review for a study that uses existing data for secondary purposes. What if I conducted a primary study in which I video-recorded 32 patientphysician interactions (16 with male patients and 16 with female patients), and I used these data to understand the role that primary care physicians play in addressing their patients’ mental health concerns. Some patients do not go directly to mental health professionals when dealing with mental health conditions. Instead, they may seek help from their primary care physician about the outward effects of mental health conditions manifesting as physical health conditions (e.g., headaches, body aches, insomnia, etc.). Let’s assume I used these data to answer my research questions about the mental health implications of such interactions, and that was all I hoped to do with the videos. You might express interest in using my videos as a secondary qualitative data source to explore the gender dynamics in patientphysician interactions in clinical settings. Again, I should note that specific permissions are needed to access such data since videos of healthcare patients are involved. So an ethics review would be required, and you will need to outline a process for obtaining permission to use the videos for a secondary purpose. Nevertheless, based on the types of questions I included in my primary study protocol, you might be able to glean some interesting information to help you address your research questions. See Table 1.4 for other real-world examples of purpose statements, research questions, and primary and secondary data.

Scientific Developments With Secondary Data Secondary data have practical applications for both single-method and mixed methods research. As you will note throughout this text, while I recommend scholars consider using secondary data for mixed methods studies, I recognize that it is not always obvious when a mixed methods study can benefit from secondary data. Likewise, a more persuasive rationale may be needed for how secondary data are used to acquire knowledge. Many scholars have advocated using secondary data and analyzing secondary quantitative data (Hyman, 1972; Johnston, 2014; Smith, 2008). Rather than overstating the points these earlier scholars made regarding the value of secondary data in single-method studies, here I focus on three scientific developments of secondary data and how they can help you think about expanding your research, be it single-method or mixed methods focused.

Are there sex differences in smoking cessation as demonstrated by the program participants? What are some advantages and disadvantages to counseling for firstgeneration students? Are there racial/ethnic differences in depression symptoms for adults in the U.S.A.? What are the unique experiences of female clergy in the Pentecostal Church?

This study aims to test the Health Belief Model by comparing male and female smokers who are in our smoking cessation program.

The purpose of this qualitative study is to explore the counseling experiences of first-generation first-year students at the University of California, Berkeley.

This study aims to examine the depressive symptoms of African Americans, Caribbean Blacks, and Whites in the United States.

This phenomenological study aims to understand the religious practices of female clergy at three Pentecostal Churches in the city.

Qualitatively oriented; your colleague is a religious scholar who has collected these data from the three churches.

Quantitatively oriented; several national data sources exist, such as the NCSa and the NSALb.

Qualitatively oriented; unique to the first-generation freshmen at the University of California, Berkeley.

Quantitatively oriented; unique to a smoking cessation program.

Study Context

Secondary data (because current, localized data to address this research question already exist)

Secondary data (because current, national data to address the research question already exist)

New data (because this inquiry is unique to the first-generation students at the University of California, Berkeley)

New data (because this inquiry is unique to the participants in this smoking cessation program)

New vs. Secondary Data (Why?)

b

The National Survey of American Life (NSAL) is the first national survey to investigate ethnic differences within the Black population by including significant numbers of Black Caribbeans.

a

The National Comorbidity Survey (NCS) was the first nationally representative survey of the prevalence and correlates of psychiatric disorders and treatment in the United States (1991 and 1992). There are NCS extensions, including a 10-year follow-up of the baseline NCS sample and a replication of the NCS (2001 and 2002).

Research Question

Purpose Statement

TABLE 1.4  ●  Real-World Examples for When to Use New Data Versus Secondary Data

Chapter 1  •  The Science of Secondary Data  

15

16  Part I  •  An Introduction to Secondary Data in Mixed Methods Research

Knowledge Pursuit as a Scientific Development of Secondary Data The first scientific development of secondary data has allowed scholars to pursue knowledge in unique and sophisticated ways. As a health or social science scholar, not only do you have your research program but you are also encouraged to think about the ways that other scholars conduct the same or similar work. There are several benefits to using secondary data for one’s research projects, such as exposure to new and different research foci, extending your understanding of a topic beyond the scope of your geographic area, and your accessibility to the breadth (or depth) provided by learning from new research participants. But one of the most fundamental benefits of secondary data is its ability to challenge you by extending your pursuit of knowledge above and beyond your current methodological boundaries and methods margins. This is your greatest gift from the original research investigators: an opportunity to think critically about a This is your greatest gift research question outside your from the original research usual conceptual and practiinvestigators: an opportunity to cal parameters. Furthermore, think about a research question countless learning opportunities come along with using outside of your usual conceptual secondary data. The ability to and practical parameters. work with data that you did not collect allows you to gain an insider’s view of a complete data set, its strengths and weaknesses, and its ability to address a series of research questions associated with your research interests. This insider’s view of a complete project can be a formative experience for new researchers who aspire to lead their own primary data collection projects someday.

Technology as a Scientific Development of Secondary Data The second scientific development in secondary data is the growth and expansion of technology. Advances in technology have allowed us to streamline our data management during three vital research stages: data collection, data storage, and data-sharing. First, technological advances in data collection have far exceeded those of the researchers who came before us. While early researchers had to depend on face-to-face interactions and the telephone to collect their data, we now have the Internet as a means for collecting both qualitative and quantitative data for our research purposes. Web- and emailbased surveys are frequently used by scholars in the health and social sciences and have resulted in more streamlined data collection methods for studies that require quantitative (e.g., online survey) or qualitative (e.g., video-recorded interview) data.

Chapter 1  •  The Science of Secondary Data  

17

Second, with access to space on the Internet and on the While early researchers had “cloud,” unlimited data storto depend on face-to-face age is a possibility, not only for the Ph.D.-trained scientist who interactions and the telephone has 1,000 data files to store but to collect their data, we now also for the stay-at-home parhave the Internet as a means ent who has 1,000 high-qualfor collecting both qualitative ity photos of their children to store. Simply put, because and quantitative data for our data storage options nowadays research purposes. are maximized or unlimited, “cloud-based” storage has become a way of life for us all, including scholars working with qualitative and quantitative data for research purposes. Finally, data-sharing is much easier now compared to 50 years ago. More people than ever before are storing and managing their data sources on the Internet and devising data-sharing plans so that others have access to their data files. Sophisticated data-sharing plans have become a requirement for federally funded research projects; therefore, there is an expectation to access qualitative and quantitative data collected using federal funds. For example, the National Institutes of Health now require applicants to include data-sharing plans in their grant proposals (https://grants.nih.gov/policy/sharing.htm). Aligned with this, health and social sciences research topics are becoming more complex, and now, these complexities are being shared using technology and data-sharing mediums. An example of this is how some Internet-based scientific journals allow authors to publish supplemental materials to accompany their accepted publications.

Big Data as a Scientific Development of Secondary Data The last of the three scientific developments in secondary data is big data. As its popularity evolves, big data are becoming an essential component of research to improve population health and well-being. But you may be asking yourself, what exactly are big data? Big data are “…data that exceed the processing capacity of conventional database systems” (Lauzon, 2012, p. 4). In other words, “big data” are too big, move too fast, or do not fit the traditional database storage and management systems structures. To gain value from big data, some scholars suggest we choose an alternative way to process it, understand it, and apply it to our world (Dumbill, 2013; Mayer-Schonberger & Cukier, 2013); the same can be said about considering big data as a scientific development in secondary data. In the current scientific discourse, big data are used by different industries, disciplines, and platforms (including business, healthcare, social science, and information technology). The current literature is broad and deep, and while big data are a topic of growing interest, it lacks an agreed-upon definition. For this text, I acknowledge two things: (1) that big data are collected at incredible rates, and (2) that big data users are finding more and more ways to capitalize on their

18  Part I  •  An Introduction to Secondary Data in Mixed Methods Research bigger, faster, and more substantial ambitions for data acquisition, management, and use. Some examples of big data are data found in mobile devices, the Internet, social networking sites, political polling, and entertainment (Brown, Chui, & Manyika, 2011; Heafner, Fitchett, & Knowles, 2016). Big data and mixed methods that use secondary data are similar in their introduction to the research world, acceptance, and application. Because the core contribution of big data is how it can help us perform at a larger scale, it can also make a variety of contributions to how we understand and apply secondary data to mixed methods to answer research questions. Just as big data researchers are seeking to extract more abundant insights and create new areas of foci as they delve deeper into phenomena of interest, so too are mixed methods researchers who are attempting to reap the benefits of secondary qualitative and quantitative data and integrating them so that the strengths outweigh the weaknesses. This will allow us to develop a complete picture of a problem under investigation. The appeal of big data is not necessarily its quantity, but [Big data’s] appeal is not rather how our understanding of the world is shaped and necessarily its quantity, but becomes a direct by-product rather how our understanding of of how we analyze and interthe world is shaped and becomes pret those data. Simply put, a direct by-product of how we to understand big data is to acknowledge the importance analyze and interpret those data. of predictions in our world; big data is not about teaching a computer to think like a person but about applying sophisticated math equations to massive quantities of data so that we can infer probabilities that improve the way we function as a society (Mayer-Schonberger & Cukier, 2013). Thus, big data’s contributions to mixed methods have only just begun. We have not yet scratched the surface of advancing research methods with big data and how they can radically change how we think about secondary data in mixed methods.

Summary This chapter introduced the science of secondary data, described ways to be epistemologically sound when using secondary data, described secondary data types and relevant examples of secondary data, and described three scientific developments in secondary data and how they have shaped knowledge acquisition. The science of secondary data must begin with understanding the distinct differences between primary and secondary data. While primary data are qualitative or quantitative data collected to address a primary research question, secondary data are used for a secondary purpose: to address a new, sometimes related, research question. Now that you have read this chapter, you should be able to describe ways to be

Chapter 1  •  The Science of Secondary Data  

epistemologically sound in your use of secondary data. There are at least three ways to conduct epistemologically sound research with secondary data. First, be realistic about your limited knowledge of the original study and acknowledge that your analysis of their information is really “secondhand.” Second, remember that knowledge is socially and historically located within a complex cultural context. Third, respect the culture of the primary research and acknowledge the power dynamics between the researchers and the people being researched. Secondary data are used to address a secondary research or evaluation question that may or may not be related to the original research goals and questions. Finally, you should be able to name three scientific developments in secondary data and how they have shaped the acquisition of knowledge: the pursuit of knowledge, technology, and the use of big data. Each has contributed in its unique ways, but all have advanced how we use secondary data in single-method and mixed methods research.

Chapter 1 Application Questions 1. What are secondary data, and what are some examples of secondary data in the health and social sciences? 2. What are some ways to be epistemologically sound in your use of secondary data? 3. What are three scientific developments in secondary data, and how have they shaped knowledge acquisition? 4. What are big data and its contributions to mixed methods with secondary data? 5. What are some real-world examples that may require the use of secondary data? 6. How does the pursuit of knowledge influence your decision to use secondary data in your single-method or mixed methods research? 7. Can technology play a role in your research that uses secondary data? If so, how can data collection, storage, and sharing be maximized in your project using technology? 8. What role, if any, does big data have in your research? Can you see big data making an essential contribution to your study as secondary data for either a current or future single-method or mixed methods project? Why or why not?

19

2 Using Secondary Data in Mixed Methods Imagine Your colleague Chris flops down next to you at the staff meeting, startling you. “Oh! I’m sorry,” he sighs. “How have you been?” You can tell by his tone that Chris is in a talkative mood, not a listening mood, so you ask how he is doing. “Man… I got feedback from my program officer about my research project, and she encouraged me to do a mixed methods study. You know I’m a stats guy, so I don’t even know where to begin!” You can see the frustration on Chris’ face, so you listen to him as he begins to express his enthusiasm for the research project; his previous studies, outcomes, and methods; and his confusion about how to proceed. He doesn’t sound excited about the possibility of doing mixed methods for his newest project, so you stop him mid-sentence, “Hey, man, is there any particular reason why you are not seriously considering a mixed methods project?” Chris scoots his chair closer to you, leans in, and whispers, “I don’t know how to do mixed methods. Plus, I was hoping to use some of my pilot data as a launching pad for this next project. Can I even use previous data in a new mixed methods project?” You lean back in your chair and smile, “Of course, you can.” If you want some valuable tips to share with your colleague, Chris, then this chapter is for you.

21

22  Part I  •  An Introduction to Secondary Data in Mixed Methods Research

Learning Objectives This chapter aims to briefly review what, why, and how to use existing data for secondary purposes in mixed methods. By the end of the chapter, you will be able to: 1. Define the knowledge-level continuum and its contributions to mixed methods, 2. Describe what using existing data for secondary purposes in mixed methods involves, 3. Explain why secondary data should be used in mixed methods, 4. Describe examples of how mixed methods can benefit from secondary data, and 5. Describe how to do mixed methods using secondary data as a team, “semi-solo,” or solo effort.

The Knowledge-Level Continuum as a Roadmap for Mixed Methods Though various definitions of mixed methods exist (Johnson, Onwuegbuzie, & Turner, 2007), scholars tend to place their definitions of mixed methods into one of four categories: a philosophy, a methodology, a method, or a community of research practice (Johnson, Onwuegbuzie, & Turner, 2007; Plano Clark & Ivankova, 2016). Philosophies are formal beliefs such as pragmatism, constructivism, and post-positivism (Plano Clark & Ivankova, 2016). When you mix philosophies, you consider that different beliefs and ideas could benefit from being brought together to advance an area of inquiry. Mixing p ­ hilosophies are very different than combining specific research techniques and tools, such as what you would find when mixing methods and mixing methodologies. Note that methods are different from methodologies ­(Hesse-Biber, 2010; ­Watkins & Gioia, 2015), though they are often used interchangeably. Methods are determined by the methodology, reflecting your perspective and philosophical stance. The methodology determines how you should frame the research question and choose the sample and, arguably, whether collecting new data or gathering existing data for secondary purposes will produce the best results ­(Hesse-Biber, 2010; Watkins & Gioia, 2015). Simply put, methods are your tools, and methodology is how you plan to use the tools. A more recent and inclusive way to define mixed methods is as a community of research practice. Plano Clark and Ivankova (2016) define mixing at the community of practice level as having individuals interested in mixed methods and regard themselves as mixed

Chapter 2  •  Using Secondary Data in Mixed Methods  

23

methods researchers, qualitative researchers, or quantitative researchers. These researchers then come together informally or formally to share their beliefs, research agendas, and substantive knowledge about a topic. Creswell defines mixed methods as: An approach to research in the social, behavioral, and health sciences in which the investigator gathers both quantitative (closed-ended) and qualitative (open-ended) data, integrates the two, and then draws interpretations based on the combined strengths of both sets of data to understand research problems. (p. 2) Creswell’s (2015) definition emphasizes mixed methods as a method, and for many budding researchers, it is a more practical way to understand mixed methods. Whether your orientation to mixed methods is a philosophy, a methodology, a method, or a community of research practice, having some sense for your approach to mixed methods is key to understanding the what and why of using existing data for secondary purposes in mixed methods. My approach to mixed methods and thinking about ways to incorporate secondary data into mixed methods is to think about mixed methods in the context of the knowledge-level continuum (see Figure 2.1). In short, I let the knowledge-level continuum guide my decisions to use existing data for secondary purposes in mixed methods. The knowledge-level continuum is a term used in social work (­Grinnell & Unrau, 2018) and anthropology (Vassallo, 1999) research to describe the

Descriptive

• Used for research topics for which some, but not a lot of, information exists • Answers the “what” and “for whom” questions about a research topic

• Used for research topics for which little to no information exists • Answers the “what” questions about a research topic

Source: Based on the framework discussed by Grinnell & Unrau, 2018.

Level of preexisting knowledge on a topic

Explanatory

• Used for research topics for which a lot of information exists • Answers the “what,” “when,” “why,” and “how” questions about a research topic

Exploratory

FIGURE 2.1  ●  The Knowledge-Level Continuum

24  Part I  •  An Introduction to Secondary Data in Mixed Methods Research process for scientific inquiry; all research studies can fall anywhere along the continuum depending on how much is already known about the topic. It encourages you to think about research as being a direct by-product of the information already known about a topic. For example, when little to no knowledge exists about a topic, an exploratory level of inquiry can explore the topic and produce more knowledge about it. Exploratory studies are at the lower end of the knowledge-level continuum not Exploratory studies are at the because they are less imporlower end of the knowledgetant but because there is less level continuum not because preexisting knowledge about the topic. Qualitative methods they are less important but are often used to conduct studbecause there is less preexisting ies with no preexisting knowlknowledge about the topic. edge. If you are studying a virtually unexplored research topic for the first time, you may not know the language used to define the topic. For example, let’s say you are a researcher interested in studying carpal tunnel symptoms among sculptors, and you could not locate any preexisting knowledge on the topic. The first step is to talk with sculptors who have carpal tunnel to learn how they describe their pain. Conducting an exploratory study that uses individual interviews or focus groups to develop a language for these sculptors’ experiences would be an appropriate first step. The next level of the knowledge-level continuum is the descriptive level, which describes the topic of interest in more detail. This can include pursuing both qualitative and quantitative methods of inquiry to deepen your understanding of a topic. For example, using the carpal tunnel example, let’s assume you first conducted an exploratory study with sculptors to develop some language they use to describe their symptoms. Your next step in this line of inquiry may be to conduct a small survey of sculptors who experience carpal tunnel symptoms. This survey would include questions asking respondents to provide more details about their family history, characteristics, health behaviors, profession, lifestyle, and any current treatments they seek to alleviate their symptoms. If you were to distribute this survey nationwide, you might be able to garner the interests of more individuals who engage in different types of artistic expression. The descriptive study would collect more detailed informa[A] descriptive study tion about this group and the would collect more detailed topic of interest. information about this group and The final level is the explanthe topic of interest. atory level, situated at the highest end of the knowledge-level

Chapter 2  •  Using Secondary Data in Mixed Methods  

25

continuum. If you have used your exploratory study to understand the language used by sculptors who experience carpal tunnel and your descriptive study to collect more detailed information about the sculptors’ background, characteristics, and health behaviors, an explanatory study can help you understand the relationships between concepts that you identified in the exploratory and descriptive studies for this topic. With research at the explanatory level, you can examine the causation of these concepts, whether associations exist, and the strength of these associations. Also, this level of research can produce inferential statistics that you may be able to generalize to the larger ­population of artists with symptoms of carpal tunnel (should they exist, of course). Using the knowledge-level continuum as a roadmap to guide your scientific inquiry can be helpWith research at the ful in your mixed methods, explanatory level, you can not only for thinking about the purpose of your study and examine the causation of these your research questions but concepts, whether associations also for the types of methexist, and the strength of these ods you consider using at associations. these various levels of knowledge acquisition ­(Grinnell & Unrau, 2018). Aligned with this is the importance of theory generation and testing in scientific inquiry. Some scholars argue that qualitative research aims to deepen our understanding of a topic and build conceptual frameworks that generate theory. Suppose you embrace this goal of qualitative research. In that case, you can think about applying the knowledge-level continuum as a roadmap for your study. You could begin by using your exploratory level of knowledge to operationalize a language for further inquiry and propose a conceptual framework for how concepts fit together. You could then use your descriptive level of investigation (i.e., the qualitative and quantitative studies) to generate details about your sample, understand the profiles of the people most affected by your topic, and strengthen your understanding of the conceptual linkages between how they operationalize their language of the topic and their lived experiences. Finally, you could use your explanatory level of knowledge to broaden your understanding of the topic and test the associations between variables of the conceptual framework you generated using your exploratory and descriptive studies. You may have noticed that exploratory and explanatory are terms that are also used often in mixed methods (Creswell & Plano Clark, 2018; Watkins, 2017a; Watkins & Gioia, 2015), most notably to describe mixed methods designs (e.g., exploratory sequential design and explanatory sequential design). A part of using the knowledge-level continuum as a roadmap in your mixed methods is knowing the purpose and goal of the core mixed methods designs. Table 2.1

26  Part I  •  An Introduction to Secondary Data in Mixed Methods Research TABLE 2.1  ● Core Mixed Methods Designs and Definitions Design Name

Definition

Convergent

A core mixed methods design where quantitative and qualitative data are collected and analyzed concurrently. Then the findings of the two data sources are interpreted collectively to generate conclusions.

Exploratory sequential

A core mixed methods design where qualitative data are collected and analyzed first, followed by collecting and analyzing quantitative data. Usually, the findings from the qualitative data are used to make decisions about the collection, analysis, and interpretation of the quantitative data.

Explanatory sequential

A core mixed methods design where quantitative data are collected and analyzed first, followed by collecting and analyzing qualitative data. Usually, the findings from the quantitative data are used to make decisions about the collection, analysis, and interpretation of the qualitative data.

provides definitions for the core mixed methods designs that have been cited in previous literature. It is not by chance that the way I have operationalized these terms in my description of the knowledge-level continuum mentioned previously is aligned with how they are also used in mixed methods designs. I hope you can decipher these two sequential mixed methods designs from one another based on what you now know about the knowledge-level continuum. I will refer to the knowledge-level continuum as a roadmap throughout this text, so you can revisit these terms as you move through the chapters.

What Is Required to Do Mixed Methods With Secondary Data? Drawing from our knowledge-level continuum roadmap for scientific inquiry, let’s delve into what mixed methods with secondary data involve. Scholars have gone to great lengths to describe the advantages of mixed methods studies apart from qualitative and quantitative single-method studies (Bazeley, 2017; Creswell & Plano Clark, 2018; Hesse-Biber, 2010; Mertens, 2009, 2017; Morse, 2009). Mixed methods may also vary slightly across disciplines (Curry & Nunez-Smith, 2014; Johnson & Christense, 2013; O’Cathain, 2009) and professions (Andrew & Halcomb, 2009; Haight & Bidwell, 2016; Johnston, 2012; Magee, et al., 2006; Sheperis & Young, 2016; Watkins, 2017a; Watkins & Gioia, 2015). Yet sparse resources provide clear guidelines for mixed methods with secondary data. Mixed methods with secondary data involve identifying, evaluating, and incorporating either one or more existing data sources into one or more components for a mixed methods project. It acknowledges the purpose of

Chapter 2  •  Using Secondary Data in Mixed Methods  

27

mixed methods, which is to collect, analyze, and integrate qualitative and quantitative data in rigorous and theoretically sound ways to encompass the breadth and depth of a phenomenon of interest (Creswell, 2015; Creswell & Plano Clark, 2007; Johnson, Onwuegbuzie, & Turner, 2007; Guetterman, Fetters, & Creswell, 2015; Tashakkori & Teddlie, 2010; Teddlie & Tashakkori, 2008). But it also acknowledges the purpose of secondary data analysis, which is to further analyze Mixed methods with existing data by addressing secondary data involve a research question like (or identifying, evaluating, and distinct from) the original data (Kitchin, 2014; H ­ ewson, incorporating either one or more 2006; Trinh, 2018; Panchenko existing data sources into one or & Samovilova, 2020). more components for a mixed Using secondary data for methods project. your mixed methods puts data in which someone has already invested time, resources, and energy to good use. Your purpose statement, research questions, and theory should guide your decision to use secondary data. How you use theory will vary depending on whether your mixed methods prioritize the quantitative phase or the qualitative phase of the study. For example, if your mixed methods study prioritizes the quantitative phase, the theory will guide your choice of the variables you plan to use. In sequential designs, prioritized phases tend to occur before other phases of the study. Suppose the prioritized quantitative phase involves using a secondary data set. In that case, the theory used by the original investigators will likely guide your understanding and selection of the preexisting variables in those data to help answer your research questions. If your mixed methods study prioritizes the qualitative phase, the theory has a different role. Rather than guide the selection of variables or theory, developing a conceptual framework is the goal of mixed methods studies that prioritize the qualitative phase. You are beginning your mixed methods study with a qualitative phase so that you can build a preliminary, conceptual framework (to test as a theory in subsequent phases of the study). If the prioritized qualitative phase involves using secondary data, you aim to use the secondary data to generate a conceptual framework that will evolve into a future theory. So knowing theory’s role in mixed methods with secondary data is critical, regardless of whether you are trying to build theory (e.g., qualitative analysis) or test theory (e.g., quantitative analysis). Novice mixed methods scholars may seem especially interested in how you can collect and combine two methods, traditionally used in isolation, and then extract from the results insight that enables you to answer your research question. But doing mixed methods with secondary data may be even more mysterious for traditional mixed methods scholars. The idea of doing mixed methods with one or more secondary data sources is valuable for your research

28  Part I  •  An Introduction to Secondary Data in Mixed Methods Research and career trajectory and can help you streamline the logistics associated with your research purpose and answer your research questions. Here, I highlight three reasons you may consider using existing data for secondary purposes in your mixed methods.

Secondary Data in Mixed Methods: Three Reasons Why Ease of use should never drive the decision-making for your research methods and data sources. Instead, you may find that using secondary data in mixed methods can help you achieve your research and career goals. Specifically, using secondary data in mixed methods can help you: (1) examine the potentially untapped possibilities to expand knowledge and understanding of a topic, (2) gauge the depth of a topic so you know how to proceed quantitatively, and (3) gauge the breadth of a topic, so you know how to proceed qualitatively.

Examine the Potentially Untapped Possibilities to Expand Knowledge Many scholars are drawn to secondary data in mixed methods to extend their current methods expertise. You are only one person, and while your passion for your research topic is solid and pursuant, there is only so much you can do and so much data you can collect in your lifetime. Furthermore, you have been trained in a specific way, and, chances are, how you view research problems and solutions are aligned with how you were taught. I note this not as a disadvantage but as an advantage. So essentially, it is healthy for you to face the reality that your education, training, and skills are unique, providing you with an opportunity to view the world through a particular lens. This makes you unique and a valuable contributor to your research area. Similarly, other researchers also contribute to science; only they may be trained in ways that differ from yours. Thus, they will make different contributions to science. Let’s assume that you are a doctoral student at a U.S.-based institution and that you have dedicated the past four years of your education, training, and research to learning about the lived experiences of men who have been diagnosed with breast cancer. Your primary work has involved collecting mixed methods data about the experiences of male breast cancer patients from the time they receive the diagnoses through their treatment. One evening, you come across a recently published paper by an Australian researcher whose body of work includes a 15-year longitudinal study of hundreds of male breast cancer survivors. Would you not want to connect with that researcher to explore possibilities for collaboration? How does the Australian researcher conduct her research? What methods does she employ? Are there cultural differences in how you two pose your research questions, collect your data, and discuss the implications for your work?

Chapter 2  •  Using Secondary Data in Mixed Methods  

29

Suppose you are a scholar who values the inclusion of existing data for secondary purposes in your mixed methods research. In that case, you might even wonder if it would be possible for you and the Australian researcher to build strong collegiality and share your data sources to extend the opportunities for advancing the current way you think about your work. This kind of collaboration and sharing of data sources is not uncommon for some disciplines and professions. Sometimes, researchers from different parts of the world build long-standing careers, maintain their research programs in their geographic area, and then connect with another scholar who may share their ideas or data to maximize the possibilities for understanding the topic and building the science for that research topic.

Gauge the Depth of a Topic, So You Know How to Proceed Quantitatively The knowledge-level continuum suggests qualitative data can be used to build a theory that can subsequently be tested quantitatively (Grinnell & Unrau, 2018; Watkins & Gioia, 2015). Qualitative research uses inductive reasoning, which begins with observations and ends with theory. Therefore, if information from a secondary qualitative data source aligns with the information needed to address your research question, you should consider using these secondary qualitative data as one of the data phases of your mixed methods research. An implied depth is associated with qualitative research; therefore, you can immerse yourself in the data, conduct a rigorous qualitative analysis, and generate a conceptual framework (i.e., preliminary theory) that connects relevant concepts about your phenomenon of interest. Such deep thought about and inquiry into secondary qualitative data can serve as a springboard for building a new quantitative data phase to your mixed methods study. Consider an example involving a secondary qualitative data source to generate a new quantitative investigation. Let’s say your mentor owns a qualitative data source on educational outcomes of court-appointed youth, and you are interested in exploring these data further. Both you and your mentor study this topic, but he collected focus group data three years ago and recently received a Department of Education grant he needs to turn his attention to for the next five years. He offers the data to you to help advance your current scholarship in this area. This use of your mentor’s secondary qualitative data can help you gauge the depth of the educational outcomes of court-appointed youth so that you can know how you want to proceed quantitatively. Namely, you could analyze the focus group data to generate a list of concepts and themes for which a preliminary theory (i.e., conceptual framework) can be created. After which, you could collect quantitative data (including items that measure the concepts and themes generated from your qualitative analysis) to generate hypotheses and then test the conceptual framework. This is a way to enhance the use of your mentor’s secondary qualitative data by repurposing it so that you have some direction in your research.

30  Part I  •  An Introduction to Secondary Data in Mixed Methods Research Both quantitative and qualitative data have advantages and disadvantages, but sometimes, researchers are reluctant to reuse secondary qualitative data. This is unfortunate given the plethora of possibilities for open-ended ­inquiries into the human experience. For example, the benefits of being able to assess behaviors bound by social and cultural contexts; reveal connections, relationships, and subjective processes that result from social phenomena; and holistically uncover the root of motivation and the factors that influence decision-making and opinions (Watkins, 2012) will strengthen your research inquiry, not weaken it. Researchers may hesitate to use (and then reuse) qualitative data sources for several reasons. For example, fewer instructional resources explain maximizing secondary qualitative data (see Beck, 2019; Hughes & T ­ arrant, 2020; Fielding & Fielding, 2000; Heaton, 1998; L ­ argan & Morris, 2019, for exceptions). But barring the anticipated challenges of learning about a data source that you did not collect, the advantages of using secondary Sometimes, researchers are qualitative data far outweigh reluctant to reuse qualitative the disadvantages. Seconddata. This is unfortunate given ary qualitative data can help you gauge the depth of a the plethora of possibilities for topic so that you know how open-ended inquiries into the to proceed quantitatively and human experience. enhance the longevity of the data by reusing it for a different purpose.

Gauge the Breadth of a Topic, So You Know How to Proceed Qualitatively Keeping the knowledge-level continuum in mind, we know that quantitative inquiry uses deductive reasoning to test and confirm (or refute) a theory (Grinnell & Unrau, 2018). Deductive reasoning involves beginning with an established theory and confirming that theory more broadly. Given the sheer number of cases that can be observed using quantitative methods, there are apparent advantages to the secondary analysis of existing quantitative data. One clear advantage is gauging the breadth of a topic before deciding how to proceed qualitatively. For example, let’s assume you are an employee at Planned Parenthood, and your supervisor has asked you to explore possibilities with the organization’s confidential client survey data. You decide to use these secondary quantitative data to understand what a­ dolescents think about the services offered by your Planned Parenthood branch. After you uncover some descriptive information about what adolescents think about the services provided, this information could be strengthened by a small, qualitative study in which you assemble groups of adolescents who represent the demographics reflected in your secondary quantitative analysis and then ask them to expound on the qualitative findings you uncovered.

Chapter 2  •  Using Secondary Data in Mixed Methods  

31

Many resources are invested in collecting primary quantitative data sources for scientific purposes. Time, energy, and human resources are a few examples, and unbeknownst to you, the data may hold the answers to some of your unanswered research questions. While the sense of ownership, control, and responsibility as an independent researcher may make you feel you should collect your own quantitative data for your mixed methods study, I would advise you to see if you can locate secondary data first before deciding to collect new data for your investigation. Having access to an existing study’s protocols, guidelines, training materials, codebooks, and field notes can make you feel like you were part of the original data collection team and streamline the process Unbeknownst to you, the for generating quantitative [secondary] data may hold results to help you proceed the answers to some of your qualitatively in your mixed unanswered research questions. methods study. I discuss this more in Chapter 3.

Examples for How Mixed Methods Could Benefit From Secondary Data With the knowledge-level continuum at its foundation (Grinnell & Unrau, 2018; Watkins, 2017a; Watkins & Gioia, 2015), mixed methods consider inductive and deductive reasoning, individually and collectively, and suggest that a single method may not adequately answer a research question (Creswell, 2015; Creswell & Tashakkori, 2007; Tashakkori & Teddlie, 1998). The beauty of mixed methods is that it allows you to extend your understanding of the data far beyond that which any single-method study can conclude. In other words, you can expound on your definition and understanding of research with mixed methods. What you glean from integrating qualitative and quantitative data will be over and above what you intended to glean from using just one method. Similarly, using existing data for secondary purposes in mixed methods is another way of answering your research questions (Watkins, ­Wharton, Mitchell, Matusko, & Kales, 2017). Mixed methods with secondary data allow you to maximize existing qualitative and quantitative data sources; the time you would use to collect qualitative and quantitative data can be streamlined The beauty of mixed because one or both data sources already exist. methods is that it allows you Consider this scenario: You to extend your understanding of have just defended your mixed the data far beyond that which methods dissertation on the any single method can conclude. role of nurses in providing mental health treatment for

32  Part I  •  An Introduction to Secondary Data in Mixed Methods Research homeless aging populations. You found little preexisting data for your sample of homeless elders, so you collected new qualitative and quantitative data for your dissertation. You have just submitted the dissertation revisions to your committee and are preparing the document for publication so that you can transition to your new postdoctoral fellowship at a different institution. While going through the dissertation, you see a couple of tangential ideas that you did not have a chance to flesh out in the dissertation, but that you would like to explore as a next step in the work. After you have completed the requirements for your doctoral degree and your degree is conferred, you may consider revisiting these tangential ideas to see if you can perform a secondary analysis of the original qualitative or quantitative data (or both). You might even supplement your secondary data with new qualitative or quantitative data from homeless elders in the city where you will be doing your postdoctoral fellowship. The previously mentioned scenario shows how a mixed methods dissertation can evolve into a series of other single-method and mixed methods studies for which the dissertation can serve as the secondary data source. Thus, extending the lifespan of your dissertation advances your mixed methods as you now have opportunities to generate conceptual frameworks you can test quantitatively (e.g., an exploratory sequential design using secondary data) or test hypotheses you can explain qualitatively (e.g., an explanatory sequential design using secondary data). By building on your dissertation research with subsequent studies incorporating your dissertation data into them, the “methods” in mixed methods can be maximized, and the “knowledge” in knowledge acquisition can be refined. Fashioning your trajectory in this way results in research that spans multiple studies and build on your previous work throughout your research career. You may be wondering what kinds of mixed methods projects would benefit from secondary data. I would argue almost any research question in the social, behavioral, and health sciences could benefit from a further study using secondary data. Table 2.2 illustrates some examples of research questions that could be answered using a mixed methods study with secondary data and the role of secondary data within these examples. Though practically any research question could benefit from using secondary data in a mixed methods study, you will need to consider several key characteristics of the data you have access to, should you wish to use them in your mixed methods study. For example, weighing the advantages (such as your ability to maximize secondary data in ways that exceeded the expectations placed on the primary data) and learning about the research processes and methodological decisions made by the original research team, will be necessary. Likewise, disadvantages exist with all research studies. Still, these might look different with research that uses secondary data, such as lacking clarity about the primary data and the processes implemented by the primary research team. Also, suppose you have few analytic skills or are not comfortable using secondary data. This could be another disadvantage delaying or even preventing you from moving forward with your mixed methods study with secondary data.

Chapter 2  •  Using Secondary Data in Mixed Methods  

33

TABLE 2.2  ● Sample Mixed Methods Projects That Could Benefit From Secondary Data Sample Research Question

Role of Secondary Data

Secondary Data Examples

How do classroom seating arrangements influence the learning success of children on the autism spectrum in K through 5 educational settings?

Previous qualitative or quantitative data can offer insight on K through 5 students on the autism spectrum and their classroom experiences. Data can be local, regional, or national.

Qualitative: Interviews with teachers about classroom seating arrangements

What are the gender differences in treatment compliance for myocardial infarction (heart attack) patients six months post-myocardial infarction?

Previous qualitative or quantitative data on treatment options, efforts, or compliance collected from various demographic groups. Data can be local, regional, or national.

Qualitative: Interviews with patients about treatment experiences

What are the social and economic challenges incoming college students face in the United States due to the Deferred Action for Childhood Arrivals (DACA) program?

Previous qualitative or quantitative data on high school and college students in the DACA program. Data can be college-specific, regional, or national.

Qualitative: Focus groups with DACA students

Quantitative: Student test scores

Quantitative: Patient vital data (temperature, blood pressure, etc.)

Quantitative: Scores on social support measures and socioeconomic measures from DACA families

Mixed Methods Solo, Semi-Solo, and With a Team Depending on the project, you may be expected to do mixed methods with secondary data while working alone. For example, for a capstone, theses, or other independent projects, your committee may expect you to work alone to achieve your research goals. Should this occur, you must learn as much as you can about mixed methods by reading, taking classes, and building your toolkit of knowledge on how to conduct the study. When completing mixed methods projects for educational purposes, advisors expect you to reach out for assistance should you have challenges that you cannot troubleshoot independently. This may not be your preferred way of doing research, but sometimes, it must happen this way, to succeed in this educational milestone.

34  Part I  •  An Introduction to Secondary Data in Mixed Methods Research The second option is doing mixed methods with secondary data, not solo, but what I would call “semi-solo.” Dissertations or other medium to large independent mixed methods projects with secondary data often occur “semisolo.” This means a researcher is responsible for completing the work but can seek consultation on a few aspects of the project while it is underway. Sometimes, dissertation committee members encourage students to identify consultants on topics for which the committee members are not experts, be it the research topic or method. For example, suppose a committee member does not know how to advise a student on integrating the qualitative and quantitative data for the dissertation. In that case, they may put the student in touch with a colleague who can offer insight and suggestions. This is an example of doing mixed methods with secondary data “semi-solo.” It involves trudging forward with the mixed methods study but seeking advice and consultation as needed, along the way. The final option is doing mixed methods with secondary data as a team. An example is when students work on their advisor’s research team and do their research under the umbrella of a larger funded research project, which often happens in the health and social sciences. In such situations, students work with secondary data, not solo or semi-solo, but while leaning on the strengths of a larger team to accomplish their research goals. I have also seen colleagues work alongside colleagues in different departments to learn how to use secondary data in mixed methods research. In these cases, it is rewarding to watch research teams assemble based on how some people’s strengths complement the weaknesses of others. Advances in social, behavioral, and health science rarely happen in isolation, and team science is full of advantages that benefit the research team and society. Therefore, if you do not have to do mixed methods with

TABLE 2.3  ● Team Options for Doing Mixed Methods With Secondary Data Option

Definition

Examples

Solo

Doing mixed methods with secondary data and having no advice, consultation, or assistance

Capstone, theses, or other small independent projects

Semisolo

Doing mixed methods with secondary data and having limited advice, consultation, and assistance from one or two people

Dissertations or other medium to large independent projects

Team

Doing mixed methods with secondary data and having advice, consultation, and assistance from three or more people

University, agency, or other large teambased projects

Chapter 2  •  Using Secondary Data in Mixed Methods  

35

secondary data alone, don’t. You may find you can maximize time and resources if you share the responsibilities of a mixed methods study with collaborators. For example, you could share the responsibility of identifying secondary data with another member of your team. With at least two people reviewing secondary data options, you are more likely to locate a data source most applicable to your mixed methods study, and share responsibilities for study conceptualization, data gathering, data analysis, and data integration. Everyone involved can learn new skills together, instead of working solo or semi-solo. However, there are also challenges to working with a research team, such as each team member’s skill level, amount of time they can dedicate to the project, work ethic, and work pace (which can be important with the sequential mixed methods designs). All things considered, sometimes, teamwork can feel less like work if you assemble the right people for your team.

Summary This chapter provided a brief overview of the what, why, and how of using secondary data in mixed methods. Specifically, the chapter sought to define the knowledge-level continuum, a direct by-product of the preexisting information on a topic, the purpose of the research, and your anticipated next steps. It describes the process for all research studies and can serve as a roadmap for mixed methods with secondary data. Now that you have read this chapter, you should know secondary data in mixed methods includes identifying, evaluating, and incorporating either one or more secondary data to serve as one or more components of your mixed methods. The chapter acknowledges the purpose of mixed methods and secondary data analysis, which is to further the investigation of secondary data by addressing a research question like the questions posed in the original study. You now know using secondary data can be advantageous in mixed methods because it provides the chance to expand your knowledge and understanding of a topic; gauge the depth of a topic, so you know how to proceed quantitatively; and gauge the breadth of a topic, so you know how to proceed qualitatively. After reading this chapter, you know mixed methods with secondary data maximize existing qualitative and quantitative data sources. The process is more streamlined because one or both data sources exist. Finally, there are circumstances when you may need to conduct mixed methods with secondary data solo, semi-solo, or with a team.

36  Part I  •  An Introduction to Secondary Data in Mixed Methods Research

Chapter 2 Application Questions 1. What is the knowledge-level continuum, and what are its contributions to mixed methods? 2. What does using secondary data in mixed methods involve? 3. Why should secondary data be used in mixed methods? 4. How can mixed methods be enhanced using secondary data? 5. What are some examples of mixed methods projects that could benefit from secondary data? 6. What are some advantages and disadvantages of doing mixed methods using secondary data solo, “semi-solo,” and with a team?

3 Evaluating Secondary Data for Mixed Methods Imagine You and your officemate, Shirl, are helping a faculty supervisor in your department with a recently funded project. You two have been tasked with evaluating publicly available data sources for their potential use in a new mixed methods project. One day, as you and Shirl sit back-to-back at your desks searching for available data sources to consider, Shirl leans back in her chair, bumping yours. “I’m tired!” she exclaims. “What are we supposed to do with these data sets once we locate them?” “I don’t know,” you respond, not looking away from your computer screen.” “But I’m assuming the boss lady will give us some guidelines for how to judge these data sources before we put them in her new project.” “That makes sense,” Shirl says as she stretches her arms above her head, “…but right now, I would prefer to make a judgment about which type of food we are going to put in our stomachs! I’m starving!” You laugh and agree that you two are long overdue for a lunch break. When you and Shirl return from lunch I imagine your faculty supervisor will be happy to share some guidelines for evaluating the data sources you located. Many of the guidelines you will need are discussed in this chapter.

37

38  Part I  •  An Introduction to Secondary Data in Mixed Methods Research

Learning Objectives This chapter will provide guidelines for evaluating secondary qualitative and quantitative data sources for your mixed methods projects. By the end of the chapter, you will be able to: 1. Understand the life cycle of research data and where secondary data lie on the cycle, 2. Name three considerations for choosing secondary qualitative and quantitative data, 3. List ten evaluation questions when incorporating secondary data into mixed methods, and 4. Apply the ten evaluation questions to a potential data source.

The Life Cycle of Research Data and a Place for Secondary Data Thanks to the increased use of existing data for secondary purposes, we have seen longevity in the life cycle of research data. The life cycle of research data is the schedule of events that occur throughout a project’s life cycle that involve the creation, preservation, and usability of the research data. The Interuniversity Consortium for Political and Social Research (ICPSR) at the University of Michigan publishes a Guide to Social Science Data and Archiving that is “… aimed at those engaged in the cycle of research, from applying for a research grant, through the data collection phase, and ultimately to the preparation of the data for deposit in a public archive” (p. 3). Readers should refer to the guide and Figure 3.1 to learn more about the life cycle of research data (https://www .icpsr.umich.edu/files/deposit/dataprep.pdf). The life cycle of research data has been applied to digital preservation and curation practices (Corti, Van den Eynden, Bishop, & Woollard, 2014). It is maximized when primary researchers develop metadata, or data about data, for secondary purposes. Building and incorporating metadata into a study (e.g., title, description, funding source, data collectors, sample description, data sources, unit of analysis, variables, instruments, codebook, etc.) can prolong the end of a research study. As you can see from Figure 3.1, the primary researchers might prepare their data for secondary use during phases 5 and 6. Once the metadata for your secondary data has been identified, you must determine ways to maximize those data for mixed methods purposes. For more information on metadata, I encourage you to visit the Data Documentation Initiative (DDI, https://ddialliance. org/), which has standards for metadata and recommends these standards be

Chapter 3  •  Evaluating Secondary Data for Mixed Methods  

39

FIGURE 3.1  ●  The Life Cycle of Research Data Phase 1 Proposal Development and Data Management Plan

> Contact archive for advice

Phase 6 Depositing Data

> Create data management plan to ensure long-teerm availability of data resources

Phase 2 Project Start-up > Make decisions about documentation form and content > Conduct pretests and pilot tests of materials and methods

> Complete relevant forms > Comply with dissemination standards and formats

Phase 3 Data Collection and File Creation

Phase 5 Preparing Data for Sharing > Address disclosure risk limitation > Determine file formats to deposit

> Contact archive for advice

Phase 4 Data Analysis > Manage master datasets and work files > Set up appropriate file structures > Back up data and documentation

> Follow best practice > For data, address dataset integrity, variable names, labels, and groups; coding; missing data > For documentation, explore use of DDI standard; include all relevant documentation elements; document constructed variables

Source: Guide to Social Science Data Preparation and Archiving: Best Practice Throughout the Data Life Cycle (6th ed.). (2012). ICPSR, University of Michigan

applied to surveys, interviews, and other observational methods in the social, behavioral, economic, and health sciences. The DDI standards can also help you document and manage different stages in the research data life cycle, including conceptualization, collection, processing, distribution, discovery, and archiving.

40  Part I  •  An Introduction to Secondary Data in Mixed Methods Research

Three Considerations When Selecting Secondary Qualitative and Quantitative Data To maximize the full potential of existing data for secondary purposes, you need systematic processes for evaluating and incorporating secondary data into your mixed methods. Therefore, I begin by offering three considerations when choosing secondary qualitative and quantitative data. I illustrate these considerations using a decision-making flowchart to guide your process for selecting secondary data for mixed methods (Figure 3.2). Begin by asking yourself, “Have I operationalized the terms in my research question?” If your answer is no, then first operationalize (e.g., define) the key terms in your research question before proceeding. If your answer is yes, then ask yourself the second question: “Have I located secondary data that will address my research question?” If you are still searching for secondary data for your study, you need to continue searching or collect new, primary data for your research. If you successfully identify secondary data, then ask yourself the final question: “Can I evaluate the secondary data using the ten review questions from this book?” If so, then you can proceed with using the existing data for secondary purposes, but if not, you should return to question one in the decision flowchart. These three considerations and ten review questions are further described below.

First Consideration: Operationalizing the Terms in Your Research Question When choosing a secondary qualitative or quantitative data source, you should first operationalize the terms of your research question. To operationalize a term means to attach meaning (or a definition) to it so it can be measured. Forgetting to do this is a common mistake made by novice researchers. Still, before beginning any research project, it is important to develop your research question and define its key terms. This is the starting point for the study. For example, if your research question is: “How do Asian and Pacific Islander women respond to the gendered stress in their communities?” the first step is to determine what you mean by “gendered stress.” How will it be defined? Will you align your definition with that from the previous literature? Or will you define it for yourself? Also, will you let the Asian and Pacific Islander women you work with define the terms for themselves? If so, how will you reconcile differences? These are just some considerations when operationalizing the terms To operationalize a term in your research question. means to attach meaning Others include questions like (or a definition) to it so it can be how might you determine the measured. sample? What eligibility criteria do your participants need

Chapter 3  •  Evaluating Secondary Data for Mixed Methods  

41

FIGURE 3.2  ●  Decision Tree for Selecting Secondary Data Sources #1: Have I operationalized the terms in my research question?

No

Yes

#2: Have I located secondary data that will address my research question?

Operationalize the terms

No

Yes

Continue searching, seek advice, or collect new data

Was my search for secondary data successful?

No

Yes

#3: Can I evaluate the secondary data using the ten review questions?

No

Return to #1

Yes

Use the secondary data

to meet to be considered “Asian and Pacific Islander” and, not to mention, “women?” Are transwomen included? So you see, operationalizing the terms in your research question is not always a straightforward step in the process. It involves some additional levels of thinking and decision-making that help guide your methods and study measures. How you operationalize the terms in your research question will determine how to move forward with your mixed methods project and incorporate secondary data into the study design. For example, let’s think back to

42  Part I  •  An Introduction to Secondary Data in Mixed Methods Research the knowledge-level continuum (see Chapter 2). If given the nature of your research question, you plan to prioritize the qualitative component of your mixed methods design (i.e., collect and analyze the qualitative data first), it will be important to include exploratory words in your research question, such as describe, expand, explore, and understand (Creswell, 2009; Creswell & Plano Clark, 2018). On the other hand, if your research question requires that you prioritize the quantitative component of your mixed methods design, your research question should include explanatory words that reflect this, such as association, relationship, examine, contrast, and extrapolate. Once the research question is developed, polished, and operationalized, you can “…apply theoretical knowledge and conceptual skills to utilize secondary data to address the research question” (Johnston, 2014, p. 620). Remember, the goal is not to identify a data source that addresses your overall research interests. Instead, if characteristics of the data source (whether qualitative or quantitative) help uncover a new aspect of your research topic en route to a deeper or broader understanding of the topic, consider operationalizing the terms of your research question so that they can be addressed using the secondary data. For example, if, after operationalizing your research question, you realize it prioritizes the qualitative component of your mixed methods design, then you should try to locate secondary qualitative data that provide deeper insight into one or two aspects of your research question. On the contrary, if your research question prioritizes the quantitative component of your mixed methods design, locate secondary quantitative data that help you generalize (i.e., infer) the findings to a larger population.

Second Consideration: Locating Secondary Data At this point, you might be wondering how do you locate secondary data for a mixed methods study. The answer is simple: You must look for it. You should begin by talking to colleagues and collaborators about the existing data sources they use for secondary purposes in their research. You can also peruse various institutional (i.e., universities) and organizational (i.e., government) resources to look for secondary data that include variables (units of quantitative measure) and concepts (units of qualitative measure) aligned with your research interests. Academic libraries can also be helpful during your search because you can solicit the help of a librarian to maneuver through information expertly. Librarians have experience and skills that will First, be sure the data save you time searching for secondary data and supplesource is appropriate for mental resources. Then, of your research question and topic, course, there is Google (or and to do this, you must evaluate your preferred Internet search the secondary data. engine), where you can use filter features and Boolean

Chapter 3  •  Evaluating Secondary Data for Mixed Methods  

43

connectors (e.g., and, for, in, etc.) to search for keywords and phrases aligned with your interests. For example, placing quotation marks around key terms (e.g., “mental health”) can specify combinations of words and phrases and narrow your search further. Your ability to locate secondary data that aligns with your research question and fits with the design of your mixed method will be necessary for the success of your mixed methods with secondary data. This may seem easy, but you may encounter challenges depending on your access to various federal, state, or institutional data resources. Identifying secondary data and then locating all the information associated with those data is fantastic. Still, you will want to make sure the data are reliable and valid (if quantitative) or credible and dependable (if qualitative; Ulin, Robinson, & T ­ olley, 2005; Watkins & Gioia, 2015). Unfortunately, locating complete and comprehensive information about some data sources may be virtually impossible. For example, the details of some secondary data may be behind the firewall of a government-protected website, or worse, embargoed (e.g., its use has been restricted) due to federal regulations or funding delays and deferrals.

Third Consideration: Evaluating Secondary Data After you have developed a research question to guide your investigation and located secondary data that address the question, your next consideration is your ability to evaluate the secondary data to see if it can help you address your research question. First, you want to be sure the data are appropriate for your research question and topic (Dale et al. 1988; Keicolt & Nathan, 1985; Smith, 2008; Stewart & Kamins, 1993), and to do this, you must evaluate the secondary data. Evaluating a secondary qualitative or quantitative data source for potential inclusion in a mixed methods study requires at least ten steps. Here, I present them in the form of evaluation questions. These evaluation questions are not meant to discourage you from using one secondary data over another. Instead, your goal should be to thoroughly assess how appropriate the secondary data are for your inquiry and to assess how congruent the primary study data are with your intention to use them for a secondary purposes. Your goal with secondary data should be to learn as much as possible about the primary study, from conceptualization to evaluation. You learn everything about secondary data by reading about the primary study. You will need to immerse yourself in most, if not all, of the primary study documentation. Obtain these documents from the primary study investigators, websites, online data repositories, or publications. Read these documents forward and backward. Next, contact the primary investigators and request a phone meeting or videoconference to discuss their original study. Ask them if they would be willing to share study documents that are not publicly available, and in some cases, you may need to make a formal request with the data owners to access the data. You may also need to obtain permission from an ethics review committee before you proceed.

44  Part I  •  An Introduction to Secondary Data in Mixed Methods Research The key to using secondary qualitative or quantitative data effectively is to conduct your very own “goodness of fit” test. The statistical definition of the goodness of fit is the extent to which observed data match the values proposed by a theory (Grinnell & Unrau, 2018). But, here, I offer an expanded definition of the term. The goodness of fit for mixed methods with secondary data is the extent to which the secondary data provides you with new information about your topic of interest. Simply put, secondary data is used to address a research question—or a part of a question—that you may not have otherwise considered. Under such conditions, using secondary data in mixed methods would be in your best interest.

Ten Questions When Evaluating a Secondary Qualitative or Quantitative Data Source Scholars who collect primary data abide by a set of guiding principles for the appropriate conduct of a qualitative or quantitative study. So too do scholars who work with secondary data (Johnston, 2014). Here, I review ten questions that can be used to evaluate secondary qualitative or quantitative data sources (Table 3.1) and that can be used to develop a secondary data evaluation guide. Fetters (2020) recommends creating a data resources table to identify potential data sources to use in mixed methods. After selecting a secondary data source evaluating it using these questions will help you critically examine the quality and utility of the secondary data using a systematic process.

TABLE 3.1  ● Ten Questions When Evaluating Secondary Data for Your Mixed Methods Research   (1) What was the purpose of the original research?   (2) Can you retrieve the data?   (3) How timely are the data?   (4) Who collected the data? How were the data collected?   (5) What types of questions were asked?   (6) How relevant are the data to your research questions?   (7) Do the variables (or concepts) you plan to use match those of the original study?   (8) What are the sampling strategies? Response rates? Missing data?   (9) Are you equipped to handle the analyses? (10) Which component of your mixed methods project will the secondary data help complete?

Chapter 3  •  Evaluating Secondary Data for Mixed Methods  

45

Question 1: What Was the Purpose of the Original Research? You should begin evaluating a secondary qualitative or quantitative data source by performing a thorough appraisal of the original research study. After reviewing some preliminary documentation, can you gauge what the original investigators were trying to achieve by conducting the original study? What was the stated purpose of the study? Were they conducting an omnibus survey that included many topics, one of which happened to be in your area of interest? Or was it an exploratory, qualitative inquiry with several open-ended questions that could assist you in developing some language around a topic that is still in the early stages of research consensus and theory development? Also, note their funding source: Was the work commissioned or supported through a competitive grant award process? As you review the primary study’s purpose, keep in mind that even if a research study is undertaken for purposes contrary to your secondary purpose, you should not exclude it from your list of potential secondary data. For example, secondary qualitative data originally designed to uncover patients’ positive views of the Affordable Care Act may not immediately appear helpful in your study that aims to illuminate the disadvantages of the Affordable Care Act for Americans. Instead, learning about the primary methodology, methods, and procedures used to explore this topic from an angle contrary to your study could help you uncover methods for the next steps in your mixed methods. This information should provide some insight into the purpose of the primary study and whether your secondary purposes are appropriate for the data. Closely reviewing the purpose, aims, research questions, and methods of the original study is essential when evaluating existing data for secondary purposes in mixed methods.

Question 2: Can You Retrieve the Data? After you have determined the purpose of the original data, find out if it is accessible to you by searching across data repositories. Data repositories are a collection of previously collected numeric or text data sets available for secondary use. Often, data repositories are part of larger institutions established for research and archiving purposes to support the data needs of those institutions. Table 3.2 provides examples of data repositories in the health and social sciences. You may need to search data repositories using the Internet to uncover information about the data source of interest. This involves a simple search for the full name of the data source (i.e., not the acronym, if one exists). You can place the full name of the secondary data in quotation marks and see what your search engine finds. The first option is often a primary website for the data source if one exists. Clicking on that URL may take you to more information about the data source. If it is well known (e.g., national, federally funded), you will likely find a website full of helpful

46  Part I  •  An Introduction to Secondary Data in Mixed Methods Research information—such as names of principal investigators, variable names (if quantitative) or concepts (if qualitative), and most importantly, access to some version of the data files. If you can retrieve the data and it is quantitative, files may either be in the raw, manipulated form (e.g., files with CSV, XLS, KML, TXT, and XML extensions), or they may have been cleaned, organized, and placed in a usable format, such as those ready for use in statistical analysis software such as SAS, SPSS, Stata, R, or M-Plus. If the secondary data is retrievable and qualitative, it may be in a format you can easily use—a word processing program file, spreadsheet, PDF, or TXT file. Regardless of how the data are presented when you retrieve them, convert them into a format you can use for your mixed methods purposes. Qualitative, files may come in various forms. For example, files may be video, audio, photos, or text. Sometimes, secondary qualitative data are not ready for analysis when you first access them. After obtaining access, you may need format the data prior to analysis. This may involve transcribing audio or video files or reformatting transcripts for consistency before analyzing them. Qualitative analysis packages like NVivo, Atlas.ti, MAXQDA, and Dedoose are just a few examples of software you can use to analyze existing data for secondary purposes in your mixed methods research. If the secondary data you want to retrieve cannot be easily downloaded from a public database, reach out to the person who maintains the data (e.g., the data manager) for assistance. Often, a contact name, telephone number, or email address will be listed on the website. If no contact person is listed, try to locate contact information for the organization or office and inquire about accessing the data. It is not uncommon data managers to request information from you—your name, institution, contact information, and a description of your plans for data use. You may also be asked to pay a fee for access to the data. Keeping track of secondary data use is important for data repositories; it is also a common way to track the impact of the project and the value of the original resources invested in the primary study.

Question 3: How Timely Are the Data? Data timeliness will be a concern for some researchers, but not for all. Your research topic and plans for dissemination influence the temporal nature of secondary qualitative or quantitative data and how they should be incorporated into mixed methods (Gray & Geraghty, 2020). For example, health and healthcare researchers usually prefer the most up-to-date primary and secondary data, as many health policy decisions are contingent upon the most recent data. So, if you are a health sciences researcher, try to include the most recent secondary qualitative and quantitative data in your mixed methods. Health researchers tend to value studies based on recent data more than those based on data collected 10 or 20 years ago. For example, data on contraception in the 1970s might not be relevant to your research unless you are studying the history of contraception.

Chapter 3  •  Evaluating Secondary Data for Mixed Methods  

47

TABLE 3.2  ● Sample Data Repositories in the Health and Social Sciences Data Repository

Description

Data Type

Inter-university Consortium for Political and Social Research (ICPSR)

ICPSR is an online data archive provided by the Institute for Social Research at the University of Michigan. With the help of over 700 academic institutions and research organizations, ICPSR has over 500,000 data files relating to social science fields, including education, aging, criminal justice, substance abuse, and terrorism. (https://www.icpsr .umich.edu/icpsrweb/)

Quantitative and qualitative

United Kingdom (U.K.) Data Service

The U.K. Data Service provides the United Kingdom’s most extensive collection of data from the U.K. on international social, economic, and population topics. It is funded by the Economic and Social Research Council, including both qualitative and quantitative data. (https://ukdataservice .ac.uk/)

Quantitative and qualitative

The Timescapes Archive

The Timescapes Archive specializes in longitudinal qualitative data and complements the U.K. Data Archive. The Archive is designed to enable the sharing and reuse of data sets that have been generated using Qualitative Longitudinal Research methods. (https://timescapesarchive.leeds.ac.uk/)

Qualitative

Qualitative Data Repository

The repository develops and publicizes common standards and methodologically informed practices for these activities and reuses and cites qualitative data. Four beliefs underpin the repository’s mission: data that can be shared and reused; evidence-based claims should be made transparently; the use of well-documented data enriches teaching; and rigorous social science requires common understandings of its research methods. (https://qdr.syr.edu/)

Qualitative

(Continued)

48  Part I  •  An Introduction to Secondary Data in Mixed Methods Research TABLE 3.2  ● (Continued) U.S. Census Bureau (DataFerrett)

The United States Census Bureau provides dozens of government surveys, including the American Community Survey (ACS), the Decennial Census of Population and Housing (1990 and 2000 available), the National Health and Nutrition Examination Survey (NHANES), and the Survey of Income and Program Participation. (https://dataferrett .census.gov/)

Quantitative

Common Core of Data (CCD)

The CCD is a U.S. Department of Education program that collects data yearly on all public schools, public school districts, and state education agencies in the United States. The data are supplied by state officials and include information that describes schools and school districts, demographics on students and staff, and revenues and expenditures. (https:// nces.ed.gov/ccd/)

Quantitative

Panel Study of Income Dynamics (PSID)

The PSID has been collected since 1968 with a nationally representative sample of individuals and their descendants, including data covering employment, income, wealth, expenditures, health, marriage, childbearing, child development, philanthropy, education, and numerous other topics. (https:// psidonline.isr.umich.edu/)

Quantitative

In the previous example, healthcare data from the 1970s might not be suitable if you are studying how today’s health insurance companies support family planning. This is because your understanding of more recent decisions around family planning and health insurance is more of a contemporary issue and may not benefit from older, outdated information. On the other hand, if your research focuses on how the political climate has evolved since the Civil Rights movement in the United States, planning to conduct a secondary analysis of qualitative or quantitative data collected in the 1950s is appropriate. If you elect not to use data from experts might question your methods, not to mention the relevance and authenticity of your work. In short, the timeliness of your secondary qualitative and

Chapter 3  •  Evaluating Secondary Data for Mixed Methods  

49

quantitative data depends on your topic of study and the audience for your study findings.

Question 4: Who Collected the Data? How Were They Collected? When you make use of secondary data, it is helpful to know who collected the data. This information can help you understand how to manage and analyze the data. For instance, survey data are often collected via faceto-face interviews, over the phone, the Internet (i.e., online surveys), and through the mail. The people hired to collect the data may be motivated by the desire to complete a certain number of interviews, which might directly impact their compensation. On the other hand, if the lead investigators conducted the interviews themselves, they may be motivated by the desire to achieve more profound scientific rigor and precision. This is not to say that lead investigators are better than hired, trained interviewers. Instead, the motivation for conducting interviews might differ, depending on who is responsible for the data collection. As the researcher interested in using the secondary data for your mixed methods study, you simply need to know who collected the data so that you can interpret the data in the context of the source (Smith, 2008). If you have ever conducted a research study from beginning to end, you know that significant time and resources are invested in data collection. You also understand how data are collected and that not all qualitative or quantitative data are collected in the same way. Data collection strategies are usually determined based on the sample under study, the methodology, and the data type needed to answer the research question. For example, a qualitative study that aims to delve into a highly sensitive topic should avoid focus groups as a data collection tool. If the topic is sensitive, stigmatizing, or shameful to focus group participants, they may be less likely to discuss the topic for fear of being judged, condemned, or embarrassed. Therefore, if the secondary data you access is a qualitative study of a sensitive topic that used focus groups, closely evaluate its usefulness for your secondary analysis.

Question 5: What Types of Questions Were Asked? For secondary data users, the types of questions posed in the original interview guide can influence your use of the data for a secondary purpose. For example, suppose the secondary data are from a quantitative source. In that case, you can look through the questionnaire used to interview study participants for items from the primary data that align with your research question. Attend to how questions were asked, the order of questions, and response choices for each item. Also, note the levels of measurement for questions that pique your interest. From previous research methods classes, you may recall

50  Part I  •  An Introduction to Secondary Data in Mixed Methods Research that there are four levels of measurement—nominal, ordinal, interval, and ratio—and that statistical analyses are determined based on the types of survey questions you want to study and the level of measurement of the response options. When defining terms, be sure the secondary data you wish to use operationalizes variables of interest in ways aligned with how you plan to operationalize those variables. These operationalized definitions are often found in the documents accompanying the secondary data. Also, consider the variables: Do they match? An imperfect match is not a significant problem, as White and Smith note, “[O]ur concerns about the match between research questions and variables were explicitly documented in reports emerging from this project and our warrant paid due regard to its limitations” (White & Smith, 2005, p. 66). I discuss this more in Chapters 4–7, where I present mixed methods designs and discuss how to incorporate secondary data into them. If the secondary data are qualitative, you will still need to pay close attention to the interview questions asked in the protocol. Attend to whether the questions are structured, semi-structured, or unstructured and observe the response patterns. You are at the mercy of question structure, the interviewer (or the focus group moderator), particularly with secondary qualitative data. Secondary quantitative data may be structured and consistent across many cases. However, despite interview guide and question order consistency, secondary qualitative data is qualitative: Different respondents interpret questions differently, and as a result, the variability in responses may be wider than those in a secondary quantitative study.

Question 6: How Relevant Are the Data to Your Research Questions? Once you have developed your research questions, retrieved the data, and studied the data collection methods, your next step is to examine how relevant the data are to your research questions. This is an important step in secondary data use, and it should not be taken lightly. However, while evaluating secondary data, it is important to maintain the authenticity and commitment to your original research interests. In other words, do not force your ideas onto those of the secondary data. There is nothing to gain from designing a research question, reviewing the questionnaire from secondary data, and then forcing the meaning of the questions from the secondary data onto your research questions. It is tempting to make things work by loosely defining your research program and searching for a secondary data set that works for you. But when using secondary data in your mixed methods, you should not compromise the quality of your inquiry for the convenience of not needing to collect primary data. The purpose of using secondary data is stay true to your original research question. If the secondary data are not a perfect match for your question, explore whether and how using the data will further your overall understanding of the topic and help craft the next inquiry phase for your overall research program.

Chapter 3  •  Evaluating Secondary Data for Mixed Methods  

51

Question 7: Do The Variables (or Concepts) Match Those of the Original Study? When evaluating secondary quantitative (or qualitative) data sources, it takes time to discern what the variables (or concepts) from the original study represent, how they were used in the original study, and how they can help you address your research question. So, though using secondary data may reduce the overall time needed to complete the project, do not underestimate the time required for deciding which documents and supplementary data files merit inclusion, assist with retrieving variables (or concepts), and clarify merging data sources (if needed). Plan for these steps before the analysis. Depending on the data source, some of the preliminary, descriptive results you need may already exist, and you may not need to run these analyses in the secondary data.

Question 8: What Are the Sampling Strategies? Response Rates? Missing Data? Once you have answered the evaluation questions and are confident that the secondary data you are evaluating is a good fit for your mixed methods study, you should evaluate how the samples were drawn and if they are sufficiently representative (quantitative) or if saturation was achieved (qualitative) before proceeding. With secondary quantitative data sources, large-scale surveys tend to use more rigorous sampling strategies to ensure that data are representative of the population. So data users can generalize the findings from the samples to the larger population. Despite these attempts, not all secondary quantitative data sources achieve this goal. Search the technical manuals and handbooks associated with the secondary data of interest to acquire information about the data collection procedures. Note if the original investigators report response rates to their original survey and if there are missing data. As a secondary analyst, you will need to decide whether response rates are sufficiently robust to enable inclusion in your mixed methods. Sampling strategies are also important when working with secondary qualitative data sources, so it is important to know how the original investigators recruited participants. Did they hang flyers in designated areas? Did they use a convenience sample? Was saturation achieved? These factors will contribute to your level of understanding of the richness and utility of the secondary qualitative data and whether they will be helpful to you as you incorporate these data into your mixed methods. There are no perfect data sources in research. So study the data mechanics, the advantages, and the disadvantages of the data you consider for your mixed methods. Ask yourself: Have any groups been excluded from the data collection process? Were any groups oversampled? Did this result in large numbers of people or not? While it is not unusual to learn that some aspects of the population are missing from data sources you are evaluating, it will be ideal if you can address the

52  Part I  •  An Introduction to Secondary Data in Mixed Methods Research omission of these individuals in your mixed methods study and propose ways to include these individuals in future data collection efforts.

Question 9: Are You Equipped to Handle the Analysis? When evaluating secondary qualitative or quantitative data to include in your mixed methods research, ask yourself if you are equipped to handle the analysis and integration of the data sources. Whether you are integrating two secondary data sources or one new and one secondary, it will be important to assess your level of training, skills, experience, and comfort in managing the data. The absence of one or more skills should not deter you from performing the analysis. On the contrary, I hope that your research question is so important that you are relentless in your pursuit of answers. Still, do not proceed with false hope and confidence that leads to a poorly planned and executed study. Instead, invest time and energy into securing support, training, and consultation to achieve a successful project. If you do not have, or cannot develop, the skills needed to conduct a mixed methods study using secondary data, collaborate with someone who does. If your secondary data is quantitative, ensure that you or someone from your team has quantitative data analysis experience. This person should be aware of the variables involved in the secondary quantitative analysis and the strengths and limitations of the secondary data. Knowing which descriptive analyses to run will provide clues that help you determine which inferential analyses will lead to the most worthwhile results. In the business of hypothesis testing, the ability to support or refute hypotheses is a platform on which entire careers are built. Approaching this work carelessly may result in questionable study findings. Similarly, if you are testing hypotheses, you should pay close attention to your study’s power and be sure you can determine the appropriate sample size required to detect an effect size with which you have confidence. If your secondary data are qualitative, you need different skills and experiences. For example, rather than meeting assumptions for testing hypotheses, you or someone from your team will need to know how to build theory using qualitative methodology and methods. Here, it is key to understand qualitative methodologies—grounded theory, phenomenology, and ethnography—and decide whether these or another type of inquiry will address the qualitative research questions you plan to explore. To conduct a rich, thorough qualitative analysis, you do not need to be a degree-holding anthropologist or sociologist. But, if you have not done the reading, received the training, and developed the confidence to undertake a high-quality study, recruit a team member who does. For example, knowing how to evolve from your transcripts to a reduced version of the data presented as quotes and then extracting ideas and concepts from those quotes that form the bases for your overarching themes will help you generate the most valuable findings from your qualitative inquiry.

Chapter 3  •  Evaluating Secondary Data for Mixed Methods  

53

Question 10: Which Component of Your Mixed Methods Project Will the Secondary Data Help Complete? Perhaps you know which component of your mixed methods study can benefit from secondary data. If this is the case, you must understand the role of the secondary data and favorably answer the previous nine evaluation questions in this chapter. Reviewing these features will help you determine whether you can use the secondary data for one or both phases of your mixed methods research. Specifically, the type of secondary data you gather helps determine their placement in your study. For example, if you retrieve qualitative data, you will decide whether you want to design a study that begins with qualitative inquiry and is followed by a quantitative investigation (i.e., exploratory sequential). However, if you retrieved quantitative data, you will decide whether analyze quantitative data and then follow up with a qualitative inquiry (i.e., explanatory sequential). Still, your research question may require both types of data to be analyzed simultaneously (i.e., convergent). In Chapters 4–7, I will introduce key points about the type, pacing, and placement of your secondary data and more closely examine how the data type influences the timing, pacing, and placement of the data in your mixed methods study.

Applying the Ten Questions to Secondary Data Now that I have covered three considerations for choosing secondary qualitative and quantitative data sources and reviewed ten questions you can use when evaluating secondary data for your mixed methods, let’s see how they work. Let’s refer back to Table 3.2, which includes sample secondary data. Here, I will review a sample data source using the ten evaluation questions. Table 3.2 includes both secondary qualitative and secondary quantitative data and the web links to the respective data sources. Let’s select the first one on the list, ICPSR. Then, Let’s choose the General Social Survey (GSS; http://gss.norc.org/), which contains a standard core of demographic, behavioral, and attitudinal questions, plus topics of particular interest. Let’s say I want to look specifically at the 2016 version of the GSS because it is merged data and includes a cultural module. First, I would create a table that looks a lot like Table 3.3. Then I would review all the information on the ICPSR and the GSS websites about GSS and fill in the responses to the evaluation questions about the utility of the GSS for my purposes. Keep in mind that these tables are for you and your research team, so try to be as detailed and comprehensive in your answers to the ten evaluation questions as you possibly can. Completing this table in its entirety, just as I have in Table 3.3, will be helpful in the research process as you refer to the GSS and review why you wanted to include this data source in your mixed methods with secondary data.

54  Part I  •  An Introduction to Secondary Data in Mixed Methods Research TABLE 3.3  ● Applying the Evaluation Questions to a Sample Secondary Data Source Data Source:

General Social Survey (GSS)

Question #

Evaluation Questions

Your Response

1

What was the purpose of the original research?

Since 1994, the GSS changed to conducting biannual surveys to gather data on contemporary American society to monitor and explain trends and constants in attitudes, behaviors, and attributes.

2

Can you retrieve the data?

Yes. The GSS website takes me directly to the data.

3

How timely are the data?

The most recent wave of data (2016) is available.

4

Who collected the data? How were the data collected?

The basic GSS design is a repeated cross-sectional survey of a nationally representative sample of noninstitutionalized adults who speak English or Spanish. The preferred interview mode is in person; however, a few interviews are done over the telephone.

5

What types of questions were asked?

Demographic, behavioral, and attitudinal questions, plus topics of particular interest. Among the topics covered are civil liberties, crime and violence, intergroup tolerance, morality, national spending priorities, psychological wellbeing, social mobility, and stress and traumatic events.

6

How relevant are the data to your research questions?

Very relevant. My research questions explore mental health outcomes among racial and ethnic groups.

7

Do the variables (or concepts) you plan to use match those of the original study?

Yes. I have an interest in mental health outcomes more broadly and suicide measures. The GSS variables will allow me to examine both.

8

What are the sampling strategies? Response rates? Missing data?

GSS participation rates are approximately 70%; the 2016 response rate of 61% was 8 points below the participation rate for 2014.

Chapter 3  •  Evaluating Secondary Data for Mixed Methods  

Data Source:

General Social Survey (GSS)

Question #

Evaluation Questions

Your Response

 9

Are you equipped to handle the analyses?

Yes. I get assistance from my research team.

10

Which component of your mixed methods project will the secondary data help complete?

The GSS will assist me with the first phase of my explanatory sequential design.

Summary This chapter provided guidelines for identifying and evaluating secondary qualitative and quantitative data for mixed methods. Now that you have read this chapter, you should be able to describe three considerations for choosing secondary qualitative and quantitative data. How you operationalize your research question and locate and evaluate secondary data to evaluate their appropriateness for your mixed methods are key elements to mixed methods with secondary data. The ten evaluation questions for assessing the utility of secondary qualitative and quantitative data include: (1) What was the purpose of the original research? (2) Can you retrieve the data? (3) How timely are the data? (4) Who collected the data? How were the data collected? (5) What types of questions were asked? (6) How relevant are the data to your research questions? (7) Do the variables (or concepts) you plan to use match those of the original study? (8) What are the sampling strategies? Response rates? Missing data? (9) Are you equipped to handle the analyses? (10) Which phase of your mixed methods project will the secondary data help you complete? Your ability to critically evaluate ­secondary qualitative and quantitative data sources is essential to addressing your mixed methods research questions.

55

56  Part I  •  An Introduction to Secondary Data in Mixed Methods Research

Chapter 3 Application Questions 1. Identify a secondary data source in your field of study and determine where it currently lies on the life cycle of research data. 2. What are the three considerations for choosing secondary qualitative and quantitative data? How do these considerations help you prepare to incorporate secondary data into mixed methods? 3. List the ten evaluation questions you can use when incorporating secondary data into your mixed methods. 4. What do the ten evaluation questions help you achieve?

PART II

Designing and Conducting Mixed Methods With Secondary Data

57

4 Convergent Design With Secondary Data Imagine Your cousin, Quinn, just landed a research associate position at his dream institution. So he feels lucky to have this great opportunity. What makes this opportunity a dream come true is that the two leading scholars in Quinn’s field are at the institution. One of the leading scholars is an ethnographer whose decade-long research has established language and descriptions for how the experiences of Quinn’s population of interest have evolved over time. The other scholar is a statistician who, using complex sample survey data, has documented the trends of Quinn’s research topic over time. Both scholars have different data sources from which they cannot publish due to administrative responsibilities. They are both enthusiastic about working with Quinn because he will be able to use their secondary data and (hopefully) publish from it. As you are catching up with Quinn via text one evening, you realize his research questions may be addressed by combining both data sources, concurrently. If he expresses an interest in mixing the two data sources using a convergent design, then you might want to share this chapter with Quinn.

59

60  Part II  •  Designing and Conducting Mixed Methods With Secondary Data

Learning Objectives This chapter reviews the convergent design and discusses how secondary data can be used for one or both study components. By the end of the chapter, you will be able to: 1. Define the features of a convergent design; 2. Describe how to bring secondary data into a convergent design; 3. Analyze and integrate secondary data for a convergent design; 4. Plan and implement a convergent design with secondary quantitative data, secondary qualitative data, and secondary qualitative and quantitative data; and 5. Identify potential challenges (and solutions) when using secondary data in a convergent design.

Features of the Convergent Design Highly regarded as the most popular core mixed methods design (­Creswell, 2015), the convergent design is a beginner-friendly mixed methods design for researchers who want to launch their mixed methods journeys in familiar territory. The familiarity of collecting and analyzing qualitative and quantitative data separately and then interpreting them jointly makes the ­convergent design popular among novice mixed methods researchers. Combining qualitative and quantitative data for the convergent design occurs by merging the results during the study’s data analysis and interpretation stage (Creswell & Plano Clark, 2018; Morse & Niehaus, 2009; Teddlie & Tashakkori, 2009; ­Watkins & Gioia, 2015). You can review the references above for more information about the convergent design. Just as beginning mixed methods researchers gravitate toward the convergent design, so do researchers who want to use secondary data in their mixed methods studies. This is because the convergent design (see Figure 4.1) involves collecting qualitative and quantitative data, often simultaneously, but not always, then using both data sources equally to address the research question. One data source is not dependent on the other; therefore, each data source (be it new or secondary) is handled separately, and the two sets of results are integrated at the end of the study. Researchers use the convergent design when they want a more comprehensive understanding of the qualitative data separate from the quantitative data, to corroborate results from different methods, or to compare multiple levels within a single system (Creswell & Plano Clark, 2018).

Chapter 4  •  Convergent Design With Secondary Data  

61

Timing (also called “pacing”) is the temporal relationFor novice researchers, the ship between the quantitative familiarity of the convergent and qualitative components for a mixed methods study design may translate into (Morse, 1991; Creswell & comfort with a convergent Plano Clark, 2018). Previous design that involves secondary scholars describe the timing data. of mixed methods research, so I will not provide those details here. But, essentially, it involves when time the data are collected and the order in which the researchers use the results from two sets of data within a single study (think “convergent” vs. “sequential”). Timing relates to all the quantitative and qualitative components, not just new data collection but also secondary data used in a mixed methods study. Therefore, it for a convergent design with secondary data, the data analysis and integration will occur at the end of the study. It is important to note that there could be any considerable time differences between when both data sets are collected, if one or both are secondary data and especially if either of them were collected for a longitudinal project.

Bringing Secondary Data Into the Convergent Design If you need some convincing, there are advantages to using secondary data in your convergent design. For example, while a traditional convergent design requires you to collect new qualitative and quantitative data, a convergent design with secondary data boasts money- and time-saving benefits when it comes to assembling a team to collect the data, incentivizing study participants, and cleaning and managing the data. For example, let’s say you decide to use two secondary data sources for your convergent design. Given the nature of the design, the data are not integrated until the data analysis and interpretation stage. So, if you evaluate a secondary qualitative data source (see Chapter 3 of this book) and it turns out those data are not a good fit for your study, you can review and evaluate a different qualitative data source without interrupting the work you plan to do with the secondary quantitative data. Building secondary data into a convergent design also presents opportunities to work with an experienced team for one or both study components. For example, let us say you decide to collaborate with a survey research team on a data set they collected two months ago. While they are still cleaning their data, you can develop a new instrument for the qualitative component of the convergent design. Finally, using secondary data in a convergent design can offer new ideas for concepts you had not considered while developing your initial purpose statement and research question. Training can sometimes limit

62  Part II  •  Designing and Conducting Mixed Methods With Secondary Data FIGURE 4.1  ● Convergent Design With New or Secondary Data

Formulate a Research Question

Select secondary (or collect new) QUANTITATIVE data

Select secondary (or collect new) QUALITATIVE data

Review secondary QUANTITATIVE data using evaluative criteria (or plan sampling, measures/instruments, and procedures for new QUANTITATIVE analysis)

Review secondary QUALITATIVE data using evaluative criteria (or plan sampling, measures/instruments, and procedures for new QUALITATIVE analysis)

Analyze secondary (or new) QUALITATIVE data

Analyze secondary (or new) QUANTITATIVE data

Produce QUANTITATIVE Results

Integrate QUALITATIVE/ QUANTITATIVE Results

Produce QUALITATIVE Results

Generate findings and overall conclusions

our thinking, so having access to a secondary data source for perusal can give us new ideas about the problem under investigation. Unlike sequential designs (which I cover in Chapters 5 and 6), where the results from the first data source influence the decisions you make about the second data source, with a convergent design, the two data sources remain separate until the final stage when the results are compared/merged to develop integrated conclusions for the study. If you use two secondary data and decide to swap the qualitative source out for a different one, this may not have a major impact on the secondary quantitative data. Though, naturally, if you select different data for one of your secondary sources, some decisions for analyzing the other data source might need to change. My point is only to acknowledge that using the previous example, you will not need to complete the qualitative study component from

Chapter 4  •  Convergent Design With Secondary Data  

63

BIG DATA BREAK 4.1. METADATA IN CONVERGENT DESIGNS WITH SECONDARY DATA There are opportunities to benefit from all big data have to offer. One way of doing this in mixed methods research is to closely review the various definitions, mappings, and other characteristics used to describe how to find, access, and use big data components. These characteristics are called metadata. Reviewing secondary metadata for your big data is ideal because it can help you learn more about the qualitative and quantitative study components before analyzing and integrating them. If a metadata catalog does not exist, it would be worth your time to create one before integrating big data into your mixed methods study. By nature, the convergent design allows for a cleaner evaluation of metadata. The quantitative and qualitative components are separate and do not mix until the analysis and integration stages. So building a catalog of a big data source during the earlier stages of your convergent design can be advantageous as you proceed through the subsequent stages of your study.

beginning to end before realizing the data are not the best fit for your study. This gives you time to review and then choose choose the most appropriate data (new or secondary) to address the research question for your convergent design.

Preparing Secondary Data for a Convergent Design The procedures for designing and implementing single-method qualitative studies (Silverman, 2011; Tolley, Ulin, Mack, Robinson, & Succop, 2016; W ­ atkins, 2012) and single-method quantitative studies (Onwuegbuzie, Leech, & Whitcome, 2008; Ross & Onwuegbuzie, 2015) to prepare oneself for a mixed methods study are outlined in several resources. Rather than covering these here, I will offer ten steps for incorporating secondary data into a convergent mixed methods design. When planning for a convergent design with secondary quantitative or qualitative data, it will be necessary to (1) gather preliminary information about the secondary data to assess topic fit and alignment; (2) gain access to the secondary data; (3) review the study files to learn about the secondary data and the intent of the original researchers; (4) take inventory of the secondary data and assess what is available, missing, incomplete, and so on; (5) check the data for study fit and appropriateness for addressing the research question; (6) reorganize (or restructure) the data for analysis; (7) develop an analysis plan for the secondary data; (8) analyze the secondary data; (9) generate results from the analysis; and (10) integrate the results from the two study components. These ten steps, and their relevance to the convergent mixed methods design, are covered explicitly in Table 4.1. Review these steps carefully as you prepare your

64  Part II  •  Designing and Conducting Mixed Methods With Secondary Data TABLE 4.1  ● Preparing Secondary Quantitative and Qualitative Data for a Convergent Design

Step

Quantitative and Qualitative Data Preparation Steps

Relevance to the Convergent design

1

Gather preliminary information about the secondary data to assess topic fit and alignment

You can save time by gathering preliminary information (e.g., summaries, published papers, websites, etc.) about secondary data sources before deciding if you will use them for your convergent design. Sometimes, the information from secondary studies is incomplete or nonexistent, so plan accordingly for the necessary time to review study information and make decisions.

2

Gain access to the secondary data

With a convergent design, the timing of your use of either data source is not contingent upon the other data source. However, you should have access to complete data files before deciding to use one or both in your convergent design. Complete data files might include a statistical spreadsheet or database for the quantitative data and interview transcripts for the qualitative data.

3

Review study files to learn about the secondary data and intent of the original researchers

Try to understand how the original researchers made decisions about their methods. A close review of their study files may involve summarizing each file, what it contains, and how it can help your convergent design.

4

Take inventory of the secondary data and assess what is available, missing, and incomplete

Secondary information on the data may be available but incomplete. A data inventory can help you make decisions about the secondary data and whether it is necessary to collect new qualitative or quantitative data. Develop a log for the secondary data, if/how the original researchers used them, and if/how you will use them for your convergent design. Contact the original researchers, if possible, for more information.

5

Check the data for study fit and appropriateness for addressing the research question

Once the secondary data are inventoried, it is easier to assess their ability to address the mixed methods study purpose and research question. The convergent design allows for flexibility in how the two data sources complement one another (i.e., decisions about both studies may occur simultaneously and may not be contingent upon completing one component before moving on to the next).

Chapter 4  •  Convergent Design With Secondary Data  

65

6

Reorganize (or restructure) the data for analysis

Restructuring privileges the secondary data as an important component of the study and acknowledges its utility apart from the new data. Focus on restructuring the secondary data in a way where you can best address the overall study’s purpose and research question.

7

Develop an analysis plan for the secondary data

Focus on operationalizing the secondary variables and text data to address your overall study’s purpose and research question. The analysis plan for the secondary data may be complementary or contradictory to the plan proposed by the original researchers. This step should be reciprocated with the data reorganizing in step 6.

8

Analyze the secondary data

The two data components for a convergent design function are separate; therefore, single-method quantitative and qualitative analysis techniques can produce whether the data are new or secondary.

9

Generate results from the analysis

This step involves looking at the results from each study component, determining what each contributes to the study purpose, and research question.

10

Integrate results from the two study components

The final step is integrating the two sets of results. This involves taking step 9 further by determining what the individual and collective results (i.e., quantitative and qualitative) contribute to the study purpose and research question. Integration may result in findings that converge or diverge from one another. Regardless, both data sources add insight otherwise unknown.

secondary data. Remember, in a convergent design, the two data sources are independent during the earlier (e.g., data collection and data analysis) stages of the study. Moreover, my ten steps should be interpreted in light of the fact that your data collection and analysis steps may occur simultaneously. Regardless, the two sources, be they new or secondary, will be integrated during the convergent design’s data analysis and interpretation stage.

Data Analysis and Integration for a Convergent Design With Secondary Data Convergent designs involve completing both single-method studies from beginning to end and then integrating the data during the analysis and interpretation stage. This section describes how to analyze and integrate data for convergent designs with secondary data. The data analysis and data integration steps may look different for the convergent design compared to other mixed methods

66  Part II  •  Designing and Conducting Mixed Methods With Secondary Data designs. So, here, I offer considerations when generating results and interpreting the findings for convergent designs where one or both data sources exist.

Analyzing Data and Integrating the Results There are at least three steps to generating results for a convergent design where one or both data sources exist: analyzing the qualitative data, analyzing the quantitative data, and then mixing or integrating the results from the qualitative and quantitative data (Creswell & Plano Clark, 2018; Morse & Niehaus, 2009; Teddlie & Tashakkori, 2009; Creswell, Plano Clark, Gutmann, & Hanson, 2003). Other authors provide step-by-step procedures for how to analyze single-method qualitative data (Grbich, 2012; Miles, Huberman, & ­Saldana, 2013; Silverman, 2011; Tolley, Ulin, Mack, Robinson, & Succop, 2016; Watkins, 2012) and single-method quantitative data (Onwuegbuzie & Combs, 2010; Rosenthal, 2012; Warner, 2008). Therefore, I will not cover such detail in this chapter. However, I encourage readers to consider how these authors distinguish between their analysis of quantitative and qualitative data for single-method studies and mixed methods studies and whether the respective data sources of a mixed methods study need to be treated differently because they become part of a more extensive mixed methods study (Bazeley, 2018; Onwuegbuzie & Combs, 2010; Tashakkori & Teddlie, 2003). For example, data sources for convergent designs usually do not rely on the results from one study component to make decisions about the other component. Guetterman, Fetters, and Creswell (2015) note that the overarching term to describe side-by-side comparisons, data transformation, and data merging is joint data displays. So, as you begin analyzing, integrating, and presenting the data for your convergent design, you may find these three techniques helpful. The first is a side-by-side comparison, which gets its name from how the qualitative and quantitative data are presented in a table, diagram, or figure side-by-side (Creswell & Zhang, 2009; Guetterman, Fetters, & Creswell, 2015). This approach involves presenting one set of results alongside (or next to) the other results to see if the results complement one another. When using a sideby-side comparison for a convergent design with secondary data, there may not be huge differences in how this side-by-side comparison is made compared to that for a convergent design with new qualitative and quantitative data. By the time you produce the results for each data component, the fact that one came from a secondary data source and the other came from a new data source may not influence how you review, compare, and integrate the results side-by-side. The second way of integrating results for a convergent design is by changing one set of results (either qualitative or quantitative) into a format that allows for a smoother merging of one set of results with the other. This is called data transformation (Creswell & Creswell, 2018; Creswell, Fetters, & Ivankova, 2004; Curry & Nunez-Smith, 2015; de Block & Vis, 2018; Shemmings, 2008; Watkins, 2017a; Watkins & Gioia, 2015) and can look like qualitative data taking the form of numbers or quantitative data being transformed in a way that

Chapter 4  •  Convergent Design With Secondary Data  

67

BIG DATA BREAK 4.2. ANALYTICS FOR SECONDARY BIG DATA IN CONVERGENT DESIGNS Big data experts suggest there is both basic and advanced analytics associated with the measures and metrics of big data. Basic analytics are used to explore the data when you are unsure what you have but believe the data have something worth exploring (think “descriptive statistics”). Advanced analytics are used to explore algorithms for complex analysis and to uncover associations within the data (think “inferential statistics”). Which big data analytic you select for your convergent design will depend on the problem you want to solve and your research question. For example, big data analytics can get quite complex, such as using social media, text, and large organizational records. But for mixed methods designs that use big data, you have the luxury of beginning with basic analytics to assess whether the secondary data can help address your study purpose and research question and then interrogating the big data with advanced analytics after you have determined the data are a good fit for your convergent design.

allows the researcher to deeply analyze the context in which the language and scope of the quantitative measures were made (Onwuegbuzie & Combs, 2011; Onwuegbuzie & Combs, 2010). When transforming secondary qualitative and quantitative data for a convergent design, researchers may default to quantifying the qualitative data (Srnka & Koeszegi, 2007) because this is more frequently done than qualifying the quantitative data. I would caution researchers not to make decisions about data transformation based on ease of use and convenience but rather assess which techniques will address the research question. Sometimes, this will involve quantifying the qualitative data, but other times, it might mean qualifying the quantitative data. For example, let us assume you have just wrapped up a convergent design with secondary data on the influence of gender stereotypes on educational performance among high school students. Your qualitative data were focus groups with high school students, but you needed to collect new quantitative data on gender norms among high school students using the gender role stereotypes scale (GRSS; Mills, Culbertson, Huffman, & Connell, 2012). If your research purpose and question involved deepening your understanding of the measures used to acquire information about gender stereotypes, perhaps you might consider taking a more grounded theory approach to your review of the quantitative measures. One way to do this would be to qualify some of the quantitative GRSS items using content analysis to review their meaning, interpretation, and application.

68  Part II  •  Designing and Conducting Mixed Methods With Secondary Data The third way of integrating findings is by merging both data sets into a table, diagram, or figure. This is a single table, diagram, or figure in which the qualitative and quantitative results and interpretation are presented (Guetterman, Fetters, & Creswell, 2015; Johnson, Grove, & Clarke, 2017; ­Watkins & Gioia, 2015). When merging the quantitative and qualitative data for a convergent design with secondary data, the table, diagram, or figure may look similar to how it would look with new data. Joint data displays The key to merging for present related concepts a convergent design is from both the quantitative and the qualitative data and to ensure that both data show where the results from components have been those components overlap, thoroughly analyzed and the converge, or diverge. For results presented in a way that instance, using the previous GRSS example, you might highlights the individual and imagine a joint data display combined contributions of the including themes from the data sources to address the secondary focus group data research questions. or a content analysis of items from the GRSS.

Interpreting the Results Though generating and integrating the findings from a convergent design with secondary data may help address the research questions for the study, it does not complete the study. When writing the results, developing a separate report section can outline how the two data sources were merged and how the research team interpreted them. This often appears in the discussion section of the report (Creswell, 1999; Creswell & Creswell, 2018; Watkins & Gioia, 2015) and should include a section that compares the results from the two data sources and whether there was convergence or divergence between the sources (Creswell & Plano Clark, 2018; Irwin & Winterton, 2011; Pluye, Grad, Levine, & Nicolau, 2009). Remember whether one or both data sources existed, or new data were collected is not as important as whether the research team can use each data source to further their understanding of the study purpose and research question (Creswell & Plano Clark, 2018; Fetters & Freshwater, 2015). Though mixed methods researchers often celebrate when their qualitative and quantitative data align, or “converge,” I must underscore the value of a convergent design for which the results from the two data sources do not align, or “diverge” (Greene, 2007; Johnson & Onwuegbuzie, 2004; Pluye, Grad, Levine, & Nicolau, 2009; Tashakkori & Teddlie, 2008). When the results do not align, this allows for expansion of the research topic and methods in subsequent studies. Before I describe the implications for a convergent design

Chapter 4  •  Convergent Design With Secondary Data  

69

where the results diverge, let us first consider an example where the results converge. Under these circumstances, a mixed methods novice might assume that if they can identify variables from the quantitative data that align well with concepts from the qualitative data, then this is the marker for a successful convergent design. On the other hand, having a convergent design with divergent results (or results from the qualitative analysis and the quantitative analysis that differ) is not the end of the world. For example, one of my earlier studies pointed to the inconsistencies in depression measures when I discovered low depression scores were reported quantitatively (Watkins, 2006) while, qualitatively, some of my participants reported current depression diagnosis (and some even reported being in therapy at the time of my study; Watkins, Green, Goodson, Guidry, & Stanley, 2007; Watkins & Neighbors, 2007). Table 4.2 illustrates a joint Some of my most interesting data display for how my qualitative and quantitative mixed methods studies have data diverged. So, whether been the ones where I assumed the results of your convergent my results would converge, and design converge or diverge, they diverged. there is important information to be gained. Mixed methods research emerged from the triangulation literature (­Hammersley, 2008; Teddlie & Tashakkori, 2010), so convergence is usually the goal for many mixed methods researchers. At its core, triangulation has multiple objectives, including serving as a validity check, generating divergent interpretations, supporting complementary information, and providing epistemological dialogue or juxtaposition (Hammersley, 2008). So, while convergence may seem to be an apparent aim of mixed methods research, its etiology involves bringing data sources together to reach more justifiable conclusions than either one source could provide alone (Greene & Caracelli, 1997). Not only can convergent results confirm the relevance of a topic with a particular population of interest, but they can also ensure that the qualitative and quantitative methods used to understand a topic are aligned with, or at least complementary to, one another. Moreover, divergent results often provide researchers with greater insight into the complex aspects of a topic, leading to more in-depth investigations of previously unexplored aspects of that topic (Teddlie & Tashakkori, 2010). Regardless, whatever you generate from your results and interpret from your conclusions will inform the future direction of your research. The very nature of incorporating secondary data into a convergent design can add an additional layer of complexity to either convergent or divergent results. I advise researchers to be realistic about the goals and objectives of integrating secondary data sources because integrating two different data sources (and potentially two different samples) comes with its own challenges. For example,

Qualitative Data (Representative Quote) “Our depression is not the same as everybody else’s depression. You don’t have time to be sad because you have too much to worry about. You’re supposed to … take care of everything. You can be depressed but not have the symptoms of depression that a normal person would. I guess that standard of depression is not the standard depression that a Black man goes through.” “I would rather somebody call me the “n” word … than experience institutionalized racism. It’s indirect. It’s the silent killer. You know, it’s like, even going for job interviews, you hand them your resume when you walk through the door, you know, you say, ‘I wonder if they knew I was Black?’ And that kind of stuff it makes you kind of anxious.” “People have so many times said, ‘I’m trying to help you out’. But in reality, they were trying to stop us. I said I would rather not talk to some people because you don’t want them to know what

Quantitative Data (% on K-6 b item)

“So sad that nothing could cheer you up” 71.4% responded None of the time

“Nervous” 62.9% responded A little of the time

“Hopeless” 71.4% responded None of the time

“Everything was an effort” 40.0% responded Some of the time

Distress/Depressive Symptoms

Sad

Nervous

Hopeless

Effort

TABLE 4.2  ● Sample Joint Data Display Combining Qualitative and Quantitative Findingsa

Participants noted the lack of help from people (QUAL) and reported only some of the time feeling like everything was an effort (QUANT)

Participants discussed the challenges of institutional racism (QUAL) yet did not report high levels of hopelessness (QUANT)

Participants discussed racial tension (QUAL) yet reported feeling nervous a little of the time (QUANT)

Participants discussed feeling depressed (QUAL) yet did not report high levels of sadness (QUANT)

Integration/Interpretation

70  Part II  •  Designing and Conducting Mixed Methods With Secondary Data

Participants discussed needing to keep their feelings inside because nothing will make it better (QUAL) yet reported feeling so sad nothing could cheer them up none of the time (QUANT)

Participants discussed challenges with trusting a society that devalues them (QUAL) yet reported feeling worthless none of the time (QUANT)

Source: (1) Watkins, D. C. (2006). The depressive symptomatology of black college men: Preliminary findings. Californian Journal of Health Promotion, 4(3), 187-197. (2) Watkins, D. C., Green, B. L., Goodson, P., Guidry, J., & Stanley, C. A. (2007). Using focus groups to explore the stressful life events of black college men. Journal of College Student Development, 48(1), 105-118. (3) Watkins, D. C., & Neighbors, H. W. (2007). An initial exploration of what’ mental health’ means to young black men. Journal of Men’s Health and Gender, 4(3), 271-282.

b

The K-6 is the Kessler-6 scale for nonspecific psychological distress. Overall, the K-6 mean for the study sample was 10.89 (Range 6–30).

(1) Watkins (2006); (2) Watkins, Green, Goodson, Guidry, and Stanley (2007); and (3) Watkins and Neighbors (2007).

“We are still taught to keep our feelings inside … because you’re dealing with this problem and you’re not really talking about it, and nothing helped … nothing makes it better … especially for African American men… I believe we have more problems than any Caucasian or whatever, because of the fact that we are Black men in America, we have more problems to deal with.”

“So sad that nothing could cheer you up” 71.4% responded None of the time

Nothing can cheer you up

a

your weaknesses are because you don’t feel like they are genuinely going to help … how can you trust a society that suspects you?”

“Worthless” 88.6% responded None of the time

Worthless

Chapter 4  •  Convergent Design With Secondary Data  

71

72  Part II  •  Designing and Conducting Mixed Methods With Secondary Data if you are employing a convergent design whereby one of the One would assume that data sources exists, and you there is no way around are going to collect new, primary data for the other source divergence for convergent (Erzberger & Kelle, 2003), the designs that use secondary data. results might complement one However, that may not always be another based on the informathe case. tion you already know about the secondary data. But convergence or divergence is not guaranteed. Again, regardless of whether your convergent design uses new or secondary data and whether the findings complement one another or not, there is something important to glean from your results (Teddlie & Tashakkori, 2010).

Planning and Implementing a Convergent Design With Secondary Data This section provides ways to implement a convergent design with secondary data. There are at least three ways to do this by incorporating into the study: (1) secondary quantitative data, (2) secondary, qualitative data, or (3) secondary data for both sources.

Convergent Designs With Secondary Quantitative Data When the quantitative data for a convergent design exist, the design diagram will look like a traditional convergent design diagram. The real distinction is not in the diagram but in the research activities, because that you will need to analyze quantitative data that has already been collected and then collect qualitative data from a new sample. However, you can also use the same sample or a subsample of the secondary quantitative data. An example is when new participants are sampled for an evaluation and then used later for a convergent design with a different research question (Kawamura, Ivankova, Kohler, & Perumean-Chaney, 2009). The new qualitative data collection for this study can be inspired by the secondary quantitative sample.

Sampling for a convergent design with secondary quantitative data When sampling for a convergent design where the quantitative data exist, you should be prepared to critically evaluate the secondary quantitative samples and make decisions based on how (and who) you want to sample for the new qualitative component of the study. Sampling techniques used in quantitative research include simple random sampling, systematic random sampling, stratified random sampling, and convenience sampling. On the other hand, sampling

Chapter 4  •  Convergent Design With Secondary Data  

73

for qualitative research is purposive and may include convenience sampling or snowball sampling (Grinnell & Unrau, 2018; ­Watkins & Gioia, 2015). Developing a thorough sampling plan is essential for a convergent design where one or both data components exist. This is because traditional convergent designs with two new data sources is more straightforward. Separate information is collected from the same people for both the qualitative and quantitative study components. So, when you use secondary quantitative data for a convergent design, this may involve a different sampling approach, as you may not have access to the same sample for your new qualitative data. If you do and if your research question calls for new qualitative data to be collected from the secondary quantitative sample, then regaining access to those people is important. Often, there are limited opportunities to tap into the sample from a secondary data source. However, it may be possible to select a quantitative sample from which demographic characteristics can be used to draw a new qualitative sample. For example, imagine that a patient satisfaction survey was delivered at a family medicine clinic in your hometown. Suppose the survey was collected from residents, and it is not possible to contact those same residents for the qualitative component. In that case, you can review the demographic characteristics of the residents and draw a new qualitative sample of people who share the characteristics of the respondents from the quantitative sample.

Qualitative data collection instruments when the quantitative data exist A convergent design with secondary quantitative data requires new qualitative data collection tools. Carefully review how the aims of the original quantitative study were accomplished and note if the measures chosen by the original research team can inform your decisions for the new qualitative data. In some cases, the secondary quantitative data may fill a gap in secondary literature, while in other cases, it may be used to understand a concept at baseline before a deeper, richer interrogation of it. Planning for the measures for a convergent design with secondary quantitative data means there are opportunities to gather information from the quantitative study files and see which aspects of the measures (e.g., survey items, indices, reports, codebook, etc.) could inspire developing new qualitative measures. Creating new qualitative data collection tools may be ideal for the type of convergent design you are undertaking. Still, it behooves you to consider existing qualitative data collection tools, such as those used in a previous single- or mixed method study. For instance, if you were interested in learning more about frequently used depression instruments, you may decide to select items from existing quantitative instruments (e.g., Patient Health Questionnaire [PHQ-9] or the Center for Epidemiologic Studies Depression Scale [CES-D]) and use those items and the structure of the questions as inspiration for a new qualitative instrument (e.g., an interview questionnaire) that asks more

74  Part II  •  Designing and Conducting Mixed Methods With Secondary Data in-depth questions about depressive symptoms. RememWhen aligning new and ber that your access to existing secondary data sources measures, plus the need for new measures, may not always with your research questions, lead to developing a new meabe clear about what the data sure from scratch. Instead, it analysis and integration will may simply mean that qualiachieve. tative data collection tools from previous studies should be vetted for their relevance and potential inclusion in your convergent design with secondary quantitative data.

Example of a convergent design with secondary quantitative data To illustrate what a convergent design with secondary quantitative data could look like, I provide an example in Case Study 4.1. In this study ­(Diagram for Case Study 4.1), Jimenez et al. (2014) employed a convergent design with secondary quantitative data from the Translating EvidenceBased Developmental Screening (TEDS) study and new qualitative data. The

CASE STUDY 4.1 CONVERGENT DESIGN WITH SECONDARY QUANTITATIVE DATA

Reference:

Jimenez, M. E., Fiks, A. G., Ramirez Shah, L., Gerdes, M., Ni, A. Y., Pati, S., & Guevara, J. P. (2014). Factors associated with early intervention referral and evaluation. Academic Pediatrics, 14(3), 315–323.

Purpose:

To inform practice-level quality improvement efforts to link the medical home to community resources like early intervention and state-level efforts to comply with Individuals with Disabilities Education Act requirements regarding identification of children with developmental delay and service delivery.

Research Questions:

None reported.

Chapter 4  •  Convergent Design With Secondary Data  

MMR Design:

Convergent design with secondary quantitative data.

Secondary Data Source:

Secondary Quantitative Data

Secondary Data Contribution:

Secondary quantitative data from the TEDS study, a US Centers for Disease Control and Prevention— funded randomized controlled trial, tested the effectiveness of developmental screening protocols compliant with the 2006 American Academy of Pediatrics developmental surveillance and screening guidelines.

New Data Source:

New Qualitative Data Collection

2008–2010 Translating Evidence-Based Developmental Screening (TEDS) study; parentchild (n = 434) dyads focused on the children with developmental concerns; A Survey.

Pediatricians (n = 9) who participated in the TEDS study had at least one case where a child failed a developmental screener but was not referred to early intervention. Interviews consisted of open-ended questions regarding pediatricians’ decision-making around developmental screening and referral to early intervention. New Data Contribution:

The study team conducted interviews with pediatricians who participated in the original 2008–2010 TEDS study to understand better the factors that influence pediatricians’ early intervention referral process. The new, qualitative data explored the screening and referral process from the pediatricians’ experiences and complemented the TEDS study findings.

Goals of Integration:

To contextualize the findings and gain deeper insight into provider factors that influence the early intervention referral process than was available in these data sets. The authors also conducted in-depth interviews with pediatricians who participated in the TEDS study.

New Knowledge Generated:

This convergent design with secondary quantitative data helped understand that detecting developmental delays among children does not always translate into a referral for services and the receipt for services.

75

76  Part II  •  Designing and Conducting Mixed Methods With Secondary Data FIGURE FOR CASE STUDY 4.1 Purpose: To inform practice-level quality improvement efforts aimed at linking the medical to community resources. Research Question: N/A

Select secondary QUANTITATIVE data from 434 children with developmental concerns from the TEDS study

Collect new QUALITATIVE interviews with nine (n = 9) pediatricians about developmental screening and referrals

Review secondary QUANTITATIVE data using evaluative criteria to determine only hildren with special needs and those with developmental concerns involving 2+ domains were associated with early intervention (EI) referrals

Plan sampling measures/instruments, and procedures for new QUALITATIVE analysis

Analyze new QUALITATIVE data

Analyze secondary QUANTITATIVE data

QUANTITATIVE Results: The referral method (i.e., fax vs. phone) was associated with complete EI evaluation

Integrate QUALITATIVE/ QUANTITATIVE Results

QUALITATIVE Results: Office processes are critical for facilitating screening and EI referrals; the ability to fax referral forms improved referral process; families influenced EI referral decisions; instruments could be a challenge

Allowed researchers to contextualize results and gain insight into the provider factors that influenced the early intervention referral process than what would have been available from either single method alone

Source: Based on Jimenez, Fiks, Ramirez Shah, Gerdes, Ni, Pati, & Guevara (2014).

quantitative data came from a nationally represented, Centers for Disease Control and Prevention—funded randomized controlled trial, to which the authors had access. This allowed them to use a subset of the quantitative sample of parent-child dyads and the electronic medical records attached to each case for a smaller qualitative study. The pediatricians also participated in the TEDS study. This convergent design with secondary quantitative data helped the study team determine that detecting developmental delays among children does not always translate into a referrals and the receipt of services.

Chapter 4  •  Convergent Design With Secondary Data  

77

Convergent Designs With Secondary Qualitative Data When the qualitative data for a convergent design exists, the same rules from the convergent design with secondary quantitative data apply. Still, the focus is now on the secondary qualitative data. After developing the purpose statement and research question, plan the procedures for both the secondary data (qualitative) and the new data (quantitative). Modeling many of the same techniques used to collect single-method quantitative data can be helpful when planning the procedures for a convergent design with secondary qualitative data. The study methods (i.e., the plans for the sample, recruitment, data collection, and data analysis) should be appropriate for the new quantitative data’s contributions and how findings from the new quantitative data will be integrated with findings from the secondary qualitative data.

Sampling for a convergent design with secondary qualitative data When choosing a sample for a convergent design where the qualitative data exist, begin by assessing what is available from the secondary qualitative data and then determining who (and what) from the secondary sample needs to be included in the convergent design to address the study purpose and research question. For example, let’s assume you have access to interview transcripts (i.e., qualitative data) from elementary school administrators, teachers, and students from a school district in your state. For your new study purpose and research question, you may only need the secondary qualitative data from teachers. In this example, the new quantitative data could be survey data from elementary school teachers in your district and the surrounding communities. Just because you have access to different types of secondary qualitative data does not mean you should use all the data in your convergent design. If you are tempted to do so, I encourage you to determine which samples are most relevant to addressing your study’s purpose and research questions. Revisiting the study’s purpose and research question is a helpful first step; you can begin by asking yourself: Do I mention a specific population of interest in my purpose and research question? Do I include data from the people I hope to learn from in my sampling plan? If you answer these questions with a resounding “yes,” then you are on the right track. If not, ask yourself why you feel compelled to The relevance of the sample include data from a sample can be a tricky matter, and that is not the focus of your study. Will it help provide this is the case whether planning a more holistic view of the for a mixed methods study that phenomenon of interest? Do involves secondary data or secondary data from a differplanning to collect new data. ent sample complement or enhance your research? Are

78  Part II  •  Designing and Conducting Mixed Methods With Secondary Data they controversial compared to the secondary data from the sample representing your population of interest? What does the previous literature say about your population of interest and information from the other sample(s) you are considering for your study? Including secondary qualitative data from a sample that is not the focus of your research is not a bad idea; however, I recommend justifying these decisions in the sampling plan with a clear description of how information from a different group will help address the research question. For example, revisiting my previously mentioned example, you are most interested in the secondary data on teachers. Perhaps you could locate secondary interview transcripts from the teachers to complement the quantitative study component. But maybe the secondary qualitative data on administrators could also provide insight into your study and provide a more holistic look at your topic of interest. If that is the case, be sure to justify the decisions in your sampling plan. Some readers may find their sample for the secondary qualitative data limited or unable to provide the level of depth needed for their topic of interest. If this is your situation, consider how the new, quantitative sample can help complete your convergent design with secondary qualitative data. Quantitative data can provide generalizable, objective information generated from larger samples of people, so they can be used to expound on or explain the findings If the secondary qualitative from the secondary qualitainstruments help understand tive data for your convergent one aspect of the research design. So a new quantitative sample of your choice may topic, how can you maximize the be precisely what you need new quantitative instrument to to provide breadth to compleachieve a more comprehensive ment the depth of the secondpicture? ary qualitative sample and data. Frustrations you have about the limitations of your secondary qualitative sample may be outweighed by excitement for making decisions about your new, quantitative sample that will be able to generate new knowledge that builds on, complements, and extends the work of previous researchers.

Quantitative data collection instruments when the qualitative data exist Planning the data collection tools for a convergent design with secondary qualitative data means decisions have already been made about the qualitative data collection tools. You must decide how you want to use the

Chapter 4  •  Convergent Design With Secondary Data  

79

secondary instruments—either in the same way or different from the original researchers. Planning for the data collection tools the same way as the original researchers might mean aligning your use of the secondary qualitative instruments and the analysis plan for the new quantitative data with those of the original researchers. You may be able to learn more about the data source from reading previously published papers on the secondary data; reviewing websites about the study (e.g., if the source was derived from a federal funding source, this information is sometimes on their website); or reaching out to the original study team to see if they would be willing to share additional information and supplemental documents about the secondary data. There are matters to consider as you plan for the quantitative data collection tools when the qualitative data exist. For example, as you make decisions about the new quantitative data collection tools, think about which instruments will help you explain, complement, and enhance the instruments that the secondary qualitative data have predetermined. In other words, because decisions for the secondary qualitative instruments have already been made (and you have less control over those), be mindful about what you do have control over. New quantitative measures can be used to clarify your understanding of the more restricted secondary qualitative instruments. Sometimes, the limits of a secondary data source can lead to creativity. So relish the opportunity to think “outside the box” with the new quantitative data for your convergent design with secondary ­qualitative data.

Example of a convergent design with secondary qualitative data Case Study 4.2 is an example of a convergent design with secondary qualitative data. In this study (Diagram for Case Study 4.2), Coolbrandt et al. (2017) employed a convergent design with secondary qualitative data using two separate data sources collected at two different times. Some members of the study team were involved in both studies. In the 2017 study, the quantitative data were discussed as a new data collection effort. The qualitative data were presented as secondary data reanalyzed from a different study conducted by the authors. The quantitative data were more recent, while the qualitative data were collected between September 2012 and May 2013 (Coolbrandt et al., 2016). Though this study might be mistaken as sequential, it is convergent because of its purpose, research question, analysis, and data integration. Integrating these two data sources improved patients’ willingness to report symptoms from home during routine cancer treatment and its burden. Communicating with healthcare professionals about symptoms was an identified theme during the interviews.

80  Part II  •  Designing and Conducting Mixed Methods With Secondary Data

CASE STUDY 4.2 CONVERGENT DESIGN WITH SECONDARY QUALITATIVE DATA Reference:

Coolbrandt, A., Steffens, E., Wildiers, H., Bruyninckx, E., Verslype, C., & Milisen, K. (2017). Use of a symptom diary during chemotherapy: A mixed methods evaluation of a patient perspective. European Journal of Oncology Nursing, 31, 37–45.

Study Purpose:

To explore the patient perspective on using a symptom diary to monitor chemotherapy-related side effects they experience at home.

Research Questions:

To what extent do patients in treatment with chemotherapy use the symptom diary, and what factors mediate their use or nonuse of the symptom diary? How do patients experience the offer and use of it during chemotherapy treatment?

MMR Design:

Convergent design with secondary qualitative data.

Secondary Data Source:

Secondary Qualitative Data

Secondary Data Source Contribution:

To determine how patients perceived being offered the diary for tracking symptoms and, if they did use it, their experiences with the diary.

New Data Source:

New Quantitative Data Collection (QUANT/n = 143/survey).

New Data Contribution:

The survey assessed whether patients used the diary; if so, how they used it (or for nonusers, why they did not use the diary), and if they stopped, why. All patients who had used the diary at least once rated a set of 21 statements that assessed their perceptions of using the diary and its value, burden, and feasibility.

Goals of Integration:

The authors used a parallel data analysis approach in which data collection and analysis were performed separately, and qualitative and quantitative findings were integrated during the interpretation and discussion. The qualitative

Chemotherapy patients (n = 17); semi-structured interviews.

Chapter 4  •  Convergent Design With Secondary Data  

findings (e.g., symptom diary as a sign of the hospital’s diligence or concern) broadened how patients valued the diary. New Knowledge Generated:

The study contributes to a better understanding of patients’ willingness to report symptoms from home during routine cancer treatment and its burden. Communicating with healthcare professionals about symptoms was a theme during the interviews.

FIGURE FOR CASE STUDY 4.2 Purpose: To explore the patient perspective on using a symptom diary for monitoring chemotherapy-related side effects they experience at home Research Question: (1) To what extent do patients in treatment with chemotherapy make use of the symptom diary, and what factors mediate their use or nonuse of the symptom diary? (2) How do patients experience the offer and use of it during chemotherapy treatment?

Collect new QUANTITATIVE data from participants who were chemotherapy patients (n = 143) being treated in a hospital

Plan sampling, measures, and procedures for new survey data

Analyze new survey data

QUANTITATIVE Results: 79% used the symptom diary; 21% did not use the symptom diary; 30% reported the diary required too much time; and 19% struggled to integrate the diary into their daily life

Select secondary QUALITATIVE data from Participants were a subset of the patients (n=17) from the quantitative sample

Review secondary semi-structured interviews using evaluative criteria Analyze secondary semi-structured interview data

Integrate QUALITATIVE/ QUANTITATIVE Results

QUALITATIVE Results: Patients valued the diary because it helped them: report symptoms more accurately; communicate with their healthcare professional; deal with symptoms at home; and contribute to their professional care

Chemotherapy patients found the symptom diaries to be feasible regarding time commitment and ease of use. Inconsistencies existed regarding healthcare providers’ use of the diary with patients

Source: Based on Coolbrandt, Steffens, Wildiers, Bruyninckx, Verslype, and Milisen (2017).

81

82  Part II  •  Designing and Conducting Mixed Methods With Secondary Data

Convergent Designs With Secondary Quantitative and Qualitative Data When both the quantitative and qualitative data sources exist for your convergent design, you do not need to collect new data. Instead, your focus will be on bringing secondary qualitative and quantitative data by sources together to answer the research question. Use to Figure 4.1 to help you imagine a convergent design where both data sources exist, and then review these secondary data sources using the review criteria from Chapter 3. Bridging the review criteria to meet your study’s goals and objectives will be critical to successfully using the secondary sources to address your research question. Begin by trying to understand the methodological decisions made by the original researchers for their quantitative and qualitative studies. This is because the intent behind the original researchers’ sampling, instruments, and procedures will inform how you repurpose their secondary data.

Sampling for a convergent design with secondary quantitative and qualitative data The sampling plan for a convergent design with secondary quantitative and qualitative data will look very different from the sampling plan for a convergent design with one secondary data source. For example, no new data will be collected for this design. So what does a sampling plan for a convergent design with two secondary data sources look like? It begins by critically reviewing whom those studies sampled and why, and how the sampling plans influenced the decisions made in the studies. Closely reviewing the secondary qualitative and quantitative sampling plans and determing how (and if) they align with your research question will determine whether you should use both secondary qualitative and quantitative data sources for your convergent design, find other secondary data sources, or collect one or two new data sources for your study. As the popularity of mixed methods grows, more and more researchers are collecting both quantitative and qualitative data for their projects. So, if you have access to a colleague’s mixed methods data, you also have access to a secondary qualitative sample from a secondary quantitative sample. This allows you to select from a sample chosen by the original researchers, which can help you address your research question. While repurposing two secondary data sources may be the plan, do not force the integration of two secondary data sources that cannot answer your research question. Reference the review criteria in Chapter 3 throughout this process and as you continue to repurpose secondary data sources.

Example of a convergent design with secondary quantitative and qualitative data Case Study 4.3 is an example of a convergent design with secondary quantitative and qualitative data. This study by Craig, Steinauer, Kuppermann, Schmittdiel, and Dehlendorf (2019) used a convergent design where

Chapter 4  •  Convergent Design With Secondary Data  

CASE STUDY 4.3 CONVERGENT DESIGN WITH SECONDARY QUANTITATIVE AND QUALITATIVE DATA Reference:

Craig, A. D., Steinauer, J., Kuppermann, M., Schmittdiel, J. A., & Dehlendorf, C. (2019). Pill, patch, or ring? A mixed methods analysis of provider counseling about combined hormonal contraception. Contraception, 99(2), 104–110.

Study Purpose:

To further explore the process and content of contraceptive counseling about combined hormonal contraceptive (CHC) methods.

Research Questions:

None reported.

MMR Design:

Convergent design with secondary quantitative and qualitative data.

Secondary Data Source #1 and #2:

Secondary Quantitative and Secondary Qualitative Data Patient-Provider Communication about Contraception study (QUANT/n = 342/pre- and post-surveys with female patients and QUAL/n = 71/recorded interviews with female patients and 38 providers).

Secondary Data Source #1 and #2 Contribution:

Surveys from a representative sample of women receiving contraception in the geographic area at the time of the study. Recorded interviews between female patients and their providers. Secondary data identified (quantitatively and qualitatively) the women who preferred combined contraceptive, the patch, the ring, no preference, and a nonhormonal method.

Goals of Integration:

• To investigate differences in counseling according to whether patients preferred a specific CHC method. • To understand the content and process of contraceptive counseling surrounding CHC in San Francisco Bay clinics between 2009 and 2012.

New Knowledge Generated:

Findings showed that CHC methods often exclude information about all available methods and comprehensive information about side effects, benefits, and logistics of the available methods. The authors benefited from using secondary quantitative and qualitative data to learn about the limitations to the information provided by healthcare professionals who work in San Francisco Bay clinics.

83

84  Part II  •  Designing and Conducting Mixed Methods With Secondary Data FIGURE FOR CASE STUDY 4.3 Purpose: To further explore the process and content of contraceptive counseling about combined hormonal contraceptive (CHC) methods. Research Question: N/A

Select secondary QUANTITATIVE data from 342 female patients about their preferences for contraceptive methods. (Providers completed a one-time survey about their demographic information.)

Review secondary QUANTITATIVE data using evaluative criteria

Select secondary QUALITATIVE interviews from 71-recorded interviews with patients and their providers

Review secondary QUALITATIVE data using evaluative criteria

Analyze secondary pre- and post-test surveys

QUANTITATIVE Results: 37% of the sample preferred nonCHC methods; 33% preferred the pill; 18% had no preference; 8% preferred the ring; and 3% preferred the patch

Integrate QUALITATIVE/ QUANTITATIVE Results

Analyze secondary interview data

QUALITATIVE Results: Patients and providers reported: how women with (and without) a preference selected contraceptive methods were selected by; the implications of counseling including side effects, use, and non-contraceptive benefits

Results revealed a lack of comprehensive counseling about CHC methods, specifically about method choice, side effects, benefits, and the logistics of use.

Source: Based on Craig, Steinauer, Kuppermann, Schmittdiel, and Dehlendorf (2019).

the  secondary quantitative and qualitative data came from their Patient-­ Provider Communication about Contraception study, a longitudinal study of women who received contraceptive counseling in the San Francisco Bay area between August 2009 and January 2012. As the primary researchers were the ones who collected the original data, previous reports from these data were published in other journals. In the current study, the authors used their secondary pre- and post-test survey data with patients and secondary interview data with patients and providers to address their study purpose. By integrating these two secondary data sources, they learned that information about combined hormonal contraceptive methods often excludes information about all available methods, as well as comprehensive information about side effects, benefits, and logistics of the available methods (Diagram for Case Study 4.3).

Chapter 4  •  Convergent Design With Secondary Data  

85

Challenges (and Solutions) When Using Secondary Data in the Convergent Design Regardless of whether the qualitative data, the quantitative data, or both exist, you will likely face challenges when integrating secondary data into a convergent design. These challenges may include lack of complementary data, few integration options with the two data sources, a weak rationale for the inclusion of two data sources, the pacing of the data analysis and integration, and the inability to address the research question with the selected data sources. A lack of complementary data can be challenging to manage with a convergent design with secondary data for one or both study components. However, just as I outlined in earlier sections of this chapter, whether the findings are complementary or not, you can gauge important information from your decision to integrate the two data sources. But, if you are trying to avoid this outcome, know the strengths and limitations of your secondary data sources, and do your best to align similarities across the two databases. Understanding the strengths and limitations of both data sources can prepare you for potential outcomes, determined whether the results will be complementary or not, and influence the The beauty of mixed overall interpretation of the methods research is that you data to generate a conclusion. Having few integration can be creative in your pursuit options with the two data of information and for how you sources can also be challengwant to use two different data ing to manage. So this is your sources to help expand new opportunity to be creative (Bazeley, 2018) with your knowledge on the topic. plans to generate findings from each data source and think about how one data source can help you glean more from the other. Are there ways to analyze the data using methods outside your discipline? How can technology be used to enhance your design? Are there secondary or tertiary concepts or variables that should be promoted to a primary focus of your study? What sources of information seem supplemental but should be used as principal data for your study? When one or both data sources exist, the need to be creative becomes even more important, as often the repurposing of a secondary data source to address a new or different research question can be exciting but also force you to stretch your brain more than usual. Another challenge when using secondary data in a convergent design is a weak rationale for including two data sources. Previous scholars have outlined the importance of using mixed methods only when it is necessary (Creswell & Plano Clark, 2018; Tashakkori & Teddlie, 2010), so be careful if, by the time you

86  Part II  •  Designing and Conducting Mixed Methods With Secondary Data have completed your convergent design with secondary data, you are facing a hard truth: that you do not have a strong rationale for why you decided to pursue a mixed methods study in the first place. One strategy for addressing this with a convergent design is to see if you have exhausted all your options for analyzing and interpreting the results on the path to answering your research question. Consulting with experienced qualitative, quantitative, and mixed methods colleagues is always a good idea when you are stuck or need ideas about other ways to think about your study. After your options have been exhausted, you may decide to share the findings as a mixed methods study or two or more single-method studies. The pacing of data analysis and integration is another potential challenge for a convergent design with secondary data. Compared to the sequential designs, the convergent design is a simultaneously occurring design by nature. So preparing two data sources may likely happen alongside one another. However, when the two data sources are analyzed and integrated is at the researcher’s discretion and will depend on access to the secondary data, comfort with analysis, and plans for integration. It is also important to note a potential bias regarding the appropriate pacing for a convergent design with secondary data. To resolve this matter, consider analyzing one data source before the other to avoid the results of one data source potentially skewing your interpretation of the other. I have seen people wait several weeks or months between the time they analyze both data sources to minimize bias. The final challenge you may face when working with a convergent design with secondary data is your inability to address the research question with the two data sources. Using the criteria in Chapter 3 to evaluate the secondary data source(s) and whether they will enhance your convergent design will be essential to avoid getting to the end of your study, only to realize that you cannot address your research question with the data sources you selected. Sometimes, despite your careful review and evaluation of data sources, the analyses produce results that do not help address the overall research question. Under such conditions, consulting with colleagues can be reassuring as you decide how best to proceed.

Summary This chapter reviewed how secondary data could enhance the convergent design. You now know that, unlike a sequential design where the results from the first data source have a direct influence on the decisions you make about the second data source, with a convergent design, the two data sources remain separate throughout the study until the merging/integration occurs during the later stages of the project. You also know ten steps to incorporate

Chapter 4  •  Convergent Design With Secondary Data  

secondary quantitative and qualitative data into a convergent design. To generate results for a convergent design where one or both data sources exist, you will need to analyze the qualitative data, analyze the quantitative data, and then mix or integrate the results from the two data sources. Side-by-side comparison, data transformation, and merging are three ways to integrate your convergent design’s secondary data. After reading this chapter, you should have a good sense of how to develop a plan for sampling, instruments, and procedures for employing a convergent design with secondary quantitative data, qualitative data, or both. Finally, the chapter ends with solutions to the challenges of working through your convergent design with secondary data.

Chapter 4 Application Questions 1. What key features of the convergent design might make using secondary data different from using secondary data with a sequential design? Make a list of how the key features of the convergent design were advantageous to the authors of Case Study 4.3. 2. What are ten steps to help you prepare secondary quantitative and qualitative data for a convergent design? Create a table like Table 4.1 for your convergent design with secondary data. Keep the left column of information from Table 4.1 but fill in the right column with information associated with your study. 3. How do the steps for a convergent design differ when there is one secondary data source compared to two secondary data sources? Compare Case Studies 4.2 and 4.3 from this chapter. How did the authors benefit from new quantitative data in Case Study 4.2 compared to how the authors helped from secondary quantitative data in Case Study 4.3? 4. Which of the three ways to integrate data for a convergent design with secondary data seem like something you would use for your convergent design? Why? 5. How do the challenges to employing a convergent design with secondary data appear in Case Study 4.1? How did the authors go about resolving them?

87

5 Exploratory Sequential Design With Secondary Data Imagine You just returned home from a mixed methods conference in the Caribbean, which was a much-deserved work/play trip since completing a large qualitative project a few months ago. You attended the conference with your classmate, Clayton, so you both could learn more about mixed methods research. One of your mentors suggested mixed methods might be an “appropriate next step” in your research. But, here, you sit now, staring at a blank screen, trying to decide if you should dig up the one-on-one, in-depth interviews from your qualitative project. Given your research topic, mixed methods seem like a logical next step in your trajectory. Admittedly, in the middle of the qualitative data collection, you thought about a few research questions that would be best answered with a survey. But, due to time, convenience, and your training, you and your team agreed a qualitative study would be the best. As with many qualitative projects, you did not have a chance to dive deeper into all the emerging themes. So, to expound on your work, you have decided to look closely at a different aspect of your secondary qualitative data (one you did not report earlier) and then build a new, quantitative component into your study to see if the findings can be generalized to a larger population. If this process sounds exciting, then this chapter is for you!

89

90  Part II •  Designing and Conducting Mixed Methods with Secondary Data

Learning Objectives This chapter reviews the exploratory sequential design and discusses how secondary data can be used for one or both study components. By the end of the chapter, you will be able to: 1. Define the features of an exploratory sequential design, 2. Describe how to bring secondary data into an exploratory sequential design, 3. Analyze and integrate secondary data for an exploratory sequential mixed methods design, 4. Plan and implement an exploratory sequential design with secondary quantitative data, secondary qualitative data, and secondary qualitative and quantitative data, and 5. Identify potential challenges (and solutions) when using secondary data in an exploratory sequential design.

Features of the Exploratory Sequential Design The exploratory sequential design is a core mixed methods design (Creswell, 2015; Plano Clark & Ivankova, 2016) that involves collecting qualitative data, analyzing those data, and then making method-based decisions about how to collect and analyze a second, quantitative study component based on the qualitative methods and results (Figure 5.1). This design has other names, such as the “sequential exploratory design” and the “sequential qualitative to quantitative design” (Plano Clark & Ivankova, 2016). Typically, researchers choose an exploratory sequential design when the research problem is qualitatively oriented, when necessary variables are not known and instruments are not available, when the researcher has limited resources and needs to collect and analyze one data type at a time, and when new questions have emerged from qualitative results (Creswell & Plano Clark, 2018; Watkins & Gioia, 2015). Thus, qualitatively oriented researchers gravitate toward the exploratory sequential design, as it allows them to begin with a research method most familiar to them. If quantitative data are not the priority in your study, you An exploratory sequential may need to use the exploratory design can be used for sequential design, where the qualitative methods and results instrument development and shape a quantitative study comassessing whether qualitative ponent. This occurs by using the themes generalize to a results from the first qualitative population. study to identify research questions and variables, develop an

Chapter 5  •  Exploratory Sequential Design With Secondary Data  

91

FIGURE 5.1  ● Exploratory Sequential Design With New and Secondary Data Formulate a research question

Select secondary (or collect new) QUALITATIVE data Review secondary QUALITATIVE data using evaluative criteria (or develop sampling, instruments, and procedures for new QUALITATIVE data collection) Analyze secondary (or new) QUALITATIVE data Produce QUALITATIVE results

Data integration and determine the need for secondary (or new) QUANTITATIVE data

Use results from QUALITATIVE data to make decisions about QUANTITATIVE samples, measures, and procedures

Review secondary QUANTITATIVE data using evaluative criteria (or develop sampling, measures/instruments, and procedures for new QUANTITATIVE data collection) Analyze secondary (or new) QUANTITATIVE data Produce QUANTITATIVE results

Data integration

Review QUALITATIVE results and how they help extend the QUANTITATIVE results

Generate findings and overall conclusions

instrument, or generate a typology to be explored during the second quantitative component (­Creswell & Plano Clark, 2018; Watkins & Gioia, 2015). The point of interface is where the two data sources meet (i.e., “mixing”; Creswell, 2015). An exploratory sequential design can be used for instrument development and assessing whether qualitative themes generalize to a population.

92  Part II •  Designing and Conducting Mixed Methods with Secondary Data

BIG DATA BREAK 5.1 TYPES OF BIG DATA WITH SEQUENTIAL DESIGNS If you use big data for your exploratory sequential design, it will be incorporated into the second quantitative study component. Your qualitative data may be new or secondary, but your quantitative study component will include secondary data. When identifying secondary big data for your mixed methods study, the sources fall into at least two categories: (1) computer-generated data and (2) human-generated data. Computer-generated data are created using a computer (or a machine). An example of computer-generated data might be web login data, such as login information for an app on your iPhone or Android phone. Human-generated data are created by humans in collaboration with machines. An example of human-generated data is any piece of information you would enter into a computer database, such as your name, birthday, country of origin, and so on. An example of an exploratory sequential study that uses big data could explore the frequency of use, comfort, and familiarity of older adults (over age 70) in Canada who use the social media application, Instagram. To maximize the benefits of big data, you could collect new focus group data from older adults and then generate qualitative themes and subthemes that examine how they describe their frequency of use, comfort, and familiarity with Instagram. Then (new or secondary) human-generated big, quantitative data could be used to track their Instagram activities over six months to generalize how closely aligned their Instagram activities are to the themes and subthemes from their focus groups.

Bringing Secondary Data Into the Exploratory Sequential Design Incorporating secondary data into an exploratory sequential design can enhance this core mixed methods design. For example, while research projects generally require a lot of time, qualitative projects often need more time to plan the study design, recruit participants, and collect data. Data collection for a qualitative study focuses more on depth than breadth; therefore, qualitatively driven projects require time and a different mindset when compared to quantitatively driven projects. Secondary qualitative data used for an exploratory sequential design could mean that you put that time and mindset toward data analysis and integration rather than planning a primary research design and collecting data. What I like most about bringing secondary qualitative data into my mixed methods studies is the opportunity to be reflexive and apply my lens to someone else’s rich, qualitative inquiry to generate a different set of themes and conceptual framing that can later be tested quantitatively.

Chapter 5  •  Exploratory Sequential Design With Secondary Data  

93

Preparing Secondary Data for an Exploratory Sequential Design An exploratory sequential design with secondary data looks like a traditional exploratory sequential design; only in place of one or both data collection steps, you will review and evaluate the secondary data using evaluation criteria from Chapter 3. Creating a diagram or figure for your exploratory sequential design with secondary data (such as Figure 5.1) can help you outline a plan for integrating the secondary data into the mixed methods procedures. A diagram or figure can also demonstrate how secondary data will enhance the exploratory sequential design. Procedures for an exploratory sequential design with secondary qualitative and quantitative data (Table 5.1) follow a somewhat different ordering than the procedures for a convergent design (Refer back to Table 4.1). I offer 12 steps for preparing secondary qualitative and quantitative data for an exploratory sequential design. TABLE 5.1  ●  Preparing Secondary Qualitative and Quantitative Data for an Exploratory Sequential Design

Step

Steps for Preparing Secondary Qualitative and Quantitative Data

Relevance to the Exploratory Sequential Design

1

Decide on the kind of data you will need for the first qualitative study component.

Though the pacing for this design is clear, the decision to use this design often comes from researcher expectations for the second quantitative component. Thus, whether the second component will include secondary data or not, the first step involves deciding whether the first qualitative component will use new or secondary data.

2

If using secondary qualitative data, assess the secondary data for topic fit and alignment. (If collecting new data, design the qualitative study.)

Design the first qualitative study’s sampling, instruments, and procedures. Determine what you already know (if using secondary qualitative data) or what you hope to learn from the qualitative component (collecting new data). This step will also determine if the secondary data can complement the new data (if applicable) and if they both address the research question. Take inventory of the secondary data to assess what is available, missing, and incomplete to determine the fit and alignment with the research question. (Continued)

94  Part II •  Designing and Conducting Mixed Methods with Secondary Data TABLE 5.1  ● (Continued)

Step

Steps for Preparing Secondary Qualitative and Quantitative Data

Relevance to the Exploratory Sequential Design

3

If using secondary qualitative data, review and evaluate the qualitative data using the evaluation criteria from Chapter 3. (If using new data, collect the qualitative data.)

The evaluation criteria from Chapter 3 will assist you in determining (1) how you want to manage the samples from the secondary qualitative data (if applicable), (2) what about the secondary qualitative data can help you answer the research question, and (3) what is missing from the qualitative data that can be supplemented by collecting new (or using secondary) quantitative data.

4

Develop an analysis plan for the secondary (or new) qualitative data.

The analysis plan for the secondary qualitative data may (or may not) be complementary to the plan proposed by the original researchers; this step should occur iteratively with step 3.

5

Analyze the secondary (or new) qualitative data.

With an exploratory sequential design, the timing for your use of the second data source is contingent upon your use of the first data source. Try to access complete data files before deciding to use them in your study.

6

Integrate the data sources by making decisions about the kind of data (new or secondary) you will need for the second quantitative study based on what you find in the first qualitative study.

If using secondary qualitative data, try to understand how the original researchers made decisions about their qualitative methods. Then, review your qualitative results (from new or secondary data), summarize them, and then make decisions about a new (or secondary) quantitative study based on the qualitative methods and results.

7

If using secondary quantitative data, assess the secondary data for topic fit and alignment. (If collecting new data, design the quantitative study.)

Design the quantitative component sampling, instruments, and procedures. Develop a log for the secondary data, if/how the original researchers used it, and if/how you will use it for your exploratory sequential design. Review secondary methods (or plans for new data collection) for appropriateness and their ability to address the research question.

Chapter 5  •  Exploratory Sequential Design With Secondary Data  

95

Steps for Preparing Secondary Qualitative and Quantitative Data

Relevance to the Exploratory Sequential Design

8

If using secondary quantitative data, review and evaluate the data with the evaluation criteria. (If working with new quantitative data, collect it.)

With the exploratory sequential design, results from the new (or secondary) qualitative data will influence selecting new (or secondary) quantitative methods. Incorporate the evaluation criteria from Chapter 3 for secondary data as needed.

9

Develop an analysis plan for the secondary (or new) quantitative data.

The analysis plan for the secondary data may (or may not) complement the plan proposed by the original researchers. Regardless, this analysis plan should address the research question.

10

Analyze the secondary (or new) quantitative data.

Use single-method quantitative analysis techniques to generate results.

11

Integrate the methods by reviewing the qualitative results and how they compare to the quantitative results.

With the exploratory sequential design, one point of interface (i.e., integration) occurs when you assess the individual qualitative and quantitative results. Then, you review them collectively to determine how the combined results help address the research question.

12

Generate overall findings and conclusions.

Results can be used to inform future research, practice, and policy. Methods can also inform the use of new or secondary data in future projects.

Step

BIG DATA BREAK 5.2 INTEGRATING SECONDARY BIG DATA INTO SEQUENTIAL DESIGNS An essential first step in any mixed methods study with big data is mining the data. Big data mining involves exploring and analyzing data to find patterns from which you can classify and predict study outcomes. With classification, you sort the data into groups. You predict the value, direction, or strength of a variable or concept captured from the data with prediction. From an analysis point of view, I like to think of classification as being achieved by running basic statistical analysis (i.e., the equivalent of descriptive statistics) and prediction being achieved by running more advanced analytics (i.e., the equivalent of inferential statistics). (Continued)

96  Part II •  Designing and Conducting Mixed Methods with Secondary Data

(Continued)

For example, thinking back to my example from Big Data Break 5.1. You may decide to mine your human-generated data from older Canadians by separating the Instagram data into groups by gender, race, age, or geographic region. Then you might run analyses to examine the frequency of use, comfort, and familiarity for your sample based on gender, race, age, or geographic region.

Data Analysis and Integration for an Exploratory Sequential Design With Secondary Data Now that I have covered the features of an exploratory sequential design and how to prepare secondary data, I will describe how to analyze and integrate the new and secondary data into your exploratory sequential mixed methods study. The data analysis and integration steps for an exploratory sequential design might look different compared to the convergent design (which I covered in Chapter 4). Here, I discuss helpful tips when generating results and interpreting the findings for an exploratory sequential design with one or two secondary data sources.

Analyzing Data and Integrating the Results At the risk of stating the obvious, the exploratory sequential design is, well. . . sequential. So, rather than waiting until you have analyzed both data sources to integrate them, data integration occurs by asking yourself: How can I make decisions about the (new or secondary) quantitative study based on what I found from the (new or secondary) qualitative study? Thus, the process and pacing you use to analyze the data for an exploratory sequential design can vary depending on whether the qualitative data exist, the quantitative data exist, or both exist. If the qualitative data exist, the analysis may involve a review, followed by sifting, sorting, and categorizing the data (Grbich, 2013; Miles, Huberman, & Saldana, 2013; Watkins, 2012, 2017b). These tasks are usually done to achieve both content-related and method-related goals. Notably, the act of sifting, sorting, and categorizing data can help you: (1) address the research question, and (2) make decisions about how to best move forward with the second quantitative study. An example might help you understand this concept better. Let’s say you just completed the first phase of an exploratory sequential design where the results from your secondary qualitative analysis Sifting, sorting, and revealed four ways oncology categorizing data involves nurses prefer to engage with their patients. These results reviewing and organizing the offer insights about a new data to determine patterns to sampling plan and survey for help determine similarities and new quantitative data collecdifferences in the data concepts. tion with oncology nurses. You can also use the secondary

Chapter 5  •  Exploratory Sequential Design With Secondary Data  

97

qualitative methods as a roadmap for (1) which nurse engagement preferences to focus on during the new quantitative data collection and (2) which analysis and integration techniques will yield findings to address your research questions. Developing a survey instrument is an example of a mixed methods study that could benefit from an exploratory sequential design with secondary qualitative data. For example, let’s say you decide to use secondary qualitative data to develop a new instrument that will include more culturally appropriate language for Asian adolescents and their knowledge, attitudes, and beliefs about standardized testing in the United States. Then, using secondary qualitative data—such as interviews with Asian adolescents or a meta-synthesis of qualitative studies on Asian adolescents—you could create survey items to test with a new sample. If you have the quantitative data for your exploratory sequential design the analysis and integration might involve (1) collecting new qualitative data, (2) identifying themes and subthemes, and then (3) determining how well the themes and subthemes map onto your secondary (quantitative) survey instrument or other quantitative files. For example, let’s say you have access to quantitative surveys from 300 math teachers who responded to questions about school climate, culture, and their preferred school placements. These data also include teachers’ demographics (e.g., race, ethnicity, gender, parents’ education, household income, etc.). Given your interest in how math teachers’ identities and backgrounds influence their preferred school placements, you decide to employ an exploratory sequential design with new qualitative data. You first observe math teachers in their classrooms and during faculty meetings to learn how they discuss their school climate and culture. You then conduct one-on-one interviews with 25 teachers to learn about their backgrounds, journeys to becoming teachers, and school placement preferences. Once you analyze these rich qualitative data, you may learn about different concepts from the observations and the one-on-one interviews that you want to map onto the secondary quantitative instrument. You will likely run statistical models with multiple outcomes for your quantitative analysis, including school culture, climate, and preferred school placements. Suppose both the qualitative and the quantitative data exist for an exploratory sequential design. In that case, the analysis step will involve an iterative, combined process of sifting, sorting, and categorizing data sources and mapping the results from one study component onto the methods for the other study component. The key to working with two secondary data sources for an exploratory sequential design is to remember that this The key to working with two mixed methods design was secondary data sources for intended to be employed sequentially. So while you an exploratory sequential design may have access to both secis to remember that this mixed ondary data sources from the methods design was intended to very beginning of the project, be employed sequentially. try to work with the qualitative data first. Starting with

98  Part II •  Designing and Conducting Mixed Methods with Secondary Data the secondary qualitative data (let’s assume they are interviews), you want to sift, sort, and categorize the interviews to determine the takeaway points that best illuminate the answer to your research question. Then, using the results from this analysis, you can create a concept map of themes and subthemes that can be reviewed side-by-side with (e.g., mapped onto) the variables from the secondary quantitative data and files (let us assume in this case it is an online survey). This process will also show you which topic areas of the survey are covered in the interviews and which are not. This mapping process can be helpful in the current study and inform future project iterations.

Interpreting the Results The interpretation step of an exploratory sequential design is another point of interface (i.e., where integration occurs). If you recall, for an exploratory sequential design, one of the first points of interface is where you make decisions about the quantitative component based on what you found from the qualitative component (see Figure 5.1). The second point of interface is the one that occurs after the quantitative data have been analyzed. This is because after both sets of data have been evaluated (and collected) and analyzed (and integrated), you will want to take a step back and spend some time looking over the qualitative results, reviewing how they led to the decisions you made for the quantitative methods of your study, and determining other useful qualitative findings to report. Keeping your research question in mind, this additional integration step will help you perform a more comprehensive review of the quantitative results, how they align with and extend the previous qualitative results, and how they all inform your overall study findings and conclusions. When interpreting the results of an exploratory sequential design with secondary data, the fact that one or both data sources already exist may pose challenges to the study you cannot ignore. For example, using secondary data sources is a great way to maximize project resources and avoid employing entirely new data collection efforts. However, it is important to note that secUsing secondary data ondary data sources sometimes underscore more gaps in a topic sources is a great way to than fill them. It may be easier maximize project resources and for someone who did not collect avoid employing entirely new the original data to identify its data collection efforts. limitations, especially when the data are repurposed to address a different, albeit expanded, research question. Consequently, it will be important to report both the strengths and limitations of your exploratory sequential design using secondary data, why secondary data sources were used for one or both study components, and how the study might have looked different had new data been incorporated into the study for one or both study components.

Chapter 5  •  Exploratory Sequential Design With Secondary Data  

99

Planning and Implementing an Exploratory Sequential Design With Secondary Data This section describes how to plan and implement an exploratory sequential mixed methods study when one or both data sources exist. There are at least three ways to do this; by incorporating secondary data into: (1) the qualitative study component, (2) the quantitative study component, or (3) both study components.

Exploratory Sequential Designs With Secondary Qualitative Data While having access to secondary data is exciting, it does not negate the importance of focusing the decisions you make about the methods on the guiding research question, which should be established at the beginning of the study (Figure 5.1). This level of focus can sometimes be lost once you review secondary data sources critically, so I will periodically remind you to check your research question throughout this book. The research question is important and can serve as a roadmap, directing the path for your study and your decision to use new or secondary data. The research question is important for any mixed methods design. But it is even more important when framing how a secondary qualitative data source will produce results that can be integrated into the study design for a new quantitative component. For example, a straightforward research question can help you decide how to dive into a secondary data source that includes 50 onehour interviews with young adults previously in foster care. While the richness of each interview could potentially sway you in one direction or another, staying focused on your research question (which may be different compared to the research question from the original study) will help you develop the appropriate sampling plan for the secondary data and operationalize the measures. So let’s begin by considering the sampling plan.

Sampling for an exploratory sequential design with secondary qualitative data When qualitative data exist for an exploratory sequential design, you must evaluate the sampling procedures for the data source to (1) choose a subsample from the secondary qualitative data for your study and (2) develop a sampling plan for the new quantitative data collection. Your challenge will be making decisions about the new quantitative sampling plan based on what you evaluate from the secondary qualitative sampling plan. For the exploratory sequential design, some researchers may find it easier to work, first, with secondary qualitative data than with secondary quantitative data (which I discuss in the next section). This is only because the limitations of secondary data sources are mainly handled during the first (qualitative) component. Meanwhile, the

100  Part II •  Designing and Conducting Mixed Methods with Secondary Data second new (quantitative) component may offer more flexibility and freedom in sampling and data collection. Given this, it is vital to glean what you can from the secondary qualitative sample before determining the next steps. For example, what can you learn from the secondary qualitative subsample that will inform your decisions for the new quantitative data? Can you recruit a new sample whose characteristics align with or complement the subsample of respondents from the secondary qualitative data? Suppose your exploratory sequential design is meant to influence outcomes for a particular group of people for a specific purpose (e.g., a neighborhood or a school-based intervention). In that case, you should precisely align the demographic characteristics of the new quantitative sample with those of the secondary qualitative sample. Moreover, if your second (quantitative) study needs to be more generalizable, collecting new data from a new quantitative sample with demographics that are similar to those from the original qualitative sample will suffice.

Quantitative data collection instruments when the qualitative data exist When the qualitative data exist for an exploratory sequential design, you may feel limited by the decisions you can make about the qualitative component of your study. This is because someone has already collected the data using an instrument they believed would answer their research question, not necessarily yours. Being strategic about maximizing the secondary qualitative data to make decisions about the measures you select for the new quantitative data is essential to completing an exploratory sequential design with secondary qualitative data. For example, secondary focus group data may prompt you to review the focus group questionnaire and then use those questions and their responses to create a new survey with questions that confirm the information generated from the focus group data. The key here is to build on the strengths of the secondary qualitative measures by maximizing ways to expound on those findings as you create the data collection tools for the quantitative study component.

Example of an exploratory sequential design with secondary qualitative data Case Study 5.1 uses two studies by the same research team to illustrate an exploratory sequential design with secondary qualitative data. In the first study, Carrington, Gephart, Verran, and Finley (2015) employed an exploratory sequential design with secondary qualitative data to develop an instrument to measure the unintended consequences of electronic health records. In the second study, Gephart, Bristol, Dye, Finley, and Carrington (2016) tested the validity and reliability of the instrument previously mentioned with acute care nurses. Rather than including instrument development and instrument testing in the same article, the authors published two peer-reviewed journal articles to maximize their reach and impact (Figure for Case Study 5.1).

Chapter 5  •  Exploratory Sequential Design With Secondary Data  

101

CASE STUDY 5.1 EXPLORATORY SEQUENTIAL DESIGN WITH SECONDARY QUALITATIVE DATA References:

Carrington, J. M., Gephart, S. M., Verran, J. A., & Finley, B. A. (2015). Development of an Instrument to Measure the Unintended Consequences of EHRs. Western Journal of Nursing Research, 37(7) 842–858. ** and ** Gephart, S. M., Bristol, A. A., Dye, J. L., Finley, B. A., & Carrington, J. M. (2016). Validity and reliability of a new measure of nursing experience with unintended consequences of electronic health records. Computers, Informatics, and Nursing, 34(10), 436–447.

Purpose:

To develop and test the psychometric adequacy of the new Carrington-Gephart Unintended Consequences of Electronic Health Records Questionnaire (CG-UCE-Q).

Research Questions:

From a preexisting qualitative data set, what are the unintended consequences encountered by nurses while using electronic health records? Using these data, what process can be used to transfer qualitative data to scale development for psychometric testing?

MMR Design:

Exploratory sequential with secondary qualitative data

Secondary Data Source:

Secondary Qualitative Data

Secondary Data Source Contribution:

Using the emergent themes derived from the qualitative data, a self-report measure was developed to assess nurses’ experience with electronic health records (EHRs). Themes (including security access, hardware issues, data entry, and irretrievability) were expanded to construct items for the instrument. An item was created verbatim when a statement was identified from the qualitative data reflecting the theme particularly well.

New Data Source:

New Quantitative Data Collection

New Data Source Contribution:

Twenty items were derived from the qualitative approach, each representing a theme. These items are joined by eight items (Ash, Sittig, Dykstra, Campbell, & Guappone, 2009) to investigate clinicians’ experience with unintended consequences (UCs) and 12 items exploring relationships of UCs with patient safety events (Carrington & Effken, 2011).

Secondary interviews from 37 registered nurses (RNs) at 2 care facilities

New survey data with acute nurses (n = 144) from a local hospital

(Continued)

102  Part II •  Designing and Conducting Mixed Methods with Secondary Data

(Continued) Goals of Integration:

To enable the development of a theory-based instrument to measure unintended consequences, move the science of efficient measurement forward, and develop an instrument.

New Knowledge Generated:

This research revealed several significant findings concerning barriers and workarounds and nurse-to-nurse communication when using the EHR and presented a fresh picture of the phenomena.

FIGURE FOR CASE STUDY 5.1 Purpose: To create and test the reliability and validity of a new instrument Research Questions: (1) From a preexisting qualitative data set, what are the unintended consequences encountered by nurses while using electronic health records? (2) Using these data, what process can be used to transfer qualitative data to scale development for psychometric testing? Selected secondary QUALITATIVE interviews from registered nurses across two care facilities Reviewed secondary QUALITATIVE data using evaluative criteria to develop categories of barriers and solutions reported by nurses Analyzed secondary QUALITATIVE data QUALITATIVE Results: A list of “workarounds” for nurses and overall themes

Data Integration: Items for the instrument (CarringtonGephart Unintended Consequences of Electronic Health Record Questionnaire [CG-UCE-Q]) were derived from QUALITATIVE themes and used to assess nurse’s experiences with EHRs

Used results from themes generated from secondary QUALITATIVE data to make decisions about samples, measures, and procedures for new QUANTITATIVE data

Used a cross-sectional exploratory, descriptive approach to survey registered nurses (n = 144) Analyzed new QUANTITATIVE data collected from CG-UCE-Q with descriptive statistics and principal component analysis with oblique rotation to evaluate construct validity QUANTITATIVE Results: The validity of the instrument was moderately supported with both the context and processes of workarounds Data Integration: The secondary QUALITATIVE data led to construct development for a new QUANTITATIVE

Reviewed how QUALITATIVE results led to the development of QUANTITATIVE results

Strong associations identified between when nurses perceived a block and altered a process to work around it to subscales in the new instrument CG-UCE-Q for electronic health record system design (P < 0.01) and technology barriers (P < 0.01)

Source: Based on Carrington, Gephart, Verran, & Finley (2015) and Gephart, Bristol, Dye, Finley, & Carrington (2016).

Chapter 5  •  Exploratory Sequential Design With Secondary Data  

103

Exploratory Sequential Designs With Secondary Quantitative Data For exploratory sequential designs where the quantitative data exist, you will collect new qualitative data, then those results will be used to further develop your understanding of the secondary quantitative data. Collecting new qualitative data that needs to be mapped onto secondary quantitative data may give you the impression you have freedom and flexibility initially but restrictions at the end. After Be creative and flexible all, the freedom and flexibility with how the data from the of the qualitative data comnew qualitative component are ponent become limited once those findings are used to make integrated into the secondary decisions about a secondary, quantitative study component. closed-ended quantitative data source, right? So how should you plan for such a study? First, have realistic expectations about incorporating the secondary quantitative data into the overall exploratory sequential study. Acknowledge what is and is not available in the secondary quantitative data and then decide how to use it to explain what you learned from the new, qualitative data. For example, once themes are generated from the new qualitative data, you may need to see which aspects of those themes can be mapped onto different properties of the secondary quantitative data (e.g., sample, measures, data, procedures, etc.) and which cannot. For ideas for linking the two data sources, I refer you to earlier sections of this chapter where I discuss sifting, sorting, categorizing, and mapping concepts from one study component onto the other. Next, let’s consider the sampling procedures for an exploratory sequential design with secondary quantitative data.

Sampling for an exploratory sequential design with secondary quantitative data When the quantitative data exist for an exploratory sequential design, you will need to develop a sampling plan for the new qualitative data and then evaluate the sampling procedures for the secondary quantitative data. Admittedly, it may be tempting to work with the data you have (i.e., quantitative) before thinking about the data you do not have (i.e., qualitative). The truth is: You may experience some difficulty trying to ignore what you learned about the secondary quantitative sample during your evaluation (Chapter 3). Rather than doing that, a more realistic goal might be to acknowledge what you learned from the quantitative sample and permit yourself to let it influence how you think about your qualitative sample. Though you cannot ignore what you learned about the secondary quantitative sample, work with the qualitative sample first. The secondary data will be used for the second half of the study, so you can still work iteratively to make decisions for the new qualitative sample, but not so much that you are distracted by the secondary quantitative sample. Instead, think of your new qualitative sample as informing the road

104  Part II •  Designing and Conducting Mixed Methods with Secondary Data map for how you will critically evaluate the more restricted information from the secondary quantitative sample. Sampling procedures for the secondary quantitative data may include but are not limited to random sampling, simple random sampling, or quota sampling (Clark, Foster, Sloan, & Bryman, 2021; Grinnell, Gabor, & Unrau, 2012; Watkins & Gioia, 2015). Evaluate the sampling plan to determine which ones the original authors used to address their research question. For example, let’s assume you have access to a nationally representative sample of college students via the Healthy Minds Study (HMS; Lipson & Eisenberg, 2018). The HMS is a robust survey with mental health and help-seeking items delivered annually to college and university students. You might be interested in a study where you collect new focus group data from college students and then use the themes derived from those data to test a model using the secondary HMS data. Or you could do a content analysis of how mental health is discussed among focus group participants to see if the relationships they discussed are also present in a factor analysis of the same (or similar) variables in the HMS. The sample for your focus groups might include students from a local college, but you could analyze national data from HMS respondents who fit the same demographic profiles as the local students.

Qualitative data collection tools when the quantitative data exist The instruments available in secondary quantitative data can be maximized if the data have diverse variables and a broad scope. This can make mapping the findings from the new qualitative data onto the secondary quantitative data source easier. qualitative instruments. The measures for an exploratory sequential design with secondary quantitative data often begin with a researcher who wants to generate some language and meaning for a topic. They then want to map the meaning of these findings onto a larger, secondary quantitative data source to see if they can achieve generalizability. Again, it is fine to acknowledge that the questions developed for a new qualitative instrument will be strongly influenced by evaluating the secondary quantitative data for an exploratory sequential design. This is because it might be unrealistic to do an exploratory sequential design with secondary quantitative instruments without looking at them before developing the new qualitative instruments. My advice is that though the new qualitative study may happen alongside your review of the secondary quantitative data and methods execute the two studies as intended: in sequence.

It might be unrealistic to do an exploratory sequential design with secondary quantitative instruments without looking at them before developing the new

Example of an exploratory sequential design with secondary quantitative data Case Study 5.2 illustrates an exploratory sequential design with secondary quantitative data. In this study, Cabrera (2011) employed an exploratory

Chapter 5  •  Exploratory Sequential Design With Secondary Data  

105

sequential design with secondary quantitative data to examine white college men’s racial ideologies and the experiences that influence how they form their racial ideologies. First, new qualitative data were collected and analyzed, providing themes around white college men’s racial ideologies across two colleges. Then, Cabrera used a secondary survey collected at one of the two colleges to test the themes generated from the new qualitative data he collected on white college men. Integrating the data allowed the author to highlight racial ideologies and privilege among white college men and explore how mixed methods and intersectionality can be used to research college students (Figure for Case Study 5.2).

CASE STUDY 5.2 EXPLORATORY SEQUENTIAL DESIGN WITH SECONDARY QUANTITATIVE DATA Reference:

Cabrera, N. L. (2011). Using a sequential exploratory mixed methods design to examine racial hyper-privilege in higher education. New Directions for Institutional Research, 151, 77–91.

Purpose:

To discuss a mixed methods approach to studying the impact of college environments on white male college students’ development of racial ideologies.

Research Question:

What is the dominant racial ideology of white male college students? What college experiences affect their development?

MMR Design: Exploratory sequential with secondary quantitative data New Data Source:

New Qualitative Data

New Data Source Contribution:

The qualitative data created four frames of participants’ racial ideologies for the author to explore quantitatively.

New interview data with 28 white male students across two colleges with a range of political orientations.

Secondary Secondary Quantitative Data Data Source: Secondary survey data from one of the colleges were collected by a team (Cabrera was a member). This survey was collected in two waves—once at the beginning of the academic year and a second time during the middle of the spring semester. Of the 593 total respondents who completed both surveys, 104 were white men. Secondary Data Source Contribution:

It allowed the researcher to explore whether the four frames white college men identified during the qualitative component are generalizable to a larger population.

Goals of Integration:

To identify either hierarchy-enhancing (in support of inequality) or hierarchy-attenuating (in support of egalitarianism) racial ideologies among white college students by first establishing ideologies germane to white male college students and then testing the generalizability of those ideologies. (Continued)

106  Part II •  Designing and Conducting Mixed Methods with Secondary Data

(Continued) New Knowledge Generated:

The results of the analysis (according to the author) implied that by leaving white male undergraduates insufficiently challenged by their racial selves during their first year of college, the institution inadvertently was helping perpetuate and support systemic racism.

FIGURE FOR CASE STUDY 5.2 Purpose: To discuss a mixed methods approach to the study of the impact of college environments on white male college students’ development of racial ideologies Research Questions: N/A Collected new QUALITATIVE interviews from 28 White male students across two colleges Completed sampling, instruments, and procedures for a new QUALITATIVE data collection Analyzed new QUALITATIVE data QUALITATIVE Results: Four racial ideologies for White male students Data integrated and determined the need for secondary QUANTITATIVE data to test and attempt to generalize the racial ideology frame with a sample of White male students

Use results from themes generated from new QUALITATIVE data to make decisions about samples, measures, and procedures for secondary QUANTITATIVE data

Reviewed secondary QUANTITATIVE data on two waves of student data at one of the two colleges sampled for the qualitative component. Of the 593 respondents who completed both waves of the survey 104 were White men Used factor analysis, ANOVA, Scheffé’s test, and ordinary least squares regression

QUANTITATIVE Results: Racial ideology construct held for white male students, who also had high racial ideologies Data Integration: The new QUALITATIVE data led to construct development and testing using a QUANTITATIVE instrument

Review how QUALITATIVE results led to the development of QUANTITATIVE results

The intersection of being White and male was strongly related to subscribing to hierarchyenhancing\racial ideologies, which is aligned with existing research

Source: Based on Cabrera, N. L. (2011).

Chapter 5  •  Exploratory Sequential Design With Secondary Data  

107

Exploratory Sequential Designs With Secondary Qualitative and Quantitative Data When the qualitative and quantitative data for an exploratory sequential design exist, the process for designing the study involves critically reviewing both secondary data sources and then generating a list of the design possibilities for the study’s sample, measures and instruments, and study procedures. I covered how to handle qualitative instruments and quantitative measures in the first two examples of exploratory sequential designs with secondary data, so I do not cover those here. Instead, I focus on the sampling plan and procedures for an exploratory sequential design with secondary qualitative and quantitative data.

Sampling for an exploratory sequential design when both qualitative and quantitative data exist The sampling plans for an exploratory sequential design with secondary qualitative and quantitative data look different from compared to a traditional exploratory sequential design. The major difference is that you will need to evaluate secondary sampling plans for two different study designs: the secondary qualitative data and the secondary quantitative data. This evaluation involves reviewing the study protocol, noting how the original study team made decisions about who to sample for the qualitative and quantitative studies and why, and deciding which samples (if different or multiple samples were used across both the qualitative and quantitative data sources) will address the research question for your mixed methods study. Depending on the original study team’s question, the samples for your study may exclude certain cases, groups, or units of data. Therefore, you should have a clear sense of how the original samples were determined and then decide if all the cases used in the original sample need to be repurposed for your new project. I want to reiterate the importance of revisiting the research question while developing the sampling plans for your exploratory sequential study with secondary data. When I feel distracted by a secondary data source, I take a step back from my data mining to dissect and reaffirm my research question. I do this by (1) writing my research question on a sheet of paper; (2) taking 5 minutes to focus on the topic, sample, method, and intended outcome as they are presented in my research question; (3) placing the words from my research question When I feel distracted by that represent each of these study components on separate a secondary data source, I post-it notes; and (4) placing take a step back from my data them along the bottom of my mining to dissect and reaffirm computer monitor in the same my research question. order in which they appear in my research question. This

108  Part II •  Designing and Conducting Mixed Methods with Secondary Data usually reminds me that my original purpose for using secondary data is not to “fish” in uncharted waters but instead to use my current research question as a net for extracting clams with potentially unknown pearls of knowledge.

Example of an exploratory sequential design with secondary quantitative and qualitative data Case Study 5.3 illustrates an exploratory sequential design with secondary quantitative and qualitative data. In this study, Watkins, Wharton, Mitchell, Matusko, and Kales (2017) used an exploratory sequential design with secondary qualitative and quantitative data to explore the role of non-spousal family support on the mental health of older, churchgoing African American men. First, secondary focus group data were used to develop a conceptual framework for non-spousal family support for older African American men. Second, these qualitative concepts were mapped onto a secondary, nationally representative data source that included a sample of 401 older African American men. Integrating the two secondary data sources helped the authors examine the impact of family support, beyond spouses and female partners, on churchgoing African American men aged 50 and older (Figure for Case Study 5.3).

CASE STUDY 5.3 EXPLORATORY SEQUENTIAL DESIGN WITH SECONDARY QUALITATIVE AND QUANTITATIVE DATA Reference:

Watkins, D. C., Wharton, T., Mitchell, J. A., Matusko, N., & Kales, H. (2017). Perceptions and receptivity of non-spousal family support: A mixed methods study of psychological distress among older, churchgoing African American men. Journal of Mixed Methods Research, 11(4): 487–509.

Purpose:

The purpose of this study was to explore the role of non-spousal family support on mental health among older, churchgoing African American men.

Research Question:

What are the social and cultural experiences of non-spousal family support for older, churchgoing African American men who experience psychological distress?

MMR Design:

Exploratory sequential design with secondary qualitative and quantitative data

Chapter 5  •  Exploratory Sequential Design With Secondary Data  

Secondary Data Source #1:

Secondary Qualitative Data

Secondary Data Source #1 Contribution:

The authors analyzed secondary focus group data with older African American men to understand their experiences with depression and psychological distress.

Secondary Data Source #2:

Secondary Quantitative Data

Secondary Data Source #2 Contribution:

The authors based their decisions on operationalizing the secondary quantitative data (i.e., selecting survey items and building statistical models to test) on the conceptual model they made from the secondary qualitative data.

Goals of Integration:

The authors made decisions about one data component based on the findings of the other. The two data sources were also sequenced, with the secondary qualitative data source being evaluated and analyzed first, followed by the review, evaluation, and analysis of the quantitative data source. Results suggested that non-spousal family support was protective against psychological distress for older, churchgoing African American men.

New Knowledge Generated:

The data integration allowed for a deeper exploration of the impact of family support on older, churchgoing African American men that extends beyond their spouses and female partners. This is important given the role of family members and the church in the lives of older African American men and their experiences with mental health challenges.

The Depression Care in African American Church Elders Study (aka “The Churches Study”) is a qualitative study of focus groups conducted with 21 older (ages 50 and older), churchgoing African American men in Michigan.

The National Survey of American Life (NSAL, 2001–2003) is the most comprehensive study of mental disorders and the mental health of Americans of African descent ever completed. The quantitative sample included 401 older (ages 50 and older), churchgoing African American men.

109

110  Part II •  Designing and Conducting Mixed Methods with Secondary Data FIGURE FOR CASE STUDY 5.3 Purpose: The purpose of this study was to explore the role of non-spousal family support on mental health among older, church-going African American men Research Questions: What are the social and cultural experiences of non-spousal family support for older, church-going African American men who experience psychological distress? Reviewed secondary QUALITATIVE data from 21 older, church-going African American men Reviewed sampling, instruments, and procedures for secondary QUALITATIVE (focus group) data AAnalyzed secondary QUALITATIVE data using the RADaR technique QUALITATIVE Results: Three overarching themes and six subthemes Data integration determined the need for quantitative data because a QUALITATIVE conceptual framework was built that needed to be tested QUANTITATIVELY Reviewed secondary QUANTITATIVE data from the National Survey of American Life (NSAL). These data included 1,217 African American male respondents, of which 401 were aged 50 and older

Used results from themes generated from secondary QUALITATIVE data to make decisions about samples, measures, and procedures for new QUANTITATIVE data

Used confirmatory factor analysis and measures/indicators of model fit

QUANTITATIVE Results: A good model fit was determined of the data to the hypothesized conceptual structure Data Integration: The secondary QUALITATIVE data led to construct development and testing using a secondary QUANTITATIVE instrument

Reviewed how QUALITATIVE results led to the development of QUANTITATIVE results

Churches alone may not provide the kind of mental health support needed by some groups of older, church-going African American men. Under such circumstances, these men may benefit from stronger connections with non-spousal family support.

Source: Watkins, Wharton, Mitchell, Matusko, and Kales (2017).

Challenges (and Solutions) When Using Secondary Data in the Exploratory Sequential Design Throughout the chapter, I have noted potential challenges with implementing an exploratory sequential design where one or both data sources exist. These challenges can occur as early as the planning stage for your mixed methods

Chapter 5  •  Exploratory Sequential Design With Secondary Data  

111

study, while others may occur during the later stages of the project. Regardless, it is a good rule of thumb to anticipate these challenges and have a plan for how to address them. In this section, I describe two challenges that may be a direct byproduct of the exploratory sequential design: (1) having new qualitative data that do not map well onto the secondary quantitative data, and (2) having secondary qualitative data that do not map well onto the secondary quantitative data. Solutions for how to address each are in the subsequent paragraphs. For an exploratory sequential design with secondary data, you may develop new qualitative data that do not map well onto the secondary quantitative data. If this happens, you can either: (1) be more creative with mapping the new qualitative findings onto the secondary quantitative data and files, or (2) consider finding a different quantitative data source. Mapping one data source onto another is not always an apples-to-apples match. Instead, see if other characteristics of the new qualitative component (e.g., sampling plan, instruments, theory, etc.) can be mapped onto the secondary quantitative data. For example, one strategy I use when integrating new qualitative data with secondary quantitative data is simply not anticipating a perfect apples-to-apples match. Instead, I focus on my qualitative findings and my understanding of the research topic, and I think about how the themes I generated present themselves (or not) in the secondary quantitative data. The unaligned characteristics you uncover while integrating the qualitative and quantitative studies are just as informative as those that are aligned. So the goal should not be for a perfect apples-to-apples match but for the new qualitative data to enhance the lens through which you see the secondary quantitative data. During integration—mainly, when decisions about the secondary quantitative data are made based on what you found from the new qualitative data—the real beauty of using secondary data occurs. What I mean is the point of interface (i.e., integration) is where your creative side can surface. The exploratory sequential design integrates qualitative results into the quantitative component’s samples, measures, and procedures. However, bringing these two components together should be more of an art than a science. In my opinion, the so-called rules of mixed methods research are all up for debate. This is because so much is still unknown about how qualitative and quantitative data can come together to address our research questions. So I encourage you to consider yourself a pioneer as you explore the uncharted territory of mixed methods using secondary data. Creativity can occur when you see the secondary quantitative methods and data through a lens influenced by the results from the new qualiConsider yourself a pioneer tative data. Look for opportunias you explore the uncharted ties that may seem impossible territory of mixed methods using at first but then become what secondary data. strengthens your use of secondary data in mixed methods.

112  Part II •  Designing and Conducting Mixed Methods with Secondary Data Suppose you have made several attempts at mapping new qualitative findings onto a secondary quantitative study and experienced more problems than solutions. In that case, you may want to either locate another secondary data source on which you can map your new qualitative results or collect new quantitative data. Remember: do not force a secondary data source into your study if it does not fit. Your primary goal should be to address your research question by any means necessary, even if that means altering the original plan for what data you use to accomplish your research goals. For the sake of a study that can more closely address your research question, choosing a different data source or collecting new data altogether is worth it. A good rule of thumb for an exploratory sequential design with secondary quantitative data is to preview (not analyze) the secondary quantitative data first and then work iteratively as you plan for the new qualitative component. This can help you identify which concepts from your qualitative inquiry can be examined using secondary data from your quantitative investigation. During your exploratory sequential design with secondary data, a second challenge you may experience is secondary qualitative data that do not map well onto the secondary quantitative data. Under these circumstances, you will need to determine if: (1) the two secondary data sources, when combined sequentially, help address your research question or (2) the two secondary data sources should serve as separate “sister studies” that contribute to your overall understanding of the research topic. Option 1 requires reassessing the need for a sequential design to address your research question and determining if your selected secondary data sources can help you reach your goal. You could also decide if a different (or new) data source is needed for the first qualitative or the second quantitative study component. Meanwhile, option 2 requires reexamining the need to keep the two study components together in either a sequential or convergent mixed methods study or separating them into two single-method studies to address your research question. Regardless of what you decide, never lose sight of your research question, as it will keep you focused on the end goal and help you determine the best roadmap for getting there (whether it requires new or secondary data). Never force two secondary data sources together; two data sources should connect in a way where they complement one another and help you address your research question. The fit may not feel natural, but it should make sense for your mixed methods purpose and research question. The exploratory sequential design with two secondary data sources may be the most intimidating of all secondary data mixed methods designs. Using two data sources collected by other researchers may make some scholars feel powerless in their research inquiries. Do not avoid using secondary data because you think you may look less like the researcher others believe you should be or because you believe you have less control over the procedures. Some of the most skilled and productive mixed methods scholars are those who can extract knowledge from familiar data sources rather than always collecting new data. As a mixed methods researcher you might not need to create new methods for

Chapter 5  •  Exploratory Sequential Design With Secondary Data  

113

the topic under study but rather ask yourself: How can the methods used to collect the qualitative and quantitative data be repurposed to expand my topic? How will secondary data sources add to what is currently known about the topic? What are the challenges/limitations to using secondary data to address my research question?

Summary This chapter defined the features of the exploratory sequential design and described how to use secondary data for one or both study components. By now, you should know that, unlike a convergent design where the two data sources are handled separately, the decisions you make about the quantitative study component are direct reflections of what you learn from the qualitative study component with an exploratory sequential design. You now know the 12 steps to prepare secondary qualitative and quantitative data for an exploratory sequential design. You also understand that analysis and integration for an exploratory sequential design with secondary data involve an evaluation, followed by a process of sifting, sorting, and categorizing the data. If the quantitative data exist, this might include collecting new qualitative data and then mapping the themes and subthemes onto a secondary survey instrument. Finally, you are prepared to handle the challenges with employing an exploratory sequential design with secondary data, including having new qualitative data that do not map well onto secondary quantitative data and secondary qualitative data that do not map well onto secondary quantitative data.

Chapter 5 Application Questions 1. What key features of the exploratory sequential design might make using secondary data different from using secondary data with a convergent design? Make a list of how the key features of the exploratory sequential design were advantageous to the authors in Case Study 5.3. 2. What are the 12 steps for preparing secondary quantitative and qualitative data for an exploratory sequential design? Create a table like Table 5.1 for your exploratory sequential design. Keep the left column of information from Table 5.1 but fill in the right column with information associated with your study.

114  Part II •  Designing and Conducting Mixed Methods with Secondary Data

3. How do the steps for an exploratory sequential design differ when the qualitative data exist compared to the quantitative data? Compare Case Studies 5.2 and 5.3 from this chapter. How did the authors benefit from secondary qualitative data in Case Study 5.1 compared to how the authors from Case Study 5.3 benefited from their new qualitative data? 4. What are two ways to integrate data for an exploratory sequential design with secondary data? How do these apply to your exploratory sequential design? 5. How do the challenges to employing an exploratory sequential design with secondary data play out in Case Study 5.1? How did the authors go about resolving them?

6 Explanatory Sequential Design With Secondary Data Imagine Your best friend Jaxon is a “numbers guy.” Or at least that is what he calls himself. He works for a policy firm, and his primary responsibility is to run statistics for statewide policymakers to determine how resources are allocated to specific regions in the state. During lunch one day, he told you about one of his projects in which the findings were incredibly skewed compared to his supervisor’s expectations. “I have seen a lot of strange things over the years,” said Jaxon, “but this one tops them all.” You were intrigued by this and asked why his firm did not do some follow-up interviews or focus groups with communitydwellers in the region to see if their thoughts aligned with what he found from the statistics. Jaxon responded with an empathetic, “I don’t know. . . . Do you want to get dessert before we leave? I need to head back to the office in a few minutes.” While your friend was focused on the caramel flan that afternoon, you were focused on the follow-up. If hearing from people could enrich his findings, why wouldn’t he take that next step? This makes perfect sense to you. If you understand the importance of the “voices behind the numbers,” then this chapter is for you.

115

116   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah

Learning Objectives This chapter reviews the explanatory sequential design and discusses how secondary data can be used for one or both study components. By the end of the chapter, you will be able to: 1. Define the features of an explanatory sequential design, 2. Describe how to bring secondary data into an explanatory sequential design, 3. Analyze and integrate data for an explanatory sequential design, 4. Plan and implement an explanatory sequential design with secondary quantitative data, secondary qualitative data, and secondary qualitative and quantitative data, and 5. Identify potential challenges (and solutions) when using secondary data in an explanatory sequential design.

Features of an Explanatory Sequential Design The explanatory sequential design is one of the three core mixed methods designs (Creswell, 2015; Plano Clark & Ivankova, 2016). It involves collecting and analyzing quantitative data, then using the results from the quantitative data to make decisions about data collection and analysis for the second qualitative study. Researchers select an explanatory sequential design to gain better insight into the quantitative data using the qualitative data and to qualitatively explain what they discovered quantitatively (Creswell & Plano Clark, 2018; Watkins & Gioia, 2015). So, just as a qualitatively oriented researcher might gravitate toward the exploratory sequential design (Chapter 5), a quantitatively oriented researcher might gravitate toward the explanatory sequential design I cover in this chapter. Decisions are made about the qualitative data based on what you find quantitatively in an explanatory sequential design. Therefore, the qualitative data and methods act as a follow-up to what you determine from analyzing the quantitative data. So the quantitative and qualitative data are not utilized apart from one another, but rather together, as the results from the quantitative component are used to build or explain the second qualitative component for this core mixed methods design. The quantitative and qualitative results from the two study components are not compared at the end of the study (like you would for the convergent design I covered in Chapter 4). Instead, the explanatory sequential design allows the researcher to begin their inquiry by acquiring statistical breadth (achieved through the quantitative study) and then going deeper into experiences, perceptions, and context (achieved through the qualitative study).

Chapter 6  •  Explanatory Sequential Design With Secondary Data   

117

A key feature of the explanatory sequential design, much The explanatory sequential like the exploratory sequential design lets you begin by design, is the timing of the study components. The quanacquiring statistical breadth titative study is completed and then going deeper first, which means you begin into human experiences, your mixed methods inquiry perceptions, and context. under the post-positivism assumption. If you are more quantitatively oriented, this will feel like a natural move toward what you already know, but then use what you find to build a qualitative study. However, for more qualitatively oriented researchers, positioning the first part of your study as quantitative may mean you need to prioritize a different set of principles in your thinking. For example, rather than focusing on the experiences and explanations expressed by study participants, you will need to begin by focusing on the frequencies, rates, and associations between variables as demonstrated by the items study participants respond to in a survey. If you are more qualitatively oriented, my advice is to acknowledge the shift in assumptions you need to employ with an explanatory sequential design. Journal about the challenges you might face (I cover these later in this chapter) and continue to seek advice and guidance from colleagues and trusted mentors.

Bringing Secondary Data Into the Explanatory Sequential Design Regardless of your methods orientation (i.e., qualitative vs. quantitative), there are several reasons why you should consider using secondary data in an explanatory sequential design. Beyond the money- and time-saving advantages noted in previous chapters, using secondary data in an explanatory sequential design can also expand your thinking for a research topic using data previously collected by you or someone else. For example, let’s say that while volunteering at your local municipal administrative offices, you stumble across a publicly available data set that outlines voter registrations in your county. You ask your supervisor whether these data have ever been analyzed, and she laughs. “Unfortunately, no. But if this is something you are interested in exploring, be my guest!” What an excellent opportunity to analyze voter registration data across your county by age, race, gender, and zip code! After you generate some descriptive and inferential statistics across these demographics, you might even be able to convince your supervisor to provide some funding for follow-up interviews with residents to gauge their reactions to the voter registration demographics in their county. This is an example of an explanatory sequential mixed methods design using secondary data, much

118   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah like Figure 6.1. As you see whether the quantitative data or the qualitative data exists before the start of your study, you will need to make decisions along the way (e.g., concerning sampling, measures/­instruments, and analyses) about how to best use those data to enhance your understanding of the research topic.

FIGURE 6.1  ● Explanatory Sequential Design With New and Secondary Data Formulate a research question

Select secondary (or collect new) QUANTITATIVE data Review secondary QUANTITATIVE data using evaluation criteria (or complete sampling measures/instruments, and procedures for new QUANTITATIVE data collection) Analyze secondary (or new) QUANTITATIVE data Produce QUANTITATIVE results

Integrate the data and determine the need for secondary (or new) QUANTITATIVE data

Use results from QUANTITATIVE data to make decisions about QUALITATIVE samples, measures, and procedures

Review secondary QUALITATIVE data using evaluation criteria (or complete sampling, instruments, and procedures for new QUALITATIVE data collection) Analyze secondary (or new) QUALITATIVE data Produce QUALITATIVE results

Data integration

Generate findings and overall conclusions

Review QUANTITATIVE results and how they compare to the QUALITATIVE results

Chapter 6  •  Explanatory Sequential Design With Secondary Data   

119

TABLE 6.1  ● Preparing Secondary Quantitative and Qualitative Data for an Explanatory Sequential Design

Step

Steps for Preparing Quantitative and Qualitative Data

Relevance to the Explanatory Sequential Design

1

Decide on the kind of data you will need for the first quantitative component for the study.

Though the pacing for this design is clear, your decision for whether to use secondary data or collect new data for the first, quantitative component will depend on the research question, access to secondary data sources, and how you plan to integrate what you find from the quantitative data into the decisions you make for the qualitative component.

2

If using secondary quantitative data, review the secondary data for topic fit and alignment. (If collecting new data, design the quantitative component.)

If using secondary quantitative data, review the data paying close attention to the study sampling, measures, and procedures. Next, determine what you already know (if using secondary quantitative data) or what you hope to learn from the quantitative study (if collecting new data). This step will also determine if the secondary quantitative data can complement the new qualitative data (if applicable) and if they both help address the research question. During your evaluation, take an inventory of the secondary data to assess what is available, missing, and incomplete to determine topic fit and alignment with the research question.

3

If using secondary quantitative data, evaluate the quantitative data using the review criteria. (If collecting new data, design the quantitative study.)

The evaluation criteria from Chapter 3 will assist you in determining (a) how you want to manage the samples from the secondary quantitative data (if applicable), (b) what about the secondary quantitative data can help you answer your research question; and (c) what is missing from the quantitative data that can be supplemented by collecting new (or using secondary) qualitative data. Try to access complete data files before deciding whether to use them in your study. (Continued)

120   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah TABLE 6.1  ● (Continued)

Step

Steps for Preparing Quantitative and Qualitative Data

Relevance to the Explanatory Sequential Design

4

Develop an analysis plan for the secondary (or new) quantitative data.

If using secondary quantitative data, the analysis plan may (or may not) be complementary to the plan proposed by the original researchers; this step should occur iteratively with step 3. Try to understand how the original researchers made decisions about their quantitative methods. Seek references and support for the statistical analysis plan if needed.

5

Analyze the secondary (or new) quantitative data.

With an explanatory sequential design, the timing for your use of the second qualitative data source is contingent upon your use of the first quantitative data source. Your quantitative analysis will generate statistical findings that you can use to make decisions about the second qualitative study.

6

Integrate by making decisions about the kind of data (new or secondary) you will need for the second qualitative study based on what you find in the first quantitative component.

There are at least two interface points (i.e., where integration happens). This is the first one, where the quantitative results were used to decide the qualitative methods. A close review of your quantitative results (from new or secondary data) may include summarizing your findings and mapping them onto a new (or secondary) qualitative study. Making sense of how the quantitative results can prompt decisions for the new (or secondary) qualitative methods is critical.

7

If using secondary qualitative data, review the secondary data for topic fit and alignment. (If using new qualitative data, design the qualitative component.)

If using secondary qualitative data, develop a log for the secondary data, if/how the original researchers used it, and if/how you will use it for your explanatory sequential design. Review secondary qualitative methods (or plans for new qualitative data collection) for appropriateness and likelihood of addressing the research question. If using new data, design the qualitative study sampling, instruments, and procedures based on the results from the quantitative component.

Chapter 6  •  Explanatory Sequential Design With Secondary Data   

Step

Steps for Preparing Quantitative and Qualitative Data

121

Relevance to the Explanatory Sequential Design

8

If using secondary qualitative data, evaluate them with the evaluation criteria. (If using new qualitative data, collect it.)

With the explanatory sequential design, results from the secondary (or new) quantitative data will influence selecting the secondary (or new) qualitative methods. Therefore, use the evaluation criteria from Chapter 3 to assess the secondary qualitative study.

9

Develop an analysis plan for the secondary (or new) qualitative data.

If using secondary qualitative data, the analysis plan may (or may not) complement the plan proposed by the original researchers. If using new data, develop a robust analysis plan that includes systematic procedures and measures of rigor. The analysis plan should address the research question.

10

Analyze the secondary (or new) qualitative data.

Use single-method qualitative analysis techniques to produce secondary (or new) qualitative results.

11

Integrate the methods by reviewing the quantitative results and how they compare to the qualitative results.

There are at least two interface points (i.e., where integration happens). This is the second one, how the quantitative results compare to the qualitative results overall. Beyond the decisions you made for the qualitative study based on the quantitative study, what did the quantitative results generate compared to the qualitative results?

12

Generate findings and conclusions.

The results can be used to inform future research, practice, and policy. Similarly, what you learn from integrating the methods can inform new or secondary data in future projects and build additional studies.

Preparing Secondary Data for an Explanatory Sequential Design An explanatory sequential design with secondary data looks like a traditional explanatory sequential design. The difference is that you will use secondary data for one or both study components rather than generate two new data sources. For the study component (quantitative or qualitative) with secondary data, you will need to use the evaluation criteria I outlined

122   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah in Chapter  3 of this book. You can also create a diagram or figure (i.e., ­Figure 6.1) to help illustrate the two study components and how you plan to lay out the steps and integrate the results from the quantitative study with the subsequent qualitative study. Table 6.1 explores the steps to prepare secondary quantitative and qualitative data for an explanatory sequential design.

Data Analysis and Integration for an Explanatory Sequential Design With Secondary Data Data analysis and integration for an explanatory sequential design with secondary data may feel comfortable if you are more quantitatively oriented. This is because running both descriptive and inferential statistics is the first step of the explanatory sequential design. Then, based on the findings you generate from the quantitative analysis, you will need to decide how to analyze the secondary (or new) qualitative data. For example, let’s assume after you analyze the secondary quantitative data, a regression model found positive associations between race and mental health coping styles among Black, White, and Native American adolescents. If your secondary qualitative data source includes focus groups with Black, White, and Native American adolescents about mental health treatment and outcomes, your review of those data could focus on how the adolescents described mental health coping styles. You may also identify differences in how the members from each racial group discussed their individual and group coping styles.

Analyzing Data and Integrating the Results Data analysis and integration for the explanatory sequential design allows you to generate quantitative results that you can use to prepare and implement a follow-up, qualitative study (Creswell & Creswell, 2018). Whether you decide to use secondary data for one or both components of your explanatory sequential design, analyzing the data will involve considering the types of statistical tests to run during the first quantitative component and then connecting the findings from those statistical tests to the second qualitative component. Researchers usually begin their quantitative data analysis by running descriptive and inferential statistics to determine some basic information about the variables used in their quantitative research and determine if relationships exist between those variables, respectively. Quantitative results help inform sampling, instruments, and procedures for an explanatory sequential design, whether secondary data are used or not. For example, let us assume your secondary quantitative results were generated from a big data source (see the Big Data Break 6.1 for a description of big data analytics). In this example, your big data source is banking data from all the Federal Deposit Insurance Corporation (better known as “FDIC”)

Chapter 6  •  Explanatory Sequential Design With Secondary Data   

123

banks on the west coast of the United States. For your quantitative analysis, you want to determine how Black, Indigenous, and People of Color (BIPOC) young adults experienced a decrease in their savings account trends due to the COVID-19 pandemic of 2020. To generate results, you may need to run regression analyses to identify trends in reduced balances pre- compared to post-COVID-19. Based on your findings, you can determine how to choose a sample, choose instruments, and plan your data collection procedures for a follow-up qualitative study that explores why young BIPOC adults might have experienced decreases in their savings accounts post-COVID-19. This is an example of how your quantitative results can inform the sampling plan, instruments, and procedures for an explanatory sequential design with secondary quantitative data. Furthermore, your findings could inform future FDIC-protected bank policies and procedures and systemic ways for protecting the savings accounts for young BIPOC adults amid a pandemic or global economic crisis. I will not detail how to analyze single-method quantitative and qualitative data here because these steps have been covered elsewhere (Grbich, 2013; Miles, Huberman, Saldana, 2019; Ott & Longnecker, 2015). However, I will cover how to think about the analysis and integrate the results for your explanatory sequential design with secondary data. First and foremost, data analysis and integration for an explanatory sequential design begins by asking yourself: How can I make decisions about the (new or secondary) qualitative component

BIG DATA BREAK 6.1 WHAT ARE BASIC BIG DATA ANALYTICS? Basic big data analytics are analysis techniques you can use if you are unsure about what is in the big data, but you know you have something valuable that requires further exploration (Hurwitz et al., 2013). You might begin this inquiry by creating simple diagrams or graphics for your big data or running descriptive statistics to determine your sample’s basic demographics. Basic big data analytics are most frequently used when you have large amounts of disparate data. The reconciliation of these data can be achieved by running the following basic analytic techniques for descriptive statistics: ••

•• ••

Slicing and dicing: Breaking down the data into smaller sets of data that are easier to analyze. Graphs and plots help explore data across different dimensions. Basic statistics such as averages and medians can help determine what you have. Basic monitoring: Monitoring large volumes of data in real-time. Anomaly identification: The act of identifying anomalies, or outliers, such as when the actual observations differ from what you expected.

124   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah based on what I find in the (new or secondary) quantitative study? As you can imagine, the process and pacing used to analyze data for an explanatory sequential design can vary depending on whether the quantitative data exist, the qualitative data exist, or both exist. If the quantitative data exist for your explanatory sequential design, the analysis and integration might involve an evaluation of the secondary data (see Chapter 3), followed by developing an analysis plan, analyzing the quantitative data, then integrating the results from the secondary quantitative data into a new qualitative study. For example, let’s assume you have access to secondary Qualtrics surveys from 450 therapists who responded to questions about U.S. climate, culture, and the most frequent Diagnostic and Statistical Manual of Mental Disorders (DSM-5) diagnosis they assign to their older (65 and older) adult clients. Your survey also includes items about the therapists’ demographics (e.g., age, race, ethnicity, gender, educational level, discipline, household income, etc.). Given your interest in how therapists’ identities and backgrounds influence the most frequent diagnoses they assign to their older adult clients, you decide to employ an explanatory sequential design using your secondary Qualtrics survey data and new qualitative data. After running a few descriptive statistics, you realize the best way to collect your qualitative data from the therapists is to conduct one-on-one interviews via Zoom (i.e., a videoconference). During your Zoom interviews with the 20 therapists who agreed to be contacted for further discussion after they completed the previous Qualtrics survey, you learn more about their backgrounds, their disciplines (e.g., social work, psychology, psychiatry, etc.), how they became therapists, and how they determine an older adult’s diagnoses based on their symptoms. Once you analyze these rich qualitative data, you may generate a list of concepts (and later themes) from the interviews that you will want to go back and map onto the secondary quantitative instrument and statistical findings. Suppose the qualitative data exist and the quantitative data for your explanatory sequential design are new. In that case, the analysis may involve reviewing the secondary qualitative data to determine topic fit and alignment and then returning to the quantitative study to flesh out the study design. I imagine as you read this, you might be thinking, “Wait. . . You want me to look at the secondary qualitative data for the second study before I flesh For designs where you are out the details for the first, collecting new data in the new quantitative study?” The short answer is “yes.” For this first phase and using secondary important reason, any design data in the second phase of the where you are collecting new study, it behooves you to peek at data in the first phase and your secondary data source. using secondary data in the second phase of the study

Chapter 6  •  Explanatory Sequential Design With Secondary Data   

125

behooves you to peek at your secondary data source of interest not only to assess for topic fit and study alignment but also to see if what you hope to find from your new quantitative results are likely to inform the decisions for how you manage the secondary qualitative data. In other words, my advice is to scan the secondary qualitative data for your explanatory sequential design to see if you believe it is robust (enough) or will include the type of concepts that could potentially explain, or be built from, the results from your first, new quantitative study. Here, the intent is not to let the secondary data source guide your decisions for the first (new) data source. Instead, it might be beneficial to think about the explanatory sequential design with secondary qualitative data as an iterative design that begins with collecting new quantitative data that you plan to integrate into a secondary qualitative study. My advice on how to achieve this integration smoothly is to (1) prioritize the goal and intent of your quantitative research; (2) accept that the new variables you plan to analyze with the quantitative study will likely be influenced in some way by the secondary qualitative data concepts; and (3) allow this inspiration to help you achieve quantitative results from the first study that can be explained or enhanced by the secondary qualitative study. Thus, the explanatory sequential design with secondary qualitative data is an iterative mixed methods design. This is a practical way to think about the best use of your time and the fit of the secondary qualitative data you plan to integrate into the study. If both the quantitative and the qualitative data exist for your explanatory sequential design—much like the exploratory sequential design with secondary data from Chapter 5—the analysis will involve an iterative, combined process of sifting, sorting, and categorizing both data sources and mapping the results from one data source onto the methods and results for the other data source. The key to working with two secondary data sources for an explanatory sequential design is to remember that this mixed methods design was intended to be employed sequentially. So, while you may have access to both secondary data sources from the very beginning of the project, you may iteratively review them but work with (i.e., manage) the quantitative data first. For example, if you have access to secondary quantitative data (let us assume for the moment you have big data from the social media application, TikTok), you may begin by running some predictive modeling analytics (see Big Data Break 6.2) to determine if previous TikTok video posting behaviors of White, The explanatory sequential middle-class teenagers can design with secondary predict future posting trends. qualitative data is an iterative Then once you generate those mixed methods design. statistics, you might decide to extract relevant information

126   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah

BIG DATA BREAK 6.2 What Are Advanced Big Data Analytics? Advanced big data analytics can provide algorithms for a complex analysis of either structured or unstructured big data. These types of analytics may include sophisticated statistical models, machine learning, neural networks, text analytics, and other advanced data-mining techniques (Hurwitz et al., 2013). However, you might be most familiar with logistic regression and other regression models to achieve your advanced analytic goals. The following are examples of advanced analytical techniques for inferential statistics: ••

••

••

Predictive modeling: A statistical or data-mining solution consisting of algorithms and techniques used on data to determine future outcomes. Text analytics: The process of analyzing unstructured text, extracting relevant information, and transforming it into structured information that can be leveraged. Other statistical and data-mining algorithms: Advanced forecasting optimization, cluster analysis, or affinity analysis.

from the actual videos themselves using text analytics from the TikTok application. This additional level of analysis will provide some context for the statistical results determined from the predictive modeling analysis. When both data sources exist for an explanatory sequential design, your quantitative results can still provide a roadmap for reviewing, evaluating, and analyzing the qualitative data and comparing the results from the quantitative data with those of the qualitative data.

Interpreting the Results Data integration for an explanatory sequential design occurs (1) when the quantitative results from the first study are incorporated into making decisions about the methods for the second qualitative study, and (2) when quantitative results are compared with the qualitative results. So there are at least two points of interface for an explanatory sequential design, whether secondary data are incorporated into the design or not. Thus, when using this design, you will not only need to make sure you explain how the results from the quantitative study helped to make decisions about the methods for the subsequent qualitative study, but you will also need to be able to describe how the quantitative results compare to the qualitative results overall. Remember, the decisions you make about your qualitative study are directly linked to your quantitative methods and results, so being able to articulate how that connection occurs will strengthen your explanatory sequential design with secondary data.

Chapter 6  •  Explanatory Sequential Design With Secondary Data   

127

Planning and Implementing an Explanatory Sequential Design With Secondary Data This section describes designing and implementing an explanatory sequential design with secondary data for one or both study components. You can do this in at least three different ways; by incorporating secondary data into (1) the first quantitative study, (2) the second qualitative study, or (3) both studies. Each is discussed in the following sections.

Explanatory Sequential Design With Secondary Quantitative Data The overall intent of the explanatory sequential design is for the qualitative study to explain or help expand on findings from the first quantitative study. Given this, you must connect the quantitative results and the qualitative methods (e.g., sampling, measures, and procedures) for your explanatory sequential design with secondary quantitative data. Therefore, the decisions you make about your methods for an explanatory sequential design that uses secondary quantitative data should be guided by your research question, which you should determine at the beginning of the study (Figure 6.1). Remember, your research question is your roadmap; it will guide your decision to use new or secondary data for your mixed methods project. The research question is essential for the other core designs, and it is equally important for guiding the decisions for an explanatory sequential design with secondary data. For example, circling back to your research question for your explanatory sequential design could mean the difference between running multivariate regression models with mediating or moderating variables or running an exploratory factor analysis versus a confirmatory factor analysis. While both analyses could be helpful to your study, which one will help you answer your mixed methods research question? Furthermore, which statistical analysis will help you generate quantitative results that guide the decisions you plan to make for [Y]our research question is your new qualitative study? your roadmap; it will guide Similarly, reaffirming the research question can help your decision to use new or you develop the appropriate secondary data for your mixed sampling plan for the secmethods project. ondary quantitative data and operationalize the measures.

Sampling for an explanatory sequential design with secondary quantitative data Sampling procedures for an explanatory sequential design with secondary quantitative data involves a thorough review of the secondary quantitative data source (see Chapter 3 for evaluation criteria) before deciding on the study’s

128   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah second, new qualitative phase. A secondary quantitative data source may have sophisticated sampling procedures or very simple ones, but how you use this information will depend on how the population is defined and how individuals from that population were represented (i.e., the sampling frame) by the original study team. Were random samples drawn using complex or straightforward sampling plans (Grinnell & Unrau, 2018; Shrout & Napier, 2011)? Do they offer various stratification levels from which you can choose (e.g., individual, family, community, policy, etc.)? Will they require sample weights and strata? These are some of the questions you will need to address to decide how to maximize the secondary quantitative sample and how to benefit from the decisions made by the original study team regarding their original quantitative sample. My advice is to allow the sampling frame for the secondary quantitative data to guide your sampling plan for the new qualitative data source. For example, suppose the secondary quantitative sample includes hospital volunteers. In that case, you will need to determine if the qualitative data you want to collect needs to be based on the experiences of those hospital volunteers or if there is another level of information you need to know (i.e., from hospital administrators or patients) to answer your overall research question. You can then answer your sampling questions (i.e., who should I sample for my new qualitative study and why?) by determining which qualitative sample will help you answer your overall research question.

Qualitative data collection instruments when the quantitative data exist A critical review of the measures for a secondary quantitative data source in an explanatory sequential design and the variable characteristics will influence your ability to proceed with the qualitative study using what you find during the quantitative study. Remember you did not collect the quantitative data, so focus your attention on studying the details of the secondary data source. Reviewing previous reports and scientific articles where the original data source was used, if applicable, can be insightful when using a secondary data source for the first time. In addition, it is beneficial to see how others described the data, which variables they highlighted and found useful, and how they discussed the validity and reliability of the measures under investigation. Different types of surveys (i.e., panel, longitudinal, etc.) can be used as an explanatory sequential design’s secondary quantitative data source. Whether limited in focus (e.g., topic-specific) or comprehensive (e.g., omnibus), you may find doing an explanatory sequential design with secondary quantitative data an efficient way to answer your research question. Big data are becoming popular for the first component of explanatory sequential designs and for answering mixed methods questions involving large samples. Like the other secondary data designs in this text (e.g., convergent designs, exploratory sequential designs, and complex designs), when both the qualitative and quantitative data phases exist, there is no need to go out and collect new data with new measures/instruments. Instead, an explanatory sequential design where both data sources exist will require a close review and evaluation

Chapter 6  •  Explanatory Sequential Design With Secondary Data   

129

of the measures used to collect the original quantitative and qualitative data. Though your review of the secondary data sources may seem to occur apart from one another, the integration occurs between the two data phases of this sequential design. Once the secondary quantitative data are evaluated and analyzed and results are produced, decisions about incorporating the secondary qualitative data will be made based on the secondary quantitative methods and results. I provide an example of this in case study 6.1.

CASE STUDY 6.1 EXPLANATORY SEQUENTIAL DESIGN WITH SECONDARY QUANTITATIVE DATA Reference:

Treviño, E., Scheele, J., & Flores, S. M. (2014). Beyond the test score: A mixed methods analysis of a college access intervention in Chile. Journal of Mixed Methods Research. 8(3): 255–265.

Purpose:

To examine the role of a college access program in the enrollment and persistence outcomes of low-income students in Chile, modeled partially after a Texas admissions program.

Research Questions:

None reported.

Secondary Data Source:

Secondary Quantitative Data

Secondary Data Source Contribution:

Secondary quantitative data helped gain valuable descriptive information about the program’s effectiveness in preparing economically disadvantaged students for college compared to students who entered through a regular admissions program.

New Data Source:

New Qualitative Data

New Data Source Contribution:

The qualitative data helped explain how the Propedéutico program gave tools that helped students ease their transition to university life.

Survey data from two random samples: one from students who enrolled in the University of Santiago de Chile (USACH) bachillerato program through Propedéutico (n = 183) and one from students who enrolled through the regular admissions process (n = 74).

Students (n = 15) participated in 15 semi-structured interviews (15–25 minutes each), covering topics such as the program’s effectiveness, students’ experiences during the college preparatory program, and students’ experiences once they entered college.

(Continued)

130   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah

(Continued) Goals of Integration:

The goal of integration was to allow the authors to consider the analyses’ agreement, complementarity, and discrepancy.

New Knowledge Generated:

This study supports culturally sensitive college preparatory programs as beneficial to students with lower GPAs and may face other significant barriers to their social and educational experiences.

FIGURE FOR CASE STUDY 6.1 Purpose: To examine the role of a college access program in the enrolment and persistence outcomes of low-income students in Chile, modeled partially after a Texas admissions program

Research Question: None reported

Select secondary QUANTITATIVE data from survey data from two random samples: one from students who enrolled in the University of Santiago de Chile (USACH) bachillerato program through Propedéutico (n = 183) and one from students who enrolled through the regular admissions process (n = 74)

Review secondary QUANTITATIVE data using evaluative criteria Analyze secondary QUANTITATIVE data QUANTITATIVE Results: At the end of H.S., students in the Propedéutico program had lower achievement than their counterparts who applied through the regular admission system Data integration determine the need for new QUALITATIVE data from students who could speak to their program experiences

Use results from QUANTITATIVE data to make decisions about QUALITATIVE samples, measures, and procedures

Complete sampling, instruments, and procedures for the new QUALITATIVE, semi-structured interview data collected from 15 students Analyze new QUALITATIVE data to gain a more complete picture of the effects of the program and general learning outcomes measures by GPA QUALITATIVE Results: Both Propedéutico program students and regular admission students reported they felt they were likely to finish college and they were confident in their capacity to excel in college Study findings were triangulated using a “convergence coding matrix” that allowed authors to consider convergence, complementarity, and discrepancy

Review QUANTITATIVE results and how they compare to the QUALITATIVE results

Though students in the Propedéutico program had lower GPAs than regular admission students, they wereable to persist in college and believed the program offered tools that eased their difficult transition to university life

Source: Based on Trevino, Scheele, & Flores (2014)

Chapter 6  •  Explanatory Sequential Design With Secondary Data   

131

Example of an explanatory sequential design with secondary quantitative data Case Study 6.1 illustrates an explanatory sequential design with secondary quantitative data. In this study, Treviño, Scheele, and Flores (2014) employed an explanatory sequential design with secondary quantitative data to examine the role of a college access program in the enrollment and persistence outcomes of low-income students in Chile. First, Treviño and colleagues used secondary survey data collected from the University of Santiago de Chile (USACH). Then they collected and analyzed new qualitative data that provided themes on the program’s effectiveness and students’ experiences during the college preparatory program and once they were attending college. Integrating the data in this way helped authors learn that despite the low-grade point averages (GPAs) from students who entered college through the program (upon entering college and after two semesters), the program students still met the institution’s academic requirements. The authors also included a convergence coding matrix in their article. This additional triangulation of the data allowed them to consider other aspects that may have influenced the achievement gap for their participants, such as motivation, personal challenges, and adaptation to college life (Figure for Case Study 6.1).

Explanatory Sequential Design With Secondary Qualitative Data The second type of explanatory sequential design with secondary data is the explanatory sequential design with new quantitative data and secondary qualitative data. This design is the opposite of the explanatory sequential design with secondary quantitative data described above. It differs in that the first quantitative phase of the design includes new data collection and analysis before moving onto the secondary qualitative data to explain or support the results produced by the new quantitative data. Therefore, different considerations must be made about the explanatory sequential design’s sample, measures, diagram, and procedures when the second study includes secondary qualitative data.

Sampling for an explanatory sequential with secondary qualitative data When sampling for an explanatory sequential design with secondary qualitative data, the first part of your study will undergo sampling procedures that model those of the traditional, core explanatory sequential design. In other words, the explanatory sequential design with secondary qualitative data will generate a quantitative data source by collecting data from a new sample to answer the research question. Quantitative sampling procedures have been covered in other research methods books (Bryman, 2016; Shrout & Napier, 2011), so I will not cover those here. However, for this type of explanatory sequential design, you have less flexibility with the qualitative portion of your

132   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah sampling plan, as those participants and those data have already been collected, and you are limited to what exists (unless you plan to collect new qualitative data to supplement what exists). The key to working with this type of explanatory sequential design is knowing the limitations of your secondary qualitative sample and using that information to determine how you develop the sampling plan for your new quantitative data. My advice is to interrogate your secondary qualitative sample to see if there are characteristics about the sample that could be included in the sampling plan for your new quantitative study. For example, knowing that your secondary qualita[I]nterrogate your secondary tive sample included caregivers qualitative sample to see of patients with dementia may mean your new quantitative if there are characteristics sample should include the careabout the sample that could be givers of patients with demenincluded in the sampling plan for tia. Of course, the final say on your new quantitative study. this will need to be guided by your research question.

Quantitative data collection instruments when the qualitative data exist The new quantitative instruments for the explanatory sequential design with secondary qualitative data can mimic the design for a core explanatory sequential design. Again, because you have creative control over how the quantitative sample is drawn and how participants are recruited, your only limitations will be those that are typical of single-method research; that is, the limitations you experience may be those associated with personnel, resources, and time (see Creswell & Plano Clark, 2018 and Watkins & Gioia, 2015 for other limitations). The new quantitative data you collect for your explanatory sequential design with secondary qualitative data can set you up for success by answering your research question and providing some direction for how you should proceed with incorporating the secondary qualitative data into the study. For the first step of this design, you have freedom and flexibility, so you may decide to use a secondary survey or scale for which validity and reliability have already been tested and confirmed. Or you may choose to create new psychometric measures that provide clarification about an understudied topic. Perhaps your new instrument is culturally sensitive and gender-appropriate for the participants you hope to recruit for the quantitative study. Regardless, there are benefits to initiating an explanatory sequential design with a quantitative study that you design and implement. As you move into the second qualitative component for this design, you might experience limitations with your secondary qualitative data. Remember, the quantitative data will provide the breadth of information about a topic for which you want to acquire some depth, using the secondary qualitative data.

Chapter 6  •  Explanatory Sequential Design With Secondary Data   

133

Example of an explanatory sequential design with secondary qualitative data Previous examples of explanatory sequential studies that use secondary qualitative data are challenging to locate, given the nature of the design. So, instead, let us consider a modified version of the Case Study 6.1 example. You may recall, in the Treviño, Scheele, and Flores (2014) article from Case Study 6.1, the authors reported an explanatory sequential design where they used secondary survey data collected from the USACH and then collected new qualitative data to understand the program’s effectiveness and students’ experiences during the college preparatory program and once they were in college. However, for Case Study 6.2, let’s swap these data sources. Rather than have secondary quantitative data and collect new qualitative data, let’s pretend Treviño and colleagues collected new quantitative data and that they plan to integrate it with some secondary qualitative data they have on file. The purpose of the original study by Treviño and colleagues (2014) could essentially stay the same as the published article, but the data sources would be different. The survey data from two random samples of students would be new. The semi-structured interviews collected from students about their experiences during the college preparatory program and their experiences once they entered college would be secondary data. The major differences between this modified example of Case Study 6.1 and the original study are which data existed first (quantitative in the original and qualitative in my modified version) and how the data are integrated. With an explanatory sequential design that uses secondary qualitative data, you would have to be familiar with the secondary qualitative data to develop a quantitative method that allows you to achieve your goals. The secondary qualitative data would inspire the new quantitative data collection and still provide generalizable findings that the qualitative data could explain. My advice for working with this design is to stay true to the overall aim of quantitative and qualitative methods and what they are intended to achieve and do not lose sight of this when collecting the new quantitative data and mixing those data into the secondary qualitative data (Figure for Case Study 6.2).

CASE STUDY 6.2 EXPLANATORY SEQUENTIAL DESIGN WITH SECONDARY QUALITATIVE DATA Reference:

[Modified version of] Treviño, E., Scheele, J., & Flores, S. M. (2014). Beyond the test score: A mixed methods analysis of a college access intervention in Chile. Journal of Mixed Methods Research. 8(3): 255–265. (Continued)

134   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah

(Continued) Purpose [same as original]:

To examine the role of a college access program in the enrollment and persistence outcomes of low-income students in Chile, modeled partially after a Texas admissions program.

Research Questions:

None reported.

New Data Source [modified]:

New Quantitative Data

New Data Source Contribution [modified]:

Secondary quantitative data helped gain valuable descriptive information about the program’s effectiveness in preparing economically disadvantaged students for college compared to students who entered through a regular admissions program.

Secondary Data Source [modified]:

Secondary Qualitative Data

Secondary Data Source Contribution [modified]:

Secondary qualitative data helped gain valuable descriptive information about the program’s effectiveness in preparing economically disadvantaged students for college compared to students who entered through a regular admissions program.

Goals of Integration [same as original]:

The goal of integration was to allow the authors to consider the analyses’ agreement, complementarity, and discrepancy.

New Knowledge Generated [same as original]:

This study supports culturally sensitive college preparatory programs as beneficial to students with lower GPAs and may face other significant barriers to their social and educational experiences.

Two random samples: one from students who enrolled in the USACH bachillerato program through Propedéutico and one from students who enrolled through the regular admissions process; total sample of students (n ~ 180); a survey.

Students (n = 15) participated in 15 semi-structured interviews (15–25 minutes each), covering topics such as the program’s effectiveness, students’ experiences during the college preparatory program, and students’ experiences once they entered college.

FIGURE FOR CASE STUDY 6.2 Purpose: To examine the role of a college access program in the enrolment and persistence outcomes of low-income students in Chile, modeled partially after a Texas admissions program Research Question: None reported

Select new QUANTITATIVE data from survey data from two random samples: one from students who enrolled in the University of Santiago de Chile (USACH) bachillerato program through Propedéutico (n = 183) and one from students who enrolled through the regular admissions process (n = 74)

Complete sampling, instruments, and procedures for the new QUANTITATIVE data Analyze new QUANTITATIVE data QUANTITATIVE Results: At the end of H.S., students in the Propedéutico program had lower achievement than their counterparts who applied through the regular admission system Data integration determine the need for new QUALITATIVE data from students who could speak to their program experiences

Use results from QUANTITATIVE data to make decisions about QUALITATIVE samples, measures, and procedures

Review secondary QUALITATIVE semi-structured interview data collected from 15 students Analyze secondary QUALITATIVE data to gain a more complete picture of the effects of the program and general learning outcomes measures by GPA QUALITATIVE Results: Both Propedéutico program students and regular admission students reported they felt they were likely to finish college and they were confident in their capacity to excel in college

Study findings were triangulated using a “convergence coding matrix” that allowed authors to consider convergence, complementarity, and discrepancy

Review QUANTITATIVE results and how they compare to the QUALITATIVE results

Though students in the Propedéutico program had lower GPAs than regular admissions students, they were able to persist in college and believed the program offered tools that eased their difficult transition to university life

Source: Based on Modified version of Trevino, Scheele, & Flores (2014).

Explanatory Sequential Designs With Secondary Quantitative and Qualitative Data Suppose you just completed a secondary analysis of a single-method quantitative data source. In that case, you may feel most comfortable with managing an explanatory sequential design where both data sources exist. However, it requires

136   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah you to perform a secondary analysis of the existing quantitative data before proceeding with a secondary analysis of the existing qualitative data. In the following section, I discuss how to plan for the sample, measures, diagram, and procedures for an explanatory sequential design where both data sources exist.

Sampling for an explanatory sequential design with secondary quantitative and qualitative data Sampling plans for explanatory sequential designs where both data sources exist involve conducting thorough reviews and evaluations of the secondary quantitative and qualitative data (see Chapter 3). The major difference between the sampling plan for a core explanatory sequential design and an explanatory sequential design with secondary quantitative and qualitative data is that you do not need to develop a sampling and recruitment plan from the first study (quantitative) to implement during the later study (qualitative). Instead, you would evaluate the secondary sampling plans from the quantitative and qualitative data sources and determine if you can answer your research question using the original researchers’ sampling plans. Your review of the secondary quantitative and qualitative data may involve first reading over the study protocols for the secondary quantitative and qualitative data to learn how the original study teams determined who they would sample and why. If multiple samples were collected across both the qualitative and quantitative data sources, you would need to determine which ones you want to incorporate into your explanatory sequential design, paying close attention to the ones that are most likely to help you answer your research question. Depending on your research question, the samples from the quantitative study may not include specific cases, groups, or units you desire, thereby influencing your selection of the individuals or groups from the secondary qualitative data source. So getting a clear understanding of how the original quantitative and qualitative samples were determined and deciding if all (or some) of the samples used in the initial studies need to be used in your explanatory sequential design is essential when making sampling decisions. My advice for making sampling decisions that foster a strong connection between the two study components is to keep revisiting your research question as you determine which samples to use from the secondary data. You may need to be creative with addressing the research question using your secondary samples but never lose sight of your study’s purpose and research question. Also, should you realize the secondary samples you had hoped to compare in your explanatory sequential design do not allow you to answer your research question or somehow drift from your overall study purpose, consider finding a different secondary data source or collecting new data for one or both study components of your mixed methods study.

Example of an explanatory sequential design with secondary quantitative and qualitative data Case study 6.3 illustrates an explanatory sequential design with secondary quantitative and qualitative data. In this study, Cracium, Gellert, and Flick (2017) sought to understand the role of positive views on aging as a

Chapter 6  •  Explanatory Sequential Design With Secondary Data   

137

CASE STUDY 6.3 EXPLANATORY SEQUENTIAL DESIGN WITH SECONDARY QUANTITATIVE AND QUALITATIVE DATA Reference:

Cracium, C., Gellert, P., & Flick, U., (2017). Aging in precarious circumstances: Do positive views on aging make a difference? The Gerontologist, 57(3): 517–528.

Purpose:

Comparing how people with insecure and secure pension plans differ concerning psychosocial resources would help them for positive aging.

Research Question:

What is the role of positive views on aging as a potential resource for health and well-being in middle-aged individuals?

Secondary QUANT Data Source:

Secondary Quantitative Data

Secondary QUANT Data Source Contribution:

The DEAS contains measures of behavioral (e.g., smoking, physical activity), social (e.g., social networks), and psychological resources (e.g., self-efficacy) for health and well-being. In addition, it offers an overview of the people growing old in Germany, targeting both individual development and social change.

Secondary QUAL Data Source:

Secondary Qualitative Data

Secondary QUAL Data Source Contribution:

Helped authors explore and compare how precarious and secure middle-aged individuals perceive their resources for positive aging. Analyses focused on how middle-aged individuals identify resources and use them to age positively.

Goals of Integration:

To use the findings from the secondary qualitative interviews to explain the findings from the secondary quantitative data set.

New Knowledge Generated:

This study supports the notion that aging individuals in precarious circumstances are disadvantaged in physical health, well-being, and behavioral, social, and psychological resources compared to secure individuals. People in

Two samples from the German Aging Survey (DEAS) included individuals classified as being in secure circumstances (n = 1,992) and those in precarious circumstances (n = 240).

Twenty (precarious group n = 10; secure group n = 10) author-collected qualitative interviews from an earlier study (October 2012–August 2013) on resources for positive aging in precarious and non-precarious middle-aged Germans.

(Continued)

138   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah

(Continued) precarious groups tend to value their flexibility compared to people in secure groups who value planning. Using mixed methods data illuminated that what aging individuals want most is autonomy, as both secure and precarious strategies could be effective for resource management.

FIGURE FOR CASE STUDY 6.3 Purpose: To compare how people with insecure and secure pension plans differ concerning psychosocial resources that would help them for positive aging. Research Question: What is the role of positive views on aging as a potential resource for health and well-being in middle-aged individuals?

Select secondary QUANTITATIVE data from the German Aging Survey (DEAS) that included individuals classified as being in secure circumstances (n = 1,992) and those in precarious circumstances (n = 240)

Review secondary QUANTITATIVE data using evaluation criteria Analyze secondary QUANTITATIVE data QUANTITATIVE Results: Findings supported Hypothesis 1 (precarious individuals will be disadvantaged in health, well-being, and resources compared to secure individuals); partially supported Hypothesis 2 (differences in resources for health and well-being in precarious and secure individuals); and supported Hypothesis 3 (precarious individuals with insecure pensions and positive views on aging will have better health and well-being than precarious individuals with fewer resources and negative views on aging).

Data integration to shed light on how individuals from the precarious and financially secure group perceive their resources for aging.

Use results from QUANTITATIVE data to make decisions about QUALITATIVE samples, measures, and procedures

Review secondary QUALITATIVE interview data collected from 20 participants (10 secure and 10 precarious). Analyze secondary QUALITATIVE data and use findings from the qualitative interviews to explain the findings from the secondary quantitative data. Analyses focused on how middleaged individuals identify resources and how they use them to age positively. QUALITATIVE Results: Themes (i.e., flexibility, social relations, etc.) on factors leading to health and well-being in old age. The main theme for the secure group was planning, while both groups highlighted their engagement in health behaviors and a “do-it-yourself” mentality.

Study findings used a triangulation principle and helped understand how participants from both group represented positive aging and how positive view on aging helped them prepare for old age.

Review QUANTITATIVE results and how they compare to the QUALITATIVE results

Precarious individuals are disadvantaged in behavioral, social, and psychological resources compared with their secure counterparts. However, participants reported a positive outlook on aging that offset inadequate resources.

Chapter 6  •  Explanatory Sequential Design With Secondary Data   

139

potential resource for health and well-being in middle-aged individuals. They used secondary data from the German Aging Survey (DEAS)—an ongoing cohort-sequential nationwide representative survey of community-dwelling adults aged 40 to 85—to test three hypotheses: Precarious individuals would be disadvantaged in health and well-being and in resources compared to secure individuals (Hypothesis 1); resources for health and well-being differ in precarious and secure individuals (Hypothesis 2); and precarious individuals with insecure pensions and positive views on aging will have better health and well-being than precarious individuals with fewer resources and negative views on aging (Hypothesis 3). After analyses were run to test these hypotheses, a secondary qualitative data source previously collected by two of the study’s authors was used to explore and compare how precarious and secure middle-aged individuals perceived their resources for aging well. Thus, data integration for the Cracium, Gellert, and Flick (2017) study occurred in at least two different stages: (1) The secondary qualitative data were used to shed light on how individuals from the precarious and financially secure group perceived their resources for aging, and (2) overall findings from the quantitative data and the qualitative data helped the authors understand how participants from both groups represented positive aging and how positive views of aging helped them prepare for old age (Figure for Case Study 6.3).

Challenges (and Solutions) When Using Secondary Data in the Explanatory Sequential Design The challenges associated with an explanatory sequential design should be considered if you use secondary data for one or both study components. The two challenges I cover here are (1) determining which quantitative results to follow-up on in the qualitative study and (2) expanding your use of the second qualitative study beyond the demographics of the first quantitative study. First, with an explanatory sequential design with secondary data, you may experience challenges determining which quantitative results to follow-up on in the second qualitative study. It can be challenging to which quantitative results are just exciting and which ones are interesting enough to follow-up on with your second qualitative study. Once you have analyzed the quantitative data, you might be tempted to run additional quantitative analyses. However, keep your overall research question at the forefront of your mind during this process. That way, you can focus on your project purpose and address the current research question. Second, it is natural to be concerned about “topic drift” (i.e., moving away from your original topic of interest) in the sequential designs when you have exciting results from the first study. When this happens to me, I jot down those “drifting” ideas in my project journal (see Chapter 8), with a promise to revisit them later. Knowing I will eventually give these ideas time and attention helps me focus on the current mixed methods project.

140   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah Like the exploratory sequential design that uses two secondary data sources, the explanatory sequential design with two secondary data sources may be intimidating to a novice. It can seem as though you have less control over the study because you did not collect either data source, but that could not be further from the truth. Think about the flexibility you have, even in light of the limitations of each data source. With both secondary quantitative and qualitative data sources, rather than focus on what you wish the data included, think about why certain information was collected, from whom the information was collected, and why these decisions were made by the original quantitative and qualitative study teams. Then think about the intent of the original study teams in conducting their studies and whether the goals of your mixed methods project are aligned with or apart from those of the original study team. Second, with an explanatory sequential design that uses secondary data, make sure you expand your use of the second qualitative study beyond the demographics from the quantitative first phase. Never lose sight of how valuable the qualitative data can be for an explanatory sequential design. What do I mean by this? I mean, after conducting a robust quantitative study, it might be tempting to use your follow-up qualitative study to expound on the quantitative survey respondents’ demographic characteristics only. For example, suppose you are conducting an explanatory sequential design to understand higher education leaders and administrators. In that case, you might first collect (or have access to) 1,000 surveys from higher education leaders from across the country. Then, for your second qualitative study, you might be tempted to use your secondary (or new) qualitative data to only focus on the demographic characteristics of the higher education leaders and administrators (e.g., race, gender, income, sexual orientation, education level, marital status, etc.). If your research questions call for such an analysis, this is acceptable. My point is to simply advise you not to lose sight of the outcome variables and how the qualitative data might provide deeper insight into the relationships between the independent and dependent variables from your quantitative analysis. When all else fails, think about it like this: The quantitative data can help you answer your “what” questions, and the qualitative data can help you answer your “why” and “how” questions. So be sure to take the answers to your “why” and “how” questions a step further when you analyze qualitative data. Overall, the limitations of using a secondary data source to answer your research questions are apparent. Still, you may be inspired by creative ways to assess your methods and address your research question when you see the secondary qualitative data through the lens of your secondary quantitative data. Know that you have options with any secondary data source. However, it is important to see the possibilities at face value and imagine the opportunities that may not seem apparent at first. Your imagined possibilities just might end up being what strengthens your overall approach and your use of secondary data to answer your mixed methods research question.

Chapter 6  •  Explanatory Sequential Design With Secondary Data   

Summary This chapter reviewed the core explanatory sequential design and considered how secondary data could be used for one or both study components. Now that you have read this chapter, you should be able to define the features of an explanatory sequential design and describe ways to bring secondary data into an explanatory sequential design effectively. I highlight explanatory sequential designs with a close eye on the sampling, measures, and procedures. You now know how to analyze and integrate data for an explanatory sequential design as I provided analysis and integration considerations for three types of explanatory sequential designs. I also provide big data breaks that underscore basic and advanced big data sources and analytic techniques. Regardless of the kinds of big data analytics employed, data mining is an essential first step in incorporating big data into a mixed methods study with secondary data. I outlined ways to plan and implement an explanatory sequential design with secondary quantitative data, secondary qualitative data, and secondary qualitative and quantitative data and provided case study examples for each. I end the chapter by identifying potential challenges (and solutions) when doing an explanatory sequential design with secondary data.

Chapter 6 Application Questions 1. What key features of the explanatory sequential design might make using secondary data different from using secondary data with a convergent design? 2. What are 10 steps to help you prepare secondary quantitative and qualitative data for an explanatory sequential design? Create a table like Table 6.1 for your explanatory sequential design with secondary data. Keep the information in the left column from Table 6.1 in your table but fill in the right column with information associated with your study. 3. How do the steps for an explanatory sequential design differ when there is one secondary data source compared to two secondary data sources? For example, compare Case Studies 6.1 and 6.3 from this chapter. How did the authors benefit from new quantitative data in Case Study 6.1 compared to the authors’ secondary quantitative data in Case Study 6.3?

141

142   Part II  •  Designing and Conducting Mixed Methods With Secondary Datah

4. Which of the three ways to integrate data for an explanatory sequential design with secondary data seem like something you would use for your explanatory sequential design? Why? 5. How do the challenges to employing an explanatory sequential design with secondary data play out in Case Study 6.3? How did the authors go about resolving these challenges?

7 Complex Applications of the Core Designs With Secondary Data Imagine You receive a call from Jordyn, a former classmate from graduate school. She is a bit frantic because her boss has asked her to do “. . .some sort of theorydriven mixed methods evaluation for customer satisfaction. . .” at her agency. Your classmate chuckles as she admits to why she called you, “. . .you seemed to actually enjoy our evaluation class in grad school!” Projects have slowed down a bit in your department, so you agree to help Jordyn. Besides, she was part of your study group in graduate school, and she frequently made the monotonous, late-night study sessions fun. After you hang up the phone, you pause for a second and then take a deep breath. You suddenly realize you sold your evaluation textbook back to the university bookstore after the class ended. Yikes! Well, maybe it’s never too late to revisit something as important as evaluation. Plus, you heard mixed methods are popular these days, and it wouldn’t hurt to brush up on your skills. Who knows? Maybe you could even use what you learn for a future project in your department. If you can identify with agreeing to help someone with an evaluation project, only to realize later you need a refresher yourself, then this chapter is for you.

143

144   Part II  •  Designing and Conducting Mixed Methods with Secondary Data

Learning Objectives In previous chapters of this book, I described ways to incorporate secondary data into the core mixed methods designs, which are convergent (Chapter 4), exploratory sequential (Chapter 5), and explanatory sequential (Chapter 6). In this chapter, I define and review complex mixed methods designs and discuss how secondary data can be used for one or both study components for a complex design. By the end of the chapter, you will be able to: 1. Define the features of complex mixed methods designs, 2. Describe how to bring secondary data into complex mixed methods designs, 3. Describe how to prepare secondary data for complex mixed methods design, 4. Review case studies of complex mixed methods designs with secondary data, and 5. Identify potential challenges (and solutions) when using secondary data in complex designs.

Features of a Complex Design I want to begin this chapter with a reminder of what I mean by “core” or “basic” mixed methods designs, as I have used this language in earlier chapters. That way, I can distinguish between core mixed methods designs and advanced or complex mixed methods designs. Though earlier mixed methods scholars would describe roughly six mixed methods designs from which to choose (­Creswell & Plano Clark, 2018; Watkins & Gioia, 2015), more recent scholars have expanded those categories to include either basic (i.e., “core”) or complex (i.e., “advanced”) designs (Creswell & Plano Clark, 2018; Creswell & Creswell, 2018; Plano Clark & Ivankova, 2016). The distinction derives from the notion that there are three basic designs with current mixed methods orientation (convergent, exploratory sequential, and explanatory sequential). All other designs, be they complex or otherwise, are just expansions of one of these core designs. In short, core designs are the foundation of complex designs (Nastasi & Hitchcock, 2016). While the basic mixed methods designs are well known, new complex design ideas emerge all the time, providing opportunities for the growth and expansion of mixed methods. Therefore, new complex designs can help shape a mixed methods core design beyond its basic features and intent. This distinction between core designs and complex designs evolved out of necessity. Plano Clark and Ivankova (2016) acknowledged the mixed methods community’s

Chapter 7  •  Complex Applications of the Core Designs With Secondary Data  

145

realization that the core mixed methods designs were far Complex designs help shape too basic for some projects a mixed methods core design and that they could be used to enhance larger complex beyond its basic features and designs if applied appropriintent. ately. Other names for complex mixed methods designs (Creswell & Creswell, 2018) are advanced applications (Plano Clark & Ivankova, 2016), advanced designs (­Creswell, 2015; Haight & Bidwell, 2016), hybrid designs (Creswell, 2010), and scaffolded designs (Fetters, 2020) and may appear as different variations of these terms in the mixed methods literature. Specific features may signal a complex mixed methods design in the literature. Not only might the name of the design include terms beyond the convergent and sequential language we see in the core designs, but they may also have multiple research phases, occur over several years, have substantial funding associated with it, and include core mixed methods designs during different stages of the project (Nastasi & Hitchcock, 2016). These are not the only features of a complex design, and frankly, given the expansive ways core designs can be added to a design, a methodology, or a theoretical framework (Plano Clark & Ivankova, 2016), it is difficult to limit the criteria of a complex mixed methods design in the pages of this chapter. Better use of time and space in this chapter would be to highlight three ways complex designs are approached in the literature, namely, by (1) embedding a secondary method (qualitative or quantitative) within a primary qualitative or quantitative design, (2) intersecting mixed methods with another methodology, and (3) intersecting mixed methods with another theoretical framework (Creswell & Plano Clark, 2018). The literature on complex designs is extensive, yet there is little consensus on how to label them. Despite this, recent scholars have noted four prominent types of complex mixed methods designs most frequently reported (Creswell & Plano Clark, 2018; Plano Clark & Ivankova, 2016). They are (1) the experimental (or intervention) design, (2) the case study design, (3) the participatory-social justice design, and (4) the evaluation design. Later in the chapter, we will consider using secondary data with these complex mixed methods designs.

Bringing Secondary Data Into Complex Designs Despite the aforementioned terms used to describe complex designs, their three approaches, and the four most prominent types, complex designs are not more rigorous than core designs. Instead, using the term “complex” signals that the design may contain more components than a core mixed methods design. Notwithstanding the additional work associated with a complex design, you may still want to consider incorporating secondary data into your

146   Part II  •  Designing and Conducting Mixed Methods with Secondary Data complex mixed methods design. Bringing secondary data into a complex design might look like bringing secondary data into any other mixed methods design (see Chapters 4–6 for examples). Moreover, the additional work required to design and conduct a complex mixed methods study should not be taken lightly. For instance, a suitable starting place may be asking yourself, “under what conditions should I consider bringing secondary data into my complex mixed methods design?” Let’s consider a program evaluation example at an agency for which you have data from previous program years. Perhaps the program you plan to evaluate is an after-school program for youth in your neighborhood. The program has been in operation for three years, and you have survey data from all the students (n  =  120) who have participated in the program thus far. Of the 120 students who have completed initial surveys, 108 are still “active participants” in the after-school program. That’s a retention rate of 90%, so the program must be doing something right. The program director has only anecdotal evidence (mainly from parents and teachers) suggesting why students seem to like the after-school program, but no empirical evidence for what the after-school program is “doing right” to retain so many students. After your initial review and preliminary analysis of the survey (i.e., quantitative) data, you see they paint a clear picture of student demographics and their general interests in after-school program areas, such as visual and performing arts, technology, athletics, and literacy. As the person tasked with evaluating the program to date, you might consider using the survey data as a launching point for your multiphase, mixed methods program evaluation to understand students’ satisfaction with the program, which program areas are most popular, and what potential changes the program director could employ to increase student satisfaction and program retention. In this example, your evaluation of this after-school program could be enhanced if you used the secondary survey data from the students, and your design might look something like Figure 7.1. The dotted line surrounding the figure is meant to indicate that this mixed methods design is not one of the core designs from Chapters 4–6, but rather a complex design in which an explanatory sequential design with secondary data has been added to a mixed methods evaluation (i.e., complex design).

Preparing Secondary Data for Complex Designs [A major] difference between secondary data for a core design and secondary data for a complex design is the context in which you prepare the data.

Preparing secondary data for a complex design will look like preparing secondary data for all the core designs. However, one significant difference between secondary data for a core design and secondary data for a complex design is the context in which you

Chapter 7  •  Complex Applications of the Core Designs With Secondary Data  

147

FIGURE 7.1  ●  Complex Evaluation Design With Secondary Data Formulate a research question

Mixed Methods Evaluation

Select secondary (or collect new) QUANTITATIVE data

Review secondary QUANTITATIVE data using evaluation criteria (or complete sampling measures/instruments, and procedures for new QUANTITATIVE data collection) Analyze secondary (or new) QUANTITATIVE data Produce QUANTITATIVE results

Integrate the data and determine the need for secondary (or new) QUANTITATIVE data

Use results from QUANTITATIVE data to make decisions about QUALITATIVE samples, measures, and procedures

Review secondary QUALITATIVE data using evaluation criteria (or complete sampling, instruments, and procedures for new QUALITATIVE data collection)

Analyze secondary (or new) QUALITATIVE data

Produce QUALITATIVE results

Data integration

Review QUANTITATIVE results and how they compare to the QUALITATIVE results

Generate findings and overall conclusions

prepare the data. For example, when preparing secondary data for a complex design you have to (step 1) decide how the core mixed methods design will be used to address your research question; (step 2) assess how the secondary data will intersect with the methodological approach/theoretical framework and the core design used to address your research question; and (step 3) evaluate

148   Part II  •  Designing and Conducting Mixed Methods with Secondary Data the core design choice, secondary data, and research question with special attention to how you plan to use each to enhance your experiment, case study, participatory-social justice work, or evaluation (i.e., complex design; see Figure 7.2). Just like for the core designs, you will need to decide whether to collect new data or use secondary data for one or both study components (i.e., quantitative or qualitative) for your complex mixed methods design. You should also review your secondary data using the review criteria I outlined in ­Chapter 3 and create a diagram or figure (i.e., Figure 7.1) to help illustrate how the core design was added to the experiment, case study, participatory-social justice work, or evaluation. Figures can help lay out the steps for your design and help you plan the analysis and integration. The core designs you apply to your complex designs will look like those I covered in Chapters 4–6. Therefore, I refer you to Tables 4.1, 5.1, and 6.1 for guidance on preparing secondary data for your complex design. Further, this is a great place to acknowledge the overlap between the core mixed methods designs (i.e., convergent, exploratory sequential, and explanatory sequential) and complex designs and remind you that complex designs are simply mixed methods designs in which a core mixed methods design is added to another research design, methodological approach, or theoretical framework.

FIGURE 7.2  ●  The Context for Preparing Data for a Complex Design

Research question

Step 1

Step 3 Core mixed methods design

Secondary data

Step 2 Methodological approach/theoretical framework

Chapter 7  •  Complex Applications of the Core Designs With Secondary Data  

149

Data analysis, integration, and interpretation for complex designs with secondary data will look very similar to the core designs with secondary data I covered in Chapters 4–6. Likewise, due to the core designs being the foundation of complex mixed methods designs, the critical steps in planning and implementing a complex mixed methods design will look similar to what I have already covered in Chapters 4–6. So I will not cover these steps here. Instead, I will present three examples (i.e., case studies) of complex mixed methods designs with secondary data from the literature. Specifically, these studies report using complex mixed methods designs with secondary quantitative data, qualitative data, or both for an evaluation, social justice framework, and a longitudinal study.

Case Studies of Complex Mixed Methods Designs With Secondary Data Case Study 1: Complex Evaluation Design With Secondary Data Case Study 7.1 illustrates a complex mixed methods design with secondary quantitative data. Shannon, Hulbig, Birdwhistell, Newell, and Neal (2015) added a convergent design to their evaluation study. The purpose of the study was “. . . to develop an enhanced probation program modeled after Hawaii’s Opportunity Probation with Enforcement (HOPE), to better provide for those high-risk/high-needs individuals in need of enhanced services as well as alleviate the over-crowding issue in the penal institutions which is partially related to individuals failing at community supervision” (p. 50). Shannon and colleagues used a complex mixed methods design to examine process and outcome evaluation data collected as part of the pilot project to understand better the implementation of Kentucky’s Supervision, Monitoring, Accountability, Responsibility, and Treatment (SMART) program and associated outcomes. This complex mixed methods design focused on answering two primary questions: (1) What were program impacts, barriers to implementation, and needed changes/areas of focus for future program years? and (2) what is the effectiveness of the SMART program (as defined in the goals via reduced positive drug screens, violations, and incarcerations)? A process evaluation (that included collecting new qualitative data) was used to answer question one. An outcome evaluation (that included using secondary quantitative data) was used to answer question two. First, to answer their process evaluation question, Shannon and colleagues collected new qualitative interview data from 48 individuals involved in implementing the SMART probation program. The interview data focused on gauging people’s views about program goals/organization, perceived impact of the program on the community/judicial system, and barriers to program implementation

150   Part II  •  Designing and Conducting Mixed Methods with Secondary Data

CASE STUDY 7.1 COMPLEX EVALUATION DESIGN WITH SECONDARY DATA Reference:

Shannon, L. M., Hulbig, S. K., Birdwhistell, S., Newell, J., & Neal, C. (2015). Implementation of an enhanced probation program: Evaluating process and preliminary outcomes. Evaluation and Program Planning, 49: 50–62.

Purpose:

To examine process and outcome evaluation data collected as part of the pilot project to understand better the implementation of the SMART program as well as associated outcomes.

Research Questions:

(1) What were program impacts, barriers to implementation, and needed changes/areas of focus for future program years? (2) What is the effectiveness of the SMART program (as defined in the goals via reduced positive drug screens, violations, and incarcerations)?

Complex Design:

Convergent mixed methods for an evaluation.

New Data Source:

New Qualitative Data

New Data Source Contribution:

The process evaluation data highlighted key findings related to SMART program implementation: (1) improvements to the overall probation system, (2) increased interpersonal communication/collaboration between key stakeholders, and (3) enhanced responsibility and accountability for probationers to address their identified needs/barriers to success.

Secondary Data Source:

Secondary Quantitative Data

Secondary Data Source Contribution:

The outcome data showed advantages of implementing enhanced SMART probation for high-risk/high-need offenders via program effectiveness. Two implications emerged: (1) random drug screening is a deterrent for continued substance use, and (2) the promise of certain consequences promotes programmatic compliance.

New interview data were collected from people (n = 48) who delivered the SMART probation program. Interview data were collected about program goals/ organization, perceived impact of the program on the community/judicial system, barriers to program delivery, and future recommendations. Data were used to conduct a process evaluation.

Collected from the Department of Corrections Kentucky Offender Management System Data were available from key stakeholders (n = 307) about the program’s impact, barriers to implementation, and future recommendations. Data were used to conduct an outcome evaluation.

Chapter 7  •  Complex Applications of the Core Designs With Secondary Data  

Goals of Integration:

Qualitative data for the process evaluation illustrated perspectives from multiple key stakeholders, while the quantitative data for the outcome evaluation provided preliminary outcomes from the SMART program.

New Knowledge Generated:

Improvements to the probation system and increased interpersonal communication and collaboration both contributed to the success of the SMART probation program.

FIGURE FOR CASE STUDY 7.1 Purpose: To examine process and outcome evaluation data collected as part of the pilot project to better understand the implementation of the SMART program as well as associated outcomes. Research Question: (1) What were program impacts, barriers to implementation, and needed changes/areas of focus for future program years? (2) What is the effectiveness of the SMART program (as defined in the goals via reduced positive drug screens, violations, and incarcerations)?

Select secondary QUANTITATIVE data from the Department of Corrections (DOC) Kentucky Offender Management System (KOMS). Data were available from key stakeholders (n = 307).

Collect new QUALITATIVE data from people (n = 48) involved in implementing the SMART probation program.

Review secondary QUANTITATIVE data using evaluative criteria to learn about the impact of the program, barriers to implementation, and future recommendations. Use these data for the outcome evaluation.

Plan sampling, measures/instruments, and procedures for a new QUALITATIVE method that aims to uncover information about program goals/organization, perceived impact of the program on the community/judicial system, and barriers to program delivery and future recommendations. Use these data for the process evaluation

Analyze secondary QUANTITATIVE data to show advantages of implementing enhanced SMART probation for high-risk/high-need offenders via program effectiveness.

Produce QUANTITATIVE results related to the outcomes of the SMART program regard to drug screening, program violations, and sentencing and incarceration costs.

Integrate QUALITATIVE/ QUANTITATIVE Results

Analyze new QUALITATIVE data to demonstrate key findings related to SMART program implementation.

Produce QUALITATIVE results that highlight (1) improvements to the overall probation system, (2) increased interpersonal communication/collaboration between key stakeholders, and (3) enhanced responsibility and accountability for probationers to address their identified needs/ barriers to success.

Generate findings and overall conclusions note improvements to the probation system and increased interpersonal communication and collaboration, which both contributed to the success of the SMART probation program.

151

152   Part II  •  Designing and Conducting Mixed Methods with Secondary Data and future recommendations. Then, to answer their outcome evaluation question, the authors used secondary data from the Kentucky Offender Management System at the Department of Corrections to assess participating site information, the level of service/case management inventory, drug screening/results, violations of probation conditions, and movements/alterations of sentencing and incarceration cost. The goal of the data integration was to use the qualitative data to understand key stakeholders’ perspectives about the SMART program and then use the quantitative data to understand preliminary outcomes from the SMART program. Integrating the data in this convergent way allowed the authors to determine how the success of the SMART program occurs. They attribute improvements to the probation system and increased interpersonal communication and collaboration to the program’s success (Figure for Case Study 7.1).

Case Study 2: Complex Embedded Design With Secondary Data Case Study 7.2 illustrates another complex mixed methods design with secondary quantitative data. Weaver-Hightower (2014) used an embedded design within a critical social justice framework in this study. The author describes the complex design as a QUAL (qual + quant→ qual) design where he embedded, within a qualitative study, a five-stage mixed methods approach that moved from qualitative analysis to quantitative analysis to qualitative analysis. The purpose of the study was to examine how Australia’s national policy on boys’ education influenced people and organizations and, in turn, how these people and organizations were influential. Weaver-Hightower described his mixed methods approach as one phase of a larger, primarily qualitative study conducted over six years. The larger study combined critical policy analysis, critical ethnography to examine Australia’s creation of the national policy on boys’ education. This is a first of its kind, and therefore, Weaver-Hightower aimed to provide a historical account of gender relations and gender-based policies in Australia, examine governmental policies that emerged out of the policy, and then develop a series of case studies of how schools incorporated the policies into their programs for boys (Weaver-Hightower, 2014). First, Weaver-Hightower reviewed several data sources, including (1) the findings and 24 committee recommendations from the report Boys: Getting it Right (Australian House of Representatives Standing Committee on Education and Training [the “Committee”], 2002), (2) official transcripts of public hearings, (3) written submissions from individuals and organizations, (4) research from the report, and (5) physical exhibits displayed at the hearings (e.g., videos, books, and slides). Then Weaver-Hightower collected new, qualitative semi-structured interview data from members of the Committee to understand their perspectives about the policy and their roles in developing the report. Their interviews were used during data analysis and validity checking stages. The author performed data transformation processes that involved quantifying qualitative data (converting qualitative codes into numbers for statistical analysis) and qualifying quantitative data (converting quantitative samples into small-scale case descriptions of each influential

Chapter 7  •  Complex Applications of the Core Designs With Secondary Data  

CASE STUDY 7.2 COMPLEX EMBEDDED DESIGN WITH SECONDARY DATA Reference:

Weaver-Hightower, M. B. (2014). A mixed methods approach for identifying influence on public policy. Journal of Mixed Methods Research, 8(2): 115–138.

Purpose:

To use a five-stage procedure to identify outside influences on the government initiatives growing out of Australia’s national-level policy on the education of boys and the subsequent programs.

Research Questions:

What arguments were made to the policymakers? Who was listed to (i.e., influential) and who was not? What political and ideological positions did influential people hold?

Complex Design

Embedded, qualitative-priority mixed methods design with critical, social justice aims.

Secondary Data Source:

Secondary Quantitative and Qualitative Data

Secondary Data Source Contribution:

The quantitative procedures solidified and challenged the author’s qualitative impressions of influence established by the author’s previous research and qualitative data collection on the topic.

New Data Source:

New Qualitative Data

New Data Source Contribution:

The qualitative interviews highlighted influential individuals and groups and how they might have been influential in the policy’s dissemination and application.

Goals of Integration:

A quantitative means for identifying influential people and organizations were followed by a qualitative case study that explained their influence.

New knowledge generated:

The integration of the methods employed in this study showed the overall result that the Australian boy’s education policy— which had nationwide dissemination and application—was the result of work by a small (primarily White male) group of influential people. These influential people developed a report that resulted in the allocation of millions of dollars, refocusing programs, and redrafting the entire nation’s gender equity policy.

Critical policy analysis was conducted alongside descriptive quantitative measures (qual + quant), which led to a separate qualitative cross-case analysis to explain the qual + quan findings.

Semi-structured interviews were conducted with members of the Australian House of Representatives Standing Committee on Education and Training.

153

154   Part II  •  Designing and Conducting Mixed Methods with Secondary Data FIGURE FOR CASE STUDY 7.2 Social Justice Aims Identify groups advantaged and disadvantaged by specific policy outcomes. Identify those with the political power to influence policy development. Challange recuperative masculinityn politics (Lingard, 2003) in boys devbates. Help create gender just (Keddie, 2007) educational practices. QUAL Study Procedures Critical ethonographic interviewing and observation (Carspecken, 1996) with policymakers, at conferences, and in schools Critical policy analysis (Prunty, 1985; Taylor et al., 1997) of policy/ report Critical discourse analysis (Gee, 2005) of documents used in formulating the policy/report (hearing transcripts, written submissions, exhibits).

Products Transcripts and fieldnotes Case studies of school programs Arguments and ideologies identified (themes) Arguments and ideologies identified (themes)

Embedded quan Phase [quan=identify politically influential individuals and organizations, or ‘influentials’]

Stage I

Stage II

Procedures Quantitize argument themes and recommendations-applied to hearings, submissions, and report

Products Agreement scores between commenters and report

Calculate ither indicators of influence (time, explicit citation)

Descriptive statistcs

Stage III Cross-tabulation of agreement scores and other indicators

List of influentials

Qualitative case Stage IV descrptions of ‘influentials’ and their impact on the policy/report. Cross-case analysis to theorize reasons for influence

Negative case and ‘positive case’ validity analysis of identified “influentials”

Stage V

Source: Adapted from Weaver-Hightower, 2014 Journal of Mixed Methods Research paper (Original reference to this diagram is in the Australian Boys’ Education Study (Weaver-Hightower, 2008).

respondent). The integration of the methods employed in this study suggested the Australian boy’s education policy—which had nationwide dissemination and application—was the result of work by a small (primarily white male) group of influential people. These influential people developed a report that resulted in the allocation of millions of dollars, refocusing programs, and redrafting the entire nation’s gender equity policy (Figure for Case Study 7.2).

Chapter 7  •  Complex Applications of the Core Designs With Secondary Data  

155

Case Study 3: Complex Longitudinal Design With Secondary Data Case Study 7.3 illustrates my final example of an explanatory sequential design with secondary quantitative data. In this study, Sligo, Nairm, and McGee (2018) added an explanatory sequential design with secondary quantitative and qualitative data to their longitudinal study. The purpose of the study was to provide a mode understanding of the relationship between aspirations and occupations by using different studies from different times. First, Sligo and colleagues used secondary quantitative data collected from the same participants throughout their lives. These “quantitative study” data were from a larger study that began in 1986 and was a multidisciplinary study of intersections between health, education, and employment among a sample of 10th-grade students (n = 878) in the South Island of New Zealand. The quantitative data included a written questionnaire about intentions and aspirations for future occupations. Participants were also asked if they believed their chances of achieving their occupations were good, fair, or poor. After analyzing these data, Sligo and colleagues found that participants’ occupation predictions as teenagers did not predict their adult occupations. The authors then used these findings to answer the question: What might account for this significant degree of divergence between aspirations and occupation? Bourdieu’s theory was applied to a different set of data, from a “Qualitative Project” to identify the structural and individual factors that might have been a barrier or a facilitator to young people’s (n = 66) fulfillment of their vocational aspirations. Though the authors analyzed interviews, visual data, interviewer notes, and reflections for the 66 participants from the Qualitative Project, they provide three narratives from participants in their study to demonstrate how the narratives could explain their findings from the quantitative study (Figure for Case Study 7.3).

CASE STUDY 7.3 COMPLEX LONGITUDINAL DESIGN WITH SECONDARY DATA Reference:

Sligo, J. L., Nairm, K. M., & McGee, R. O. (2018). Rethinking integration in mixed methods research using data from different eras: Lessons from a project about teenage vocational behavior. International Journal of Social Research Methodology, 21(1): 63–75.

Purpose:

To provide a mode understanding of the relationship between aspirations and occupations using different studies from different times. (Continued)

156   Part II  •  Designing and Conducting Mixed Methods with Secondary Data

(Continued) Research Questions:

(1) Do young people’s vocational aspirations predict their adult occupations? (2) Why or why not?

Complex Design:

Explanatory sequential mixed methods for a longitudinal project.

Secondary Data Source #1:

Secondary Quantitative Data

Secondary Data Source #1 Contribution:

Comparing participants’ vocational aspirations from when they were 15 to their occupations at 32 showed that their teenage aspirations did not predict their adult occupations.

Secondary Data Source #2:

Secondary Qualitative Data

Secondary Data Source #2 Contribution:

The Qualitative Project’s participants provided rich data about what impacted their potential to follow their teenage aspirations into adulthood.

Goals of Integration:

Using a theoretical analysis during data integration, the authors found the quantitative study demonstrated an overall pattern of a discrepancy between teenage aspirations and adult occupations. The Qualitative Project provided possible explanations for why young people rarely end up in the occupations they aspire for as teenagers.

New Knowledge Generated:

By revisiting data from different eras, the authors understood how young New Zealanders in the twenty-first century struggle with structural boundaries that make it challenging to achieve their desired occupations without the relevant social capital they need to follow their vocational aspirations.

The quantitative study (n = 878) is a longitudinal study with the same participants throughout their lives, comparing teenage occupational aspirations data and actual occupations from adulthood. The subset of participants analyzed in this study was followed since they were 15 years old and provided their employment information when they were 32 years old (2004/2005).

The Qualitative Project (n = 66) included interviews with participants from the quantitative study and noted the similarities and differences across participants and within their lives. Narratives from 3 of the 66 participants were highlighted in this study.

Chapter 7  •  Complex Applications of the Core Designs With Secondary Data  

157

FIGURE FOR CASE STUDY 7.3 Purpose: To provide understanding of the relationship between aspirations and occupations by using different studies from different times

Research Question: (1) Do young people’s vocational aspirations predict their adult occupations? (2) Why or why not?

Select secondary QUANTITATIVE data from the quantitative study (n = 878), a longitudinal study with the same participants over many years allowing for the comparison of teenage occupational aspirations and actual occupations in adulthood

Review secondary QUANTITATIVE data using evaluation criteria Analyze secondary QUANTITATIVE data QUANTITATIVE Results: Comparing participants’ vocational aspirations from when they were 15 to their occupations at 32 showed that their teenage aspirations did not predict their adult occupations

Data integration to shed light on how individuals from the precarious and financially secure group perceive their resources for aging

Use results from QUANTITATIVE data to make decisions about QUALITATIVE methods

Review secondary QUALITATIVE data (n = 66) included interviews with participants from the quantitative study Analyze secondary QUALITATIVE data to note the similarities and differences across participants and within their lives; narratives from 3 of the 66 participants were highlighted in this study QUALITATIVE Results: Themes generated provided rich data about what impacted their potential to follow their teenage aspirations into adulthood

Quantitative findings showed a discrepancy between teenage aspirations and adult occupations and the qualitative findings provided possible explanations for why young people rarely end up in the occupations they aspire for as teenagers

Review QUANTITATIVE results and how they compare to the QUALITATIVE results

Comparing secondary data from different eras helped the authors understand how young New Zealanders in the twenty-first century struggle with structural boundaries that make it challenging for them to achieve their desired occupations without the relevant social capital they need to follow their vocational aspirations

158   Part II  •  Designing and Conducting Mixed Methods with Secondary Data

Challenge (and Solution) When Using Secondary Data in Complex Designs If you use secondary data with complex mixed methods designs, potential challenges should be anticipated. Remember that designing and conducting mixed methods using the core designs might be challenging for a mixed ­methods novice. So it behooves readers to understand that complex designs receive their name because they add layers of complexity to their design and conduct, one that goes beyond the core designs. As you work through the details for your complex mixed methods designs, you may face some of the challenges I have covered in Chapters 4–6, when using secondary data in convergent, exploratory sequential, and explanatory sequential designs. Another challenge when using secondary data in complex designs is dealing with additional, method-related assumptions when adding a core design to your methodological approach. I have already outlined the ways complex mixed methods designs have assumptions you must navigate beyond the core designs. However, depending on your comfort level with the core designs and your chosen complex design, you may accidentally overlook some additional assumptions that could weaken your study. For example, let’s assume you plan to add an exploratory sequential design to an experimental study with secondary focus group data on HIV-positive pet owners across the country. You know from your readings that you must first complete the qualitative Depending on your comfort study before deciding on level with core designs the quantitative study for this exploratory sequential and complex designs, you may design. You want to conduct accidentally overlook some a randomized controlled trial additional assumptions that for your second quantitative could weaken your study. study component, which will require you to collect a random sample of study participants. However, analyzing the secondary qualitative data first may uncover findings that lead you down a different path for your second, new quantitative study component. For instance, what if you learn midway into your qualitative analysis that your research budget has been cut by 50%. Do you now need to collect your quantitative data from your local hospital and not from respondents across the country? While frustrating, all hope is not lost. It may just be more challenging to collect the number of quantitative responses you initially proposed, and it may also be challenging to randomize your sample. Due to unfortunate circumstances beyond your control, your sample may need to be convenient and not random. Thus, your plans for adding an exploratory sequential design to a complex experimental study can no longer occur. Once your sample changes from random to convenient, you can no longer meet the assumptions of the

Chapter 7  •  Complex Applications of the Core Designs With Secondary Data  

159

randomized controlled trial. Therefore, you will need to make some changes to your complex design. I included this example to simply illustrate that sometimes you may have a plan in place for your complex design, and some unforeseen circumstances (either within your control or out of your control) may result in your need to change the research design. When this happens to me, I am (obviously) disappointed, but then I remind myself that all hope is not lost. A delay in my research plan may not necessarily mean I will be denied the opportunity forever. I can always submit another proposal to fund my project or find other ways to support the research now or in the foreseeable future.

Summary This chapter reviewed complex mixed methods designs and discussed how secondary data could be used for one or both study components. Now that you have read this chapter, you should be able to define the features of a complex design and describe the ways to bring a core mixed methods design with secondary data (i.e., convergent, exploratory sequential, and explanatory sequential) into a complex design. I highlighted how scholars define complex designs in the literature (e.g., advanced applications, advanced designs, hybrid designs, and scaffolded designs). I also described ways to bring secondary data into complex mixed methods designs. I then provided a description, table, and figure for three case studies where authors have added a core design to an evaluation, embedded mixed methods design, and longitudinal study. I ended the chapter by identifying a potential challenge (and solution) when incorporating a core design with secondary data into a complex mixed methods design.

Chapter 7 Application Questions 1. What key features of complex designs might make using secondary data different from using secondary data with one of the core mixed methods designs? 2. What are three contextual steps to help you prepare secondary quantitative and qualitative data for a complex design? 3. How do the steps for a complex design with secondary differ when used in an evaluation, an embedded mixed methods study, and a longitudinal study? Compare Case Studies 7.1–7.3 from this chapter. How were the authors able to benefit from secondary qualitative data compared to secondary quantitative data?

160   Part II  •  Designing and Conducting Mixed Methods with Secondary Data

4. In what ways did the authors for Case Study 7.3 add additional levels of complexity to their core explanatory sequential design? How did these levels of complexity help achieve the overall study goals and objectives? 5. This chapter offered an example where budget cuts changed the research design for a complex mixed methods study. How might changes to your research team be a challenge to designing and conducting a complex design with secondary data?

Before, During, and After

PART III

Writing Mixed Methods With Secondary Data

161

8 Early-Stage and Active Project Writing for Mixed Methods With Secondary Data Imagine Your best friend, Donald, received a brand-new leather journal for his birthday. It has his initials monogrammed on the front cover, and when he pulled it out of the box, he laughed, “. . .I guess this is too nice for my tacky sketches and daily musings.” Since Donald just passed his comprehensive exams for his doctoral program, he has decided to use the journal for his dissertation research, still “to be determined.” His program adviser has strongly recommended he consider doing a mixed methods dissertation, perhaps one that could be a spin-off from the adviser’s recent fieldwork on the impact of the Deferred Action for Childhood Arrivals (DACA) legislation on college students in the United States. Donald tells you he will tuck the leather journal away for safekeeping until he is ready to “officially” start on his dissertation. He does not plan to begin analyzing the secondary data for his dissertation for at least another three or four months. You advise him otherwise, insisting that it is never too early to begin collecting ideas about his dissertation research. Besides, you can sense he is looking forward to writing in the journal, so why not start now? If you want some specific ideas to share with Donald about how he can make good use of his new journal, then this chapter is for you. 163

164   Part III  •  Writing Mixed Methods with Secondary Data

Learning Objectives This chapter reviews the writing that occurs during the early and active stages of a mixed methods study with secondary data. Specifically, I outline some examples of early-stage and active writing and tips for maximizing your writing during the early and active project of your mixed methods with secondary data. By the end of the chapter, you will be able to: 1. Define what early-stage and active writing are and what they look like for mixed methods with secondary data, 2. Identify the types of documents you can develop and refine during the early and active project stages of mixed methods with secondary data, and 3. List some tips to maximize your early-stage and active writing for mixed methods with secondary data.

Early-Stage and Active Project Writing: Defined The early-stage writing for your mixed methods with secondary data includes the introductory information, budding ideas, and the preliminary work plans you develop for your project before it begins. During the early stages of your mixed methods study, your writing processes can vary based on how you generate ideas for your research and writing. For example, sometimes the process of early-stage writing can look informal (i.e., writing ideas about the study design on Post-it® notes and placing them on the bathroom mirror), while other times, the process will look much more formal (i.e., holding regular team meetings and documenting study details discussed during those meetings). For many people, the early-stage writing process may be a combination of both. My early-stage writing is an iterative process involving regular meetings with my teams to discuss findings from our previous studies, new ideas for our next projects, and potential funding mechanisms for our ideas. We usually identify one or more notetakers for these meetings so that none of our good ideas are lost as we brainstorm, debate, and sketch our ideas on the whiteboard or giant sheets of Post-it® super sticky easel pads. Maybe I am biased, but my teams are composed of some of the most brilliant minds this generation has to offer. This is because, when research teams come together, magic happens. Nevertheless, after one of these brainstorming meetings, I can usually come up with other ideas while driving home from the office or, ironically, when I am in the shower (when my laptop and project journal are nowhere in sight). Under less formal working conditions (i.e., my car or the shower), I find myself either jotting down my idea on the back of a store receipt or using my cell phone to record an audio memo. My team meetings are part of my formal writing processes (because we are more organized, planned, and intentional in our

Chapter 8  •  Early-Stage and Active Project Writing for Mixed Methods...  

165

discussions and documentation). My car and shower You may find both [formal epiphanies can be considered and informal writing] part of my informal writing processes. Like me, you may processes useful for your mixed find both processes useful for methods with secondary data. your mixed methods with secondary data. Active project writing occurs while your project is underway. It includes the various ways you gather information, document, and track the ongoing procedures, protocols, and findings associated with your mixed methods with secondary data. Active project writing is unlike early-stage writing in that you are no longer planning your research design and drafting work plans. Instead, you are carrying out your study and making a concerted effort to monitor the day-to-day successes (and failures). Because of the sheer volume of information collected during a mixed method project, especially a mixed methods project with secondary data, you may find yourself overwhelmed by all the information you need to track. The kind of information you need to manage to conduct your mixed methods with secondary data will require some planning (i.e., early-stage writing) and executing (i.e., active project writing) on your part. Later, I share examples of early-stage and active project writing and some tips that have helped me create useful writing and organizational strategies over the years.

Examples of Early-Stage Writing It is helpful to start writing about your mixed methods study during the early stages of the planning process. The files you produce during the early-stage writing for your mixed methods research with secondary data may begin as drafts and undergo several revisions. After developing and refining these files, you should end up with documents that help you design, conduct, and evaluate your mixed methods project. You, can produce several documents to help you plan your mixed methods project with secondary data. Here, I present five examples of early-stage writing: journaling to plan; developing research proposals; planning with diagrams, tables, and figures; drafting study protocols; and building training materials. You can begin working on any or all of these during the early stages of your mixed methods with secondary data (Table 8.1). This list is not meant to be exhaustive but rather a start to the early-stage writing process.

Early-Stage Writing Example 1: Journaling to Plan It may be the aspiring ethnographer in me, but I have always loved to journal. Ever since I was a child, it has been refreshing to reflect on my day by penning a narrative about the things that occurred, my reactions to them, and how I plan to make tomorrow a better day. The satisfaction I experienced journaling

166   Part III  •  Writing Mixed Methods with Secondary Data TABLE 8.1  ● Examples of Early-Stage Writing for Mixed Methods With Secondary Data Writing Type

Application

Journaling to plan

Draft initial ideas, preliminary studies, and early plans for the study.

Developing research proposals

Treat the proposal as a journey and not a destination and relish in the process of developing the study design to inform later stages of the work.

Planning with diagrams, tables, and figures

Sketch preliminary diagrams, tables, and figures to think through the early study designs, methods, procedures, and analysis.

Drafting study protocols

Workshop preliminary study protocols with tentative documents that evolve during the early stages of the project.

Building training materials

Draft background information about the topic, team members’ roles, and additional information about the study design and methods.

as a young person has shaped the way I do my research, and thankfully, it has turned into a valuable skill I now apply to my research projects. Journaling to plan is helpful for all research projects, but especially mixed methods with secondary data. Whether you decide to keep an electronic or a paper planning journal is less critical. What is most important is understanding that journaling to plan at the early stages of a mixed methods project is an essential part of the research process. So much so that whenever I advise aspiring mixed methods researchers, journaling is among my top recommendations. I believe in the value of journaling for all research projects, not just mixed methods projects or those that use secondary data. I have found journaling to be a good practice during the early, midpoint, and later stages of my mixed methods studies and never regret documenting the information in my planning journal at the beginning stages of my projects with secondary data. At this point, you may be asking yourself what should I journal about during the early stages of mixed methods with secondary data. My advice is to not hold yourself to strict rules for journaling about your research. Defining the parameters of your planning journal is just another level of pressure you do not need. For me, “anything goes” when it comes to early-stage journaling for my mixed methods projects with secondary data, and I have found it helpful to keep my options open. For example, I have found it beneficial to journal answers to questions about my assumptions and position in the research and how this affects my interpretation of the secondary data. Who from my

Chapter 8  •  Early-Stage and Active Project Writing for Mixed Methods...  

167

research team can help me identify, evaluate, and analyze the project’s secondary data? I also journal about ideas I have for our timeline and answer questions such as what is the best- and worst-case scenarios for our inclusion of secondary data? What does the current literature state about my research topic, and has secondary data been used previously? These are all important questions to ask for your mixed methods with secondary data, and there are others. When my projects use secondary quantitative or qualitative data, I also use my planning journal to outline what I know about the secondary data sources. For example, these journal pages are where I record the secondary data inventory, list of related variables (or constructs) from the new (or secondary) data, potential sources of information I should incorporate into my study to help with data integration, and previous studies that used the secondary data I plan to use for my analysis. The journal can also document preliminary studies and how the results from these studies could potentially inform the next steps in the research. Furthermore, you can use the journal to outline alignment and connections between different quantitative and qualitative data sources. Identifying constructs across various types of data early on can help you develop integration opportunities you can apply When my projects use later. During the early stages secondary quantitative or of my studies, I also use the qualitative data, I also use my planning journal to document planning journal to outline what my evaluation (­ Chapter 3) of the secondary data source I know about the secondary data to assess its applicability and sources. appropriateness for my mixed methods study.

Early-Stage Writing Example 2: Developing Research Proposals For many readers (e.g., students, principal investigators, research teams, etc.), the early-stage writing process will involve seeking external approval and securing funding for your project. This approval process usually begins with developing a proposal that describes the project’s details for a research review committee or board (i.e., thesis/dissertation committee, funding agency, or institutional review board [IRB], etc.) who must review and approve the project before you can proceed. Thus, I would classify the research proposals you write for your mixed methods studies with secondary data as a type of earlystage writing for your project. Other authors have described how to write successful mixed methods dissertation proposals (DeCuir-Gunby & Schutz, 2017; ­Rudestam & Newton, 2015; Terrell, 2016; Watkins & Gioia, 2015) and grant proposals (Creswell et al., 2011; Curry & Nunez-Smith, 2015), so I will not cover that here. Instead, I will underscore the value of treating your mixed methods proposal with secondary data as a journey and not just a destination.

168   Part III  •  Writing Mixed Methods with Secondary Data When I am working on a mixed methods project, I first draft the specific aims of my research proposal and then begin preparing the various sections of the proposal (i.e., background, theoretical framework, study design, data collection, data analysis, etc.). Often, when I get halfway through the study design or data collection section, I notice something I wrote in my data collection section changes the way I think about my specific aims or research questions. This means I need to go back to the beginning of my proposal and rewrite the specific aims or another section that reflects the changes I made later in the proposal. So, for me, the process of writing a research proposal is iterative, not linear; yet budding researchers are often taught to treat the research proposal as a product and not a process. What would happen if you treated the proposal stages of your mixed methods with secondary data like a process and not a product? What if you focused on the evolution of your ideas while drafting the proposal and less on the need to just get it done? I encourage you to think more intuitively about your research proposals and take stock in the idea of it being a process that evolves as you work on it (and may even change after you have already begun the project! However, significant changes are rare and should be handled with caution). I, too, am happy when the proposal stage of my mixed methods project is completed. But, over the years, I have learned to appreciate the process of developing my research proposals, especially those that include secondary data. For instance, as you develop the proposal for your mixed methods with secondary data, the process of documenting the secondary data inventory, variables, and potential analysis plan based on what you learned from the primary researchers can be rewarding. Dissecting how someone else made decisions about their research methods to inform the decisions you make for using their data is an exercise in critical thinking and advanced data organization and management. Not to mention, even when our proposals are not approved or receive funding support, this does not mean our hard work was for naught. On the contrary, this is an opportunity to use the proposal that was not approved or denied funding What would happen if you as a starting place for a future treated the proposal stage for project. This level of thinking your mixed methods project with about our work is more likely secondary data like a process to occur if we treat our mixed methods proposals with secand not a product? ondary data as a journey and not merely a destination.

Early-Stage Writing Example 3: Planning With Diagrams, Tables, and Figures The mixed methods community does not spend enough time discussing the importance of developing diagrams, tables, and figures during the early stages of projects. I argue that there are benefits to developing diagrams, tables, and figures as a part of the early stage of writing and planning for your mixed

Chapter 8  •  Early-Stage and Active Project Writing for Mixed Methods...  

169

methods with secondary data. For example, sometimes, it is difficult for me to write about my early-stage mixed methods designs if all the ideas about the secondary data sources are still in my head. Rather than succumb to the pressure of writing eloquent sentences about my research project at such an early stage, I prefer to sketch a diagram to see if the visual elements of my ideas for integrating two secondary data sources (or one secondary and one new data source) match up with the ideas in my head. Think back to the procedural diagrams mentioned earlier in the book. Your initial drafts of your procedural diagrams with secondary data can begin at the early writing stage. I encourage you to take advantage of the early contributions of diagrams, tables, and figures as you plan your mixed methods projects with secondary data. As I noted above, creating diagrams can help you clarify your ideas visually and preliminarily when you are not ready to commit to words. You can also use your planning journal (see the first early-stage writing example above) to develop tables and figures for your mixed methods project’s secondary data sources. For instance, after evaluating a potential secondary quantitative data source for an explanatory sequential design, you may decide to create a table or figure that helps outline the various options you have for integrating aspects of the quantitative data into the new (or secondary) qualitative methods. Perhaps a figure showing the steps involved in integrating the secondary quantitative data into a new (or secondary) qualitative study component would be beneficial (see Chapter 6). A data sources table that aligns the new and secondary data to your project’s variables and hypotheses is another valuable resource to create during the early writing process. Due to the importance of visual elements of mixed methods studies at various stages of the research process (Fetters, 2020; Guetterman, Fetters, & Creswell, 2015; [C]reating diagrams can Plano Clark & Ivankova, 2016; help you clarify your ideas Watkins & Gioia, 2015), I will visually and preliminarily when also discuss how you can make you are not ready to commit to use of diagrams, tables, and figures after your mixed methwords. ods project with secondary data has ended (Chapter 9).

Early-Stage Writing Example 4: Drafting Study Protocols The early writing for your mixed methods with secondary data might involve drafting new study protocols or rigorously reviewing existing protocols. The early writing process might also involve reviewing and documenting your evaluation of secondary quantitative or qualitative study protocols (see Chapter 3 for evaluation criteria). Again, this work can be completed in a journal and further illustrated using diagrams, tables, or figures. However, a straightforward narrative or description of your understanding of the

170   Part III  •  Writing Mixed Methods with Secondary Data secondary data might also suffice. Suppose you are collecting new data for one of your study components. In that case, the early writing for your study protocols might include reviewing previous literature (for substantive information about previous research designs and study methods used) and outlining a plan for new data collection, given your research question, study purpose, and resources. You can use early-stage writing to outline the details for planning, creating, and testing, the secondary data study protocols. For example, suppose I am conducting an exploratory sequential design with secondary quantitative data. In that case, I might draft the interview questionnaires and study protocols for my study’s qualitative component during the early stages of my mixed methods studies. I could use a similar approach for the secondary quantitative survey instruments I incorporate into this example. Because the quantitative survey in this example exists, I could use my early-stage writing to document previous measures of scale validity and reliability and note the likelihood of the scales producing strong Cronbach’s alphas in my study given the way they have been used in studies by previous authors whose samples included similar (or different) characteristics compared to my sample. These interview questionnaires and scale validity and reliability notes are the types of study protocol documents you can submit to a human subjects review board (e.g., IRB) for review, oversight, and approval prior to conducting your mixed methods project with secondary data.

Early-Stage Writing Example 5: Developing Training Materials So far, I have highlighted four examples of early-stage writing for your mixed methods with secondary data: journaling to plan, developing research proposals; planning with diagrams, tables, and figures; and drafting study protocols. Here, I want to underscore how you might consider including these early-stage documents, or portions of them, in the training materials for your mixed methods projects with secondary data. If you plan to work with others (i.e., a team) for your mixed methods with secondary data, it behooves you to create training materials for your team members. These training materials might include background information about the topic under study, team members’ roles, and (depending on your team members’ prior experience with mixed methods research and secondary data) some additional information about the study you plan to conduct. While the information you include in the study training materials might vary based on the project, your resources, your timeline, and so on, your training materials are what helps familiarize your research team with the project, the use of mixed methods, and how you plan to incorporate secondary data. Begin by identifying facilities and resources as part of the mixed methods study infrastructure. Also, think about permissions needed for obtaining secondary data and related costs. A vital part of training a team is ensuring everyone is on the same page about the study purpose, research question(s), and timeline. It

Chapter 8  •  Early-Stage and Active Project Writing for Mixed Methods...  

171

is also important that the people responsible for the new data collection and secondary data review and evaluation have been trained. When drafting training materials for your mixed methods with secondary data, it is essential to note the skills you and your team require to complete the work. The skills needed to analyze new data are similar, yet slightly different from the skills needed to review, evaluate, and analyze secondary data. Notably, extra steps must occur with the secondary data that will require a keen eye and open-mindedness about the role of the data in the project and how they can be integrated with new (or secondary) data to address the mixed methods research question. My advice is to list the specific skills you think are necessary to complete the work. Your research team may already possess some of the skills needed, while there may be other skills that will require additional training. The early-stage writing you do for your mixed methods studies is not merely to document your work at this stage of the project. It can also set your team up for success and ensure consistency across the project information they receive, how they receive it, and how they plan to achieve the study goals. In short, the investment you make in earlystage journaling for planning; developing research proposThe early writing you do for als; planning with diagrams, your mixed methods studies tables, and figures; drafting is not merely to document your study protocols; and developing training materials just work at this stage. might be what sets you and your team up for success.

Examples of Active Project Writing While I am in the middle of a mixed methods project with secondary data, the writing that occurs looks different from the early-stage writing I did at the beginning of the project. For example, unlike my early-stage writing, where I might integrate my formal and informal writing processes to achieve desired results, I usually take a more formal and systematic approach to tracking my writing while a project is underway. Active writing serves multiple purposes; here, I will note three: (1) it helps document the information you need to answer your research question(s); (2) it keeps track of the decisions you make (and those made by your team) as you navigate the day-to-day responsibilities of the project; and (3) it provides draft protocols and procedures for future iterations of the project. In the next section, I provide examples of active writing and describe ways they can be used for your mixed methods with secondary data. There are many writing opportunities for you to engage in during an active mixed methods project with secondary data. During this stage of your mixed methods research, the types of writing you do will depend heavily on your mixed methods design, whether you use secondary data for one or both study components, and the information you have (or need to collect) to complete

172   Part III  •  Writing Mixed Methods with Secondary Data your project. Despite the various elements of your study, there are a few active writing documents I believe you should create for all mixed methods projects with secondary data. Here, I present four examples of active writing you can do while your project is underway (Table 8.2). This is not an exhaustive list, so feel free to expand on it with other documents you find useful while your mixed methods project with secondary data is active.

Active Project Writing Example 1: Journaling to Track Journaling during an active mixed methods project is just as important as journaling before the project begins. In fact, despite the amount and the degree to which I communicate with my research team during our active mixed methods studies, I still use my tracking journal because writing in it during my projects is a necessary “conversation” with myself. It helps me keep track of my ideas, thoughts, and the study’s progress. Indeed, the conversations with my team are beneficial. Still, I need additional time, alone, to wrestle with some of the day-to-day project activities. Sometimes, this occurs while we are in the process of collecting new data, evaluating secondary data, or analyzing large quantities of data for my mixed methods projects. The opportunity to log the activities of my project while it is underway helps me think through some of the nuances I may not pick up on while in meetings with my team, discussing my work with colleagues, or working with communities. Earlier in this chapter, I emphasized the importance of journaling to plan for your mixed methods study with secondary data. In a similar vein, I want TABLE 8.2  ● Examples of Active Project Writing for Mixed Methods with Secondary Data Writing Type

Application

Journaling to track

Draft in-progress ideas, study plans, and decisions while the study is underway.

Keeping field notes

Document the decisions made for both the new and secondary data used for the mixed methods study.

Tracking with diagrams, tables, and figures

Create draft diagrams, tables, and figures to work through the early study designs, methods, procedures, and analysis.

Reviewing and developing codebooks

Review codebooks (for secondary data) and develop codebooks (for new data) while your mixed methods study is underway. While working, your team may generate helpful information and subsequent resources (from journaling, keeping field notes, and creating diagrams, tables, and figures) that can be included in the codebooks. Bringing codebooks together will enhance data integration and interpretation.

Chapter 8  •  Early-Stage and Active Project Writing for Mixed Methods...  

173

to underscore the importance of journaling to track the details of your data review, collection, and analysis during an active mixed methods project with secondary data. My project tracking journal includes everything from independent and dependent variables for my statistical analysis to preliminary codes and thematic constructs from my transcript-based content analysis. I like to treat my journal as an audit trail or a “paper trail” for all project-related decisions. There is something about tracking the individual study components, methods, and results in one place that provides a certain level of comfort and satisfaction to me as a mixed methods researcher. Also, having the details for the two study components in one tracking journal helps me build a better data analysis and integration plan. My quantitative and qualitative data may take several different forms over the life of the project, and I document these in my project tracking journal. For instance, the data may begin as separate entities, then come together (i.e., integrate, or “mix”), then separate again, and then come together again. When your mixed methods project uses secondary data, it is essential to have a place to track each stage of the project as you unpack the various data features and steps involved when working with secondary and new data sources. So the value in keeping a project tracking journal extends beyond the initial project planning stages to include providing a place to keep various field notes, diagrams, tables, figures, and preliminary codebooks, which I outline below. The creativity that occurs while journaling during an active mixed methods project with secondary data makes it worth your time. For example, while journaling about your secondary data, you may uncover a new unit of analysis (individual vs. agency) or a new way to integrate data from your discipline with data from a different discipline (e.g., integrating data on the environmental causes of depression in school-age children with blueprints drawn by a landscape architect). My best ideas about how to present my findings to various stakeholders (e.g., the scientific community, lay community, political officials, etc.) come from my project tracking journals.

Active Project Writing Example 2: Keeping Field Notes Most researchers will not argue about the importance of field notes for documenting their research activities. So I will not cover that here. Instead, I want to highlight the importance of keeping field notes for new and secondary data for a mixed methods project. It can be helpful to keep notes about what occurs while you are collecting new data “in the field” (hence the term “field notes”) and then integrate those notes with the notes about your secondary data. Naturally, your mixed methods design determines the field notes you keep and whether you decide to use secondary data for one or both study components. For example, let’s say you are conducting an explanatory sequential design with secondary quantitative data. Since you already have access to your quantitative data, you may decide to journal about your evaluation (Chapter 3) of the secondary quantitative data in your project tracking journal and then use a different section of the same journal to document the field notes for your new qualitative data collection efforts.

174   Part III  •  Writing Mixed Methods with Secondary Data While collecting your qualitative data, your field notes can help track various aspects of your data, how elements of the new data align with your review of the secondary data, and how they contribute to your research questions and overall project goals. If you wonder about the potential overlap between your active project journal and your active project field notes, that is good. Essentially, you might be wondering what the differences are and if you might need to keep your journal and field notes in two separate places. On the one hand, my field notes are captured in the project journal because the journals I keep for my mixed methods projects with secondary data are usually an all-encompassing book, or log, of all activities associated with my project. On the other hand, my field notes are generally in a separate section of my project journal, dedicated to whichever study component’s activities require me to enter the field and complete new data collection. So you see, the field notes section of my journal may have more details about my intimate knowledge about the new data I need to collect for my project. I place my notes from the secondary data in a different section of the journal, all together. It is my responsibility to bring them together at some point, but isolating and focusing on each data component, apart from the other, is a valuable exercise for integrating new and secondary data in mixed methods. In short, your tracking journal may be where you keep all your notes for both study components, regardless of whether you use secondary data or not.

. . .the field notes you keep are determined by your mixed methods design and whether you decide to use secondary data for one or both study components.

Active Project Writing Example 3: Tracking With Diagrams, Tables, and Figures Creating diagrams, tables, and figures to track your mixed method project with secondary data can help you organize your study’s information while your research is underway. It might be tempting to only think about diagrams, tables, and figures at the beginning (i.e., during the planning) or the end (i.e., during the reporting) of your mixed methods with secondary data. However, I encourage you to begin thinking about how you can use diagrams, tables, and figures long before you reach the data collection and analysis phases of your study. I hope with the multiple steps you must complete to design and conduct mixed methods with secondary data, you think about various ways to maximize your active writing by using diagrams, tables, and figures. For example, I frequently create tables that disaggregate my conceptual framework’s constructs and link them to new and secondary data, as well as primary and secondary outcomes, for my mixed methods project. Though I may have developed design diagrams and procedural diagrams (Watkins & Gioia, 2015) during the early

Chapter 8  •  Early-Stage and Active Project Writing for Mixed Methods...  

175

writing stages, I might refine those diagrams during my active project writing. For example, before analyzing secondary quantitative data, I create tables for my study measures, primary outcomes, and reliability and validity measures. Oftentimes existing, supplemental files need to be critically examined before data analysis, integration, and interpretation takes place. Similarly, secondary qualitative data must be reviewed, evaluated, and inspected for appropriate measures of rigor (e.g., credibility, dependability, dependability, and transferability; Guba [B]egin thinking about & Lincoln, 1989; ­Tolley, Ulin, how you can use diagrams, Mack, Robinson, & Succop, tables, and figures long before 2016) before data analysis, you reach the data collection and integration and interpretation occurs. analysis phase of your study. Let’s consider an example where you need to analyze secondary quantitative data for your mixed methods project and draft a few table templates before creating them in Microsoft Excel® and filling them in with results from your statistical output. Sketching these tables in your tracking journal is a great way to draft preliminary table rows and column headers. (If there are previous publications from your secondary data source, then it would be important to review the tables from those publications and use them to draft your table templates.) Similarly and depending on your statistical analysis plan for the quantitative data, you may need to sketch some regression models (complete with mediating and moderating variables, if applicable) before running the analysis and reviewing the output. Suppose you are in the process of analyzing secondary qualitative data for a mixed methods project. In that case, you might consider developing a diagram or a figure that illustrates your project’s overarching themes before using them to make decisions about your project’s quantitative methods (if your study is sequential) or merging them with the quantitative results (if your study is convergent). If the study is sequential and you plan to collect new data for the second study component, you might sketch the sections of the survey questionnaire and develop questions that fall under each section. Qualitative themes from the first component could even inspire the need for valid and reliable surveys you can embed in a new quantitative questionnaire. For this example, the quantitative data could be new or secondary, and you could use your tracking journal to sketch out how you integrate and interpret the results from the two study components.

Active Project Writing Example 4: Developing Codebooks When you began reading this section of the chapter, I bet the first active project example that came to your mind was the project codebook. This makes sense because most researchers default to conversations about the codebook,

176   Part III  •  Writing Mixed Methods with Secondary Data regardless of whether they are discussing the qualitative or quantitative component for their mixed methods project. Qualitative codebooks usually include qualitative study protocols, codes, themes, and contextual information used to organize and interpret the qualitative data. Quantitative codebooks usually have survey items, variable names, variable labels, syntax, output files, and additional information to help describe ways to best understand and analyze the quantitative data. Reviewing codebooks (for secondary data) and developing codebooks (for new data) may occur during the active project writing stages of mixed methods research. How you handle codebooks depends on whether you use secondary data for both study components or one secondary data source and one new data source. Regardless, codebooks are necessary for completing your project’s mixed methods data analysis, integration, and interpretation. If you are doing mixed methods with secondary data, my advice is to begin reviewing (or developing) the codebooks while your mixed methods research is underway, if not sooner. This is because while you are working on your project, your team may generate helpful information and subsequent resources (from journaling, keeping field notes, and creating diagrams, tables, and figures) that can be included in the codebooks while the mixed methods study is active and after it is completed. Suppose you gain access to a codebook that is part of a secondary data set. In that case, you will need to make decisions about whether to (i) use the secondary TABLE 8.3  ● Considerations When Working With a Secondary Quantitative and Qualitative Data Codebook

Quantitative

Codebook Consideration

Qualitative

Decide whether the variable labels, variable structures, and levels of measurement align with your plans for analysis and integration.

Use the secondary data from the codebook in whole or in part

Decide whether secondary codes (and definitions), coding structures, constructs, and themes align with your analysis and integration plans.

Decide whether to recode variables (e.g., collapsing categories from a Likert scale) for analysis and integration.

Recode the secondary data from the codebook you plan to use

Decide whether to recode constructs (e.g., change language based on context) for analysis and integration.

Decide whether to develop a new quantitative codebook that redefines and reclassify variables for analysis and integration.

Develop a brandnew codebook from scratch

Decide whether to rename secondary codes (and develop new definitions), reorganize coding structures, and reframe qualitative context for analysis and integration.

Chapter 8  •  Early-Stage and Active Project Writing for Mixed Methods...  

177

data from the codebook in whole or in part, (ii) recode the secondary data from the codebook you plan to use for your project, or (iii) develop a brand-new codebook. Depending on the research question and the fit of the secondary data based on the evaluation criteria (Chapter 3), you may find yourself doing all three. Table 8.3 illustrates what each task looks like for secondary quantitative and qualitative data. I have worked with research teams that determined the primary outcome of the study would be a codebook (i.e., no papers, presentations, or other products at this stage). I have also worked with (and led my own) teams where the codebook was not the primary study deliverable. Instead, another project deliverable (e.g., policy brief, book chapter, peer-reviewed manuscript, etc.) was the focus of the work; the codebook was a secondary project deliverable we developed along the way.

Tips to Maximize Early-Stage and Active Project Writing Investments in early-stage and active project writing will pay off. But you must engage in these practices to make the most of your early-stage and active project writing experiences. I encourage you to consider some additional tips to maximize those early and active project writing stages your mixed methods with secondary data. In the following sections, I review five tips (Table 8.4) you can apply to your work.

Tip #1: Start Early Experts agree on the importance of starting early when writing up research (Clark, Foster, Sloan, & Bryman, 2021; Watkins & Gioia, 2015). In theory, this is a reasonable recommendation. Yet, in practice, few scholars start the writing process early due to intrinsic (e.g., fear, anxiety, impostor syndrome, etc.) and extrinsic (e.g., money, time, people to assist, etc.) forces that delay the start of the work. I understand that sometimes the forces that influence our abilities to start early are out of our control. But rather than focus on that, perhaps a more practical question we should answer is: how early is too early? Also, how do you define “start?” I believe it is never too early to start writing about mixed methods with secondary data. So start now. And while your definition of “start” may vary from project to project (think back to the informal vs. formal writing process described earlier in the chapter), anything you do related to the project is, by definition, starting the work. Just as you read in the vignette at the beginning of this chapter, Donald was going to wait until he “officially” started his mixed methods dissertation before he began using his new journal. But why wait? What I have learned over the years is that though there are official start dates to projects for funders and research oversight committees, I, personally, do not have official project start dates. I suggest you begin outlining the initial thoughts for your mixed methods with secondary data as early as possible. The earlier you start thinking about your project ideas and ways to incorporate secondary data into a mixed

178   Part III  •  Writing Mixed Methods with Secondary Data TABLE 8.4  ● Tips for Maximizing Early-Stage and Active Project Writing No.

Writing Tip

Application

1

Start early

Start thinking about the project and tracking the work as soon as possible

2

Review previous literature

Acknowledge how well you understand the studies before yours and if your research helps fill the gaps in those studies

3

Use templates for inspiration

Track how other researchers have documented their mixed methods with secondary data and use their templates as inspiration

4

Use technology

Choose technology that helps, not hinders, your progress

5

Collaborate

Expand the definition of collaborating to include thought-leaders who may offer a one-time consultation on your mixed methods project

methods study design, the better. I have never regretted documenting my thoughts about a project early and often and in the same place (e.g., a journal). This has saved me time and resources (e.g., money, staff effort, etc.) in the long term.

Tip #2: Review Previous Literature I do not need to convince you of the importance of literature reviews. There are books about using mixed methods to synthesize the literature (Heyvaert, Hannes, & Onghena, 2017), not to mention books that present strategies for systematically reviewing and synthesizing literature (Garrard, 2016; Galvan & Galvan 2017; Onwuegbuzie & Frels, 2016; Machi & McEvoy, 2012). A significant part of writing for mixed methods with secondary data must include a thorough search and synthesis of previous studies highlighting the substantive topic under investigation and the methods used to examine that topic. The success of early-stage and active project writing for mixed methods with secondary data depends on how thoroughly you understand the studies that have come before yours. For example, let’s assume you are planning to develop a new focus group study from the results of secondary survey data about adverse childhood experiences (ACES) among low-income children in Baltimore. If you review the literature and see that focus groups with parents have been used for most studies to date, then consider if your use of a focus group will add something new and innovative to the existing literature on the topic. If not, one alternative might be to build the qualitative study component using a different data collection method (e.g., individual interviews,

Chapter 8  •  Early-Stage and Active Project Writing for Mixed Methods...  

179

participant observation, ethnography, etc.) to expand on previous research while addressing your research question. Your review of prior literature is not merely to build a case for the need for the study. It can also justify using data collection methods to understand the topic better and expound on previous methods used to study the topic. In mixed methods with secondary data, reviewing the literature is another way to learn about secondary data sources on your topic and explore whether you can use these secondary data for your project.

Tip #3: Use Templates for Inspiration Early-stage and active project writing will sometimes require forms, spreadsheets, charts, or project management software to evaluate the secondary data for a mixed methods project. It can be intimidating to take on a mixed methods project with secondary data and then be expected to generate new tracking programs, systems, and data logs at the beginning of your project or while your project is underway. My advice is to let previous models inspire you. Do not recreate the wheel (so to speak). Instead, see how other researchers have documented their mixed methods with secondary data. Sometimes, this kind of information can be found in journal articles (see Gradinger, Elston, Asthana, Martin, & Byng, 2019; Lindsey & Bulloch, 2014 for examples), and sometimes, you may have to search outside of the scientific literature (e.g., blog posts, social media, news media, etc.) to learn about the types of files your colleagues use to track their mixed methods studies. This would also be an excellent opportunity to talk with colleagues and relevant stakeholders (who might be proficient in a single-method qualitative or quantitative study) about how they track their project activities. Some might use forms, templates, Gantt charts, or project management software to track their active project writing. Others might use synchronous programs that allow them to share files with their team (i.e., Google Drive), search Google to locate sample models, or perhaps just create their own. My point here is to encourage you to think beyond your cur. . .talk with colleagues and rent tools and begin asking relevant stakeholders about people in (and outside) your how they track their project research circle about the tools activities. . . that make their active projects easier to manage. Then consider adapting their tools for your purposes. For mixed methods with secondary data, you may not need to look far to find applicable models to adapt for tracking your early-stage and active project writing. For example, suppose you can access secondary quantitative data from a government agency, like the Centers for Disease Control and Prevention (CDC). In that case, you might be able to access tracking and technical assistance resources directly from the CDC’s website. Well-resourced secondary

180   Part III  •  Writing Mixed Methods with Secondary Data data depositories (see Chapter 3 for a list of data repositories in the health and social sciences) often have forms, templates, charts, and data management systems you can use or adapt for your purposes. These kinds of resources may inspire you to format your active project writing in a way that maximizes your creation and use of the files you produce and inform how your future mixed methods with secondary data are designed, implemented, and evaluated.

Tip #4: Use Technology Technology has changed how we do research, influencing study recruitment, data collection, data analysis, and even reporting our project findings. Therefore, I encourage you to consider using technology for your early-stage and active project writing. Whether you prefer programs in the Microsoft Office® Suite, Google Workspace, via Apple, or a different suite altogether, various word processing and spreadsheet structures are intuitive and offer a range of options for your early-stage and active project writing. While each has its unique features, they all have key elements to help log your early-stage and active project writing. For example, if a suite is linked to the cloud (e.g., Google), you can do synchronous and asynchronous writing with other members your research team. As for the mechanics of synchronous versus asynchronous communication with your team during early-stage and active project writing, Janet Salmons (2016) offers a useful time response curriculum ­(Figure 8.1) that can be adapted for online teamwork. These options range from synchronicity (i.e., focused, real-time dialogue) to asynchronous (i.e., time-lapses between messages and responses). Though Salmons’ diagram illustrates variations in data collection and the communication exchanges between research teams and study participants, I encourage readers to also consider these options as they think about maximizing FIGURE 8.1  ●  Time Response Continuum Synchronicity Foucsed real-time dialogue

Synchronous Exchange in real time—other events may also occur

Near-Synchronous Near-immediate, ongoing post and response

Asynchronous Time lapse between message and response

Time-Response Continuum Videoconference, video chat or call

Podcast, vodcast

Text message or chat Shared applications

email Image or file sharing

Web conference, or webinar MOOGs, virtual worlds

Link to recording Posts to forum, blog or social network

Source: Salmons, J. (2016). Doing Qualitative Research Online, p. 46.

Chapter 8  •  Early-Stage and Active Project Writing for Mixed Methods...  

181

technology for themselves and their research teams when doing mixed methods with secondary data. An example of applying Salmons’ continuum to mixed methods with secondary data is to consider using a Google Doc to draft preliminary ideas for a convergent design that uses two secondary data sources. Suppose you were to share the link to this Google Doc with five of your team members. In that case, you might discover some of your team members working in the document at the same time during the same 3-hour period each day (i.e., simultaneously), while others may work in the document in 10–15-minute spurts sporadically, over several days. This process maximizes early and active project writing as somewhere in between a synchronous and near-synchronous process for mixed methods with secondary data. Another example might help elucidate the importance of this tip and illustrate how it has benefited me. A few years ago, I was doing a project that involved delivering an online program, for which I was the group “facilitator,” and three of my doctoral students served as the group “managers.” These titles simply meant I was responsible for sharing the daily program content with participants, and my students were responsible for monitoring our participants’ responses to the content. While this project was underway, I found it helpful to create a team journal for my students and me to log our behaviors and reactions to the ways our participants responded to the program content. It made sense for us to develop a Google Doc and use it as our team journal. We wrote in this team journal regularly, dated our notes, and placed our initials next to each entry. This allowed my group managers to track when I posted content, how I thought program participants might respond to it, and suggestions for engaging seemingly quiet (i.e., minimally engaged) participants in the program. My group managers also used our team journal to track and log how they interacted with the participants throughout the program. Having a cloudbased team journal that allowed us to follow our program delivery was the best way to monitor our progress and hold one another accountable for the work we agreed to complete for the project. For mixed methods with secondary data, technology can be used to document secondary data sources, share secondary data files, map secondary data concepts onto new data collection methods, and outline the integration process. The entire process for evaluating secondary data for a mixed methods project (Chapter 3) can be created, examined, and managed using technology.

Tip #5: Collaborate Though I understand institutional or structural barriers mean some readers (e.g., students and independent scholars) may not be able to collaborate with others on their mixed methods projects with secondary data, in this section I will describe the ways collaborating during the early-stage and active project writing processes for mixed methods with secondary data can be advantageous to your overall study goals and objectives. My definition of “collaborate” in this context is flexible and fluid. While I recognize and have advocated for collaborations in mixed methods with secondary data throughout this text, I

182   Part III  •  Writing Mixed Methods with Secondary Data do not want to limit the definition of collaboration to mean only the people who can help you evaluate, collect, or analyze the data for your mixed methods with secondary data. If possible, you should definitely work with those people. Yet it is also important to develop a list of thought-partners who might not have a specific role in the day-to-day operations of your project but who might be willing to engage with you via email or videoconference to talk through the early and active stages of your project, the challenges you face, and the decisions you make along the way. Given the ways technology and social media have revolutionized the way we connect with colleagues, you might also consider joining a virtual mixed methods community and using that space to exchange ideas with novice and experienced mixed methods researchers alike. Connections such as these can also mean you widen the community of colleagues who can help you identify secondary data sources for your mixed methods projects. I believe the best collaborations are reciprocal. So be sure to glean from others’ wisdom in those spaces and joyfully provide some of your own as you see others working through their mixed methods projects. For example, if you create or join a community of mixed methods scholars, there might be colleagues in the group who not only can share secondary data sources with you but also can help you think about evaluation criteria for the secondary data (Chapter 3), review your procedural diagrams, and share resources they come across related to your work. In exchange, they would expect the same support from you for their work. I have found collaborating during early-stage and active writing for my mixed methods with secondary data to have numerous advantages to my project relevance, direction, and support. I have benefited from face-to-face collaborations and virtual communities that connect mixed methods scholars from across multiple continents. One of the things I have learned about mixed methods research is that the more you do it, the more you learn, and the more it changes. Staying in touch with people with varying levels of knowledge about and experience with mixed methods (i.e., from experts to budding scholars) will help you maximize the early-stage and active writing for your project and help you maximize secondary data. These people can provide feedback on your work, help you structure your research design and work plan, remind you to keep the audience in mind, and share some of their own experiences designing, doing, and evaluating mixed methods with secondary data.

Summary This chapter presented the writing that occurs in your mixed methods with secondary data during your project’s early and active stages. Now that you have read this chapter, you should be able to define what early-stage and active project writing are and what they look like for mixed methods with secondary data. Early-stage writing is any preliminary information, ideas,

Chapter 8  •  Early-Stage and Active Project Writing for Mixed Methods...  

183

and work plans you draft and document for your mixed methods research at the beginning of the project. Active project writing involves the various ways you gather information, document, and track the ongoing procedures, protocols, and findings associated with your mixed methods study. I also provide five examples of early-stage writing (i.e., journaling for planning purposes; developing research proposals; planning with diagrams, tables, and figures; drafting study protocols; and developing training materials) and four examples of active writing (i.e., journaling for tracking purposes; keeping field notes; tracking with diagrams, tables, and figures; and developing codebooks) to consider in your mixed methods with secondary data. I end the chapter with five tips for maximizing early-stage and active project writing for mixed methods studies with secondary data. Starting early, reviewing previous literature, using templates for inspiration, using technology, and collaborating should inspire you and generate other tips as you engage in the early and active writing for your mixed methods with secondary data.

Chapter 8 Application Questions 1. Whether they do single-method studies or mixed methods projects, many scholars find it helpful to document the early stages of research projects to keep track of their plans, actions, and emerging ideas. When thinking about your research area and your mixed methods with secondary data, why might it be helpful to write during the early stages? 2. Which early-stage examples from this chapter seem most helpful in your research? Do you think these practices are specific to your discipline or topic under study? Explain. 3. This chapter describes documents you can produce from the earlystage writing of mixed methods with secondary data. Compile a list of the documents or files you think are essential during the early writing process. How might this list apply to your research project? 4. This chapter describes documents that could be developed during active project writing for your mixed methods with secondary data. Which documents or files described in this chapter might be most helpful to you, and how do they help your active project writing? 5. When thinking about your mixed methods with secondary data, are there any advantages and disadvantages to using an electronic (i.e., computer) versus a paper (i.e., hard copy) system for early and active project writing for some mixed methods designs (e.g., convergent, sequential, complex, etc.) over others?

9 Reporting Mixed Methods With Secondary Data Imagine Your neighbor, Lucy, has just completed a big research project at work, and she is describing it to you one August afternoon as the two of you lean over the fence separating your backyards. “I didn’t think we would ever finish the data collection for this project. Now, my boss is ready to begin the next project, and all I want to do is rest!” “Geez,” you sigh as you bend down to swat a mosquito from your leg, “You all spent so much time collecting the data; I know you can’t wait to analyze it and share it with others.” Lucy puts her head down and shakes it slowly. “That’s the thing,” she says, “my boss wants me to lead the report writing for this project, and I don’t know where to start. I know what to do when we collect new data, but this project used some archived data. Hmm. . . I don’t know. I guess I’ll just figure it out.” You nod your head and think about the challenge before Lucy. You bid farewell to your neighbor, and as you walk back to your house, you think about how resources for writing research proposals seem to outnumber those describing how to report and disseminate research findings. Lucy seems overwhelmed by the idea of putting together the final report, which is not uncommon among researchers. If this scenario sounds familiar, then this chapter is for you. Who knows? You might even learn something helpful to share with Lucy.

185

186   Part III  •  Writing Mixed Methods with Secondary Data

Learning Objectives This chapter discusses how to write up the final report for a mixed methods project with secondary data. Compared to early-stage and active project writing, there are different matters to consider while reporting mixed methods with secondary data. I outline some reporting examples and provide tips for maximizing writing opportunities after you have completed your mixed methods project with secondary data. By the end of the chapter, you will be able to: 1. Define reporting for mixed methods with secondary data, 2. Identify project deliverables for mixed methods with secondary data, 3. Describe challenges with reporting mixed methods with secondary data, and 4. List some tips for reporting mixed methods with secondary data.

Reporting Mixed Methods With Secondary Data Now that we have covered early-stage and active project writing (Chapter 8), I want to describe the type of writing that occurs at the end of your mixed methods project with secondary data, also referred to as “reporting.” Reporting a mixed methods project with secondary data is unlike early-stage writing, where you plan and design your research. It is also unlike active project writing, where you work through the day-to-day elements of your research. Reporting is the process used to gather, integrate, interpret, and disseminate what you learn from your mixed methods research (Leech, Onwuegbuzie, & Combs, 2011; O’Cathain, 2010; Watkins & Gioia, 2015). The investment you make in creating outlines and drafts for your manuscripts, conference presentations, and final reports during early-stage and active project writing (­Chapter 8) pay off when you are ready to develop project deliverables (i.e., report your findings). This chapter offers tips for sharing the results from mixed methods with secondary data by answering questions like What deliverables can be produced from mixed methods with secondary data? What information should be included in those deliverables to describe your decision to use secondary data in mixed methods research? Depending on the motivation behind the mixed methods with secondary data, you may decide to draft manuscripts, conference presentations, final reports, and other documents after your project has ended. Several authors have described the importance of producing outlines and manuscript drafts when reporting research (Badke, 2017; Wolcott, 2009), so I will not cover that. Instead, I will highlight considerations for drafting manuscripts, conference presentations, and final reports for your mixed methods with secondary data.

Chapter 9  •  Reporting Mixed Methods With Secondary Data  

187

Let’s begin by considering the needs of your intended audience (i.e., readers of your report). Though you have invested weeks or months (maybe even years) planning, designing, and conducting your mixed methods study with secondary data, your readers will want to read a report that is concise and to the point. Your drafts should be written for the intended audience (Fetters, 2020; Watkins, 2017a, 2017b) and include as much or as little information as you need to communicate your point. You may find most audiences want to know your study’s primary outcomes; in short, an answer to your guiding research question(s). On the other hand, a different set of readers might find value in knowing your methodological trajectory, or what led you to choose a mixed methods design over a single-method design, and what prompted you to use secondary data rather than collect new data. One way to address your readers’ various needs and maximize the findings from your mixed methods with secondary data is to diversify the number of project deliverables you produce from your mixed methods study.

Project Deliverables for Mixed Methods With Secondary Data Reporting your mixed methods with secondary data is more than sharing project outcomes. It can also include sharing additional findings, experiences, and nuances you learned while planning and conducting the research. This is especially true for mixed methods that use secondary data, as for some readers, the techniques used may be new and unexplored. For instance, think about your colleagues who have trouble integrating theory into their mixed methods with secondary data. They could benefit from learning how you successfully integrated and applied theory from an existing data source in your mixed methods with secondary data (i.e., a theory application report). Or you may work alongside colleagues who are having trouble recruiting participants for the new quantitative study they hope to integrate into their secondary qualitative data for their mixed methods project (i.e., a study participant report). You can create one or more of these project deliverables during the reporting stage of your mixed methods project with secondary data. Table 9.1 outlines five potential project deliverables from mixed methods with secondary data, potential challenges for each, and a sample report from the literature. Though not an exhaustive list, these should inspire other deliverables in your field and for the topic under study. Mixed methods yield large volumes of information. When you include secondary data, the large volumes of data become more complex to manage as you make your way through someone else’s methodological decisions, data, and outcomes. Therefore, when developing deliverables for mixed methods projects with secondary data, it is imperative to consider multiple ways to share what was gained while planning and conducting the research. The techniques you use may be new and unexplored to you but could guide future project plans for you or someone else.

188   Part III  •  Writing Mixed Methods with Secondary Data TABLE 9.1  ● Potential Deliverables When Reporting Mixed Methods with Secondary Data Challenges When Using Secondary Data

Deliverable

Definition

Sample Reports

Project outcome report

Describes the results and overall outcomes for mixed methods with secondary data. This may include the results for one or multiple project outcomes and at one point in time or over a period of time.

Being clear about the origins of the secondary data and how repurposing them addresses the overall research question and study purpose in a way new data cannot.

Kang, B., Pan, W., Karel, M. J., McConnell, E. S. (2021). Rejection of care and aggression among older veterans with dementia: The influence of background factors and interpersonal triggers, Journal of the American Medical Directors Association,22(7): 1435—1441.

Protocol and procedure report

Describes the procedures for mixed methods with secondary data from conceptualization to study completion. Includes study protocols (i.e., questionnaires, training, research conduct, human subjects, etc.). May include barriers and facilitators to conducting the study.

Identifying and securing supplemental files that clearly describe the protocols and procedures used by the original study team to collect, analyze, and report the original data.

Trotter II, R. T., Camplain, R., Eaves, E. R., Fofanov, V. Y., Dmitrieva, N. O., Hepp, C.M., Warren, M., Barrios, B. A., Pagel, N., Mayer, A., & Baldwin, J. A. (2018). Health disparities and converging epidemics in jail populations: protocol for a mixed methods study. JMIR Research Protocols, 7(10):e10337.

Theory application report

Describes how theories or conceptual frameworks were generated before, applied during, or tested after mixed methods with secondary data.

Having a comprehensive understanding of how theory was applied to the original study team’s research.

Doi, L., Morrison, K., Astbury, R., et al. (2020). Study protocol: A mixed methods realist evaluation of the Universal Health Visiting Pathway in Scotland, BMJ Open, 10: e042305.

Chapter 9  •  Reporting Mixed Methods With Secondary Data  

Challenges When Using Secondary Data

189

Deliverable

Definition

Sample Reports

Study participant report

Describes participant demographics and how researchers engage with the study participants/ respondents (e.g., sampling, recruitment, retention, etc.) for mixed methods with secondary data.

Secondary data may exclude some, or most, information about the study participants, demographics, recruitment, and retention strategies.

Arends, I., Bultmann, U., Shaw, W. S., van Rhenen, W., Roelen, C., Nielsen, K., van der Klink, J. J. (2014). How to engage occupational physicians in recruitment of research participants: A mixed methods study of challenges and opportunities. Journal of Occupational Rehabilitation, 24 (1): 68–78.

Recommendations and solutionoriented report

Summarizes recommendations (i.e., “take-aways”) using solutiontargeted language determined by the study team after a mixed methods study with secondary data has been completed. They may also be called practice articles or “white papers.”

Knowing whether the original study team would have come to the same recommendations based on the intent of their initial study.

Rankin-Williams, A. C., Geoffroy, E. M., Schell, E. S., Mguntha, A. M. (2017). How can male rates of HIV testing be increased? Recommendations from a mixed methods study in southern Malawi, International Health, 9(6), 367–373.

Challenges (and Solutions) When Reporting Mixed Methods With Secondary Data While the project deliverables for mixed methods with secondary data are promising, you may face challenges when developing these deliverables. It may be challenging to know what information to include in the project deliverables for a mixed methods study with secondary data because of challenges associated with reporting mixed methods research in general. Here, I outline seven challenges you may face when reporting mixed methods with secondary data and techniques to overcome them.

190   Part III  •  Writing Mixed Methods with Secondary Data First, there is the challenge of limited examples. There are not (yet, at least) many examples of mixed methods research that use secondary data you can use as models. Current examples are sometimes difficult to find because an author’s use of secondary data is not always recognizable via article searches. This presents a challenge because models that outline the strategies used by other researchers are helpful when developing reports of your own. Mixed methods are evolving, and new tools and techniques are emerging rapidly. However, sometimes researchers developing and growing the knowledge base on new mixed methods techniques, particularly those with secondary data, do not disseminate their findings, leaving no blueprint for other researchers interested in adopting similar approaches in their work. Second, there is the challenge of extensive information. Overall, mixed methods studies are hard to report due to the vast amount of information included and managed in these projects. The growing body of research on reporting mixed methods acknowledges the robust nature of mixed methods studies and the challenges with reporting (Leech, Onwuegbuzie, & Combs, 2011). When using secondary data, there may be an abundance of information to be organized, managed, and interpreted to complete the mixed methods project. Therefore, using the strategies outlined in this book and evaluating the secondary data to assess their fit with your mixed methods study are essential for success. The third is the challenge of terminology. While writing this book, I grappled with whether I should use “secondary data” or “existing data” to describe the types of data you might incorporate into a mixed methods project. For fear that readers would get lost in my descriptions, or worse, be confused by repetitive uses of the terms “primary” and “secondary,” I opted to use “secondary data” and “existing data for secondary purposes” interchangeably throughout this text. You may face similar challenges as you decide how to write up your mixed methods with secondary data. My advice is to choose which terms you will use to discuss the secondary data early in your mixed methods project. Beyond the challenge of terminology are challenges associated with using terms like data “collection” versus “gathering” or “assembling.” You will need to decide how these terms are defined and operationalized in your mixed methods with secondary data. For instance, one could argue that new data are “collected,” while secondary data are “gathered” or “assembled.” In short, it is important to make decisions about the language used to describe your secondary data and how they are generated so that you and your readers can be aligned in your understanding and interpretation of the methods and the findings. The fourth challenge when reporting mixed methods with secondary data is the challenge of method depiction. When using secondary data for one or both study components, it is essential to depict when the secondary data are used. I have modeled this throughout this book by naming a study component based on its mixed methods design, the method, and when a data source is new or secondary. For example, you might consider naming a mixed methods project “an explanatory sequential design with secondary survey data,” in which you

Chapter 9  •  Reporting Mixed Methods With Secondary Data  

191

do a secondary analysis (which signals analyzing existing data) of census data (which signals a quantitative data source) and use that to develop a new (which signals new data collection) focus group questionnaire (which signals a qualitative data source) for your project. The fifth challenge is the challenge of accurate acknowledgment. When using secondary data in mixed methods, accurately describing and giving proper credit to the secondary data origins, team, and sources is vital. After evaluating the secondary data source using review criteria (Chapter 3), it may feel like an enormous amount of work has been done on your part and the part of your team. However, your evaluation of the secondary data would have never been possible had the original project team not completed their primary data collection. Therefore, when writing up the final report and other deliverables for mixed methods with secondary data, be sure to acknowledge and cite the original researchers appropriately. Do this in such a way where your use of their secondary data and documents does not duplicate or plagiarize previous deliverables but instead highlights the value and contributions of their original research. The sixth challenge is the challenge of differentiating samples. When writing up mixed methods with secondary data, distinguish between the samples used and whether they are secondary or new. If the mixed methods with secondary data use a convergent design, clearly describe which data sources exist and which ones require new data collection efforts. If the mixed methods design uses a sequential design, explain the connection made to or from the secondary data. When deciding how to write up mixed methods for which both the qualitative and quantitative data components are derived from secondary sources, this requires a close and critical eye on the study components, from the beginning to the end. Pay close attention to the secondary data, methods, and samples. As I have noted in previous chapters, journal about how you will manage new or secondary quantitative and qualitative samples throughout the life of the project. The seventh challenge when writing up mixed methods with secondary data is the challenge of clarity. It may be difficult to clarify what procedures you underwent, even if some or all data exist. In other sections of this book, I acknowledge mixed methods are not for the faint of heart. Adding complexity to an already complicated research method means some researchers would rather avoid these additional challenges than face them head-on. So, whether the data you are reporting in your mixed methods research are new or secondary, be transparent and honest about how you will use those data.

Tips for Reporting Mixed Methods With Secondary Data Several scholars have outlined research reporting guidelines, techniques, and tips for both single-method (Clark, et al., 2021; Rocco & Hatcher, 2011; Thyer, 2008; Verhoeven, 2011; Wolcott, 2009) and mixed methods studies (Bazeley,

192   Part III  •  Writing Mixed Methods with Secondary Data TABLE 9.2  ● Ten Tips for Writing Mixed Methods Reports With Secondary Data Report Section

Tip

Introduction/ background

1: Introduce the value-added by using secondary data to achieve the study purpose and answer the research questions

Methods

2: Outline the advantages of using secondary data in the study design

Methods

3: Discuss alignment between the secondary data and the research question

Methods

4: Describe the appropriateness of the secondary data for successful analysis and integration

Results

5: Align the secondary results with study findings and inferences

Results

6: Present the secondary results so they clearly illustrate their contributions to the study

Discussion

7: Report how using secondary data contributed to theory

Discussion

8: Share limitations when using secondary data

Discussion

9: Discuss how using secondary data contributed to furthering knowledge on the topic

Discussion/ conclusion

10: Discuss how using secondary data contributed to furthering mixed methods

2015; Creswell & Plano Clark, 2018; Fetters, 2020; Watkins & Gioia, 2015). I refer you to these sources for more information as some of these sources offer guidelines for handling various stages in the reporting process. For instance, some sources provide advice about handling manuscript rejections, which are a normal part of the research reporting process. With most rejections, poorly done or poorly explained research methods are the main reasons for an unfavorable editor decision (Fetters & Freshwater, 2015). So much of the editor’s decision is based on how you present the findings of your mixed methods with secondary data. To strengthen the quality of your submission, I propose 10 tips for reporting mixed methods with secondary data and the corresponding sections of your final report in which this information can be found (Table 9.2).

Tip #1: Introduce the Value-Added by Using Secondary Data to Achieve the Study Purpose and Answer the Research Questions Your final report’s introduction and background sections are where you review the literature, make a case for why the research study is needed, and outline the valuable contributions it will make to the field. I recommend using

Chapter 9  •  Reporting Mixed Methods With Secondary Data  

193

this section of the final report to expound on the value-added using secondary data in your mixed methods project. The easiest way to introduce the valueadded by using secondary data is to use guiding questions to frame your writing. For example, asking questions such as: How will using secondary data help me achieve my study purpose in ways that I may not be able to do if I collect new data? (see Chapters 2 and 3 for help with addressing these questions).

Tip #2: Outline Advantages of Using Secondary Data in the Study Design After presenting the idea of secondary data in the introduction and background section of your final report, you can outline the advantages of using secondary data in the study design and methods sections (e.g., theory or methodological framework, study design, and sampling procedures). Provide a clear justification and scientific purpose for using secondary data in your mixed methods project. Therefore, describe the advantages of using secondary data to achieve your study purpose and address the mixed methods research questions. Some questions that can help you outline the benefits of using secondary data in the study design include: How would the design differ had you not used secondary data? What about the secondary data enhanced your ability to execute the mixed methods study in ways that would have been different had you used new data? (see Chapters 2 and 3 for help with addressing these questions). The methods section also discusses the characteristics of the participants who provided the secondary data, how the data were collected, and other information known about the secondary data sources. The key here is to focus on the advantages of using secondary data in your mixed methods study and note the enhancements secondary data offer to the project. Detailing the benefits of using secondary data in the mixed methods report’s study design section is essential. Readers may wonder why you did not collect new data to address your research questions. So being transparent about what secondary data add to the study will help readers understand your decision to use secondary data and help them follow the context, conditions, and decisions you used for your project.

Tip #3: Discuss Alignment Between the Secondary Data and the Research Question When using secondary data in mixed methods studies, align the secondary data with study concepts. Researchers using secondary data in mixed methods cannot change the secondary data. Instead, they must evaluate the secondary data quality and fit, and find ways to map their overall study goals and objectives onto those found in the secondary data. When writing up the report for a mixed methods study with secondary data, it is essential to discuss how and why you aligned the secondary data with the overall study concepts. In mixed methods studies where one data source is secondary and the other is new, this

194   Part III  •  Writing Mixed Methods with Secondary Data alignment will help readers see how you connect new and . . .discuss how and why you existing concepts. When writaligned the secondary data ing the methods section, think with the overall study concepts. about these guiding questions: How did the secondary data capture study concepts? How are secondary data being operationalized to understand the topic under study better? (see Chapter 3 for help with addressing these questions). Another suggestion is to use the methods section of your mixed methods report to outline the measures of rigor operationalized in the qualitative and quantitative procedures. Be sure to refer to the measures of rigor most appropriate for qualitative research (i.e., credibility, dependability, confirmability, and transferability) and the measures of rigor most appropriate for quantitative research (i.e., validity, reliability, objectivity, and generalizability; Ulin, Robinson, & Tolley, 2004). This improves the quality and the trustworthiness of your mixed methods with secondary data and assures readers that you genuinely understand the difference between quantitative and qualitative methods. Furthermore, these additional steps confirm that you applied and reported the appropriate measures to assess study rigor based on each research method, rather than using blanket quantitative measures of rigor for both the quantitative and qualitative study components. If previous reports exist for your secondary data, those reports can also be used to report on the quality of your secondary data source. Furthermore, existing data used for secondary purposes might seem more reliable if their use across multiple studies is acknowledged, confirmed, and amplified.

Tip #4: Describe the Appropriateness of Secondary Data for Successful Analysis and Integration Whether the secondary data are qualitative, quantitative, or both, you must describe the data’s appropriateness for addressing the research question once analyzed and integrated for mixed methods purposes. This information can be included in the methods section of the final mixed methods report. It can outline the analysis plan, the quantitative and qualitative methods used, and how researchers accounted for secondary data sources. Integration is essential to all mixed methods research. Therefore, it is also helpful to describe the appropriateness of using secondary data during the integration or “mixing” stage of your mixed methods with secondary data. Suppose you or your team made special considerations when integrating secondary data with other new or existing qualitative and quantitative data. This information should be described in the methods section of the final report. An example of a special consideration might be with secondary qualitative data, such as focus group videos, that need to undergo multiple iterations of data reduction or data transformation (e.g., transcription, creating respondent profiles, listing study site descriptions, conducting individual analysis, doing

Chapter 9  •  Reporting Mixed Methods With Secondary Data  

195

group-based analysis, etc.) before the findings can be integrated with the new or secondary quantitative data. Also, it would be important to determine which iteration (i.e., version) of the secondary data (from the data reduction and transformation process) was most appropriate for integration to address the research question(s). This information is necessary for any research project but is especially important in mixed methods with secondary data.

Tip #5: Align the Secondary Results With Study Findings and Inferences Using secondary data in mixed methods is more than just repurposing an available data source for a secondary purpose. It must also involve aligning the secondary data with your study’s findings and inferences. When using secondary data to generate results for a mixed methods project, it is important to discuss how the results of those secondary data determine—both by themselves and when integrated with the results from the other study component—how your overall goals were met and how the research questions were answered. For example, when using a secondary qualitative data source in a convergent design, closely assess the qualitative results generated from the secondary data, then describe how those results, both by themselves and when integrated with the new, or secondary, quantitative results, help to generate findings and inferences that build new knowledge for the topic under study. Additionally, the inferences determined by the secondary data should be interpreted in context. Some guiding questions to ensure this occurs are: Were the secondary data collected during different times under different economic or political circumstances? Were the secondary data collected during a different developmental stage of life for the population under study? (See Chapter 3 for help with addressing these questions.) These questions help align the ­secondary qualitative and quantitative data with study findings and inferences for your mixed methods research.

Tip #6: Present the Secondary Results So They Clearly Illustrate Their Contributions to the Study The final report for a mixed methods study with secondary data should include diagrams, tables, and figures, especially if such illustrations are appropriate for the audience. Creating visual representations of mixed methods with secondary data is helpful with interpretations of the study concepts, findings, inferences, and the ability to communicate them to others. Just like early and active project writing stages, creating diagrams, tables, and figures during the reporting stage can further develop previous interpretations of the data and organize those interpretations. Some readers may be thinking about joint data displays as diagrams, tables, or figures to produce in the late writing stage of a mixed methods project with secondary data (Guetterman, Fetters, & Creswell, 2015). I agree this

196   Part III  •  Writing Mixed Methods with Secondary Data is an excellent first step, but let’s take this idea a step further. My best diagrams, tables, and figures come from my report writing. Many of the joint data displays for my mixed methods studies are produced while drafting my final reports. Therefore, I encourage readers to take the visual elements of their mixed methods with secondary data a step further and think about other ways and other features to include in the diagrams, tables, and figures for the project (Fetters, 2020). For example, photographs from a photovoice project could be incorporated into a joint data display and network analysis maps, big data tables, geographic information system maps, and screenshots from social media posts. Joint data displays can be created in Microsoft Word, Excel, ­PowerPoint, or cloud-based programs such as Google Docs, Sheets, or Slides. But I have also seen colleagues make them in more sophisticated graphic design platforms such as Canva or Adobe Spark. The end goal should be to present the secondary data so that it illustrates its significant contributions to the mixed methods project and address the study purpose and research question(s).

Tip #7: Report How Using Secondary Data Contributed to Theory Using secondary data in your mixed methods project helps you advance the science of mixed methods and secondary data sources. Therefore, you need to also report on how the use of secondary data contributed to theory. Regardless of which mixed methods design (e.g., convergent, exploratory sequential, explanatory sequential, complex etc.) you choose for your mixed methods with secondary data, your overall study aim and purpose should build on theory or generate new conceptual frameworks to be tested in the future. The discussion section of your final report from your mixed methods with secondary data should propose other conceptual frameworks that would benefit from using secondary data to address the research questions. Some guiding questions for this section of the report include: Did using secondary data build on theory in ways that could not have been accomplished by collecting new data? Are there ways to integrate the theory from the secondary data with theory from new data that extends the scientific knowledge beyond what was previously known? (see Chapter 3 for help with addressing these questions). You can answer these types of questions in the discussion section of your mixed methods report.

Tip #8: Share Limitations When Using Secondary Data All research has limitations, and mixed methods with secondary data are no exception. Just as you should describe the advantages of using secondary data in your final report, I advise you to point out how using secondary data might have presented some challenges to your mixed methods study. Consider the seven challenges I described earlier in this chapter and include study limitations in the final report. These are helpful to readers because they help interpret results in the context of your study limitations. When writing the limitations sections for my mixed methods studies, I tend to think about using this as

Chapter 9  •  Reporting Mixed Methods With Secondary Data  

197

an opportunity to “call out” the challenges I experienced Rather than waiting for doing the research before someone else to identify the readers have a chance to call them out for me. I advise you potential problems with your to do the same. Rather than mixed methods using secondary waiting for someone else to data, do it yourself. identify the potential problems with your mixed methods using secondary data, do it yourself. Another way to think about the limitations of your mixed methods study with secondary data is to underscore vital aspects of the science to which the next researcher should pay close attention. For example, let’s assume a limitation of your mixed methods with secondary data was your use of a selfreported question: “Do you have depression?” instead of using a valid and reliable measure for depressive symptoms like the PHQ-9 or the CES-D. In the limitations section of your final report, you can note the challenges with using a single self-report item for depression and the benefits of using more valid and reliable measures such as the PHQ-9 and the CES-D. This acknowledgment will alert future researchers to the study design and psychometric advantages of using depression scales in their studies. Not to mention appropriately prompting readers to consider the psychometric properties of the measures they adopt from secondary data sources. Another example of a limitation is dealing with missing data in secondary quantitative data sources. Furthermore, sometimes questions on the survey change each time they are administered. Other potential limitations for secondary qualitative data include the accuracy of transcripts, challenges with interpreting interviews conducted by multiple interviewers, and not following up with the study participants to probe further.

Tip #9: Discuss How Using Secondary Data Contributed to Furthering Knowledge on the Topic When writing the discussion section of your mixed methods with secondary data, think about how you will use the results from secondary data to understand the research topic. Discuss how the final report makes a novel contribution to the literature and the substantive area of inquiry. For instance, reflecting on your mixed methods project with secondary data overall, how did your research question, study design, evaluation, and analysis of the secondary data and data integration build on previous literature on the topic? Did your findings complement those of previous studies, or did they contradict them? Looking ahead, how can future studies build on your work? Recount how using secondary data compares to previous research on the topic and how the studies that come after yours have a roadmap for further developing the research area.

198   Part III  •  Writing Mixed Methods with Secondary Data

Tip #10: Discuss How Using Secondary Data Contributed to Furthering Mixed Methods Just as it is important to discuss how using secondary data can further our understanding of a topic, it is also important for researchers who do mixed methods with secondary data to report how their study contributes to furthering the science of mixed methods. Because mixed methods are an evolving methodology and method (Creswell & Plano Clark, 2018), new discoveries happen every day. Therefore, it will be important to document how using secondary data in your mixed methods study helps further the science of mixed methods overall. Other researchers can learn from your decisions and procedures. For example, if a limitation of your research was that you only had access to secondary focus group data and you think the science could have been enhanced by new in-depth interviews, then describing ways in-depth interviews could have been valuable to the study would be a noteworthy recommendation in your final report. Also noteworthy is how your mixed methods with sec[D]ocument how using ondary data can be interpreted secondary data in your mixed in the context of the knowlmethods study helps further the edge-level continuum. You science of mixed methods. will remember from Chapter 2 that the knowledge-level continuum is a road map used to describe the process for scientific inquiry. All research studies can fall anywhere along the continuum depending on how much is already known about the topic. Think about how the knowledge-level continuum applies to your research. What information was already known about your topic, and based on that, did you decide to use a convergent, sequential, or complex mixed methods design with secondary data? In what ways has your study furthered our understanding of the previous, current, and future science of mixed methods? Based on the results from your mixed methods with secondary data, what might other researchers do to follow up on your findings? As you think about yourself as a mixed methods researcher, take your contributions to the field seriously and frame how your use of secondary data in mixed methods makes a novel contribution to the literature on mixed methods research overall. Some guiding questions you can use are: In what ways has your use of secondary data contributed to the science of mixed methods overall? Did you adopt a technique, tool, or strategy from a different field? Have you invented a design, process, or system never done before? These questions can be addressed in the discussion and conclusion sections or your mixed methods report. Remember: when reporting mixed methods with secondary data, you risk losing your audience if you incorporate all the data into your final reports. Conversely, by omitting too much, you risk excluding the study context that helps readers attach meaning to your conclusions. Only include the necessary data to achieve your study goals.

Chapter 9  •  Reporting Mixed Methods With Secondary Data  

Summary This chapter described the writing that occurs after your mixed methods with secondary data project has ended. After reading this chapter, you should be able to define what reporting is and what it looks like for mixed methods with secondary data. Unlike early-stage writing and active project writing, which help plan and track your mixed methods studies with secondary data, report writing is meant to help you organize and share your findings. I also use this chapter to identify the five kinds of reports that may come from your mixed methods with secondary data: project outcome reports, study protocol and procedure reports, theory application reports, study participant reports, recommendations and solutionoriented reports. I end the chapter with tips for maximizing your report writing and guiding questions to accompany each recommendation. By applying these tips and thinking about your responsibility to report mixed methods with secondary data, you further develop your topic and the science of mixed methods.

Chapter 9 Application Questions 1. This chapter identifies project deliverables for mixed methods projects with secondary data. When thinking about your research area and your mixed methods project, what are two to three potential deliverables from your project and the potential audiences for each? 2. Review the five examples of project deliverables reports in this chapter. Which do you see most frequently in your field or discipline? Which do you see most often for your topic under study? 3. Locate an example of a mixed methods report with secondary data in your field and critically review it. Is it a project outcome report, study protocol and procedure report, theory application report, participant report, or a recommendations and solutions report? Is it a different type of report altogether? 4. Using this chapter’s 10 mixed methods reporting tips, create an outline for a potential deliverable. Write out how you will address each tip for your mixed methods project with secondary data. 5. After reviewing the 10 tips for reporting in this chapter, which reporting practices are most closely aligned with your discipline or topic under study?

199

• Glossary • Active project writing: The writing that occurs while your project is underway involves the various ways you gather information, document, and track the ongoing procedures, protocols, and findings associated with your mixed methods with existing data. Administrative records: Documents related to organization functions (such as managing the facilities, finances, and personnel) and agreements, contracts, meetings, legal actions, and so on. Advanced analytics: See analytics. Advanced big data analytics: See analytics. Analytics: The systematic computational analysis of data or statistics. Analytics, advanced: The analytics used to explore algorithms for complex analysis and to uncover associations within the data (i.e., the analytics you might associate with “inferential statistics”). Analytics, advanced big data: Involves complex analysis of either structured or unstructured big data. May include sophisticated statistical models, machine learning, neural networks, text analytics, and other advanced data-mining techniques. Analytics, basic: The analytics used to explore data when you are unsure what you have but believe the data have something worth exploring (i.e., the analytics you might associate with “descriptive statistics”). Analytics, basic big data: Most often used when you have large amounts of disparate data. These data can be reconciled by running some basic analytic techniques for descriptive statistics, such as slicing and dicing, basic monitoring, and anomaly identification. Anomaly identification: The act of identifying anomalies or outliers in data, such as when the actual observation differs from what you expected. See analytics, basic big data. Audio recording: An electronic sound file used to capture activity or human experience. Basic analytics: See analytics. Basic big data analytics: See analytics.

201

202   Secondary Data in Mixed Methods Research Basic monitoring: Monitoring large volumes of data in real time. See analytics, basic big data. Big data: Existing data that exceed the processing capacity of conventional database systems. Often these data are too big, move too fast, or do not fit the traditional database storage and management systems structures. Big data mining: Exploring and analyzing big data to find patterns to classify and predict study outcomes. Classification: A component of big data mining that involves sorting the big data into groups. From an analysis point of view, classification is achieved by running basic statistical analysis (i.e., the equivalent of descriptive statistics). Cohort and other longitudinal surveys: A cohort study is a particular form of a longitudinal study (panel study) that sample a cohort (a group of people who share a defining characteristic, typically who experienced a common event in a selected period, such as birth or graduation), performing a cross-section at intervals through time. A cohort study is a panel study, but a panel study is not always a cohort study as individuals in a panel study do not always share a common characteristic. Complex (advanced) mixed methods design: An expansion of the core (basic) mixed methods design. Computer-generated data: A type of big data created using a computer (or a machine). An example of computer-generated data might be web login data, such as login information for an app on an iPhone or Android device. Concept: A unit of qualitative measure. Conceptual framework: With the knowledge-level continuum as a roadmap, a conceptual framework generates theory and strengthens your understanding of how concepts fit together. Also known as preliminary theory. Convergent design: A core mixed methods design where quantitative and qualitative data are collected and analyzed separately, often simultaneously; then the findings of the two data sources are interpreted collectively to generate conclusions. Core (basic) mixed methods designs: The foundation of complex (advanced) mixed methods designs. Data evaluation guide: Questions to guide the evaluation of existing qualitative and quantitative data sources. Data repository: A collection of previously collected numeric or text data sets available for secondary use. Often, data repositories are part of larger institutions established for research and archiving purposes to support the data needs of those institutions.

Glossary  203

Data transformation: One way of integrating results for a mixed methods design; changing one set of results (either qualitative or quantitative) into a format that allows for a smoother merging of one set of results with the other. This can look like qualitative data taking the form of numbers or quantitative data being transformed to allow the researcher to deeply analyze the context in which the language and scope of the quantitative measures were made. Deductive reasoning: Beginning with an established theory and working to confirm that theory more broadly. The reasoning behind quantitative research. Descriptive studies: Studies at the descriptive level of the knowledge-level continuum have a moderate level of preexisting knowledge. Descriptive studies can use both quantitative and qualitative methods of inquiry to deepen one’s understanding of a topic. Document (text): A written record of information. Early-stage writing: For your mixed methods with existing data, this writing stage includes the introductory information, budding ideas, and preliminary work plans you develop for your mixed methods study before the project begins. Epistemology: The study of the distinction between acceptable belief and opinion. The theory of knowledge, especially about its methods, validity, and scope. Existing data: Often called “secondary” data. They are not new and undergo secondary analysis to address a research question. Existing data are usually somehow related to the original, primary study goals and research questions. However, they can be used for secondary purposes. Explanatory sequential design: A core mixed methods design where quantitative data are collected and analyzed first, followed by collecting and analyzing qualitative data. Usually, the findings from the quantitative data are used to make decisions about the collection, analysis, and interpretation of the qualitative data. Explanatory studies: Studies at the explanatory level of the knowledge-level continuum have the highest level of preexisting knowledge. With research at the explanatory level, you can examine causation, whether relationships exist, and the strength of these relationships. Also, this level of research can produce inferential statistics that you may generalize to the larger population. Exploratory studies: Studies at the exploratory level of the knowledge-level continuum have the lowest level (i.e., limited amount/quality) of preexisting knowledge about the topic. Exploratory studies lie at the far end of the knowledge-level continuum. Exploratory sequential design: A core mixed methods design where qualitative data are collected and analyzed first, followed by collecting and analyzing quantitative data. Usually, the findings from the qualitative data are used to make decisions about the collection, analysis, and interpretation of the quantitative data; thus,

204   Secondary Data in Mixed Methods Research the qualitative results shape a quantitative component. An exploratory sequential design can be used for instrument development and assessing whether qualitative themes generalize to a population. Focus group (group interview): A discussion by three or more people led by a group moderator, or facilitator, on a specific topic. Human-generated data: A type of big data created by humans in collaboration with machines. An example of human-generated data is any piece of information you would enter into a computer database, such as your name, birthday, country of origin, and so on. Individual interview: One-on-one interviews between two people, one being the interviewer and the other being the interviewee. Inductive reasoning: Beginning with observations to note patterns and ending with theory. The reasoning behind qualitative research. Joint data displays: Visual representations such as diagrams, tables, and figures that illustrate the mixed methods with existing data and interpret the study concepts, findings, inferences, and the ability to communicate them to others. Knowledge-level continuum: A term used in social work and anthropology to describe the process for scientific inquiry and the amount of preexisting knowledge on a given topic. It encourages you to think about research as being a direct by-product of the preexisting information about a topic and the purpose of the intended study. Life cycle of research data: The schedule of events that occur throughout a project’s life cycle involves the creation, preservation, and usability of the research data. Measures of rigor: The steps taken to ensure the qualitative and quantitative procedures were conducted rigorously. Quantitative measures of rigor are validity, reliability, objectivity, and generalizability. Qualitative measures of rigor are credibility, dependability, confirmability, and transferability. Medical record: The systematic documentation of a single patient’s medical history and care across time within one healthcare provider’s jurisdiction. The terms medical record, health record, and medical chart are used interchangeably. Merging: A way of integrating findings in a single table, diagram, or figure in which the qualitative and quantitative results and the interpretation are presented. Metadata: Data about data. Includes the various definitions, mappings, and other characteristics used to describe how to find, access, and use big data components. Examples include title, description, funding source, data collectors, sample description, data sources, unit of analysis, variables, instruments, codebook, and so on. Metadata catalog: A list of the qualitative and quantitative study components, compiled before analyzing and integrating them.

Glossary  205

Methodological trajectory: The path that led you to choose a mixed methods design over a single-method design and prompted you to use existing data rather than collect new data. Mixed methods: The rigorous and epistemological application and integration of qualitative and quantitative research approaches to draw interpretations based on the combined strengths of both approaches to influence research, practice, and policy. Mixed methods with secondary data: Identifying, evaluating, and incorporating either one or more existing data sources into one or more components for a mixed methods project. It acknowledges the purpose of mixed methods, which is to collect, analyze, and integrate qualitative and quantitative data in rigorous and theoretically sound ways to encompass the breadth and depth of a phenomenon of interest Operationalize the terms: To attach meaning (or a definition) to a term so that it can be measured. Population-based existing quantitative data: Population-based studies are epidemiology studies in which a defined population is followed up and observed longitudinally to assess exposure and outcome relationships. Prediction: A component of big data mining that involves predicting the value, direction, or strength of a variable or concept captured from big data. From an analysis point of view, prediction is achieved by running more advanced analytics (i.e., the equivalent of inferential statistics). Primary data: New qualitative or quantitative data collected to address a fundamental research question. Primary data are also called “new” or “original” data for a study. Prioritize the qualitative phase: In sequential designs, prioritized phases take precedence over other study phases. If your mixed methods study prioritizes the qualitative phase, developing a conceptual framework is the goal. Prioritize the quantitative phase: In sequential designs, prioritized phases occur before other phases of the study. If your mixed methods study prioritizes the quantitative phase, the theory will guide your choice of the variables you plan to use. Project outcome report: A project deliverable that describes the results and overall outcomes for mixed methods with existing data. This may include the results for one or multiple project outcomes and at one time or over a period. Protocol and procedure report: A project deliverable that describes the p ­ rocedures— from conceptualization to study completion—for mixed methods with existing data. Includes study protocols (i.e., questionnaires, training, research conduct, human subjects, etc.). May consist of barriers and facilitators of the project. Recommendations and solution-oriented report: A project deliverable that summarizes recommendations (i.e., “take-aways”) using solution-targeted language

206   Secondary Data in Mixed Methods Research determined by the study team after a mixed methods study with existing data has been completed. Also called practice articles or “white papers.” Reporting: The process used to gather, integrate, interpret, and disseminate what you learn from your mixed methods research. Secondary data: See existing data. Secondary qualitative data: Previously collected text or image data. Also referred to as “secondary qualitative data” or “qualitative secondary data.” Secondary quantitative data: Previously collected numeric data. Also referred to as “secondary quantitative data” or “quantitative secondary data.” Semi-solo option: When conducting mixed methods with existing data, you proceed with limited advice, consultation, and assistance from one to two people. Side-by-side comparison: Qualitative and quantitative data presented in a table, diagram, or figure side-by-side. Showing one set of results alongside (or next to) the other results to see if the results complement one another. Single-method study: Unlike a mixed methods study, a single-method study employs one quantitative or qualitative research method. Slicing and dicing: Breaking down data into smaller sets of data that are easier to analyze. Graphs, plots help explore data across different dimensions. Basic statistics such as averages and medians can clue you into what you have. See analytics, basic big data. Solo option: When conducting mixed methods with existing data, you proceed with no advice, consultation, or assistance. Study participant report: A project deliverable that describes participant demographics and how researchers engage with the study participants/ respondents (e.g., sampling, recruitment, retention, etc.) for mixed methods with existing data. Team option: An option when conducting mixed methods with existing data during which you proceed with advice, consultation, and assistance from three or more people. Theory: A product of qualitative inquiry and for testing during the quantitative inquiry developed using conceptual frameworks and deepened understanding. Theory application report: This project deliverable describes how theories or conceptual frameworks were generated before, applied during, or tested after mixed methods with existing data. Variables: Units of quantitative measure. Video recording: An electronic video file that is used to capture the human experience.

• References • Andrews, L., Higgins, A., Andrews, M. W., & Lalor, J. G. (2012). Classic grounded theory to analyses secondary data: Reality and reflections. The Grounded Theory Review, 11(1), 12–26. Arends, I., Bultmann, U., Shaw, W. S., van Rhenen, W., Roelen, C., Nielsen, K., & van der Klink, J. J. (2014). How to engage occupational physicians in recruitment of research participants: A mixed methods study of challenges and opportunities. Journal of Occupational Rehabilitation, 24(1): 68–78. doi: 10.1007/s10926-013-9452-y Badke, W. (2017). Research Strategies: Finding Your Way Through the Information Fog. Bloomington: iUniverse. Basken, P. (2014). NIH awards $32 million to tackle big data in medicine. Chronicle of Higher Education. Retrieved from http://chronicle.com/article/NIH-Awards-32Million-to/149323/? cid=wc&utm_source=wc&utm_medium=en Bazeley, P. (2015). Writing up Multimethod and Mixed Methods Research for Diverse Audiences. In: Hesse-Biber, SN and Johnson, RB eds, The Oxford Handbook of Multimethod and Mixed Methods Research Inquiry. New York, NY: Oxford University Press, 296–313. Bazeley, P. (2018). Integrating analyses in mixed methods research. Thousand Oaks, CA: Sage Publications. Beck, C. T. (2019). Secondary qualitative data analysis in the health and social sciences. London, UK: Routledge. Bishop, E. (2009). Ethical sharing and reuse of qualitative data. Australian Journal of Social Issues, 44(3): 255–72. Bishop, L. (2007). A reflexive account of reusing qualitative data: Beyond primary/ secondary dualism, Sociological Research Online, 12(3) Retrieved from http://www .socresonline.org.uk/12/3/2.html Brown, B., Chui, M., & Manyika, J. (2011). Are you ready for the era of ‘big data?’ The McKinsey Quarterly, 4, 24–35. Cabrera, N. L. (2011). Using a sequential exploratory mixed methods design to examine racial hyperprivilege in higher education. New Directions for Institutional Research, 151, 77–91. doi: 10.1002/ir.400

207

208   Secondary Data in Mixed Methods Research Carrington, J. M., Gephart, S. M., Verran, J. A., & Finley, B. A. (2015). Development of an instrument to measure the unintended consequences of EHRs. Western Journal of Nursing Research, 37(7), 842–858. doi: 10.1177/0193945915576083 Cameron, R. (2011). Mixed methods research: The five Ps framework. The Electronic Journal of Business Research Methods, 9(2), 96–108. Available online at www.ejbrm.com Clark, T., Foster, L., Sloan, L., & Bryman, A. (2021). Bryman’s Social Research Methods. Sixth Edition. Oxford, United Kingdom: Oxford University Press. Coolbrandt, A., Dierckx de Casterlé, B., Wildiers, H., Aertgeerts, B., Van der Elst, E., van Achterberg, T., & Milisen, K. (2016). Dealing with chemotherapy-related symptoms at home: A qualitative study in adult patients with cancer. European Journal of Cancer Care, 25(1), 79–92. doi: 10.1111/ecc.12303 Coolbrandt, A., Steffens, E., Wildiers, H., Bruyninckx, E., Verslype, C., & Milisen, K. (2017). Use of a symptom diary during chemotherapy: A mixed methods evaluation of a patient perspective. European Journal of Oncology Nursing, 31, 37–45. doi: 10.1016/j.ejon.2017.09.003 Corti, L., & Backhouse, G. (2005). Acquiring qualitative data for secondary analysis. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(2), Art. 36, doi: 10.17169/fqs-6.2.459. Corti, L. (2012). Recent development in archiving qualitative research data. International Journal of Social Research Methodology, 15, 281–290. Corti, L. Van den Eynden, V., Bishop, L., & Woollard, M. (2014). Managing and ­sharing research data: A guide to good practice. Los Angeles, CA: Sage Publications. Cracium, C., Gellert, P., & Flick, U., (2017). Aging in precarious circumstances: Do positive views on aging make a difference? The Gerontologist, 57(3): 517–528. Craig, A. D., Steinauer, J., Kuppermann, M., Schmittdiel, J. A., & Dehlendorf, C. (2019). Pill, patch or ring? A mixed methods analysis of provider counseling about combined hormonal contraception. Contraception, 99(2), 2014–110. doi: 10.1016/j. contraception.2018.09.001 Creswell, J. W. (1999). Mixed-method research: Introduction and application. In G. J. Cizek (Ed.), Handbook of educational policy (pp. 455–472). San Diego, CA: Academic Press. Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches. Los Angeles: Sage. Creswell, J. W. (2009). Mapping the field of mixed methods research [Editorial]. Journal of Mixed Methods Research, 3(2), 95–108. Creswell, J. W. (2015). A concise introduction to mixed methods research. Thousand Oaks, CA: Sage.

References  209

Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, Quantitative, and Mixed Methods Approaches. 5th Edition. Thousand Oaks, CA: Sage Publications. Creswell, J. W., Fetters, M. D., & Ivankova, N. V. (2004). Designing a mixed methods study in primary care. The Annals of Family Medicine, 2(1), 7–12. doi: 10.1370/ afm.104 Creswell, J. W. Goodchild, L. F., & Turner, P. (1996). Integrated qualitative and quantitative research: Epistemology, history, and designs. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 11, pp. 90–136). New York, NY: Agathon Press. Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Thousand Oaks, CA: Sage Publications. Creswell, J. W., & Zhang (2009). The application of mixed methods designs to trauma research. Journal of Traumatic Stress, 22(6), 612–621. doi: 10.1002/jts.20479 Creswell, J. W., Plano Clark, V. L., Gutmann, M. L., & Hanson, W. E. (2003). Advanced mixed methods research designs. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 209–240). Thousand Oaks, CA: Sage Publications. Creswell, J. W., Klassen, A. C., Plano Clark, V. L., & Clegg Smith, K. (2011). Best practices for mixed methods research in the health sciences. Washington, DC: Office of Behavioral and Social Sciences Research. Creswell, J. W., & Tashakkori, A. (2007). Developing publishable mixed methods manuscripts [Editorial]. Journal of Mixed Methods Research, 1(2), 107–111. doi:10.1177/155868980 6298644 Curry, L., & Nunez-Smith, M. (2015). Mixed methods in health sciences research: A practical primer (Mixed Methods Research Series 1). Thousand Oaks, CA: Sage. Dale, A., Arbor, S., & Proctor, M. (1988). Doing secondary analysis (Contemporary Social Research Series No. 17). London: Unwin Hyman Ltd. de Block, D., & Vis, B. (2018). Addressing the challenges related to transforming qualitative into quantitative data in qualitative comparative analysis. Journal of Mixed Methods Research, 13(4), 503–535. doi: 10.1177/1558689818770061 DeCuir-Gunby, J. T., & Schutz, P. A. (2017). Developing a mixed methods proposal: A practical guide for beginning researchers (Vol. 5). Sage Publications. Dehlendorf, C., Tharayil, M., Anderson, N., Gbenedio, K., Wittman, A., & Steinauer, J. (2014). Counseling about IUDs: A mixed methods analysis. Perspectives on Sexual and Reproductive Health, 46(3), 133–140. doi: 10.1363/46e0814 Erzberger, C. & Kelle, U. (2003). Making inferences in mixed methods: The rules of integration. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 457–488). Thousand Oaks, CA: Sage.

210   Secondary Data in Mixed Methods Research Doi, L., Morrison, K., Astbury, R., et al. (2020). Study protocol: A mixed methods realist evaluation of the Universal Health Visiting Pathway in Scotland, BMJ Open, 10: e042305 doi: 10.1136/ bmjopen-2020-042305 Doolan, D. M., & Froelicher, E. S. (2009). Using an existing data set to answer new research questions: A methodological review. Research and Theory for Nursing Practice: An International Journal, 23(3), 203–215. doi: 10.1891/1541-6577. 23.3.203 Dumbill, E. (2013). Making sense of big data. 1(1): 1–2. Accessed on September 13, 2017 from doi: 10.1089/big.2012.1503 Erzberger, C., Kelle, U., 2003. Making inferences in mixed methods: The rules of integration. In: Tashakkori, A., Teddlie, C. (Eds.), Handbook of Mixed Methods in Social & Behavioural Research. Sage, Thousand Oaks, pp. 457–488. Fetters, M. D. (2020). The mixed methods research workbook. Los Angeles, CA: Sage. Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designs—Principles and practices. Health Services Research, 48(6), 2134–2156. Fetters, M. D., & Freshwater, D. (2015). The 1 + 1 = 3 Integration Challenge. Journal of Mixed Methods Research, 9(2), 115–117. Fetters, M. D., & Freshwater, D. (2015). Publishing a methodological mixed methods research article. Journal of Mixed Methods Research, 9(3), 203–213. doi: 10.1177/1558689815594687 Fielding, N. G., and Fielding, J. L. (2000). Resistance and adaptation to criminal identity: Using secondary analysis to evaluate classic studies of crime and deviance, Sociology, 34(4), 671–689. Garrard, J. (2016). Health science literature reviews made easy. Jones & Bartlett Learning, 5th Edition. Galvan, J. L. & Galvan, M. C. (2017). Writing Literature Reviews: A Guide for Students of the Social and Behavioral Sciences (3rd Ed.). Glendale, CA: Pyrczak Publishing. Gephart, S. M., Bristol, A. A., Dye, J. L., Finley, B. A., & Carrington, J. M. (2016). Validity and reliability of a new measure of nursing experience with unintended consequences of electronic health records. Computers, Informatics, and Nursing, 34(10), 436–447. doi: 10.1097/CIN.0000000000000285 Glaser, B. G. (1963). Retreading research materials: The use of secondary analysis by the independent researcher. American Behavioral Scientist, 6(10), 11–14. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3–8.

References  211

Gradinger, F., Elston, J., Asthana, S., Martin, S., & Byng, R. (2019). Reflections on the Researcher-inResidence model co-producing knowledge for action in an integrated care organisation: A mixed methods case study using an impact survey and field notes, Evidence and Policy: A Journal of Research, Debate and Practice, 15, 197–215. Gray, J. & Geraghty, R. (2020). Using quantitative data in qualitative secondary analysis. In K. Hughes, and A. Tarrant (eds.) Qualitative Secondary Analysis. 1st ed, 3–18. London, UK: Sage Publications. Grbich, C. (2012). Qualitative Data Analysis: An Introduction. Thousand Oaks, CA: Sage. Greene, J. C. (2007). Mixed methods in social inquiry (Vol. 9). San Francisco, CA: Jossey-Bass, John Wiley & Sons. Greene, J. C. & Caracelli, V. J. (1997). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. San Francisco, CA: Jossey-Bass. Grinnell, R. M., Jr. & Unrau, Y. A. (2018). Social work research and evaluation: Foundations of evidence-based practice (11th ed.). New York, NY: Oxford University Press. Guetterman, T. C., Fetters, M. D., & Creswell, J. W. (2015). Integrating quantitative and qualitative results in health science mixed methods research through joint displays. Annals of Family Medicine, 13, 554–561. Guba, E. G. & Y. S. Lincoln (1989). Fourth Generation Evaluation. Newbury Park, CA: Sage. Haight, W. & Bidwell, L. N. (2016). Mixed Methods Research for Social Work Integrating Methodologies to Strengthen Practice and Policy. Oxford, United Kingdom: Oxford University Press. Hakim, C. (1982). Secondary analysis in social research: A guide to data sources and method examples. London, UK: George Allen & Uwin. Hammersley, M (1997). Research note. Qualitative data archiving: some reflections on its prospects and problems, Sociology, 31(1), 131–42. Hammersley, M. (2008). Troubles with triangulation. In M. M. Bergman (Ed.), Advances in mixed methods research (pp. 22–36). Thousand Oaks, CA: Sage. Hammersley, M. (2009). Can we re-use qualitative data via secondary analysis? Notes on some terminological and substantive issues, Sociological Research Online, 15(1), at http://www.socresonline.org.uk/15/1/5.html Heafner, T. L., Fitchett, P. G., & Knowles, R. T. (2016). Using big data, large-scale studies, secondary datasets, and secondary data analysis as tools to inform social studies teaching and learning. In A. R. Crowe and A. Cuena (Eds.). Rethinking social studies teacher education in the twenty-first century. Springer Publishers.

212   Secondary Data in Mixed Methods Research Heaton, J. (1998). Secondary analysis of qualitative data. Social Research Update, Guildford: Surrey University ISR. Heaton, J. (2008). Secondary analysis of qualitative data: An overview. Historical Social Research, 33(3), 33–45. Hesse-Biber, S. N. (2010). Mixed methods research: Merging theory with practice. New York, NY: Guilford. Hewson, C. (2006). Secondary analysis. In Jupp, V. (ed.). The Sage Dictionary of Research Methods, London: Sage Publications. Heyvaert, M., Hannes, K., & Onghena, P. (2017). Using mixed methods research synthesis for literature reviews. Thousand Oaks, CA: Sage. Hinds, P., Vogel, R., & Clarke-Steffen, L. (1997). The possibilities and pitfalls of doing secondary analysis of a qualitative data set, Qualitative Health Research, 7(3): 408–424 Hughes, K. & Tarrant, A. (2020). An introduction to qualitative secondary analysis. In Hughes, K and Tarrant, A. (eds.) Qualitative Secondary Analysis. 1st ed, 3–18. London, UK: Sage Publications. Hurwitz, J., Nugent, A., Halper, F. & Kaufman, M. (2013). Big Data for Dummies. Hoboken, NJ: John Wiley & Sons. Hyman, H. H. (1972). Secondary analysis of sample surveys. Principles, procedures, and potentialities. New York, NY: Wiley. Irwin, S. & Winterton, M. (2011). Debates in qualitative secondary analysis: Critical reflections. Timescapes Working Paper Series No. 4: An ESRC qualitative longitudinal study. University of Leeds. Retrieved from http://www.timescapes.leeds.ac.uk/ assets/files/WP4-March-2011.pdf Jimenez, M. E., Fiks, A. G., Ramirez Shah, L., Gerdes, M., Ni, A. Y., Pati, S., & Guevara, J. P. (2014). Factors associated with early intervention referral and evaluation: A mixed methods analysis. Academic Pediatrics, 14(3), 315–323. doi: 10.1016/j. acap.2014.01.007 Johnson, R. B. & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational researcher, 33(7), 14–26. doi: 10.3102/0013189X033007014 Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1, 112–133. doi:10.1177/1558689806298224 Johnson, R. E., Grove, A. L., & Clarke, A. (2017). Pillar integration process: A joint display technique to integrate data in mixed methods research. Journal of Mixed Methods Research, 13(3), 301–320. doi: 10.1177/1558689817743108

References  213

Johnston, M. P. (2012). School librarians as technology integration leaders: Enablers and barriers to leadership enactment. School Library Research, 15(1). Johnston, M. P. (2014). Secondary method: A method of which the time has come. Qualitative and Quantitative Methods in Libraries, 3(3), 619–626. Kang, B., Pan, W., Karel, M. J., & McConnell, E. S. (2021). Rejection of care and aggression among older veterans with dementia: The influence of background factors and interpersonal triggers, Journal of the American Medical Directors Association. doi: 10.1016/ j.jamda.2021.03.032 Kawamura, Y., Ivankova, N., Kohler, C., & Perumean-Chaney, S. (2009). Utilizing mixed methods to assess parasocial interaction of one entertainment-education program audience. International Journal of Multiple Research Approaches, 3(1), 88–104. doi: 10.5172/mra.455.3.1.88 Kiecolt, K. J. & Nathan, L. E. (1985). Secondary analysis of survey data. Sage University Paper Series on Quantitative Applications in the Social Sciences, 53. Kitchin, R. (2014). The data revolution: Big data, open data, data infrastructures and their consequences. Thousand Oaks, CA: Sage Publications. Largan, C. & Morris, T. (2019). Qualitative secondary research: A step-by-step guide. London, UK: Sage Publications. Lauzon, D. (2012). Introduction to big data. Accessed on August 11, 2017. http://­ publicationslist.org/data/a.april/ref-389/introduction_to_bigdata.pdf Leech, N. L., Onwuegbuzie, A. J., & Combs, J. P. (2011). Writing Publishable Mixed Research Articles: Guidelines for Emerging Scholars in the Health Sciences and Beyond. Mixed Methods Research in the Health Sciences, 5(1):7–24. Lindsey, R. & Bulloch, S. (2014). A sociologist’s field notes to the mass observation archive: A consideration of the challenges of ‘re-using’ mass observation data in a longitudinal mixed methods study. Sociological Research Online, 19(3), 8. Lipson, S. K. & Eisenberg, D. (2018). Mental health and academic attitudes and expectations in university populations: Results from the healthy minds study. Journal of Mental Health, 27(3), 205–213. doi: 10.1080/09638237.2017.1417567 Logan, T. (2020). A practical, iterative framework for secondary data analysis in educational research. Australian Educational Researcher, 47(3), 129–148. Machi, L. A. & McEvoy, B. T. (2012). The literature review: Six steps to success (2nd ed.). Thousand Oaks, CA: Corwin Press. MacLean, L. M., Meyer, M., & Estable, A. (2004). Improving accuracy of transcripts in qualitative research. Qualitative Health Research, 14(1), 113–123. doi: 10.1177/1049732303259804

214   Secondary Data in Mixed Methods Research Magee, T., Lee, S. M., Giuliano, K. K., & Munro, B. (2006). Generating new knowledge from existing data: The use of large data sets for nursing research. Nursing Research, 55(2), S50–S56. Mayer-Schonberger, V. & Cukier, K., 2013. Big Data: A Revolution that will Transform how we Live, Work and Think. London: John Murray. Maxwell, J. A. (2016). Expanding the history and range of mixed methods research. Journal of Mixed Methods Research, 10(1) 12–27. Miles, M. B., Huberman, A. M., & Saldana, J. (2013). Qualitative data analysis: A methods sourcebook. Thousand Oaks, CA: Sage. Mills, M. J., Culbertson, S. S., Huffman, A. H., & Connell, A. R. (2012). Assessing gender biases: Development and initial validation of the gender role stereotypes scale. Gender in Management: An International Journal, 27(8), 520–540. doi: 10.1108/17542411211279715 Minnis, A. M., Mavedzenge, S. N., Luecke, E., & Dehlendorf, C. (2014). Provider counseling to young women seeking family planning services. Perspectives on Sexual Reproductive Health, 46(4), 223–231. doi: 10.1363/46e1414 Morse, J. M. (1991). Approaches to qualitative-quantitative methodological triangulation. Nursing Research, 40(1), 120–123. Morse, J. M. & Niehaus, L. (2009). Mixed method design: Principles and procedures. Walnut Creek, CA: Left Coast Press. Nastasi, B. K. & Hitcock, J. H. (2016). Mixed methods research and culture-specific intervention: Program design and evaluation. Thousand Oaks, CA: Sage. O’Cathain, A. (2010). Assessing the quality of mixed methods research: Toward a comprehensive framework. In A. Tashakkori & C. Teddlie (Eds.), SAGE handbook of mixed methods in social & behavioral research (2nd ed., pp. 531–558). Thousand Oaks, CA: Sage Publications. Onwuegbuzie, A. J. & Combs, J. P. (2010). Emergent data analysis techniques in mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), The SAGE handbook of mixed methods in social & behavioral research (2nd ed., pp. 397–430). Thousand Oaks, CA: Sage Publications. Onwuegbuzie, A. J. & Combs, J. P. (2011). Data analysis in mixed research: A primer. International Journal of Education, 3(1), 1–25. doi: 10.5296/ije.v3i1.618 Onwuegbuzie, A. J. & Frels, R. (2016). Seven steps to a comprehensive literature review. London: Sage. Onwuegbuzie, A. J., Leech, N. L., & Whitcome, J. A. (2008). A framework for making quantitative educational research articles more reader-friendly for practitioners. Quality & Quantity, 42(1), 75–87. doi: 10.1007/s11135-006-9037-3

References  215

Ott, R. L. & Longnecker, M. T. (2015). An introduction to statistical methods and data analysis. Scarborough ON, Canada: Nelson Education. Panchenko, L. & Samovilova, N. (2020). Secondary data analysis in educational research: Opportunities for PhD students. SHS Web of Conferences, 75, pp. 1–7. Pienta, A. M., O’Rourke, J. M., & Franks, M. M. (2011). Getting started: Working with secondary data. In K. H. Trzesniewski, M. B. Donnellan, & R. E. Lucas (Eds.), Secondary data analysis: An introduction for psychologists (pp. 13–25). Washington, DC: American Psychological Association. Plano Clark, V. L. & Ivankova, N. V. (2016). Mixed methods research: A guide to the field. Los Angeles, CA: Sage. Pluye, P., Grad, R. M., Levine, A., & Nicolau, B. (2009). Understanding divergence of quantitative and qualitative data (or results) in mixed methods studies. International Journal of Multiple Research Approaches, 3(1), 58–72. doi: 10.5172/mra.455.3.1.58 Polit, D. & Beck, C. (2016). Nursing research: Generating and assessing evidence for nursing practice (10th ed.). Philadelphia: Wolters Kluwer. Prada-Ramallal, G., Takkouche, B., & Figueiras, A. (2017). Diverging conclusions from the same meta-analysis in drug safety: Source of data (primary versus secondary) takes a toll. Drug Safety, 40(4): 351–358. doi: 10.1007/s40264-016-0492-z Rankin-Williams, A. C., Geoffroy, E. M., Schell, E. S., & Mguntha, A. M. (2017). How can male rates of HIV testing be increased? Recommendations from a mixed methods study in southern Malawi, International Health, 9(6), 367–373. doi: 10. 1093/inthealth/ihx042 Rocco, T. & Hatcher, T, (2011). The handbook of scholarly writing and publishing. San Francisco, CA: Jossey-Bass. Rosenthal, J. A. (2012). Statistics and data interpretation for social work. New York, NY: Springer Publishing Company. Ross, A. & Onwuegbuzie, A. J. (2015). Complexity of quantitative analyses used in mixed research articles published in a flagship mathematics education journal. International Journal of Multiple Research Approaches, 8(1), 63–73. doi: 10.5172/mra.2014.8.1.63 Rudestam, K. E. & Newton, R. R. (2015). Surviving your dissertation: A comprehensive guide to content and process (4th ed.). Thousand Oaks, CA: Sage. Sale, J. E., Lohfeld, L. H., & Brazil, K. (2002). Revisiting the quantitative-­qualitative debate: Implications for mixed methods research. Quality and Quantity, 36(1), 43–53. doi: 10.1023/A:1014301607592 Salmons, J. (2016). Doing Qualitative Research Online. Los Angeles, CA: Sage. Schutt, R. K. (2007). The Blackwell Encyclopedia of Sociology, Volume III, Ritzer, G. (Ed.), Oxford: Blackwell.

216   Secondary Data in Mixed Methods Research Shannon, L. M., Hulbig, S. K., Birdwhistell, S., Newell, J., & Neal, C. (2015). Implementation of an enhanced probation program: Evaluating process and ­preliminary outcomes. Evaluation and Program Planning, 49, 50–62. doi: 10.1016/j .evalprogplan.2014.11.004 Shemmings, D. (2008). ‘Quantifying’ qualitative data: An illustrative example of the use of Q methodology in psychosocial research. Qualitative Research in Psychology, 3(2), 147–165. doi: 10.1191/1478088706qp060oa Sherif, V. (2018). Evaluating Preexisting Qualitative Research Data for Secondary Analysis [37 paragraphs]. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 19(2), Art. 7. doi: 10.17169/fqs-19.2.2821 Shrout, P. E. & Napier, J. L. (2011). Analyzing survey data with complex sampling designs. In K. H. Trzesniewski, M. B. Donnellan, & R. E. Lucas (Eds.), Secondary data analysis: An introduction for psychologists (pp. 63–81). American Psychological Association. doi: 10.1037/12350-004 Silverman, D. (2011). Interpreting qualitative data: A guide to the principles of qualitative research. London: Sage Publications. Sligo, J. L., Nairm, K. M., & McGee, R. O. (2018). Rethinking integration in mixed methods research using data from different eras: Lessons from a project about teenage vocational behavior. International Journal of Social Research Methodology, 21(1): 63–75. Smith, A. K., Ayanian, J. Z., Covinsky, K. E., Landon, B. E., McCarthy, E. P., Wee, C. C., & Steinman, M. A. (2011). Conducting high-value secondary dataset analysis: An introductory guide and resources. Journal of General Internal Medicine, 28(8), 920–929. doi: 10.1007/s11606-010-1621-5 Smith, E. (2008). Using secondary data in educational and social research. New York, NY: McGraw-Hill Education. Smith, E. (2011). Special issue on using secondary data in educational research. International Journal of Research and Method in Education, 34(3), 219–221. Sobal, J. (1981). Teaching with secondary data. Teaching Sociology, 8(2), 149–170. Srnka, K. J. & Koeszegi, S. T. (2007). From words to numbers: How to transform qualitative data into meaningful quantitative results. Schmalenbach Business Review, 59(1), 29–57. doi: 10.1007/BF03396741 Stewart, D. W. & Kamins, M. A. (1993). Secondary research: Information sources and methods. Newbury Park, CA: Sage Publications. Sturtevant, D. & Wimmer, J. S. (2014). Success and challenges of measuring program impacts: An international study of an infant nutrition program for AIDS orphans. Evaluation and Program Planning, 42, 50–56. doi: 10.1016/j.evalprogplan.2013.09.004 Tashakkori, A. & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: Sage Publications.

References  217

Tashakkori, A. & Teddlie, C. (Eds.) (2003). Handbook of mixed methods in social & behavioral research. Thousand Oaks, CA: Sage. Tashakkori, A. & Teddlie, C. (2008). Introduction to mixed method and mixed model studies in the social and behavioral sciences. In V. L. Plano Clark & J. W. Creswell (Eds.), The mixed methods reader (pp. 7–26). Portland, OR: Sage. Tashakkori, A. & Teddlie, C. (Eds.) (2010). SAGE handbook of mixed methods in social & behavioral research (2nd ed., pp. 581–611). Thousand Oaks, CA: Sage. Teddlie, C. & Tashakkori, A. (2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks, CA: Sage Publications. Teddlie, C. & Tashakkori, A. (2010). Overview of contemporary issues in mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed., pp. 1–41). Thousand Oaks, CA: Sage. Terrell, S. R. (2016). Writing a proposal for your dissertation. New York, NY: The Guilford Press. Thyer, B. A. (2008). Preparing research articles. Pocket Guides to Social Work Research Series. Oxford, United Kingdom: Oxford University Press. Tolley, E. E., Ulin, P. R., Mack, N., Robinson, E. T., & Succop, S. M. (2016). Qualitative methods in public health: A field guide for applied research (2nd ed.). San Francisco, CA: Jossey-Bass. Treviño, E., Scheele, J., & Flores, S. M. (2014). Beyond the test score: A mixed methods analysis of a college access intervention in Chile. Journal of Mixed Methods Research. 8(3): 255–265. Trinh, Q. D. (2018). Understanding the impact and challenges of secondary data analysis. Urologic Oncology, 36(4): 163–164. doi: 10.1016/j.urolonc.2017.11.003. Trotter II, R. T., Camplain, R., Eaves, E. R., Fofanov, V. Y., Dmitrieva, N. O., Hepp, C. M., Warren, M., Barrios, B. A., Pagel, N., Mayer, A., & Baldwin, J. A. (2018). Health disparities and converging epidemics in jail populations: Protocol for a mixed methods study. JMIR Research Protocols, 7(10): e10337. Ulin, P. R., Robinson, E. T., & Tolley, E. E. (2005). Qualitative methods in public health: A field guide for applied research. San Francisco, CA: Jossey-Bass. Vassallo, P. (1999). The knowledge continuum - organizing for research and scholarly communication. Internet Research, 9(3), 232–242. Verhoeven, N. (2011). Doing research: The hows and whys of applied research. Lyceum Books. Watkins, D. C. (2006). The depressive symptomatology of black college men: Preliminary findings. Californian Journal of Health Promotion, 4(3), 187–197.

218   Secondary Data in Mixed Methods Research Watkins, D. C. (2012). Qualitative research: The importance of conducting research that doesn’t ‘count.’ Health Promotion Practice, 13(2), 153–158. Watkins, D. C. (2017a). Mixed methods research in social work. Encyclopedia of Social Work Online. New York, NY: Oxford University Press. Watkins, D. C. (2017b). Rapid and rigorous qualitative data analysis: The RADaR technique for applied research. International Journal of Qualitative Methods, 16(1): 1–9. doi: 10.1177/1609406917712131 Watkins, D. C. & Gioia, D. (2015). Pocket guides to social work research methods series: Mixed methods research. New York, NY: Oxford University Press. Watkins, D. C., Green, B. L., Goodson, P., Guidry, J., & Stanley, C. A. (2007). Using focus groups to explore the stressful life events of black college men. Journal of College Student Development, 48(1), 105–118. Watkins, D. C. & Neighbors, H. W. (2007). An initial exploration of what ‘mental health’ means to young black men. Journal of Men’s Health and Gender, 4(3), 271–282. Watkins, D. C., Wharton, T., Mitchell, J. A., Matusko, N., & Kales, H. (2017). Perceptions and receptivity of non-spousal family support: A mixed methods study of psychological distress among older, church-going African American men. Journal of Mixed Methods Research, 11(4): 487–509. doi: 10.1177/1558689815622707 Wang, C. & Burris, M. A. (1997). Photovoice: Concept, methodology, and use for participatory needs assessment. Health Education & Behavior, 24(3), 369–387. doi: 10.1177/109019819702400309 Warner, R. M. (2008). Applied statistics: From bivariate through multivariate techniques. Thousand Oaks, CA: Sage Publications. Weaver-Hightower, M. B. (2014). A mixed methods approach for identifying influence on public policy. Journal of Mixed Methods Research, 8(2): 115–138. White, P. & Smith, E. (2005). What can PISA tell us about teacher shortages? European Journal of Education, 40(1), 93–112. Wolcott, H. F. (2008). Writing up qualitative research. Thousand Oaks, CA: Sage Publications.

• Index • active project, xx, 179, 181, 183, 186, 195, 199 active project stages, 164 Active Writing, 164–83, 201 address, xv–xx, 4–5, 14–16, 18–19,

analysis steps, 65, 97 analytics, 67, 126, 201–2, 206 Analyze, 60, 62, 76, 81, 84, 91, 94–95, 118, 120–21, 130, 135, 138, 147, 151, 157

39–44, 51–52, 63–65, 67–68, 82,

Analyze Data Online, 5

84–86, 93–98, 111–13, 139–40,

analyzing big data, 202

147, 150–51, 193, 195–96

Analyzing Data, 66, 95–96, 122

adolescents, 30, 122 adulthood, 156–57 advanced analytics, 67, 95, 201, 205 advanced big data analytics, 126, 201 advanced big data sources, 141 African American, 15, 71, 110 churchgoing, 108–9

analyzing secondary quantitative data, 14, 175 approaches, xiii–xiv, 5, 23, 66, 140, 145, 170, 190, 205 parallel data analysis, 80 Australia, 152–53 available data sources, xvi, 4, 37, 195

older, 108–9 aging, 47, 136–39, 157 aims, 15, 22, 45, 49, 73, 151, 168

barriers, 102, 130, 134, 149–51, 155, 188, 206

alcohol consumption, 5, 8

basic analytics, 67, 201

alignment, 8–9, 63–64, 93–94, 119–20,

basic big data, 201–2, 206

124–25, 192, 194 analysis, 6, 9–10, 26, 38–39, 46, 51–52, 55, 63, 65–66, 79–80, 86, 96–98, 123–27, 130–31, 137–41, 154, 166–67, 172–73, 175–76, 203–5 mixed methods, 83, 129, 133

basic big data analytics, 123, 201 benefits of big data, 92 big data, xix–xx, 17–19, 63, 67, 92, 95, 123, 125, 128, 202, 204–5 identifying secondary, 92 unstructured, 126, 201

analysis of existing data, 6

big data analytics, 67, 123, 141

analysis of quantitative and

Big Data Break, 63, 67, 92, 95–96, 123,

qualitative data, 66 analysis of quantitative and

125–26, 141 big data mining, 95, 202, 205

qualitative data for single-

big data source, 63, 123

method studies, 66

big data tables, 196

analysis plan, 63, 65, 79, 94–95, 120–21, 124, 194 statistical, 120, 175

Black, Indigenous, and People of Color (BIPOC), 123 Bristol, 100–102

219

220   Secondary Data in Mixed Methods Research Cabrera, 104–6, 110 Carrington, 100–102

convergent design, 59–69, 71–87, 93, 96, 113, 116, 141, 149, 191, 195, 202

categorizing data, 96

Coolbrandt, 79–81

CCD (Common Core of Data), 48

core designs, 127, 143–59

census data, 4, 10, 191

Core Mixed Methods Designs and

CHC (combined hormonal contraceptive), 83–84 CHC methods, 83–84 codebooks, 31, 38, 73, 172, 175–77, 205 cohort, 10–11, 202 collecting new data, 22, 93–94, 100, 112, 119, 124, 136, 170, 172–73, 196

Definitions, 26 Craig, 82–84 Creswell, xvi, xx, 23, 27, 31, 60, 66, 68, 90–91, 145, 167, 169 Creswell & Plano Clark, xx, 25–27, 42, 60–61, 66, 68, 85, 90–91, 144–45, 192, 198

colleges, 104–6, 129–31, 133–35 white, 105 college students, 33, 104, 163 white male, 105–6 Common Core of Data (CCD), 48 communities, xviii, xxi, 22, 40, 128, 172–73, 182

DACA (Deferred Action for Childhood Arrivals), 33, 163 146–48, 150–53, 155–57, 173–76, 190–91, 201–2 data analysis, 61, 65, 85–86, 92, 96, 149, 152, 168, 172–73, 175, 180

community center, 10–11

convergent study’s, 65

community college, 11–12

quantitative, 122

community of research practice, 22–23

secondary qualitative, xvi

complex analysis, 67, 126, 201 complex designs, 144–49, 152, 158–60 complex mixed methods designs, xx, 144–46, 149, 152, 158–59, 198 computer database, 92, 204

study’s, 60 data analysis and data integration steps, 65 data analysis and integration, 61, 65, 85–86, 92, 96, 122–23

computer-generated data, 92, 202

data analysis and integration plan, 173

concepts, 25, 29, 42, 46, 52, 54, 68–69,

data analysis and integration steps, 96

73, 95–97, 124–25, 193–95, 202,

data analysis and interpretation stage, 61

204–5

data analysis and validity, 152

conceptual frameworks, xvii, 25, 27, 29, 32, 108, 188, 196, 202, 205–6 Conducting Mixed Methods, 60–160 consultation, 34, 52, 178, 206 content, 39, 83–84, 181 content analysis, 67–68, 104

data analysis procedures, xvi, 193 database systems, 17, 202 data collection, 6, 10, 16, 19, 47, 49, 77, 80, 168, 174–75, 180, 185 convergent design’s, 65 primary, xix, 191

contraception, 46, 83–84

data collection and data analysis, 65

Contribution, 18–19, 75, 80, 101, 105,

data collection efforts, 52, 98

109, 129, 134, 137, 150, 153, 156, 192, 195–96, 198 contributions to mixed methods, 19, 22, 36 convergence, 68–69, 72, 130, 135

data collection methods, 50, 179 streamlined, 16 data collection procedures, 51, 123 data collection process, 51 data collection steps, 93

Index  221

data collection tools, 49, 78–79, 100

design, xvi–xviii, 61, 65, 85–86,

new qualitative, 73

89–90, 92–94, 104, 119–20, 124–

data collectors, 38, 205

26, 131–33, 144–46, 148, 152–53,

data components, 65–66, 73, 109, 174 big, 63, 204

164–66, 168, 192–93, 197–98 single-method, 187, 205

data evaluation guide, 44, 202

design and secondary data, 146

data features, 173

Designing, 57, 60–160

data files, xvi, 17, 46–47

developing codebooks, 172, 175–76, 183

complete, 64, 94, 119 Data in Complex Designs, 158 data integration, 91, 96, 102, 106,

developing deliverables, 187 developing deliverables for mixed methods projects, 187

109–10, 126, 130, 135, 138–39, 147,

developing diagrams, 168

152, 156–57, 167

development, xiii, 9, 39, 101–2, 105–6,

data integration steps, 65 data management, xvi, 16

110, 137 diagrams, 66, 68, 93, 122–23, 131,

data mining, 107, 141

136, 148, 165–66, 168–76, 183,

data phases, 29, 129, 136

195–96, 204, 206

data reduction, 194–95

procedural, 169, 174, 182

data repositories, xiii, 45–47, 180, 202

discrepancy, 130, 134–35, 156–57

data review, 173

disparate data, 123, 201

Data Service, 47

dissertation research, 32, 163

data sets, xvi–xvii, 37, 47, 61, 68, 75, 102

dissertations, xvii–xix, 32, 34, 163

data source

documentation, 39, 43, 165

first, 86, 94 second, 62, 86, 94 data sources, xvi–xix, 10, 28–29, 35,

documents, 11, 13, 39, 43, 50–51, 164–67, 170–73, 181, 183, 186, 198, 201, 203 Donald, xxii, 163, 177

37–38, 42–43, 45–46, 49–55, 59–60, 62–66, 68–69, 79–80, 82, 85–87, 96–99, 101, 109–13, 125– 26, 128–29, 133–37 perfect, 51

early intervention referral process, 75–76 early-stage and active writing for mixed methods, 164–65, 167–83

data sources for convergent designs, 66

early stages, 45, 164–70, 183

data timeliness, 46

early stages of mixed methods, 166

data transformation, 66–67, 87,

education, xix, 12–13, 28, 47–48, 97,

194, 203

105, 152–53, 155

DEAS, 137–39

EHRs (electronic health records), 101–2

deductive reasoning, 30–31, 203

electronic health records. See EHRs

Deferred Action for Childhood

evaluating, 26–27, 35, 37, 40, 43, 45,

Arrivals (DACA), 33, 163

51, 104, 126, 146, 190–91

deliverables, 186–89, 191 depression, 70, 109, 197 depth, 16, 27–30, 35, 78, 92, 126, 132, 205 descriptive studies, 24–25, 203

Evaluating Secondary Data, 43–44, 50, 53, 172, 181 Evaluating Secondary Data for Mixed Methods, 39–55

222   Secondary Data in Mixed Methods Research evaluation

focus group data, 29, 100

mixed methods program, 146

focus group questionnaire, 100, 191

outcome, 149–51

focus groups, 24, 33, 49, 67, 71, 92,

evaluation criteria, 93–95, 118–19, 121,

104, 109–10, 115, 122, 178

127, 138, 147, 157, 169, 177, 182 evaluation questions, 19, 38, 43, 51, 53–56 evaluative criteria, 62, 76, 81, 84, 91, 102, 130, 151 existing data, xiii–xiv, xvii–xviii, 4–6, 10, 12, 27, 190, 194, 201–6 existing data for secondary purposes, xviii, 14, 22–23, 28–29, 31, 38, 40, 46, 190 existing data for secondary purposes in mixed methods, 22–23, 31

gender stereotypes, 67 General Social Survey, 53–55 GPAs (grade point averages), 130–31, 135 grade point averages (GPAs), 130–31, 135 groups, 10, 13, 24, 46, 51, 95–96, 107, 110, 136, 138–39, 143, 153–54, 181–82 secure, 137–39, 157 GSS, 53–55

Existing Data in Education and Social Research, 12

HMS, 104

existing data sources, 26–27, 42, 187, 205

human experience, 13, 30, 201, 206

explanatory level, 25, 203

human-generated data, 92, 96, 204

explanatory sequential design, 25, 32,

Hypothesis, 52, 139

55, 116–42, 146, 155, 158, 160, 169, 173, 190 traditional, 121 explanatory sequential design’s sample, 131 explanatory sequential design to gain, 116 explanatory sequential mixed methods, 156 exploratory and descriptive studies, 25 Exploratory Sequential Design, 90–113, 204 exploratory studies, 24, 203

ICPSR (Inter-university Consortium for Political and Social Research), 38–39, 47, 53 implementation, 149–51 improvements, 150–51 inclusion of existing data for secondary purposes, 29 individuals precarious, 138–39 secure, 137–39 inferences, 192, 195, 204 influential people, 153–54 information, 8–10, 13–14, 23–25,

Fetters, 27, 44, 66, 68, 145, 169, 187, 192, 195–96 findings, 26, 51–52, 65–66, 75, 77–78, 85–86, 95–98, 102–4, 115,

29–30, 45, 48–49, 64, 72–73, 78–79, 83–85, 87, 128, 132, 140–41, 165–67, 170–72, 186–87, 189–90, 192–95, 203–4

120–23, 136–39, 151–53, 155,

contact, 46

164–65, 186–87, 190, 192, 195,

descriptive, 30, 129, 134

198–99, 201–4

helpful, 45, 172, 176

Finley, 100–102

identifying, 11–12

Index  223

metadata, 38

keeping field notes, 172–73, 176, 183

preliminary, 63–64, 182

knowledge, 6, 8–10, 13–14, 16, 19,

inquiry, 6, 9, 15, 22, 24–25, 29, 43, 50, 52, 116, 123, 197, 203 scientific, 24–26, 198, 204 institutional data resources, 43 instrument development, 90–91,

24–25, 28, 32–33, 35, 106, 108, 151, 153, 156 knowledge-level continuum, 22–26, 29–31, 35–36, 42, 198, 202–4 Kuppermann, 82–84

100, 204 instruments, 76, 79, 82, 87, 90–91,

life cycle of research data, 38–39, 56, 204

93–94, 100–102, 104, 106–7,

limitations of secondary data sources, 99

110–11, 118, 120, 122–23

literature, xvii, xx, 145, 149, 159, 178–79,

instrument to measure, 100–101 integrating big data, 63

192, 197–98 logistics, 28, 83–84

Integrating Secondary Big Data, 95 integration, 65, 74–75, 80, 82–83,

machines, 92, 202, 204

85–86, 95–98, 109, 111, 120–25,

map, 97, 104, 111–13, 124, 181, 193

129–30, 134, 148–49, 153–56,

maximize project resources, 98

175–76, 194–95

measurement, 49–50, 176

intent, 5, 63–64, 82, 125, 127, 140, 144–45, 189, 191 interest, xiii–xv, xviii, 5, 8–9, 24–25, 27, 29, 43–44, 46, 49–51, 53–54, 59, 77–78, 124–25 population of, 69, 77–78

measures, 67, 73, 99–104, 106–7, 110–11, 118–19, 121, 127–31, 135–38, 141, 147, 194, 197 new, 74, 101 reliable, 197 measures and metrics, 67

interest in secondary data, 8

measures and metrics of big data, 67

Internet, xvi, 16–18, 45, 49

measures/instruments, 62, 91, 118, 151

interpretation, 5–6, 8, 13, 23, 26,

measures of rigor, 175, 194, 204

67–68, 80, 85–86, 172, 175–76,

medical records, 10–11, 204

190, 195, 203–5

mental health, 14, 43, 71, 104, 108–10

interpretation stage, 60–61, 65

mentor, 29, 89

Inter-university Consortium for Political

metadata, 38, 63, 204

and Social Research. See ICPSR interview data, 149–50 interview questionnaires, 73, 170 interviews, 13, 33, 49, 54, 75, 79, 81, 83, 97–99, 115, 117, 152, 155 Ivankova, xx, 5, 22, 66, 72, 90, 116, 144–45, 169

metadata catalog, 63, 205 methodological approach/theoretical framework, 147–48 methodological decisions, xviii, 32, 82, 187 methodology, 22–23, 49, 145, 198 methods, 21–23, 25–28, 31–32, 45, 76–77, 83–85, 95, 121, 125–27,

joint data display, 66, 68–69, 195–96, 204 journaling, 165–66, 170, 172–73, 176, 183

153–54, 165–66, 169–70, 172–73, 178–79, 190–92 available, 83–84 single, 31, 76

224   Secondary Data in Mixed Methods Research Methods Designs, xx, 174 methods research, 1, 85

mixed methods purposes, 8–9, 38, 46, 194

methods section, 193–94

mixed methods questions, 5, 128

middle-aged individuals, 137–39

mixed methods report, 194, 196, 198

mixed methods, xiii, xv–xvi, xviii–xxii,

mixed methods research, xiii–xx, xxii,

5, 8, 10, 18–56, 82, 85, 143–45,

4–56, 69, 129, 132–33, 153–55,

163–83, 186–99, 201, 203–6

170–71, 186, 190–91, 194, 202,

mixed methods approaches, xiii, 106, 152 mixed methods colleagues, 86 mixed methods community, 144 virtual, 182 mixed methods core design, 144–45 mixed methods crystallizes, 5

204, 206 mixed methods research, 89, 108, 111, 183 mixed methods research and secondary data, 170 Mixed Methods Research Series, xiii–xiv

mixed methods data, 82, 138

mixed methods research studies, xiii

mixed methods data analysis, 176

mixed methods scholars, 27, 144, 182

mixed methods design for researchers, 60 mixed methods designs, xiii, xv, xx,

traditional, 27 mixed methods studies, xiii, xvii– xviii, 26–27, 66, 69, 166–67,

25–26, 42, 60, 116, 125, 144–48,

170–71, 179, 183, 190–91, 193,

159, 171, 173, 190–91, 202–5

196, 199

mixed methods designs early-stage, 169

narratives, 155–57

sequential, 26, 35

National Comorbidity Survey (NCS), 15

mixed methods dissertation, xvii, xix, 31–32, 163, 177 mixed methods dissertation proposals, 167 Mixed Methods Evaluation, 147 mixed methods journey, xxi mixed methods literature, 145 mixed methods project, xvii–xviii, xx,

National Survey of American Life (NSAL), 15, 109–10 new data, 15, 68, 75, 77, 82, 101, 105, 119–21, 128–29, 150, 153, 171–72, 174–76, 187–88, 193 new data collection, 61, 94, 131, 170,191 new data collection efforts, 79, 98, 191

21, 26–27, 165, 168, 171, 173–77,

new data collection methods, 181

179, 181, 186–87, 190, 195

new data sources, 66, 73, 75, 80, 82,

mixed methods project and address, 196 mixed methods project and draft, 175 mixed methods projects, xvi, 36, 38,

121, 134, 169, 173, 176 new qualitative data, 73–74, 97, 103–6, 111, 113–14, 119–21, 124, 129–33, 150–51, 153

41, 55, 165–66, 168–70, 172–75,

new qualitative data source, 128

178–79, 181–83, 186–87, 190,

new qualitative data to supplement, 132

193, 196–97, 199

new qualitative instruments, 73, 104

completing, 33 mixed methods proposals, xviii, 167

new qualitative sample, 73, 103–4 new quantitative component, 99

Index  225

new quantitative data, 67, 77, 79, 87, 95, 99–100, 102, 112, 131–35, 141, 187 new quantitative data and secondary qualitative data, 131

possibilities of secondary qualitative data, 13 potential data sources, 38, 44 potential limitations for secondary qualitative data, 197

new quantitative sample, 78, 100, 132

power dynamics, 9, 19

non-spousal family support, 108–10

precarious circumstances, 137–38

NSAL (National Survey of American

prediction, 18, 95, 205

Life), 15, 109–10 nurses, 31, 101–2

preexisting data sources, xvii preexisting data sources to address, xvii preexisting knowledge, 23–24, 203–4

Onwuegbuzie & Combs, 66–67 operationalize, 25, 40–41, 50, 55, 99, 127, 205 organizations, 30, 150, 152–54

preparing secondary data, 63, 93, 121, 146–48 Preparing Secondary Data for Complex Designs, 146

original data, 4, 27, 45, 84, 98, 188, 205

presecondary data source, 62

original research, xviii, 16, 44–45,

primary data, 4–5, 10, 14, 18, 32, 40,

54–55, 191 original researchers, 63–65, 79, 82, 94–95, 120–21, 136, 191 outcome evaluation data, 149–51 outcomes data, 150, 173 outline, 14, 93, 164, 167, 169–70, 173, 181, 186, 189–90, 192–94, 199

44, 46, 49–50, 72, 77, 205 primary data collection projects, 16 primary researchers, 38, 84, 168 process, xxi–xxii, 14, 31–32, 35, 40–41, process evaluation, 149–51 process evaluation data, 150 process for evaluating secondary

pacing, 53, 61, 85–86, 93, 96, 119, 124 participants, 9, 13–15, 69–71, 81, 117, 131–32, 138–39, 155–58, 180–81, 189, 193, 197 participatory-social justice work, 148 patients, 11, 14, 33, 45, 80–81, 83–84, 96, 128, 132

data, 181 process of analyzing secondary qualitative data, 175 products, 154, 168, 177, 206 program, xxi, 15, 33, 129–31, 134–35, 146, 149–53, 180–81 after-school, 146

chemotherapy, 80–81

college access, 129–31, 134–35

female, 14, 83–84

preparatory, 129–31, 133–34

pediatricians, 75–76

regular admissions, 129, 134

perfect data sources in research, 51

program years, 149–51

planning, 72–73, 77–79, 92, 99, 165–66,

project, xx, 19, 21, 33–35, 50–53, 92,

168, 170–71, 174, 178, 183, 187 planning journal, 166–67, 169 Plano Clark, xx, 5, 22, 66, 90, 116, 144–45, 169

95, 143, 145, 164–68, 170–79, 181–83, 185–86, 190–91, 205–6 independent, 33–34 pilot, 149–51

population, 48, 51, 90–91, 128, 195, 204

project activities, 172, 179

positive aging, 137–38

project deliverables, 186–87, 189, 199

226   Secondary Data in Mixed Methods Research project deliverables for mixed methods, 186–87, 189 project deliverables for mixed methods projects, 199 project journal, 139, 164, 174

qualitative data components, 68, 103 qualitative data set, 101 qualitative data sources, 29–30, 61, 82, 136, 140, 167, 191 second, 120

project outcome reports, 199, 205

qualitative methodologies, 52

project purpose, 139

qualitative methods, 24, 90, 94, 97,

project purpose and address, 139

120–21, 127, 133, 157, 169, 194, 203

project tracking journal, 173

qualitative phase, 27, 139, 205

Propedéutico, 129–30, 134–35

Qualitative Project, 89, 92, 155–56

proposal, 39, 159, 166–68

qualitative research, 25, 29, 72, 194, 204

protocols, 9, 14, 50, 165–66, 169–71,

QUALITATIVE results, 90–91, 94–95,

176, 183, 188, 199, 201, 206 providers, 83–84

98, 102, 106, 110–11, 116, 118, 121, 126, 130, 135, 138

Public-use Data Analysis System, 5

QUALITATIVE samples, 103, 118, 128,

qualitative analysis, 27, 29, 52, 62, 69,

Quantitative, 33, 47–48, 62, 64, 76, 81,

130, 132, 135–36, 138, 191 76, 152, 158 qualitative component, 42, 61, 63,

84, 103, 106, 118–21, 135, 138, 147, 151

73, 93, 98, 100, 105–6, 119–20,

Quantitative and Qualitative Data, 135

123, 170

quantitative component, 42, 89, 98,

qualitative data, 26, 62–64, 66–70, 75–79, 82–85, 87, 91–92, 94–97,

116, 119–20, 204 quantitative data, xv–xviii, 16–18,

99–103, 105–6, 108, 113–14, 116,

26–27, 31–34, 46–49, 55–56,

118–26, 128–33, 135–37, 139–41,

60–62, 66–69, 72–73, 78–79,

151–53, 173–74, 202–4

90–98, 102–4, 106–8, 110–11,

analyzing, 26, 203 analyzing secondary, 175 collected new, 133 collecting, 90

113–14, 116, 118–21, 124–28, 138–41, 203–6 quantitative data collection instruments, 78, 100, 132

collecting new, 97, 113

quantitative data components, 191

de-identified, 13

quantitative data experience, 52

existing, 136

quantitative data for secondary

I, 167 included collecting new, 149 integrating new, 111 longitudinal, 47 single-method, 66 Qualitative Data Codebook, 176

purposes, xviii quantitative data source by sampling, 131 quantitative data source by sampling to answer, 131 quantitative data sources, 8, 35, 38,

qualitative data collection, 89, 153

40, 43–45, 53, 55, 107, 109, 111,

qualitative data collection

131, 136

instruments, 73, 128 qualitative data collection tools, 61, 74, 78, 104

closed-ended, 103 collecting primary, 31 evaluating secondary, 51

Index  227

first, 120

researchers, xiii–xv, xvii–xix, 6–7,

potential secondary, 169

9–10, 23–24, 28–30, 60, 67, 69,

single-method, 135

90, 104–5, 112, 116, 178–79,

quantitative measures, 42, 67, 73, 107, 194, 203, 206 quantitative phase, 27, 205 quantitative sample, 73, 76, 78, 81, 91, 103, 109, 128, 132 questions, xiii–xv, 8–9, 13–14, 23–24, 40, 43–46, 48–55, 73–74, 100–101,

189–91, 193–94, 197–98 mixed methods, 173, 198 research program, xv, 9, 16, 29, 50 original, 50 research projects, xv, 5, 8, 16, 21, 40, 92, 166, 169, 183, 195 funded, xvi, 17

104–5, 128–29, 140, 153, 155–56,

research proposals, 167–68, 185

192–98

research purposes, xvi, 10–12, 16–17,

attitudinal, 53–54 guiding, 193–96, 198–99 important, xx, 167

28, 67 research question, 4–6, 13–16, 27–29, 31–32, 40–45, 49–55, 59–65, 67–68, 76–79, 81–82, 84–86,

race, 96–97, 117, 122, 124, 140

93–100, 107–8, 110–13, 118–21,

racial ideologies, 105–6

127–28, 135–36, 138–40, 147–48,

random sampling, 72, 104 =recommendations, 150–52, 177, 189, 198–99, 206

192–97 fundamental, 4–5, 205

recruitment, 77, 180, 189, 206

guiding, 99, 187 research questions and variables, 50, 90

referrals, 75–76

research teams, xvi, xix, 8–9, 32,

registered nurses (RNs), 101–2 relationships, 25, 30, 42, 101, 104, 122, 140, 155, 157, 203 relevant examples of secondary data, 18 report, xx, 68, 70, 73, 89, 98, 152–54, 185–90, 192–94, 196, 198–99 participant, 187, 199 reporting, 174, 180, 186–87, 190–91, 199, 206 reporting mixed methods, 185–99 reports for mixed methods, xx report symptoms, 79, 81

34–35, 53, 55, 68, 164, 167, 170– 72, 177, 180–81 research topic, 5, 8, 23, 25, 28, 31, 42, 46, 68, 111–12, 117 resources, xv–xvii, 12, 27, 31, 35, 39, 42, 137–39, 169–70, 172, 176, 178, 180 psychological, 137–38 respective data sources, 53, 66 response rates, 44, 51, 54–55 responses, 6, 50–51, 53–55, 100, 180–81 results, 50–52, 60–63, 65–66, 68–69,

repurpose, 82

72, 76, 81–82, 84–87, 90–91,

research, 8–10, 14, 19, 23–25, 27–29,

94–99, 116, 118, 120–26, 129–31,

31–35, 48–49, 77–78, 101–2, 152–53, 166–67, 174, 177–78, 186–87, 192, 196–98, 202–4

151, 153–54, 188, 195–98, 203, 205–6 results complement, 66, 206

research data, 38–39, 56, 204

retrieve, 44–46, 53–55

research design, xix, 148, 159–60,

review, 10–11, 44–45, 53, 60–64,

165, 182

66–67, 73, 84, 91, 93–96, 99–100,

228   Secondary Data in Mixed Methods Research 118–20, 129–30, 135–36, 138, 147–48, 157, 170–71, 174–75, 177–79, 199 review case studies, 144 review case studies of complex mixed methods designs, 144 review criteria, 82, 119, 148, 191 Reviewed secondary QUALITATIVE data, 102, 110 Reviewed secondary QUANTITATIVE data, 106, 110 RNs (registered nurses), 101–2

scholars, xiii–xiv, 10, 12, 14, 16–17, 22, 25–26, 28–29, 44, 59, 183 schools, 12, 48, 152, 154 science, 4, 9, 11, 28–29, 102, 111, 197–98 science of mixed methods, 196, 198–99 science of mixed methods and secondary data sources, 196 science of secondary data, 3–19 Scientific Development of Secondary Data, 14, 16–17

role in mixed methods, 27

scientific developments, 4, 14, 16–19

role of non-spousal family support,

scientific developments in secondary

108, 110 role of secondary data, 32–33

data, 4, 17–19 sculptors, 24–25 secondary analysis, xiii, xvi, xviii, 4,

Sample Data Repositories, 47

6, 12, 30, 32, 48–49, 135–36, 191

sample data source, 53

secondary analysis of existing data, 12

Sample Mixed Methods Projects, 33

secondary analysis to address, 4, 203

Sample Reports, 187–89

Secondary Big Data, 67

Sample Research Question, 33

secondary data, xiii–xx, 3–19, 22–23,

samples, 11, 25, 49, 51, 69, 71–73, 77–78, 102–4, 106–8, 123, 132, 136–37, 158, 170, 191 original, 107 random, 128–30, 133–35, 158

26–27, 31–36, 38, 40–46, 49–53, 55–57, 59–199, 203, 205–6 analyzing, 7 located, 40–41 qualitative, 12, 203

Sample Secondary Data Source, 54

secondary data analysis, xv, 6–7, 27, 35

sampling, 72, 82, 87, 91, 99–100,

secondary data and documents, 191

103, 107, 118–20, 127, 131, 136,

secondary data and help, 193

189, 193

secondary data designs, 128

convenience, 72–73 quantitative results help inform, 122 sampling decisions, 136 sampling frame, 128 sampling plan, 62, 73, 77–78, 81–82, 99, 103–4, 107, 111, 123, 127–28, 132, 136 Sampling plans for explanatory sequential designs, 136 sampling procedures, 99, 103–4, 127, 131 sampling strategies, 44, 51, 54–55

secondary data for complex mixed methods design, 144 secondary data for knowledge acquisition, 4 secondary data for mixed methods studies, 14 secondary data for single-method studies, xvi secondary data for successful analysis, 192 secondary data for topic fit and alignment, 93–94, 119–20

Index  229

secondary data in complex designs, 144, 158 secondary data in mixed methods, xvi, xxii, 10, 18, 21–35, 174, 191, 193, 195, 198 secondary data in mixed methods research, xiii–xviii, xx, xxii, 4–56, 186, 202, 204, 206 secondary data in mixed methods studies, 193 secondary data in single-methods studies, 10 secondary data inventory, 167–68 secondary data of interest, 51 secondary data project, 199 secondary data review, 170

120–21, 124–25, 131–35, 137–41, 156–59, 203 secondary qualitative data collection tools, 73 secondary qualitative data concepts, 125 secondary qualitative data in Case Study, 114 secondary qualitative data in research studies, 12 secondary qualitative data in social science research, 12 secondary qualitative data on administrators, 78 secondary qualitative data source, xvii, 12–14, 29, 51, 61, 99, 109, 122, 125, 136, 139

Secondary Data Source Examples, 33

secondary qualitative instruments, 79

secondary data source guide, 125

secondary qualitative sample, 78,

secondary data source of interest, 125 secondary data sources, xvii–xviii, 8–10, 61–62, 73–75, 79–80, 82,

100, 132 secondary quantitative analysis, 30, 52 secondary quantitative data, 10, 12,

84–87, 96–99, 107–8, 112–13,

72–77, 84, 87, 90–91, 94–95,

124–25, 128–29, 140–41, 167, 169,

98–99, 103–7, 109, 111–13,

175–76, 181–82, 193–94, 196–97

118–20, 122–25, 127–31, 133–34,

integrating, 69 selected, 112 secondary data source to address, 85 secondary data source to answer, 140 secondary data to address, xviii, 5, 42, 113, 196 secondary data to answer, 140 secondary data to report, 198 secondary data types, 18 secondary focus group data, 68, 100, 108–9, 158, 198

137–38, 140–41, 149–52, 155–57, 169–70 secondary quantitative data and files, 98, 111 secondary quantitative data in Case Study, 87, 141 secondary quantitative data set, xvii,137 secondary quantitative data sources, xvii, 10–11, 51, 62, 104, 127–28, 197

secondary interview data, 84

secondary quantitative results, 122, 129

secondary purposes, xviii, 4–5, 7, 12,

secondary quantitative sample, 72–73,

14, 22–23, 28–29, 31, 38, 40, 42–43, 45–46, 49, 194–95

82, 103–4, 128 secondary results, 192, 195

secondary qualitative analysis, 12, 96

secondary samples, 77, 103, 136

secondary qualitative data, 12–13,

secondary sampling plans, 107, 136

29–30, 45–46, 50–51, 77–81, 83–84, 89–94, 97–102, 109–14,

secondary survey data, 105, 131, 133, 146, 178, 190

230   Secondary Data in Mixed Methods Research second type of explanatory sequential design, 131

team members, 52, 166, 170, 181 TEDS, 74–76

Selecting Secondary Data Sources, 41

teenage aspirations, 156–57

sequential design, 25, 27, 86–87,

Texas admissions program, 129–30,

90–100, 103–5, 107–8, 110–14,

134–35

116–17, 121–22, 125, 127–29,

text analytics, 126, 201

139–40, 158, 204–5

text data sets, 45, 202

services, 30, 75–76

theory application report, 187, 199, 206

Shannon, 149–50

tips for reporting mixed methods, 186,

single-method quantitative data, 66, 77 single-method studies, xvi, 10, 14, 65–66, 86, 112, 183, 206 SMART program, 149–52 Social Research, 12, 47 software, project management, 179

191–92 Tips for Writing Mixed Methods Reports, 192 tools, 22, 129–30, 135, 179, 198 online data analysis, 5 topic, xiv–xvi, 8–10, 16–17, 23–25,

Steinauer, 82–84

28–30, 34–35, 42–45, 48–50,

STEM students, 12

53–54, 69, 78, 85, 113, 178–79,

steps for preparing secondary qualitative, 93 steps for preparing secondary qualitative and quantitative data, 93 students, xix, xxi, 3, 13, 15, 33–34, 104, 129–31, 133–35, 146, 181

194–95, 197–99, 203–4 sensitive, 49 topic fit and alignment, 63–64, 93–94, 119–20, 124 training, xxi, 28, 52, 61, 89, 152–53, 170–71, 188, 206

disadvantaged, 129, 134

training materials, 31, 170

first-generation, 15

Treviño, 129, 131, 133

low-income, 129–31, 134–35 students work, 34

unintended consequences (UCs), 101–2

studies, xiv–xv, 12, 14, 16, 68–69, 79,

United States, 10–11, 15, 33, 48, 97,

82, 154–55, 157, 167, 170, 178, 195, 197, 203 study’s purpose, 65, 77, 136

123, 163 USACH (University of Santiago de Chile), 129–31, 133–35

survey data, 49, 77, 81, 129–30, 133, 135, 146 complex sample, 59 secondary Qualtrics, 124 symptom diary, 80–81

variables, 25, 27, 38–39, 42, 50–52, 54, 90, 95, 98, 104, 122, 167–68, 205–6 Verran, 100–102

symptoms, 24, 79, 81, 124 synchronous, 180–81

Watkins, xiii, xxii, 5, 25–26, 30–31,

teachers, 33, 77–78, 97, 146

Weaver-Hightower, 152–54

team, 33–36, 52–53, 61, 73, 75–76,

work plans, 182–83

63, 66, 69, 71, 108, 110

79, 105, 128, 140, 164, 170–72, 176–77, 179–80, 188–89, 191 team journal, 181

preliminary, 164, 203 Writing Mixed Methods, xx, 164–98 Writing Mixed Methods Reports, 192